|
web
newsgroups
|
|||||||||||||||||||||||
|
|||||||||||||||||||||||
ThreadingI'm building a windows service which will monitor the file system for new
files, process the information in any new file that appears, and put that information into a database. My worry is that many files may appear (not necessarily at once, but close enough) and will fire the file system watcher events, opening up many threads. Is there a way I can limit the threads that open, yet still keep the new file information in some sort of queue to process when a thread is free? Is there a better approach so I wouldn't be drowing system resources and constantly opening and closing db connections? Thanks for any advice. -Jason > I'm building a windows service which will monitor the file system for new Others may disagree, but this is how I'd approach this problem.> files, process the information in any new file that appears, and put that > information into a database. My worry is that many files may appear (not > necessarily at once, but close enough) and will fire the file system watcher > events, opening up many threads. Is there a way I can limit the threads > that open, yet still keep the new file information in some sort of queue to > process when a thread is free? Is there a better approach so I wouldn't be > drowing system resources and constantly opening and closing db connections? I would have the main service thread and one worker thread. The main thread blocks on a timer and the worker thread blocks on file watching. The worker thread's only job is to set the timer's interval to a minimum value when it sees file activity of interest. The main thread processes some (or all, but I like some better) files in the directory when the timer wakes it up. As long as it does some work, it leaves the timer interval at minimum value. During periods of idleness, it raises the interval toward a ceiling. For example, the minimum could be 10 milliseconds, the maximum 10 seconds, and the raising operation could be to multiply by 10, so the values the interval takes on are 10, 100, 1000, and 10000 milliseconds. At transition from 1000 to 10000, the main thread can close the db connection. This design is resource friendly on several fronts. It also overcomes a problem with file watcher technology wherein some file watcher events can be lost if file activity happens fast enough. All the file watcher thread needs to do is see one file activity to wake up the main thread, and the main thread will eventually process everything it finds in the directory. In other words, the file watcher is used only as a trigger. All that matters is that it sees something, and specifically what files it sees is of no consequence. You will need to make access to the timer interval thread-safe. I guess I would use synclock, and I would have the worker thread read the interval and then update it only if it was unequal to the minimum value. I would create the timer on the main thread, so the worker probably needs to do thread marshalling to update the interval. This design does single thread db access. If you really need many threads performing concurrent db accesses, then this design is not for you. From your original post, it sounds like the db updates can happen in any order (ie roughly as files show up in the directory) and that db parallelism is not an issue. Other responders may suggest your original design with thread pooling. I don't like it, never have, never will, and I'm kind of obstinate about it. But in theory, it could work for you. It is just not my cup of tea.
Sorting Arraylist Of Objects
System Idle Process force cast of object to mytype... How to ensure DB changes complete? mouse freezes in Visual Studio CreateToolhelp32Snapshot Pop up dialog after Main Form displayed Datatable display help help Can't access controls programmatically when inside FormView control MenuItem vs. ToolStripMenuItem |
|||||||||||||||||||||||