Home All Groups Group Topic Archive Search About

System.IO.FileSystemWatcher is missing alot of files.

Author
7 Feb 2006 12:22 PM
Philip Wagenaar
I have a Windows Form application that monitors a directory using the
System.IO.FileSystemWatcher.

On the event created I run a sub that sends an xml file over http to a
service.

When I copy 7000 files in the directory to be monitored only 123 are sent to
the service.

What might be going wrong?

More info:

The sending of the XML is more then just the http post. I also have to do
some stuff with xml first before sending it.

Author
7 Feb 2006 12:38 PM
Al Reid
Show quote Hide quote
"Philip Wagenaar" <philip.wagenaar@online.nospam> wrote in message news:1A09EA20-078F-4A5A-AAB7-B909F676D3F5@microsoft.com...
> I have a Windows Form application that monitors a directory using the
> System.IO.FileSystemWatcher.
>
> On the event created I run a sub that sends an xml file over http to a
> service.
>
> When I copy 7000 files in the directory to be monitored only 123 are sent to
> the service.
>
> What might be going wrong?
>
> More info:
>
> The sending of the XML is more then just the http post. I also have to do
> some stuff with xml first before sending it.

It's possible that you are missing events because it takes longer to process the files than it takes to generate the events and thus
some get lost.  I had a similar situation using the FileSystemWatcher and I solved it by placing the event info into a FIFO and had
a separate worker thread unloading the FIFO and processing the events.  This completely eliminated the problem.  Perhaps this
approach would be appropriate here as well.


--
Al Reid
Author
7 Feb 2006 12:46 PM
Philip Wagenaar
You mean working with threads?

Is it possible to post a sample?

Show quoteHide quote
"Al Reid" wrote:

> "Philip Wagenaar" <philip.wagenaar@online.nospam> wrote in message news:1A09EA20-078F-4A5A-AAB7-B909F676D3F5@microsoft.com...
> > I have a Windows Form application that monitors a directory using the
> > System.IO.FileSystemWatcher.
> >
> > On the event created I run a sub that sends an xml file over http to a
> > service.
> >
> > When I copy 7000 files in the directory to be monitored only 123 are sent to
> > the service.
> >
> > What might be going wrong?
> >
> > More info:
> >
> > The sending of the XML is more then just the http post. I also have to do
> > some stuff with xml first before sending it.
>
> It's possible that you are missing events because it takes longer to process the files than it takes to generate the events and thus
> some get lost.  I had a similar situation using the FileSystemWatcher and I solved it by placing the event info into a FIFO and had
> a separate worker thread unloading the FIFO and processing the events.  This completely eliminated the problem.  Perhaps this
> approach would be appropriate here as well.
>
>
> --
> Al Reid
>
>
>
Author
7 Feb 2006 12:52 PM
Al Reid
"Philip Wagenaar" <philip.wagenaar@online.nospam> wrote in message news:5FCD0C34-9C97-41C5-AC95-015D2A92A4BD@microsoft.com...
> You mean working with threads?
>
> Is it possible to post a sample?
>
> "Al Reid" wrote:
>

Yes, I mean working with threads.  If I get a chance, I'll see if I can't skinny down my working code and post the basics.

--
Al Reid

Show quoteHide quote
> >
> > It's possible that you are missing events because it takes longer to process the files than it takes to generate the events and
thus
> > some get lost.  I had a similar situation using the FileSystemWatcher and I solved it by placing the event info into a FIFO and
had
> > a separate worker thread unloading the FIFO and processing the events.  This completely eliminated the problem.  Perhaps this
> > approach would be appropriate here as well.
> >
> >
> > --
> > Al Reid
> >
> >
> >
Author
7 Feb 2006 1:30 PM
Al Reid
"Philip Wagenaar" <philip.wagenaar@online.nospam> wrote in message news:5FCD0C34-9C97-41C5-AC95-015D2A92A4BD@microsoft.com...
> You mean working with threads?
>
> Is it possible to post a sample?
>

I found my original proof of concept code.  It was implemented as a Windows Service.  Most of the concepts were obtained from Google
searches, etc.

=================================================

Imports System

Imports System.Threading

Module General

Public goFIFO As New Collections.Queue

Public gblnStopThreads As Boolean

Public gstrSourceDir As String

Public gstrDestinationDir As String

Public goLogger As New CLog

Public goEventlog As EventLog

Public gintIdle As Integer

Public gintInterval As Integer

End Module

-----------------------------

Imports System.Threading

Imports System.io

Public Class WorkerThread

Public Shared Sub ProcessFiles()

Dim e As Object

Dim strFilePath As String

Dim strDestDir As String

Do Until gblnStopThreads And (goFIFO.Count = 0)

Try

If goFIFO.Count > 0 Then

e = goFIFO.Dequeue

If TypeOf e Is RenamedEventArgs Then

Dim strOldName As String = CType(e, RenamedEventArgs).OldFullPath.Replace(gstrSourceDir, gstrDestinationDir)

strDestDir = CType(e, RenamedEventArgs).FullPath.Replace(gstrSourceDir, gstrDestinationDir)

Rename(strOldName, strDestDir)

ElseIf TypeOf e Is FileSystemEventArgs Then

strFilePath = CType(e, FileSystemEventArgs).FullPath

strDestDir = strFilePath.Replace(gstrSourceDir, gstrDestinationDir)

If (GetAttr(strFilePath) And FileAttribute.Directory) = FileAttribute.Directory Then

If Not Directory.Exists(strDestDir) Then

strFilePath = strFilePath

strDestDir = strFilePath.Replace(gstrSourceDir, gstrDestinationDir)

Directory.CreateDirectory(strDestDir)

End If

ElseIf New FileInfo(strDestDir).Directory.Exists = False Then

strFilePath = strFilePath

strDestDir = strFilePath.Replace(gstrSourceDir, gstrDestinationDir)

Directory.CreateDirectory(Path.GetDirectoryName(strDestDir))

End If

If Not (GetAttr(strFilePath) And FileAttribute.Directory) = FileAttribute.Directory Then

FileCopy(strFilePath, strDestDir)

End If

End If

Thread.Sleep(gintInterval)

Else

Thread.Sleep(gintIdle)

End If

Catch ex As Exception

goEventlog.WriteEntry(ex.Message, EventLogEntryType.Error)

End Try

Loop

End Sub

End Class



-----------------------------



Imports System.ServiceProcess

Imports System

Imports System.Threading

Imports System.io

Imports Microsoft.Win32

Public Class MirrorService

Inherits System.ServiceProcess.ServiceBase

Private WithEvents moWatcher As FileSystemWatcher

Protected Overrides Sub OnStart(ByVal args() As String)

Me.AutoLog = True

goEventlog = Me.EventLog()

Process.GetCurrentProcess().PriorityClass = ProcessPriorityClass.High

gstrSourceDir = "C:\WatcherTest\Source"

gstrDestinationDir = "F:\WatcherTest\Destination"

gintIdle = 5000

gintInterval = 100

Dim oWorker As New WorkerThread

Dim oWorkerThread As New Thread(New ThreadStart(AddressOf oWorker.ProcessFiles))

' Start the thread.

oWorkerThread.Start()

moWatcher = New FileSystemWatcher

moWatcher.InternalBufferSize = 16384

moWatcher.Path = gstrSourceDir

moWatcher.IncludeSubdirectories = True

moWatcher.EnableRaisingEvents = True

End Sub

Protected Overrides Sub OnStop()

gblnStopThreads = True

End Sub

Private Sub moWatcher_CreatedOrChanged(ByVal sender As Object, ByVal e As System.IO.FileSystemEventArgs) Handles moWatcher.Changed,
moWatcher.Created

Try

If e.ChangeType <> WatcherChangeTypes.Deleted Then

goFIFO.Enqueue(e)

End If

Catch ex As Exception

goEventlog.WriteEntry(ex.Message, EventLogEntryType.Error)

End Try

End Sub

Private Sub moWatcher_Renamed(ByVal sender As Object, ByVal e As System.IO.RenamedEventArgs) Handles moWatcher.Renamed

goFIFO.Enqueue(CType(e, RenamedEventArgs))

End Sub

End Class

===================================================



Sorry that the cut and paste lost the formatting.  I hope this helps.

==

Al Reid
Author
7 Feb 2006 1:38 PM
Philip Wagenaar
Thanks! This really helped me alot!

Show quoteHide quote
"Al Reid" wrote:

> "Philip Wagenaar" <philip.wagenaar@online.nospam> wrote in message news:5FCD0C34-9C97-41C5-AC95-015D2A92A4BD@microsoft.com...
> > You mean working with threads?
> >
> > Is it possible to post a sample?
> >
>
> I found my original proof of concept code.  It was implemented as a Windows Service.  Most of the concepts were obtained from Google
> searches, etc.
>
> =================================================
>
> Imports System
>
> Imports System.Threading
>
> Module General
>
> Public goFIFO As New Collections.Queue
>
> Public gblnStopThreads As Boolean
>
> Public gstrSourceDir As String
>
> Public gstrDestinationDir As String
>
> Public goLogger As New CLog
>
> Public goEventlog As EventLog
>
> Public gintIdle As Integer
>
> Public gintInterval As Integer
>
> End Module
>
> -----------------------------
>
> Imports System.Threading
>
> Imports System.io
>
> Public Class WorkerThread
>
> Public Shared Sub ProcessFiles()
>
> Dim e As Object
>
> Dim strFilePath As String
>
> Dim strDestDir As String
>
> Do Until gblnStopThreads And (goFIFO.Count = 0)
>
> Try
>
> If goFIFO.Count > 0 Then
>
> e = goFIFO.Dequeue
>
> If TypeOf e Is RenamedEventArgs Then
>
> Dim strOldName As String = CType(e, RenamedEventArgs).OldFullPath.Replace(gstrSourceDir, gstrDestinationDir)
>
> strDestDir = CType(e, RenamedEventArgs).FullPath.Replace(gstrSourceDir, gstrDestinationDir)
>
> Rename(strOldName, strDestDir)
>
> ElseIf TypeOf e Is FileSystemEventArgs Then
>
> strFilePath = CType(e, FileSystemEventArgs).FullPath
>
> strDestDir = strFilePath.Replace(gstrSourceDir, gstrDestinationDir)
>
> If (GetAttr(strFilePath) And FileAttribute.Directory) = FileAttribute.Directory Then
>
> If Not Directory.Exists(strDestDir) Then
>
> strFilePath = strFilePath
>
> strDestDir = strFilePath.Replace(gstrSourceDir, gstrDestinationDir)
>
> Directory.CreateDirectory(strDestDir)
>
> End If
>
> ElseIf New FileInfo(strDestDir).Directory.Exists = False Then
>
> strFilePath = strFilePath
>
> strDestDir = strFilePath.Replace(gstrSourceDir, gstrDestinationDir)
>
> Directory.CreateDirectory(Path.GetDirectoryName(strDestDir))
>
> End If
>
> If Not (GetAttr(strFilePath) And FileAttribute.Directory) = FileAttribute.Directory Then
>
> FileCopy(strFilePath, strDestDir)
>
> End If
>
> End If
>
> Thread.Sleep(gintInterval)
>
> Else
>
> Thread.Sleep(gintIdle)
>
> End If
>
> Catch ex As Exception
>
> goEventlog.WriteEntry(ex.Message, EventLogEntryType.Error)
>
> End Try
>
> Loop
>
> End Sub
>
> End Class
>
>
>
> -----------------------------
>
>
>
> Imports System.ServiceProcess
>
> Imports System
>
> Imports System.Threading
>
> Imports System.io
>
> Imports Microsoft.Win32
>
> Public Class MirrorService
>
> Inherits System.ServiceProcess.ServiceBase
>
> Private WithEvents moWatcher As FileSystemWatcher
>
> Protected Overrides Sub OnStart(ByVal args() As String)
>
> Me.AutoLog = True
>
> goEventlog = Me.EventLog()
>
> Process.GetCurrentProcess().PriorityClass = ProcessPriorityClass.High
>
> gstrSourceDir = "C:\WatcherTest\Source"
>
> gstrDestinationDir = "F:\WatcherTest\Destination"
>
> gintIdle = 5000
>
> gintInterval = 100
>
> Dim oWorker As New WorkerThread
>
> Dim oWorkerThread As New Thread(New ThreadStart(AddressOf oWorker.ProcessFiles))
>
> ' Start the thread.
>
> oWorkerThread.Start()
>
> moWatcher = New FileSystemWatcher
>
> moWatcher.InternalBufferSize = 16384
>
> moWatcher.Path = gstrSourceDir
>
> moWatcher.IncludeSubdirectories = True
>
> moWatcher.EnableRaisingEvents = True
>
> End Sub
>
> Protected Overrides Sub OnStop()
>
> gblnStopThreads = True
>
> End Sub
>
> Private Sub moWatcher_CreatedOrChanged(ByVal sender As Object, ByVal e As System.IO.FileSystemEventArgs) Handles moWatcher.Changed,
> moWatcher.Created
>
> Try
>
> If e.ChangeType <> WatcherChangeTypes.Deleted Then
>
> goFIFO.Enqueue(e)
>
> End If
>
> Catch ex As Exception
>
> goEventlog.WriteEntry(ex.Message, EventLogEntryType.Error)
>
> End Try
>
> End Sub
>
> Private Sub moWatcher_Renamed(ByVal sender As Object, ByVal e As System.IO.RenamedEventArgs) Handles moWatcher.Renamed
>
> goFIFO.Enqueue(CType(e, RenamedEventArgs))
>
> End Sub
>
> End Class
>
> ===================================================
>
>
>
> Sorry that the cut and paste lost the formatting.  I hope this helps.
>
> ==
>
> Al Reid
>
>
>