Home All Groups Group Topic Archive Search About

System.IO.File.Copy + NTFS Streams + special ACLs

Author
26 May 2009 5:29 PM
Peter Gibbons
Hello,

when I try to copy a file with System.IO.File.Copy I get a
"System.IO.IOException: The operation completed successfully".

File summary information was added to the file with then Windows XP
Explorer file property page. It writes the information to an alternate
NTFS Data Stream.

Additionally the user only has "Create file" permission in the
destination directory because it is used for archiving purposes.

Is it possible to copy only the primary data stream of files with .Net
built in functions to work around this?


Regards,

Peter

Author
26 May 2009 9:11 PM
Mike
Peter Gibbons wrote:
> Hello,
>
> when I try to copy a file with System.IO.File.Copy I get a
> "System.IO.IOException: The operation completed successfully".

How are you copying the file?  That sounds like a OVERLAPPED I/O
(IOCP) completion status.  You are using some async copy method.

> File summary information was added to the file with then Windows XP
> Explorer file property page. It writes the information to an alternate
> NTFS Data Stream.
>
> Additionally the user only has "Create file" permission in the
> destination directory because it is used for archiving purposes.
>
> Is it possible to copy only the primary data stream of files with .Net
> built in functions to work around this?

Good question.

Since streams are supports at the RTL level, I wonder if using a "."
at the end of the file name will do this?  I would ask in the
microsoft.public.win32.programmer.kernel fora because I think you can
probably do this via the device file name.

One way for sure:

Create your own "COPYFILE" function opening the main file name,
creating a target and a loop to read/write X KBytes blocks. That for
sure will only give you the main stream.

That is what I do in our p-code compiler where the client has HTML
tagged code (ie, <SCRIPT language="WCBASIC"> and it saves the source
in a server side meta file (xxxx.wcx:wcc) but the main stream
(xxxx.wcx) is the compiled code.   When the RTE opens, copies to cache
and runs it. It only sees the compiled stream (xxxx.wcx), not the source.

--
Author
27 May 2009 6:31 AM
Peter Gibbons
Hello,

>> when I try to copy a file with System.IO.File.Copy I get a
>> "System.IO.IOException: The operation completed successfully".
>
> How are you copying the file?  That sounds like a OVERLAPPED I/O (IOCP)
> completion status.  You are using some async copy method.

I use System.IO.File.Copy. It doesn't support overlapped IO. The thrown
exception is documented as "An I/O error has occurred." It's weird and
misleading that it is presented as "The operation completed successfully".

Show quoteHide quote
>> File summary information was added to the file with then Windows XP
>> Explorer file property page. It writes the information to an alternate
>> NTFS Data Stream.
>>
>> Additionally the user only has "Create file" permission in the
>> destination directory because it is used for archiving purposes.
>>
>> Is it possible to copy only the primary data stream of files with .Net
>> built in functions to work around this?
>
> Good question.
>
> Since streams are supports at the RTL level, I wonder if using a "." at
> the end of the file name will do this?

Adding a . doesn't make a difference in this case.

> I would ask in the
> microsoft.public.win32.programmer.kernel fora because I think you can
> probably do this via the device file name.

I know it can be done with unmanages code / pinvoke but I'd like to
avoid it.

> One way for sure:
>
> Create your own "COPYFILE" function opening the main file name, creating
> a target and a loop to read/write X KBytes blocks. That for sure will
> only give you the main stream.
>
> That is what I do in our p-code compiler where the client has HTML
> tagged code (ie, <SCRIPT language="WCBASIC"> and it saves the source in
> a server side meta file (xxxx.wcx:wcc) but the main stream (xxxx.wcx) is
> the compiled code.   When the RTE opens, copies to cache and runs it. It
> only sees the compiled stream (xxxx.wcx), not the source.
>
> --

Regards,

Peter
Author
27 May 2009 9:00 AM
Mike
Peter Gibbons wrote:

> I use System.IO.File.Copy. It doesn't support overlapped IO. The thrown
> exception is documented as "An I/O error has occurred." It's weird and
> misleading that it is presented as "The operation completed successfully".

+1. Does seem odd.

>> Since streams are supports at the RTL level, I wonder if using a "."
>> at the end of the file name will do this?
>
> Adding a . doesn't make a difference in this case.

Right.

Rolling your own "CopyFile" function will work:

    Sub CopyFile(byval src as string, tar as string)
        Dim sr As New FileStream(src,FileMode.Open,FileAccess.Read)
        Dim sw As New FileStream(tar,FileMode.Create,FileAccess.Write)
        Dim b(4*1024) As Byte
        Do
          dim n as integer = sr.Read(b, 0, b.Length)
          if n <= 0 then exit do
          sw.write(b,0,n)
        Loop
        sw.Close()
        sr.Close()
    end sub

usage:

    try
      CopyFile("afs1.txt","afs2.txt")
    catch ex as exception
       WriteLine(ex.message)
       WriteLine(ex.stacktrace)
    end try

To create the test, I used notepad to create the AFS1.TXT file and the
meta stream

     NotePad afs1.txt

I typed junk lines and saved.

     NotePad afs1.txt:meta.  <--- NOTE THE ENDING DOT!

I typed junk lines and saved.

You can confirm that a straight DOS copy will copy the streams:

     copy afs1.txt junk.txt
     notepad junk.txt:meta.  <--- NOTE THE ENDING DOT!

You will see the meta data.

Then I ran the above applet to copy the file to afs2.txt, and tested
to see if the the meta stream was copied as well:

     NotePad afs2.txt:meta.  <--- NOTE THE ENDING DOT!

Notepad asked if you want to create it. :-)

Hope this provides you the solution you seek.

--
Author
27 May 2009 5:23 PM
Peter Gibbons
Hello Mike,

thanks for the example code. I changed it a little to my needs.
I'd like to preserve the LastWriteTime like the normal copy functions
but I get a "System.UnauthorizedAccessException" due to the NTFS
Permissions because the user only has "Create file" permission in the
destination directory.


   Sub CopyFileWithoutStreams(ByVal sourceFileName As String, ByVal
destFileName As String, Optional ByVal buffersize As Integer = &HFFFFUI)
     Dim sr As New System.IO.FileStream(sourceFileName,
IO.FileMode.Open, IO.FileAccess.Read, IO.FileShare.Read, buffersize,
IO.FileOptions.SequentialScan)
     Dim sw As New System.IO.FileStream(destFileName,
IO.FileMode.CreateNew, IO.FileAccess.Write, IO.FileShare.None,
buffersize, IO.FileOptions.SequentialScan Or IO.FileOptions.WriteThrough)
     Dim b(buffersize) As Byte
     Dim n As Integer
     Do
       n = sr.Read(b, 0, buffersize)
       If n <= 0 Then Exit Do
       sw.Write(b, 0, n)
     Loop
     sw.Close()
     sr.Close()
     System.IO.File.SetLastWriteTimeUtc(destFileName,
System.IO.File.GetLastWriteTimeUtc(sourceFileName))
   End Sub

Regards,

Peter


Mike wrote:
Show quoteHide quote
> Peter Gibbons wrote:
>
>> I use System.IO.File.Copy. It doesn't support overlapped IO. The
>> thrown exception is documented as "An I/O error has occurred." It's
>> weird and misleading that it is presented as "The operation completed
>> successfully".
>
> +1. Does seem odd.
>
>>> Since streams are supports at the RTL level, I wonder if using a "."
>>> at the end of the file name will do this?
>>
>> Adding a . doesn't make a difference in this case.
>
> Right.
>
> Rolling your own "CopyFile" function will work:
>
>    Sub CopyFile(byval src as string, tar as string)
>        Dim sr As New FileStream(src,FileMode.Open,FileAccess.Read)
>        Dim sw As New FileStream(tar,FileMode.Create,FileAccess.Write)
>        Dim b(4*1024) As Byte
>        Do
>          dim n as integer = sr.Read(b, 0, b.Length)
>          if n <= 0 then exit do
>          sw.write(b,0,n)
>        Loop
>        sw.Close()
>        sr.Close()
>    end sub
>
> usage:
>
>    try
>      CopyFile("afs1.txt","afs2.txt")
>    catch ex as exception
>       WriteLine(ex.message)
>       WriteLine(ex.stacktrace)
>    end try
>
> To create the test, I used notepad to create the AFS1.TXT file and the
> meta stream
>
>     NotePad afs1.txt
>
> I typed junk lines and saved.
>
>     NotePad afs1.txt:meta.  <--- NOTE THE ENDING DOT!
>
> I typed junk lines and saved.
>
> You can confirm that a straight DOS copy will copy the streams:
>
>     copy afs1.txt junk.txt
>     notepad junk.txt:meta.  <--- NOTE THE ENDING DOT!
>
> You will see the meta data.
>
> Then I ran the above applet to copy the file to afs2.txt, and tested to
> see if the the meta stream was copied as well:
>
>     NotePad afs2.txt:meta.  <--- NOTE THE ENDING DOT!
>
> Notepad asked if you want to create it. :-)
>
> Hope this provides you the solution you seek.
>
> --
Author
27 May 2009 9:10 PM
Mike
Ok, the user doesn't have "Modify" access? or Write Attributes?

So how are you going to handle that?  Impersonate a higher level account?

If this is a system application, not just a end-user application, it
can make sense to raise the process credentials so you can do your
system work.

--



Peter Gibbons wrote:
Show quoteHide quote
> Hello Mike,
>
> thanks for the example code. I changed it a little to my needs.
> I'd like to preserve the LastWriteTime like the normal copy functions
> but I get a "System.UnauthorizedAccessException" due to the NTFS
> Permissions because the user only has "Create file" permission in the
> destination directory.
>
>
>   Sub CopyFileWithoutStreams(ByVal sourceFileName As String, ByVal
> destFileName As String, Optional ByVal buffersize As Integer = &HFFFFUI)
>     Dim sr As New System.IO.FileStream(sourceFileName, IO.FileMode.Open,
> IO.FileAccess.Read, IO.FileShare.Read, buffersize,
> IO.FileOptions.SequentialScan)
>     Dim sw As New System.IO.FileStream(destFileName,
> IO.FileMode.CreateNew, IO.FileAccess.Write, IO.FileShare.None,
> buffersize, IO.FileOptions.SequentialScan Or IO.FileOptions.WriteThrough)
>     Dim b(buffersize) As Byte
>     Dim n As Integer
>     Do
>       n = sr.Read(b, 0, buffersize)
>       If n <= 0 Then Exit Do
>       sw.Write(b, 0, n)
>     Loop
>     sw.Close()
>     sr.Close()
>     System.IO.File.SetLastWriteTimeUtc(destFileName,
> System.IO.File.GetLastWriteTimeUtc(sourceFileName))
>   End Sub
>
> Regards,
>
> Peter
>
>
> Mike wrote:
>> Peter Gibbons wrote:
>>
>>> I use System.IO.File.Copy. It doesn't support overlapped IO. The
>>> thrown exception is documented as "An I/O error has occurred." It's
>>> weird and misleading that it is presented as "The operation completed
>>> successfully".
>>
>> +1. Does seem odd.
>>
>>>> Since streams are supports at the RTL level, I wonder if using a "."
>>>> at the end of the file name will do this?
>>>
>>> Adding a . doesn't make a difference in this case.
>>
>> Right.
>>
>> Rolling your own "CopyFile" function will work:
>>
>>    Sub CopyFile(byval src as string, tar as string)
>>        Dim sr As New FileStream(src,FileMode.Open,FileAccess.Read)
>>        Dim sw As New FileStream(tar,FileMode.Create,FileAccess.Write)
>>        Dim b(4*1024) As Byte
>>        Do
>>          dim n as integer = sr.Read(b, 0, b.Length)
>>          if n <= 0 then exit do
>>          sw.write(b,0,n)
>>        Loop
>>        sw.Close()
>>        sr.Close()
>>    end sub
>>
>> usage:
>>
>>    try
>>      CopyFile("afs1.txt","afs2.txt")
>>    catch ex as exception
>>       WriteLine(ex.message)
>>       WriteLine(ex.stacktrace)
>>    end try
>>
>> To create the test, I used notepad to create the AFS1.TXT file and the
>> meta stream
>>
>>     NotePad afs1.txt
>>
>> I typed junk lines and saved.
>>
>>     NotePad afs1.txt:meta.  <--- NOTE THE ENDING DOT!
>>
>> I typed junk lines and saved.
>>
>> You can confirm that a straight DOS copy will copy the streams:
>>
>>     copy afs1.txt junk.txt
>>     notepad junk.txt:meta.  <--- NOTE THE ENDING DOT!
>>
>> You will see the meta data.
>>
>> Then I ran the above applet to copy the file to afs2.txt, and tested
>> to see if the the meta stream was copied as well:
>>
>>     NotePad afs2.txt:meta.  <--- NOTE THE ENDING DOT!
>>
>> Notepad asked if you want to create it. :-)
>>
>> Hope this provides you the solution you seek.
>>
>> --
Author
28 May 2009 7:01 PM
Peter Gibbons
Hello Mike,

the applikation runs as a service under a least privilege account that
only has the "create file" permission in the archive directory, not
more, not less. Elevation isn't possible and shouldn't be neccessary.
Files that don't have NTFS alternate data streams can be copied without
problems. The LastWriteTime is also kept.

Regards,

Peter

Mike wrote:
Show quoteHide quote
> Ok, the user doesn't have "Modify" access? or Write Attributes?
>
> So how are you going to handle that?  Impersonate a higher level account?
>
> If this is a system application, not just a end-user application, it can
> make sense to raise the process credentials so you can do your system work.
>
> Troubleshooting Exceptions: System.UnauthorizedAccessException
> http://msdn.microsoft.com/en-us/library/18b8kx07.aspx
>
> Code Access Security Basics
> http://msdn.microsoft.com/en-us/library/33tceax8.aspx
>
> See the Secure Class Libraries and Requesting permission links. > --
>
>
>
> Peter Gibbons wrote:
>> Hello Mike,
>>
>> thanks for the example code. I changed it a little to my needs.
>> I'd like to preserve the LastWriteTime like the normal copy functions
>> but I get a "System.UnauthorizedAccessException" due to the NTFS
>> Permissions because the user only has "Create file" permission in the
>> destination directory.
>>
>>
>>   Sub CopyFileWithoutStreams(ByVal sourceFileName As String, ByVal
>> destFileName As String, Optional ByVal buffersize As Integer = &HFFFFUI)
>>     Dim sr As New System.IO.FileStream(sourceFileName,
>> IO.FileMode.Open, IO.FileAccess.Read, IO.FileShare.Read, buffersize,
>> IO.FileOptions.SequentialScan)
>>     Dim sw As New System.IO.FileStream(destFileName,
>> IO.FileMode.CreateNew, IO.FileAccess.Write, IO.FileShare.None,
>> buffersize, IO.FileOptions.SequentialScan Or IO.FileOptions.WriteThrough)
>>     Dim b(buffersize) As Byte
>>     Dim n As Integer
>>     Do
>>       n = sr.Read(b, 0, buffersize)
>>       If n <= 0 Then Exit Do
>>       sw.Write(b, 0, n)
>>     Loop
>>     sw.Close()
>>     sr.Close()
>>     System.IO.File.SetLastWriteTimeUtc(destFileName,
>> System.IO.File.GetLastWriteTimeUtc(sourceFileName))
>>   End Sub
>>
>> Regards,
>>
>> Peter
>>
>>
>> Mike wrote:
>>> Peter Gibbons wrote:
>>>
>>>> I use System.IO.File.Copy. It doesn't support overlapped IO. The
>>>> thrown exception is documented as "An I/O error has occurred." It's
>>>> weird and misleading that it is presented as "The operation
>>>> completed successfully".
>>>
>>> +1. Does seem odd.
>>>
>>>>> Since streams are supports at the RTL level, I wonder if using a
>>>>> "." at the end of the file name will do this?
>>>>
>>>> Adding a . doesn't make a difference in this case.
>>>
>>> Right.
>>>
>>> Rolling your own "CopyFile" function will work:
>>>
>>>    Sub CopyFile(byval src as string, tar as string)
>>>        Dim sr As New FileStream(src,FileMode.Open,FileAccess.Read)
>>>        Dim sw As New FileStream(tar,FileMode.Create,FileAccess.Write)
>>>        Dim b(4*1024) As Byte
>>>        Do
>>>          dim n as integer = sr.Read(b, 0, b.Length)
>>>          if n <= 0 then exit do
>>>          sw.write(b,0,n)
>>>        Loop
>>>        sw.Close()
>>>        sr.Close()
>>>    end sub
>>>
>>> usage:
>>>
>>>    try
>>>      CopyFile("afs1.txt","afs2.txt")
>>>    catch ex as exception
>>>       WriteLine(ex.message)
>>>       WriteLine(ex.stacktrace)
>>>    end try
>>>
>>> To create the test, I used notepad to create the AFS1.TXT file and
>>> the meta stream
>>>
>>>     NotePad afs1.txt
>>>
>>> I typed junk lines and saved.
>>>
>>>     NotePad afs1.txt:meta.  <--- NOTE THE ENDING DOT!
>>>
>>> I typed junk lines and saved.
>>>
>>> You can confirm that a straight DOS copy will copy the streams:
>>>
>>>     copy afs1.txt junk.txt
>>>     notepad junk.txt:meta.  <--- NOTE THE ENDING DOT!
>>>
>>> You will see the meta data.
>>>
>>> Then I ran the above applet to copy the file to afs2.txt, and tested
>>> to see if the the meta stream was copied as well:
>>>
>>>     NotePad afs2.txt:meta.  <--- NOTE THE ENDING DOT!
>>>
>>> Notepad asked if you want to create it. :-)
>>>
>>> Hope this provides you the solution you seek.
>>>
>>> --
Author
27 May 2009 10:43 PM
Mike
Pete,

Check these out:

Troubleshooting Exceptions: System.UnauthorizedAccessException
http://msdn.microsoft.com/en-us/library/18b8kx07.aspx

Code Access Security Basics
http://msdn.microsoft.com/en-us/library/33tceax8.aspx

See the Secure Class Libraries and Requesting permission links.

--

Peter Gibbons wrote:
Show quoteHide quote
> Hello Mike,
>
> thanks for the example code. I changed it a little to my needs.
> I'd like to preserve the LastWriteTime like the normal copy functions
> but I get a "System.UnauthorizedAccessException" due to the NTFS
> Permissions because the user only has "Create file" permission in the
> destination directory.
>
>
>   Sub CopyFileWithoutStreams(ByVal sourceFileName As String, ByVal
> destFileName As String, Optional ByVal buffersize As Integer = &HFFFFUI)
>     Dim sr As New System.IO.FileStream(sourceFileName, IO.FileMode.Open,
> IO.FileAccess.Read, IO.FileShare.Read, buffersize,
> IO.FileOptions.SequentialScan)
>     Dim sw As New System.IO.FileStream(destFileName,
> IO.FileMode.CreateNew, IO.FileAccess.Write, IO.FileShare.None,
> buffersize, IO.FileOptions.SequentialScan Or IO.FileOptions.WriteThrough)
>     Dim b(buffersize) As Byte
>     Dim n As Integer
>     Do
>       n = sr.Read(b, 0, buffersize)
>       If n <= 0 Then Exit Do
>       sw.Write(b, 0, n)
>     Loop
>     sw.Close()
>     sr.Close()
>     System.IO.File.SetLastWriteTimeUtc(destFileName,
> System.IO.File.GetLastWriteTimeUtc(sourceFileName))
>   End Sub
>
> Regards,
>
> Peter
>
>
> Mike wrote:
>> Peter Gibbons wrote:
>>
>>> I use System.IO.File.Copy. It doesn't support overlapped IO. The
>>> thrown exception is documented as "An I/O error has occurred." It's
>>> weird and misleading that it is presented as "The operation completed
>>> successfully".
>>
>> +1. Does seem odd.
>>
>>>> Since streams are supports at the RTL level, I wonder if using a "."
>>>> at the end of the file name will do this?
>>>
>>> Adding a . doesn't make a difference in this case.
>>
>> Right.
>>
>> Rolling your own "CopyFile" function will work:
>>
>>    Sub CopyFile(byval src as string, tar as string)
>>        Dim sr As New FileStream(src,FileMode.Open,FileAccess.Read)
>>        Dim sw As New FileStream(tar,FileMode.Create,FileAccess.Write)
>>        Dim b(4*1024) As Byte
>>        Do
>>          dim n as integer = sr.Read(b, 0, b.Length)
>>          if n <= 0 then exit do
>>          sw.write(b,0,n)
>>        Loop
>>        sw.Close()
>>        sr.Close()
>>    end sub
>>
>> usage:
>>
>>    try
>>      CopyFile("afs1.txt","afs2.txt")
>>    catch ex as exception
>>       WriteLine(ex.message)
>>       WriteLine(ex.stacktrace)
>>    end try
>>
>> To create the test, I used notepad to create the AFS1.TXT file and the
>> meta stream
>>
>>     NotePad afs1.txt
>>
>> I typed junk lines and saved.
>>
>>     NotePad afs1.txt:meta.  <--- NOTE THE ENDING DOT!
>>
>> I typed junk lines and saved.
>>
>> You can confirm that a straight DOS copy will copy the streams:
>>
>>     copy afs1.txt junk.txt
>>     notepad junk.txt:meta.  <--- NOTE THE ENDING DOT!
>>
>> You will see the meta data.
>>
>> Then I ran the above applet to copy the file to afs2.txt, and tested
>> to see if the the meta stream was copied as well:
>>
>>     NotePad afs2.txt:meta.  <--- NOTE THE ENDING DOT!
>>
>> Notepad asked if you want to create it. :-)
>>
>> Hope this provides you the solution you seek.
>>
>> --