On Windows fs.writeFile deletes all alternate data streams

[f0zi]

• This only affects Windows.
• Tested on node 6 and 8.
• Subsystem: fs

When calling fs.writeFile to write to a file that has NTFS alternate data streams all the streams are deleted.

To reproduce, first create a file with alternate data streams (e.g. from the command line):

echo File content > test.txt
echo Stream content > test.txt:stream

You can see the streams with dir /r.

Then, write to the file:

fs.writeFile("test.txt", "Hello there", err => err ? console.error(err) : console.log("Ok"))

Run dir /r to see if the data streams are still there.

Expected: data streams are there, ready for reading
Observed: data streams are gone

Trackback to this issue.

[richardlau]

CC @nodejs/platform-windows

[seishun]

This behavior is consistent with both CRT and Python.

Source.cpp

#include <stdio.h>

int main() {
		auto f = fopen("test.txt", "w");
		fputs("Hello there", f);
		fclose(f);
}

CRT

C:\Users\Nikolai\source\repos\Project1\Debug\test>dir /r
 Volume in drive C has no label.
 Volume Serial Number is 48D0-99F5

 Directory of C:\Users\Nikolai\source\repos\Project1\Debug\test

03/02/2018  17:39    <DIR>          .
03/02/2018  17:39    <DIR>          ..
03/02/2018  17:39                15 test.txt
                                 17 test.txt:stream:$DATA
               1 File(s)             15 bytes
               2 Dir(s)   8,012,750,848 bytes free

C:\Users\Nikolai\source\repos\Project1\Debug\test>..\Project1.exe

C:\Users\Nikolai\source\repos\Project1\Debug\test>dir /r
 Volume in drive C has no label.
 Volume Serial Number is 48D0-99F5

 Directory of C:\Users\Nikolai\source\repos\Project1\Debug\test

03/02/2018  17:39    <DIR>          .
03/02/2018  17:39    <DIR>          ..
03/02/2018  17:39                11 test.txt
               1 File(s)             11 bytes
               2 Dir(s)   7,966,208,000 bytes free

Python

C:\Users\Nikolai\source\repos\Project1\Debug\test>dir /r
 Volume in drive C has no label.
 Volume Serial Number is 48D0-99F5

 Directory of C:\Users\Nikolai\source\repos\Project1\Debug\test

03/02/2018  17:41    <DIR>          .
03/02/2018  17:41    <DIR>          ..
03/02/2018  17:41                15 test.txt
                                 17 test.txt:stream:$DATA
               1 File(s)             15 bytes
               2 Dir(s)   7,789,568,000 bytes free

C:\Users\Nikolai\source\repos\Project1\Debug\test>python
Python 2.7.11 (v2.7.11:6d1b6a68f775, Dec  5 2015, 20:32:19) [MSC v.1500 32 bit (Intel)] on win32
Type "help", "copyright", "credits" or "license" for more information.
>>> with open("test.txt", "w") as f: f.write("hello there")
...
>>> ^Z


C:\Users\Nikolai\source\repos\Project1\Debug\test>dir /r
 Volume in drive C has no label.
 Volume Serial Number is 48D0-99F5

 Directory of C:\Users\Nikolai\source\repos\Project1\Debug\test

03/02/2018  17:41    <DIR>          .
03/02/2018  17:41    <DIR>          ..
03/02/2018  17:41                11 test.txt
               1 File(s)             11 bytes
               2 Dir(s)   7,769,198,592 bytes free

So the behavior of the 'w' flag won't be changed.

To preserve data in alternate streams, consider using the 'r+' flag and ftruncate instead:

C:\Users\Nikolai\source\repos\Project1\Debug\test>node
> var fd = fs.openSync('test.txt', 'r+')
undefined
> fs.ftruncateSync(fd)
undefined
> fs.writeSync(fd, 'Hello there')
11
> fs.closeSync(fd)
undefined
>
(To exit, press ^C again or type .exit)
>

C:\Users\Nikolai\source\repos\Project1\Debug\test>dir /r
 Volume in drive C has no label.
 Volume Serial Number is 48D0-99F5

 Directory of C:\Users\Nikolai\source\repos\Project1\Debug\test

03/02/2018  17:41    <DIR>          .
03/02/2018  17:41    <DIR>          ..
03/02/2018  17:52                11 test.txt
                                 17 test.txt:stream:$DATA
               1 File(s)             11 bytes
               2 Dir(s)   6,506,446,848 bytes free

C:\Users\Nikolai\source\repos\Project1\Debug\test>cat test.txt
Hello there

[WSLUser]

Is this workaround effective in WSL land? Any Windows file editor (in my case and the case of a few others VS Code) resets the permissions back to 755 and chmod after saving isn't always practical or in some cases even possible. Since we now have the ability to preserve Linux permissions in Windows with build 17134+, we need a way to address this issue so we can maintain our permissions after saving a file.

[seishun]

Does WSL store Linux permissions in alternate data streams? If not, then this issue is unrelated.

[nickjj]

@seishun This stuff is above my pay grade but I am a VSCode user plagued by this issue.

I was Googling around to see how WSL saves its metadata and came across this blog post from Microsoft: https://blogs.msdn.microsoft.com/wsl/2016/06/15/wsl-file-system-support/.

The blog post mentions "In addition, if a file has any file capabilities, these are stored in an alternate data stream for the file". Keep in mind that blog post is 2 years old. Since the latest stable Windows release, these metadata attributes can be modified by users.

It sounds like they do use alternate data streams.

[WSLUser]

Yes it does. See the referenced issue OP as well as the OP for microsoft/WSL#3165. There are actual repro steps in the WSL issue.

[WSLUser]

And here's the blog to piggy back off the blog mentioned above that discusses the new behavior in Windows: https://blogs.msdn.microsoft.com/commandline/2018/01/12/chmod-chown-wsl-improvements/

[seishun]

If they are indeed stored in alternate data streams, then the workaround should work.

[nickjj]

What's the workaround and how would you apply it to VSCode Insiders that's installed today?

[WSLUser]

The workaround mentioned above. But this is a workaround. Are there any plans to actually support alternate datastreams vice using a workaround? Workaround ≠ fix

[seishun]

See #18549 (comment). It needs to be fixed on VS Code's side. I submitted a PR for that 3 months ago: microsoft/vscode#42899

[WSLUser]

Hmm I see. Wonder what caused it to fail.

[f0zi]

@seishun I have to say I'm thinking differently about that workaround now.

The reason I opened this issue is not that fs.openSync with "w" overwrites alternate data streams, it's that fs.writeFile overwrites alternate data streams. Maybe I should have mentioned that apart from the behavior in VS Code I also tripped over this in my own code where I was saving mails from an IMAP session where there is no guarantee weather I get the mail body (main stream) or the attributes (alternate stream) first.

I ended up writing an alternate fs.writeFile with your workaround.

I now think fs.writeFile should use "r+" and truncate instead.

If (as a user of fs) you really want to wipe a file with all its streams, I would suggest fs.unlink.

[seishun]

fs.writeFile defaults to 'w' per the documentation and it's unlikely to change. However, you can override it by passing the options object: https://nodejs.org/api/fs.html#fs_fs_writefile_file_data_options_callback

What I think might happen is deprecating the use of fs.writeFile without options if there is strong evidence of it causing unexpected behavior. But that would need to be discussed in a separate issue.

[f0zi]

Ok but can I pass an option that uses "r+" and truncates the file?

Consider the following code lines:

fs.writeFile("test.txt", "Hello there", err => err ? console.error(err) : console.log("Ok"))
fs.writeFile("test.txt:alt1", "Alternative one", err => err ? console.error(err) : console.log("Ok"))
fs.writeFile("test.txt:alt2", "Second stream", err => err ? console.error(err) : console.log("Ok"))

Right now, the only thing that decides the final result of the file is the order these commands are executed in. The behavior of writing the alternate streams differs from the one writing the main stream.

That means it is inconsistent any way you look at it. Either it fails to keep the alternate streams when you write the main stream or it fails to wipe all other streams when you write one of the alternate streams.

[f0zi]

I just tried, fs.writeFile("test.txt", "Hello there",{ flag: "r+" }, err => err ? console.error(err) : console.log("Ok")) does seem to do the right thing. This is probably what I will be using in my code.

Edit: And I have to take it back. The file or stream has to exist in order for "r+" to work.

I leave it to you guys to decide what a sensible default would be on Windows. @seishun, thanks a lot for your feedback, I really appreciate it.