-
Notifications
You must be signed in to change notification settings - Fork 29.8k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
fs: behaviour of readFile and writeFile with file descriptors #23433
Comments
Another option:
|
See also #22554 for various behaviour of |
As for a use case for |
@sam-github I included that option in the OP. @vsemozhetbyt I have included that issue also in the list of references. Let's start recording what we think would be the expected behaviour of |
I checked few other languages to see how they handle file descriptors. Python
Other languages which I checked were Ruby, Java, C, and C++. They all provide their own abstractions over file descriptors. Using file descriptors are not recommended, so not a lot of fd based functions are available in them. |
I think step zero should be to update the documentation to specify the current behavior. If we change that behavior, fine, but there should be no reason to not have correct docs while that is being discussed. Is there already a PR open to update the docs? If not, I'm happy to do it or let @thefourtheye or anyone else do it. |
Would it be a bit confusing to state 3 different behaviors in docs? Especially callback/sync fd variant (merging) vs promise fd variant (appending)? |
It will be a bit confusing, but hopefully less confusing than the current situation of having the same behavior but incorrectly documented. |
I'm for deprecating |
Adding |
I find all four proposed options acceptable (assuming Option 1 involves documenting the current behavior, which given the open PR, it seems like it does). |
(Although for option 3: If we throw, then we shouldn't do anything afterwards, right? So either it's not throwing and it's emitting a warning instead and then trying from the beginning of the file, or it's throwing an error and giving up? Or am I misunderstanding it?) |
I'm ok with options 1 and 4, with a slight preference for 1 based on the concern of breaking existing code. |
@Trott I updated Option 3 now. It either throws or works from the beginning of the file. |
So … one thing I can potentially see people doing is I’m definitely okay with the current behavior, i.e. option 1. |
@thefourtheye Oh, yes, I’m a fan of #23709. I just didn’t really see that as an option here, so I thought it falls under option 1? :) |
@addaleax I updated the option 1 now. |
I’m for option 1 as well. |
@thefourtheye Was option 1 updated to include changes from #23709 or did I just miss that the first time I looked at it. I think there should be an option were we leave functionality as is but update the documentation. That is what I thought 1 was originally. |
@mhdawson Oh yes, Option 1 was updated after @addaleax's comment. I'll land #23706 tomorrow (provided there are no suggestions by then), which documents the current behaviour. As it is anyway going to be landed, I didn't want to include that in the options. Just to be thorough, I have included that as Option 5 now. |
not sure if we're still voting on this one, I weighed in @ #23709, I'd prefer option 1. |
@ChALkeR This would be tricky, as there is no offset passed by the users. So, if we are to emit warnings, only option is to emit them if file descriptors are used with |
@thefourtheye What I described above is (1) + a warning about behaviour change emitted the first time That can't be from our JS api, but perhaps we could use lseek/_lseek internally and at least emit the warning on supported platforms (if not all platforms have that)? We shouldn't emit needless warnings -- writing to an fd with zero offset is fine, that behaviour won't change and doesn't need a warning. |
Seems to be consensus around option 1, or at least that's the closest to consensus? Is that correct? |
+1 on option 1 from my end. |
I prefer Option 1. |
I'd be OK with going the option 1 + warning route through a single major cycle for a pure option 1. I'm concerned about performance implications though, if we need to reach down in to do an lseek to figure out whether a warning is warranted, can we put that in the path in such a way as to not make reads slower? Since here's a bunch of layers of indirection between |
@ChALkeR Seek is not supported by libuv. @bnoordhuis's comment in this thread
kind of gives a reason why it is not included in libuv. Do you mean, using
|
This seems to be progressing to a conclusion. I'm going to remove the |
I think seek is needed #24923 |
@kghost The outcome of this issue is that |
@thefourtheye Sorry for the delay 😞 . After further thoughts about this, I withdraw my suggestion — the increased maintaining costs indeed would not pay off here. |
Out of the 18 TSC members (including @Trott), 12 have expressed their opinions in the table in the OP. With six abstentions and nine votes for Option 1, Option 1 wins. @nodejs/tsc please let me know, if there are any objections here. Thanks everyone for participating and driving this issue to closure. |
As per the decision in nodejs#23433, the `fs.writeFile` will always write from the current position if it is used with file descriptors. This patch updates it. Ref: nodejs#23709
As per the decision in #23433, the `fs.writeFile` will always write from the current position if it is used with file descriptors. This patch updates it. Ref: #23709 PR-URL: #25080 Reviewed-By: Daniel Bevenius <daniel.bevenius@gmail.com> Reviewed-By: Weijia Wang <starkwang@126.com> Reviewed-By: Matteo Collina <matteo.collina@gmail.com> Reviewed-By: Anto Aravinth <anto.aravinth.cse@gmail.com> Reviewed-By: Colin Ihrig <cjihrig@gmail.com> Reviewed-By: James M Snell <jasnell@gmail.com> Reviewed-By: Vse Mozhet Byt <vsemozhetbyt@gmail.com> Reviewed-By: Rod Vagg <rod@vagg.org> Reviewed-By: Trivikram Kamat <trivikr.dev@gmail.com>
As per the decision in nodejs#23433, the `fs.writeFile` will always write from the current position if it is used with file descriptors. This patch updates it. Ref: nodejs#23709 PR-URL: nodejs#25080 Reviewed-By: Daniel Bevenius <daniel.bevenius@gmail.com> Reviewed-By: Weijia Wang <starkwang@126.com> Reviewed-By: Matteo Collina <matteo.collina@gmail.com> Reviewed-By: Anto Aravinth <anto.aravinth.cse@gmail.com> Reviewed-By: Colin Ihrig <cjihrig@gmail.com> Reviewed-By: James M Snell <jasnell@gmail.com> Reviewed-By: Vse Mozhet Byt <vsemozhetbyt@gmail.com> Reviewed-By: Rod Vagg <rod@vagg.org> Reviewed-By: Trivikram Kamat <trivikr.dev@gmail.com>
This problem has been discussed multiple times without reaching any conclusion. That is why opening a separate issue to discuss.
Documented Expectations
As per our docs
readFile
is defined asand
writeFile
is defined asCurrent State of
readFile
&writeFile
:When a regular file path is passed:
The file is either read from the beginning or completely overwritten. This complies with the documentation.
When a file descriptor is passed:
readFile
The file is read from the current position in the file (the current position is maintained by the file descriptor itself).
readFile problem reproduction
writeFile
Writes from the beginning of the file, without replacing the contents of the file. For example, if the existing content in the file is
ABCD
and the newly written data is12
, then the actual contents of the file after both thewriteFile
s is12CD
.writeFile problem reproduction
These two behaviours deviate from the documented expectations.
Cases for changing the behaviour to work from the beginning
#9671 (It presents the case with
readFile
)Cases for keeping the current behaviour
Its intuitive for people to expect read/write from the current position till the end of the file. (For example, read a header from a file and then decide whether to read the rest of the file or not.)
Proposed courses of action
writeFile
also to write from the current file position just likereadFile
reads from the current file position, and be done with it. (semver-major)readFile
andwriteFile
to work from the beginning of the file. (semver-major)a. If the file descriptor is not seekable, an error will be thrown from libuv.
This is not an exhaustive list. Please feel free to propose more ideas.
PS: This comment by @sam-github summarises my personal expectations from the API.
X
X
X
X
X
X
X
X
X
X
+ warn *X
X
X
X
X
X
X
9
1
1
2
4
cc @nodejs/fs
Tagging @nodejs/tsc (Might be a bit early, but this has been discussed before. So tagging to get more inputs)
Tagging @seishun @vsemozhetbyt @sam-github @jorangreef @addaleax from the previous discussions.
If needed we can tag the collaborators to get their expectations.
Previous discussions on this topic happened in #9671, #10809, #10853, #13572,
#22554
The text was updated successfully, but these errors were encountered: