fs: support special files in promises.readFile #21497

TimothyGu · 2018-06-24T05:18:43Z

A size of 0 may indicate that the file is a Linux special file, which may still have content when read(). Instead of returning early, use the same approach as fs.readFile():

node/lib/internal/fs/read_file_context.js

Lines 74 to 88 in 6ced651

    
           if (this.size === 0) { 
        
             buffer = this.buffer = Buffer.allocUnsafeSlow(kReadFileBufferLength); 
        
             offset = 0; 
        
             length = kReadFileBufferLength; 
        
           } else { 
        
             buffer = this.buffer; 
        
             offset = this.pos; 
        
             length = Math.min(kReadFileBufferLength, this.size - this.pos); 
        
           } 
        
           const req = new FSReqWrap(); 
        
           req.oncomplete = readFileAfterRead; 
        
           req.context = this; 
        
           read(this.fd, buffer, offset, length, -1, req);

Additionally, only the return code of 0 indicates end of file, and we should not use bytesRead !== chunkSize as a indication of EOF. Per POSIX:

The value returned may be less than nbyte if the number of bytes left in the file is less than nbyte, if the read() request was interrupted by a signal, or if the file is a pipe or FIFO or special file and has fewer than nbyte bytes immediately available for reading. For example, a read() from a file associated with a terminal may return one typed line of data.

Refs: http://pubs.opengroup.org/onlinepubs/9699919799/functions/read.html
Refs: https://groups.google.com/forum/#!topic/nodejs-dev/rxZ_RoH1Gn0
Fixes: #21331

Checklist

make -j4 test (UNIX), or vcbuild test (Windows) passes
tests and/or benchmarks are included
commit message follows commit guidelines

nodejs-github-bot · 2018-06-24T05:18:44Z

@TimothyGu build started: https://ci.nodejs.org/blue/organizations/jenkins/node-test-pull-request-lite-pipeline/detail/node-test-pull-request-lite-pipeline/139/pipeline

devsnek · 2018-06-24T05:22:44Z

lib/internal/fs/promises.js

  if (size > kMaxLength)
    throw new ERR_FS_FILE_TOO_LARGE(size);

  const chunks = [];
-  const chunkSize = Math.min(size, 16384);
-  let totalRead = 0;
+  const chunkSize = size === 0 ? 16384 : Math.min(size, 16384);


might wanna factor that number out into a constant

TimothyGu · 2018-06-24T16:31:15Z

CI: https://ci.nodejs.org/job/node-test-pull-request/15584/

joyeecheung · 2018-06-24T16:35:45Z

lib/internal/fs/promises.js

@@ -125,6 +125,10 @@ async function writeFileHandle(filehandle, data, options) {
  } while (remaining > 0);
 }

+// Note: This is different from kReadFileBufferLength used for non-promisified
+// fs.readFile.
+const kReadFileMaxChunkSize = 16384;


Out of curiosity, is there context for this?

Not really, it was there when I found it so I kept it that way. I just thought it was interesting to note.

I guess we'd have to ask @jasnell about it.

Fixes: nodejs#21331

TimothyGu · 2018-07-04T14:59:07Z

CI: https://ci.nodejs.org/job/node-test-pull-request/15726/ (linuxone passes after re-run)

PR-URL: #21497 Fixes: #21331 Refs: http://pubs.opengroup.org/onlinepubs/9699919799/functions/read.html Refs: https://groups.google.com/forum/#!topic/nodejs-dev/rxZ_RoH1Gn0 Reviewed-By: Gus Caplan <me@gus.host> Reviewed-By: Anna Henningsen <anna@addaleax.net> Reviewed-By: Trivikram Kamat <trivikr.dev@gmail.com> Reviewed-By: Colin Ihrig <cjihrig@gmail.com> Reviewed-By: Benjamin Gruenbaum <benjamingr@gmail.com>

TimothyGu · 2018-07-04T16:16:09Z

Landed in 5057dd4.

PR-URL: #21497 Fixes: #21331 Refs: http://pubs.opengroup.org/onlinepubs/9699919799/functions/read.html Refs: https://groups.google.com/forum/#!topic/nodejs-dev/rxZ_RoH1Gn0 Reviewed-By: Gus Caplan <me@gus.host> Reviewed-By: Anna Henningsen <anna@addaleax.net> Reviewed-By: Trivikram Kamat <trivikr.dev@gmail.com> Reviewed-By: Colin Ihrig <cjihrig@gmail.com> Reviewed-By: Benjamin Gruenbaum <benjamingr@gmail.com>

TimothyGu requested a review from jasnell June 24, 2018 05:18

nodejs-github-bot added the fs Issues and PRs related to the fs subsystem / file system. label Jun 24, 2018

devsnek approved these changes Jun 24, 2018

View reviewed changes

addaleax approved these changes Jun 24, 2018

View reviewed changes

joyeecheung reviewed Jun 24, 2018

View reviewed changes

trivikr approved these changes Jun 24, 2018

View reviewed changes

cjihrig approved these changes Jun 24, 2018

View reviewed changes

benjamingr approved these changes Jun 24, 2018

View reviewed changes

TimothyGu added 3 commits July 4, 2018 10:55

fs: support pseudofiles in promises.readFile

16233aa

Fixes: nodejs#21331

fixup! fs: support pseudofiles in promises.readFile

2eeb52b

fixup! fs: support pseudofiles in promises.readFile

f81f15c

TimothyGu force-pushed the fs-promise branch from c8ebfd5 to f81f15c Compare July 4, 2018 14:56

TimothyGu closed this Jul 4, 2018

TimothyGu deleted the fs-promise branch July 4, 2018 16:16

targos mentioned this pull request Jul 17, 2018

v10.7.0 proposal #21851

Merged

TimothyGu mentioned this pull request Aug 18, 2018

fs.promises.readFile returns 0 bytes from /dev/fd/NNN #22391

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

fs: support special files in promises.readFile #21497

fs: support special files in promises.readFile #21497

TimothyGu commented Jun 24, 2018 •

edited

Loading

nodejs-github-bot commented Jun 24, 2018

devsnek Jun 24, 2018

TimothyGu Jun 24, 2018

TimothyGu commented Jun 24, 2018

joyeecheung Jun 24, 2018

TimothyGu Jun 24, 2018

TimothyGu Jun 24, 2018

TimothyGu commented Jul 4, 2018 •

edited

Loading

TimothyGu commented Jul 4, 2018

	if (this.size === 0) {
	buffer = this.buffer = Buffer.allocUnsafeSlow(kReadFileBufferLength);
	offset = 0;
	length = kReadFileBufferLength;
	} else {
	buffer = this.buffer;
	offset = this.pos;
	length = Math.min(kReadFileBufferLength, this.size - this.pos);
	}

	const req = new FSReqWrap();
	req.oncomplete = readFileAfterRead;
	req.context = this;

	read(this.fd, buffer, offset, length, -1, req);

fs: support special files in promises.readFile #21497

fs: support special files in promises.readFile #21497

Conversation

TimothyGu commented Jun 24, 2018 • edited Loading

Checklist

nodejs-github-bot commented Jun 24, 2018

devsnek Jun 24, 2018

Choose a reason for hiding this comment

TimothyGu Jun 24, 2018

Choose a reason for hiding this comment

TimothyGu commented Jun 24, 2018

joyeecheung Jun 24, 2018

Choose a reason for hiding this comment

TimothyGu Jun 24, 2018

Choose a reason for hiding this comment

TimothyGu Jun 24, 2018

Choose a reason for hiding this comment

TimothyGu commented Jul 4, 2018 • edited Loading

TimothyGu commented Jul 4, 2018

TimothyGu commented Jun 24, 2018 •

edited

Loading

TimothyGu commented Jul 4, 2018 •

edited

Loading