Download.dataset partially downloads ( corrupts) files?

[travatine]

Hi,

Download.dataset() uses a "fs.createWriteStream()" to create a writable stream.
The writeablestream end() function is not explicitly being called.

For small files ( less than 64K), this results in incomplete files / data loss / corruption.

 public static async dataSet(session: AbstractSession, dataSetName: string, options: IDownloadOptions = {}): Promise<IZosFilesResponse> {
...
 const writeStream = IO.createWriteStream(destination);

            // Use specific options to mimic ZosmfRestClient.getStreamed()
            const requestOptions: IOptionsFullResponse = {
                resource: endpoint,
                reqHeaders,
                responseStream: writeStream,
                normalizeResponseNewLines: !options.binary,
                task: options.task
            };

            // If requestor needs etag, add header + get "response" back
            if (options.returnEtag) {
                requestOptions.reqHeaders.push(ZosmfHeaders.X_IBM_RETURN_ETAG);
                requestOptions.dataToReturn = [CLIENT_PROPERTY.response];
            }

            const request: IRestClientResponse = await ZosmfRestClient.getExpectFullResponse(session, requestOptions);

Might it possible for "end()" to be called on "writeStream" , after the "getExpectFullResponse" call, so the stream data is written to the file immediately?

Thanks

[github-actions]

Thank you for creating a bug report. If you haven't already, please ensure you have provided steps to reproduce it and as much context as possible.

[Shofiya2003]

Hello
I would like to work on this issue. Would suggest a solution as soon as possible.
@zFernand0

[travatine]

Hi @Shofiya2003 ,

For some additional context,
I was trying to download a dataset , using Download.dataset(),
Then immediately reading the local file.

The problem was that, the download.datset() function returns before the data is actually written to the file. (Because end() was not being called?)

This is similar to the problem described in ,
https://stackoverflow.com/questions/61913033/node-js-closing-a-file-after-writing .

Adding a call to end() on the write stream, and waiting for the finish event might help:
E.g.

writeStream.end();
await once(writeStream, 'finish');