GOT not able to handle Passthrough streams #2111
Comments
|
got version Form data Library |
|
Can you try https://github.com/octet-stream/form-data instead? |
|
Hi @szmarczak fd.set("file", {
type: "text/plain",
name: "upload.txt",
[Symbol.toStringTag]: "File",
stream() {
return getFileStream();
}
});const encoder = new FormDataEncoder(fd);
console.log('Length', encoder.contentLength);
fd.set("file", new BlobFromStream(stream, encoder.contentLength), 'upload.txt');The above two ways are taken from new fromdata-node library itself |
|
After some debugging, I found that Old FormData library is unable to calculate content length when I am trying to pipe stream with passthrough |
|
Can you please send a full example with the new |
|
Hi @szmarczak Please find the full example below /**
* App that will upload the file to the server
*/
import express from 'express';
import { createReadStream } from "node:fs";
import { dirname, join, resolve } from 'node:path';
import { PassThrough, pipeline, Readable, Transform } from "node:stream";
import { got } from 'got';
import { fileURLToPath } from 'node:url';
import cors from 'cors';
import { FormData } from 'formdata-node'
import { FormDataEncoder } from "form-data-encoder"
import axios from 'axios';
const __filename = fileURLToPath(import.meta.url);
const __dirname = dirname(__filename);
const app = express();
const filePath = resolve(join(__dirname,'../assets/upload.txt'));
app.use(cors());
const getFileStream = () => {
const fileStream = createReadStream(filePath);
const passThrough = new PassThrough();
// fileStream.pipe(passThrough);
// return passThrough;
return fileStream;
};
app.get('/', (req, res) => {
res.send('Hello World!');
});
app.post('/sendFile', async (req, res) => {
try {
const fd = new FormData();
fd.set("file", {
type: "text/plain",
name: "upload.txt",
[Symbol.toStringTag]: "File",
stream() {
return getFileStream();
}
});
const encoder = new FormDataEncoder(fd);
await got.post('http://localhost:3001/upload', {
body: Readable.from(encoder),
headers: encoder.headers
});
// Below code works with old Formdata library not checked with new Formdata library
// await axios({
// url: 'http://localhost:3001/upload',
// method: 'post',
// data: fd,
// });
res.send('uploaded');
} catch (error) {
res.send('uploaded error');
console.log(error);
}
});
app.listen(3000, () => {
console.log('Example app listening on port 3000!');
});second way /**
* App that will upload the file to the server
*/
import express from 'express';
import { createReadStream } from "node:fs";
import { dirname, join, resolve } from 'node:path';
import { PassThrough, pipeline, Readable, Transform } from "node:stream";
import { got } from 'got';
import { fileURLToPath } from 'node:url';
import cors from 'cors';
import { FormData } from 'formdata-node'
import { FormDataEncoder } from "form-data-encoder"
import axios from 'axios';
class BlobFromStream {
#stream
size
constructor(stream: any, size: any) {
this.#stream = stream
this.size = size
}
stream() {
return this.#stream
}
get [Symbol.toStringTag]() {
return "Blob"
}
}
const __filename = fileURLToPath(import.meta.url);
const __dirname = dirname(__filename);
const app = express();
const filePath = resolve(join(__dirname,'../assets/upload.txt'));
app.use(cors());
const getFileStream = () => {
const fileStream = createReadStream(filePath);
const passThrough = new PassThrough();
// fileStream.pipe(passThrough);
// return passThrough;
return fileStream;
};
app.get('/', (req, res) => {
res.send('Hello World!');
});
app.post('/sendFile', async (req, res) => {
try {
const fd = new FormData();
const stream = getFileStream();
fd.set("file", {
type: "text/plain",
name: "upload.txt",
[Symbol.toStringTag]: "File",
stream() {
return stream;
}
});
const encoder = new FormDataEncoder(fd);
fd.set("file", new BlobFromStream(stream, encoder.contentLength), 'upload.txt');
await got.post('http://localhost:3001/upload', {
body: Readable.from(encoder),
headers: encoder.headers
});
// Below code works with old Formdata library not checked with new Formdata library
// await axios({
// url: 'http://localhost:3001/upload',
// method: 'post',
// data: fd,
// });
res.send('uploaded');
} catch (error) {
res.send('uploaded error');
console.log(error);
}
});
app.listen(3000, () => {
console.log('Example app listening on port 3000!');
});Example node server to upload files /**
* App that will upload the file to the server
*/
import express from 'express';
import multer from 'multer';
import cors from 'cors';
const app = express();
app.use(cors());
const upload = multer({ dest: '../uploads/' });
app.get('/', (req, res) => {
res.send('Hello World!');
});
app.post('/upload', upload.any(), (req, res) => {
console.log(req.files);
res.send('uploaded');
setTimeout(() => {
console.clear();
}, 10000);
})
app.listen(3001, () => {
console.log('Example app listening on port 3001!');
console.clear();
}); |
|
The first one definitely should work, I've opened octet-stream/form-data-encoder#9 to track this. |
|
In the second example you're incorrectly passing |
|
Can you try https://github.com/jimmywarting/FormData ? |
|
@szmarczak I tried with the above package. The Process is similar to what formdata-node package does for streams. |
|
As I mentioned here, you need to avoid |
|
@octet-stream @szmarczak Removing the Content-Length header fixed the issue. /**
* App that will upload the stream to the server using got
*/
import express from "express";
import { createReadStream } from "node:fs";
import { dirname, join, resolve } from "node:path";
import { PassThrough, Readable } from "node:stream";
import { got } from "got";
import { fileURLToPath } from "node:url";
import cors from "cors";
import { FormData } from "formdata-node";
import { FormDataEncoder } from "form-data-encoder";
const __filename = fileURLToPath(import.meta.url);
const __dirname = dirname(__filename);
const app = express();
const filePath = resolve(join(__dirname, "../assets/upload.txt"));
app.use(cors());
/**
* Imitate streams coming from other destinations
*/
const getFileStream = () => {
const fileStream = createReadStream(filePath);
const passThrough = new PassThrough();
fileStream.pipe(passThrough);
return passThrough;
};
app.post("/sendFile", async (req, res) => {
try {
const fd = new FormData();
fd.set("file", {
type: "text/plain",
name: "upload.txt",
[Symbol.toStringTag]: "File",
stream() {
return getFileStream();
},
});
const encoder = new FormDataEncoder(fd);
const headers = {
"Content-Type": encoder.headers["Content-Type"],
};
await got.post("http://localhost:3001/upload", {
body: Readable.from(encoder),
headers: headers,
});
res.send("uploaded");
} catch (error) {
res.send("uploaded error");
console.log(error);
}
});
app.listen(3000, () => {
console.log("Example app listening on port 3000!");
});Thanks a lot for the help @octet-stream @szmarczak |
|
We need to add a test |
|
I already tested it in form-data-encoder - it will not return |
|
Actually, I think I can add a test and send a PR if you want :) |
4 tasks
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Current Behaviour
Hi, I am trying to upload the file stream to another server.
Please check the code below
Hi, I am creating a file stream like the below snippet.
I have to use passthrough for some reasons.
If I do something like the above I get an error
'ERR_GOT_REQUEST_ERROR'
But if I change getFileStream function like below it works basically I am not using Passthrough
If I do the same in Axios and node fetch it's working with passthrough.
Expected Behaviour:
It should upload the file
Node Version
v14.18.2
The text was updated successfully, but these errors were encountered: