Add support for downloading input images from S3 bucket.

[theoway]

What is the problem?

There should be a way for NodeODM to directly get input images from a S3 bucket/folder and use it for processing. Currently, one has to upload it from their machine to a remote instance.

What should be the expected behavior?

There should be a field where one can simply enter the name of the S3 path to images folder, and be able to process it, saving the results in the folder specified by the user.

How can we reproduce this? (What steps did you do to trigger the problem? Be detailed)

It's a feature request.

[theoway]

How I'm going to implement this:

Here, I'll add a similar function to download_images() from the S3 path

NodeODM/libs/S3.js

Line 77 in 6acd372

uploadPaths: function(srcFolder, bucket, dstFolder, paths, cb, onOutput){

Modifications will have to be made in places like these:

NodeODM/index.js

Lines 201 to 284 in 6acd372

	/** @swagger
	* /task/new:
	* post:
	* description: Creates a new task and places it at the end of the processing queue. For uploading really large tasks, see /task/new/init instead.
	* tags: [task]
	* consumes:
	* - multipart/form-data
	* parameters:
	* -
	* name: images
	* in: formData
	* description: Images to process, plus optional files such as a GEO file (geo.txt), image groups file (image_groups.txt), GCP file (*.txt) or seed file (seed.zip). If included, the GCP file should have .txt extension. If included, the seed archive pre-polulates the task directory with its contents.
	* required: false
	* type: file
	* -
	* name: zipurl
	* in: formData
	* description: URL of the zip file containing the images to process, plus an optional GEO file and/or an optional GCP file. If included, the GCP file should have .txt extension
	* required: false
	* type: string
	* -
	* name: name
	* in: formData
	* description: An optional name to be associated with the task
	* required: false
	* type: string
	* -
	* name: options
	* in: formData
	* description: 'Serialized JSON string of the options to use for processing, as an array of the format: [{name: option1, value: value1}, {name: option2, value: value2}, ...]. For example, [{"name":"cmvs-maxImages","value":"500"},{"name":"time","value":true}]. For a list of all options, call /options'
	* required: false
	* type: string
	* -
	* name: skipPostProcessing
	* in: formData
	* description: 'When set, skips generation of point cloud tiles.'
	* required: false
	* type: boolean
	* -
	* name: webhook
	* in: formData
	* description: Optional URL to call when processing has ended (either successfully or unsuccessfully).
	* required: false
	* type: string
	* -
	* name: outputs
	* in: formData
	* description: 'An optional serialized JSON string of paths relative to the project directory that should be included in the all.zip result file, overriding the default behavior.'
	* required: false
	* type: string
	* -
	* name: dateCreated
	* in: formData
	* description: 'An optional timestamp overriding the default creation date of the task.'
	* required: false
	* type: integer
	* -
	* name: token
	* in: query
	* description: 'Token required for authentication (when authentication is required).'
	* required: false
	* type: string
	* -
	* name: set-uuid
	* in: header
	* description: 'An optional UUID string that will be used as UUID for this task instead of generating a random one.'
	* required: false
	* type: string
	* responses:
	* 200:
	* description: Success
	* schema:
	* type: object
	* required: [uuid]
	* properties:
	* uuid:
	* type: string
	* description: UUID of the newly created task
	* default:
	* description: Error
	* schema:
	* $ref: '#/definitions/Error'
	*/
	app.post('/task/new', authCheck, taskNew.assignUUID, taskNew.uploadImages, (req, res, next) => {

I'd like to know your thoughts on this, and any directions on implementing this are much appreciated :)

[fosteman]

Also interested in doing something similar. My thought was to mount GCP bucket onto the local filesystem with gcloud api.

[FrankvVeelen]

Did you make any progress on this? I'm trying to do the same thing and would love to know if you got it working.

[theoway]

Did you make any progress on this? I'm trying to do the same thing and would love to know if you got it working.

No, I'm working on a different feature in the ODM engine. You're welcome to add this enhancement :)

[utya]

Hello everyone. I would like to update this issue. Currently, I have make some research and realized that there are two ways to transfer an image from S3 to ODM. The first method is to add an S3 photo upload feature in the ODM python Docker itself. And the second method is to add an S3 photo upload feature to the nodeODM and then pass these images into python. I have some draft code. Can I pull request it?

[pierotofy]

Always feel free to open pull requests.

[Saijin-Naib]

This sounds fascinating! I look forward to seeing what you have, but Piero and Stephen likely need to weigh in first.

[utya]

Hi. My ODM container is configured to download images from an S3 bucket upon each startup. I'm planning to modify NodeODM to pass S3 bucket parameters for each task. However, I need to confirm how NodeODM handles ODM Docker containers for each task. Does NodeODM initiate a new ODM Docker container for each task, or does it reuse an existing container? Understanding this will help me ensure that my modifications work as intended.

Thanks

[kauly]

Hello, I think that similar results can be achieved using the zipUrl param of this route:

https://github.com/OpenDroneMap/NodeODM/blob/master/docs/index.adoc#post-tasknew

Just zip the images before upload them to S3