Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

temp folder refactor #5000

Merged
merged 7 commits into from
Dec 26, 2023
Merged

Conversation

mnaamani
Copy link
Member

@mnaamani mnaamani commented Dec 12, 2023

Fixes #4977

  • Temp folder can now be configured with argument with a fully qualified path, with a check that it is not the same as the uploads folder.
  • Added check to ensure logs folder is not the same directory as the uploads folder.
  • Improved initialization of the uploads directory, reading data objects from the uploads folder filters out any files/subfolders without a name matching the data-object-id (ie. a number)
  • Improved initialization of temp directory. It will not be recursively deleted on startup (risky).
  • Renamed the 'dev' commands to 'util' commands although I have some reservation about using the cleanup command outside of the context of the running server.. Perhaps we should drop them.

@mnaamani mnaamani changed the title storage-node: temp folder refactor temp folder refactor Dec 12, 2023
@mnaamani mnaamani marked this pull request as ready for review December 21, 2023 13:39
Copy link
Contributor

@zeeshanakram3 zeeshanakram3 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks Good. Can you please add --tempFolder flag in docker-compose file, as an example on how to use it?

logger.warn(
'It is recommended to specify a unique file path for temporary files.' +
'For now a temp folder under the uploads folder will be used.' +
'In future this will be warning will become and error!'
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

In future this will be warning will become and error!

Can you please rephrase that?

Copy link
Member Author

@mnaamani mnaamani Dec 23, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Will rephrase the whole warning to:

You did not specify a path to the temporary directory. A temp folder under the uploads folder will
be used. In a future release passing an absolute path to a temporary directory with the tempFolder argument
will become mandatory.

WDYT?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

done in a287fdc

Comment on lines +169 to +171
if (flags.logFilePath && path.relative(flags.logFilePath, flags.uploads) === '') {
this.error('Paths for logs and uploads must be unique.')
}
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
if (flags.logFilePath && path.relative(flags.logFilePath, flags.uploads) === '') {
this.error('Paths for logs and uploads must be unique.')
}
if (flags.logFilePath && path.relative(flags.logFilePath, flags.uploads) === '..') {
this.error('Paths for logs and uploads must be unique.')
}

Shouldn't it be path.relative(flags.logFilePath, flags.uploads) === '..', I tested the path.relative function as path.relative('/uploads/temp', '/uploads') and I got .. as return value

Copy link
Member Author

@mnaamani mnaamani Dec 23, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Here is was only trying to protect against the two paths being the same location. I was not concerned with temp or logs being subfolders of the main uploads directory. That is why I was comparing with '' null string.

Originally I was doing if (lags.logFilePath === flags.uploads) { } but I decided to use path.relative() to take into account trailing / path separator..

Given that I added proper filtering of filenames when they are loaded from uploads directory, and if we assume the file names of logs and temp files will not clash with the object id name, perhaps this constraint is not really necessary at all?

Comment on lines +241 to +242
if (path.relative(tempFolder, flags.uploads) === '') {
this.error('Paths for temporary and uploads folders must be unique.')
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
if (path.relative(tempFolder, flags.uploads) === '') {
this.error('Paths for temporary and uploads folders must be unique.')
if (path.relative(tempFolder, flags.uploads) === '..') {
this.error('Paths for temporary and uploads folders must be unique.')

Please see the other comment

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

commented here #5000 (comment)

@mnaamani
Copy link
Member Author

Looks Good. Can you please add --tempFolder flag in docker-compose file, as an example on how to use it?

Added in 3dd8e0a

Copy link
Contributor

@zeeshanakram3 zeeshanakram3 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM. I believe the version bump and CHANGELOG will be done in separate release PR?

@mnaamani
Copy link
Member Author

LGTM. I believe the version bump and CHANGELOG will be done in separate release PR?

Yes will do that next.

@mnaamani mnaamani merged commit c4789f9 into Joystream:master Dec 26, 2023
23 checks passed
@mnaamani mnaamani deleted the colossus-specify-temp-folder branch March 12, 2024 10:36
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

Successfully merging this pull request may close these issues.

colossus: fully qualified path for temp folder
2 participants