Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Dumping all crops into the same dir seems potentially messy and inefficient #3502

Closed
jonfroehlich opened this issue Feb 26, 2024 · 3 comments · Fixed by #3521
Closed

Dumping all crops into the same dir seems potentially messy and inefficient #3502

jonfroehlich opened this issue Feb 26, 2024 · 3 comments · Fixed by #3521
Assignees

Comments

@jonfroehlich
Copy link
Member

Related to #3468

With @hoominchu's auto-cropping code, should we avoid dumping all crops from all cities into the same directory? Some filesystems have small limits (e.g., 65k files per dir). Moreover, it takes a long time to open large dirs. I might suggest organizing them by city and then label type (so, we would have a Zurich, Switzerland dir and then dirs like curb ramps, missing curb ramps, etc.)

@misaugstad
Copy link
Member

Sounds like our filesystem if zfs, which has a max number of files per dir of 281,474,976,710,656. So running into that limit won't be an issue. Though we could always group them into sub directories if we find that easier.

@jonfroehlich
Copy link
Member Author

I think we should clean this up a bit. If I use WinSCP or another UI tool to try and load those dirs, it can take a very long time.

Perhaps you can address this when we work on #3506.

Also, these files are marked as "temp"--shouldn't that also be addressed? Maybe that's another existing Issue that should be linked.

@misaugstad
Copy link
Member

Also, these files are marked as "temp"--shouldn't that also be addressed? Maybe that's another existing Issue that should be linked.

Yes, that will be fixed as part of #3468!

Hoping to tackle all three of these issues in the next week or so!

@misaugstad misaugstad self-assigned this Mar 6, 2024
@misaugstad misaugstad mentioned this issue Apr 3, 2024
@misaugstad misaugstad moved this to done / on test servers in Mikey Task Board May 30, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
Status: done / on test servers
Development

Successfully merging a pull request may close this issue.

2 participants