Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Implement new cached metadata system #39

Open
stv0g opened this issue Oct 18, 2022 · 3 comments
Open

Implement new cached metadata system #39

stv0g opened this issue Oct 18, 2022 · 3 comments

Comments

@stv0g
Copy link
Owner

stv0g commented Oct 18, 2022

We currently use S3 object tags to store some metadata information about each upload.

This works well for most use-cases, however has the disadvantage that the object tags are not easily modifiable.
Or flat out not supported by some cloud hosters (Wasabi).

Some planned features require that we can modify upload metadata (#10, #11).
Hence, I've thought about moving the metadata to a dedicated S3 object.
There is still the disadvantage, that S3 does not really support atomic operations on objects so we still have a hard time tracking download numbers (e.g. any sort of counter). But I already have an idea of implementing some sort of local or distributed (Redis) caching layer in front of those metadata objects which could solve this.

@mrg358
Copy link

mrg358 commented Nov 11, 2022

I just tested out Gos3, so dropping my results.

  • AWS works perfectly, no issue.
  • Contabo S3 (ceph) results in error "Upload failed: Final checksum mismatch", upload is completed though. On second attempt to upload the same file, it completes without error.
  • Garage (selfhosted S3) sadly doesn't work b/c of the missing ObjectTagging API.

I will keep using AWS, but it would be so nice to have broader S3 compatibility.
Very nice app indeed. Thank you for your work!

@literon36
Copy link

ContaboS3 (ceph) results in error "Upload failed: Final checksum mismatch", upload is completed though. On second attempt to upload the same file, it completes without error.

this also now happens at the demo instance https://gose.0l.de ... (@stv0g)
I've tried multiple browsers & devices so it should not just be a me-issue.

@stv0g
Copy link
Owner Author

stv0g commented Dec 27, 2023

Thanks for the input. I am currently running a bit low on resources since I am occupied with my other projects. But contributions in the direction of a new metadata system are very welcome :)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

3 participants