-
Notifications
You must be signed in to change notification settings - Fork 879
NG: Transactional storage #626
Comments
I don't expect miracles unfortunately. Distributed store are unlikely to grow wings... On the other hand, we had problems with that aspect because of the design / workflow of registry v1. That might include using "locks" on the transactional storage (redis). |
On Tue, Oct 28, 2014 at 01:40:29PM -0700, Olivier Gambier wrote:
Redis supports this already, we just need to add at least |
The content-addressability part is up to the registry, not to the storage itself (drivers shouldn't need to know anything about that).
That was the same problem with v1... such an approach is not practical with any crowded storage. About the rest, I would rather keep the "transactional storage" as a "simple cache" and not as full-blown requirement. |
On Tue, Oct 28, 2014 at 03:00:58PM -0700, Olivier Gambier wrote:
If you're going to have content-addressability I'd definately put it
That just means your garbage collection process takes longer to
How can a cache add transactionality? |
That would make for code duplication into every driver, doesn't solve race conditions problems, requires the drivers to do more work, and make it more difficult to fix the adressibility model if we want to change it (eg: multiple version of tarsums for example). So, it's no on this :-) |
On Tue, Oct 28, 2014 at 03:26:33PM -0700, Olivier Gambier wrote:
So I'd use a simple hash for content addressability. For another If you go ahead with a fancy hash anyway, extending the API to have:
would be a fairly small change. And the rest of the streaming |
On Tue, Oct 28, 2014 at 03:26:33PM -0700, Olivier Gambier wrote:
Why not? With content-addressable storage (even with |
|
On Tue, Oct 28, 2014 at 03:54:50PM -0700, Olivier Gambier wrote:
Ah, that is another reason why calculating the name in the storage
I'd be surprised if any language folks would use wouldn't have a sha1 a. Use a hash that folks have already implemented (e.g. sha256), with
There's no need for a generic move here, we just get the name for |
On Tue, Oct 28, 2014 at 03:13:29PM -0700, W. Trevor King wrote:
Another possible optimization is that you can make the initial, Is it worth avoiding copies on S3? It looks like S3 has a transparent |
We have very few layers above 500MB, not to mention above 5GB. Either way, I still fail to understand what benefit we could possibly get from delegating (the same) intelligence into (each and every) drivers :-). |
On Wed, Oct 29, 2014 at 10:18:51AM -0700, Olivier Gambier wrote:
If you want a 5GB cap on layers, then that will work (assuming other
It's not the same intelligence. Maybe I have a dozen repositories |
We are using this discussion in the ongoing roadmap for distribution but don't have actionable work based on this issue. Closing for now. |
We've had some difficulties ensuring atomic transactions with the current registry. This makes it difficult to do things like robustly refcount images (or anything else where you'd need to touch metadata in several locations at once). I've been poking around to get a feel for how other folks handle transactions with huge.
This is not my area of expertise though, and its certainly possible that there's a mainstream, open-source, transactional storage solution for large binary blobs; I just can't find it ;).
The text was updated successfully, but these errors were encountered: