Feature/ingest cmd #1864
Feature/ingest cmd #1864
Conversation
cmd/lakectl/cmd/ingest.go
Outdated
| resp, err := client.StageObjectWithResponse(ctx, | ||
| lakefsURI.Repository, | ||
| lakefsURI.Ref, &api.StageObjectParams{ | ||
| Path: key, | ||
| }, | ||
| api.StageObjectJSONRequestBody{ | ||
| Checksum: e.ETag, | ||
| PhysicalAddress: e.Address, | ||
| SizeBytes: e.Size, | ||
| }, | ||
| ) |
There was a problem hiding this comment.
Optimization: parallelize the calls instead of using a single worker that perform the calls one after the other.
There was a problem hiding this comment.
As discussed, let's open a seperate issue
| var staged int64 | ||
| var stagedBytes int64 | ||
| client := getClient() | ||
| err := store.Walk(ctx, from, func(e store.ObjectStoreEntry) error { |
There was a problem hiding this comment.
A simple progress bar will help the user understand that things are progressing and in which rate. See import progress bars as an example.
There was a problem hiding this comment.
currently possible with --verbose which will print each individual iobject being staged. Progress bars are nice for things that are done interactively, I feel the use case here is more "do this before running my job in airflow" in which case the output will appear in a log file, rendering the progress bar unreadable.. Happy to hear objections or suggestions though..
Closes #1848