Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add a lakeFS / XID time slug decoder #8323

Open
wants to merge 1 commit into
base: master
Choose a base branch
from

Conversation

arielshaqed
Copy link
Contributor

@arielshaqed arielshaqed commented Oct 30, 2024

Add a lakeFS / XID time slug decoder

Decodes object names nicely, e.g.

❯ lakectl -c fs --pre-sign=false stat lakefs://repo/main/50m
Path: 50m
Modified Time: 2024-10-28 17:52:50 +0200 IST
Size: 52428800 bytes
Human Size: 52.4 MB
Physical Address: s3://treeverse-ariels-test-us/repos/s3-dev-repo/data/gcnl78vqmpl9eb7rqlr0/csfr6kfqmpl9eb7rqlt0,jMHnLFOwK2RtCVV_aE0iClDtfP7ehfORTH_siX9V5e8
Checksum: 4d98d5ac224a2a14a2b7efac5d100757-10
Content-Type: application/octet-stream

❯ go run ./contrib/decoder/decode_partition/ csfr6kfqmpl9eb7rqlt0
2024-10-28 15:52:49 +0000 UTC (asc)
3945-03-05 08:07:11 +0000 UTC (desc)

But does not decode the partition name ("desc"); not sure why not.

❯ go run ./contrib/decoder/decode_partition/ gcnl78vqmpl9eb7rqlr0
2039-09-29 13:48:19 +0000 UTC (asc)
3930-04-04 10:11:41 +0000 UTC (desc)

(Happy to modify if reviewer can explain this!)

Decodes object names nicely, e.g.
```sh
❯ lakectl -c ~/.lakectl.s3.yaml fs --pre-sign=false stat lakefs://repo/main/50m
Path: 50m
Modified Time: 2024-10-28 17:52:50 +0200 IST
Size: 52428800 bytes
Human Size: 52.4 MB
Physical Address: s3://treeverse-ariels-test-us/repos/s3-dev-repo/data/gcnl78vqmpl9eb7rqlr0/csfr6kfqmpl9eb7rqlt0,jMHnLFOwK2RtCVV_aE0iClDtfP7ehfORTH_siX9V5e8
Checksum: 4d98d5ac224a2a14a2b7efac5d100757-10
Content-Type: application/octet-stream

❯ go run ./contrib/decoder/decode_partition/ csfr6kfqmpl9eb7rqlt0
2024-10-28 15:52:49 +0000 UTC (asc)
3945-03-05 08:07:11 +0000 UTC (desc)
```

But does _not_ decode the partition name ("desc"); not sure why not.
```sh
❯ go run ./contrib/decoder/decode_partition/ gcnl78vqmpl9eb7rqlr0
2039-09-29 13:48:19 +0000 UTC (asc)
3930-04-04 10:11:41 +0000 UTC (desc)
```

(Happy to modify if reviewer can explain this!)
@arielshaqed arielshaqed added area/tools Improvements or additions to tooling and scripting exclude-changelog PR description should not be included in next release changelog minor-change Used for PRs that don't require issue attached labels Oct 30, 2024
Copy link

E2E Test Results - Quickstart

11 passed

Copy link

E2E Test Results - DynamoDB Local - Local Block Adapter

13 passed

@N-o-Z
Copy link
Member

N-o-Z commented Oct 30, 2024

Add a lakeFS / XID time slug decoder

Decodes object names nicely, e.g.

❯ lakectl -c fs --pre-sign=false stat lakefs://repo/main/50m
Path: 50m
Modified Time: 2024-10-28 17:52:50 +0200 IST
Size: 52428800 bytes
Human Size: 52.4 MB
Physical Address: s3://treeverse-ariels-test-us/repos/s3-dev-repo/data/gcnl78vqmpl9eb7rqlr0/csfr6kfqmpl9eb7rqlt0,jMHnLFOwK2RtCVV_aE0iClDtfP7ehfORTH_siX9V5e8
Checksum: 4d98d5ac224a2a14a2b7efac5d100757-10
Content-Type: application/octet-stream

❯ go run ./contrib/decoder/decode_partition/ csfr6kfqmpl9eb7rqlt0
2024-10-28 15:52:49 +0000 UTC (asc)
3945-03-05 08:07:11 +0000 UTC (desc)

But does not decode the partition name ("desc"); not sure why not.

❯ go run ./contrib/decoder/decode_partition/ gcnl78vqmpl9eb7rqlr0
2039-09-29 13:48:19 +0000 UTC (asc)
3930-04-04 10:11:41 +0000 UTC (desc)

(Happy to modify if reviewer can explain this!)

Bug in xid!?!?! 😲😲😲😲
They use uint32 when dealing with golang Time objects instead of uint64 😢
This has major implications on us - we need to understand how to resolve this...

@N-o-Z
Copy link
Member

N-o-Z commented Nov 6, 2024

@arielshaqed what would you like to do with this? I can approve it but given what we found this won't be very useful for partitions

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
area/tools Improvements or additions to tooling and scripting exclude-changelog PR description should not be included in next release changelog minor-change Used for PRs that don't require issue attached
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants