Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: fast directory listings with DAG Size column #9481

Merged
merged 4 commits into from
Dec 12, 2022
Merged

Conversation

hacdias
Copy link
Member

@hacdias hacdias commented Dec 8, 2022

Closes #9058.

Checklist

  • Use .Dag.Get instead of .Unixfs.Ls to fetch the directories (take into consideration sharded)
    • Instead, improved Unixfs.Ls with UseCumulativeSize to use the DAG Size, but skip remaining calculations. See comment.
  • Remove Gateway.FastDirIndexThreshold option as it is no longer relevant
  • Add "Preview as" and "Download as" buttons to the listing header with multiple formats. descoped for now
  • Tooltip on Size with explanation.
  • feat: add UseCumulativeSize UnixfsLs option interface-go-ipfs-core#95
    • Approved, merged and released
    • Dependency updated here

image

Loads bafybeiggvykl7skb2ndlmacg2k5modvudocffxjesexlod2pfvg5yhwrqm (10k items) extremely fast, and with size!

@hacdias hacdias self-assigned this Dec 8, 2022
@Jorropo Jorropo self-requested a review December 8, 2022 13:24
@hacdias hacdias force-pushed the refactor/9058 branch 2 times, most recently from 8b9ff9a to 233ff43 Compare December 8, 2022 13:31
@hacdias hacdias changed the title refactor: use Dag Size instead of UnixFS Size in Listing refactor: improve directory listing through .Dag.Get and add 'Download as' links Dec 8, 2022
@hacdias hacdias changed the title refactor: improve directory listing through .Dag.Get and add 'Download as' links refactor: faster directory listing with .Dag.Get, add 'Download as' links Dec 8, 2022
@hacdias hacdias marked this pull request as ready for review December 8, 2022 13:41
@hacdias hacdias requested a review from lidel as a code owner December 8, 2022 13:41
Copy link
Member

@lidel lidel left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Overall this is step in good direction.

Would be nice to include this (#9058) in 0.18.0-rc1, so the "Size" in dir listing is the same as Tsize in ?format=json response added by #9335.

We have a lot of things in flight for 0.18.0-rc1, so this may slip to rc2 (or 0.19).
Small asks below.

assets/dir-index-html/src/dir-index.html Outdated Show resolved Hide resolved
assets/dir-index-html/dir-index.html Outdated Show resolved Hide resolved
assets/dir-index-html/src/dir-index.html Outdated Show resolved Hide resolved
docs/config.md Show resolved Hide resolved
core/corehttp/gateway_handler_unixfs_dir.go Outdated Show resolved Hide resolved
@hacdias hacdias requested a review from lidel December 8, 2022 14:06
@hacdias
Copy link
Member Author

hacdias commented Dec 8, 2022

@Jorropo @lidel as per @Jorropo's suggestion, I instead added an option to Unixfs.Ls to make it use the cumulative DAG size instead. We can keep the same API, and it's now blazing fast with that option and includes size!

core/coreapi/unixfs.go Outdated Show resolved Hide resolved
Copy link
Contributor

@Jorropo Jorropo left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@hacdias hacdias changed the title refactor: faster directory listing with .Dag.Get, add 'Download as' links refactor: faster directory listing, add 'Download as' links Dec 8, 2022
@hacdias hacdias force-pushed the refactor/9058 branch 2 times, most recently from d280dc3 to 845ebef Compare December 12, 2022 11:38
We've just introduced JSON and CBOR support, and need additional work to
figure put how to communicate these options in a way that leverages
verifiable CAR over untrusted TARs.

Removing them for now.
@lidel lidel changed the title refactor: faster directory listing, add 'Download as' links feat: fast directory listings with size column Dec 12, 2022
@lidel lidel changed the title feat: fast directory listings with size column feat: fast directory listings with DAG Size column Dec 12, 2022
@lidel lidel self-assigned this Dec 12, 2022
Copy link
Member

@lidel lidel left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM, merging for inclusion in Kubo 0.18.0-rc1 (so the sizes here match Tsize from ?format=json)

  • Removed "Preview as" and "Download as" buttons as discussed today – we don't want to announce these features widely yet, probably need proper designer to prototype / refresh dir listing before we add them.
  • Confirmed that the removed optional field does not crash clients that had it defined manually
  • Switched to released dependency
  • Added release notes

@lidel lidel merged commit 7bdb341 into master Dec 12, 2022
@lidel lidel deleted the refactor/9058 branch December 12, 2022 20:45
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
No open projects
Archived in project
Development

Successfully merging this pull request may close these issues.

gateway/dir-index-html: switch dir listing sizes to Tsize
3 participants