Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

go-libp2p v0.21.0 #1514

Closed
15 of 41 tasks
marten-seemann opened this issue May 20, 2022 · 5 comments · Fixed by #1648
Closed
15 of 41 tasks

go-libp2p v0.21.0 #1514

marten-seemann opened this issue May 20, 2022 · 5 comments · Fixed by #1648

Comments

@marten-seemann
Copy link
Contributor

marten-seemann commented May 20, 2022 •
edited by MarcoPolo
Loading

🗺 What's left for release

Monitoring and Resource Management

Swarm Sync Notification

Misc

🔦 Highlights

Resource Management

  • Default resource management config limits will now autoscale depending on the machine size.
  • You can now specify an allowlisted set of multiaddrs that can still connect to you even if you've reached your normal limits (still limited by the allowlist limits).
  • Resource manager now exposes opencensus metrics and provides a default grafana dashboard.

Swarm Notifications are now synchronous

If your protocol depended on being given notifications in an async context and taking its time processing that, you will slow down your whole go-libp2p application. See #1562 for more details. This only matters if you implemented a network.Notifiee and registered to receive notifications on swarm network events.

Changelog

Contributors

Contributor Commits Lines ± Files Changed
Marten Seemann 217 +85564/-25980 1143
Marco Munizaga 78 +6722/-2489 150
vyzo 64 +6393/-666 130
Steven Allen 31 +1357/-745 54
Jeromy 5 +1032/-64 16
Jorropo 6 +168/-33 14
hareku 3 +172/-6 7
Marcin Rataj 2 +50/-124 5
Ivan Trubach 1 +98/-74 6
Jakub Sztandera 1 +101/-62 1
Christian Stewart 3 +89/-48 14
Raúl Kripalani 2 +69/-62 9
Masih H. Derkani 3 +84/-13 5
Julien Muret 1 +60/-7 2
Lars Gierth 1 +20/-29 4
Cole Brown 4 +27/-19 4
Chao Fei 2 +15/-30 9
Nuno Diegues 2 +25/-18 9
Daniel Martí 1 +15/-6 1
Wiktor Jurkiewicz 1 +13/-5 1
Matt Robenolt 1 +15/-1 2
gammazero 1 +7/-6 3
aarshkshah1992 2 +8/-2 2
Aaron Riekenberg 1 +4/-4 4
Rod Vagg 3 +3/-3 3
Adrian Lanzafame 1 +3/-3 1
Dmitriy Ryajov 2 +2/-3 2
millken 1 +1/-1 1
Matt Joiner 1 +1/-1 1
Leo Balduf 1 +1/-1 1
Didrik Nordström 1 +2/-0 1
Adin Schmahmann 1 +1/-1 1

✅ Release Checklist

  • Stage 0 - Finishing Touches
    • Go through relevant libp2p repos looking for unreleased changes that should make it into the release. If you find any, cut releases.
    • Run go get -u ./... to see if there are any out-of-date deps that look important. If there are, bubble them. Try to avoid directly updating indirect deps in go-libp2p's go.mod when possible.
    • Make sure local tests are passing.
  • Stage 1 - Upstream Testing
    • Create testing branches in lotus & go-ipfs with the new go-libp2p release and run CI/tests. Many upstream projects are tested in CI, but lotus & go-ipfs are not.
    • (someday) Run upstream testground tests. Unfortunately, this is too time consuming at the moment.
      • (someday) Run bitswap testground tests.
      • (someday) Run DHT testground tests.
  • Stage 2 - Infrastructure Testing
    • How: Using the testing branches created above, work with the infrastructure team to deploy the new libp2p versions to our infrastructure.
    • Where:
      • A go-ipfs gateway.
        • Deploy
        • Analyze
          • Look at pprof profile dumps, especially CPU profiles and heap allocation profiles, noting any significant changes.
      • A go-ipfs bootstrapper.
        • Deploy
        • Analyze
          • Look at pprof profile dumps, especially CPU profiles and heap allocation profiles, noting any significant changes.
          • Check peers (e.g., ipfs swarm peers) to make sure we're connecting to peers on all transports.
          • Check advertised addresses and protocols (e.g., ipfs id) to make sure they're sane.
  • Stage 3 - Release
    • Tag the release on master.
    • Publish the release through the GitHub UI, adding the release notes. Some users rely on this to receive notifications of new releases.
    • Announce the release on the discuss.libp2p.io.
  • Stage 4 - Update Upstream
  • Make required changes to the release process.
@marten-seemann marten-seemann changed the title v0.21.0 go-libp2p v0.21.0 May 20, 2022
@marten-seemann marten-seemann mentioned this issue Jun 21, 2022
Closed
MarcoPolo added a commit that referenced this issue Jul 7, 2022
@MarcoPolo
Release v0.21.0
59bbbe4 
See #1514
@MarcoPolo MarcoPolo mentioned this issue Jul 7, 2022
Merged
@BigLep BigLep linked a pull request Jul 23, 2022 that will close this issue
Release v0.21.0 #1648
Merged
@BigLep
Copy link
Contributor

BigLep commented Jul 25, 2022 •
edited by MarcoPolo
Loading

I'm putting here for now but happy to move somewhere else. I wanted a table to see the summary of the deployment status to ipfs.io gateways and how it pertains to go-libp2p so we can figure out how to unblock this release.

This is my understanding from looking at the last 24 hours: https://protocollabs.grafana.net/d/8zlhkKTZk/gateway-slis-precomputed?from=now-24h&to=now&viewPanel=1369&var-location=All&var-interval=30m&var-ipfsbank=All&orgId=1

Bank # Configuration go-ipfs version Resource manager enabled Has worse performance Other notes
1 https://github.com/protocol/bifrost-infra/blob/master/ansible/inventories/bifrost/group_vars/gateway_ipfs_bank1.yml go_ipfs_repo: "ipfs/go-ipfs-private"
go_ipfs_version: "bifrost-0.13.0-steb-patch-bitswap-2f2580"		 Yes No
2 https://github.com/protocol/bifrost-infra/blob/master/ansible/inventories/bifrost/group_vars/gateway_ipfs_bank2.yml go_ipfs_version: "bifrost-0.13.0-steb-patch-bitswap-2f2580"
go_ipfs_repo: "ipfs/go-ipfs-private"		 Yes No
3 https://github.com/protocol/bifrost-infra/blob/master/ansible/inventories/bifrost/group_vars/gateway_ipfs_bank3.yml go_ipfs_version: "bifrost-0.13.0-steb-patch-bitswap-2f2580"
go_ipfs_repo: "ipfs/go-ipfs-private"		 Yes Yes
4 https://github.com/protocol/bifrost-infra/blob/master/ansible/inventories/bifrost/group_vars/gateway_ipfs_bank4.yml go_ipfs_deploy_bin_strategy: git
go_ipfs_version: "rcgmr-auto-scale"		 No Yes
5 https://github.com/protocol/bifrost-infra/blob/master/ansible/inventories/bifrost/group_vars/gateway_ipfs_bank5.yml go_ipfs_version: v0.14.0-rc1
go_ipfs_name: kubo		 No Yes
6 https://github.com/protocol/bifrost-infra/blob/master/ansible/inventories/bifrost/group_vars/gateway_ipfs_bank6.yml go_ipfs_version: v0.14.0-rc1
go_ipfs_name: kubo		 No Yes

Snapshotted graph:
image

@MarcoPolo
Copy link
Collaborator

1 & 2 have rcmgr enabled

@MarcoPolo
Copy link
Collaborator

MarcoPolo commented Jul 25, 2022 •
edited
Loading

I wrote this as my update, but I’ll reproduce here:

  • Due to time and priority constraints we're going to focus on answering the question "Does the new go-libp2p release make things worse?" instead of "What is causing an increase in TTFB". I'd much rather find the root cause here, but we have to balance that with getting these new improvements, obs, and rcmgr improvements out there.
  • To answer the former question we're going to rebase the go-libp2p RC PR on the Kubo 0.14 and compare it with Kubo 0.14 on the other nodes.

@BigLep
Copy link
Contributor

BigLep commented Jul 25, 2022 •
edited
Loading

@MarcoPolo : thanks for the catch that bank 1 and 2 have resource manager enabled. I should have looked at the config. I have updated the table.

So to summarize, it seems performance is helped by resource management enforced.
When resource management disabled, W=we see worse performance regardless if go-libp2p v0.21 is used or not.

As a result, why do we need to "rebase the go-libp2p RC PR on the Kubo 0.14 and compare it with Kubo 0.14 on the other nodes"? Is this just to fully confirm that go-libp2p 0.21 isn't making things worse since we previously tested it with a version of go-ipfs with a lesser amount of commits than Kubo 0.14?

(Happy to cover verbally during 1 on 1.)

@MarcoPolo
Copy link
Collaborator

MarcoPolo commented Jul 27, 2022 •
edited
Loading

I've rebased and deployed go-libp2p v0.21-rc on Kubo 0.14. PR here: ipfs/kubo#9074.

After running this for a bit I don't think go-libp2p has obviously negatively affected anything. Thus I'm going to continue with the release.
Screen Shot 2022-07-27 at 1 07 00 PM
(Vertical blue line is when bank4 was deployed)

Note that there is massive variance between these banks even if they're running the same thing. So I wouldn't put to much stock on the spike of bank 4 at the end.

MarcoPolo added a commit that referenced this issue Jul 27, 2022
@MarcoPolo
Release v0.21.0 (#1648)
3b3605c 
See #1514
@lidel lidel mentioned this issue Sep 5, 2022
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

Release v0.21.0 libp2p/go-libp2p
4 participants
@BigLep @MarcoPolo @marten-seemann and others