[Tracking issue] Release for the "over the hump" copy repo consolidation

[BigLep]

Done criteria

There is a Kubo release that has:

1. All the repos listed below copied into this repo (with deprecated types signposts).
2. A changelog following Document the release process #170
3. No refactors applied on top.

In addition, the repos that are copied in have:

1. Deprecated types
2. Readme update with a signpost to Boxo
3. Issues/PRs closed or copied
4. Commit history maintained

List of repos and their status

The determination of this plan was handled in #174

Repo	Destination	Copied?	Issues Hand Copied?	README notice of repo status added?	Deprecated Types?	Notes
ipfs/go-bitswap	boxo/bitswap	✔️	✔️ Issues were transferred	✔️	✔️	This repo was fully migrated (not just copied). Archived.
ipfs/go-ipfs-files	boxo/files	✔️	✔️ Issues were transferred	✔️	✔️	This repo was fully migrated (not just copied). Archived.
ipfs/go-ipfs-config	kubo/config	✔️	✔️ Issues were transferred	✔️	✔️	This repo was fully migrated (not just copied). Archived.
ipfs/go-block-format	🪜 Being moved of Boxo	n/a	✔️ Issues moved back	✔️ Reverted README notice	✔️ Reverted	We are leaving this repo as it was pre-2023 as a separate repo from Boxo.
ipld/go-car	boxo/ipld/car	✔️	🪜 Issues will not be copied since the repo is actively maintained by others.	🪜 No readme notice will be added since the repo is actively maintained by others.	🪜 No types will be deprecated.	Decisions here were made in #218, #218 (comment)
ipfs/go-merkledag	boxo/ipld/merkledag	✔️	🪜 Issues will not be copied since the repo is maintained by others.	✔️ No "not maintained" notice was added, but a pointer to Boxo was included.	🪜 No types will be deprecated.	Decisions here were made in #218. The docs changes are in ipfs/go-merkledag#101
ipfs/interface-go-ipfs-core	boxo/coreiface	✔️	✔️ Issues were transferred to either boxo or kubo/client/rpc	✔️	✔️	we will keep non-kubo parts in boxo, and move kubo-specific parts to kubo repo. Archived.
ipfs/go-unixfs	boxo/ipld/unixfs	✔️	✔️	✔️	✔️	Archived.
ipfs/go-pinning-service-http-client	boxo/pinning/remote/client	✔️	✔️	✔️	✔️	Archived.
ipfs/go-path	boxo/path	✔️	✔️	✔️	✔️	Archived.
ipfs/go-namesys	boxo/namesys	✔️	✔️	✔️	✔️	Archived.
ipfs/go-mfs	boxo/mfs	✔️	✔️	✔️	✔️	Archived.
ipfs/go-ipfs-provider	boxo/provider	✔️	✔️	✔️	✔️	Archived.
ipfs/go-ipfs-pinner	boxo/pinning/pinner	✔️	✔️	✔️	✔️	Archived.
ipfs/go-ipfs-keystore	boxo/keystore	✔️	✔️	✔️	✔️	Archived.
ipfs/go-filestore	boxo/filestore	✔️	✔️	✔️	✔️	Archived.
ipfs/go-ipns	boxo/ipns	✔️	✔️	✔️	✔️	Archived.
ipfs/go-blockservice	boxo/blockservice	✔️	✔️	✔️	✔️	Archived.
ipfs/go-ipfs-chunker	boxo/chunker	✔️	✔️	✔️	✔️	Archived.
ipfs/go-fetcher	boxo/fetcher	✔️	✔️	✔️	✔️	Archived.
ipfs/go-ipfs-blockstore	boxo/blockstore	✔️	✔️	✔️	✔️	Archived.
ipfs/go-ipfs-posinfo	boxo/filestore/posinfo	✔️	✔️	✔️	✔️	Archived.
ipfs/go-ipfs-util	boxo/util	✔️	✔️	✔️	✔️	We don't want util dirs, but we'll refactor later. Archived.
ipfs/go-ipfs-ds-help	boxo/datastore/dshelp	✔️	✔️	✔️	✔️	Archived.
ipfs/go-verifcid	boxo/verifcid	✔️	✔️	✔️	✔️	Archived.
ipfs/go-ipfs-exchange-offline	boxo/exchange/offline	✔️	✔️	✔️	✔️	Archived.
ipfs/go-ipfs-routing	boxo/routing	✔️	✔️	✔️	✔️	Archived.
ipfs/go-ipfs-exchange-interface	boxo/exchange	✔️	✔️	✔️	✔️	Archived.

• We want to make it easy for consumers to migrate, so we want to keep package names the same as much as possible, and import paths should be trivial to update. This takes priority over organized directory structures within libipfs, we will be refactoring later anyway. The focus here is on consolidation and making it easy to migrate.
• Anything that is only applicable to Kubo should go to Kubo, not go-libipfs

"not maintained" README messaging

This messaging will be posted in all the READMEs for repos that have been copied into Boxo that are not maintained any more: #202 (comment)

[BigLep]

Reminder for anyone watching this issue that populating and reviewing this issue is tracked in #174

[Jorropo]

Feedback:

ipfs/go-unixfs	libipfs/unixfs

I think it should be libipfs/unixfs/something-something because we want to make more than one unixfs implementation.

[aschmahmann]

2023-03-14 standup conversation:

1. Don't include ipfs/go-ipfs-delay and ipfs/go-detect-race since aren't updated and we aren't planning to refactor.
2. Bring interface-go-ipfs-core into go-libipfs for now but then clean it up further after.
3. ipfs/go-filestore and ipfs/go-ipfs-posinfo - seems reasonable to bring in so we can refactor with the code that leverages it in go-unixfs
4. ipfs/go-fs-lock - leave it alone (or maybe bring into kubo)
5. ipfs/go-ipfs-cmds - leave alone, useful for someone building a daemon and not something go-libipfs refactors should touch
6. ipfs/go-peertaskqueue and ipfs/go-ipfs-pq - leave alone for now, we'll copy if we need to
7. ipfs/bbloom - leave alone
8. ipfs/go-ipfs-blocksutil - leave alone, only really used for tests

[guseggert]

I have updated the table based on the notes above, thank you for the feedback.

[lidel]

• re: interface-go-ipfs-core and comment from @hacdias – yes its a fankencode at this point, will take time to clean up, but that will be easier once we move it to libipfs/coreiface as planned in the table above
  • @hacdias afaik ErrOffline is used in Kubo when you ask for a block that is not in local repo, but you run with ipfs daemon --offline – so its Kubo-specific

[guseggert]

Random thought this morning: moving interfaces may silently break fx plugins that use fx.Replace or fx.Decorate to replace implementations at runtime. For consumers who upgrade Kubo without upgrading their plugin to use the libipfs interface, the result would be that their custom implementation would suddenly be ignored. Not a blocker, but we can communicate this in release notes.

[BigLep]

Not a blocker, but we can communicate this in release notes.

Good callout Gus. Is there a WIP branch this work is starting in for Kubo. Maybe add a "TODO" in changelog 0.20.md so we don't forget?

[BigLep]

Per 2023-04-11 maintainer conversation, we won't be doing the steps of copying in issues and marking repos as "not maintained" until after IPFS Thing.

[BigLep]

We are starting the "not maintained" campaign. I updated #202 (comment) with links to our improved docs/FAQs.

[BigLep]

@guseggert : I wanted to make sure you saw my comments in ipfs/interface-go-ipfs-core#113 (comment) as I believe they'll be relevant for each repo we're touching as part of this.

[guseggert]

I've added a tool to Boxo which adds deprecated notices to all of the exported types in a repo, which should make it quick and less error-prone to deprecate a repo. Run the tool, update the readme, cut a release. #297

[hacdias]

@BigLep all repositories are now deprecated (types deprecated and release) and have the message in the README. Thanks @Jorropo for reviewing.

[BigLep]

@hacdias and @Jorropo : thanks for getting the readme messages and deprecated types and releases handled. Good stuff!

The last remaining item is to handle the open issues/PRs and copy in anything we think is relevant.

I put all open issues/PRs from the repos into a board so can easily track status: https://github.com/orgs/ipfs/projects/27/views/1
(python gist to do this)

I removed PRs/issues I know we're not going to touch (e.g., dependabot, unified CI).

We look to have ~120 items to go through.

I added two custom fields:

• consolidation owner - We can use this to assign ownership of figuring out what we're doing with the issue. I did this in case we didn't want to overload the existing "assignee" value. We can switch to just using "assignee" though.
• next step - With this we can be clear about what is next for a given issue/PR (even if we're not closing it). I have these values (and we can add more):

I know we don't have bandwidth to process these yet, but at least have this in place where can track.

Let me know if you have any questions/suggestions.

[hacdias]

Looking at the board, these are the repositories that have issues that need to be handled:

Repository	±#Issues	Owner	Done
go-blockservice	3	@Jorropo	✔️
go-fetcher	3	@Jorropo	✔️
go-filestore	1	@lidel	✔️
go-ipfs-blockstore	8	@lidel	✔️
go-ipfs-chunker	5	@hacdias	✔️
go-ipfs-ds-help	1	@aschmahmann	✔️
go-ipfs-exchange-interface	1	@aschmahmann	✔️
go-ipfs-pinner	1	@aschmahmann	✔️
go-ipfs-posinfo	1	@hacdias	✔️
go-ipfs-provider	6	@hacdias	✔️
go-ipfs-routing	1	@hacdias	✔️
go-ipfs-util	1	@Jorropo	✔️
go-ipns	6	@hacdias	✔️
go-mfs	25	@aschmahmann	✔️
go-namesys	4	@hacdias	✔️
go-path	4	@hacdias	✔️
go-pinning-service-http-client	6	@Jorropo	✔️
go-unixfs	16	@Jorropo	✔️
go-verifcid	3	@hacdias	✔️
interface-go-ipfs-core	21	@lidel	✔️

I think it makes sense to assign a single person to each of the repositories.

Messages

Here are some messages that I think we can use for the issues. I already took care of a few issues and have used this messages.

Merged with others

This repository is no longer maintained and has been copied over to [Boxo](https://github.com/ipfs/boxo). We are now in the process of reviewing issues and PRs and moving the ones that are still relevant. This issue has been merged with a few other issues, and moved to **LINK**.

(Learn more at the [FAQs for the Boxo repo copying/consolidation effort](https://github.com/ipfs/boxo/wiki/Copied-or-Migrated-Repos-FAQ).)

Closed

This repository is no longer maintained and has been copied over to [Boxo](https://github.com/ipfs/boxo). We are now in the process of reviewing issues and PRs and moving the ones that are still relevant. This issue is no longer relevant, and therefore is being closed.

(Learn more at the [FAQs for the Boxo repo copying/consolidation effort](https://github.com/ipfs/boxo/wiki/Copied-or-Migrated-Repos-FAQ).)

[BigLep]

@hacdias : I like the plan. Looks like we just need to assign owners to repos now and a deadline to get it done by :)

Concerning messaging:

1. I think it would generally be good to add a layer of indirection for messaging so we can add more FAQs if things emerge. To that end, maybe add something like this to each prompt: ``(Learn more at the FAQs for the Boxo repo copying/consolidation effort.)```
2. For the "Moved" bucket, assuming you transferred the issue, the old URL will still work and the new URL will be self-evident. You won't need "This issue has been moved to LINK."

Is there anything special you want people to do for closing issues or setting next steps in https://github.com/orgs/ipfs/projects/27/views/3 ? (No pressure to use it - just asking so it's clear for maintainers.)

[hacdias]

@BigLep

Is there anything special you want people to do for closing issues or setting next steps in github.com/orgs/ipfs/projects/27/views/3 ?

Not really, I just need to assign people to the repositories and ping them.

[BigLep]

2023-06-06 maintainer conversation:
Realistically there is very few if any of these repos where we'd make code changes in Boxo the next 6 months. As a result, the approach will be:

1. Assign triage owners for each repo
2. Triage owners look for clear and pure gold that should be migrated. Otherwise close and invite people to open a new issue in Boxo. Messaging could look something like:

This repository is no longer maintained and has been copied over to [Boxo](https://github.com/ipfs/boxo).  In an effort to avoid noise and crippling in the Boxo repo from the weight of issues of the past, we are closing most issues and PRs in this repo.  Please feel free to open a new issue in Boxo (and reference this issue) if resolving this issue is still critical for unblocking or improving your usecase.

You can learn more in the [FAQs for the Boxo repo copying/consolidation effort](https://github.com/ipfs/boxo/wiki/Copied-or-Migrated-Repos-FAQ).

FYI that I added a FAQ entry: https://github.com/ipfs/boxo/wiki/Copied-or-Migrated-Repos-FAQ#why-were-most-of-the-existing-open-issues-and-prs-not-moved-over-to-boxo

[hacdias]

@ipfs/maintainers just wanted to check if the plan is to remove boxo/ipld/car and point people at ipld/go-car instead. AFAIK, we decided to keep only the copy in go-car and to remove ours. But I haven't seen that work yet. I think we should prioritise it to limit confusion.

[BigLep]

@hacdias :

just wanted to check if the plan is to remove boxo/ipld/car and point people at ipld/go-car instead. AFAIK, we decided to keep only the copy in go-car and to remove ours. But I haven't seen that work yet. I think we should prioritise it to limit confusion.

Agreed this needs to close out and remove boxo/car. We're unblocked for this now. I added a comment: #218 (comment)

[BigLep]

I realized this issue never made clear that when we copied in repos, we copied in their commit history as well. https://github.com/guseggert/repo-migration-tools was used. I added an FAQ entry about this as well: https://github.com/ipfs/boxo/wiki/Copied-or-Migrated-Repos-FAQ#was-the-git-history-maintained-for-repos-that-were-copied-in

[lidel]

Q: are we archiving repos after we are done with issue cleanup at https://github.com/orgs/ipfs/projects/27/views/6?

I did not see it on the checklist, but I did that for:

• https://github.com/ipfs/go-filestore (kubo-specific experiment, moved to boxo
• https://github.com/ipfs/interface-go-ipfs-core (kubo specific, moved to boxo and/or kubo/client/rpc)
• https://github.com/ipfs/go-ipfs-blockstore (moved to boxo, nobody did anything here since 2021)

Why I think archiving is a good call

Looked at commit history, there was no active development, just legacy of issues, no maintainer.

My understanding was that there is no point in cleaning up issues and PRs if someone opens new issue next week,
and we have to triage two places every Monday during GO Triage, but lmk if something else was agreed.

[BigLep]

Thanks @lidel ! I'm not against archiving especially if we are the only ones who realistically have been contributing. The approach that was originally laid out was just to make clear that the repo isn't maintained and point to the boxo version and then come back through later and archive. We aren't on the hook for triaging anything in the repos. We had this FAQ entry: https://github.com/ipfs/boxo/wiki/Copied-or-Migrated-Repos-FAQ#will-the-not-maintained-ipfs-repos-be-left-around-to-rot

I realize though that it may not be clear for people if the repo isn't archived and there is just a signpost message. If it makes more sense to just archive the repo now, I'm not against it. I didn't push on archiving now to minimize the disruption where possible at this point.

[hacdias]

Agreeing with @lidel, I also went again through my assigned repos and archived them too. Most of them were go-ipfs specific and/or did not have an active commit history. I think it's best to do that, than just putting a sign in the README. It's more clear. If someone wants a repository, they can reach us out and we can always unarchive it.

[hacdias]

Updates:

• All issues are reviewed and ported if needed. Closed the project at https://github.com/orgs/ipfs/projects/27.
• All repositories in this list have been archived.
• Updated the table in the main comment of this issue to reflect the current status.

I'm closing this issue as it seems complete. Thanks to everyone involved for the time reviewing issues and moving PRs around.