Make this repo public

[BigLep]

Done Criteria

This repo or some slice of if it is public.

Why Important

1. Enable others to learn from it so they don't have to reinvent the wheel
2. We reference this repo from other public repos (recent example: chore(build): trim and compress genesis CARs in build lotus#12439 and feat: use dnsaddr for butterflynet bootstrap addresses lotus#12448)
3. (bonus/maybe) get external contributions

Notes

1. The key thing is to ensure we're not exposing any secrets.

Tasks

Tasks

Preview Give feedback

1. Decide on approach
2. Create lotus-infra-archive
3. Give context on lotus-infra-archive: https://github.com/filecoin-project/lotus-infra-archive/pull/1
4. Update lotus-infra readme to be public facing: docs: update readme with brief repo history #1636
5. Prepare script for bulk transferring issues: Make this repo public #1631 (comment)
6. Bulk transfer issues
7. Mark lotus-infra-archive as archived: Make lotus-infra public and archive lotus-infra-archive github-mgmt#78
8. Mark lotus-infra as public: Make lotus-infra public and archive lotus-infra-archive github-mgmt#78

[BigLep]

@masih : I created this issue so we could leave any notes if/as we discover why we can't make this public without more work.

I didn't capture enough of the details of what could potentially go wrong if we made this public today. I know there is stuff we'd ideally want to separate out, but if there is no harm in someone seeing it, I would vote for getting this out there so folks who are tracking Lotus PRs referencing issues here can see them. Basically, I want to make sure we note the worst that could happen if we made this public today. (I'm not saying we'll do that, but I'm asking for understanding.)

[masih]

Thank you for capturing this. I too would like to make this repo public. I'll timebox to take a closer look at this by EOW.

[masih]

I have reviewed the code in this repository and generally I am happy for the code to be made public, with the following caveats:

• This repository uses a combination of Terraform, bash scripts and Ansible to manage butterflynet deployment. Revealing them would show the inner workings of how the system is set up and perhaps can make it easier to find misconfigurations that could ultimately result in security incidents. Having said that, butterflynet is ephemeral in nature, and is reset often.
• The way Ansible manages sensitive information is using encryption at rest where the vault key is shared across engineers and CI to be able to interact with the inventory. The fact that the private key is shared in this way is itself a security weak-link. If for whatever reason that key leaks we will need to rotate all the encrypted files in this repo along with the butterflynet network. The repo uses appropriate gitignore entities to avoid accidental leakage. Nevertheless having a single key plus making encrypted sensitive data public would increase the risk of leakage.
• The terraform state files can contain sensitive information. All the state files use remote state management on S3, except one (the remote s3 storage itself). The state files are not checked into this repo and are appropriately ignored by git to avoid accidental commits.
• On the off chance that some secret ends up on any pushed git commit we will need to rotate the secret entirely. Removing the branch / commit is not enough.
• I have not reviewed the commit history of this repo. This repo is very old relatively speaking and has gone through significant changes most of which have been made redundant removed. I point this out as a small reminder that git history will also be made public which may or may not be desirable. We can re-create this repo with a different name for example if revealing history poses a concern.

In terms of partial separation, separating terraform or ansible from this repo would leave a public observer sufficiently confused about the bits that remain private and I don't think that quite achieves what we want to achieve in this issue.

[rjan90]

I have not reviewed the commit history of this repo. This repo is very old relatively speaking and has gone through significant changes most of which have been made redundant removed. I point this out as a small reminder that git history will also be made public which may or may not be desirable. We can re-create this repo with a different name for example if revealing history poses a concern.

There is also a lot of closed out issues, and I'm not sure what kind of information has been shared in these historical issues: https://github.com/filecoin-project/lotus-infra/issues?page=1&q=is%3Aissue+is%3Aclosed. Re-creating the repo with a different name seems ideal, just in the event that some of these issues were never intended to see the public.

[BigLep]

Thanks for the writeup @masih and review @rjan90 . I'm summarize the risks in my head as:

1. This repo has always been private and thus it's hard to know what in the past (whether issues, PR comments, or commit history) lurks that shouldn't be exposed. The "this repo is private" may have enabled cutting corners or not cleaning things up, and the current team members don't have full history to have a strong enough pulse.
2. Sensitive information has some guardrails in place (e.g., encryption at rest, gitignores), but it isn't fool-proof and doesn't have multiple layers of protection. As a result, it's certainly feasible a secret gets leaked which then requires key rotations. This would be annoying to do (an unwanted use of time), but is not catastrophic given the butterfly nature.

Here's how I'm thinking about the options:

Dimension	Proposal 1	Proposal 2	Proposal 3	Proposal 4
What is it	Do Nothing	Rename this repo to lotus-infra-priv or lotus-infra-archive Create a new public repo starting from the current state that we use going forward. Keep this repo private. Transfer open issues from this repo to the new public repo. Archive this repo.	Create a new private repo (lotus-infra-archive). Transfer old issues to it. Archive lotus-infra-archive. Make this repo public.	Make this repo public as-is (no risk mitigations)
Ability work in public	🔴 Poor. Same as current.	🟢 Good	🟢 Good	🟢 Good
Risk of exposing information that didn't mean to	🟢 None	🔵 Low since no git or issue history	🟡 Medium since git history is still in place and any comments in old PRs since PRs can be transferred.	🟠 Highest given the repo had years of work under the assumption of being a private repo
Developer time to execute	🟢 None	🟡 Most time will be spent on setting up the ~10 secrets that the repo needs (see here). A little bit of time transferring open or recently closed issues.	🟡 Most time will be spent transferring old issues to the archive repo. (There is no bulk transfer option in GitHub UI but can easily use ChatGPT script to use GitHub API.)	🟢 Almost none (quick PR to github-mgmt to change visibility)
Developer time in case of secret "check-in"	🟢 None	🟡 Some since need to revoke leaked secret and update secrets in GitHub	🟡 Same	🟡 Same
README updates	Add note about why this repo is private and link to this issue	Add note to both repos explaining their relationship and purpose.	Add note to both repos explaining their relationship and purpose.	Add a note about the purpose of this repo

Are there other dimensions or proposals we should consider?

Maybe one other step we can take is to scan issues/prs/commits for keywords like:

• revert
• secret
• password

I did look through these and didn't see anything of concern but I don't have a super trained eye here.

Assuming the above check comes up clear from @masih or @rjan90, my vote would be to go with "Proposal 3" because it isn't disruptive to current work, has us working in public, and does some risk mitigation.

That said, this isn't absolutely essential to do, so I can understand if the sentiment is "lets not expose ourselves to unnecessary sidetracks in case there is something in the commit history that would be really problematic to expose".

[masih]

+1 for proposal 3. Moving issues to a new repo should be quick. I believe there is a built in support for it in github and there are not that many.

[BigLep]

To execute on proposal 3, I created a task list in the issue description. Various PRs have been opened. I'll next work on identifying the issues to migrate/transfer and the commands for doing so.

[BigLep]

I propose to:

1. keep all issues created in 2024 (regardless of whether opened or closed) and all open issues: https://github.com/filecoin-project/lotus-infra/issues?q=is%3Aissue+created%3A%3E2024-01-01+
2. transfer all closed issues that were created before 2024: https://github.com/filecoin-project/lotus-infra/issues?q=is%3Aissue+is%3Aclosed+created%3A%3C2024-01-01+sort%3Acreated-desc

The command I would run on the command line to do the transfer is:

# Inspired from https://jloh.co/posts/bulk-migrate-issues-github-cli/
cd lotus-infra
gh issue list --search "is:issue is:closed created:<2024-01-01 sort:created-desc" -L 600 --json number \
    jq -r '.[] | .number' | \
    xargs -I% gh issue transfer % https://github.com/filecoin-project/lotus-infra-archive

You can confirm that this is the set of issues you'd expect with gh issue list --search "is:issue is:closed created:<2024-01-01 sort:created-desc" -L 600

@rjan90 or @masih : once I get review/approval from you of the above command, I will execute it.

[BigLep]

@masih @rjan90 : once I get approval for the items in the task list, I'll execute to close this out (assuming there isn't more feedback to incorporate first).

[rjan90]

The checklist items looks good to me, and I have given a approval a couple of them already, except for:

• Prepare script for bulk transferring issues: Make this repo public #1631 (comment)
• Mark lotus-infra-archive as archived: Make lotus-infra public and archive lotus-infra-archive github-mgmt#78
• Mark lotus-infra as public: Make lotus-infra public and archive lotus-infra-archive github-mgmt#78

I will address those tomorrow 2024-09-20.

[masih]

@BigLep Just to confirm my understanding:

• What is the final repo name that will be made public?

If we intend to use lotus-infra avoid broken links (assuming the rename dance below would work and github auto redirect does not mess with things if a new repo with the same name is created), then I think the task list should include:

• rename the current lotus-infra to something other than that name.
• create a new repo, named lotus-infra.
• transfer sources.
• transfer issues.

Or is there something that I have missed?

[BigLep]

@masih : my thinking is that this current filecoin-project/lotus-infra repo would be made public. A subset of the issues would be transferred to lotus-infra-archive. lotus-infra-archive will be private and be in archived stated. GitHub will automatically handle redirection from https://github.com/filecoin-project/lotus-infra/issues/# links to issues that now live in lotus-infra-archive.

In summary, the only repo created is lotus-infra-archive but it doesn't have any source. It only has archived issues. I was planning to leave source (including history) completely in tact in this repo as is. This is what was meant by proposal 3.

[masih]

the only repo created is lotus-infra-archive but it doesn't have any source. It only has archived issues.

The primary source of unknown risk is git history itself more than issues. Having re-read the proposal 3 I would classify risk of exposing unintended things as high since git history will be made public.

This is fine as long as we are ok with that risk.

[BigLep]

@rjan90 : gave verbal approval on 2024-09-20 of the issues I'm keeping around in #1631 (comment)

[BigLep]

the only repo created is lotus-infra-archive but it doesn't have any source. It only has archived issues.

The primary source of unknown risk is git history itself more than issues. Having re-read the proposal 3 I would classify risk of exposing unintended things as high since git history will be made public.

This is fine as long as we are ok with that risk.

Fair, I agree the risk is higher than I highlighted it in the original table.

Per verbal with @rjan90 on 2024-09-20, I'm still going to proceed given:

1. No hard objections
2. We imagine there are easier / more "juicy" to target than this repo's commit history
3. We'll have secret scanning turned on for the repo
4. Many of the things that this repo had have been moved out (e.g., Chainsafe taking over snapshots)
5. If there is a resource that gets abused, we will take on mitigation of rotating secrets.

[BigLep]

I transferred as many issues as possible with:

gh issue list --search "is:issue is:closed created:<2024-01-01 sort:created-desc" -L 600 --json number | \
    jq -r '.[] | .number' | \
    xargs -I% gh issue transfer % https://github.com/filecoin-project/lotus-infra-archive

Some issues refuse to migrate (even through the UI). They are:

Nothing stood out to me. Anything that is of concern should just be deleted.

[BigLep]

This should be resolved once github-mgmt workflow runs: https://github.com/filecoin-project/github-mgmt/actions/runs/10966021272

[BigLep]

Closing as repo is now public