-
Notifications
You must be signed in to change notification settings - Fork 80
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
proposal for build multi arch images #159
Changes from 3 commits
301d2dd
c73be51
e6dccd5
95f1485
f4cb2ad
8067b12
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,123 @@ | ||
# Proposal: `Multi Arch Support` | ||
|
||
Author: `Zhipeng Yu / yuzp1996` | ||
|
||
Discussion: `none` | ||
|
||
## Background | ||
|
||
More and more companies need to support multi-architecture solutions, some of the more mainstream x86 and arm64 architectures, this proposal is to provide a solution to produce multi-architecture images | ||
|
||
## Plan | ||
|
||
When the release 2.3 of Harbor is released, the arm64 Harbor can be issued at the same time | ||
|
||
Release 2.3 of Harbor is tentatively scheduled to be released in mid-June 2021 | ||
|
||
## Target | ||
|
||
### Short Term Target | ||
|
||
We can provide support for different architectures in a different Harbor sub-projects. | ||
|
||
For example, in the harbor-arm repo, we can maintain the build logic and other related information for arm64 images | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Why would it need to be in separate repositories? (I've built and run Harbor on arm64 successfully, and it didn't require any code changes -- just a base image that was supported, as seen in goharbor/harbor#13788) I think the PostgreSQL major version upgrade problem we discussed over there in goharbor/harbor#13788 (comment) is very relevant to this work -- without some way to upgrade between major versions of PostgreSQL, this is going to get very complicated very quickly (and is the reason that PR got more complicated and added a separate There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. We will discuss major version upgrades of PostgreSQL at a meeting every Wednesday at 5 pm. You are welcome to join in. The meeting link is https://VMware.zoom.com.cn/j/94528344426?pwd=aVpmMWtweHNXenRBOFhQNjgvQlZ0UT09 There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Oh, nice! Which timezone is that in? There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Sorry, I didn't realize this. 😂 We are in Beijing now. We discussed your mention of PostgreSQL in comment. As far as I know Harbor Team will provide some solution to resolve this problem in two weeks. But we are not sure if we will encounter the same problem when building arm64 image base photon 4.0. We need do a test about this. If Harbor Team can provide a solution to solve this problem within two weeks, then this may not be a problem any more There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Yeah, fair enough -- if upstream stays on photon 2.0 and downstream is on photon 4.0, neither will have an issue today, but it's one more delta between upstream and downstream, and it will be a serious issue for both eventually (photon 2.0 won't be maintained forever, and even it is, the PostgreSQL version in it won't be 😅). There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Sorry ! I am not sure if I understand the delta between upstream and downstream that you have mentioned. 😭 The base images in upstream have upgrade to photon 4.0 in this PR which has been merged to master. https://github.com/goharbor/harbor/pull/14551/files I think we will develop based on upstream master branch. Then the arm images will use the latest version of PostgreSQL in photon 4.0 so maybe we don't need to care about the major version problem for now? If I did not understand the question you mentioned correctly please feel free to correct me Thank you very much! There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Ohhhh, I hadn't seen goharbor/harbor#14551 yet 😬🙈 That's definitely going to be a problem for upgrades unless the PostgreSQL upgrade problem is solved. 😬😬😬 |
||
|
||
### Long Term Target | ||
|
||
When the harbor is released, images of different architectures can be directly generated and push to registry. When deploying harbor, users do not need to distinguish the difference in architecture | ||
|
||
## Implementation | ||
|
||
### Glossary | ||
|
||
Upstream: github repository goharbor/harbor | ||
|
||
Downstream: github repository like harbor-arm harbor-loongson and so on | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Again, not understanding the need for these to be separate? If they're treated as downstreams, they'll always be second-class citizens. If they require code changes, the ideal would be to get those upstream, and have those tested upstream too. There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. The solution you mentioned is an ideal solution. However, Harbor Team has no plans to do multi-architecture support in the upstream this year, because the workload brought about by this is huge and it is easy to cause delays in release. There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Maybe a reasonable middle-ground would be adding a GitHub Action upstream that just verifies that the code still compiles successfully for any architectures that get built/tested downstream? I think Go's cross-compilation support is probably enough to accomplish that without QEMU or any major slowdown, and it would help make sure that the downstream project is more likely to have success. |
||
|
||
### Upstream changes | ||
|
||
#### Goal | ||
|
||
Provides the ability to compile binary and image of a specific architecture based on environment variable parameters. (In the initial stage, the upstream only guarantees that the image of the amd64 architecture can be successfully built, and other versions are guaranteed by the corresponding downstream repository) | ||
|
||
#### Changes | ||
|
||
1. Increase the environment variable GOARCH, the default is amd64, which can be overwritten during execution | ||
|
||
2. When executing make compile to generate binary, it can generate binary of different architectures according to the environment variable GOARCH | ||
|
||
3. Add a file named adapter.sh, the content of the file is empty. Before executing compile and build commands, please call adapter.sh scripts. This shell script file can be overwritten by downstream users, and downstream users can write their own modifications in this file, such as overriding the addresses of third-party dependencies or replacing Dockerfile. | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. This is really strange IMO -- Go has top-notch, first-class support for multiple architectures and cross-compiling, so it really feels like a backwards approach to treat them as secondary and even downstream. There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. The upstream will be responsible for compiling different architecture binaries, while the downstream will not be responsible for this The downstream will use the way that upstream provided to get the different architecture binary. There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. I don't quite understand why we need to put this upstream. Could you give a concrete example as to what this script may look like and what it does? There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. The file will not perform any operations in the upstream. This is just an empty file. It looks like a placeholder and should be overwritten by downstream repositories. There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. I believe what @tianon and @reasonerjt mean is that this script should not be called in the upstream at all. Since the downstream repository is responsible for executing the whole build process, it is enough to have the There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. no problem. I agree with this |
||
|
||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Are there any considerations, as to what upstream should be careful with, in order not to break the down stream workflows? There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. There is nothing that upstream to care about because it is an empty file and should be overwritten by the downstream repository. Downstream will care about the content of this file. In the upstream repository just need call this shell file before executing the compilation and build tasks in the makefile to ensure that this shell can be executed. There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. I believe that the focus of the question here is not regarding the script, but regarding any general changes in the core project, for example:
|
||
### Downstream changes | ||
|
||
#### Goal | ||
|
||
Maintain the build and test of an architecture images | ||
|
||
#### Change | ||
|
||
This comment was marked as outdated.
Sorry, something went wrong. There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. readme in upstream can link to there |
||
1. Introduce the harbor repository into the downstream repository through git Submodules | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. It might be personal, but I've never had anything but a bad time using Git Submodules, especially at larger scales (which I think is another strong argument against having other architectures support be totally separated from the core product code). One concrete issue is that they're tied to a specific commit without any real history or notation of intent, so it becomes difficult to determine which branch that commit comes from and why that branch was chosen, for example. There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. maybe we can suggest a better way to achieve the same goals? I believe that the requirement here is to build downstream repos on upstream have the following changes:
using @tianon any suggestion how can we achieve that using Github Actions will be of great help There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. From what I can find, the only way to get the downstreams to build automatically whenever there's a commit on the upstream is to add something in the upstream's GitHub actions with an access token for the downstream and have it use that token to explicitly trigger an event like No matter how this downstream repository is consuming the upstream code, it's going to be complicated to coordinate -- I imagine each release of Harbor is going to require slightly different downstream code (like the addition of the new metrics service in 2.2, for example), so I'm not sure what the best solution for mapping those and keeping them up to date is going to be. 😬 |
||
|
||
2. Add a file named adapter.sh, responsible for making changes to areas that cannot be covered by upstream changes | ||
|
||
3. Add a Makefile (which can be called when performing github actions) to achieve such as overwriting adapter.sh or other functions | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. maybe there will be many dockefiles |
||
|
||
4. Add Dockerfile if necessary | ||
|
||
5. Add github action files to maintain the build and test in them. The content of the github action should be modified based on the upstream github action, but it should be kept in sync at all times | ||
|
||
### Test | ||
|
||
We want to use the upstream test method to test, use docker-compose to deploy Harbor and then use the shell script to execute the test | ||
|
||
However, there are currently the following issues that need to be resolved | ||
|
||
1. During testing, docker-compose will be used to deploy Harbor and test, but it seems that docker-compose [only provides x86 binary files](https://github.com/docker/compose/releases), and we need to compile other versions by ourselves. We can try to install docker-compose with pip | ||
|
||
2. Images of different architectures may not be able to run using the runner provided by github action. [Self-hosted runners](https://docs.github.com/en/actions/hosting-your-own-runners) should be used to ensure that images of different architectures can run tests. | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. block issue There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. FWIW, Travis has support for a few alternative architectures, but the product quality/reliability has seen a steady decline since their acquisition. We might be able to use the Alternatively, now that Harbor is part of the CNCF sandbox, we might be able to convince them to sponsor something like an Amazon A1 instance for a self-hosted arm64 runner. There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Thanks for your advice. We may solve this problem through the third comment you mentioned. |
||
|
||
|
||
### Image strategy | ||
|
||
#### Repository | ||
|
||
The image will be hosted under the goharbor project of Docker Hub | ||
|
||
#### Tag | ||
|
||
Add the architecture name after the generated image version. For example: goharbor/harbor-core:v2.2.1-arm64 | ||
|
||
|
||
### Related Implementation | ||
|
||
There are already some PRs and repository that have implemented this function, we can make full use of these contents | ||
|
||
Changes provided by this [PR](https://github.com/goharbor/harbor/pull/13788) can build amd64 and arm64 image simultaneously | ||
|
||
This repository https://github.com/querycap/harbor have created arm64 and amd64 image successfully | ||
|
||
|
||
|
||
## Development Iteration | ||
|
||
After several iterations, we need to ensure that the content in adapter.sh and github actions files in downstream repository is as small as possible,After each iteration, more code can be extracted and merge to upstream。 | ||
|
||
After the adapter.sh and github actions files of the downstream code disappear, we can no longer maintain the downstream repository. | ||
|
||
|
||
|
||
## Release | ||
|
||
When the goharbor/harbor is released, check out the same release branch under the downstream repository, and check the corresponding commit in git Submodules, execute the pipeline to generate the corresponding image and push it to the docker hub | ||
|
||
Readme.md or release notes needs to be updated in different branches to inform the images related information, such as location and corresponding tags | ||
|
||
|
||
## Notify Users | ||
|
||
We provide repository addresses that support other architectures in goharbor/harbor release notes. Users can directly find repository addresses of different architectures according to the link. | ||
|
||
|
||
## Block Item | ||
|
||
1. Images of different architectures may not be able to run using the runner provided by github action. [Self-hosted runners](https://docs.github.com/en/actions/hosting-your-own-runners) should be used to ensure that images of different architectures can run tests. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Maybe add a table describing an overview of the maturity model? For example:
- Introduce development practices
- Upstream changes trigger downstream build
- Ensure upstream master code have at least a daily build and test
- Run release tests in the new architecture
- Enable master build and PR build checks
- Active maintainance on upstream
- Offer official multi-architecture tag for users
- Offiline installer build process and architecture