build: add OpenAPI spec bundling, stop using URL references #3288
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
petermetz
requested review from
RafaelAPB,
takeutak,
izuru0,
jagpreetsinghsasan,
VRamakrishna,
sandeepnRES and
outSH
as code owners
jagpreetsinghsasan
approved these changes
Jun 4, 2024
petermetz
force-pushed
the
chore-vendor-openapi-json-files-v200alpha2
branch
from
12ef49d
to
7ef7302
Compare
|
@hyperledger/cacti-maintainers I've found a more efficient and less brittle way to do this type of vendoring automatically by way of this concept called bundling the open API specifications. It was originally a feature of the (now deprecated) swagger-cli tool, but the successor of it also implements it and it appears to work seamlessly based on my tests. I'll send an update to this pull request shortly with the refactored version which will be slightly less verbose but equivalent functinality. |
petermetz
force-pushed
the
chore-vendor-openapi-json-files-v200alpha2
branch
from
7ef7302
to
41b5fa4
Compare
|
@hyperledger/cacti-maintainers I've pushed the latest changes and now this is once again ready for your review. I made sure to update the pull request description with the new changes, please re-read carefully! |
petermetz
changed the title
petermetz
added a commit
to petermetz/cacti
that referenced
this pull request
Jun 12, 2024
On a high level this is a find & replace operation where the occurrences of the first bullet point were replaced with the second bullet point: * `"$ref": "https://raw.githubusercontent.com/hyperledger/cactus/v2.0.0-alpha.2` * `"$ref": "../../../../..` The firs bullet point above is called a URL reference while the second one is called a REMOTE references (remote as in a different spec file on the file-system). 1. With this change, we unlock the release process being able to issue code that is working on the latest OpenAPI specifications that we are cross-referencing from one package to another. 2. Previously you had to manually update the references in about a hundred and fifty locations to make sure that the versions are bumped but after this change this happens automatically as the newly introduced bundling process and the usage of the REMOTE references instead of URL references. 3. The problem so far with the release process was that with the URL references we dependended on the existence of a pushed git tag for a successful release build. But we cannot git push the tag before having performed a successful release build, so this was a chicken-egg problem that had to be somehow untangled from its circular dependency hell and this change is what makes it happen by no longer depending on the git tags having been pushed to the upstream repository. Related to, but does not yet fix: https://github.com/hyperledger/cacti/issues/2175 Depends on hyperledger-cacti#3288 Signed-off-by: Peter Somogyvari <peter.somogyvari@accenture.com>
5 tasks
petermetz
added a commit
to petermetz/cacti
that referenced
this pull request
Jun 12, 2024
1. After this change the steps within the release management documentation should work without issues. 2. Currently the process is (was) broken due to our reliance on URL references within the OpenAPI specifications which created a chicken-egg problem with the release tag issuance and the building of the source code to be released. This change depends on the other pull requests that are refactoring the cross-package OpenAPI specification references: Depends on hyperledger-cacti#3288 Depends on hyperledger-cacti#3315 Signed-off-by: Peter Somogyvari <peter.somogyvari@accenture.com>
5 tasks
outSH
approved these changes
Jun 13, 2024
There was a problem hiding this comment.
@petermetz Looks great, sorry for late review. I've left few comment but none of them are critical, you can look into them in future PR or here if there's no rush :)
petermetz
force-pushed
the
chore-vendor-openapi-json-files-v200alpha2
branch
from
41b5fa4
to
f91a8be
Compare
**IMPORTANT:** From now on, if you are changing the OpenAPI specification of any given package within Cacti, please make sure to edit the template file instead of editing the openapi.json specific file directly because changes in the openapi.json file will be overwritten by the codegen script the next time you run it. This slight alteration in the development flow is the least intrusive solution I could find to resolving our issues with the release automation. This change enables us to have our openapi.json files work without having remote and URL references in them (which was a blocker issue for release automation). 1. The openapi.json files that we used to have are now called openapi.tpl.json where the tpl stands for template. Their content is equivalent to what openapi.json files used to have prior to this commit. 2. These template specs are fed into the bundler tool which then spits out the files which then are saved as openapi.json files. The big change is that these bundled versions are no longer containing any remote nor URL references, only local ones. 3. This means that we still get project-wide re-use of schema types from packages such as cactus-core-api, but we no longer suffer from the additional complexities of having to deal with remote and URL references. 4. The scirpt that performs the bundling is callable separately by executing this command ```sh yarn tools:bundle-open-api-tpl-files ``` 5. The `yarn tools:bundle-open-api-tpl-files` is also embedded as a warmup step of the larger `codegen` script so there is no need usually to call the bundling script separately. 6. The heavylifting in terms of bundling is done by the tooling script that can be found here: `tools/bundle-open-api-tpl-files.ts`. On a high level what it does is loop through existing `openapi.tpl.json` files throughout the project and then renders their bundled version next to it as `openapi.json` which then can be used by all of our tools as a self contained version of the template file which *does* still have the remote and URL references in it. More information on what URL and remote references are can be read here on the official OpenAPI website: https://swagger.io/docs/specification/using-ref/ Signed-off-by: Peter Somogyvari <peter.somogyvari@accenture.com>
petermetz
force-pushed
the
chore-vendor-openapi-json-files-v200alpha2
branch
from
f91a8be
to
870634b
Compare
petermetz
added a commit
to petermetz/cacti
that referenced
this pull request
Jun 13, 2024
On a high level this is a find & replace operation where the occurrences of the first bullet point were replaced with the second bullet point: * `"$ref": "https://raw.githubusercontent.com/hyperledger/cactus/v2.0.0-alpha.2` * `"$ref": "../../../../..` The firs bullet point above is called a URL reference while the second one is called a REMOTE references (remote as in a different spec file on the file-system). 1. With this change, we unlock the release process being able to issue code that is working on the latest OpenAPI specifications that we are cross-referencing from one package to another. 2. Previously you had to manually update the references in about a hundred and fifty locations to make sure that the versions are bumped but after this change this happens automatically as the newly introduced bundling process and the usage of the REMOTE references instead of URL references. 3. The problem so far with the release process was that with the URL references we dependended on the existence of a pushed git tag for a successful release build. But we cannot git push the tag before having performed a successful release build, so this was a chicken-egg problem that had to be somehow untangled from its circular dependency hell and this change is what makes it happen by no longer depending on the git tags having been pushed to the upstream repository. Related to, but does not yet fix: https://github.com/hyperledger/cacti/issues/2175 Depends on hyperledger-cacti#3288 Signed-off-by: Peter Somogyvari <peter.somogyvari@accenture.com>
petermetz
added a commit
that referenced
this pull request
Jun 14, 2024
On a high level this is a find & replace operation where the occurrences of the first bullet point were replaced with the second bullet point: * `"$ref": "https://raw.githubusercontent.com/hyperledger/cactus/v2.0.0-alpha.2` * `"$ref": "../../../../..` The firs bullet point above is called a URL reference while the second one is called a REMOTE references (remote as in a different spec file on the file-system). 1. With this change, we unlock the release process being able to issue code that is working on the latest OpenAPI specifications that we are cross-referencing from one package to another. 2. Previously you had to manually update the references in about a hundred and fifty locations to make sure that the versions are bumped but after this change this happens automatically as the newly introduced bundling process and the usage of the REMOTE references instead of URL references. 3. The problem so far with the release process was that with the URL references we dependended on the existence of a pushed git tag for a successful release build. But we cannot git push the tag before having performed a successful release build, so this was a chicken-egg problem that had to be somehow untangled from its circular dependency hell and this change is what makes it happen by no longer depending on the git tags having been pushed to the upstream repository. Related to, but does not yet fix: https://github.com/hyperledger/cacti/issues/2175 Depends on #3288 Signed-off-by: Peter Somogyvari <peter.somogyvari@accenture.com>
petermetz
added a commit
to petermetz/cacti
that referenced
this pull request
Jun 14, 2024
1. After this change the steps within the release management documentation should work without issues. 2. Currently the process is (was) broken due to our reliance on URL references within the OpenAPI specifications which created a chicken-egg problem with the release tag issuance and the building of the source code to be released. This change depends on the other pull requests that are refactoring the cross-package OpenAPI specification references: Depends on hyperledger-cacti#3288 Depends on hyperledger-cacti#3315 Signed-off-by: Peter Somogyvari <peter.somogyvari@accenture.com>
petermetz
added a commit
that referenced
this pull request
Jun 14, 2024
1. After this change the steps within the release management documentation should work without issues. 2. Currently the process is (was) broken due to our reliance on URL references within the OpenAPI specifications which created a chicken-egg problem with the release tag issuance and the building of the source code to be released. This change depends on the other pull requests that are refactoring the cross-package OpenAPI specification references: Depends on #3288 Depends on #3315 Signed-off-by: Peter Somogyvari <peter.somogyvari@accenture.com>
fazzatti
pushed a commit
to fazzatti/cacti
that referenced
this pull request
Jun 24, 2024
On a high level this is a find & replace operation where the occurrences of the first bullet point were replaced with the second bullet point: * `"$ref": "https://raw.githubusercontent.com/hyperledger/cactus/v2.0.0-alpha.2` * `"$ref": "../../../../..` The firs bullet point above is called a URL reference while the second one is called a REMOTE references (remote as in a different spec file on the file-system). 1. With this change, we unlock the release process being able to issue code that is working on the latest OpenAPI specifications that we are cross-referencing from one package to another. 2. Previously you had to manually update the references in about a hundred and fifty locations to make sure that the versions are bumped but after this change this happens automatically as the newly introduced bundling process and the usage of the REMOTE references instead of URL references. 3. The problem so far with the release process was that with the URL references we dependended on the existence of a pushed git tag for a successful release build. But we cannot git push the tag before having performed a successful release build, so this was a chicken-egg problem that had to be somehow untangled from its circular dependency hell and this change is what makes it happen by no longer depending on the git tags having been pushed to the upstream repository. Related to, but does not yet fix: https://github.com/hyperledger/cacti/issues/2175 Depends on hyperledger-cacti#3288 Signed-off-by: Peter Somogyvari <peter.somogyvari@accenture.com>
fazzatti
pushed a commit
to fazzatti/cacti
that referenced
this pull request
Jun 24, 2024
1. After this change the steps within the release management documentation should work without issues. 2. Currently the process is (was) broken due to our reliance on URL references within the OpenAPI specifications which created a chicken-egg problem with the release tag issuance and the building of the source code to be released. This change depends on the other pull requests that are refactoring the cross-package OpenAPI specification references: Depends on hyperledger-cacti#3288 Depends on hyperledger-cacti#3315 Signed-off-by: Peter Somogyvari <peter.somogyvari@accenture.com>
sandeepnRES
pushed a commit
to sandeepnRES/cacti
that referenced
this pull request
Jul 30, 2024
On a high level this is a find & replace operation where the occurrences of the first bullet point were replaced with the second bullet point: * `"$ref": "https://raw.githubusercontent.com/hyperledger/cactus/v2.0.0-alpha.2` * `"$ref": "../../../../..` The firs bullet point above is called a URL reference while the second one is called a REMOTE references (remote as in a different spec file on the file-system). 1. With this change, we unlock the release process being able to issue code that is working on the latest OpenAPI specifications that we are cross-referencing from one package to another. 2. Previously you had to manually update the references in about a hundred and fifty locations to make sure that the versions are bumped but after this change this happens automatically as the newly introduced bundling process and the usage of the REMOTE references instead of URL references. 3. The problem so far with the release process was that with the URL references we dependended on the existence of a pushed git tag for a successful release build. But we cannot git push the tag before having performed a successful release build, so this was a chicken-egg problem that had to be somehow untangled from its circular dependency hell and this change is what makes it happen by no longer depending on the git tags having been pushed to the upstream repository. Related to, but does not yet fix: https://github.com/hyperledger/cacti/issues/2175 Depends on hyperledger-cacti#3288 Signed-off-by: Peter Somogyvari <peter.somogyvari@accenture.com>
sandeepnRES
pushed a commit
to sandeepnRES/cacti
that referenced
this pull request
Jul 30, 2024
1. After this change the steps within the release management documentation should work without issues. 2. Currently the process is (was) broken due to our reliance on URL references within the OpenAPI specifications which created a chicken-egg problem with the release tag issuance and the building of the source code to be released. This change depends on the other pull requests that are refactoring the cross-package OpenAPI specification references: Depends on hyperledger-cacti#3288 Depends on hyperledger-cacti#3315 Signed-off-by: Peter Somogyvari <peter.somogyvari@accenture.com>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
IMPORTANT: From now on, if you are changing the OpenAPI specification of any given
package within Cacti, please make sure to edit the template file instead of editing the
openapi.json specific file directly because changes in the openapi.json file will be
overwritten by the codegen script the next time you run it.
This slight alteration in the development flow is the least intrusive solution I could find
to resolving our issues with the release automation.
This change enables us to have our openapi.json files work without having remote and URL
references in them (which was a blocker issue for release automation).
tpl stands for template. Their content is equivalent to what openapi.json files used to
have prior to this commit.
then are saved as openapi.json files. The big change is that these bundled versions are
no longer containing any remote nor URL references, only local ones.
cactus-core-api, but we no longer suffer from the additional complexities of having to deal
with remote and URL references.
yarn tools:bundle-open-api-tpl-filesis also embedded as a warmup step of thelarger
codegenscript so there is no need usually to call the bundling script separately.here:
tools/bundle-open-api-tpl-files.ts. On a high level what it does is loop throughexisting
openapi.tpl.jsonfiles throughout the project and then renders their bundledversion next to it as
openapi.jsonwhich then can be used by all of our tools as a selfcontained version of the template file which does still have the remote and URL references
in it.
More information on what URL and remote references are can be read here on the official
OpenAPI website: https://swagger.io/docs/specification/using-ref/
Signed-off-by: Peter Somogyvari peter.somogyvari@accenture.com
Pull Request Requirements
upstream/mainbranch and squashed into single commit to help maintainers review it more efficient and to avoid spaghetti git commit graphs that obfuscate which commit did exactly what change, when and, why.-sflag when usinggit commitcommand. You may refer to this link for more information.Character Limit
A Must Read for Beginners
For rebasing and squashing, here's a must read guide for beginners.