ci(codegen): fix the yarn codegen job in ci.yaml #2618
Assignees
Labels
bug
Something isn't working
dependencies
Pull requests that update a dependency file
Flaky-Test-Automation
Issues related to test stability (which is a long running issue that can never fully be solved)
P1
Priority 1: Highest
Tests
Anything related to tests be that automatic or manual, integration or unit, etc.
Milestone
Comments
petermetz
added
bug
petermetz
added a commit
to petermetz/cacti
that referenced
this issue
Aug 20, 2023
WORK IN PROGRESS Fixes hyperledger-cacti#2618 Signed-off-by: Peter Somogyvari <peter.somogyvari@accenture.com>
petermetz
added a commit
to petermetz/cacti
that referenced
this issue
Aug 22, 2023
WORK IN PROGRESS Fixes hyperledger-cacti#2618 Signed-off-by: Peter Somogyvari <peter.somogyvari@accenture.com>
petermetz
added a commit
to petermetz/cacti
that referenced
this issue
Aug 22, 2023
WORK IN PROGRESS Fixes hyperledger-cacti#2618 Signed-off-by: Peter Somogyvari <peter.somogyvari@accenture.com>
petermetz
added a commit
to petermetz/cacti
that referenced
this issue
Aug 22, 2023
WORK IN PROGRESS Fixes hyperledger-cacti#2618 Signed-off-by: Peter Somogyvari <peter.somogyvari@accenture.com>
petermetz
added a commit
to petermetz/cacti
that referenced
this issue
Aug 22, 2023
WORK IN PROGRESS Fixes hyperledger-cacti#2618 Signed-off-by: Peter Somogyvari <peter.somogyvari@accenture.com>
petermetz
added a commit
to petermetz/cacti
that referenced
this issue
Aug 22, 2023
WORK IN PROGRESS Fixes hyperledger-cacti#2618 Signed-off-by: Peter Somogyvari <peter.somogyvari@accenture.com>
petermetz
added a commit
to petermetz/cacti
that referenced
this issue
Aug 23, 2023
WORK IN PROGRESS Fixes hyperledger-cacti#2618 Signed-off-by: Peter Somogyvari <peter.somogyvari@accenture.com>
petermetz
added a commit
to petermetz/cacti
that referenced
this issue
Aug 23, 2023
WORK IN PROGRESS Fixes hyperledger-cacti#2618 Signed-off-by: Peter Somogyvari <peter.somogyvari@accenture.com>
petermetz
added a commit
to petermetz/cacti
that referenced
this issue
Aug 23, 2023
BREAKING CHANGES: 1. Fabric connector endpoints are using base64 encoding instead of JSON when needing to transfer otherwise binary data (UInt8Array) 2. Same as the above for the Iroha 1 and Iroha 2 connectors. Primary changes: ----------------- 1. Eliminated incorrect usage of the "format": "binary" declarations in the OpenAPI specifications where the format was declared as binary but the data was still traveling as JSON strings despite it (divergence between our spec .json files and the implementation under the hood) 2. Updated the tests and their related endpoints to use base64 encoding instead of JSON serializing UInt8Array instances: It is more efficient this way and also has the benefit of honoring the specification. The reason why it's more efficient is because JSON serializing a typed UInt8Array results in an object literal that maps every single byte to a property of said object, roughly tripling the size of the network transfer. It probably also is slower to serialize/deserialize but I haven't actually measured this. 3. All of the packages are now using the OpenAPI generator v6.6.0 instead of some being stuck on v5.2.1. This might cause build issues in kotlin API clients but because we don't have a freeze on breaking changes right now (leading up to v2.0.0) it is the right time to make drastic changes like this. Fixes hyperledger-cacti#2618 Signed-off-by: Peter Somogyvari <peter.somogyvari@accenture.com>
petermetz
added a commit
to petermetz/cacti
that referenced
this issue
Aug 23, 2023
BREAKING CHANGES: 1. Fabric connector endpoints are using base64 encoding instead of JSON when needing to transfer otherwise binary data (UInt8Array) 2. Same as the above for the Iroha 1 and Iroha 2 connectors. Primary changes: ----------------- 1. Eliminated incorrect usage of the "format": "binary" declarations in the OpenAPI specifications where the format was declared as binary but the data was still traveling as JSON strings despite it (divergence between our spec .json files and the implementation under the hood) 2. Updated the tests and their related endpoints to use base64 encoding instead of JSON serializing UInt8Array instances: It is more efficient this way and also has the benefit of honoring the specification. The reason why it's more efficient is because JSON serializing a typed UInt8Array results in an object literal that maps every single byte to a property of said object, roughly tripling the size of the network transfer. It probably also is slower to serialize/deserialize but I haven't actually measured this. 3. All of the packages are now using the OpenAPI generator v6.6.0 instead of some being stuck on v5.2.1. This might cause build issues in kotlin API clients but because we don't have a freeze on breaking changes right now (leading up to v2.0.0) it is the right time to make drastic changes like this. 4. The yarn_codegen CI job in ci.yaml will now fail if it produced unintended side-effects in version control: If any files have been changed by it that are under version control then it fails. This lets contributors know that they need to execute the `yarn codegen` script locally and then include the results in their commit diff. Fixes hyperledger-cacti#2618 Signed-off-by: Peter Somogyvari <peter.somogyvari@accenture.com>
petermetz
added a commit
to petermetz/cacti
that referenced
this issue
Aug 23, 2023
BREAKING CHANGES: 1. Fabric connector endpoints are using base64 encoding instead of JSON when needing to transfer otherwise binary data (UInt8Array) 2. Same as the above for the Iroha 1 and Iroha 2 connectors. Primary changes: ----------------- 1. Eliminated incorrect usage of the "format": "binary" declarations in the OpenAPI specifications where the format was declared as binary but the data was still traveling as JSON strings despite it (divergence between our spec .json files and the implementation under the hood) 2. Updated the tests and their related endpoints to use base64 encoding instead of JSON serializing UInt8Array instances: It is more efficient this way and also has the benefit of honoring the specification. The reason why it's more efficient is because JSON serializing a typed UInt8Array results in an object literal that maps every single byte to a property of said object, roughly tripling the size of the network transfer. It probably also is slower to serialize/deserialize but I haven't actually measured this. 3. All of the packages are now using the OpenAPI generator v6.6.0 instead of some being stuck on v5.2.1. This might cause build issues in kotlin API clients but because we don't have a freeze on breaking changes right now (leading up to v2.0.0) it is the right time to make drastic changes like this. 4. The yarn_codegen CI job in ci.yaml will now fail if it produced unintended side-effects in version control: If any files have been changed by it that are under version control then it fails. This lets contributors know that they need to execute the `yarn codegen` script locally and then include the results in their commit diff. Fixes hyperledger-cacti#2618 Signed-off-by: Peter Somogyvari <peter.somogyvari@accenture.com>
petermetz
added a commit
to petermetz/cacti
that referenced
this issue
Aug 23, 2023
1. If we run the codegen script and it generates a diff (e.g. changes files that are under version control) that means that we have to send it back to the person who is making the pull request and ask them to include those changes in their commit so that we are adhering to the idea of reproducible builds as much as possible. 2. These (env var) parameters of ./tools/ci.sh are now true by default: - `TOOLS_VALIDATE_BUNDLE_NAMES_DISABLED` - `CUSTOM_CHECKS_DISABLED` - `FULL_BUILD_DISABLED` 3. Refactored the custom-check that verifies the Open API specs' license parameters (`tools/custom-checks/check-open-api-json-specs.ts`) so that it does not fail for missing "identifier" properties UNLESS the spec's version is 3.1.0 or higher. This was necessary because we cannot yet migrate the specs to v3.1.0 due to the generator not supporting it properly (it ends up mangling all of the types in the generated code from their actual type to `any`) 4. Additional logging was also added to the script mentioned in 3) to make it easier in the future to debug what is the problem. 5. Added these environment variables to control the behavior of the `tools/custom-checks/check-open-api-json-specs.ts` script: - `CACTI_CUSTOM_CHECKS_REQUIRED_OPENAPI_SPEC_VERSION` - `CACTI_CUSTOM_CHECKS_REQUIRED_OPENAPI_LICENSE_SPDX` - `CACTI_CUSTOM_CHECKS_REQUIRED_OPENAPI_LICENSE_URL` Related to hyperledger-cacti#2618 Signed-off-by: Peter Somogyvari <peter.somogyvari@accenture.com>
petermetz
added a commit
to petermetz/cacti
that referenced
this issue
Aug 23, 2023
BREAKING CHANGES: 1. Fabric connector endpoints are using base64 encoding instead of JSON when needing to transfer otherwise binary data (UInt8Array) 2. Same as the above for the Iroha 1 and Iroha 2 connectors. Primary changes: ----------------- 1. Removed the `"identifier": "Apache-2.0",` section from all OpenAPI specification documents (./**/openapi.json) project-wide because the OpenAPI generator is not yet compatible with it and because of that the new property just makes it crash with a false-negative validation error. 2. Downgraded all OpenAPI specification documents version from `v3.1.0` to `v3.0.3` (which is how it used to be) because the generator is not yet compatible with it and it had caused types to be mangled. 3. Eliminated incorrect usage of the "format": "binary" declarations in the OpenAPI specifications where the format was declared as binary but the data was still traveling as JSON strings despite it (divergence between our spec .json files and the implementation under the hood) 4. Updated the tests and their related endpoints to use base64 encoding instead of JSON serializing UInt8Array instances: It is more efficient this way and also has the benefit of honoring the specification. The reason why it's more efficient is because JSON serializing a typed UInt8Array results in an object literal that maps every single byte to a property of said object, roughly tripling the size of the network transfer. It probably also is slower to serialize/deserialize but I haven't actually measured this. 5. All of the packages are now using the OpenAPI generator v6.6.0 instead of some being stuck on v5.2.1. This might cause build issues in kotlin API clients but because we don't have a freeze on breaking changes right now (leading up to v2.0.0) it is the right time to make drastic changes like this. Fixes hyperledger-cacti#2618 Signed-off-by: Peter Somogyvari <peter.somogyvari@accenture.com>
petermetz
added a commit
to petermetz/cacti
that referenced
this issue
Aug 23, 2023
BREAKING CHANGES: 1. Fabric connector endpoints are using base64 encoding instead of JSON when needing to transfer otherwise binary data (UInt8Array) 2. Same as the above for the Iroha 1 and Iroha 2 connectors. Primary changes: ----------------- 1. Removed the `"identifier": "Apache-2.0",` section from all OpenAPI specification documents (./**/openapi.json) project-wide because the OpenAPI generator is not yet compatible with it and because of that the new property just makes it crash with a false-negative validation error. 2. Downgraded all OpenAPI specification documents version from `v3.1.0` to `v3.0.3` (which is how it used to be) because the generator is not yet compatible with it and it had caused types to be mangled. 3. Eliminated incorrect usage of the "format": "binary" declarations in the OpenAPI specifications where the format was declared as binary but the data was still traveling as JSON strings despite it (divergence between our spec .json files and the implementation under the hood) 4. Updated the tests and their related endpoints to use base64 encoding instead of JSON serializing UInt8Array instances: It is more efficient this way and also has the benefit of honoring the specification. The reason why it's more efficient is because JSON serializing a typed UInt8Array results in an object literal that maps every single byte to a property of said object, roughly tripling the size of the network transfer. It probably also is slower to serialize/deserialize but I haven't actually measured this. 5. All of the packages are now using the OpenAPI generator v6.6.0 instead of some being stuck on v5.2.1. This might cause build issues in kotlin API clients but because we don't have a freeze on breaking changes right now (leading up to v2.0.0) it is the right time to make drastic changes like this. Fixes hyperledger-cacti#2618 Signed-off-by: Peter Somogyvari <peter.somogyvari@accenture.com>
petermetz
added a commit
to petermetz/cacti
that referenced
this issue
Aug 28, 2023
1. If we run the codegen script and it generates a diff (e.g. changes files that are under version control) that means that we have to send it back to the person who is making the pull request and ask them to include those changes in their commit so that we are adhering to the idea of reproducible builds as much as possible. 2. These (env var) parameters of ./tools/ci.sh are now true by default: - `TOOLS_VALIDATE_BUNDLE_NAMES_DISABLED` - `CUSTOM_CHECKS_DISABLED` - `FULL_BUILD_DISABLED` 3. Refactored the custom-check that verifies the Open API specs' license parameters (`tools/custom-checks/check-open-api-json-specs.ts`) so that it does not fail for missing "identifier" properties UNLESS the spec's version is 3.1.0 or higher. This was necessary because we cannot yet migrate the specs to v3.1.0 due to the generator not supporting it properly (it ends up mangling all of the types in the generated code from their actual type to `any`) 4. Additional logging was also added to the script mentioned in 3) to make it easier in the future to debug what is the problem. 5. Added these environment variables to control the behavior of the `tools/custom-checks/check-open-api-json-specs.ts` script: - `CACTI_CUSTOM_CHECKS_REQUIRED_OPENAPI_SPEC_VERSION` - `CACTI_CUSTOM_CHECKS_REQUIRED_OPENAPI_LICENSE_SPDX` - `CACTI_CUSTOM_CHECKS_REQUIRED_OPENAPI_LICENSE_URL` Related to hyperledger-cacti#2618 Signed-off-by: Peter Somogyvari <peter.somogyvari@accenture.com>
petermetz
added a commit
to petermetz/cacti
that referenced
this issue
Aug 28, 2023
BREAKING CHANGES: 1. Fabric connector endpoints are using base64 encoding instead of JSON when needing to transfer otherwise binary data (UInt8Array) 2. Same as the above for the Iroha 1 and Iroha 2 connectors. Primary changes: ----------------- 1. Removed the `"identifier": "Apache-2.0",` section from all OpenAPI specification documents (./**/openapi.json) project-wide because the OpenAPI generator is not yet compatible with it and because of that the new property just makes it crash with a false-negative validation error. 2. Downgraded all OpenAPI specification documents version from `v3.1.0` to `v3.0.3` (which is how it used to be) because the generator is not yet compatible with it and it had caused types to be mangled. 3. Eliminated incorrect usage of the "format": "binary" declarations in the OpenAPI specifications where the format was declared as binary but the data was still traveling as JSON strings despite it (divergence between our spec .json files and the implementation under the hood) 4. Updated the tests and their related endpoints to use base64 encoding instead of JSON serializing UInt8Array instances: It is more efficient this way and also has the benefit of honoring the specification. The reason why it's more efficient is because JSON serializing a typed UInt8Array results in an object literal that maps every single byte to a property of said object, roughly tripling the size of the network transfer. It probably also is slower to serialize/deserialize but I haven't actually measured this. 5. All of the packages are now using the OpenAPI generator v6.6.0 instead of some being stuck on v5.2.1. This might cause build issues in kotlin API clients but because we don't have a freeze on breaking changes right now (leading up to v2.0.0) it is the right time to make drastic changes like this. Fixes hyperledger-cacti#2618 Signed-off-by: Peter Somogyvari <peter.somogyvari@accenture.com>
petermetz
added a commit
that referenced
this issue
Aug 28, 2023
1. If we run the codegen script and it generates a diff (e.g. changes files that are under version control) that means that we have to send it back to the person who is making the pull request and ask them to include those changes in their commit so that we are adhering to the idea of reproducible builds as much as possible. 2. These (env var) parameters of ./tools/ci.sh are now true by default: - `TOOLS_VALIDATE_BUNDLE_NAMES_DISABLED` - `CUSTOM_CHECKS_DISABLED` - `FULL_BUILD_DISABLED` 3. Refactored the custom-check that verifies the Open API specs' license parameters (`tools/custom-checks/check-open-api-json-specs.ts`) so that it does not fail for missing "identifier" properties UNLESS the spec's version is 3.1.0 or higher. This was necessary because we cannot yet migrate the specs to v3.1.0 due to the generator not supporting it properly (it ends up mangling all of the types in the generated code from their actual type to `any`) 4. Additional logging was also added to the script mentioned in 3) to make it easier in the future to debug what is the problem. 5. Added these environment variables to control the behavior of the `tools/custom-checks/check-open-api-json-specs.ts` script: - `CACTI_CUSTOM_CHECKS_REQUIRED_OPENAPI_SPEC_VERSION` - `CACTI_CUSTOM_CHECKS_REQUIRED_OPENAPI_LICENSE_SPDX` - `CACTI_CUSTOM_CHECKS_REQUIRED_OPENAPI_LICENSE_URL` Related to #2618 Signed-off-by: Peter Somogyvari <peter.somogyvari@accenture.com>
petermetz
added a commit
that referenced
this issue
Aug 28, 2023
BREAKING CHANGES: 1. Fabric connector endpoints are using base64 encoding instead of JSON when needing to transfer otherwise binary data (UInt8Array) 2. Same as the above for the Iroha 1 and Iroha 2 connectors. Primary changes: ----------------- 1. Removed the `"identifier": "Apache-2.0",` section from all OpenAPI specification documents (./**/openapi.json) project-wide because the OpenAPI generator is not yet compatible with it and because of that the new property just makes it crash with a false-negative validation error. 2. Downgraded all OpenAPI specification documents version from `v3.1.0` to `v3.0.3` (which is how it used to be) because the generator is not yet compatible with it and it had caused types to be mangled. 3. Eliminated incorrect usage of the "format": "binary" declarations in the OpenAPI specifications where the format was declared as binary but the data was still traveling as JSON strings despite it (divergence between our spec .json files and the implementation under the hood) 4. Updated the tests and their related endpoints to use base64 encoding instead of JSON serializing UInt8Array instances: It is more efficient this way and also has the benefit of honoring the specification. The reason why it's more efficient is because JSON serializing a typed UInt8Array results in an object literal that maps every single byte to a property of said object, roughly tripling the size of the network transfer. It probably also is slower to serialize/deserialize but I haven't actually measured this. 5. All of the packages are now using the OpenAPI generator v6.6.0 instead of some being stuck on v5.2.1. This might cause build issues in kotlin API clients but because we don't have a freeze on breaking changes right now (leading up to v2.0.0) it is the right time to make drastic changes like this. Fixes #2618 Signed-off-by: Peter Somogyvari <peter.somogyvari@accenture.com>
sandeepnRES
pushed a commit
to sandeepnRES/cacti
that referenced
this issue
Dec 21, 2023
1. If we run the codegen script and it generates a diff (e.g. changes files that are under version control) that means that we have to send it back to the person who is making the pull request and ask them to include those changes in their commit so that we are adhering to the idea of reproducible builds as much as possible. 2. These (env var) parameters of ./tools/ci.sh are now true by default: - `TOOLS_VALIDATE_BUNDLE_NAMES_DISABLED` - `CUSTOM_CHECKS_DISABLED` - `FULL_BUILD_DISABLED` 3. Refactored the custom-check that verifies the Open API specs' license parameters (`tools/custom-checks/check-open-api-json-specs.ts`) so that it does not fail for missing "identifier" properties UNLESS the spec's version is 3.1.0 or higher. This was necessary because we cannot yet migrate the specs to v3.1.0 due to the generator not supporting it properly (it ends up mangling all of the types in the generated code from their actual type to `any`) 4. Additional logging was also added to the script mentioned in 3) to make it easier in the future to debug what is the problem. 5. Added these environment variables to control the behavior of the `tools/custom-checks/check-open-api-json-specs.ts` script: - `CACTI_CUSTOM_CHECKS_REQUIRED_OPENAPI_SPEC_VERSION` - `CACTI_CUSTOM_CHECKS_REQUIRED_OPENAPI_LICENSE_SPDX` - `CACTI_CUSTOM_CHECKS_REQUIRED_OPENAPI_LICENSE_URL` Related to hyperledger-cacti#2618 Signed-off-by: Peter Somogyvari <peter.somogyvari@accenture.com>
sandeepnRES
pushed a commit
to sandeepnRES/cacti
that referenced
this issue
Dec 21, 2023
BREAKING CHANGES: 1. Fabric connector endpoints are using base64 encoding instead of JSON when needing to transfer otherwise binary data (UInt8Array) 2. Same as the above for the Iroha 1 and Iroha 2 connectors. Primary changes: ----------------- 1. Removed the `"identifier": "Apache-2.0",` section from all OpenAPI specification documents (./**/openapi.json) project-wide because the OpenAPI generator is not yet compatible with it and because of that the new property just makes it crash with a false-negative validation error. 2. Downgraded all OpenAPI specification documents version from `v3.1.0` to `v3.0.3` (which is how it used to be) because the generator is not yet compatible with it and it had caused types to be mangled. 3. Eliminated incorrect usage of the "format": "binary" declarations in the OpenAPI specifications where the format was declared as binary but the data was still traveling as JSON strings despite it (divergence between our spec .json files and the implementation under the hood) 4. Updated the tests and their related endpoints to use base64 encoding instead of JSON serializing UInt8Array instances: It is more efficient this way and also has the benefit of honoring the specification. The reason why it's more efficient is because JSON serializing a typed UInt8Array results in an object literal that maps every single byte to a property of said object, roughly tripling the size of the network transfer. It probably also is slower to serialize/deserialize but I haven't actually measured this. 5. All of the packages are now using the OpenAPI generator v6.6.0 instead of some being stuck on v5.2.1. This might cause build issues in kotlin API clients but because we don't have a freeze on breaking changes right now (leading up to v2.0.0) it is the right time to make drastic changes like this. Fixes hyperledger-cacti#2618 Signed-off-by: Peter Somogyvari <peter.somogyvari@accenture.com>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
CI Logs Attachment
2023-08-14T07-19-09-cacti-ci-yarn-codegen-crash.log
Likely cause
The older version of the generator that we use in parallel (v.5.2.1) has a different schema defined for the spec files by default and our changes to the license object are not compatible with that (only the 6.x version)
Potential solution is to upgrade all the generators to 6.x but I have some vague memory of this not being feasible right now because the way some of our specs are written that breaks compilation of the generated code.
The text was updated successfully, but these errors were encountered: