From a1e5a4db66e511e00abf3985379ef3f0756fede6 Mon Sep 17 00:00:00 2001 From: Ben Howard Date: Tue, 4 Aug 2020 18:16:35 -0600 Subject: [PATCH 1/8] cosa/cli-spec: add envVar CLI support --- cosa/cli-spec.md | 64 ++++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 64 insertions(+) create mode 100644 cosa/cli-spec.md diff --git a/cosa/cli-spec.md b/cosa/cli-spec.md new file mode 100644 index 00000000..05a1f323 --- /dev/null +++ b/cosa/cli-spec.md @@ -0,0 +1,64 @@ +# CoreOS Assemlber Config Specification +- Subject ID: COSA.20200804 +- Superceeds: None +- Conflicts: None + +## Summary + +The RHCOS development pipelines use a "jobspec" to describe the pipeline runs. This enhancement proposes a file-based interface based loosely on the RHCOS jobspec. + +## Goals + +1. Provide a file-based interface into COSA +1. Prep for breaking up the Pipeline monoliths +1. Provide a migration path for RHCOS pipelines +1. Prepare for pipeline Nirvana (where *COS pipelines can live in peace and harmony) + +## Why? + +The innovation of the RHCOS JobSpec was it provided a clear seperation of configuration from the code; the logic was conflated. Further it the jobspec allowed the same logic to be run with different inputs. + +To support this, a "CliSpec" is being introduced. + +## YAML to EnvVars + +* For Bash Pythonic CLIs, each argument will have an envVar added. +* At entry into COSA, the optional CliSpec will be checked +* Values will be parsed and emit as EnvVars by a common entry point. + +## Example: + +For the `coreos-assembler build` command: +``` +build: + force: false + force_nocache: false + skip_prune: false + parent: + tag: + ``` + + Would render the envVars: + ``` + COSA_BUILD_FORCE=false + COSA_BUILD_FORCE_NOCAHCE=false + COSA_BUILD_SKIP_PRUNE=false + ... + ``` + +Then in `cmd-build` the defaults would be: +``` +FORCE="${COSA_BUILD_FORCE:-}" +FORCE_IMAGE="${COSA_BUILD_FORCE_IMAGE:-}" +SKIP_PRUNE="${COSA_BUILD_SKIP_PRUNE:-0}" +... +``` + +## PR Standard + +A thorough review of the COSA code-base, a comparison between the FCOS and RHCOS pipelines shows that each command will need to be handled seperately. + +Therefore, for each command updated to modify this interface: +- The prefix `COSA_` will be used for envVars (e.g. `COSA_BUILDEXTEND_AZURE`) +- No calculated values. Commands may not parse the spec file itself. +- One command per PR. \ No newline at end of file From 8d4616b71bcd903cbf0e1faecb7e6fdbb7a6b98a Mon Sep 17 00:00:00 2001 From: Ben Howard Date: Wed, 5 Aug 2020 09:47:33 -0600 Subject: [PATCH 2/8] PR feedback update Signed-off-by: Ben Howard --- cosa/20200804-cfg-spec.md | 107 ++++++++++++++++++++++++++++++++++++++ cosa/cli-spec.md | 64 ----------------------- 2 files changed, 107 insertions(+), 64 deletions(-) create mode 100644 cosa/20200804-cfg-spec.md delete mode 100644 cosa/cli-spec.md diff --git a/cosa/20200804-cfg-spec.md b/cosa/20200804-cfg-spec.md new file mode 100644 index 00000000..f06ef1a7 --- /dev/null +++ b/cosa/20200804-cfg-spec.md @@ -0,0 +1,107 @@ +# CoreOS Assemlber Config Specification +- Subject ID: COSA.20200804 +- Superceeds: None +- Conflicts: None + +## Summary + +The RHCOS development pipelines use a "jobspec" to describe the pipeline runs. This enhancement proposes a file-based interface based loosely on the RHCOS jobspec. + +## Goals + +1. Provide a file-based interface into COSA +1. Prep for breaking up the Pipeline monoliths +1. Provide a migration path for RHCOS pipelines +1. Prepare for pipeline Nirvana (where *COS pipelines can live in peace and harmony) + +## Why? + +The innovation of the RHCOS JobSpec was it provided a clear seperation of configuration from the code; the logic was conflated. Further the jobspec allowed the logic to be run with different inputs. + +To support this, a "COSAspec" is being introduced to provide a CLI file-interface. + +## YAML to EnvVars + +* For Bash and Pythonic CLIs, each argument will have an envVar added. +* At entry into COSA, the optional COSAspec will be checked +* Values will be parsed and emit as EnvVars by a common entry point. +* The EnvVars will set defaults to the CLI commands + +Note: The Mantle code is GoLang based. The use of envVars is supported by the Cobra CLI, but needs further research and consideration. The Mantle code is not considered in this design standard. + +## Types + +* List values will be joined to comma seperated strings +* Dicts/Map will be flattened with an `_` between levels +* Boolean values should be 0, 1, True, False, true, false. +* Number types are dealt with + +Due to the lack of cohesion, the mix of Python, Bash and GoLang, each component will nessesarily handle empty and null values. + + +## Example: + +For the `coreos-assembler build` command: +``` +build: + force: false + force_nocache: false + skip_prune: false + parent: + tag: +``` + +Would render the envVars: +``` +COSA_BUILD_FORCE=false +COSA_BUILD_FORCE_NOCAHCE=false +COSA_BUILD_SKIP_PRUNE=false +... +``` + +Then in `cmd-build` the defaults would be: +``` +FORCE="${COSA_BUILD_FORCE:-}" +FORCE_IMAGE="${COSA_BUILD_FORCE_IMAGE:-}" +SKIP_PRUNE="${COSA_BUILD_SKIP_PRUNE:-0}" +... +``` + +## List value example + +``` +extend: + aws: + regions: + - us-east-1 + - us-west-1 +``` +Would render at `COSA_EXTEND_AWS_REGIONS="us-east-1,us-west-1"` + +## Dict example + +``` +foo: + bar: + baz: + 1: true + 2: true +``` + +Would render as `COSA_FOO_BAR_BAZ_1=true` and `COSA_FOO_BAR_BAZ_2=true` + +## PR Standard + +A thorough review of the COSA code-base, a comparison between the FCOS and RHCOS pipelines shows that each command will need to be handled seperately. + +Therefore, for each command updated to modify this interface: +- The prefix `COSA_` will be used for envVars (e.g. `COSA_EXTEND_AZURE`) +- No calculated values. Commands may not parse the spec file itself. +- One command per PR. + + +Naming convention for commands (in order of precidence): +- `cmd-buildextend-*`: `COSA_EXTEND_*` +- `cmd-build`: `COSA_BUILD_*` +- `cmd-.*`: `COSA__*` +- `.*`: no interface diff --git a/cosa/cli-spec.md b/cosa/cli-spec.md deleted file mode 100644 index 05a1f323..00000000 --- a/cosa/cli-spec.md +++ /dev/null @@ -1,64 +0,0 @@ -# CoreOS Assemlber Config Specification -- Subject ID: COSA.20200804 -- Superceeds: None -- Conflicts: None - -## Summary - -The RHCOS development pipelines use a "jobspec" to describe the pipeline runs. This enhancement proposes a file-based interface based loosely on the RHCOS jobspec. - -## Goals - -1. Provide a file-based interface into COSA -1. Prep for breaking up the Pipeline monoliths -1. Provide a migration path for RHCOS pipelines -1. Prepare for pipeline Nirvana (where *COS pipelines can live in peace and harmony) - -## Why? - -The innovation of the RHCOS JobSpec was it provided a clear seperation of configuration from the code; the logic was conflated. Further it the jobspec allowed the same logic to be run with different inputs. - -To support this, a "CliSpec" is being introduced. - -## YAML to EnvVars - -* For Bash Pythonic CLIs, each argument will have an envVar added. -* At entry into COSA, the optional CliSpec will be checked -* Values will be parsed and emit as EnvVars by a common entry point. - -## Example: - -For the `coreos-assembler build` command: -``` -build: - force: false - force_nocache: false - skip_prune: false - parent: - tag: - ``` - - Would render the envVars: - ``` - COSA_BUILD_FORCE=false - COSA_BUILD_FORCE_NOCAHCE=false - COSA_BUILD_SKIP_PRUNE=false - ... - ``` - -Then in `cmd-build` the defaults would be: -``` -FORCE="${COSA_BUILD_FORCE:-}" -FORCE_IMAGE="${COSA_BUILD_FORCE_IMAGE:-}" -SKIP_PRUNE="${COSA_BUILD_SKIP_PRUNE:-0}" -... -``` - -## PR Standard - -A thorough review of the COSA code-base, a comparison between the FCOS and RHCOS pipelines shows that each command will need to be handled seperately. - -Therefore, for each command updated to modify this interface: -- The prefix `COSA_` will be used for envVars (e.g. `COSA_BUILDEXTEND_AZURE`) -- No calculated values. Commands may not parse the spec file itself. -- One command per PR. \ No newline at end of file From 927c2011854269b814dc8d2254a3b658cf3fe64c Mon Sep 17 00:00:00 2001 From: Ben Howard Date: Wed, 5 Aug 2020 15:11:22 -0600 Subject: [PATCH 3/8] Update cosa/20200804-cfg-spec.md Co-authored-by: Jonathan Lebon --- cosa/20200804-cfg-spec.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/cosa/20200804-cfg-spec.md b/cosa/20200804-cfg-spec.md index f06ef1a7..7635bd0c 100644 --- a/cosa/20200804-cfg-spec.md +++ b/cosa/20200804-cfg-spec.md @@ -100,7 +100,7 @@ Therefore, for each command updated to modify this interface: - One command per PR. -Naming convention for commands (in order of precidence): +Naming convention for commands (in order of precedence): - `cmd-buildextend-*`: `COSA_EXTEND_*` - `cmd-build`: `COSA_BUILD_*` - `cmd-.*`: `COSA__*` From 8cf6fafc4ca4a38f32563e1bc9cf7e2a9b2c8aa1 Mon Sep 17 00:00:00 2001 From: Ben Howard Date: Wed, 5 Aug 2020 15:11:30 -0600 Subject: [PATCH 4/8] Update cosa/20200804-cfg-spec.md Co-authored-by: Jonathan Lebon --- cosa/20200804-cfg-spec.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/cosa/20200804-cfg-spec.md b/cosa/20200804-cfg-spec.md index 7635bd0c..aa92da34 100644 --- a/cosa/20200804-cfg-spec.md +++ b/cosa/20200804-cfg-spec.md @@ -103,5 +103,5 @@ Therefore, for each command updated to modify this interface: Naming convention for commands (in order of precedence): - `cmd-buildextend-*`: `COSA_EXTEND_*` - `cmd-build`: `COSA_BUILD_*` -- `cmd-.*`: `COSA__*` +- `cmd-.*`: `COSA__*` - `.*`: no interface From 326fdce14f67ca3d748f5c4a28a462480499adf4 Mon Sep 17 00:00:00 2001 From: Ben Howard Date: Wed, 5 Aug 2020 15:11:36 -0600 Subject: [PATCH 5/8] Update cosa/20200804-cfg-spec.md Co-authored-by: Jonathan Lebon --- cosa/20200804-cfg-spec.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/cosa/20200804-cfg-spec.md b/cosa/20200804-cfg-spec.md index aa92da34..029167dd 100644 --- a/cosa/20200804-cfg-spec.md +++ b/cosa/20200804-cfg-spec.md @@ -5,7 +5,7 @@ ## Summary -The RHCOS development pipelines use a "jobspec" to describe the pipeline runs. This enhancement proposes a file-based interface based loosely on the RHCOS jobspec. +The RHCOS development pipelines use a "jobspec" to describe the pipeline runs. This enhancement proposes a YAML-based interface based loosely on the RHCOS jobspec. ## Goals From d40ac497beecf7292e64d73b3ff9d1703d0e8e23 Mon Sep 17 00:00:00 2001 From: Ben Howard Date: Wed, 5 Aug 2020 15:27:31 -0600 Subject: [PATCH 6/8] Update cosa/20200804-cfg-spec.md Co-authored-by: Jonathan Lebon --- cosa/20200804-cfg-spec.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/cosa/20200804-cfg-spec.md b/cosa/20200804-cfg-spec.md index 029167dd..f8a27115 100644 --- a/cosa/20200804-cfg-spec.md +++ b/cosa/20200804-cfg-spec.md @@ -36,7 +36,7 @@ Note: The Mantle code is GoLang based. The use of envVars is supported by the Co * Boolean values should be 0, 1, True, False, true, false. * Number types are dealt with -Due to the lack of cohesion, the mix of Python, Bash and GoLang, each component will nessesarily handle empty and null values. +Due to the lack of cohesion, the mix of Python, Bash and GoLang, each component will necessarily handle empty and null values. ## Example: From 27ff39e7ff981346d651cbb99e1593ad9d277c6c Mon Sep 17 00:00:00 2001 From: Ben Howard Date: Wed, 5 Aug 2020 15:27:38 -0600 Subject: [PATCH 7/8] Update cosa/20200804-cfg-spec.md Co-authored-by: Jonathan Lebon --- cosa/20200804-cfg-spec.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/cosa/20200804-cfg-spec.md b/cosa/20200804-cfg-spec.md index f8a27115..d6895b74 100644 --- a/cosa/20200804-cfg-spec.md +++ b/cosa/20200804-cfg-spec.md @@ -1,4 +1,4 @@ -# CoreOS Assemlber Config Specification +# CoreOS Assembler Config Specification - Subject ID: COSA.20200804 - Superceeds: None - Conflicts: None From 8e234eb132e1bc964ab594acb8e2c3d38061f470 Mon Sep 17 00:00:00 2001 From: Ben Howard Date: Thu, 6 Aug 2020 14:39:06 -0600 Subject: [PATCH 8/8] Update 20200804-cfg-spec.md --- cosa/20200804-cfg-spec.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/cosa/20200804-cfg-spec.md b/cosa/20200804-cfg-spec.md index d6895b74..8b0961c4 100644 --- a/cosa/20200804-cfg-spec.md +++ b/cosa/20200804-cfg-spec.md @@ -34,7 +34,7 @@ Note: The Mantle code is GoLang based. The use of envVars is supported by the Co * List values will be joined to comma seperated strings * Dicts/Map will be flattened with an `_` between levels * Boolean values should be 0, 1, True, False, true, false. -* Number types are dealt with +* Number types are strings and the consuming code is repsonsbile for casting to the type. Due to the lack of cohesion, the mix of Python, Bash and GoLang, each component will necessarily handle empty and null values.