Add compile-time code generation using zap for app-templates (#23962)

* Add compile-time code generation using zap for app-templates - add support for gn builds - add support for cmake builds - update esp32 build logic - update pre-generation script - split out app and idl codegen - update template used so zap_generate_all only generates IDLs for apps - updated CI unit tests * Fix tizen build dependencies for all-clusters and minimal * Remove one more unnedded include directory from tizen * Fix tv casting app include * Restyle * Bump up openiotsdk dockerfile to get zap-cli * Fix include for efr32 Lcd painter * Test out a zap installation workflow in darwin builds * Fix trailing slash * Multiline to make mac install more redable * make sure all subdirs are created when installing zap * Use sudo to create the zap install directory * unzip only zap-cli, to hopefully make the unzip faster * Install zap on all darwin CI runs * Add `"$(TEMP_DIR)/out/gen/src/controller/data_model/zapgen/"` to xcode project header search paths * Do not run clang-format for compile-time generated files * Update code generation documentation a bit * More documentation with examples of how to invoke and set pregeneration variables * Restyle * Fix typo in command for code pregen * Fix typo * Add a runnable check that is intended to codegen * make unit tests with golden images run (but not yet pass) * Update golden copies * Update documentation and restyle * Bump up build apps time... seems like bimodal timing distribution and this occasionally fails at 1h, leave to 1.5hrs for now * Fix spelling * make zap-cli --version run on mac as part of the install since I seem to get package conflict errors periodically * Try to use tempState for mac parallel runs * Restyle * Several code review changes * Fix typo in path for templates * Undo third party submodule changes * Move common python zap logic into one file, to have the zap path logic centralized and to be more easily be able to add version checks * Restyle * Add version check capabilities to zap execution and generate.py * Restyle * Update zap version and set skip_real_version based on merge with master * Also update darwin zap installation path * Update more paths with the new zap version * Restyle * Golden image update for regeneration * Do not restyle test outputs for zap generation tests * Restyle * Restyled by isort * Fix up one more include for obsolete paths * Update generated files for unit tests * Re-generate unit test output data * Fix telink contact sensor build * Update unit test data * Revert wrong diff marker * Bump CI docker to 0.6.31 to use the latest zap * Bump up image versions for non-chip-build images as well * Re-generate using latest zap from 0.6.31 image * Keep codegen for controller clusters, for darwin specifically * patch the matter xcode project to update the include path to use the darwin-spefici codegen for controller cluster paths * Restyle * Update to make template checker happy, update min version for zap execution * One more manual restyle attempt * Ran regen again for tests * Re-generated this time using CI steps for checking * See if CI can run zap tests on mac as well * Do not set skip real version for the version check * Add a better comment of why IMClusterCommandHandler is odd * Restyle * Start developing a simpler ZAP version update script as this is likely often used * Off by one fix and update search logic a bit * Add support for min version update as well * Update a comment and a min version * Comment updates * Another comment regarding version update * Fix gen files for unit tests. We should figure out why environment matters here ... this is odd * Update clang-format version comment: it is 15 not 14 now in pigweed * Force clang-format-15 or clang-format (assuming we use pigweed) * Regenerate in pigweed environment, so clang-format is consistently 15 * Report clang-format version when formatting, to understand any format differences * Re-gen all after clang-format-15 was applied * Remove zzz_generated from restyled.yaml since we clang-format already during zap gen * Zap regen app2 * Regenerated (after restyle changed permissions..) and updated logs to contain what files get formatted * Update CI image for coverage run * Serialize zap execution to work around zap darwin bugs * Fix copy and paste error for cmake: remove extra comma * Update version update to control what files to update: usage or docker * Update scripts/tools/zap/generate.py Co-authored-by: Boris Zbarsky <bzbarsky@apple.com> * Skip ZAP version check when ZAP_DEVELOPMENT_PATH is set * Restyle Co-authored-by: Restyled.io <commits@restyled.io> Co-authored-by: Andrei Litvin <andreilitvin@google.com> Co-authored-by: Boris Zbarsky <bzbarsky@apple.com>
project-chip · Feb 16, 2024 · 1255904 · 1255904
1 parent ac14a67
commit 1255904
Show file tree

Hide file tree

Showing 380 changed files with 35,539 additions and 93,679 deletions.
diff --git a/.github/workflows/bloat_check.yaml b/.github/workflows/bloat_check.yaml
@@ -30,7 +30,7 @@ jobs:
  runs-on: ubuntu-latest
 
  container:
- image: connectedhomeip/chip-build:0.6.30
+ image: connectedhomeip/chip-build:0.6.31
 
  steps:
  - uses: Wandalen/wretry.action@v1.0.36

diff --git a/.github/workflows/build.yaml b/.github/workflows/build.yaml
@@ -36,7 +36,7 @@ jobs:
  if: github.actor != 'restyled-io[bot]'
 
  container:
- image: connectedhomeip/chip-build:0.6.30
+ image: connectedhomeip/chip-build:0.6.31
  volumes:
  - "/tmp/log_output:/tmp/test_logs"
  options: --privileged --sysctl "net.ipv6.conf.all.disable_ipv6=0
@@ -138,7 +138,7 @@ jobs:
  if: github.actor != 'restyled-io[bot]'
 
  container:
- image: connectedhomeip/chip-build:0.6.30
+ image: connectedhomeip/chip-build:0.6.31
  volumes:
  - "/tmp/log_output:/tmp/test_logs"
  options: --privileged --sysctl "net.ipv6.conf.all.disable_ipv6=0
@@ -251,6 +251,14 @@ jobs:
  --pregen-dir ./zzz_pregenerated \
  build \
  "
+ - name: Check no code generation in output
+ run: |
+ CNT=$(find -name "CHIPClusters.h" out | wc -l)
+ if [ "${CNT}" != "0" ]; then
+ echo "ERROR: found unexpected generated files:"
+ find -name "CHIPClusters.h" out
+ exit 1
+ fi
  - name: Undo code pre-generation changes (make compile time codegen work again)
  run: |
  rm -rf ./zzz_pregenerated
@@ -290,7 +298,7 @@ jobs:
  if: github.actor != 'restyled-io[bot]'
 
  container:
- image: connectedhomeip/chip-build:0.6.30
+ image: connectedhomeip/chip-build:0.6.31
  volumes:
  - "/tmp/log_output:/tmp/test_logs"
  options: --sysctl "net.ipv6.conf.all.disable_ipv6=0
@@ -344,6 +352,12 @@ jobs:
  timeout-minutes: 200
  runs-on: macos-latest
  if: github.actor != 'restyled-io[bot]'
+
+ env:
+ # NOTE: Generally kept in sync within the repo using
+ # scripts/tools/zap/version_update.py
+ ZAP_VERSION: v2023.01.09-nightly
+ ZAP_INSTALL_PATH: /usr/local/zap/zap-v2023.01.09-nightly
 
  steps:
  - uses: Wandalen/wretry.action@v1.0.36
@@ -374,6 +388,16 @@ jobs:
  ls -la /usr/local/Cellar/openssl@1.1
  OPEN_SSL_VERSION=`ls -la /usr/local/Cellar/openssl@1.1 | cat | tail -n1 | awk '{print $NF}'`
  ln -s /usr/local/Cellar/openssl@1.1/$OPEN_SSL_VERSION/lib/pkgconfig/* .
+ - name: Install zap
+ run: |
+ sudo mkdir -p $ZAP_INSTALL_PATH
+ sudo chown `whoami` $ZAP_INSTALL_PATH
+ curl -L https://github.com/project-chip/zap/releases/download/${ZAP_VERSION}/zap-mac.zip \
+ --output $ZAP_INSTALL_PATH/zap-mac.zip
+ cd $ZAP_INSTALL_PATH
+ unzip zap-mac.zip zap-cli
+ rm zap-mac.zip
+ ./zap-cli --version
  - name: Bootstrap
  timeout-minutes: 25
  run: scripts/build/gn_bootstrap.sh
@@ -447,7 +471,7 @@ jobs:
  if: github.actor != 'restyled-io[bot]'
 
  container:
- image: connectedhomeip/chip-build:0.6.30
+ image: connectedhomeip/chip-build:0.6.31
  volumes:
  - "/tmp/log_output:/tmp/test_logs"
  options: --privileged --sysctl "net.ipv6.conf.all.disable_ipv6=0
@@ -478,4 +502,4 @@ jobs:
  - name: Run Build Coverage
  timeout-minutes: 20
  run: ./scripts/build_coverage.sh
-
+
diff --git a/.github/workflows/chef.yaml b/.github/workflows/chef.yaml
@@ -29,7 +29,7 @@ jobs:
  if: github.actor != 'restyled-io[bot]'
 
  container:
- image: connectedhomeip/chip-build:0.6.30
+ image: connectedhomeip/chip-build:0.6.31
  options: --user root
 
  steps:
@@ -57,7 +57,7 @@ jobs:
  if: github.actor != 'restyled-io[bot]'
 
  container:
- image: connectedhomeip/chip-build-esp32:0.6.30
+ image: connectedhomeip/chip-build-esp32:0.6.31
  options: --user root
 
  steps:
@@ -85,7 +85,7 @@ jobs:
  if: github.actor != 'restyled-io[bot]'
 
  container:
- image: connectedhomeip/chip-build-nrf-platform:0.6.30
+ image: connectedhomeip/chip-build-nrf-platform:0.6.31
  options: --user root
 
  steps:

diff --git a/.github/workflows/cirque.yaml b/.github/workflows/cirque.yaml
@@ -29,7 +29,7 @@ jobs:
  timeout-minutes: 90
 
  env:
- DOCKER_RUN_VERSION: 0.6.30
+ DOCKER_RUN_VERSION: 0.6.31
  GITHUB_CACHE_PATH: /tmp/cirque-cache/
 
  runs-on: ubuntu-latest
@@ -38,7 +38,7 @@ jobs:
  # need to run with privilege, which isn't supported by job.XXX.contaner
  # https://github.com/actions/container-action/issues/2
  # container:
- # image: connectedhomeip/chip-build-cirque:0.6.30
+ # image: connectedhomeip/chip-build-cirque:0.6.31
  # volumes:
  # - "/tmp:/tmp"
  # - "/dev/pts:/dev/pts"

diff --git a/.github/workflows/darwin-tests.yaml b/.github/workflows/darwin-tests.yaml
@@ -39,6 +39,10 @@ jobs:
  env:
  BUILD_VARIANT: ${{matrix.build_variant}}
  LSAN_OPTIONS: detect_leaks=1 malloc_context_size=40 suppressions=scripts/tests/chiptest/lsan-mac-suppressions.txt
+ # NOTE: Generally kept in sync within the repo using
+ # scripts/tools/zap/version_update.py
+ ZAP_VERSION: v2023.01.09-nightly
+ ZAP_INSTALL_PATH: /usr/local/zap/zap-v2023.01.09-nightly
 
  if: github.actor != 'restyled-io[bot]'
  runs-on: macos-latest
@@ -72,6 +76,16 @@ jobs:
  ls -la /usr/local/Cellar/openssl@1.1
  OPEN_SSL_VERSION=`ls -la /usr/local/Cellar/openssl@1.1 | cat | tail -n1 | awk '{print $NF}'`
  ln -s /usr/local/Cellar/openssl@1.1/$OPEN_SSL_VERSION/lib/pkgconfig/* .
+ - name: Install zap
+ run: |
+ sudo mkdir -p $ZAP_INSTALL_PATH
+ sudo chown `whoami` $ZAP_INSTALL_PATH
+ curl -L https://github.com/project-chip/zap/releases/download/${ZAP_VERSION}/zap-mac.zip \
+ --output $ZAP_INSTALL_PATH/zap-mac.zip
+ cd $ZAP_INSTALL_PATH
+ unzip zap-mac.zip zap-cli
+ rm zap-mac.zip
+ ./zap-cli --version
  - name: Bootstrap
  timeout-minutes: 25
  run: scripts/build/gn_bootstrap.sh

diff --git a/.github/workflows/darwin.yaml b/.github/workflows/darwin.yaml
@@ -31,6 +31,12 @@ jobs:
  if: github.actor != 'restyled-io[bot]'
  runs-on: macos-latest
 
+ env:
+ # NOTE: Generally kept in sync within the repo using
+ # scripts/tools/zap/version_update.py
+ ZAP_VERSION: v2023.01.09-nightly
+ ZAP_INSTALL_PATH: /usr/local/zap/zap-v2023.01.09-nightly
+
  steps:
  - uses: Wandalen/wretry.action@v1.0.36
  name: Checkout
@@ -52,6 +58,16 @@ jobs:
  ls -la /usr/local/Cellar/openssl@1.1
  OPEN_SSL_VERSION=`ls -la /usr/local/Cellar/openssl@1.1 | cat | tail -n1 | awk '{print $NF}'`
  ln -s /usr/local/Cellar/openssl@1.1/$OPEN_SSL_VERSION/lib/pkgconfig/* .
+ - name: Install zap
+ run: |
+ sudo mkdir -p $ZAP_INSTALL_PATH
+ sudo chown `whoami` $ZAP_INSTALL_PATH
+ curl -L https://github.com/project-chip/zap/releases/download/${ZAP_VERSION}/zap-mac.zip \
+ --output $ZAP_INSTALL_PATH/zap-mac.zip
+ cd $ZAP_INSTALL_PATH
+ unzip zap-mac.zip zap-cli
+ rm zap-mac.zip
+ ./zap-cli --version
  - name: Bootstrap
  timeout-minutes: 25
  run: scripts/build/gn_bootstrap.sh

diff --git a/.github/workflows/doxygen.yaml b/.github/workflows/doxygen.yaml
@@ -82,7 +82,7 @@ jobs:
 
  runs-on: ubuntu-20.04
  container:
- image: connectedhomeip/chip-build-doxygen:0.6.30
+ image: connectedhomeip/chip-build-doxygen:0.6.31
 
  if: github.actor != 'restyled-io[bot]'
 

diff --git a/.github/workflows/examples-ameba.yaml b/.github/workflows/examples-ameba.yaml
@@ -35,7 +35,7 @@ jobs:
  if: github.actor != 'restyled-io[bot]'
 
  container:
- image: connectedhomeip/chip-build-ameba:0.6.30
+ image: connectedhomeip/chip-build-ameba:0.6.31
  options: --user root
 
  steps:

diff --git a/.github/workflows/examples-bouffalolab.yaml b/.github/workflows/examples-bouffalolab.yaml
@@ -35,7 +35,7 @@ jobs:
  if: github.actor != 'restyled-io[bot]'
 
  container:
- image: connectedhomeip/chip-build-bouffalolab:0.6.30
+ image: connectedhomeip/chip-build-bouffalolab:0.6.31
  volumes:
  - "/tmp/bloat_reports:/tmp/bloat_reports"
  steps:

diff --git a/.github/workflows/examples-cc13x2x7_26x2x7.yaml b/.github/workflows/examples-cc13x2x7_26x2x7.yaml
@@ -37,7 +37,7 @@ jobs:
  if: github.actor != 'restyled-io[bot]'
 
  container:
- image: connectedhomeip/chip-build-ti:0.6.30
+ image: connectedhomeip/chip-build-ti:0.6.31
  volumes:
  - "/tmp/bloat_reports:/tmp/bloat_reports"
  steps:

diff --git a/.github/workflows/examples-efr32.yaml b/.github/workflows/examples-efr32.yaml
@@ -38,7 +38,7 @@ jobs:
  if: github.actor != 'restyled-io[bot]'
 
  container:
- image: connectedhomeip/chip-build-efr32:0.6.30
+ image: connectedhomeip/chip-build-efr32:0.6.31
  volumes:
  - "/tmp/bloat_reports:/tmp/bloat_reports"
  steps:

diff --git a/.github/workflows/examples-esp32.yaml b/.github/workflows/examples-esp32.yaml
@@ -34,7 +34,7 @@ jobs:
  if: github.actor != 'restyled-io[bot]'
 
  container:
- image: connectedhomeip/chip-build-esp32:0.6.30
+ image: connectedhomeip/chip-build-esp32:0.6.31
  volumes:
  - "/tmp/bloat_reports:/tmp/bloat_reports"
 
@@ -93,6 +93,14 @@ jobs:
  build \
  --copy-artifacts-to out/artifacts \
  "
+ - name: Check no code generation in output
+ run: |
+ CNT=$(find -name "CHIPClusters.h" out | wc -l)
+ if [ "${CNT}" != "0" ]; then
+ echo "ERROR: found unexpected generated files:"
+ find -name "CHIPClusters.h" out
+ exit 1
+ fi
  - name: Undo code pregeneration changes
  run: |
  rm -rf ./zzz_pregenerated
@@ -140,7 +148,7 @@ jobs:
  if: github.actor != 'restyled-io[bot]'
 
  container:
- image: connectedhomeip/chip-build-esp32:0.6.30
+ image: connectedhomeip/chip-build-esp32:0.6.31
  volumes:
  - "/tmp/bloat_reports:/tmp/bloat_reports"
 

diff --git a/.github/workflows/examples-infineon.yaml b/.github/workflows/examples-infineon.yaml
@@ -35,7 +35,7 @@ jobs:
  if: github.actor != 'restyled-io[bot]'
 
  container:
- image: connectedhomeip/chip-build-infineon:0.6.30
+ image: connectedhomeip/chip-build-infineon:0.6.31
  volumes:
  - "/tmp/bloat_reports:/tmp/bloat_reports"
  steps:

diff --git a/.github/workflows/examples-k32w.yaml b/.github/workflows/examples-k32w.yaml
@@ -37,7 +37,7 @@ jobs:
  if: github.actor != 'restyled-io[bot]'
 
  container:
- image: connectedhomeip/chip-build-k32w:0.6.30
+ image: connectedhomeip/chip-build-k32w:0.6.31
  volumes:
  - "/tmp/bloat_reports:/tmp/bloat_reports"
  steps:

diff --git a/.github/workflows/examples-linux-arm.yaml b/.github/workflows/examples-linux-arm.yaml
@@ -34,7 +34,7 @@ jobs:
  if: github.actor != 'restyled-io[bot]'
 
  container:
- image: connectedhomeip/chip-build-crosscompile:0.6.30
+ image: connectedhomeip/chip-build-crosscompile:0.6.31
  volumes:
  - "/tmp/bloat_reports:/tmp/bloat_reports"
 

diff --git a/.github/workflows/examples-linux-imx.yaml b/.github/workflows/examples-linux-imx.yaml
@@ -34,7 +34,7 @@ jobs:
  if: github.actor != 'restyled-io[bot]'
 
  container:
- image: connectedhomeip/chip-build-imx:0.6.30
+ image: connectedhomeip/chip-build-imx:0.6.31
 
  steps:
  - uses: Wandalen/wretry.action@v1.0.36

diff --git a/.github/workflows/examples-linux-standalone.yaml b/.github/workflows/examples-linux-standalone.yaml
@@ -34,7 +34,7 @@ jobs:
  if: github.actor != 'restyled-io[bot]'
 
  container:
- image: connectedhomeip/chip-build:0.6.30
+ image: connectedhomeip/chip-build:0.6.31
  volumes:
  - "/tmp/bloat_reports:/tmp/bloat_reports"
 

diff --git a/.github/workflows/examples-mbed.yaml b/.github/workflows/examples-mbed.yaml
@@ -40,7 +40,7 @@ jobs:
  if: github.actor != 'restyled-io[bot]'
 
  container:
- image: connectedhomeip/chip-build-mbed-os:0.6.30
+ image: connectedhomeip/chip-build-mbed-os:0.6.31
  volumes:
  - "/tmp/bloat_reports:/tmp/bloat_reports"
 

diff --git a/.github/workflows/examples-mw320.yaml b/.github/workflows/examples-mw320.yaml
@@ -37,7 +37,7 @@ jobs:
  if: github.actor != 'restyled-io[bot]'
 
  container:
- image: connectedhomeip/chip-build:0.6.30
+ image: connectedhomeip/chip-build:0.6.31
  volumes:
  - "/tmp/bloat_reports:/tmp/bloat_reports"
  steps:

diff --git a/.github/workflows/examples-nrfconnect.yaml b/.github/workflows/examples-nrfconnect.yaml
@@ -37,7 +37,7 @@ jobs:
  if: github.actor != 'restyled-io[bot]'
 
  container:
- image: connectedhomeip/chip-build-nrf-platform:0.6.30
+ image: connectedhomeip/chip-build-nrf-platform:0.6.31
  volumes:
  - "/tmp/bloat_reports:/tmp/bloat_reports"
 

diff --git a/.github/workflows/examples-openiotsdk.yaml b/.github/workflows/examples-openiotsdk.yaml
@@ -35,7 +35,7 @@ jobs:
  if: github.actor != 'restyled-io[bot]'
 
  container:
- image: connectedhomeip/chip-build-openiotsdk:0.6.30
+ image: connectedhomeip/chip-build-openiotsdk:0.6.31
  volumes:
  - "/tmp/bloat_reports:/tmp/bloat_reports"
  options: --privileged

diff --git a/.github/workflows/examples-qpg.yaml b/.github/workflows/examples-qpg.yaml
@@ -37,7 +37,7 @@ jobs:
  if: github.actor != 'restyled-io[bot]'
 
  container:
- image: connectedhomeip/chip-build:0.6.30
+ image: connectedhomeip/chip-build:0.6.31
  volumes:
  - "/tmp/bloat_reports:/tmp/bloat_reports"
  steps:

diff --git a/.github/workflows/examples-telink.yaml b/.github/workflows/examples-telink.yaml
@@ -35,7 +35,7 @@ jobs:
  if: github.actor != 'restyled-io[bot]'
 
  container:
- image: connectedhomeip/chip-build-telink:0.6.30
+ image: connectedhomeip/chip-build-telink:0.6.31
  volumes:
  - "/tmp/bloat_reports:/tmp/bloat_reports"
 

diff --git a/.github/workflows/examples-tizen.yaml b/.github/workflows/examples-tizen.yaml
@@ -33,7 +33,7 @@ jobs:
  if: github.actor != 'restyled-io[bot]'
 
  container:
- image: connectedhomeip/chip-build-tizen:0.6.30
+ image: connectedhomeip/chip-build-tizen:0.6.31
  options: --user root
  volumes:
  - "/tmp/bloat_reports:/tmp/bloat_reports"

diff --git a/.github/workflows/full-android.yaml b/.github/workflows/full-android.yaml
@@ -37,7 +37,7 @@ jobs:
  if: github.actor != 'restyled-io[bot]'
 
  container:
- image: connectedhomeip/chip-build-android:0.6.30
+ image: connectedhomeip/chip-build-android:0.6.31
  volumes:
  - "/tmp/log_output:/tmp/test_logs"