Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

chore: track customization in templates #1876

Merged
merged 35 commits into from
Oct 17, 2023

Conversation

JoeWang1127
Copy link
Contributor

@JoeWang1127 JoeWang1127 commented Oct 3, 2023

Append env_vars in all kokoro build configuration files (*.cfg in .kokoro/) which contain an env_vars (key: JOB_TYPE, value: integration)

When you want to append the environment variable too all templated .cfg which has an env_vars (key: JOB_TYPE, value: integration), follow the steps below:

  1. Create a partial file in the repository root directory.
    Note that the file extension should be .yaml.
  2. Write customization entries in the partial file.
  3. Create a pull request containing changes in step 1 - 2
  4. Confirm the customizations appear in the configuration file in the same pull request after the OwlBot post processor runs.

Modify default env_vars in Kokoro build templates

You don’t need to modify existing env_vars defined in the templates because if there are multiple env_vars with the same key, the last value will be used.

Please refer to Maps in proto 3 language guide.

Example

Append env_vars in configuration files with an env_vars (key: JOB_TYPE, value: integration)

  1. Add a partial file .integration-partials.yaml to the repository root.
  2. In .integration-partials.yaml, add the following section:
integration: |
  env_vars: {
    key: "INTEGRATION_TEST_ARGS"
    value: "-P bigtable-emulator-it"
  }
  
  env_vars: {
    key: "GCLOUD_PROJECT"
    value: "gcloud-devel"
  }

  1. Create a pull request containing the changes in step 1 - 2.

After the change

The following section will be appended to

  • .kokoro/nightly/integration.cfg
  • .kokoro/nightly/java11-integration.cfg
  • .kokoro/presubmit/integration.cfg
env_vars: {
  key: "INTEGRATION_TEST_ARGS"
  value: "-P bigtable-emulator-it"
}

env_vars: {
  key: "GCLOUD_PROJECT"
  value: "gcloud-devel"
}

Because these files contain env_vars (key: JOB_TYPE, value: integration)

For env_var entry with key GCLOUD_PROJECT, the default value is overwritten by customized value (gcloud-devel) that appears later in the file.

Fix #1798.

@JoeWang1127
Copy link
Contributor Author

JoeWang1127 commented Oct 3, 2023

Kokoro CI error:

>               raise CalledProcessError(retcode, process.args,
                                         output=stdout, stderr=stderr)
E               subprocess.CalledProcessError: Command '['docker', 'build', '-t', 'owlbot-php-test', '-f', 'docker/owlbot/php/Dockerfile', '.']' returned non-zero exit status 1.

/[usr/local/lib/python3.10/subprocess.py:526](https://cs.corp.google.com/piper///depot/google3/usr/local/lib/python3.10/subprocess.py?l=526): CalledProcessError
---------------------------- Captured stderr setup -----------------------------
DEPRECATED: The legacy builder is deprecated and will be removed in a future release.
            Install the buildx component to build images with BuildKit:
            [https://docs.docker.com/go/buildx/](https://www.google.com/url?q=https://docs.docker.com/go/buildx/&sa=D)

unable to prepare context: unable to evaluate symlinks in Dockerfile path: lstat /tmpfs/src/github/synthtool/tests/fixtures/java_templates/partials/docker: no such file or directory

Full log.

@JoeWang1127
Copy link
Contributor Author

Changed test directory and only lint failed:

would reformat /tmpfs/src/github/synthtool/synthtool/gcp/common.py
would reformat /tmpfs/src/github/synthtool/synthtool/languages/java.py
would reformat /tmpfs/src/github/synthtool/tests/test_language_java.py

@JoeWang1127 JoeWang1127 marked this pull request as ready for review October 4, 2023 14:21
synthtool/gcp/common.py Outdated Show resolved Hide resolved
synthtool/gcp/partials.py Outdated Show resolved Hide resolved
synthtool/gcp/partials.py Outdated Show resolved Hide resolved
@JoeWang1127 JoeWang1127 marked this pull request as draft October 5, 2023 00:50
@JoeWang1127 JoeWang1127 marked this pull request as ready for review October 12, 2023 20:02
synthtool/gcp/partials.py Outdated Show resolved Hide resolved
synthtool/gcp/common.py Outdated Show resolved Hide resolved
synthtool/gcp/partials.py Outdated Show resolved Hide resolved
Copy link
Member

@suztomo suztomo left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Think about the possibility of not needing partial_files arguments to java_library and _load_generic_metadata. If you still feel it's better to let users to explicitly specify the partial files, then I'm good. Approving either case.

synthtool/gcp/partials.py Outdated Show resolved Hide resolved
Copy link
Contributor

@chingor13 chingor13 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Generally, looks ok to me

synthtool/gcp/common.py Outdated Show resolved Hide resolved
@JoeWang1127 JoeWang1127 changed the title feat: track customization in templates chore: track customization in templates Oct 17, 2023
@chingor13 chingor13 merged commit a792419 into master Oct 17, 2023
8 checks passed
@chingor13 chingor13 deleted the feat/track-customization-in-templates branch October 17, 2023 17:31
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Owlbot should keep track of custom modifications to templates
4 participants