feat: generate a GAPIC library from api definition #3208

JoeWang1127 · 2024-09-17T16:32:01Z

In this PR:

Generate a GAPIC library from api definition (proto, service yaml), rather than from googleapis repository.

To generate a GAPIC library with hermetic build image, the user has to prepare the api definition to a directory and set --api-definition-path to this directory.

This feature allows the user to generate a GAPIC library from any protos, rather than checking in protos in googleapis repository.

Internal ticket: b/362705386

diegomarquezp · 2024-09-18T20:35:36Z

library_generation/test/integration_tests.py

@@ -40,21 +41,21 @@
 repo_prefix = "https://github.com/googleapis"
 output_dir = shell_call("get_output_folder")
 # this map tells which branch of each repo should we use for our diff tests
-committish_map = {
+commitish_map = {


Fun fact: for a long time I thought it only was committish (two ts).
I happens to be an alternative convenience term coming from commit-ish.

The git glossary offers either commit-ish or committish. I always thought this would be too much to start a campaign to update all config yamls and its definition.

Thanks Diego! That's good to know! Yeah I agree we should follow the naming conventions. Maybe an enhancement in the backlog.

blakeli0 · 2024-09-19T17:11:56Z

library_generation/cli/entry_point.py

 def generate(
    baseline_generation_config_path: str,
    current_generation_config_path: str,
    repository_path: str,
+    api_definition_path: str,


I think this is more like a root folder of all api definitions. e.g. The proto_path configure in generation_config.yaml is a relative path to this path right?

Maybe change this to api_definitions_path or api_defs_path?

Changed it to api_definitions_path.

blakeli0 · 2024-09-19T17:12:43Z

.github/scripts/hermetic_library_generation.sh

 # run hermetic code generation docker image.
 docker run \
  --rm \
  -u "$(id -u):$(id -g)" \
  -v "$(pwd):${workspace_name}" \
  -v "${m2_folder}":/home/.m2 \
+  -v "${api_def_dir}:${workspace_name}/api" \


This is essentially googleapis folder, maybe we can just use /googleapis to make it easier to understand?

I try to use a generalized name here to specify that the script can generate from any protos, not necessarily from googleapis.

The docker image can be used by any protos, but I think hermetic_library_generation.sh is only going to be used for generating googleapis. For generating other protos, the volumn name actually doesn't really matter, it's good as long as the volumn and api-definition-path match.
That being said, it's fine if you want to make it generic, but maybe change it to apis to imply that it could contain multiple.

but I think hermetic_library_generation.sh is only going to be used for generating googleapis

Makes sense, I changed it to googleapis.

library_generation/README.md

blakeli0 · 2024-09-19T17:19:55Z

library_generation/test/integration_tests.py

@@ -40,21 +41,21 @@
 repo_prefix = "https://github.com/googleapis"
 output_dir = shell_call("get_output_folder")
 # this map tells which branch of each repo should we use for our diff tests
-committish_map = {
+commitish_map = {


Thanks Diego! That's good to know! Yeah I agree we should follow the naming conventions. Maybe an enhancement in the backlog.

blakeli0 · 2024-09-19T17:25:07Z

Thanks Joe! I think this is a good first step, but it still requires us to have a generation_config.yaml and parsing googleapis committish from it. As a follow up, can we try to see how to generate a library with only a folder? Not relying on generation_config.yaml?

JoeWang1127 · 2024-09-19T18:56:22Z

As a follow up, can we try to see how to generate a library with only a folder? Not relying on generation_config.yaml?

Sure, I'll create a follow up PR to remove googleapis_commitish in the generation config.

Generation config can be optional after all necessary parameters are read from service yaml.

blakeli0 · 2024-09-20T03:04:13Z

.github/scripts/hermetic_library_generation.sh

 # run hermetic code generation docker image.
 docker run \
  --rm \
  -u "$(id -u):$(id -g)" \
  -v "$(pwd):${workspace_name}" \
  -v "${m2_folder}":/home/.m2 \
+  -v "${api_def_dir}:${workspace_name}/api" \


The docker image can be used by any protos, but I think hermetic_library_generation.sh is only going to be used for generating googleapis. For generating other protos, the volumn name actually doesn't really matter, it's good as long as the volumn and api-definition-path match.
That being said, it's fine if you want to make it generic, but maybe change it to apis to imply that it could contain multiple.

blakeli0 · 2024-09-20T03:08:53Z

library_generation/DEVELOPMENT.md

+ * `-v /path/to/google-cloud-java:/workspace` maps the host machine's
+   google-cloud-java folder to the /workspace folder. 
+   The image is configured to perform changes in this directory.
+   Note that this setup assumes the api definition resides in host machine's


Can we change this example to use the api_definition_path parameter?

sonarcloud · 2024-09-20T17:43:37Z

Quality Gate passed for 'gapic-generator-java-root'

Issues
0 New issues
0 Accepted issues

Measures
0 Security Hotspots
0.0% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarCloud

sonarcloud · 2024-09-20T17:46:49Z

Quality Gate passed for 'java_showcase_integration_tests'

Issues
0 New issues
0 Accepted issues

Measures
0 Security Hotspots
0.0% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarCloud

🤖 I have created a release *beep* *boop* --- <details><summary>2.46.0</summary> ## [2.46.0](v2.45.0...v2.46.0) (2024-09-23) ### Features * expose property in GrpcTransportChannel if it uses direct path. ([#3170](#3170)) ([9a432f7](9a432f7)) * generate a GAPIC library from api definition ([#3208](#3208)) ([b6b5d7b](b6b5d7b)) * Metrics tracer addAttribute map overload ([#3202](#3202)) ([1a988df](1a988df)) ### Bug Fixes * generate pr description with repo level change ([#3182](#3182)) ([edd2168](edd2168)) ### Dependencies * update dependency com.google.errorprone:error_prone_annotations to v2.32.0 ([#3192](#3192)) ([b280706](b280706)) * update dependency com.google.errorprone:error_prone_annotations to v2.32.0 ([#3193](#3193)) ([ed0cd17](ed0cd17)) * update dependency filelock to v3.16.1 ([#3210](#3210)) ([703ac3d](703ac3d)) * update dependency idna to v3.10 ([#3201](#3201)) ([211c3ec](211c3ec)) * update dependency org.threeten:threetenbp to v1.7.0 ([#3205](#3205)) ([c88a722](c88a722)) * update dependency org.threeten:threetenbp to v1.7.0 ([#3206](#3206)) ([3e9fbac](3e9fbac)) * update dependency platformdirs to v4.3.3 ([#3200](#3200)) ([b62b05d](b62b05d)) * update dependency platformdirs to v4.3.6 ([#3209](#3209)) ([227ffa5](227ffa5)) * update dependency urllib3 to v2.2.3 ([#3194](#3194)) ([f69d511](f69d511)) * update dependency virtualenv to v20.26.5 ([#3212](#3212)) ([d3ef97a](d3ef97a)) * update google api dependencies ([#3183](#3183)) ([02eea8d](02eea8d)) * update google auth library dependencies to v1.26.0 ([#3216](#3216)) ([0b369e9](0b369e9)) * update google auth library dependencies to v1.27.0 ([#3221](#3221)) ([a3cb9e7](a3cb9e7)) * update googleapis/java-cloud-bom digest to 06f632d ([#3198](#3198)) ([49dcd35](49dcd35)) * update googleapis/java-cloud-bom digest to e7d8909 ([#3207](#3207)) ([de497ee](de497ee)) * update opentelemetry-java monorepo to v1.42.1 ([#3189](#3189)) ([38117d8](38117d8)) * Upgrade Protobuf-Java to v3.25.5 ([#3217](#3217)) ([860c1bc](860c1bc)) </details> --- This PR was generated with [Release Please](https://github.com/googleapis/release-please). See [documentation](https://github.com/googleapis/release-please#release-please). Co-authored-by: release-please[bot] <55107282+release-please[bot]@users.noreply.github.com>

JoeWang1127 added 2 commits September 17, 2024 12:20

feat: generate library from local api definition

7673a5c

fix broken tests

574e80c

product-auto-label bot added the size: m Pull request size is medium. label Sep 17, 2024

JoeWang1127 added 12 commits September 17, 2024 13:27

copy if dir exists

f59b5c8

copy api definition in IT

69df0e1

copy api to temp dir

8f917de

debug

d9bd2df

debug

b5491c5

copy to dir

0c7e347

remove debug output

6455d49

change readme

02e6ae2

change development guide

855d601

change doc

c6af16e

download api def

1600dfb

remove api def in IT

e3139ae

JoeWang1127 changed the title ~~feat: local protos~~ feat: generate a GAPIC library from api definition Sep 18, 2024

JoeWang1127 marked this pull request as ready for review September 18, 2024 15:55

JoeWang1127 requested review from diegomarquezp and blakeli0 September 18, 2024 15:55

diegomarquezp approved these changes Sep 18, 2024

View reviewed changes

blakeli0 reviewed Sep 19, 2024

View reviewed changes

JoeWang1127 added 5 commits September 19, 2024 14:32

change comment

a3e9840

test missing dep

69e23df

restore deps

f4f2748

update doc

bec08b0

Merge branch 'main' into feat/local-protos

a304595

blakeli0 reviewed Sep 20, 2024

View reviewed changes

Merge branch 'main' into feat/local-protos

59e9160

JoeWang1127 added 2 commits September 20, 2024 08:10

change dir name

2d8ce35

change doc

b3ef951

JoeWang1127 requested a review from blakeli0 September 20, 2024 13:04

change to api_definitions_path

5817ea9

blakeli0 approved these changes Sep 20, 2024

View reviewed changes

JoeWang1127 merged commit b6b5d7b into main Sep 20, 2024
49 of 50 checks passed

JoeWang1127 deleted the feat/local-protos branch September 20, 2024 19:11

release-please bot mentioned this pull request Sep 20, 2024

chore(main): release 2.46.0 #3187

Merged

JoeWang1127 mentioned this pull request Nov 4, 2024

docs: modify hermetic build docs #3331

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat: generate a GAPIC library from api definition #3208

feat: generate a GAPIC library from api definition #3208

JoeWang1127 commented Sep 17, 2024 •

edited

Loading

diegomarquezp Sep 18, 2024

blakeli0 Sep 19, 2024

blakeli0 Sep 19, 2024

blakeli0 Sep 20, 2024

JoeWang1127 Sep 20, 2024

blakeli0 Sep 19, 2024

JoeWang1127 Sep 19, 2024

blakeli0 Sep 20, 2024

JoeWang1127 Sep 20, 2024

blakeli0 Sep 19, 2024

blakeli0 commented Sep 19, 2024

JoeWang1127 commented Sep 19, 2024

blakeli0 Sep 20, 2024

blakeli0 Sep 20, 2024

JoeWang1127 Sep 20, 2024

sonarcloud bot commented Sep 20, 2024

sonarcloud bot commented Sep 20, 2024

feat: generate a GAPIC library from api definition #3208

feat: generate a GAPIC library from api definition #3208

Conversation

JoeWang1127 commented Sep 17, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

blakeli0 commented Sep 19, 2024

JoeWang1127 commented Sep 19, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

sonarcloud bot commented Sep 20, 2024

Quality Gate passed for 'gapic-generator-java-root'

sonarcloud bot commented Sep 20, 2024

Quality Gate passed for 'java_showcase_integration_tests'

JoeWang1127 commented Sep 17, 2024 •

edited

Loading