Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: add generate_repo.py #2431

Merged
merged 67 commits into from
Feb 7, 2024
Merged
Changes from 1 commit
Commits
Show all changes
67 commits
Select commit Hold shift + click to select a range
8c4bd4a
feat: add generate_repo.py
JoeWang1127 Jan 30, 2024
f4bccaa
refactor model
JoeWang1127 Jan 30, 2024
b2b3fe4
refactor
JoeWang1127 Jan 30, 2024
e804991
add private methods
JoeWang1127 Jan 30, 2024
737dbbd
change config file
JoeWang1127 Jan 30, 2024
362fc27
refactor to download repo only once
JoeWang1127 Jan 31, 2024
a8cd899
use absolute path
JoeWang1127 Jan 31, 2024
449f3d5
do not remove files as owlbot cli will take care of it
JoeWang1127 Jan 31, 2024
ae4726c
move pre generation to utilities
JoeWang1127 Jan 31, 2024
c134ebf
remove new-client scripts
JoeWang1127 Jan 31, 2024
23e4732
move api download function to utilities
JoeWang1127 Jan 31, 2024
f853130
add helper function
JoeWang1127 Jan 31, 2024
1478e4e
fix unit tests
JoeWang1127 Jan 31, 2024
cd90e04
add copyright
JoeWang1127 Jan 31, 2024
5b60d25
refactor
JoeWang1127 Jan 31, 2024
997cff5
regenerate root pom.xml
JoeWang1127 Jan 31, 2024
6894a55
generate gapic libraries bom using python
JoeWang1127 Feb 1, 2024
99afa38
Merge branch 'main' into feat/generate_repo
JoeWang1127 Feb 1, 2024
a7b210a
add unit tests
JoeWang1127 Feb 1, 2024
d3bc0f6
add unit tests
JoeWang1127 Feb 1, 2024
1edbb21
do not pull git repository
JoeWang1127 Feb 2, 2024
0d14629
add unit test
JoeWang1127 Feb 2, 2024
b7c6728
refactor
JoeWang1127 Feb 2, 2024
e76647c
remove irrelevant test data
JoeWang1127 Feb 2, 2024
ec8d51e
do not use enum
JoeWang1127 Feb 2, 2024
818e3a9
use apigeeconnect in yaml
JoeWang1127 Feb 2, 2024
42cc2ee
remove customized constructor
JoeWang1127 Feb 3, 2024
d9fe30c
add get_library_name
JoeWang1127 Feb 3, 2024
b047df7
separate cli and logic
JoeWang1127 Feb 3, 2024
21fa068
code format
JoeWang1127 Feb 3, 2024
f7613c1
add integration test
JoeWang1127 Feb 3, 2024
90385b1
remove shell integration test
JoeWang1127 Feb 3, 2024
3423183
code format
JoeWang1127 Feb 3, 2024
5338681
use git, rather than subprocess
JoeWang1127 Feb 3, 2024
63a7a48
remove pyenv
JoeWang1127 Feb 3, 2024
4ee961f
add unit tests
JoeWang1127 Feb 3, 2024
c2b7b21
change workflow
JoeWang1127 Feb 3, 2024
9c583cf
set git user
JoeWang1127 Feb 3, 2024
5bdd2c6
restore mvn command
JoeWang1127 Feb 3, 2024
4d0f70a
add issue_tracker to LibraryConfig
JoeWang1127 Feb 3, 2024
066af7f
add api_reference to LibraryConfig
JoeWang1127 Feb 3, 2024
e30f5c1
compare generation result in integration test
JoeWang1127 Feb 3, 2024
d1022fb
surpress mvn output
JoeWang1127 Feb 3, 2024
342df91
only generate .repo-metadat.json once
JoeWang1127 Feb 3, 2024
7cfd7c3
create library_path if it's not exist
JoeWang1127 Feb 4, 2024
047e250
remove destination path in GeneratationConfig
JoeWang1127 Feb 4, 2024
2d71f54
remove destination path in GeneratationConfig
JoeWang1127 Feb 4, 2024
9416899
add unit tests
JoeWang1127 Feb 4, 2024
f948f31
remove delete_if_exists
JoeWang1127 Feb 4, 2024
ab2d103
add log
JoeWang1127 Feb 4, 2024
ce12090
change assert
JoeWang1127 Feb 5, 2024
0610047
handle special cases in gapic-libraries-bom/pom.xml
JoeWang1127 Feb 5, 2024
984bc83
refactor compre_poms.py
JoeWang1127 Feb 5, 2024
2535b16
compare gapic-libraries-bom/pom.xml and pom.xml in integration test
JoeWang1127 Feb 5, 2024
14e1c92
Revert "compare gapic-libraries-bom/pom.xml and pom.xml in integratio…
JoeWang1127 Feb 5, 2024
1c2d9c1
compare pom.xml in integration test
JoeWang1127 Feb 5, 2024
2aad329
format config file
JoeWang1127 Feb 5, 2024
5adf0ee
covert transport
JoeWang1127 Feb 5, 2024
b9a717c
enable diff_files
JoeWang1127 Feb 5, 2024
2e0f3a9
refactor according to code review
JoeWang1127 Feb 6, 2024
2c0f9b2
code format
JoeWang1127 Feb 6, 2024
e592f71
code format
JoeWang1127 Feb 6, 2024
369ca21
add python linter
JoeWang1127 Feb 6, 2024
c17d321
change job name
JoeWang1127 Feb 6, 2024
a0dc4cb
Merge branch 'main' into feat/generate_repo
JoeWang1127 Feb 6, 2024
0e2f22b
Merge branch 'main' into feat/generate_repo
JoeWang1127 Feb 7, 2024
c1ef755
bring back config for java-bigtable
JoeWang1127 Feb 7, 2024
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
add unit tests
JoeWang1127 committed Feb 1, 2024

Verified

This commit was created on GitHub.com and signed with GitHub’s verified signature. The key has expired.
commit d3bc0f66b9665952b3812044466ead76fe63e2f2
12 changes: 6 additions & 6 deletions library_generation/model/generation_config.py
Original file line number Diff line number Diff line change
@@ -28,22 +28,22 @@ class GenerationConfig:
def __init__(
self,
gapic_generator_version: str,
grpc_version: Optional[str],
protobuf_version: Optional[str],
googleapis_commitish: str,
owlbot_cli_image: str,
synthtool_commitish: str,
destination_path: Optional[str],
libraries: List[LibraryConfig],
destination_path: Optional[str] = None,
grpc_version: Optional[str] = None,
protobuf_version: Optional[str] = None,
):
self.gapic_generator_version = gapic_generator_version
self.grpc_version = grpc_version
self.protobuf_version = protobuf_version
self.googleapis_commitish = googleapis_commitish
self.owlbot_cli_image = owlbot_cli_image
self.synthtool_commitish = synthtool_commitish
self.destination_path = destination_path
self.libraries = libraries
self.destination_path = destination_path
self.grpc_version = grpc_version
self.protobuf_version = protobuf_version


def from_yaml(path_to_yaml: str):
110 changes: 79 additions & 31 deletions library_generation/test/unit_tests.py
Original file line number Diff line number Diff line change
@@ -24,11 +24,30 @@

from library_generation import utilities as util
from library_generation.model.gapic_config import GapicConfig
from library_generation.model.generation_config import GenerationConfig
from library_generation.model.gapic_inputs import parse as parse_build_file
from library_generation.model.library_config import LibraryConfig

script_dir = os.path.dirname(os.path.realpath(__file__))
resources_dir = os.path.join(script_dir, "resources")
build_file = Path(os.path.join(resources_dir, "misc")).resolve()
library_1 = LibraryConfig(
api_shortname="baremetalsolution",
name_pretty="Bare Metal Solution",
product_documentation="https://cloud.google.com/bare-metal/docs",
api_description="Bring your Oracle workloads to Google Cloud with Bare Metal Solution and jumpstart your cloud journey with minimal risk.",
gapic_configs=list(),
library_name="bare-metal-solution",
rest_documentation="https://cloud.google.com/bare-metal/docs/reference/rest",
rpc_documentation="https://cloud.google.com/bare-metal/docs/reference/rpc",
)
library_2 = LibraryConfig(
api_shortname="secretmanager",
name_pretty="Secret Management",
product_documentation="https://cloud.google.com/solutions/secrets-management/",
api_description="allows you to encrypt, store, manage, and audit infrastructure and application-level secrets.",
gapic_configs=list(),
)


class UtilitiesTest(unittest.TestCase):
@@ -121,82 +140,68 @@ def test_delete_if_exists_preexisting_temp_files_succeeds(self):
self.assertEqual(0, len(os.listdir(temp_dir)))

def test_client_inputs_parse_grpc_only_succeeds(self):
build_file = Path(os.path.join(resources_dir, "misc")).resolve()
parsed = parse_build_file(build_file, "", "BUILD_grpc.bazel")
self.assertEqual("grpc", parsed.transport)

def test_client_inputs_parse_grpc_rest_succeeds(self):
build_file = Path(os.path.join(resources_dir, "misc")).resolve()
parsed = parse_build_file(build_file, "", "BUILD_grpc_rest.bazel")
self.assertEqual("grpc+rest", parsed.transport)

def test_client_inputs_parse_rest_succeeds(self):
build_file = Path(os.path.join(resources_dir, "misc")).resolve()
parsed = parse_build_file(build_file, "", "BUILD_rest.bazel")
self.assertEqual("rest", parsed.transport)

def test_client_inputs_parse_empty_include_samples_succeeds(self):
build_file = Path(os.path.join(resources_dir, "misc")).resolve()
parsed = parse_build_file(build_file, "", "BUILD_include_samples_empty.bazel")
self.assertEqual("false", parsed.include_samples)

def test_client_inputs_parse_include_samples_false_succeeds(self):
build_file = Path(os.path.join(resources_dir, "misc")).resolve()
parsed = parse_build_file(build_file, "", "BUILD_include_samples_false.bazel")
self.assertEqual("false", parsed.include_samples)

def test_client_inputs_parse_include_samples_true_succeeds(self):
build_file = Path(os.path.join(resources_dir, "misc")).resolve()
parsed = parse_build_file(build_file, "", "BUILD_include_samples_true.bazel")
self.assertEqual("true", parsed.include_samples)

def test_client_inputs_parse_empty_rest_numeric_enums_succeeds(self):
build_file = Path(os.path.join(resources_dir, "misc")).resolve()
parsed = parse_build_file(
build_file, "", "BUILD_rest_numeric_enums_empty.bazel"
)
self.assertEqual("false", parsed.rest_numeric_enum)

def test_client_inputs_parse_rest_numeric_enums_false_succeeds(self):
build_file = Path(os.path.join(resources_dir, "misc")).resolve()
parsed = parse_build_file(
build_file, "", "BUILD_rest_numeric_enums_false.bazel"
)
self.assertEqual("false", parsed.rest_numeric_enum)

def test_client_inputs_parse_rest_numeric_enums_true_succeeds(self):
build_file = Path(os.path.join(resources_dir, "misc")).resolve()
parsed = parse_build_file(build_file, "", "BUILD_rest_numeric_enums_true.bazel")
self.assertEqual("true", parsed.rest_numeric_enum)

def test_client_inputs_parse_no_gapic_library_returns_proto_only_true(self):
build_file = Path(os.path.join(resources_dir, "misc")).resolve()
# include_samples_empty only has a gradle assembly rule
parsed = parse_build_file(build_file, "", "BUILD_include_samples_empty.bazel")
self.assertEqual("true", parsed.proto_only)

def test_client_inputs_parse_with_gapic_library_returns_proto_only_false(self):
build_file = Path(os.path.join(resources_dir, "misc")).resolve()
# rest.bazel has a java_gapic_library rule
parsed = parse_build_file(build_file, "", "BUILD_rest.bazel")
self.assertEqual("false", parsed.proto_only)

def test_client_inputs_parse_gapic_yaml_succeeds(self):
build_file = Path(os.path.join(resources_dir, "misc")).resolve()
parsed = parse_build_file(
build_file, "test/versioned/path", "BUILD_gapic_yaml.bazel"
)
self.assertEqual("test/versioned/path/test_gapic_yaml.yaml", parsed.gapic_yaml)

def test_client_inputs_parse_no_gapic_yaml_returns_empty_string(self):
build_file = Path(os.path.join(resources_dir, "misc")).resolve()
parsed = parse_build_file(
build_file, "test/versioned/path", "BUILD_no_gapic_yaml.bazel"
)
self.assertEqual("", parsed.gapic_yaml)

def test_client_inputs_parse_service_config_succeeds(self):
build_file = Path(os.path.join(resources_dir, "misc")).resolve()
parsed = parse_build_file(
build_file, "test/versioned/path", "BUILD_service_config.bazel"
)
@@ -205,14 +210,12 @@ def test_client_inputs_parse_service_config_succeeds(self):
)

def test_client_inputs_parse_no_service_config_returns_empty_string(self):
build_file = Path(os.path.join(resources_dir, "misc")).resolve()
parsed = parse_build_file(
build_file, "test/versioned/path", "BUILD_no_service_config.bazel"
)
self.assertEqual("", parsed.service_config)

def test_client_inputs_parse_service_yaml_succeeds(self):
build_file = Path(os.path.join(resources_dir, "misc")).resolve()
parsed = parse_build_file(
build_file, "test/versioned/path", "BUILD_service_yaml.bazel"
)
@@ -221,7 +224,6 @@ def test_client_inputs_parse_service_yaml_succeeds(self):
)

def test_client_inputs_parse_no_service_yaml_returns_empty_string(self):
build_file = Path(os.path.join(resources_dir, "misc")).resolve()
parsed = parse_build_file(
build_file, "test/versioned/path", "BUILD_no_service_yaml.bazel"
)
@@ -240,14 +242,14 @@ def test_remove_version_from_returns_self(self):
)

def test_get_version_from_returns_current(self):
versions_file = f"{resources_dir}/misc/testversions.txt"
versions_file = f"{resources_dir}/misc/versions.txt"
artifact = "gax-grpc"
self.assertEqual(
"2.33.1-SNAPSHOT", util.get_version_from(versions_file, artifact)
)

def test_get_version_from_returns_released(self):
versions_file = f"{resources_dir}/misc/testversions.txt"
versions_file = f"{resources_dir}/misc/versions.txt"
artifact = "gax-grpc"
self.assertEqual("2.34.0", util.get_version_from(versions_file, artifact, True))

@@ -259,21 +261,10 @@ def test_generate_prerequisite_files_success(self):
f"{library_path}/owlbot.py",
]
self.__cleanup(files)
library = LibraryConfig(
api_shortname="baremetalsolution",
name_pretty="Bare Metal Solution",
product_documentation="https://cloud.google.com/bare-metal/docs",
api_description="Bring your Oracle workloads to Google Cloud with Bare Metal Solution and jumpstart your cloud journey with minimal risk.",
gapic_configs=list(),
library_name="bare-metal-solution",
rest_documentation="https://cloud.google.com/bare-metal/docs/reference/rest",
rpc_documentation="https://cloud.google.com/bare-metal/docs/reference/rpc",
)
proto_path = "google/cloud/baremetalsolution/v2"
transport = "grpc"

util.generate_prerequisite_files(
library=library,
library=library_1,
proto_path=proto_path,
transport=transport,
library_path=library_path,
@@ -291,6 +282,40 @@ def test_generate_prerequisite_files_success(self):
)
self.__cleanup(files)

def test_prepare_repo_monorepo_success(self):
gen_config = self.__get_a_gen_config(2)
repo_config = util.prepare_repo(
gen_config=gen_config,
library_config=gen_config.libraries,
repo_path=f"{resources_dir}/misc",
)
self.assertEqual("output", Path(repo_config.output_folder).name)
library_path = sorted([Path(key).name for key in repo_config.libraries])
self.assertEqual(
["java-bare-metal-solution", "java-secretmanager"], library_path
)

def test_prepare_repo_monorepo_failed(self):
gen_config = self.__get_a_gen_config(2)
self.assertRaises(
FileNotFoundError,
util.prepare_repo,
gen_config,
gen_config.libraries,
f"{resources_dir}/non-exist",
)

def test_prepare_repo_split_repo_success(self):
gen_config = self.__get_a_gen_config(1)
repo_config = util.prepare_repo(
gen_config=gen_config,
library_config=gen_config.libraries,
repo_path=f"{resources_dir}/misc",
)
self.assertEqual("output", Path(repo_config.output_folder).name)
library_path = sorted([Path(key).name for key in repo_config.libraries])
self.assertEqual(["misc"], library_path)

def __compare_files(self, expect: str, actual: str):
with open(expect, "r") as f:
expected_lines = f.readlines()
@@ -302,6 +327,29 @@ def __compare_files(self, expect: str, actual: str):
first=[], second=diff, msg="Unexpected file contents:\n" + "".join(diff)
)

@staticmethod
def __get_a_gen_config(num: int):
"""
Returns an object of GenerationConfig with one or two of
LibraryConfig objects. Other attributes are set to empty str.

:param num: the number of LibraryConfig objects associated with
the GenerationConfig. Only support one or two.
:return: an object of GenerationConfig
"""
if num > 1:
libraries = [library_1, library_2]
else:
libraries = [library_1]

return GenerationConfig(
gapic_generator_version="",
googleapis_commitish="",
owlbot_cli_image="",
synthtool_commitish="",
libraries=libraries,
)

@staticmethod
def __cleanup(files: List[str]):
for file in files:
5 changes: 4 additions & 1 deletion library_generation/utilities.py
Original file line number Diff line number Diff line change
@@ -248,6 +248,7 @@ def prepare_repo(
:param repo_path: the path to which the generated repository goes
:param language: programming language of the library
:return: a RepoConfig object contained repository information
:raise FileNotFoundError if there's not versions.txt in repo_path
"""
output_folder = sh_util("get_output_folder")
print(f"output_folder: {output_folder}")
@@ -280,14 +281,16 @@ def prepare_repo(
else:
print("this is a standalone library")
if repo_path is None:
destination_path = libraries.keys()[0]
destination_path = list(libraries.keys())[0]
repo_path = f"{output_folder}/{destination_path}"
clone_out = sh_util(
f'git clone "https://github.com/googleapis/{"".join(libraries)}.git"',
cwd=output_folder,
)
print(clone_out)
versions_file = f"{repo_path}/versions.txt"
if not Path(versions_file).exists():
raise FileNotFoundError(f"{versions_file} is not found.")

return RepoConfig(
output_folder=output_folder,