Release 0.15.0 (2024-01-23)

### Features * Add an ability to add default tags * Add AWS cpu utilization and memory utilization dashboards * Add bazel configs prod and non_prod * Add change notifier error count to the AWS metrics dashboard * Add delta based realtime updates publisher * Add delta record limited file writer * Add GCP system metrics dashboard for kv server * Add gcp_project_id flag when --platform=gcp is specified * add go script for pushing gcp docker image * Add optional logical_commit_time param to wasm macros. * Add sharded realtime message batcher * Add sharding data locality terraform parameters * Add sharding support for publishing engine * Add system metrics to log cpu and memory utilization * Add UDF timeout parameter value to terraform * Allow prefix path segments for gRPC http paths. * Allow setting logging verbosity level through a parameter * Data locality support * Enable prettier pre-commit hook for JavaScript * Expose http_api_paths tf variable to allow prefix matching * Log error metrics for the change notifier * Log metrics in the retries with new Telemetry API * Migrate realtime metrics to new Telemetry API * Remove envoy health checks * Remove metrics recorder from blob storage client * Remove metrics recorder from data orchestrator * Remove metrics recorder from data reader * Set UDF timeout from parameters * Support passing in shard metadata via data_cli flags * Update default tretyakov aws env settings * Update v1 to return status per key ### Bug Fixes * Add all record values to switch cases * add data-bucket-id back as a flag suffix. * Add run_all_tests bazel config * bazel config for emscripten no longer needs tool chain resolution * Check grpc channel connection before sending requests * **deps:** Upgrade data-plane-shared-libraries to 44d1d64 2024-01-08 * envoy.yaml permission and pre-commit fix * Integrate RomaService changes from common repo * Make some cache logging more precise. * Missing absl deps * Prevent Config from being copied into RomaService * rectify kv_service_port in terraform. * Remove jaeger as it has been deprecated * Remove local platform for common repo and fix CloudPlatform import * Update 'X-allow-fledge' header to 'Ad-Auction-Allowed' according to the spec. * update GCP tentb environment * Update UDFs to fix breakages due to FunctionBindingObjectV2 * Upgrade otel collector to the latest version ### Dependencies * **deps:** Upgrade data-plane-shared to commit f0d0b89 2023-11-14 ### GCP: Features * **GCP:** Add external load balancer and envoy. * **GCP:** store Terraform state remotely in GCS bucket ### Documentation * [Sharding] Add info about NUMA clusters * Add docs for server cpu and memory profiling * Add info on default wasm macro linkopts and memory limits. * Add state transition sequence diagrams for aws/gcp sharded/nonsharded configurations * Data locality * Doc for aws private communication setup * Sharding -- add a link to a tool that sets shard id in the delta metadata * Specify --init flag for docker run commands Bug: N/A Change-Id: I85fdf31ca798b372b4fd90599a255e2c8f3348ff GitOrigin-RevId: 0b7051a3ad7ae73039edb6c1d067b5c749a8a082
privacysandbox · Jan 29, 2024 · 9930c86 · 9930c86
1 parent 186e758
commit 9930c86
Show file tree

Hide file tree

Showing 341 changed files with 11,041 additions and 3,106 deletions.
diff --git a/.bazelignore b/.bazelignore
@@ -1,3 +1,3 @@
 bazel-kv-server
 tools/wasm_example/
-google_internal/
+google_internal/piper/
diff --git a/.bazelrc b/.bazelrc
@@ -11,6 +11,9 @@ build --config=noexcept
 # Disable some ROMA error checking
 build --per_file_copt=.*sandboxed_api.*@-Wno-return-type
 
+build:run_all_tests --cache_test_results=no
+test:run_all_tests --test_verbose_timeout_warnings
+
 # Broadly disable C++ Exceptions
 # https://google.github.io/styleguide/cppguide.html#Exceptions
 build:noexcept --copt=-fno-exceptions
@@ -35,11 +38,9 @@ build:clang --client_env=CC=clang
 build:clang --cxxopt=-std=c++17
 build:clang --host_cxxopt=-std=c++17
 build:clang --client_env=BAZEL_CXXOPTS=-std=c++17
+build:clang --per_file_copt=external/nitrokmscli_.*\.c@-Wno-int-conversion
 
-# Required to run wasm_cc_binary
-build:emscripten --incompatible_enable_cc_toolchain_resolution
-
-# Required to use protos in wasm_cc_binary
+# Required to use protos in wasm_cc_binary/inline_wasm_cc_binary
 build:emscripten --per_file_copt=.*zlib.*@-Wno-deprecated-non-prototype
 build:emscripten --per_file_copt=.*utf8_range.*@-Wno-unused-function
 
@@ -82,32 +83,32 @@ build:ubsan --copt=-fno-omit-frame-pointer
 build:ubsan --linkopt:-fsanitize=undefined
 build:ubsan --linkopt=-lubsan
 
-# --config local_instance: builds the service to run with the instance=local flag
+# --config local_instance: builds assuming the instance is local
 build:local_instance --//:instance=local
 build:local_instance --@google_privacysandbox_servers_common//:instance=local
 
-# --config local_instance: builds the service to run in AWS
+# --config aws_instance: builds assuming the instance is on AWS
 build:aws_instance --//:instance=aws
 build:aws_instance --@google_privacysandbox_servers_common//:instance=aws
 
-# --config local_instance: builds the service to run in GCP
+# --config gcp_instance: builds assuming the instance is on GCP
 build:gcp_instance --//:instance=gcp
 build:gcp_instance --@google_privacysandbox_servers_common//:instance=gcp
 
-# --config aws_platform: builds the service to run with AWS
+# --config local_platform: builds the service to run locally
 build:local_platform --//:platform=local
-build:local_platform --@google_privacysandbox_servers_common//:platform=local
 
 # --config aws_platform: builds the service to run with AWS
 build:aws_platform --//:platform=aws
 build:aws_platform --@google_privacysandbox_servers_common//:platform=aws
-build:aws_platform --@google_privacysandbox_servers_common//scp/cc/public/cpio/interface:platform=aws
 
-# --config gcp_platform: builds the service to run with AWS
+# --config gcp_platform: builds the service to run with GCP
 build:gcp_platform --//:platform=gcp
 build:gcp_platform --@google_privacysandbox_servers_common//:platform=gcp
-build:gcp_platform --@google_privacysandbox_servers_common//scp/cc/public/cpio/interface:platform=gcp
 
 try-import %workspace%/builders/.coverage.bazelrc
 coverage --test_tag_filters=-nocoverage
 coverage --test_size_filters=-enormous
+
+build:non_prod --@google_privacysandbox_servers_common//:build_flavor=non_prod
+build:prod --@google_privacysandbox_servers_common//:build_flavor=prod
diff --git a/.pre-commit-config.yaml b/.pre-commit-config.yaml
@@ -19,6 +19,7 @@ exclude: (?x)^(
              .bazel_output/.*|
              builders/.*|
              .*/BUILD|
+             public/data_loading/data_loading_generated.h|
              testing/functionaltest-system/.*|
              version.txt
          )$
@@ -94,7 +95,7 @@ repos:
     description: Run terraform via docker to format Terraform files
     language: script
     pass_filenames: false
-    entry: tools/terraform fmt -write=true -recursive production/terraform
+    entry: builders/tools/terraform fmt -write=true -recursive production/terraform
     types_or:
     - terraform
 
@@ -103,8 +104,9 @@ repos:
   hooks:
   - id: prettier
     types_or:
-    - markdown
+    - javascript
     - json
+    - markdown
     exclude: (?x)^(
                  testing/functionaltest/.*/test_cases/.*|
                  production/packaging/aws/data_server/nitro-pcr0/.*

diff --git a/BUILD.bazel b/BUILD.bazel
@@ -13,6 +13,7 @@
 # limitations under the License.
 
 load("@bazel_skylib//rules:common_settings.bzl", "string_flag")
+load("@io_bazel_rules_go//go:def.bzl", "nogo")
 
 package(default_visibility = ["//:__subpackages__"])
 
@@ -101,6 +102,7 @@ config_setting(
     },
     visibility = [
         "//components:__subpackages__",
+        "//production/packaging:__subpackages__",
         "//tools:__subpackages__",
     ],
 )
@@ -164,3 +166,9 @@ EOF""",
 exports_files([
     "buf.yaml",
 ])
+
+nogo(
+    name = "kv_nogo",
+    vet = True,
+    visibility = ["//visibility:public"],
+)
diff --git a/CHANGELOG.md b/CHANGELOG.md
@@ -2,6 +2,88 @@
 
 All notable changes to this project will be documented in this file. See [commit-and-tag-version](https://github.com/absolute-version/commit-and-tag-version) for commit guidelines.
 
+## 0.15.0 (2024-01-23)
+
+
+### Features
+
+* Add an ability to add default tags
+* Add AWS cpu utilization and memory utilization dashboards
+* Add bazel configs prod and non_prod
+* Add change notifier error count to the AWS metrics dashboard
+* Add delta based realtime updates publisher
+* Add delta record limited file writer
+* Add GCP system metrics dashboard for kv server
+* Add gcp_project_id flag when --platform=gcp is specified
+* add go script for pushing gcp docker image
+* Add optional logical_commit_time param to wasm macros.
+* Add sharded realtime message batcher
+* Add sharding data locality terraform parameters
+* Add sharding support for publishing engine
+* Add system metrics to log cpu and memory utilization
+* Add UDF timeout parameter value to terraform
+* Allow prefix path segments for gRPC http paths.
+* Allow setting logging verbosity level through a parameter
+* Data locality support
+* Enable prettier pre-commit hook for JavaScript
+* Expose http_api_paths tf variable to allow prefix matching
+* Log error metrics for the change notifier
+* Log metrics in the retries with new Telemetry API
+* Migrate realtime metrics to new Telemetry API
+* Remove envoy health checks
+* Remove metrics recorder from blob storage client
+* Remove metrics recorder from data orchestrator
+* Remove metrics recorder from data reader
+* Set UDF timeout from parameters
+* Support passing in shard metadata via data_cli flags
+* Update default tretyakov aws env settings
+* Update v1 to return status per key
+
+
+### Bug Fixes
+
+* Add all record values to switch cases
+* add data-bucket-id back as a flag suffix.
+* Add run_all_tests bazel config
+* bazel config for emscripten no longer needs tool chain resolution
+* Check grpc channel connection before sending requests
+* **deps:** Upgrade data-plane-shared-libraries to 44d1d64 2024-01-08
+* envoy.yaml permission and pre-commit fix
+* Integrate RomaService changes from common repo
+* Make some cache logging more precise.
+* Missing absl deps
+* Prevent Config from being copied into RomaService
+* rectify kv_service_port in terraform.
+* Remove jaeger as it has been deprecated
+* Remove local platform for common repo and fix CloudPlatform import
+* Update 'X-allow-fledge' header to 'Ad-Auction-Allowed' according to the spec.
+* update GCP tentb environment
+* Update UDFs to fix breakages due to FunctionBindingObjectV2
+* Upgrade otel collector to the latest version
+
+
+### Dependencies
+
+* **deps:** Upgrade data-plane-shared to commit f0d0b89 2023-11-14
+
+
+### GCP: Features
+
+* **GCP:** Add external load balancer and envoy.
+* **GCP:** store Terraform state remotely in GCS bucket
+
+
+### Documentation
+
+* [Sharding] Add info about NUMA clusters
+* Add docs for server cpu and memory profiling
+* Add info on default wasm macro linkopts and memory limits.
+* Add state transition sequence diagrams for aws/gcp sharded/nonsharded configurations
+* Data locality
+* Doc for aws private communication setup
+* Sharding -- add a link to a tool that sets shard id in the delta metadata
+* Specify --init flag for docker run commands
+
 ## 0.14.0 (2023-11-07)
 
 

diff --git a/README.md b/README.md
@@ -73,6 +73,7 @@ changes.
 -   [FLEDGE K/V server trust model](https://github.com/privacysandbox/fledge-docs/blob/main/key_value_service_trust_model.md)
 -   [Local server quickstart guide](/docs/developing_the_server.md)
 -   [AWS server user deployment documentation](/docs/deploying_on_aws.md)
+-   [GCP server user deployment documentation](/docs/deploying_on_gcp.md)
 -   [Integrating the K/V server with FLEDGE](/docs/integrating_with_fledge.md)
 -   [FLEDGE K/V server sharding explainer](https://github.com/privacysandbox/fledge-docs/blob/main/key_value_sharding.md)
 -   Operating documentation

diff --git a/WORKSPACE b/WORKSPACE
@@ -13,11 +13,11 @@ python_deps("//builders/bazel")
 
 http_archive(
     name = "google_privacysandbox_servers_common",
-    # commit 3a47a5e 2023-11-01
-    sha256 = "d1a5cc31686e04f7c192f103600431f804689fbea54722250484ef38f9c8c7bf",
-    strip_prefix = "data-plane-shared-libraries-bd2efade9f130c7c8e0dbe964183204c6ee220ce",
+    # commit f198e86 2024-01-16
+    sha256 = "979d467165bef950ed69bc913e9ea743bfc068714fb43cf61e02b52b910a5561",
+    strip_prefix = "data-plane-shared-libraries-f198e86307028ad98c38a8ed1c72189eefa97334",
     urls = [
-        "https://github.com/privacysandbox/data-plane-shared-libraries/archive/bd2efade9f130c7c8e0dbe964183204c6ee220ce.zip",
+        "https://github.com/privacysandbox/data-plane-shared-libraries/archive/f198e86307028ad98c38a8ed1c72189eefa97334.zip",
     ],
 )
 
@@ -47,24 +47,17 @@ data_plane_shared_deps4()
 load(
     "//third_party_deps:cpp_repositories.bzl",
     "cpp_repositories",
-    "emscripten_repositories",
 )
 
 cpp_repositories()
 
-EMSCRIPTEN_VER = emscripten_repositories()
-
 load("//third_party_deps:container_deps.bzl", "container_deps")
 
 container_deps()
 
-load("//third_party_deps:emscripten_deps1.bzl", "emscripten_deps1")
-
-emscripten_deps1()
-
-load("//third_party_deps:emscripten_deps2.bzl", "emscripten_deps2")
+load("@io_bazel_rules_docker//go:image.bzl", go_image_repos = "repositories")
 
-emscripten_deps2(EMSCRIPTEN_VER)
+go_image_repos()
 
 # googleapis
 http_archive(
@@ -125,3 +118,10 @@ install_deps()
 load("//third_party_deps:rules_closure_repositories.bzl", "rules_closure_repositories")
 
 rules_closure_repositories()
+
+# Use nogo to run `go vet` with bazel
+load("@io_bazel_rules_go//go:deps.bzl", "go_register_toolchains", "go_rules_dependencies")
+
+go_rules_dependencies()
+
+go_register_toolchains(nogo = "@//:kv_nogo")
diff --git a/components/cloud_config/BUILD.bazel b/components/cloud_config/BUILD.bazel
@@ -17,6 +17,7 @@ load("@rules_cc//cc:defs.bzl", "cc_library", "cc_test")
 
 package(default_visibility = [
     "//components:__subpackages__",
+    "//production/packaging:__subpackages__",
 ])
 
 cc_library(
@@ -28,6 +29,7 @@ cc_library(
         "parameter_client.h",
     ],
     deps = [
+        "//public:constants",
         "@com_github_google_glog//:glog",
         "@com_google_absl//absl/container:flat_hash_map",
         "@com_google_absl//absl/flags:flag",
@@ -58,26 +60,17 @@ cc_library(
     ],
 )
 
-# TODO(b/299623229): use CPIO parameter client for both aws and gcp's local/cloud client.
-selects.config_setting_group(
-    name = "gcp_platform_local_instance",
-    match_all = [
-        "//:gcp_platform",
-        "//:local_instance",
-    ],
-)
-
 cc_test(
     name = "parameter_client_test",
     size = "small",
     srcs = selects.with_or({
         "//:aws_platform": [
             "parameter_client_aws_test.cc",
         ],
-        "//:gcp_instance": [
+        "//:gcp_platform": [
             ":parameter_client_gcp_test.cc",
         ],
-        ("//:local_platform", ":gcp_platform_local_instance"): [
+        "//:local_platform": [
             ":parameter_client_local_test.cc",
         ],
     }),
@@ -87,12 +80,12 @@ cc_test(
             "@aws_sdk_cpp//:core",
             "@aws_sdk_cpp//:ssm",
         ],
-        "//:gcp_instance": [
+        "//:gcp_platform": [
             "@com_google_googletest//:gtest",
             "@google_privacysandbox_servers_common//scp/cc/public/cpio/interface:cpio",
             "@google_privacysandbox_servers_common//scp/cc/public/cpio/mock/parameter_client:parameter_client_mock",
         ],
-        ("//:local_platform", ":gcp_platform_local_instance"): [
+        "//:local_platform": [
             "//components/data/common:mocks",
             "//components/util:sleepfor_mock",
             "@com_github_google_glog//:glog",
@@ -108,19 +101,19 @@ cc_library(
     name = "parameter_client",
     srcs = selects.with_or({
         "//:aws_platform": ["parameter_client_aws.cc"],
-        "//:gcp_instance": [":parameter_client_gcp"],
-        ("//:local_platform", ":gcp_platform_local_instance"): [":parameter_client_local"],
+        "//:gcp_platform": [":parameter_client_gcp"],
+        "//:local_platform": [":parameter_client_local"],
     }),
     hdrs = [
         "parameter_client.h",
     ],
-    deps = select({
+    deps = selects.with_or({
         "//:aws_platform": [
             "//components/errors:aws_error_util",
             "@aws_sdk_cpp//:core",
             "@aws_sdk_cpp//:ssm",
         ],
-        "//:gcp_instance": [
+        "//:gcp_platform": [
             "@google_privacysandbox_servers_common//scp/cc/public/core/interface:errors",
             "@google_privacysandbox_servers_common//scp/cc/public/cpio/interface:cpio",
             "@google_privacysandbox_servers_common//scp/cc/public/cpio/interface/parameter_client",
@@ -151,6 +144,7 @@ cc_library(
             "@aws_sdk_cpp//:ec2",
         ],
         "//:gcp_instance": [
+            "//components/errors:gcp_error_util",
             "@com_github_googleapis_google_cloud_cpp//:compute_instances",
             "@com_google_absl//absl/flags:flag",
             "@com_google_absl//absl/log:check",
@@ -167,6 +161,7 @@ cc_library(
         "@com_google_absl//absl/container:flat_hash_set",
         "@com_google_absl//absl/status",
         "@com_google_absl//absl/status:statusor",
-        "@google_privacysandbox_servers_common//src/cpp/telemetry:metrics_recorder",
+        "@com_google_absl//absl/strings",
+        "@google_privacysandbox_servers_common//src/cpp/util/status_macro:status_macros",
     ],
 )
diff --git a/components/cloud_config/instance_client.h b/components/cloud_config/instance_client.h
@@ -21,7 +21,6 @@
 #include "absl/container/flat_hash_map.h"
 #include "absl/container/flat_hash_set.h"
 #include "absl/status/statusor.h"
-#include "src/cpp/telemetry/metrics_recorder.h"
 
 // TODO: Replace config cpio client once ready
 namespace kv_server {
@@ -56,8 +55,7 @@ using DescribeInstanceGroupInput =
 // Client to perform instance-specific operations.
 class InstanceClient {
  public:
-  static std::unique_ptr<InstanceClient> Create(
-      privacy_sandbox::server_common::MetricsRecorder& metrics_recorder);
+  static std::unique_ptr<InstanceClient> Create();
   virtual ~InstanceClient() = default;
 
   // Retrieves all tags for the current instance and returns the tag with the