Release 0.16.0 (2024-04-05)

### Features

* Add cache hit or miss metrics
* Add coorindator specific terraform parameters
* Add data loading prefix allowlist parameter
* Add default PAS UDF
* Add E2E latency for GetKeyValues and GetKeyValueSet in sharded lookup
* Add file groups and file group reader logic
* Add go fmt to pre-commit
* Add key prefix support to blob storage client
* Add LogContext and ConsentedDebugConfiguration proto to v2 API and internal lookup API
* Add prod and nonprod build flag
* Add request context to wrap metrics context
* Add support for configuring directory allowlist
* Add wiring for prefix allowlist (actual impl in follow up cl)
* Allow overrides for coordinators endpoints in nonprod mode
* Allow to disable v1 key not found entry in response
* Create separate metrics context map for internal lookup server
* Deprecate metrics recorder for internal lookup
* Deprecate metrics recorder for internal server
* Deprecate metrics recorder for sharded lookup
* deprecate metrics recorder for V1 server and handler
* Deprecate metrics recorder from cache
* Enable simulation system send realtime udpates
* Enable TCMalloc for KV Server and benchmarks
* Explicitly enable core dumps
* Implement deletion cutoff max timestamp per directory
* Load data files and allow notifications from configured prefix
* Load prefix files on startup and handle prefix blob notifications
* Log common request metrics
* Migrate from glog to absl log
* Partition data loading metrics by delta file name
* Pass request context from hooks to downstream components
* Pass request context to udf hooks
* Read telemetry config from cloud parameter
* Revamp AWS metrics dashboard
* Revamp GCP metrics dashboard
* Set udf_min_log_level from parameter store.
* Support content type proto for v2 api
* Support content type proto for v2 api response
* Update cache interface and blob data location to pass prefix
* Update start_after to use a map from prefix to start_after
* Use file groups for loading snapshots
* Write logs to an Otel endpoint

### Bug Fixes

* Actually load all files in a snapshot file group
* **AWS:** Filter out unavailable zones.
* Correct an error in kokoro_release.
* Correct format for image tag.
* Correct typo for internal dev's service_mesh_address.
* Correct typos in GCP deployment guide.
* Crash server if default UDF fails to load.
* Delete non-active certificate before creating a new one.
* Fix filtering logic for prefixed blobs
* Fix permissions for data-loading-blob-prefix-allowlist
* Make GCP nat optional.
* Parse delta filename from notification before validating it
* Remove glog dependency for record_utils
* Remove temp dir only if it's successfully created.
* Rename class to ThreadManager
* Set retain_initial_value_of_delta_metric flag for aws metrics exporter
* Update a outdated hyperlink.
* Update common repo to pick up the AWS metrics dimension fix
* Update GCP Terraform with ability to delete unhealthy instance.
* Update tf variables to use gorekore instead of kelvingorekore
* Use blob key instead of prefixed basename

### GCP: Fixes

* **GCP:** Make sure server is connected to otel collector before reaching to ready state

### GCP: Features

* **GCP:** Applying Terraform pulls docker image with new tag.
* **GCP:** Make service mesh address configurable.
* **GCP:** Make subnet ip cidr configurable.
* **GCP:** Make xlb/envoy optional.

### Documentation

* Add ad retrieval explainer.
* Add docs for directory support
* Add PA and PAS folders
* Add PAS developer guide
* Add public docs for file groups
* Ads retreival explainer update.

### Dependencies

* **deps:** Add clang-tidy bazel config
* **deps:** Add cpp_nowarn bazel config
* **deps:** Upgrade bazel to 6.5.0
* **deps:** Upgrade build-system to 0.55.1
* **deps:** Upgrade build-system to 0.55.2
* **deps:** Upgrade build-system to 0.57.0
* **deps:** Upgrade data-plane-shared repo
* **deps:** Upgrade data-plane-shared repo to 1684674 2024-02-09
* **deps:** Upgrade data-plane-shared-libraries to 1fbac46
* **deps:** Upgrade pre-commit hooks

Bug: N/A
Change-Id: If188118c8459f412bcedaa2e2ee670f8c0045727
GitOrigin-RevId: 7e6c7c71d308a2c0f6401af2b96c5acedfd39f58

.bazelrc

@@ -19,12 +19,12 @@ test:run_all_tests --test_verbose_timeout_warnings
 build:noexcept --copt=-fno-exceptions
 # Grant exceptions to some dependencies so they can use exceptions
 build:noexcept --per_file_copt=.*boost.*@-fexceptions
-build:noexcept --per_file_copt=.*cc/aws/proxy.*@-fexceptions
-build:noexcept --per_file_copt=.*cc/roma.*@-fexceptions
+build:noexcept --per_file_copt=.*src/aws/proxy.*@-fexceptions
+build:noexcept --per_file_copt=.*src/roma.*@-fexceptions
 build:noexcept --per_file_copt=.*oneTBB.*@-fexceptions
 build:noexcept --per_file_copt=.*com_github_nghttp2_nghttp2.*@-fexceptions
-build:noexcept --per_file_copt=.*cc/core.*@-fexceptions
-build:noexcept --per_file_copt=.*cc/cpio.*@-fexceptions
+build:noexcept --per_file_copt=.*src/core.*@-fexceptions
+build:noexcept --per_file_copt=.*src/cpio.*@-fexceptions
 
 test --test_output=errors
 # Disable ICU linking for googleurl.
@@ -40,9 +40,17 @@ build:clang --host_cxxopt=-std=c++17
 build:clang --client_env=BAZEL_CXXOPTS=-std=c++17
 build:clang --per_file_copt=external/nitrokmscli_.*\.c@-Wno-int-conversion
 
+build:cpp_nowarn --copt=-Werror
+build:cpp_nowarn --per_file_copt=external/.*@-Wno-error
+
+build:clang-tidy --aspects @bazel_clang_tidy//clang_tidy:clang_tidy.bzl%clang_tidy_aspect
+build:clang-tidy --output_groups=report
+build:clang-tidy --@bazel_clang_tidy//:clang_tidy_config=//:clang_tidy_config
+
 # Required to use protos in wasm_cc_binary/inline_wasm_cc_binary
 build:emscripten --per_file_copt=.*zlib.*@-Wno-deprecated-non-prototype
 build:emscripten --per_file_copt=.*utf8_range.*@-Wno-unused-function
+build:emscripten --per_file_copt=.*protobuf.*@-Wno-deprecated-declarations
 
 # Address sanitizer, set action_env to segregate cache entries
 build:asan --action_env=PRIVACY_SANDBOX_SERVERS_ASAN=1
@@ -106,9 +114,14 @@ build:aws_platform --@google_privacysandbox_servers_common//:platform=aws
 build:gcp_platform --//:platform=gcp
 build:gcp_platform --@google_privacysandbox_servers_common//:platform=gcp
 
+# --config prod_mode: builds the service in prod mode
+build:prod_mode --//:mode=prod
+build:prod_mode --@google_privacysandbox_servers_common//:build_flavor=prod
+
+# --config prod_mode: builds the service in prod mode
+build:nonprod_mode --//:mode=nonprod
+build:nonprod_mode --@google_privacysandbox_servers_common//:build_flavor=non_prod
+
 try-import %workspace%/builders/.coverage.bazelrc
 coverage --test_tag_filters=-nocoverage
 coverage --test_size_filters=-enormous
-
-build:non_prod --@google_privacysandbox_servers_common//:build_flavor=non_prod
-build:prod --@google_privacysandbox_servers_common//:build_flavor=prod

.bazelversion

@@ -1 +1 @@
-6.3.2
+6.5.0

.pre-commit-config.yaml

@@ -27,7 +27,7 @@ exclude: (?x)^(
 fail_fast: true
 repos:
 - repo: https://github.com/pre-commit/pre-commit-hooks
-  rev: v4.4.0
+  rev: v4.5.0
   hooks:
   - id: end-of-file-fixer
   - id: fix-byte-order-marker
@@ -43,6 +43,12 @@ repos:
   - id: check-executables-have-shebangs
   - id: detect-private-key
 
+- repo: https://github.com/tcort/markdown-link-check
+  rev: v3.11.2
+  hooks:
+    - id: markdown-link-check
+      args: [-c .precommit_configs/markdown-link-check.json]
+
 - repo: https://github.com/jumanjihouse/pre-commit-hooks
   rev: 3.0.0
   hooks:
@@ -54,12 +60,12 @@ repos:
     exclude: ^(google_internal|builders/images)/.*$
 
 - repo: https://github.com/bufbuild/buf
-  rev: v1.26.1
+  rev: v1.29.0
   hooks:
   - id: buf-format
 
 - repo: https://github.com/pre-commit/mirrors-clang-format
-  rev: v16.0.6
+  rev: v17.0.6
   hooks:
   - id: clang-format
     types_or:
@@ -100,7 +106,7 @@ repos:
     - terraform
 
 - repo: https://github.com/pre-commit/mirrors-prettier
-  rev: v3.0.3
+  rev: v3.1.0
   hooks:
   - id: prettier
     types_or:
@@ -113,7 +119,7 @@ repos:
              )$
 
 - repo: https://github.com/DavidAnson/markdownlint-cli2
-  rev: v0.9.2
+  rev: v0.12.1
   hooks:
   - id: markdownlint-cli2
     name: lint markdown
@@ -148,7 +154,13 @@ repos:
     - --quiet
 
 - repo: https://github.com/psf/black
-  rev: 23.7.0
+  rev: 24.2.0
   hooks:
   - id: black
     name: black python formatter
+
+- repo: https://github.com/tekwizely/pre-commit-golang
+  rev: v1.0.0-rc.1
+  hooks:
+  - id: go-fmt
+    name: go format

.precommit_configs/markdown-link-check.json

@@ -0,0 +1,27 @@
+{
+  "ignorePatterns": [
+    {
+      "pattern": "^tg/"
+    },
+    {
+      "pattern": "^http://localhost"
+    },
+    {
+      "pattern": "^https://demo.kv-server.your-domain.example/"
+    },
+    {
+      "pattern": "^demo.kv-server.your-domain.example:8443"
+    }
+  ],
+  "replacementPatterns": [
+    {
+      "pattern": "^/",
+      "replacement": "{{BASEURL}}/"
+    }
+  ],
+  "timeout": "20s",
+  "retryOn429": true,
+  "retryCount": 5,
+  "fallbackRetryDelay": "30s",
+  "aliveStatusCodes": [200, 206]
+}

BUILD.bazel

@@ -12,6 +12,7 @@
 # See the License for the specific language governing permissions and
 # limitations under the License.
 
+load("@bazel_skylib//lib:selects.bzl", "selects")
 load("@bazel_skylib//rules:common_settings.bzl", "string_flag")
 load("@io_bazel_rules_go//go:def.bzl", "nogo")
 
@@ -107,6 +108,69 @@ config_setting(
     ],
 )
 
+string_flag(
+    name = "mode",
+    build_setting_default = "prod",
+    values = [
+        "prod",
+        "nonprod",
+    ],
+)
+
+config_setting(
+    name = "prod_mode",
+    flag_values = {
+        ":mode": "prod",
+    },
+    visibility = [
+        "//components:__subpackages__",
+        "//tools:__subpackages__",
+    ],
+)
+
+config_setting(
+    name = "nonprod_mode",
+    flag_values = {
+        ":mode": "nonprod",
+    },
+    visibility = [
+        "//components:__subpackages__",
+        "//tools:__subpackages__",
+    ],
+)
+
+selects.config_setting_group(
+    name = "aws_prod",
+    match_all = [
+        "//:aws_platform",
+        "//:prod_mode",
+    ],
+)
+
+selects.config_setting_group(
+    name = "aws_nonprod",
+    match_all = [
+        "//:aws_platform",
+        "//:nonprod_mode",
+    ],
+)
+
+selects.config_setting_group(
+    name = "gcp_prod",
+    match_all = [
+        "//:gcp_platform",
+        "//:prod_mode",
+    ],
+)
+
+selects.config_setting_group(
+    name = "gcp_nonprod",
+    match_all = [
+        "//:gcp_platform",
+        "//:nonprod_mode",
+    ],
+)
+
 exports_files(
     [".bazelversion"],
 )

CHANGELOG.md

@@ -2,6 +2,116 @@
 
 All notable changes to this project will be documented in this file. See [commit-and-tag-version](https://github.com/absolute-version/commit-and-tag-version) for commit guidelines.
 
+## 0.16.0 (2024-04-05)
+
+
+### Features
+
+* Add cache hit or miss metrics
+* Add coorindator specific terraform parameters
+* Add data loading prefix allowlist parameter
+* Add default PAS UDF
+* Add E2E latency for GetKeyValues and GetKeyValueSet in sharded lookup
+* Add file groups and file group reader logic
+* Add go fmt to pre-commit
+* Add key prefix support to blob storage client
+* Add LogContext and ConsentedDebugConfiguration proto to v2 API and internal lookup API
+* Add prod and nonprod build flag
+* Add request context to wrap metrics context
+* Add support for configuring directory allowlist
+* Add wiring for prefix allowlist (actual impl in follow up cl)
+* Allow overrides for coordinators endpoints in nonprod mode
+* Allow to disable v1 key not found entry in response
+* Create separate metrics context map for internal lookup server
+* Deprecate metrics recorder for internal lookup
+* Deprecate metrics recorder for internal server
+* Deprecate metrics recorder for sharded lookup
+* deprecate metrics recorder for V1 server and handler
+* Deprecate metrics recorder from cache
+* Enable simulation system send realtime udpates
+* Enable TCMalloc for KV Server and benchmarks
+* Explicitly enable core dumps
+* Implement deletion cutoff max timestamp per directory
+* Load data files and allow notifications from configured prefix
+* Load prefix files on startup and handle prefix blob notifications
+* Log common request metrics
+* Migrate from glog to absl log
+* Partition data loading metrics by delta file name
+* Pass request context from hooks to downstream components
+* Pass request context to udf hooks
+* Read telemetry config from cloud parameter
+* Revamp AWS metrics dashboard
+* Revamp GCP metrics dashboard
+* Set udf_min_log_level from parameter store.
+* Support content type proto for v2 api
+* Support content type proto for v2 api response
+* Update cache interface and blob data location to pass prefix
+* Update start_after to use a map from prefix to start_after
+* Use file groups for loading snapshots
+* Write logs to an Otel endpoint
+
+
+### Bug Fixes
+
+* Actually load all files in a snapshot file group
+* **AWS:** Filter out unavailable zones.
+* Correct an error in kokoro_release.
+* Correct format for image tag.
+* Correct typo for internal dev's service_mesh_address.
+* Correct typos in GCP deployment guide.
+* Crash server if default UDF fails to load.
+* Delete non-active certificate before creating a new one.
+* Fix filtering logic for prefixed blobs
+* Fix permissions for data-loading-blob-prefix-allowlist
+* Make GCP nat optional.
+* Parse delta filename from notification before validating it
+* Remove glog dependency for record_utils
+* Remove temp dir only if it's successfully created.
+* Rename class to ThreadManager
+* Set retain_initial_value_of_delta_metric flag for aws metrics exporter
+* Update a outdated hyperlink.
+* Update common repo to pick up the AWS metrics dimension fix
+* Update GCP Terraform with ability to delete unhealthy instance.
+* Update tf variables to use gorekore instead of kelvingorekore
+* Use blob key instead of prefixed basename
+
+
+### GCP: Fixes
+
+* **GCP:** Make sure server is connected to otel collector before reaching to ready state
+
+
+### GCP: Features
+
+* **GCP:** Applying Terraform pulls docker image with new tag.
+* **GCP:** Make service mesh address configurable.
+* **GCP:** Make subnet ip cidr configurable.
+* **GCP:** Make xlb/envoy optional.
+
+
+### Documentation
+
+* Add ad retrieval explainer.
+* Add docs for directory support
+* Add PA and PAS folders
+* Add PAS developer guide
+* Add public docs for file groups
+* Ads retreival explainer update.
+
+
+### Dependencies
+
+* **deps:** Add clang-tidy bazel config
+* **deps:** Add cpp_nowarn bazel config
+* **deps:** Upgrade bazel to 6.5.0
+* **deps:** Upgrade build-system to 0.55.1
+* **deps:** Upgrade build-system to 0.55.2
+* **deps:** Upgrade build-system to 0.57.0
+* **deps:** Upgrade data-plane-shared repo
+* **deps:** Upgrade data-plane-shared repo to 1684674 2024-02-09
+* **deps:** Upgrade data-plane-shared-libraries to 1fbac46
+* **deps:** Upgrade pre-commit hooks
+
 ## 0.15.0 (2024-01-23)
 
 

README.md

@@ -38,7 +38,7 @@ versions are released. The query API conforms to the
 [API explainer](https://github.com/WICG/turtledove/blob/main/FLEDGE_Key_Value_Server_API.md). At the
 moment, to load data, instead of calling the mutation API, you would place the data as files into a
 location that can be directly read by the server. See more details in the
-[data loading guide](/docs/loading_data.md).
+[data loading guide](/docs/data_loading/loading_data.md).
 
 Currently, this service can be deployed to 1 region of your choice with more regions to be added
 soon. Monitoring and alerts are currently unavailable.
@@ -72,18 +72,18 @@ changes.
 -   [FLEDGE K/V server API explainer](https://github.com/WICG/turtledove/blob/main/FLEDGE_Key_Value_Server_API.md)
 -   [FLEDGE K/V server trust model](https://github.com/privacysandbox/fledge-docs/blob/main/key_value_service_trust_model.md)
 -   [Local server quickstart guide](/docs/developing_the_server.md)
--   [AWS server user deployment documentation](/docs/deploying_on_aws.md)
--   [GCP server user deployment documentation](/docs/deploying_on_gcp.md)
--   [Integrating the K/V server with FLEDGE](/docs/integrating_with_fledge.md)
--   [FLEDGE K/V server sharding explainer](https://github.com/privacysandbox/fledge-docs/blob/main/key_value_sharding.md)
+-   [AWS server user deployment documentation](/docs/deployment/deploying_on_aws.md)
+-   [GCP server user deployment documentation](/docs/deployment/deploying_on_gcp.md)
+-   [Integrating the K/V server with FLEDGE](/docs/protected_audience/integrating_with_fledge.md)
+-   [FLEDGE K/V server sharding explainer](https://github.com/privacysandbox/protected-auction-services-docs/blob/main/key_value_service_sharding.md)
 -   Operating documentation
-    -   [Data loading API and operations](/docs/loading_data.md)
+    -   [Data loading API and operations](/docs/data_loading/loading_data.md)
     -   [Generating and loading UDF files](/docs/generating_udf_files.md)
     -   Error handling explainer (_to be published_)
 -   Developer guide
     -   [Codebase structure](/docs/repo_layout.md)
-    -   [Working with Terraform](/production/terraform/README.md)
-    -   [Contributing to the codebase](/docs/CONTRIBUTING.md)
+    -   [Working with Terraform](/docs/deployment/working_with_terraform.md)
+    -   [Contributing to the codebase](/docs/contributing.md)
 -   [Code of conduct](/docs/CODE_OF_CONDUCT.md)
 -   [Change log](/CHANGELOG.md)