AIP-157 Partial response implementation #717

sayan-biswas · 2024-02-21T13:03:14Z

In the present implementation, a user cannot filter the data in the response message, a feature provided by modern API implementations. This is particularly needed for the List APIs as these APIs return the whole object in the list. In scenario where the list and object definition are big, there's a lot network bandwidth wasted because the response of the List API is mostly used to display a list of resources and the Get API is then used to fetch the individual resource.
This feature provides an easy and basic way to filter the response message across all APIs, which reduces the size of the payload and hence allows faster transfer over network.

Changes

This feature add capabilities to filter the response message from across all APIs.
AIP detail:
https://google.aip.dev/157

Submitter Checklist

These are the criteria that every PR should meet, please check them off as you review them:

Has Docs included if any changes are user facing
Has Tests included if any functionality added or changed
Tested your changes locally (if this is a code change)
Follows the commit message standard
Meets the Tekton contributor standards (including functionality, content, code)
Has a kind label. You can add a comment on this PR that contains /kind <type>. Valid types are bug, cleanup, design, documentation, feature, flake, misc, question, tep
Release notes block below has been updated with any user-facing changes (API changes, bug fixes, changes requiring upgrade notices or deprecation warnings)
Release notes contain the string "action required" if the change requires additional action from users switching to the new release

Release Notes

API: Request message allows an additional parameter `fields` to filter the response message.

tekton-robot · 2024-02-21T13:03:17Z

Skipping CI for Draft Pull Request.
If you want CI signal for your change, please convert it to an actual PR.
You can still manually trigger a test run with /test all

tekton-robot · 2024-02-21T13:03:18Z

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by:
To complete the pull request process, please ask for approval from sayan-biswas after the PR has been reviewed.

The full list of commands accepted by this bot can be found here.

Needs approval from an approver in each of these files:

OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

sayan-biswas · 2024-02-28T20:18:37Z

/test all

tekton-robot · 2024-02-28T20:22:55Z

The following is the coverage report on the affected files.
Say /test pull-tekton-results-go-coverage to re-run this coverage report

File	Old Coverage	New Coverage	Delta
internal/fieldmask/fieldmask.go	Do not exist	70.1%

sayan-biswas · 2024-02-28T20:45:47Z

/test all

tekton-robot · 2024-02-28T20:49:52Z

The following is the coverage report on the affected files.
Say /test pull-tekton-results-go-coverage to re-run this coverage report

File	Old Coverage	New Coverage	Delta
internal/fieldmask/fieldmask.go	Do not exist	70.1%

sayan-biswas · 2024-02-29T12:14:24Z

/test pull-tekton-results-integration-tests

sayan-biswas · 2024-03-02T07:40:30Z

/test pull-tekton-results-integration-tests

gabemontero

finally got around to taking a pass at this @sayan-biswas

of course, being brand new / a novice to all this, consider that perspective when reviewing my feedback ;-)

docs/api/README.md

gabemontero · 2024-04-12T14:18:48Z

internal/fieldmask/fieldmask.go

+	mask := make(FieldMask)
+	for _, path := range paths {
+		current := mask
+		fields := strings.Split(path, ".")


put a comment above this split with some sample paths to illustrate why we are splitting on .

let's not make the coder have to do up to your README change to undersatand what is going on

Added reference and an example in godoc format.

awesome thanks

internal/fieldmask/fieldmask.go

internal/fieldmask/fieldmask_test.go

internal/fieldmask/fieldmask.go

gabemontero · 2024-04-12T15:22:36Z

internal/fieldmask/fieldmask.go

+					reflect.Set(fd, protoreflect.ValueOfBytes(b))
+				}
+			}
+		}


feels like we should have a default case to at least log unexpected messages, no?

I would say no, because this is simply filtering. Any unknown cases will just be ignored.
And for logging, there's are some restrictions here. We cannot throw error because then the client will receive the error. We cannot bring logger from the context, then the implementation will have dependency and can't be used as a library.
Although we can still log with normal golang's fmt or logger library, but that will break structured logging.

valid reasoning

I still wonder about being able to debug in the future "why did my filter not catch 'X'"

could we emit events for the errors in this switch?

could we emit events for the default case? Or are there too many types besides MessageKind and BytesKind?

Emit the events where? You mean kubernetes events?

yes k8s events .... api server has a k8s client so in theory could emit events to help with debug, and you don't have the logging pain you noted earlier

gabemontero · 2024-04-12T15:26:25Z

cmd/api/main.go

@@ -169,6 +171,7 @@ func main() {
 			grpc_zap.UnaryServerInterceptor(grpcLogger, zapOpts...),
 			grpc_auth.UnaryServerInterceptor(determineAuth),
 			prometheus.UnaryServerInterceptor,
+			fieldmask.UnaryServerInterceptor(),


shouldn't we have a feature enablement check of some sort ? presumably a client could send the extra headers or encoded data and it is OK to ignore ?

I have added a feature flag implementation to handle all features control.

feature gate looks good @sayan-biswas good think .... in looking at it though I raised some questions around changes you made to my recently added tuning options

I'll submit a separate review for those

tekton-robot · 2024-04-20T09:46:00Z

The following is the coverage report on the affected files.
Say /test pull-tekton-results-go-coverage to re-run this coverage report

File	Old Coverage	New Coverage	Delta
internal/fieldmask/fieldmask.go	Do not exist	70.1%

tekton-robot · 2024-05-02T22:46:03Z

The following is the coverage report on the affected files.
Say /test pull-tekton-results-go-coverage to re-run this coverage report

File	Old Coverage	New Coverage	Delta
cmd/api/main.go	9.3%	9.1%	-0.2
internal/fieldmask/fieldmask.go	Do not exist	70.1%
pkg/api/server/config/config.go	Do not exist	0.0%
pkg/api/server/config/config.go	Do not exist	0.0%
pkg/api/server/features/features.go	Do not exist	85.3%

This feature add capabilities to filter the response message from all the APIs. AIP detail: https://google.aip.dev/157

tekton-robot · 2024-05-02T23:02:54Z

The following is the coverage report on the affected files.
Say /test pull-tekton-results-go-coverage to re-run this coverage report

File	Old Coverage	New Coverage	Delta
cmd/api/main.go	9.3%	9.1%	-0.2
internal/fieldmask/fieldmask.go	Do not exist	70.1%
pkg/api/server/config/config.go	Do not exist	0.0%
pkg/api/server/config/config.go	Do not exist	0.0%
pkg/api/server/features/features.go	Do not exist	85.3%

gabemontero

just a doc bit I think @sayan-biswas

gabemontero · 2024-05-03T12:33:23Z

config/base/env/config

@@ -6,13 +6,7 @@ DB_NAME=tekton-results
 DB_SSLMODE=disable
 DB_SSLROOTCERT=
 DB_ENABLE_AUTO_MIGRATION=true
-DB_MAX_IDLE_CONNECTIONS=10


I'm assuming you did this @sayan-biswas because I didn't need to set these defaults when I added the tuning options ? If so, thanks for the correction.

Or was this accidental? ...If it was, in hindsight, I does probably make sense to not set these, even to the defaults. (though 10 does not line up with the default as described in the doc.

Users can always add these to their config map.

gabemontero · 2024-05-03T12:35:58Z

cmd/api/README.md

-| DB_ENABLE_AUTO_MIGRATION | Auto-migrate the database on startup (create/update schemas). For further details, refer to <https://gorm.io/docs/migration.html> | true (default)                                                                               |
-| PROFILING                | Enable profiling server                                                                                                           | false  (default)                                                                             |
-| PROFILING_PORT           | Profiling Server Port                                                                                                             | 6060  (default)                                                                              |
-| DB_MAX_IDLE_CONNECTIONS  | The number of idle database connections to keep open                                                                              | 2 (default for golang, but specific database drivers may have settings for this too)         |


You deleted all my tuning options here in the README @sayan-biswas .... we need to minially document them, even if they are not specified in the config/base/env/config file

Ohh.. Not sure why how this was missed. I'll rebase again to include the changes.

gabemontero · 2024-05-03T12:38:09Z

cmd/api/main.go

@@ -169,6 +171,7 @@ func main() {
 			grpc_zap.UnaryServerInterceptor(grpcLogger, zapOpts...),
 			grpc_auth.UnaryServerInterceptor(determineAuth),
 			prometheus.UnaryServerInterceptor,
+			fieldmask.UnaryServerInterceptor(),


feature gate looks good @sayan-biswas good think .... in looking at it though I raised some questions around changes you made to my recently added tuning options

I'll submit a separate review for those

gabemontero · 2024-05-03T12:43:34Z

I don't seem to have permission to resolve comment threads @sayan-biswas

Feel free to resolve all of mine except #717 (review) and the one about submitting k8s events (at least until you respond to my clarification there).

gabemontero · 2024-05-08T12:12:23Z

bump @sayan-biswas on my last round of comments - #717 (review)

khrm

@sayan-biswas Can you rebase it again without removing Gabe's performance tuning changes?

tekton-robot · 2024-07-15T19:39:40Z

@sayan-biswas: PR needs rebase.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

sayan-biswas · 2024-07-18T23:32:54Z

@sayan-biswas Can you rebase it again without removing Gabe's performance tuning changes?

Sorry! I was caught up with other activities. I will rebase soon.

sayan-biswas added the kind/feature Categorizes issue or PR as related to a new feature. label Feb 21, 2024

sayan-biswas self-assigned this Feb 21, 2024

tekton-robot added release-note Denotes a PR that will be considered when it comes time to generate release notes. do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. labels Feb 21, 2024

tekton-robot requested review from avinal and enarha February 21, 2024 13:03

tekton-robot added the size/XL Denotes a PR that changes 500-999 lines, ignoring generated files. label Feb 21, 2024

sayan-biswas force-pushed the partial-response branch from f603652 to 50dd2b9 Compare February 28, 2024 18:18

tekton-robot added size/XXL Denotes a PR that changes 1000+ lines, ignoring generated files. and removed size/XL Denotes a PR that changes 500-999 lines, ignoring generated files. labels Feb 28, 2024

sayan-biswas force-pushed the partial-response branch 2 times, most recently from 32027d4 to 08b524b Compare February 28, 2024 19:57

sayan-biswas requested review from adambkaplan and khrm February 28, 2024 20:17

sayan-biswas force-pushed the partial-response branch from 08b524b to 4ea8c19 Compare February 28, 2024 20:44

sayan-biswas marked this pull request as ready for review February 28, 2024 20:45

tekton-robot removed the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label Feb 28, 2024

gabemontero reviewed Apr 12, 2024

View reviewed changes

sayan-biswas force-pushed the partial-response branch 2 times, most recently from 0acbbac to 840e4cc Compare May 2, 2024 22:40

AIP-157 Partial response implementation

6b8e353

This feature add capabilities to filter the response message from all the APIs. AIP detail: https://google.aip.dev/157

sayan-biswas force-pushed the partial-response branch from 840e4cc to 6b8e353 Compare May 2, 2024 22:57

gabemontero suggested changes May 3, 2024

View reviewed changes

khrm reviewed Jul 4, 2024

View reviewed changes

tekton-robot added the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Jul 15, 2024

AIP-157 Partial response implementation #717

Are you sure you want to change the base?

AIP-157 Partial response implementation #717

Conversation

sayan-biswas commented Feb 21, 2024 • edited Loading

Changes

Submitter Checklist

Release Notes

tekton-robot commented Feb 21, 2024

tekton-robot commented Feb 21, 2024

sayan-biswas commented Feb 28, 2024

tekton-robot commented Feb 28, 2024

sayan-biswas commented Feb 28, 2024

tekton-robot commented Feb 28, 2024

sayan-biswas commented Feb 29, 2024

sayan-biswas commented Mar 2, 2024

gabemontero left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

tekton-robot commented Apr 20, 2024

tekton-robot commented May 2, 2024

tekton-robot commented May 2, 2024

gabemontero left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

gabemontero commented May 3, 2024 • edited Loading

gabemontero commented May 8, 2024

khrm left a comment

Choose a reason for hiding this comment

tekton-robot commented Jul 15, 2024

sayan-biswas commented Jul 18, 2024

sayan-biswas commented Feb 21, 2024 •

edited

Loading

gabemontero commented May 3, 2024 •

edited

Loading