Added a prototype for generating jsonschema files #112

kommendorkapten · 2023-07-28T08:18:12Z

WIP to generate JSON schema files, as coined by @woodruffw (see Slack: https://sigstore.slack.com/archives/C03SZ6SHU90/p1690490585606889)

Closes #67

Consider this PR WIP so far.

There are some general topics I would like to bring up for discussion too related to the current Docker image we are using for building the protoc compiler that I discovered when working on this. I'll create a separate issue for that.

Release Note

Added support for generating JSON schema files

Documentation

N/A

Signed-off-by: Fredrik Skogman <kommendorkapten@github.com>

kommendorkapten · 2023-07-28T14:57:40Z

The reason I added a new Docker image is that namely/protoc-all is too old to build newer Go programs, see issue #113

woodruffw

This looks pretty good! I did a quick pass over the generated schemas and nothing looks way off 🙂

I've asked @jleightcap to take a look at using these to generate the Rust crate, once he's finished with some conformance suite work.

Dockerfile.jsonschema

Makefile

kommendorkapten · 2023-07-31T09:22:21Z

Yeah, we should pin the versions and make sure we perform this build for the all target. I’ll be away for vacation now for two weeks, but can take a stab when I’m back, and looking into creating a single new image that covers all languages so we can drop the namely image. If any of you are in a hurry to get the JSON schema in now, feel free to modify this PR. Maintainers will have permission to do so.

…

On Fri, 28 Jul 2023 at 22:42, Cody Soyland ***@***.***> wrote: ***@***.**** commented on this pull request. ------------------------------ On Makefile <#112 (comment)> : Should we add jsonschema to the list of targets for make all <https://github.com/sigstore/protobuf-specs/blob/main/Makefile#L19>? This way, it will become part of the CI check <https://github.com/sigstore/protobuf-specs/blob/main/.github/workflows/generate.yml> to make sure generated files are always updated. — Reply to this email directly, view it on GitHub <#112 (review)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ACSDVRCYWYDJZT4EM2ZYDODXSQPZXANCNFSM6AAAAAA23FGSP4> . You are receiving this because you authored the thread.Message ID: ***@***.***>

woodruffw · 2023-07-31T13:51:31Z

If any of you are in a hurry to get the JSON schema in now, feel free to modify this PR. Maintainers will have permission to do so.

I'm going to have @jleightcap work off of the schema in this PR, so I'll take over! Enjoy your vacation!

Signed-off-by: William Woodruff <william@trailofbits.com>

woodruffw · 2023-07-31T14:15:15Z

Dockerfile.jsonschema

+# This is required to get the field_behavior.proto file
+# NOTE: --filter=tree:0 performs a treeless clone; we do this to optimize cloning
+# this otherwise relatively heavy repository.
+RUN git clone --filter=tree:0 https://github.com/googleapis/googleapis.git


NB: This repository doesn't appear to be versioned in any way, and isn't packaged in alpine. I could check out a fixed revision if we're like more consistency here.

I'd like to pin it to a fixed revision.

Hmm, this is moderately annoying: we can't do a fast-path clone with a specific revision. I'll see about just downloading it directly as a ZIP or similar and unarchiving.

Did it with a checkout after all; the performance shouldn't be too bad.

Signed-off-by: William Woodruff <william@trailofbits.com>

woodruffw · 2023-07-31T15:29:55Z

Dockerfile.jsonschema

@@ -0,0 +1,8 @@
+# 3.18.2
+FROM alpine@sha256:25fad2a32ad1f6f510e528448ae1ec69a28ef81916a004d3629874104f8a7f70
+RUN apk add --update protoc=3.21.12-r2 protobuf-dev=3.21.12-r2 go=1.20.5-r0 git=2.40.1-r0


NB: I picked these version numbers not because they're the most recent, but because they're consistent with Alpine's world version for the Alpine version chosen. More recent versions could probably be made to work, but I figured this was the path of least resistance.

znewman01

This LGTM, except I would like the googleapis/googleapis repo pinned too :)

znewman01 · 2023-07-31T19:33:34Z

Dockerfile.jsonschema

+# This is required to get the field_behavior.proto file
+# NOTE: --filter=tree:0 performs a treeless clone; we do this to optimize cloning
+# this otherwise relatively heavy repository.
+RUN git clone --filter=tree:0 https://github.com/googleapis/googleapis.git


I'd like to pin it to a fixed revision.

Signed-off-by: William Woodruff <william@trailofbits.com>

haydentherapper · 2023-07-31T20:02:09Z

Dockerfile.jsonschema

+# 3.18.2
+FROM alpine@sha256:25fad2a32ad1f6f510e528448ae1ec69a28ef81916a004d3629874104f8a7f70
+RUN apk add --update protoc=3.21.12-r2 protobuf-dev=3.21.12-r2 go=1.20.5-r0 git=2.40.1-r0
+RUN go install github.com/chrusty/protoc-gen-jsonschema/cmd/protoc-gen-jsonschema@1.4.1


Do you know if dependabot will suggest updates for Go dependencies in Dockerfiles? Otherwise we might want to add a hack like https://github.com/sigstore/rekor/blob/main/hack/tools/go.mod

haydentherapper · 2023-07-31T20:04:38Z

Should the jsonschema be versioned and have releases cut along with the other languages? Otherwise, it'll always be from HEAD. We could have folders for each version, /schema/v0.2/? Not sure if there's precedent for this.

woodruffw · 2023-07-31T20:07:25Z

Should the jsonschema be versioned and have releases cut along with the other languages? Otherwise, it'll always be from HEAD. We could have folders for each version, /schema/v0.2/? Not sure if there's precedent for this.

Hmm, good call -- we want to use this as the building block for the Rust bindings, so we should probably version it. IMO that can be done either by tagging (like the other bindings) or by having directories for either version; I don't think it'll make a major difference to the Rust bindings either way.

haydentherapper · 2023-07-31T20:09:13Z

Is it sufficient to use an existing tag like https://github.com/sigstore/protobuf-specs/releases/tag/v0.2.0? Otherwise we could have its own tag, releases/jsonschema/vXYZ, but I can't think of a case where we'll have the json schema version deviate.

znewman01 · 2023-07-31T20:10:49Z

Is it sufficient to use an existing tag like https://github.com/sigstore/protobuf-specs/releases/tag/v0.2.0? Otherwise we could have its own tag, releases/jsonschema/vXYZ, but I can't think of a case where we'll have the json schema version deviate.

In the past, we liked versioning each set of bindings independently to handle changes to the generation process. E.g. if there was a bug in the protoc-jsonschema generator. I think we should default to that if possible.

haydentherapper · 2023-07-31T20:12:26Z

SG, so we can just add a step to the release instructions to add a tag for release/json/vX.Y.Z, and we shouldn't need another release workflow.

woodruffw · 2023-07-31T20:13:32Z

SGTM! I'll add a note to the release instructions.

Signed-off-by: William Woodruff <william@trailofbits.com>

Added a prototype for generating jsonschema files

e3f3dd8

Signed-off-by: Fredrik Skogman <kommendorkapten@github.com>

kommendorkapten requested review from znewman01, woodruffw and haydentherapper July 28, 2023 08:18

Removed unused changes to Dockerfile

c2e04f5

Signed-off-by: Fredrik Skogman <kommendorkapten@github.com>

kommendorkapten mentioned this pull request Jul 28, 2023

Drop namely/protoc-all in favour of a custom Dockerfile? #113

Open

woodruffw previously approved these changes Jul 28, 2023

View reviewed changes

haydentherapper reviewed Jul 28, 2023

View reviewed changes

Dockerfile.jsonschema Outdated Show resolved Hide resolved

codysoyland reviewed Jul 28, 2023

View reviewed changes

Makefile Outdated Show resolved Hide resolved

woodruffw added 3 commits July 31, 2023 10:09

Dockerfile.jsonschema: pin apk versions

1a9f032

Signed-off-by: William Woodruff <william@trailofbits.com>

Makefile: add jsonschema to all

4f5b42d

Signed-off-by: William Woodruff <william@trailofbits.com>

gen/jsonschema: regen

2025521

Signed-off-by: William Woodruff <william@trailofbits.com>

woodruffw dismissed their stale review via 2025521 July 31, 2023 14:10

Dockerfile.jsonschema: optimize git clone slightly

240f520

Signed-off-by: William Woodruff <william@trailofbits.com>

woodruffw reviewed Jul 31, 2023

View reviewed changes

woodruffw requested review from codysoyland and haydentherapper July 31, 2023 14:15

woodruffw added 2 commits July 31, 2023 10:22

Merge remote-tracking branch 'upstream/main' into jsonschema-wip

8f605d6

jsonschema: regen

fce50a6

Signed-off-by: William Woodruff <william@trailofbits.com>

woodruffw reviewed Jul 31, 2023

View reviewed changes

znewman01 previously approved these changes Jul 31, 2023

View reviewed changes

Dockerfile.jsonschema: check out a fixed revision

56f3f97

Signed-off-by: William Woodruff <william@trailofbits.com>

woodruffw dismissed znewman01’s stale review via 56f3f97 July 31, 2023 20:00

haydentherapper reviewed Jul 31, 2023

View reviewed changes

znewman01 previously approved these changes Jul 31, 2023

View reviewed changes

RELEASE: document JSON schema process

8feb7bb

Signed-off-by: William Woodruff <william@trailofbits.com>

woodruffw dismissed znewman01’s stale review via 8feb7bb July 31, 2023 20:14

woodruffw requested review from woodruffw, haydentherapper and znewman01 July 31, 2023 20:14

haydentherapper approved these changes Jul 31, 2023

View reviewed changes

woodruffw merged commit 9abc8c2 into sigstore:main Jul 31, 2023
3 checks passed

kommendorkapten deleted the jsonschema-wip branch August 14, 2023 06:14

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Added a prototype for generating jsonschema files #112

Added a prototype for generating jsonschema files #112

kommendorkapten commented Jul 28, 2023

kommendorkapten commented Jul 28, 2023

woodruffw left a comment

kommendorkapten commented Jul 31, 2023 via email

woodruffw commented Jul 31, 2023

woodruffw Jul 31, 2023

znewman01 Jul 31, 2023

woodruffw Jul 31, 2023

woodruffw Jul 31, 2023

woodruffw Jul 31, 2023

znewman01 left a comment

znewman01 Jul 31, 2023

haydentherapper Jul 31, 2023

haydentherapper commented Jul 31, 2023

woodruffw commented Jul 31, 2023

haydentherapper commented Jul 31, 2023

znewman01 commented Jul 31, 2023

haydentherapper commented Jul 31, 2023

woodruffw commented Jul 31, 2023

Added a prototype for generating jsonschema files #112

Added a prototype for generating jsonschema files #112

Conversation

kommendorkapten commented Jul 28, 2023

Release Note

Documentation

kommendorkapten commented Jul 28, 2023

woodruffw left a comment

Choose a reason for hiding this comment

kommendorkapten commented Jul 31, 2023 via email

woodruffw commented Jul 31, 2023

woodruffw Jul 31, 2023

Choose a reason for hiding this comment

znewman01 Jul 31, 2023

Choose a reason for hiding this comment

woodruffw Jul 31, 2023

Choose a reason for hiding this comment

woodruffw Jul 31, 2023

Choose a reason for hiding this comment

woodruffw Jul 31, 2023

Choose a reason for hiding this comment

znewman01 left a comment

Choose a reason for hiding this comment

znewman01 Jul 31, 2023

Choose a reason for hiding this comment

haydentherapper Jul 31, 2023

Choose a reason for hiding this comment

haydentherapper commented Jul 31, 2023

woodruffw commented Jul 31, 2023

haydentherapper commented Jul 31, 2023

znewman01 commented Jul 31, 2023

haydentherapper commented Jul 31, 2023

woodruffw commented Jul 31, 2023