feat: Include dockerized protoc (#404)

* Include dockerized protoc

* Allow relative paths

* Allow usage of dockerized protoc in any directory

* Remove protoc.zip after installing in docker

* Small spike on some 'setup:...' commands.

* Set default value for PWD to project root directory

* Make update-bins Windows compatible, and allow running from any directory

* Fix issue where bin files were directly generated for dependencies of integration test

* Avoid find warning about argument order

* Support specifying multiple integration tests in codegen.sh

* Support running codegen.sh from any directory

* Define local alternatives for docker commands

* Update readme with development and contribution workflow

* Upgrade to protoc 3.19.1

* Add dockerized protoc documentation

* Recompile ts without docker

* Recompile proto files

* Recompile proto files

* Rename bin2pbjs to proto2pbjs.

* Fix file.bin case.

* Update workflow to use bin2ts.

* Run bin2ts output.

Co-authored-by: Stephen Haberman <stephen.haberman@gmail.com>

.github/workflows/build.yml

@@ -28,10 +28,9 @@ jobs:
  run: ./integration/pbjs.sh
  # This will fail if any git-tracked file has changed
  - name: Codegen
- run: |
- cd integration
- ./codegen.sh
- git status && git diff --exit-code
+ run: yarn bin2ts
+ - name: Diff Output
+ run: git status && git diff --exit-code
  - name: test
  run: yarn test
  env:

README.markdown

@@ -18,7 +18,7 @@
  - [Supported options](#supported-options)
  - [Only Types](#only-types)
  - [NestJS Support](NESTJS.markdown)
-- [Building](#building)
+- [Development](#development)
 - [Assumptions](#assumptions)
 - [Todo](#todo)
 - [OneOf Handling](#oneof-handling)
@@ -362,15 +362,69 @@ Kudos to our sponsors:
 
 If you need ts-proto customizations or priority support for your company, you can ping me at [via email](mailto:stephen.haberman@gmail.com).
 
-# Building
+# Development
 
-After running `yarn install`, run `./integration/pbjs.sh` to create the integration test types. These pbjs-generated files are not currently checked in.
+**Requirements**
 
-After this, the tests should pass.
+- [Docker](https://www.docker.com) or [protoc](https://github.com/protocolbuffers/protobuf/releases) v3.19.1
+- `yarn` — `npm install -g yarn`
 
-After making changes to `ts-proto`, you can run `cd integration` and `./codegen.sh` to re-generate the test case `*.ts` output files that are in each `integration/<test-case>/` directory.
+**Setup**
 
-The test suite's proto files (i.e. `simple.proto`, `batching.proto`, etc.) currently have serialized/`.bin` copies checked into git (i.e. `simple.bin`, `batching.bin`, etc.), so that the test suite can run without having to invoke the `protoc` build chain. I.e. if you change the `simple.proto`/etc. files, you'll need to run `./integration/update-bins.sh`, which does require having the `protoc` executable available.
+The commands below assume you have **Docker** installed. To use a **local** copy of `protoc` without docker, use commands suffixed with `:local`
+
+- Check out the [repository]() for the latest code.
+- Run `yarn install` to install the dependencies.
+- Run `yarn build:test` or `yarn build:test:local` to generate the test files.
+ > _This runs the following commands:_
+ > - `proto2bin` — Converts integration test `.proto` files to `.bin`.
+ > - `bin2ts` — Runs `ts-proto` on the `.bin` files to generate `.ts` files.
+ > - `proto2pbjs` — Generates a reference implementation using `pbjs` for testing compatibility.
+- Run `yarn test`
+
+**Workflow**
+
+- Modifying the plugin implementation:
+ - Run `yarn bin2ts` or `yarn bin2ts:local`. 
+ _Since the proto files were not changed, you only need to regenerate the typescript files._
+ - Run `yarn test` to verify the typescript files are compatible with the reference implementation, and pass other tests.
+- Updating or adding `.proto` files in the integration directory:
+ - Run `yarn build:test` to regenerate the integration test files.
+ - Run `yarn test` to retest.
+
+**Contributing**
+
+- Run `yarn build:test` and `yarn test` to make sure everything works.
+- Run `yarn prettier` to format the typescript files.
+- Commit the changes:
+ - Also include the generated `.bin` files for the tests where you added or modified `.proto` files. 
+ > These are checked into git so that the test suite can run without having to invoke the `protoc` build chain.
+ - Also include the generated `.ts` files.
+- Create a pull request
+
+**Dockerized Protoc**
+
+The repository includes a dockerized version of `protoc`, which is configured in [docker-compose.yml](docker-compose.yml).
+
+It can be useful in case you want to manually invoke the plugin with a known version of `protoc`.
+
+Usage:
+
+```bash
+# Include the protoc alias in your shell.
+. aliases.sh
+
+# Run protoc as usual. The ts-proto directory is available in /ts-proto.
+protoc --plugin=/ts-proto/protoc-gen-ts_proto --ts_proto_out=./output -I=./protos ./protoc/*.proto
+
+# Or use the ts-protoc alias which specifies the plugin path for you.
+ts-protoc --ts_proto_out=./output -I=./protos ./protoc/*.proto
+```
+
+- All paths must be relative paths _within_ the current working directory of the host. `../` is not allowed
+- Within the docker container, the absolute path to the project root is `/ts-proto`
+- The container mounts the current working directory in `/host`, and sets it as its working directory.
+- Once `aliases.sh` is sourced, you can use the `protoc` command in any folder.
 
 # Assumptions
 

aliases.sh

@@ -0,0 +1,11 @@
+#!/usr/bin/env bash
+PROJECT_ROOT=$(realpath $(dirname "$BASH_SOURCE"))
+PROJECT_ROOT_DOCKER="//ts-proto" # double slash to support git bash on windows
+
+# Alias docker-compose to make it usable from anywhere
+function _docker-compose() { docker-compose -f $PROJECT_ROOT/docker-compose.yml "$@"; }
+
+function protoc() { _docker-compose run --rm protoc "$@"; }
+function protoc-sh() { _docker-compose run --rm --entrypoint sh -- protoc "$@"; }
+function protoc-build() { _docker-compose build protoc; }
+function ts-protoc { protoc --plugin=$PROJECT_ROOT_DOCKER/protoc-gen-ts_proto "$@"; }

docker-compose.yml

@@ -0,0 +1,14 @@
+version: "3"
+services:
+ protoc:
+ build:
+ context: .
+ dockerfile: "protoc.Dockerfile"
+ args:
+ BASE: "node:17-alpine3.14"
+ BIN: "/ts-proto"
+ PROTOC_VERSION: "3.19.1"
+ working_dir: "/host"
+ volumes:
+ - "${PWD:-.}:/host"
+ - ".:/ts-proto"

integration/codegen.sh

@@ -9,23 +9,26 @@
 #
 # Updates generated output for all integration tests.
 #
-# ./codegen.sh simple
+# ./codegen.sh simple value
 #
-# Updates generated output only for the 'simple' integration test.
+# Updates generated output only for the 'simple' and 'value' integration test.
 #
 # Each integration test can optionally have a `parameters.txt` file that will
 # be used as the ts-proto_opt... args for generating that test's code.
 
-dir=${1:-*}
+INTEGRATION_DIR=$(realpath $(dirname "$BASH_SOURCE"))
 
-N=6
+# Run the code generator in parallel, with one process per core.
+N=$(nproc)
+
+echo "Generating typescript code for integration tests using ${N} cores..."
 
 dir=.
 if [ -n "${1}" ]; then
- dir=$1
+ dir="${@}"
 fi
 
-list=$(find "$dir" -name "*.bin" -type f | grep -v dump-response.bin)
+list=$(find $dir -name "*.bin" -type f | grep -v dump-response.bin)
 
 for file in $list; do
  echo "${file}"
@@ -39,7 +42,7 @@ for file in $list; do
  fi
 
  ((i=i%N)); ((i++==0)) && wait
- ../node_modules/.bin/ts-node ./codegen.ts "${dir}" "${file}" "${params}" &
+ "${INTEGRATION_DIR}/../node_modules/.bin/ts-node" "${INTEGRATION_DIR}/codegen.ts" "${dir}" "${file}" "${params}" &
 done
 
 wait

integration/grpc-js/google/protobuf/struct.ts

@@ -60,8 +60,8 @@ export interface Struct_FieldsEntry {
 /**
  * `Value` represents a dynamically typed value which can be either
  * null, a number, a string, a boolean, a recursive struct value, or a
- * list of values. A producer of value is expected to set one of that
- * variants, absence of any variant indicates an error.
+ * list of values. A producer of value is expected to set one of these
+ * variants. Absence of any variant indicates an error.
  *
  * The JSON representation for `Value` is JSON value.
  */

integration/meta-typings/google/protobuf/timestamp.ts

@@ -57,7 +57,16 @@ export const protobufPackage = 'google.protobuf';
  * .setNanos((int) ((millis % 1000) * 1000000)).build();
  *
  *
- * Example 5: Compute Timestamp from current time in Python.
+ * Example 5: Compute Timestamp from Java `Instant.now()`.
+ *
+ * Instant now = Instant.now();
+ *
+ * Timestamp timestamp =
+ * Timestamp.newBuilder().setSeconds(now.getEpochSecond())
+ * .setNanos(now.getNano()).build();
+ *
+ *
+ * Example 6: Compute Timestamp from current time in Python.
  *
  * timestamp = Timestamp()
  * timestamp.GetCurrentTime()
@@ -177,7 +186,7 @@ export const protoMetadata: ProtoMetadata = {
  javaPackage: 'com.google.protobuf',
  javaOuterClassname: 'TimestampProto',
  javaMultipleFiles: true,
- goPackage: 'github.com/golang/protobuf/ptypes/timestamp',
+ goPackage: 'google.golang.org/protobuf/types/known/timestamppb',
  ccEnableArenas: true,
  objcClassPrefix: 'GPB',
  csharpNamespace: 'Google.Protobuf.WellKnownTypes',
@@ -186,21 +195,21 @@ export const protoMetadata: ProtoMetadata = {
  location: [
  {
  path: [4, 0],
- span: [126, 0, 137, 1],
+ span: [135, 0, 146, 1],
  leadingDetachedComments: [],
  leadingComments:
- ' A Timestamp represents a point in time independent of any time zone or local\n calendar, encoded as a count of seconds and fractions of seconds at\n nanosecond resolution. The count is relative to an epoch at UTC midnight on\n January 1, 1970, in the proleptic Gregorian calendar which extends the\n Gregorian calendar backwards to year one.\n\n All minutes are 60 seconds long. Leap seconds are "smeared" so that no leap\n second table is needed for interpretation, using a [24-hour linear\n smear](https://developers.google.com/time/smear).\n\n The range is from 0001-01-01T00:00:00Z to 9999-12-31T23:59:59.999999999Z. By\n restricting to that range, we ensure that we can convert to and from [RFC\n 3339](https://www.ietf.org/rfc/rfc3339.txt) date strings.\n\n # Examples\n\n Example 1: Compute Timestamp from POSIX `time()`.\n\n Timestamp timestamp;\n timestamp.set_seconds(time(NULL));\n timestamp.set_nanos(0);\n\n Example 2: Compute Timestamp from POSIX `gettimeofday()`.\n\n struct timeval tv;\n gettimeofday(&tv, NULL);\n\n Timestamp timestamp;\n timestamp.set_seconds(tv.tv_sec);\n timestamp.set_nanos(tv.tv_usec * 1000);\n\n Example 3: Compute Timestamp from Win32 `GetSystemTimeAsFileTime()`.\n\n FILETIME ft;\n GetSystemTimeAsFileTime(&ft);\n UINT64 ticks = (((UINT64)ft.dwHighDateTime) << 32) | ft.dwLowDateTime;\n\n // A Windows tick is 100 nanoseconds. Windows epoch 1601-01-01T00:00:00Z\n // is 11644473600 seconds before Unix epoch 1970-01-01T00:00:00Z.\n Timestamp timestamp;\n timestamp.set_seconds((INT64) ((ticks / 10000000) - 11644473600LL));\n timestamp.set_nanos((INT32) ((ticks % 10000000) * 100));\n\n Example 4: Compute Timestamp from Java `System.currentTimeMillis()`.\n\n long millis = System.currentTimeMillis();\n\n Timestamp timestamp = Timestamp.newBuilder().setSeconds(millis / 1000)\n .setNanos((int) ((millis % 1000) * 1000000)).build();\n\n\n Example 5: Compute Timestamp from current time in Python.\n\n timestamp = Timestamp()\n timestamp.GetCurrentTime()\n\n # JSON Mapping\n\n In JSON format, the Timestamp type is encoded as a string in the\n [RFC 3339](https://www.ietf.org/rfc/rfc3339.txt) format. That is, the\n format is "{year}-{month}-{day}T{hour}:{min}:{sec}[.{frac_sec}]Z"\n where {year} is always expressed using four digits while {month}, {day},\n {hour}, {min}, and {sec} are zero-padded to two digits each. The fractional\n seconds, which can go up to 9 digits (i.e. up to 1 nanosecond resolution),\n are optional. The "Z" suffix indicates the timezone ("UTC"); the timezone\n is required. A proto3 JSON serializer should always use UTC (as indicated by\n "Z") when printing the Timestamp type and a proto3 JSON parser should be\n able to accept both UTC and other timezones (as indicated by an offset).\n\n For example, "2017-01-15T01:30:15.01Z" encodes 15.01 seconds past\n 01:30 UTC on January 15, 2017.\n\n In JavaScript, one can convert a Date object to this format using the\n standard\n [toISOString()](https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/Date/toISOString)\n method. In Python, a standard `datetime.datetime` object can be converted\n to this format using\n [`strftime`](https://docs.python.org/2/library/time.html#time.strftime) with\n the time format spec \'%Y-%m-%dT%H:%M:%S.%fZ\'. Likewise, in Java, one can use\n the Joda Time\'s [`ISODateTimeFormat.dateTime()`](\n http://www.joda.org/joda-time/apidocs/org/joda/time/format/ISODateTimeFormat.html#dateTime%2D%2D\n ) to obtain a formatter capable of generating timestamps in this format.\n\n\n',
+ ' A Timestamp represents a point in time independent of any time zone or local\n calendar, encoded as a count of seconds and fractions of seconds at\n nanosecond resolution. The count is relative to an epoch at UTC midnight on\n January 1, 1970, in the proleptic Gregorian calendar which extends the\n Gregorian calendar backwards to year one.\n\n All minutes are 60 seconds long. Leap seconds are "smeared" so that no leap\n second table is needed for interpretation, using a [24-hour linear\n smear](https://developers.google.com/time/smear).\n\n The range is from 0001-01-01T00:00:00Z to 9999-12-31T23:59:59.999999999Z. By\n restricting to that range, we ensure that we can convert to and from [RFC\n 3339](https://www.ietf.org/rfc/rfc3339.txt) date strings.\n\n # Examples\n\n Example 1: Compute Timestamp from POSIX `time()`.\n\n Timestamp timestamp;\n timestamp.set_seconds(time(NULL));\n timestamp.set_nanos(0);\n\n Example 2: Compute Timestamp from POSIX `gettimeofday()`.\n\n struct timeval tv;\n gettimeofday(&tv, NULL);\n\n Timestamp timestamp;\n timestamp.set_seconds(tv.tv_sec);\n timestamp.set_nanos(tv.tv_usec * 1000);\n\n Example 3: Compute Timestamp from Win32 `GetSystemTimeAsFileTime()`.\n\n FILETIME ft;\n GetSystemTimeAsFileTime(&ft);\n UINT64 ticks = (((UINT64)ft.dwHighDateTime) << 32) | ft.dwLowDateTime;\n\n // A Windows tick is 100 nanoseconds. Windows epoch 1601-01-01T00:00:00Z\n // is 11644473600 seconds before Unix epoch 1970-01-01T00:00:00Z.\n Timestamp timestamp;\n timestamp.set_seconds((INT64) ((ticks / 10000000) - 11644473600LL));\n timestamp.set_nanos((INT32) ((ticks % 10000000) * 100));\n\n Example 4: Compute Timestamp from Java `System.currentTimeMillis()`.\n\n long millis = System.currentTimeMillis();\n\n Timestamp timestamp = Timestamp.newBuilder().setSeconds(millis / 1000)\n .setNanos((int) ((millis % 1000) * 1000000)).build();\n\n\n Example 5: Compute Timestamp from Java `Instant.now()`.\n\n Instant now = Instant.now();\n\n Timestamp timestamp =\n Timestamp.newBuilder().setSeconds(now.getEpochSecond())\n .setNanos(now.getNano()).build();\n\n\n Example 6: Compute Timestamp from current time in Python.\n\n timestamp = Timestamp()\n timestamp.GetCurrentTime()\n\n # JSON Mapping\n\n In JSON format, the Timestamp type is encoded as a string in the\n [RFC 3339](https://www.ietf.org/rfc/rfc3339.txt) format. That is, the\n format is "{year}-{month}-{day}T{hour}:{min}:{sec}[.{frac_sec}]Z"\n where {year} is always expressed using four digits while {month}, {day},\n {hour}, {min}, and {sec} are zero-padded to two digits each. The fractional\n seconds, which can go up to 9 digits (i.e. up to 1 nanosecond resolution),\n are optional. The "Z" suffix indicates the timezone ("UTC"); the timezone\n is required. A proto3 JSON serializer should always use UTC (as indicated by\n "Z") when printing the Timestamp type and a proto3 JSON parser should be\n able to accept both UTC and other timezones (as indicated by an offset).\n\n For example, "2017-01-15T01:30:15.01Z" encodes 15.01 seconds past\n 01:30 UTC on January 15, 2017.\n\n In JavaScript, one can convert a Date object to this format using the\n standard\n [toISOString()](https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/Date/toISOString)\n method. In Python, a standard `datetime.datetime` object can be converted\n to this format using\n [`strftime`](https://docs.python.org/2/library/time.html#time.strftime) with\n the time format spec \'%Y-%m-%dT%H:%M:%S.%fZ\'. Likewise, in Java, one can use\n the Joda Time\'s [`ISODateTimeFormat.dateTime()`](\n http://www.joda.org/joda-time/apidocs/org/joda/time/format/ISODateTimeFormat.html#dateTime%2D%2D\n ) to obtain a formatter capable of generating timestamps in this format.\n\n\n',
  },
  {
  path: [4, 0, 2, 0],
- span: [130, 2, 20],
+ span: [139, 2, 20],
  leadingDetachedComments: [],
  leadingComments:
  ' Represents seconds of UTC time since Unix epoch\n 1970-01-01T00:00:00Z. Must be from 0001-01-01T00:00:00Z to\n 9999-12-31T23:59:59Z inclusive.\n',
  },
  {
  path: [4, 0, 2, 1],
- span: [136, 2, 18],
+ span: [145, 2, 18],
  leadingDetachedComments: [],
  leadingComments:
  ' Non-negative fractions of a second at nanosecond resolution. Negative\n second values with fractions must still have non-negative nanos values\n that count forward in time. Must be from 0 to 999,999,999\n inclusive.\n',