Outdated proto/google/events/cloud/firestore/v1/data.proto

[pierophp]

The proto/google/events/cloud/firestore/v1/data.proto file does not work properly with the Firestore message.

Steps to reproduce

1. Go to Cloud Run
2. Write a Function
3. Select the Runtime "Node 22"
4. Add Trigger -> Firestore trigger
5. Save Trigger
6. Create
7. Wait it to deploy
8. Change any document on Firestore

The console.log provided in the example is broken. The entire object is incorrectly placed in the name field, as shown below:

A binary file as base64 example:

CuUBCscBCngKDRILc351ZWxsby1wb2MSHgoGZHJpdmVyGhRLcnVQeWlGcWNIdGtDOEJRYlZ1bxIpChFpbl9wcm9ncmVzc19yb3V0ZRoUWlF6M0tEekVMT3F4ZFJreDVNQmwSHAoEam9icxoUNlNpcUdTS3lnU3RzbGd2UXA2dmMaDQoCaWQSBxDvkTiYAQEaIQoEZGVwcxIZShcKBxDtkTiYAQEKDIoBBjkxOTc5MJgBARoZCgZzdGF0dXMSD4oBCWJsb2NrZWQxM5gBASoMCJO2urwGEJj66LUDMgsIufvcugYQoMXwDhLlAQrHAQp4Cg0SC3N+dWVsbG8tcG9jEh4KBmRyaXZlchoUS3J1UHlpRnFjSHRrQzhCUWJWdW8SKQoRaW5fcHJvZ3Jlc3Nfcm91dGUaFFpRejNLRHpFTE9xeGRSa3g1TUJsEhwKBGpvYnMaFDZTaXFHU0t5Z1N0c2xndlFwNnZjGg0KAmlkEgcQ75E4mAEBGiEKBGRlcHMSGUoXCgcQ7ZE4mAEBCgyKAQY5MTk3OTCYAQEaGQoGc3RhdHVzEg+KAQlibG9ja2VkMTKYAQEqDAiZsbq8BhCYjbGOAzILCLn73LoGEKDF8A4aCAoGc3RhdHVz

Using protoc to show the structure:

echo "BASE64_STRING" | base64 -d > message.bin
protoc --decode_raw < message.bin

It show this structure:

1 {
  1 {
    1 {
      1 {
        2: "s~uello-poc"
      }
      2 {
        1: "driver"
        3: "KruPyiFqcHtkC8BQbVuo"
      }
      2 {
        1: "in_progress_route"
        3: "ZQz3KDzELOqxdRkx5MBl"
      }
      2 {
        1: "jobs"
        3: "6SiqGSKygStslgvQp6vc"
      }
    }
    3 {
      1: "id"
      2 {
        2: 919791
        19: 1
      }
    }
    3 {
      1: "deps"
      2 {
        9 {
          1 {
            2: 919789
            19: 1
          }
          1 {
            17: "919790"
            19: 1
          }
        }
      }
    }
    3 {
      1: "status"
      2 {
        17: "blocked13"
        19: 1
      }
    }
  }
  5 {
    1: 1737399059
    2: 918175000
  }
  6 {
    1: 1733770681
    2: 31204000
  }
}
2 {
  1 {
    1 {
      1 {
        2: "s~uello-poc"
      }
      2 {
        1: "driver"
        3: "KruPyiFqcHtkC8BQbVuo"
      }
      2 {
        1: "in_progress_route"
        3: "ZQz3KDzELOqxdRkx5MBl"
      }
      2 {
        1: "jobs"
        3: "6SiqGSKygStslgvQp6vc"
      }
    }
    3 {
      1: "id"
      2 {
        2: 919791
        19: 1
      }
    }
    3 {
      1: "deps"
      2 {
        9 {
          1 {
            2: 919789
            19: 1
          }
          1 {
            17: "919790"
            19: 1
          }
        }
      }
    }
    3 {
      1: "status"
      2 {
        17: "blocked12"
        19: 1
      }
    }
  }
  5 {
    1: 1737398425
    2: 835471000
  }
  6 {
    1: 1733770681
    2: 31204000
  }
}
3 {
  1: "status"
}

[pierophp]

I've been able to make it work, but I not sure the names are correct:
data.proto

// Copyright 2023 Google LLC
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
//     https://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.

syntax = "proto3";

package google.events.cloud.firestore.v1;

import "google/protobuf/struct.proto";
import "google/protobuf/timestamp.proto";
import "google/type/latlng.proto";

option csharp_namespace = "Google.Events.Protobuf.Cloud.Firestore.V1";
option php_namespace = "Google\\Events\\Cloud\\Firestore\\V1";
option ruby_package = "Google::Events::Cloud::Firestore::V1";

// The data within all Firestore document events.
message DocumentEventData {
  // A Document object containing a post-operation document snapshot.
  // This is not populated for delete events.
  Document value = 1;

  // A Document object containing a pre-operation document snapshot.
  // This is only populated for update and delete events.
  Document old_value = 2;

  // A DocumentMask object that lists changed fields.
  // This is only populated for update events.
  DocumentMask update_mask = 3;
}

// A set of field paths on a document.
message DocumentMask {
  // The list of field paths in the mask.
  // See [Document.fields][google.cloud.firestore.v1.events.Document.fields]
  // for a field path syntax reference.
  repeated string field_paths = 1;
}

message DocumentIdKeyValue {
    string key = 1;  
    string value = 3;  
}

message DocumentIds {
    string name = 1;  
    repeated DocumentIdKeyValue collections = 2;  
}

message DocumentData {
  DocumentIds ids = 1;
  // The document's fields.
  //
  // The map keys represent field names.
  //
  // A simple field name contains only characters `a` to `z`, `A` to `Z`,
  // `0` to `9`, or `_`, and must not start with `0` to `9`. For example,
  // `foo_bar_17`.
  //
  // Field names matching the regular expression `__.*__` are reserved. Reserved
  // field names are forbidden except in certain documented contexts. The map
  // keys, represented as UTF-8, must not exceed 1,500 bytes and cannot be
  // empty.
  //
  // Field paths may be used in other contexts to refer to structured fields
  // defined here. For `map_value`, the field path is represented by the simple
  // or quoted field names of the containing fields, delimited by `.`. For
  // example, the structured field
  // `"foo" : { map_value: { "x&y" : { string_value: "hello" }}}` would be
  // represented by the field path `foo.x&y`.
  //
  // Within a field path, a quoted field name starts and ends with `` ` `` and
  // may contain any character. Some characters, including `` ` ``, must be
  // escaped using a `\`. For example, `` `x&y` `` represents `x&y` and
  // `` `bak\`tik` `` represents `` bak`tik ``.
  map<string, Value> fields = 3;  
}

// A Firestore document.
message Document {
  // The resource name of the document. For example:
  // `projects/{project_id}/databases/{database_id}/documents/{document_path}`
  DocumentData data = 1;
  // The time at which the document was created.
  //
  // This value increases monotonically when a document is deleted then
  // recreated. It can also be compared to values from other documents and
  // the `read_time` of a query.
  google.protobuf.Timestamp create_time = 5;

  // The time at which the document was last changed.
  //
  // This value is initially set to the `create_time` then increases
  // monotonically with each change to the document. It can also be
  // compared to values from other documents and the `read_time` of a query.
  google.protobuf.Timestamp update_time = 6;
}

// A message that can hold any of the supported value types.
message Value {
  // Must have a value set.
  oneof value_type {
    // A null value.
    google.protobuf.NullValue null_value = 11;

    // A boolean value.
    bool boolean_value = 1;

    // An integer value.
    int64 integer_value = 2;

    // A double value.
    double double_value = 3;

    // A timestamp value.
    //
    // Precise only to microseconds. When stored, any additional precision is
    // rounded down.
    google.protobuf.Timestamp timestamp_value = 10;

    // A string value.
    //
    // The string, represented as UTF-8, must not exceed 1 MiB - 89 bytes.
    // Only the first 1,500 bytes of the UTF-8 representation are considered by
    // queries.
    string string_value = 17;

    // A bytes value.
    //
    // Must not exceed 1 MiB - 89 bytes.
    // Only the first 1,500 bytes are considered by queries.
    bytes bytes_value = 18;

    // A reference to a document. For example:
    // `projects/{project_id}/databases/{database_id}/documents/{document_path}`.
    string reference_value = 5;

    // A geo point value representing a point on the surface of Earth.
    google.type.LatLng geo_point_value = 8;

    // An array value.
    //
    // Cannot directly contain another array value, though can contain an
    // map which contains another array.
    ArrayValue array_value = 9;

    // A map value.
    MapValue map_value = 6;
  }
}

// An array value.
message ArrayValue {
  // Values in the array.
  repeated Value values = 1;
}

// A map value.
message MapValue {
  // The map's fields.
  //
  // The map keys represent field names. Field names matching the regular
  // expression `__.*__` are reserved. Reserved field names are forbidden except
  // in certain documented contexts. The map keys, represented as UTF-8, must
  // not exceed 1,500 bytes and cannot be empty.
  map<string, Value> fields = 1;
}

[marcgovi]

Hi Piero, did you change anything from the template that's pre-populated for you when creating the Firestore Function?

On my end everything works as expected.

[pierophp]

Hi @marcgovi, I did not change anything.

My Firestore:

My function:

I've been able to reproduce it outside Cloud Run:
https://github.com/pierophp/firestore-event-test

node test.mjs

There is any other way that I can help you get more info?

[pierophp]

@marcgovi Can you post the message encoded as base64?

 console.log(Buffer.from(cloudEvent.data).toString('base64'));

[marcgovi]

Which event type are you using in the Eventarc trigger?

Notice that there are two Firestore modes:

• Native: google.cloud.firestore.document.v1.XXX
• Datastore: google.cloud.datastore.entity.v1.XXX

And both have different schemas:

• Native schema: proto/google/events/cloud/firestore/v1/data.proto
• Datastore schema: proto/google/events/cloud/datastore/v1/data.proto

I have the impression that you might have been using the wrong data schema.

[pierophp]

@marcgovi That is it!

I was using "google.cloud.datastore.entity.v1.written", but after switching to "google.cloud.firestore.document.v1.written", it now works.

What’s the difference between these two?

It’s not very intuitive, "google.cloud.datastore.X" appears first in the list, but the code uses "google.cloud.firestore.X" proto by default. It should be smart enough to generate the correct proto based on your selection.

[marcgovi]

Great :)
I've created an internal ticket for the Functions UI team to improve the template so that the code and schema proto loaded by default correspond to the chosen trigger event type (b/391681029).

To answer your question though, you would use one trigger event type family over the other depending on what Firestore mode you are using (https://cloud.google.com/datastore/docs/firestore-or-datastore). Ultimately, if you are in Native mode you should use google.cloud.firestore.document.v1.XXX and if you are in Datastore mode you should use google.cloud.datastore.entity.v1.XXX.

You could also wonder why having a Native mode database triggers an event for an event type that's designed for Datastore mode, or vice-versa. This is due to the Firestore event interoperability https://cloud.google.com/firestore/docs/eventarc#datastore_mode_and_native_mode_event_interoperability.