feat: implements datacollection function

+ Adds proto definition for ConversationBit.
+ Changes ConversationBit.Created field to int64.
+ Adds error-handling in client side JS.

.github/workflows/go.yml

@@ -11,7 +11,7 @@ on:
 
 jobs:
 
-  build:
+  build_server:
     runs-on: ubuntu-latest
     steps:
     - uses: actions/checkout@v4
@@ -24,3 +24,17 @@ jobs:
     - name: Build
       working-directory: ./server
       run: go build -v ./...
+
+  build_service:
+    runs-on: ubuntu-latest
+    steps:
+    - uses: actions/checkout@v4
+
+    - name: Set up Go
+      uses: actions/setup-go@v4
+      with:
+        go-version: '1.23.1'
+
+    - name: Build data collection
+      working-directory: ./services/data-collection
+      run: go build -v .

docs/Instructions.md

@@ -45,3 +45,43 @@ $ docker build . -t myherodotus -f Dockerfile --build-arg BUILD_VER=Herodotus
 $ docker tag myherodotus us-west1-docker.pkg.dev/${PROJECT_ID}/my-herodotus/base-image:${SEMVER}
 $ docker push us-west1-docker.pkg.dev/${PROJECT_ID}/my-herodotus/base-image:${SEMVER}
 ```
+
+## Update protobuf files
+
+This project uses [protocol buffer][protobuf] files to define types shared between the
+different services and surfaces of the application. These files must be updated and the output
+files regenerated when new fields are added.
+
+**NOTE**: The tool registry used by `buf` has a rate limit of 10 unauthenticated requests
+per hour :/. If this quota is reached, use [protoc directly instead][#protoc] 
+
+1. Install the [`buf` CLI][buf].
+
+    ```sh
+    $ GO111MODULE=on go install github.com/bufbuild/buf/cmd/buf@v1.47.2
+    ```
+
+1. Check the installation.
+
+    ```sh
+    buf --version
+    ```
+
+1. Build the protos. From the `protos/` directory, run the following command.
+
+    ```sh
+    buf build
+    ```
+
+1. Generate the protos. From the `protos/` directory, run the following command.
+
+    ```sh
+    buf generate
+    ```
+
+### Generate with protoc-gen-go {#protoc}
+
+1. 
+
+[buf]:      https://buf.build/docs/tutorials/getting-started-with-buf-cli
+[protobuf]: https://protobuf.dev/getting-started/gotutorial/

docs/Services.md

@@ -0,0 +1,53 @@
+# Services
+
+The MyHerodotus app has several microservices running behind the scenes to assist
+with data collection, evaluation, and model tuning.
+
+## Data collection
+
+The [Firestore-to-BigQuery](../services/data-collection/) service updates a
+BigQuery table with user data and responses from the MyHerodotus app. The service
+is triggered by a specific event: when a document is updated in the Firestore database.
+This event is used for data collection because it only occurs when a user has rated
+a response provided by the app.
+
+All data collected has had PII removed from it, specifically first name, last name,
+age, and email addresses. This list of deidentified info types is configurable in
+the app.
+
+The following code shows the equivalent gcloud command for exporting from Firestore.
+
+```sh
+$ gcloud firestore export gs://myherodotus --database=l200 --collection-ids=HerodotusDev,Conversations
+```
+
+### Deploy the service to Cloud functions
+
+To deploy the `data-collection` function to Cloud Run, run the following command from the
+`data-collection/` directory. Be sure to set the project ID using `gcloud config set project`.
+
+**IMPORTANT**: Make sure that `$PROJECT_ID` and `$DATASET_NAME` env vars are set before deploying
+the function!
+
+```sh
+$ gcloud functions deploy data-collection \
+  --gen2 \
+  --runtime=go121 \
+  --region="us-west1" \
+  --trigger-location="us-west1" \
+  --source=. \
+  --entry-point=CollectData \
+  --set-env-vars PROJECT_ID=${PROJECT_ID},DATASET_NAME=${DATASET_NAME},BUILD_VER=Herodotus \
+  --trigger-event-filters="type=google.cloud.firestore.document.v1.updated" \
+  --trigger-event-filters="database=l200" \
+  --trigger-event-filters-path-pattern=document='Herodotus/{userId}/Conversations/{conversationId}'
+```
+
+### Sources
+
++ https://cloud.google.com/functions/docs/calling/cloud-firestore
++ https://cloud.google.com/functions/docs/tutorials/storage
++ https://cloud.google.com/functions/docs/calling/eventarc
++ https://cloud.google.com/eventarc/docs/reference/supported-events#cloud-firestore
++ https://cloud.google.com/bigquery/docs/loading-data-cloud-firestore#python
++ https://cloud.google.com/firestore/docs/manage-data/export-import#gcloud

protos/buf.gen.yaml

@@ -0,0 +1,18 @@
+version: v2
+managed:
+  enabled: false
+  #override:
+  #    - file_option: go_package_prefix
+  #      path: conversation.proto
+  #      value: myherodotus.com/datacollection
+plugins:
+  - remote: buf.build/protocolbuffers/go
+    out: ../server
+    opt:
+    - module=myherodotus.com/main
+  - remote: buf.build/protocolbuffers/go
+    out: ../services/data-collection
+    opt:
+    - module=myherodotus.com/datacollection
+inputs:
+  - directory: .

protos/buf.yaml

@@ -0,0 +1,10 @@
+# For details on buf.yaml configuration, visit https://buf.build/docs/configuration/v2/buf-yaml
+version: v2
+modules:
+    - path: .
+lint:
+  use:
+    - STANDARD
+breaking:
+  use:
+    - FILE

protos/conversation.proto

@@ -0,0 +1,14 @@
+syntax = "proto3";
+package myherodotus;
+
+//option go_package = "myherodotus.com/main"; 
+
+message ConversationBit {
+  string bot_response = 1;
+  string user_query = 2;
+  string model = 3;
+  string prompt = 4;
+  int64  created = 5;
+  int32  token_count = 6;
+  string rating = 7;
+}