The get_tree operation should not require a page_token

[amonshiz]

#905 implemented the get_tree functionality of the build.bazel.remote.execution.v2.ContentAddressableStorage. The request type, GetTreeRequest, includes a field for a page_token which is to be the next_page_token component of a prior GetTreeResponse object.

However, the implementation in #905 assumes that every get_tree request will include a specifically formatted page_token value. The format is not a component of the API and is not required to be consistent between implementations. In fact, the "reference" implementation via buildfarm does explicitly does not require an initial page_token: https://github.com/bazelbuild/bazel-buildfarm/blob/635d9fed03bedc076d019dcf4f430f20744eafe8/src/main/java/build/buildfarm/common/TreeIterator.java#L51-L52.

While experimenting with NativeLink locally using the remotetool tool I noticed that NativeLink was failing to download a directory given a digest because of the missing page_token value. The errors were in the form of

  2024-09-07T03:32:26.190652Z ERROR nativelink_service::cas_server: error: status: Internal, message: "Failed to parse `size_bytes` in `page_token` : Failed on get_tree() command", details: [], metadata: MetadataMap { headers: {} }
    at nativelink-service/src/cas_server.rs:374
    in nativelink_service::cas_server::get_tree with request: GetTreeRequest { instance_name: "main", root_digest: Some(Digest { hash: "501f8a609222375221dc00498ef446c2c867bc219c189e5afff25fc5611d5f66", size_bytes: 171 }), page_size: 0, page_token: "", digest_function: Unknown }
    in nativelink_util::task::http_executor
    in nativelink::services::http_connection with remote_addr: 192.168.65.1:16451, socket_addr: 0.0.0.0:50051

How to Verify

Below is how I set up the test of the get_tree functionality.

1. Checkout remote-api-sdks repo:

   $ git clone git@github.com:bazelbuild/remote-apis-sdks.git

2. Build remotetool

   $ cd remote-apis-sdks/go/cmd/remotetool/
   $ go build -o remotetool

3. Set up simple NativeLink in-memory CAS only configuration

   $ cat <<EOF > inmemory.json
   {
     "stores": {
       "CAS_MAIN_STORE": {
         "memory": {
           "eviction_policy": {
             // 1gb.
             "max_bytes": 1000000000,
           }
         }
       }
     },
     "servers": [{
       "listener": {
         "http": {
           "socket_address": "0.0.0.0:50051",
         }
       },
       "services": {
         "cas": {
           "main": {
             "cas_store": "CAS_MAIN_STORE"
           }
         },
         "capabilities": {
           "main": {}
         },
         "bytestream": {
           "cas_stores": {
             "main": "CAS_MAIN_STORE",
           }
         }
       }
     }]
   }
   EOF

4. Run NativeLink via Docker (detached) using in memory CAS configuration

   docker run \
         -d \
         -v (pwd):/config \
         -v (pwd)/cas:/cas \
         -v (pwd)/cas-tmp:/cas-tmp \
         -p 50051:50051 \
         ghcr.io/tracemachina/nativelink:v0.5.1 \
         /config/inmemory.json

5. Generate a small file and upload using remotetool's upload_dir operation which uses BatchUpdateBlobs method.

   $ dd if=/dev/urandom of=random_file bs=1024 count=10 && remotetool \
         --operation=upload_dir \
         -service localhost:50051 \
         -instance main \
         --alsologtostderr \
         --v 1 \
         -service_no_security \
         --path (realpath random_file) \
         --json=-
   10+0 records in
   10+0 records out
   10240 bytes transferred in 0.000584 secs (17534247 bytes/sec)
   I0908 22:17:38.346414   67228 client.go:708] Connecting to remote execution instance main
   I0908 22:17:38.347213   67228 client.go:709] Connecting to remote execution service localhost:50051
   I0908 22:17:38.441458   67228 tool.go:342] Computing Merkle tree rooted at /Users/amonshiz/Developer/remote-apis-sdks/go/cmd/remotetool/random_file
   I0908 22:17:38.442267   67228 tool.go:352] Directory root digest: 8e935fd85e05de7e4497c41c5ccd17e8849e124a811725ab005fb1c31b6e4ac4/76
   I0908 22:17:38.442316   67228 tool.go:353] Directory stats: 1 files, 1 directories, 0 symlinks, 10316 total bytes
   I0908 22:17:38.442330   67228 tool.go:354] Uploading directory 8e935fd85e05de7e4497c41c5ccd17e8849e124a811725ab005fb1c31b6e4ac4/76 rooted at /Users/amonshiz/Developer/remote-apis-sdks/go/cmd/remotetool/random_file to CAS.
   {
     "InputFiles": 1,
     "InputDirectories": 1,
     "InputSymlinks": 0,
     "TotalInputBytes": 10316,
     "RootDigest": {
       "Hash": "8e935fd85e05de7e4497c41c5ccd17e8849e124a811725ab005fb1c31b6e4ac4",
       "Size": 76
     },
     "CountBlobs": 2,
     "CountCacheMisses": 2,
     "BytesTransferred": 10316,
     "BytesCacheMisses": 10316,
     "Error": ""
   }

6. Attempt to download the uploaded blob using download_dir which uses GetTree.

   $ remotetool \
         --operation=download_dir \
         -service localhost:50051 \
         -instance main \
         --alsologtostderr \--v 1 \
         -service_no_security \
         --path download_random_file \
         -digest  8e935fd85e05de7e4497c41c5ccd17e8849e124a811725ab005fb1c31b6e4ac4/76 rooted
   I0908 22:17:53.717675   67439 client.go:708] Connecting to remote execution instance main
   I0908 22:17:53.718197   67439 client.go:709] Connecting to remote execution service localhost:50051
   I0908 22:17:53.725175   67439 tool.go:316] Cleaning contents of download_random_file.
   I0908 22:17:53.725405   67439 tool.go:324] Downloading input root 8e935fd85e05de7e4497c41c5ccd17e8849e124a811725ab005fb1c31b6e4ac4/76 to download_random_file.
   I0908 22:17:53.740954   67439 cas_download.go:168] call failed with err=rpc error: code = Internal desc = Failed to parse `size_bytes` in `page_token` : Failed on get_tree() command, retrying.
   I0908 22:17:53.883324   67439 cas_download.go:168] call failed with err=rpc error: code = Internal desc = Failed to parse `size_bytes` in `page_token` : Failed on get_tree() command, retrying.
   I0908 22:17:54.146807   67439 cas_download.go:168] call failed with err=rpc error: code = Internal desc = Failed to parse `size_bytes` in `page_token` : Failed on get_tree() command, retrying.
   I0908 22:17:54.512517   67439 cas_download.go:168] call failed with err=rpc error: code = Internal desc = Failed to parse `size_bytes` in `page_token` : Failed on get_tree() command, retrying.
   I0908 22:17:54.907342   67439 cas_download.go:168] call failed with err=rpc error: code = Internal desc = Failed to parse `size_bytes` in `page_token` : Failed on get_tree() command, retrying.
   I0908 22:17:55.305253   67439 cas_download.go:168] call failed with err=rpc error: code = Internal desc = Failed to parse `size_bytes` in `page_token` : Failed on get_tree() command, retrying.
   F0908 22:17:55.305481   67439 main.go:112] error downloading directory for digest 8e935fd85e05de7e4497c41c5ccd17e8849e124a811725ab005fb1c31b6e4ac4/76: rpc error: code = Internal desc = retry budget exhausted (6 attempts): Failed to parse `size_bytes` in `page_token` : Failed on get_tree() command

7. Verify the errors in the NativeLink Docker logs

   $ docker ps
   CONTAINER ID   IMAGE                                    COMMAND                  CREATED          STATUS          PORTS                      NAMES
   1799ac5707ca   ghcr.io/tracemachina/nativelink:v0.5.1   "/nix/store/jlkla38c…"   32 seconds ago   Up 31 seconds   0.0.0.0:50051->50051/tcp   admiring_allen
   
   $ docker logs admiring_allen
     2024-09-09T02:17:26.073395Z  WARN nativelink: Ready, listening on 0.0.0.0:50051
       at src/bin/nativelink.rs:774
   
     2024-09-09T02:17:53.736782Z ERROR nativelink_service::cas_server: error: status: Internal, message: "Failed to parse `size_bytes` in `page_token` : Failed on get_tree() command", details: [], metadata: MetadataMap { headers: {} }
       at nativelink-service/src/cas_server.rs:374
       in nativelink_service::cas_server::get_tree with request: GetTreeRequest { instance_name: "main", root_digest: Some(Digest { hash: "8e935fd85e05de7e4497c41c5ccd17e8849e124a811725ab005fb1c31b6e4ac4", size_bytes: 76 }), page_size: 0, page_token: "", digest_function: Unknown }
       in nativelink_util::task::http_executor
       in nativelink::services::http_connection with remote_addr: 192.168.65.1:59814, socket_addr: 0.0.0.0:50051
   
     2024-09-09T02:17:53.881142Z ERROR nativelink_service::cas_server: error: status: Internal, message: "Failed to parse `size_bytes` in `page_token` : Failed on get_tree() command", details: [], metadata: MetadataMap { headers: {} }
       at nativelink-service/src/cas_server.rs:374
       in nativelink_service::cas_server::get_tree with request: GetTreeRequest { instance_name: "main", root_digest: Some(Digest { hash: "8e935fd85e05de7e4497c41c5ccd17e8849e124a811725ab005fb1c31b6e4ac4", size_bytes: 76 }), page_size: 0, page_token: "", digest_function: Unknown }
       in nativelink_util::task::http_executor
       in nativelink::services::http_connection with remote_addr: 192.168.65.1:59814, socket_addr: 0.0.0.0:50051
   
     2024-09-09T02:17:54.144514Z ERROR nativelink_service::cas_server: error: status: Internal, message: "Failed to parse `size_bytes` in `page_token` : Failed on get_tree() command", details: [], metadata: MetadataMap { headers: {} }
       at nativelink-service/src/cas_server.rs:374
       in nativelink_service::cas_server::get_tree with request: GetTreeRequest { instance_name: "main", root_digest: Some(Digest { hash: "8e935fd85e05de7e4497c41c5ccd17e8849e124a811725ab005fb1c31b6e4ac4", size_bytes: 76 }), page_size: 0, page_token: "", digest_function: Unknown }
       in nativelink_util::task::http_executor
       in nativelink::services::http_connection with remote_addr: 192.168.65.1:59814, socket_addr: 0.0.0.0:50051
   
     2024-09-09T02:17:54.510484Z ERROR nativelink_service::cas_server: error: status: Internal, message: "Failed to parse `size_bytes` in `page_token` : Failed on get_tree() command", details: [], metadata: MetadataMap { headers: {} }
       at nativelink-service/src/cas_server.rs:374
       in nativelink_service::cas_server::get_tree with request: GetTreeRequest { instance_name: "main", root_digest: Some(Digest { hash: "8e935fd85e05de7e4497c41c5ccd17e8849e124a811725ab005fb1c31b6e4ac4", size_bytes: 76 }), page_size: 0, page_token: "", digest_function: Unknown }
       in nativelink_util::task::http_executor
       in nativelink::services::http_connection with remote_addr: 192.168.65.1:59814, socket_addr: 0.0.0.0:50051
   
     2024-09-09T02:17:54.905387Z ERROR nativelink_service::cas_server: error: status: Internal, message: "Failed to parse `size_bytes` in `page_token` : Failed on get_tree() command", details: [], metadata: MetadataMap { headers: {} }
       at nativelink-service/src/cas_server.rs:374
       in nativelink_service::cas_server::get_tree with request: GetTreeRequest { instance_name: "main", root_digest: Some(Digest { hash: "8e935fd85e05de7e4497c41c5ccd17e8849e124a811725ab005fb1c31b6e4ac4", size_bytes: 76 }), page_size: 0, page_token: "", digest_function: Unknown }
       in nativelink_util::task::http_executor
       in nativelink::services::http_connection with remote_addr: 192.168.65.1:59814, socket_addr: 0.0.0.0:50051
   
     2024-09-09T02:17:55.303216Z ERROR nativelink_service::cas_server: error: status: Internal, message: "Failed to parse `size_bytes` in `page_token` : Failed on get_tree() command", details: [], metadata: MetadataMap { headers: {} }
       at nativelink-service/src/cas_server.rs:374
       in nativelink_service::cas_server::get_tree with request: GetTreeRequest { instance_name: "main", root_digest: Some(Digest { hash: "8e935fd85e05de7e4497c41c5ccd17e8849e124a811725ab005fb1c31b6e4ac4", size_bytes: 76 }), page_size: 0, page_token: "", digest_function: Unknown }
       in nativelink_util::task::http_executor
       in nativelink::services::http_connection with remote_addr: 192.168.65.1:59814, socket_addr: 0.0.0.0:50051