Proper documentation for the remote cache API

[asquithea]

Describe the feature you'd like to request

The documentation for providing a custom remote caching server merely says:

You can see the endpoints / requests needed here.

There are a couple of projects out there trying to provide a good faith conforming implementation of the remote caching API. For example: https://github.com/fox1t/turborepo-remote-cache

The docs for this project have to be followed very precisely or frustration ensues - for example, creating the undocumented config.json file, instead of attempting to follow the published official docs.

Trying to trouble-shoot as an end user is frustratingly difficult. For example:

• turbo link fails silently though the official docs tease that it might work
• turbo run does not log results of failed HTTP calls
• the difference between slug and teamId and how to force one or the other goes unexplained
• the presence of new (optional?) undocumented POST requests that look like they're carrying telemetry

I think this could be largely resolved by publishing a specification and making some small tweaks to the client so that custom remote caches are usable simply by reading the published docs (which are otherwise excellent).

Describe the solution you'd like

• An OpenAPI specification?
• Some form of API version negotation
• Slightly more helpful logging when a request fails (at -vv level, I suggest) - e.g. providing status codes
• Sufficient documentation to use the client with custom remote caches without needing to read the source code

Describe alternatives you've considered

• A reference implementation for self-hosting would be awesome, but I do appreciate that the main point of the Vercel acquisition is to pull developers towards the paid ecosystem.

• From a different angle, some CI systems (e.g. GitLab) can save and restore cache directories. This works quite well with turbo, but the cache grows without bounds so it slows down fairly quickly. Having an option to progressively trim the local cache directory (a bit like yarn's offline cache) would make this work a lot better and perhaps provide a less sophisticated alternative to remote caching.

• Looking to the future, if the ability to provide a custom remote cache server is something that will be going away in future for commercial reasons, honestly it would be really useful to know that now.

[nathanhammond]

There are two particular things in this issue which I can provide a bit of clarity around. The main thing we are limited on is time and competing priorities.

1. We would love to build prepackaged CI setups that enable using the CI cache without issue.
2. We intend to write an API specification for our remote cache server interface looks like. (And an associated test suite.)

Most importantly, we explicitly do not intend to make it in any way difficult to implement an alternative remote artifact server for Turborepo. Right now the behavior is unspecified and subject to some upcoming changes, but it is not by accident that you can see 100% of the client code for interfacing with the server.

[jlarmstrongiv]

Related #994

[aperez900907]

any update on the doc and reference for the needed endpoints for local cache servers? thanks

[quangphuchuynh95]

Is there any update on this?
Today I try this command but it seems that there is nothing happened remotely.

turbo run build --api="http://localhost:3000" --token="xxxxxxxxxxxxxxxxx" --remote-only

I have a simple express server that listens to all request and then log but I don't see anything happening.

[aperez900907]

will vercel remote api cache server specifications be documented for someone to be allowed to create a local alternative solution for a self-hosted remote cache?

[anthonyshew]

As @nathanhammond has mentioned, this documentation will come in the future as the API stabilizes. Will close since that day hasn't come yet!

[tknickman]

Reopening to mark as completed by #7235 & #7236!