apiproxy

apiproxy is a proxy for HTTP/REST APIs with configurable cache timeouts, etc.

Documentation: https://sourcegraph.com/github.com/sourcegraph/apiproxy/tree

Installation

go get github.com/sourcegraph/apiproxy

Usage

apiproxy supports 3 modes of usage: as a standalone server, as a Go client, and as a Go HTTP server handler.

As a standalone HTTP proxy server

Running apiproxy as a standalone HTTP proxy server lets you access it from any HTTP client on any host.

The included apiproxy program runs a proxy server with a specified target URL:

$ go install github.com/sourcegraph/apiproxy/cmd/apiproxy
$ apiproxy http://api.example.com
2013/09/13 21:19:57 Starting proxy on :8080 with target http://api.example.com

Once launched, HTTP requests to http://localhost:8080 will be proxied to http://api.example.com and the responses cached according to the HTTP standard.

See apiproxy -h for more information.

As a Go client http.RoundTripper

Clients can use apiproxy.RevalidationTransport to modify the caching behavior of HTTP requests by setting a custom apiproxy.Validator on the transport. The Validator is used to determine whether a cache entry for a URL is still valid at a certain age.

A Validator can be created by wrapping a func(url *url.URL, age time.Duration) bool function with apiproxy.ValidatorFunc(...) or by using the built-in GitHub API implementation, MaxAge.

The RevalidationTransport can be used in an http.Client that is passed to external libraries, to give control over HTTP requests when using libraries whose only configuration point is an http.Client.

The file service/github/client_test.go contains a full example using the go-github library, summarized here:

transport := &apiproxy.RevalidationTransport{
  Transport: httpcache.NewMemoryCacheTransport(),
  Check: (&githubproxy.MaxAge{
    User:         time.Hour * 24,
    Repository:   time.Hour * 24,
    Repositories: time.Hour * 24,
    Activity:     time.Hour * 12,
  }).Validator(),
}
httpClient := &http.Client{Transport: transport}

client := github.NewClient(httpClient)

Now HTTP requests initiated by go-github will be subject to the caching policy set by the custom RevalidationTransport.

You can also inject a Cache-Control: no-cache header to a specific request if you use apiproxy.RequestModifyingTransport as follows:

// Wrap our transport from above in a RequestModifyingTransport.
transport = &apiproxy.RequestModifyingTransport{Transport: transport}
transport.Override(regexp.MustCompile(`^/repos/sourcegraph/apiproxy$`), apiproxy.NoCache, true)

// Now this call to the GitHub API will carry a `Cache-Control: no-cache` header.
client.Repositories.Get("sourcegraph", "apiproxy")

As a Go server http.Handler

The function apiproxy.NewCachingSingleHostReverseProxy(target *url.URL, cache Cache) *httputil.ReverseProxy returns a simple caching reverse proxy that you can use as an http.Handler.

You can wrap the handler's Transport in an 'apiproxy.RevalidationTransport` to specify custom cache timeout behavior.

The file cmd/apiproxy/apiproxy.go contains a full example, summarized here:

proxy := apiproxy.NewCachingSingleHostReverseProxy("https://api.github.com", httpcache.NewMemoryCache())
cachingTransport := proxy.Transport.(*httpcache.Transport)
cachingTransport.Transport = &apiproxy.RevalidationTransport{
  Check: apiproxy.ValidatorFunc(func(url *url.URL, age time.Duration) bool {
    // only revalidate expired cache entries older than 30 minutes
    return age > 30 * time.Minute
  }),
}
http.Handle("/", handlers.CombinedLoggingHandler(os.Stdout, proxy))
http.ListenAndServe(":8080", nil)

Examples

The included cmd/chirper/chirper.go example program helps demonstrate apiproxy's features. It returns a constantly updating JSON array of "chirps" at the path /chirps.

1. Run go run example/chirper/chirper.go in one terminal window.
2. Install apiproxy: go install github.com/sourcegraph/apiproxy/cmd/apiproxy
3. Run apiproxy -http=:8080 -never-revalidate http://localhost:9090 in another terminal window.

Now, let's make a request to the chirper API via apiproxy. Since this is our first request, apiproxy will fetch the response from the chirper HTTP server.

$ curl http://localhost:8080/chirps

Notice that apiproxy hit the chirper HTTP server: chirper logged the message "Listed chirps".

But next time we make the same request, apiproxy won't need to hit chirper, because the response has been cached and we are using the -never-revalidate option to treat cache entries as though they never expire.

$ curl http://localhost:8080/chirps

Note that we didn't hit chirper (it didn't log "Listed chirps").

However, if we pass a Cache-Control: no-cache header in our request, apiproxy will ignore the cache and hit chirper:

$ curl -H 'Cache-Control: no-cache' q:8080/chirps

Note that chirper logs "Listed chirps" after this request.

Cache backends

Any cache backend that implements httpcache.Cache suffices, including:

• httpcache.MemoryCache, instantiated with NewMemoryCache() *MemoryCache
• diskcache.Cache, instantiated with diskcache.New(basePath string) *Cache
• s3cache.Cache, instantiated with s3cache.New(bucketURL string) *Cache (requires env vars S3_ACCESS_KEY and S3_SECRET_KEY)

Contributing

Patches and bug reports welcomed! Report issues and submit pull requests using GitHub.