Implement remote resolver for storage indexer

[mtojek]

This PR implements hijack handlers for the storage indexer to redirect package zip, artifacts, static requests to https://package-storage.elastic.co/.

It uses HTTP 303 to pass the request. It was the easiest way to implement because I don't have to write content streaming at all. No need for Accept-Range tests, download tests, etc.

Breaking changes:

• Package Registry with storage indexer passes all requests to the Package Storage, it means that some resources won't be available to download except for statics (docs, imgs, etc.)

Spotted bugs:

• Package Storage v2 doesn't return correct content-types: SVG served via legacy EPR, SVG from Package Storage v2

Local testing:

go run . -feature-storage-indexer \
  -storage-indexer-bucket-internal gs://mtojek-temp/package-storage-v2 \
  -log-level debug

How to prepare the development bucket (cursor.json and search-index-all.json):

$ gsutil ls -r gs://mtojek-temp/package-storage-v2/v2
gs://mtojek-temp/package-storage-v2/v2/:
gs://mtojek-temp/package-storage-v2/v2/

gs://mtojek-temp/package-storage-v2/v2/metadata/:
gs://mtojek-temp/package-storage-v2/v2/metadata/
gs://mtojek-temp/package-storage-v2/v2/metadata/cursor.json

gs://mtojek-temp/package-storage-v2/v2/metadata/1/:
gs://mtojek-temp/package-storage-v2/v2/metadata/1/
gs://mtojek-temp/package-storage-v2/v2/metadata/1/search-index-all.json

Questions:

• Should we move indices to the public bucket and just keep the queue-publishing with package candidates in the private bucket? I don't remember why have we decided to put indices in the internal one...

Fixes #767

[elasticmachine]

💚 Build Succeeded

the below badges are clickable and redirect to their specific view in the CI or DOCS

Expand to view the summary

Build stats

• Start Time: 2022-06-27T11:12:24.727+0000

• Duration: 6 min 24 sec

Test stats 🧪

Test	Results
Failed	0
Passed	207
Skipped	0
Total	207

🤖 GitHub comments

To re-run your PR in the CI, just comment with:

• /test : Re-trigger the build.

[mtojek]

Status update:

I tried to use Google Storage client to reach out the public bucket, but failed as http.ServeContent needs objectHandle to implement io.ReadSeekCloser. I found the community implementation, but I can't guarantee that it's stable, but worked. I decided to revert.

I tried with a generic http.Client, but failed to implement streaming as it's mandatory to serve Range requests. I wasn't able to figure out how to implement it nicely. Reverted.

Now, I'm working on enabling the L7 proxy mode when the response can be streamed directly from Package Storage. I don't know what I do with streaming resources from the zip file. In the worst case, we will have to unpack and store them in the public bucket.

[jsoriano]

Added a suggestion about explicitly configuring the package to indicate that it requires redirects, instead of relying on the indexer, let me know what you think. I think this may require less refactors.

[jsoriano]

I wouldn't call it "Hijack", here we are doing a legit redirect 🙂

Some suggestions:

Suggested change

	storageIndexer.HijackArtifactsHandler(w, r, aPackage)
	storageIndexer.RedirectArtifactsHandler(w, r, aPackage)

Suggested change

	storageIndexer.HijackArtifactsHandler(w, r, aPackage)
	storageIndexer.RemoteArtifactsHandler(w, r, aPackage)

[mtojek]

I can apply those renamings, not a problem :)

I found an analogy with http.Hijacker, which lets a handler take over the TCP connection. In this case, we let Storage Indexer handlers process requests, which include a redirection.

[jsoriano]

Ok, let's keep this terminology if already present in the standard library :)

[jsoriano]

Thanks for addressing all the comments! 🚀