feat: Add Filesystem based remote store to support airgap.

[vaikas]

Signed-off-by: Ville Aikas vaikas@chainguard.dev

Please fill in the fields below to submit a pull request. The more information that is provided, the better.

Fixes #
Release Notes:

• Add file_store that implements RemoteStore backed by https://pkg.go.dev/io/fs. This is handy if you want to run airgap and bring your own repo.

Types of changes:

• [ x ] New feature (non-breaking change which adds functionality)

Description of the changes being introduced by the pull request:

Add Filesystem based remote store that the client can use. Motivation is to support airgapped environments which can't make network calls. The interface is golang io/fs.FS so that it can be backed by anything that supports the interface instead of only relying on disk.

I may have missed something, but I can't find a way to use it for airgap environments as is, since initializing requires network calls.

Please verify and check that the pull request fulfills the following requirements:

• Tests have been added for the bug fix or new feature
• Docs have been added for the bug fix or new feature

[znewman01]

You've pointed out a few things you need to fix, otherwise looking good

[vaikas]

@znewman01 @asraa please take particular look at this commit. Based on this here:
https://github.com/theupdateframework/go-tuf/blob/master/client/client.go#L843

I believe returning an error is what should be happening, so modified things accordingly.

[vaikas]

Ho hum:
golang/go#51442

--- FAIL: TestBadDirs (1.51s)
    testing.go:1090: TempDir RemoveAll cleanup: remove C:\Users\RUNNER~1\AppData\Local\Temp\TestBadDirs1564658038\001\repository-that-isfile: The process cannot access the file because it is being used by another process.
FAIL
coverage: 83.3% of statements

[vaikas]

lol, pasted to wrong issue, sorry, my bad:

FWIW, this has been wired in through sigstore here:
sigstore/sigstore#715
And
cosign here:
https://github.com/vaikas/cosign/tree/air-gap

And finally in scaffolding I use a local filesystem based custom TUF root here:
sigstore/scaffolding#382

https://github.com/sigstore/scaffolding/actions/runs/3139401394/jobs/5099778521
If you look at the 'Untar the repository from the fetched secret' section, and it's using a cosign built from here:
https://github.com/vaikas/cosign/tree/air-gap

[joshuagl]

golang/go#51442

The linked issue implies this is fixed in go 1.19, but that's the version in use on the failing tests? 😕

Should we implement the suggested workaround in golang/go#51442 (comment)

There is also a workaround that you can apply for your own specific tests independent of the Go version: you can defer (or t.Cleanup) your own function that iterates os.RemoveAll on the directory returned by TempDir until it is successfully removed.

[vaikas]

Thanks @joshuagl I added a loop through the files we create.

[JustinCappos]

How will metadata expiration be handled in the air gapped case? Is timestamp expiration a problem? Do you plan to just generate very long lifetimes for metadata files?

[trishankatdatadog]

How will metadata expiration be handled in the air gapped case? Is timestamp expiration a problem? Do you plan to just generate very long lifetimes for metadata files?

See the same issue and some ideas about it here

[joshuagl]

This is a cool feature, thanks for working on it @vaikas !

dabde9f makes me a little sad, but I can understand why you made that choice. I'll file an issue to try and resolve this in future.

[vaikas]

Thank you @joshuagl!!!