Using hackage security #412
Conversation
|
(Travis fails because the server is started without the required security files in place.) |
|
There is a merge conflict here, fixed in https://github.com/ocharles/hackage-server ( |
|
I will create a new PR that fixes the tests, and then rebase this stuff on that. |
|
I've rebased this PR on In short: working on this, will indicate when it's ready to be merged. |
|
Ok guys. This is done! Tests pass! They won't yet on Travis because it relies on yet-unreleased versions of An overview of the commits in this PR:
|
|
I have a snapshot of the Hackage server state on my local machine (updated yesterday). I intended to run the server (using the same commit as on the live server right now) on this, then update to my PR and test the migration. However, I cannot run the server on my snapshot: I don't know what this means. I tried asking the server to create a checkpoint and updated my local snapshot after that, but that didn't help either. It's a little worrying that this is not working, makes me wonder about the integrity of the server state on the live server. (I tried running my local server on a Hackage snapshot in the past and it worked back then.) |
This should make it easier for people to experiment with the server, and will also facilitate continous integration.
(rather than using the mirrors from mirrors.json)
In two places the mirror client was using openTempFile and then renaming that temp file to become a permanent file; use openBinaryTempFileWithDefaultPermissions instead.
Although some SHA types made it into the server state, swapping these two
packages does not require a DB migration because they are fully binary
compatible (as well as compatible in their base16-encoded representations, of
course):
``` haskell
fromRight :: Either a b -> b
fromRight (Right b) = b
serializeOld :: Old.Digest Old.SHA256State -> BS.Strict.ByteString
serializeNew :: New.SHA256Digest -> BS.Strict.ByteString
serializeOld = Serialize.encode
serializeNew = Serialize.encode
deserializeOld :: BS.Strict.ByteString -> Old.Digest Old.SHA256State
deserializeNew :: BS.Strict.ByteString -> New.SHA256Digest
deserializeOld = fromRight . Serialize.decode
deserializeNew = fromRight . Serialize.decode
showOld :: Old.Digest Old.SHA256State -> String
showNew :: New.SHA256Digest -> String
showOld = show
showNew = show
readOld :: String -> Old.Digest Old.SHA256State
readNew :: String -> New.SHA256Digest
readOld = fromRight . readDigest
readNew = fromRight . readDigest
oldToNewSerialized :: Old.Digest Old.SHA256State -> New.SHA256Digest
oldToNewSerialized = undefined
test :: (Show a, Show b, Eq a)
=> (BS.Lazy.ByteString -> a) -> (a -> b) -> (b -> a) -> IO ()
test mk conv convBack = do
print origHash
print convHash
print origHash'
print $ origHash == origHash'
where
origHash = mk "Input string"
convHash = conv origHash
origHash' = convBack convHash
main :: IO ()
main = do
-- old -> new -> old, through binary form
test Old.sha256 (deserializeNew . serializeOld) (deserializeOld . serializeNew)
-- new -> old -> new, through binary form
test New.sha256 (deserializeOld . serializeNew) (deserializeNew . serializeOld)
-- old -> new -> old, through base16 (string) form
test Old.sha256 (readNew . showOld) (readOld . showNew)
-- new -> old -> new, through base16 (string) form
test New.sha256 (readOld . showNew) (readNew . showOld)
```
(This test is no longer in the code base because I've removed the dependency on
SHA completely, seems a bit silly to leave it in just so we can test this.)
This should improve efficiency of hash calculation for packages and for the
index significantly.
This will enable us to reproduce precisely the same TUF files after a server restart or a backup import, which is essential for the backup roundtrip. This commit _just_ introduces the new state; I've committed this separately from actually making using of this state because this bit requires a (simple) migration.
The SafeCopy instance will be necessary to return a TUFFile from an acid-state transaction; the simplification is that we removed the expiry time. The latter was originally intended for cache control, but we don't want to use expiry times for that, so it's now redundant. Removing it allows for some simplifications in the code, removes a possible runtime exception (not all files _have_ an expiry time), and if we now introduce a SafeCopy instances it makes sense to do it now or else we might have to introduce migrations later. This is a bit of an annoying change, because `TUFFile` used to have a phantom type argument indicating what kind of TUFFile it was (this type argument was instantiated with the corresponding type from the hackage-security library). Sadly, `deriveSafeCopy` is not very smart and when applied to `TUFFile` it added a spurious `SafeCopy` requirement on the phantom type argument. To avoid this problem, in this commit we remove the phantom type argument and instead introduce a bunch of newtype wrappers.
.. in preparation for doing the TUF update in acid-state.
(unnecessarily, at least). The DB state now has enough information to recreate the TUF files at any one point. We take advantage of this to make sure that if the TUF cache is updating itself, we only increment the timestamp/snapshot version if something actually changed, or the existing files are older than 1 hour. The cabal tests (including backup roundtrips) now pass! Yay! \o/
The hackage-security library now uses Int54 instead of Int everywhere (for annoying JSON reasons). In this commit I prove a SafeCopy instance for Int54 which translates to Int, so that we remain backwards binary compatible. I do change to Int54 in a few datatypes, but only in datatypes that are not (yet) serialized on the running server.
|
Note that I have not modified the generation of the source distribution; I don't know if we want to make sure that the example TUF files as part of that; possibly we do. I've opened a separate ticket #465 about this. |
|
Merged. 765ec00 |
This PR makes a few improvements to the
hackage-mirrorclient, and adds instructions to theREADMEfor setting up the security infrastructure.