snapshot key trust data unavailable. Has a notary repository been initialized?

[ghost]

I am trying to test the docker content feature with the help of instructions at https://docs.docker.com/engine/security/trust/trust_sandbox/. (When I tried last time , everything worked well) So, I made almost everything work but stumbling upon a new issue now. I ran the container linked with notary server etc. (exactly as per the instructions on the above given link). When I am trying to push the image it is giving me the error of - "
Error: trust data missing for remote repository sandboxregistry:5000/test/trusttest or remote repository not found: snapshot key trust data unavailable. Has a notary repository been initialized?"

My notary server is initialized - Screenshot attached. Can someone help me in solving this issue?

I did not do any changes in the notary and it is exactly as per the instructions on the website. I saw this post - https://forums.docker.com/t/cant-get-notary-content-trust-delegation-to-work/9029/9 but I still could not resolve it.

[ghost]

@cyli @riyazdf @endophage Quick help should be helpful

[ghost]

So, looking at this page https://github.com/docker/docker/blob/master/api/client/trust.go
says that my error might be because of Repositorynotexist, but as you can see my sandboxrepository that generated basing on the instructions on website is also running and active.

But one thing to note - > whenever I run the repository (private sandbox for testing this docker content trust), it gives be lot of errors on the console I run it (screenshot attached). Container runs but gets some errors like the one in screenshot. Will it have anything to do with this error?

Is this error because of non proper initialization of notary server or sandboxregistry?

[riyazdf]

hi @mkonakan: could you try adding -D to your push command (ex: docker -D push sandboxregistry:...) so we can see more debug logs?

[ghost]

@riyazdf Thanks for the quick response.

Seems issue is with 404 HTTP status?

[riyazdf]

Additionally, which version of notary client, server, and signer are you running?

[cyli]

It seems to be 404-ing when getting a snapshot key specifically, which is
odd. Just a sanity check myself, is this a fresh (recent) checkout of the
notary server? Because the new docker automatically uses server snapshot
signing but very old versions of the notary server did not support it.

Could you also please attach the compose logs? Specifically the server and
signer logs?

On Wednesday, June 1, 2016, mkonakan notifications@github.com wrote:

@riyazdf https://github.com/riyazdf Thanks for the quick response.

Seems issue is with 404 HTTP status?

[image: debug logs]
https://cloud.githubusercontent.com/assets/18063453/15730813/e2aa1376-2824-11e6-9398-b3d07dfa13b7.PNG

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
#762 (comment), or mute
the thread
https://github.com/notifications/unsubscribe/AANn4BxVXeHdu4hvT2qbIGd_5kY0hJFFks5qHi7bgaJpZM4IsFGE
.

[ghost]

@riyazdf
I can see a filename NOTARY-VERSION with 1.0-rc1 in it. Not sure if its the same version for entire package.

@cyli It seems to be getting the latest notary from link in this command - "git clone -b trust-sandbox https://github.com/docker/notary.git", also distribution project from link in -"$ git clone https://github.com/docker/distribution.git"

[cyli]

@mkonakan oh oops sorry I missed the -b in the instructions. It's telling
you to check out a really old branch that doesn't support server snapshot
signing. Can you try the same command without the -b?

That should be getting you the latest version that should support it.

On Wednesday, June 1, 2016, mkonakan notifications@github.com wrote:

@riyazdf https://github.com/riyazdf
Can you help me how I can extract the versions of them? I have done
exactly in the same way, it was instructed on this link -
https://docs.docker.com/engine/security/trust/trust_sandbox/.
In the folder where the file was extracted, I can see a filename
NOTARY-VERSION with 1.0-rc1 in it. Not sure if its the same version for
entire package.

@cyli https://github.com/cyli As I said, I followed the instructions on
above link and it seems to be getting the latest notary from link in this
command - "git clone -b trust-sandbox https://github.com/docker/notary.git",
also distribution project from link in -"$ git clone
https://github.com/docker/distribution.git"

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
#762 (comment), or mute
the thread
https://github.com/notifications/unsubscribe/AANn4MnxHr_YhVdM-UTkE5IDWh9-RXxCks5qHjbPgaJpZM4IsFGE
.

[ghost]

@cyli It says fatal: repository 'trust-sandbox' does not exist when I try it without -b. with -b , it clones.

[cyli]

Er sorry, I wasn't clear - I meant without that branch argument entirely.
So "git clone https://github.com/docker/notary.git". :)

On Wednesday, June 1, 2016, mkonakan notifications@github.com wrote:

@cyli https://github.com/cyli It says fatal: repository 'trust-sandbox'
does not exist when I try it without -b. with -b , it clones.

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
#762 (comment), or mute
the thread
https://github.com/notifications/unsubscribe/AANn4Kaa3Zg-E185xQEpDR8J9n3xAIbtks5qHjyXgaJpZM4IsFGE
.

[ghost]

No. your fine. Its my mistake. I never worked with git as I am not a developer. So, do not know it.

I tried what you said and then now, some other error started popping in - attached

[ghost]

@cyli I updated my base machine hosts file based on the container names now. Is there something I am missing?

Also, I am not sure server and signer containers are getting destroyed frequently. I am not doing any changes to them. Known bug?

Edit: Signer and server containers are getting destroyed after my pull attempt.

[cyli]

@mkonakan Apologies for the slow response, I am still trying to replicate. The best way to ensure the containers are destroyed are to docker-compose down in the directory where you docker-compose up'ed.

You also shouldn't need the container names in your /etc/hosts files. It's weird that it's trying to reach a different IP - I'm not sure what 192.168.200.2 is, although port 53 is the DNS port, so maybe it's trying to do some sort of DNS lookup? I would expect it to be trying to contact notaryserver at 127.0.0.1, given your hosts file, and certainly not at port 53.

[ghost]

@cyli that IP is my DNS address and it is doing an DNS lookup for notaryserver address but not sure why it is getting failed.

I am not sure the necessity for issuing docker-compose down, I said that containers are getting destroyed by themselves when I am issuing that pull command and I am not sure why they are getting destroyed.

[cyli]

@mkonakan So I've come across a couple issues so far with the doc:

1. was the removal of the -b trust-sandbox argument from the git-clone command
2. not sure if you're running into errors like:
   Warning: failed to get default registry endpoint from daemon (Error response from daemon: client is newer than server (client API version: 1.24, server API version: 1.23)). Using system default: https://index.docker.io/v1/ Error response from daemon: client is newer than server (client API version: 1.24, server API version: 1.23)
   if you are currently running an older docker version - if so, we can address that
3. The command to: docker run -it -v /var/run/docker.sock:/var/run/docker.sock --link notary_notaryserver_1:notaryserver --link sandboxregistry:sandboxregistry notarysandbox should be changed to docker run -it -v /var/run/docker.sock:/var/run/docker.sock --link notary_server_1:notaryserver --link sandboxregistry:sandboxregistry notarysandbox, otherwise it doesn't even work for me.

If you've already addressed these and are still getting the error, please let me know. I'm still continuing on, but am on very slow hotel wifi, so it's slow going :)

[ghost]

@cyli

1. I did make the change
2. I did face it but then resolved it
3. I did change it when I changed the -b argument

Yes, still this issue

[cyli]

@mkonakan Hmm... is it possible you've encountered the same issue as #753? What happens if you docker run -it -v /var/run/docker.sock:/var/run/docker.sock -e GODEBUG=netdns=cgo --link notary_server_1:notaryserver --link sandboxregistry:sandboxregistry notarysandbox? (also, after you run, can you do docker ps, figure out which container you're running in, and docker inspect it to confirm that it's actually linked to the notary server?)

[ghost]

@cyli

1. This is working - Its the problem with default Go resolver usage, I was able to sign the images with no error now..
2. Also, would request you to update the documents accordingly so that people would not stumble on this like me :)
3. Finally, thanks a ton for your support even staying in hotel :D

[cyli]

@mkonakan Thanks lots for trying content trust, and reporting these issues with the docs! Will definitely make that update. :)

[ghost]

@cyli I was just checking through this again. In later steps, there are also many things to be changed so that it would not confuse the people who are trying this.

1. Seems the sandbox registry image is built based on GOLANG base image, so docker exec -it sandboxregistry would give us an error and we need to change that to "docker exec -it sandboxregistry /bin/sh"
2. Error messages - All the error messages current link seems to be of old version. It displays a meaningful statement now instead of "no trust available etc" and also displays "manifest unknown" instead of "filesystem verification failed... etc".

Making these changes should be helpful for people who are trying to work with this latest version of docker content trust.

[cyli]

@mkonakan Thank you for all the suggestions! I will add those to the change.

[ghost]

@cyli I was also wondering the necessity for step 7 in "testing malicious images" in that section. I do not think it is needed there as we can check even without that that verification was failed. Is there anything I am missing?

[cyli]

@mkonakan It's to prove that it will not download the image, and it will not be there when you do docker images.

[cyli]

@mkonakan Out of curiosity, do you have more than 2 DNS servers configured on the host? (I am trying to figure out why the GODEBUG=netdns=cgo thing worked for you - I haven't been able to replicate that particular failure).

[ghost]

@cyli Haha, I was actually thinking the reason for docker rmi in other angles...Dint think too simple :P

I do have only one DNS server. In my opinion the reason for error being that the Go resolver not looking at hosts file to figure out the address of notaryserver from hosts file and trying to resolve on its own... I did not dig on it too much after this reasoning.

[innocentme1]

@cyli Did you do any changes to the tutorial on website? I have done the changes mentioned above, it worked pretty well for sometime and then it started giving me errors like attached. Also, the server and signer containers are getting killed automatically - not sure why. In between the docker ps commands in screenshot I made a compose up command to bring the containers. Kindly help!

[cyli]

@innocentme1 No, no change to https://docs.docker.com/engine/security/trust/trust_sandbox/

moby/moby#23195 was just merged today, if that's what you're referring to, but that change is not reflected at https://docs.docker.com/engine/security/trust/trust_sandbox/ yet.

If you are following the old instructions, everything should still work as before. I'm not sure what's causing the new network issues. :|

Could you try giving the new instructions a shot, and see if that helps?

[innocentme1]

@cyli it shows no route to host when containers are dead (as expected) and connection refused when container is up.

[cyli]

Are containers continuing to kill themselves with the new
instructions too? Could you try removing all existing containers first?

On Monday, June 13, 2016, Innocentme notifications@github.com wrote:

@cyli https://github.com/cyli it shows no route to host when containers
are dead (as expected) and connection refused when container is up.

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
#762 (comment), or mute
the thread
https://github.com/notifications/unsubscribe/AANn4G8WJE-YQkP1xYCnCh6RexdY4pe9ks5qLhv-gaJpZM4IsFGE
.

[cyli]

@innocentme1 By removing the pipe to /dev/null, the notary_server container should print out why it's failing to connect to the mysql container, as opposed to just the "waiting for notarymysql to come up" message (which only lets us know that something went wrong, but not what).

You may have to rebuild (docker-compose build) before running docker-compose up again.

[innocentme1]

I did rebuild - let me give a try again

@cyli I am also parallelly trying the new instructions (Yeah, I have to make this setup live ASAP as its highly critical for us here) and It gives me the following error. I am running compose version 1.7.1 which crosses the minimum requirement mentioned by u the instructions.

error:
ERROR: The Compose file './docker-compose.yml' is invalid because:
Unsupported config option for services.volumes: 'notarycerts'
Unsupported config option for services.networks: 'sandbox'

[cyli]

@innocentme1 There should not be any indentation in front of your "volumes" and "networks" declaration - they should be on the same indentation level as services, which should be none.

[cyli]

@innocentme1

Yeah, I have to make this setup live ASAP as its highly critical for us here

Just to be clear, also, the new instructions are for sandbox testing - if you want to get a live signing server up, you probably want a real DB with persistent data. In which case, you probably still want the compose file in the notary repo.

[innocentme1]

@cyli I rebuilt again and now below are the logs. This seems to be opening tcp on 10.5.74.105 on 3306. this IP is internal virtual IP seems to be load balancer. I am not sure why it is dialing to it on 3306 ( 3306 is port we opened for mysql? )

dial tcp 10.5.74.105:3306: getsockopt: connection refused
waiting for notarymysql to come up.
dial tcp 10.5.74.105:3306: getsockopt: connection refused
waiting for notarymysql to come up.

Also, yeah I knew that they are for sandbox testing. I am currently evaluating docker content trust itself and I am supposed to provide my report in 2-3 days time after which we might end up using this in our docker environment where we will set up our own notary.

[cyli]

@innocentme1 Yep, 3306 is indeed the mysql port. So if you do ping mysql on your local machine, outside of any containers, it hits 10.5.74.105?

[innocentme1]

@cyli Ahhh....yes...!

[cyli]

:| I think you're hitting the same weird DNS bug as #753. Which I've been asking around about, but no one else seems to be able to replicate (let me see if I can come up with a very simple test case that fails in the same way).

In the meantime though, I think you might have to add the -GODEBUG=netdns=cgo environment variables to the compose file as per https://github.com/docker/notary/pull/753/files#diff-8857124ebdcf7d878ada27a8dd023ea4R10.

[innocentme1]

@cyli

1. I am happy that you got someone to replicate your debug issue :P and I am sad that it was me :|
2. I tried the new setup instructions - working perfectly. Man, working with the old and new instructions , I can say that this new ones are way simple and better than the old ones. Also, I am not sure if its the case with you too- I found the signing , push and pull mechanism to be "very slow" in this new setup trustsandbox container
3. We encountered this last time but when I added this -GODEBUG=netdns=cgo with -e flag while running / launching container , it was solved (Remember? the one we debugged in starting of this thread - I am the ghost , had to del my old account for some reasons) but I am not surew why is it not working now :/ Let me try adding this to compose file and try it and will let you know.
4. Can you quickly reflect the changes on the website so that people need not struggle

[cyli]

@innocentme1

I found the signing , push and pull mechanism to be "very slow" in this new setup trustsandbox container

I didn't notice much of a difference in my setup, but since docker-in-docker writes images to a docker volume, it might depend on your volume driver.

We encountered this last time but when I added this -GODEBUG=netdns=cgo with -e flag while running / launching container , it was solved

Yep, I remember - is the mysql address on your network new? Not sure why it worked before and just started failing recently.

Can you quickly reflect the changes on the website so that people need not struggle

I'm not sure when the next doc build for docker/docker is - I can submit a PR to the existing docs, but I'm not sure when they're rebuilt.

I would also like to track down the GODEBUG issue a little bit before listing it as a workaround, since I don't really know how to describing the exact case in which it might be a fix. Unfortunately I'm completely out of my depth here with regards to the networking. If you wouldn't mind - I apologize for the trouble - but if you hit the exact DNS issue with trying:

1. docker run --rm -i --name mysql alpine nc -l -p 4444 in one terminal
2. docker run --rm -i --link mysql alpine sh -c "echo hello | nc mysql 4444" in another terminal?

could you file a bug with docker/docker with your exact docker client/server details and ubuntu setup? They might have better ideas about what questions to ask to figure out what's causing this.

I've been looking for issues which might match, but I haven't found any yet (I probably just don't know what I'm looking for)

[innocentme1]

@cyli

I am not sure about that address being added newly. I tried this last night at home too but still same issue.

Sure. I will do verify that and file to docker/docker sometime this weekend? I have some important tasks to work on as the setup is up and running now.

Also, I did do the changes in the compose files and rebuilt them. It works perfectly now :) so adding this GODEBUG to run command is not helping but adding it in compose file and making changes is working good.

[cyli]

@innocentme1 Whenever you have free time, thanks so much for your patience and perserverance . :)

[innocentme1]

@cyli thanks.

1. In case, we need to check test the rotating keys option in this test setup we made, how do we do that? I am little confused with the notary client CLI thing described here : https://docs.docker.com/notary/advanced_usage/ . In our setup, sandbox (I am still running with old instructions set) is the client which will be having keys and pulls metadata from notary server right? So, do I need to rotate keys on this and they will be synced with notary server?

P.S: This TUF and entire Notary architecture is kind of really taking quite some time to understand. Any specific resource you would suggest for me to take a look at for in-depth understanding of moving this Notary to production environment and also rotating keys etc.

1. When moving to production, we need to change the server , signer, and root ca certificates in /fixtures folder, thats it "w.r.t to certificates" right?

[endophage]

@innocentme1 the TUF specification is a good place to start to understand how Notary is attempting to do things. There are some nitty gritty implementation differences and we currently only support a non-configurable threshold of 1. We've also implemented a custom prioritization for traversing delegations (which the spec specifically leaves open for users to do) in docker, but the notary CLI will do a pre-order depth first search of the delegation tree for a given target.

You are correct that you just need to replace the keys and certs in /fixtures.

[innocentme1]

@endophage: I did go through it and took quite some time to understand everything. It is fine in my head now but the way Docker implemented it via Notary is little confusing to me now. I will try to figure that out.

Also. I am yet to research the delegations part. Thats on my next to-do list. Your reply answered my second question but not the first one. Can you throw some light on it too?

[endophage]

I'll do my best to answer your key rotation question.

The client and server do not syncronize keys. The client holds some private keys (always root and targets, sometimes snapshot) and the server holds the others (always timestamp, sometimes snapshot). Currently we only support the rotation of keys held by the client (we have started work on rotating keys held by the server), and the transfer of snapshot signing responsibility between the server and client (if the client owns the snapshot key it can be rotated, if the server owns the snapshot key, currently it cannot be rotated).

When you rotate a key held on the client, a new keypair is generated, the existing public key is removed from the appropriate role in root.json, and the newly generated public key is assigned to that role in root.json.

If it is the special case of the root key being rotated, the old root role is saved in root.json as root.<version> where <version> is the version number of root.json indicating the last version for which that root role was valid. The old public root key is also kept in the keys map in the root.json. This information is only used as a hint to publishers which old root keys need to be used to sign a root.json. It is not used by consumers at all.

[innocentme1]

@endophage sounds good as of now. Will let u know as I move on.

Also, Incase if I am using docker hub instead of registries for storing my signed images. do I need to add any env variable like "export DOCKER_CONTENT_TRUST_SERVER=https://notaryserver:4443" which we do for registries? I tried without and with giving "notary.docker.io" but it gives me an error like "Error establishing connection to notary repository: failed to create private key store in directory:" Am I missing something?

[endophage]

If you are using the docker CLI you need to provide DOCKER_CONTENT_TRUST_SERVER. If you are using the notary CLI you should provide the same server URL as the -s flag. Alternatively (notary CLI only), if you only interact with one server (or at least, with one server much more than others), you can use a notary config file: https://docs.docker.com/notary/reference/client-config/

[innocentme1]

@endophage I did already input that flag even in docker CLI and it still gives me error. Below for reference

mkonakan@ubuntu:$ docker push mkonakan/ubuntu:latest
The push refers to a repository [docker.io/mkonakan/ubuntu]
5f70bf18a086: Layer already exists
a3b5c80a4eba: Layer already exists
7f18b442972b: Layer already exists
3ce512daaf78: Layer already exists
7aae4540b42d: Layer already exists
latest: digest: sha256:15e2ba0b5e9d02f62a16332bf714732202d14b5930be31ec84421ee90ecff822 size: 1334
Signing and pushing trust metadata
Error establishing connection to notary repository: failed to create private key store in directory: /home/mkonakan/.docker/trust
failed to create private key store in directory: /home/mkonakan/.docker/trust
mkonakan@ubuntu:$ printenv | grep DOCKER
DOCKER_CONTENT_TRUST=1
DOCKER_CONTENT_TRUST_SERVER=https://notary.docker.io
mkonakan@ubuntu:~$

Edit: I am able to telnet to notary.docker.io on 443.

[endophage]

That looks more like there's a permissions problem with your ~/.docker directory. Could you post the output of ls -la ~ | grep docker and ls -la ~/.docker/.

[innocentme1]

It is indeed a permission issue. Rectified. Sorry. What confused me is the statement : "Error establishing connection to notary repository" before the failed to create..... Why would it fail establishing connection because of permissions? :/ I think little more clear statement should be good.

[endophage]

Yeah, I think that Error establishing connection prefix comes from Docker rather than Notary. Still our job to fix it but if memory serves it was phrased that way because almost all the errors Notary returns up to Docker are network related.

[innocentme1]

@endophage @cyli

hmm. I am little confused on rotating keys part. I just pushed a signed repository to my docker hub and then I am trying to rotate the keys. The command "notary key list" says - " no signing keys found" even I have all the keys in my ./docker/trust....folder. Attached is just a reference - it also has target keys etc. Only if I am able to list them, I can get the GUN and then rotate them right?

[endophage]

Try notary -d ~/.docker/trust key list

Notary is an independent project from Docker and by default puts all data in the ~/.notary directory. It's necessary to point it at the ~/.docker/trust directory to see docker related trust data.

[innocentme1]

@endophage Sorry for delayed response and thanks for helping me all the time.

1. When I rotate the keys , it gives me the error as "*fatal : client is offline". I tried to check if Internet is reachable - infact , it is! Not sure what else does this error mean? Anything else to be done from my side?

This also means that the key is not rotated because I tested it as below
a. Rotated the targets key
b. Pushed to my docker hub account with old and new keys still present on my machine: Push was successful
c. Removed the old key, kept the newly rotated key and then pushed it again: it was failure. I also pushed it with debug mode which indicated that it is looking for old targets key. Though the new one is created, it is not being looked for as it seems not to be synced with docker notary (client offline error).

notary -d ~/.docker/trust/ key rotate docker.io/mkonakan/wordpress targets --server-managed=false
Enter passphrase for new targets key with ID 0b9e7c7 (docker.io/mkonakan/wordpress):
Repeat passphrase for new targets key with ID 0b9e7c7 (docker.io/mkonakan/wordpress):

* fatal: client is offline

1. Is there any documentation that is made for rotating keys etc.operations on Notary CLI? I could not actually find. I only found brief intro to it on "advanced users guide for notary cli etc. page" I am feeling strange about how to know where abouts of using entire CLI in production environment. Documentation on Docker website regarding this particular one seems little confusing and not in proper formatted as it is done for other documentations. Is it that I still need to spend some time to get dig through it or else is the documentation still not full-fledged?
2. Do you think entire DCT is production ready?

[cyli]

@innocentme1 Can you try running the notary command with a -D to get debug output? notary -D -d ~/.docker/trust ...

[innocentme1]

@cyli Hey, below are messages of our interest

ERRO[0000] could not reach https://notary-server:4443: Get https://notary-server:4443/v2/: dial tcp: lookup notary-server on 127.0.1.1:53: no such host
INFO[0000] continuing in offline mode

Surprised. Do I need to change this host address somewer? add some config file or something?

[cyli]

@innocentme1 Are you using the trustsandbox (new instructions) setup? (e.g. are you executing the notary commands from within the trustsandbox container? If so, I think we linked that one as notaryserver, not notary-server.

[innocentme1]

@cyli figured out the issue. Thanks!

I am running on my base machine and I passed the trust server address with -s flag and it works fine now!