Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Cannot start container #91

Closed
0x646e78 opened this issue Jun 30, 2014 · 20 comments
Closed

Cannot start container #91

0x646e78 opened this issue Jun 30, 2014 · 20 comments

Comments

@0x646e78
Copy link

Hi. Maybe I'm missing something obvious, but I'm new to docker and also gitlab. I can't get the container to start.

First, thanks for the great resources, I started with your postgres image and that worked really well.

In short I run 'run', it shows briefly in ps and then disappears.
/var/log/docker can be found at:
https://gist.github.com/auraltension/412f1b13d3342f43579e#file-gistfile1-txt

docker run --name='gitlab' -i -t --rm --link postgresql:postgresql -e "DB_USER=gitlab" -e "DB_PASS=rOrUQUqCvPLqEL8PSwC9a" -e "DB_NAME=gitlabhq_production" -p 10022:995 -p 10080:8080 -e "GITLAB_PORT=10080" -e "GITLAB_SSH_PORT=10022" -v /opt/gitlab/data:/home/git/data sameersbn/gitlab:latest app:rake gitlab:setup

User: root Password: deih7eey9uKo
Starting openssh server...
Starting redis server...
Running gitlab rake task...
Instance method "lock!" is already defined in ActiveRecord::Base, use generic helper instead or set StateMachine::Machine.ignore_method_conflicts = true.
This will create the necessary database tables and seed the database.
You will lose any previous data stored in the database.
Do you want to continue (yes/no)? yes

gitlabhq_production already exists
-- enable_extension("plpgsql")
-> 0.0351s
-- create_table("broadcast_messages", {:force=>true})
-> 0.0410s
-- create_table("deploy_keys_projects", {:force=>true})
....
-- initialize_schema_migrations_table()
-> 0.0013s
Adding limits to schema.rb for mysql

== Seed from /home/git/gitlab/db/fixtures/production/001_admin.rb
2014-06-30T12:48:07Z 191 TID-epb7s INFO: Sidekiq client with redis options {:url=>"redis://localhost:6379", :namespace=>"resque:gitlab"}

Administrator account created:

login.........admin@local.host
password......5iveL!fe

docker run --name='gitlab' -d --link postgresql:postgresql -e "DB_USER=gitlab" -e "DB_PASS=rOrUQUqCvPLqEL8PSwC9a" -e "DB_NAME=gitlabhq_production" -p 10022:995 -p 10080:8080 -e "GITLAB_PORT=10080" -e "GITLAB_SSH_PORT=10022" -v /opt/gitlab/data:/home/git/data sameersbn/gitlab:latest

af05128da659f9c56ef730ce843cb631e551b797d20e6e56ddda2ac3a976fda4

root@jake:/home/auraltension# docker ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
027c3589a4d8 sameersbn/gitlab:latest /app/init app:start 6 seconds ago Up 6 seconds 22/tcp, 443/tcp, 80/tcp, 0.0.0.0:10022->995/tcp, 0.0.0.0:10080->8080/tcp gitlab

root@jake:~# docker attach gitlab
2014/06/30 12:53:48 You cannot attach to a stopped container, start it first

root@jake:/home/auraltension# docker ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES

root@jake:~# docker logs gitlab
User: root Password: Uzu4mi6ua9Eh
Starting openssh server...
Starting redis server...
Starting nginx...
Migrating database...
Compiling assets. Please be patient, this could take a while...
Instance method "lock!" is already defined in ActiveRecord::Base, use generic helper instead or set StateMachine::Machine.ignore_method_conflicts = true.

@sameersbn
Copy link
Owner

@auraltension try deleting the folder /opt/gitlab/data/tmp on the host and run the command again

rm -rf /opt/gitlab/data/tmp
docker run --name='gitlab' -it --rm --link postgresql:postgresql \
-p 10022:995 -p 10080:8080 \
-e "DB_USER=gitlab" -e "DB_PASS=rOrUQUqCvPLqEL8PSwC9a" -e "DB_NAME=gitlabhq_production" \
-e "GITLAB_PORT=10080" -e "GITLAB_SSH_PORT=10022" \
-v /opt/gitlab/data:/home/git/data \
sameersbn/gitlab:latest

@sameersbn
Copy link
Owner

@auraltension if it does not work, paste the output of the above command in this thread

@0x646e78
Copy link
Author

0x646e78 commented Jul 2, 2014

Thanks for the response

root@jake:~# rm -rf /opt/gitlab/data/tmp

root@jake:~#  docker run --name='gitlab' -it --rm --link postgresql:postgresql \
> -p 10022:995 -p 10080:8080 \
> -e "DB_USER=gitlab" -e "DB_PASS=rOrUQUqCvPLqEL8PSwC9a" -e "DB_NAME=gitlabhq_production" \
> -e "GITLAB_PORT=10080" -e "GITLAB_SSH_PORT=10022" \
> -v /opt/gitlab/data:/home/git/data \
> sameersbn/gitlab:latest
User: root Password: phu8ri3Uquue
Starting openssh server...
Starting redis server...
Starting nginx...
Migrating database...
Compiling assets. Please be patient, this could take a while...
Instance method "lock!" is already defined in ActiveRecord::Base, use generic helper instead or set StateMachine::Machine.ignore_method_conflicts = true.

root@jake:~# docker ps a
CONTAINER ID        IMAGE                         COMMAND             CREATED             STATUS              PORTS               NAMES
3de9ff3a7a2d        sameersbn/postgresql:latest   /init               2 days ago          Up 41 hours         5432/tcp            postgresql 

I looked up the 'instance method "LOCK"' warning and theres an issue thread in Gitlab, but they claim it is a warning and gitlab should still work ?

@sameersbn
Copy link
Owner

@auraltension is that the end of the output?
The "lock" message seems to be harmless, ignore it.

@0x646e78
Copy link
Author

0x646e78 commented Jul 2, 2014

Yeah, That ^ is a complete copy of my terminal. Confirmed by docker logs.

@sameersbn
Copy link
Owner

@auraltension that does not look right. I am not able to pinpoint what could be the issue you are facing so we might have to do a few tests.

First, can you confirm if the instructions in the quick start guide works for you and you have a working gitlab instance?

docker run --name='gitlab' -i -t --rm \
-p 10022:22 -p 10080:80 \
-e "GITLAB_PORT=10080" -e "GITLAB_SSH_PORT=10022" \
sameersbn/gitlab:7.0.0

If it works, I suggest you delete the gitlab user and gitlabhq_production database from the postgresql server and re-create it using the instructions in the linking to postgresql database section in the readme and then run the following command to initialize the database:

docker run --name='gitlab' -i -t --rm \
-p 10022:22 -p 10080:80 --link postgresql:postgresql \
-e "GITLAB_PORT=10080" -e "GITLAB_SSH_PORT=10022" \
sameersbn/gitlab:7.0.0 app:rake gitlab:setup

and then this command to start the container:

docker run --name='gitlab' -i -t --rm \
-p 10022:22 -p 10080:80 --link postgresql:postgresql \
-e "GITLAB_PORT=10080" -e "GITLAB_SSH_PORT=10022" \
sameersbn/gitlab:7.0.0

If this still does not work, I recommend you paste the logs of the above commands here.

@sameersbn
Copy link
Owner

@auraltension another thing. can you tell me which postgresql image you are using? meaning is it sameersbn/postgresql or some other?

@0x646e78
Copy link
Author

0x646e78 commented Jul 3, 2014

ohh after spotting the following in my syslog I think this actually may be caused by my grsecurity patched kernel:

grsec: From X.X.X.X: denied RWX mmap of <anonymous mapping> by /usr/bin/ruby2.1[ruby2.1:15836] uid/euid:1000/1000 gid/egid:1000/1000, parent /usr/bin/sudo[sudo:15835] uid/euid:0/0 gid/egid:0/0
grsec: From X.X.X.X: denied RWX mmap of <anonymous mapping> by /usr/bin/ruby2.1[ruby2.1:15836] uid/euid:1000/1000 gid/egid:1000/1000, parent /usr/bin/sudo[sudo:15835] uid/euid:0/0 gid/egid:0/0
grsec: From X.X.X.X: denied resource overstep by requesting 4096 for RLIMIT_CORE against limit 0 for /usr/bin/ruby2.1[ruby2.1:15836] uid/euid:1000/1000 gid/egid:1000/1000, parent /usr/bin/sudo[sudo:15835] uid/euid:0/0 gid/egid:0/0
docker0: port 2(vethfe87) entered disabled state
device vethfe87 left promiscuous mode
docker0: port 2(vethfe87) entered disabled state
grsec: From X.X.X.X: unmount of /dev/mapper/docker-202:0-132502-2f41dadbcc31458c49da11cc9cd1455e176c7fd71e2c8825680d168d994da0b0 by /usr/bin/docker[docker:22841] uid/euid:0/0 gid/egid:0/0, parent /sbin/init[init:1] uid/euid:0/0 gid/egid:0/0

There is no /usr/bin/ruby2.1 on the host system so I guess this is within the container. I'll get a chance to play much later tonight and update the thread.

@0x646e78
Copy link
Author

0x646e78 commented Jul 3, 2014

I rolled to a non grsecurity patched kernel and then started getting different issues.

First, to add to my above comment, that syslog I had been from running the quickstart command set. And yes, I'm using the latest sameersbn/postgresql. The OS is Debian 7.5, on a Xen based VPS.

I realised this may be useful also

$ docker version
Client version: 1.0.1
Client API version: 1.12
Go version (client): go1.2.1
Git commit (client): 990021a
Server version: 1.0.1
Server API version: 1.12
Go version (server): go1.2.1
Git commit (server): 990021a

As I'm on a Linode VPS I rolled to one of their kernels, and tried again.

root@jake:/home/auraltension# uname -a
Linux jake 3.15.3-x86_64-linode44 #1 SMP Tue Jul 1 15:51:04 EDT 2014 x86_64 GNU/Linux

First, I rm'd gitlab, and tried again with my original full run command, which also failed, but with one extra line of output: su: System error

root@jake:/home/auraltension# docker logs gitlab
Creating SSH2 RSA key; this may take some time ...
Creating SSH2 DSA key; this may take some time ...
Creating SSH2 ECDSA key; this may take some time ...
Creating SSH2 ED25519 key; this may take some time ...
invoke-rc.d: policy-rc.d denied execution of restart.
Starting openssh server...
Starting redis server...
Starting nginx...
Migrating database...
Compiling assets. Please be patient, this could take a while...
Instance method "lock!" is already defined in ActiveRecord::Base, use generic helper instead or set StateMachine::Machine.ignore_method_conflicts = true.
su: System error

I then went back to basics, but now everything had started to respond with a chpasswd error.

$ sudo docker run --name='gitlab' -it --rm -p 10022:22 -p 10080:80 -v /opt/gitlab/data:/home/git/data -e "GITLAB_PORT=10080" -e "GITLAB_SSH_PORT=10022" sameersbn/gitlab:7.0.0
chpasswd: (user root) pam_chauthtok() failed, error:
System error
chpasswd: (line 1, user root) password not changed

Relevant syslog

Jul  3 13:26:23 jake kernel: EXT4-fs (dm-2): barriers disabled 
Jul  3 13:26:23 jake kernel: EXT4-fs (dm-2): mounted filesystem with ordered data mode. Opts: discard 
Jul  3 13:26:23 jake kernel: EXT4-fs (dm-2): barriers disabled 
Jul  3 13:26:23 jake kernel: EXT4-fs (dm-2): mounted filesystem with ordered data mode. Opts: discard 
Jul  3 13:26:23 jake kernel: netlink: 1 bytes leftover after parsing attributes in process `docker'. 
Jul  3 13:26:23 jake kernel: device veth8967 entered promiscuous mode 
Jul  3 13:26:23 jake kernel: IPv6: ADDRCONF(NETDEV_UP): veth8967: link is not ready 
Jul  3 13:26:23 jake kernel: IPv6: ADDRCONF(NETDEV_CHANGE): veth8967: link becomes ready 
Jul  3 13:26:23 jake kernel: docker0: port 2(veth8967) entered forwarding state 
Jul  3 13:26:23 jake kernel: docker0: port 2(veth8967) entered forwarding state 
Jul  3 13:26:23 jake kernel: docker0: port 2(veth8967) entered disabled state 
Jul  3 13:26:23 jake kernel: device veth8967 left promiscuous mode 
Jul  3 13:26:23 jake kernel: docker0: port 2(veth8967) entered disabled state 

Last section of strace

connect(5, {sa_family=AF_FILE, path="/var/run/docker.sock"}, 23) = 0
getsockname(5, {sa_family=AF_FILE, NULL}, [2]) = 0
getpeername(5, {sa_family=AF_FILE, path="/var/run/docker.sock"}, [23]) = 0
clock_gettime(CLOCK_REALTIME, {1404394048, 239847201}) = 0
clock_gettime(CLOCK_REALTIME, {1404394048, 239903886}) = 0
clock_gettime(CLOCK_REALTIME, {1404394048, 239968125}) = 0
write(5, "POST /v1.12/containers/288a980f0"..., 248) = 248
futex(0x14787d8, FUTEX_WAKE, 1)         = 1
read(5, "HTTP/1.1 200 OK\r\nContent-Type: a"..., 4096) = 68
ioctl(1, SNDCTL_TMR_TIMEBASE or TCGETS, {B38400 opost isig icanon echo ...}) = 0
ioctl(1, SNDCTL_TMR_START or TCSETS, {B38400 -opost -isig -icanon -echo ...}) = 0
                                                                                 clock_gettime(CLOCK_REALTIME, {1404394048, 242021085}) = 0 
                clock_gettime(CLOCK_REALTIME, {1404394048, 242164046}) = 0
                                                                          clock_gettime(CLOCK_REALTIME, {1404394048, 242304903}) = 0 
         clock_gettime(CLOCK_REALTIME, {1404394048, 242407119}) = 0
                                                                   clock_gettime(CLOCK_REALTIME, {1404394048, 242525765}) = 0 
  clock_gettime(CLOCK_REALTIME, {1404394048, 242638900}) = 0
                                                            write(1, "", 0)                         = 0
                                                                                                       read(5, 0xc21009a000, 4096)             = -1 EAGAIN (Resource temporarily unavailable) 
                                                                  clock_gettime(CLOCK_REALTIME, {1404394048, 243034482}) = 0
 read(0, chpasswd: (user root) pam_chauthtok() failed, error:
System error
chpasswd: (line 1, user root) password not changed
 <unfinished ... exit status 1>

I did a quick google and saw an issue you had raised in regards to this, so this may be better in another issue thread. It's intertesting that docker worked this first time after switching kernels, but now errors out. Tried rebooting but no joy. I'm now left wondering how to proceed, Linode claim docker works well, and their users seem to largely concur. The postgres is working. I'm thinking of switching back to my custom kernel and compare configs with those of the Ubuntu versions which had been claimed to have solved this latest issue, and doesing a bit of grsecurity forum reading.

Will keep you posted.

@0x646e78
Copy link
Author

0x646e78 commented Jul 3, 2014

Oh, it seems that issue was not solved by upgrading the host, but the container ubuntu version. hrmm

@sameersbn
Copy link
Owner

@auraltension what did you change the container ubuntu version to? one thing, you may want to use the latest gitlab image in which I have removed the use of chpasswd command so the above chpasswd related issue will not show.
Next make sure you have installed the latest docker version.
All this does not help, then check if selinux is enabled on the host. If yes, disable it to check if it is the cause of the issue.

@sameersbn
Copy link
Owner

@auraltension I have a feeling that the linode vps is has disabled the container networking and enabled host networking for docker. The above chpasswd error is a know error when host networking is used. But since the latest image docker pull sameersbn/gitlab:latest does not use chpasswd command anymore i think your issue should get resolved.

@0x646e78
Copy link
Author

0x646e78 commented Jul 3, 2014

Sorry, I didn't upgrade the container image. I was referencing the issue you had raised elsewhere. I'm off to bed and so will tinker more over the weekend.

@0x646e78
Copy link
Author

0x646e78 commented Jul 7, 2014

Went back to the stock Linode kernel and tried with latest.

sudo docker run --name='gitlab' -it --rm -p 10022:22 -p 10080:80 -v /opt/gitlab/data:/home/git/data -e "GITLAB_PORT=10080" -e "GITLAB_SSH_PORT=10022" sameersbn/gitlab:latest

Results in the 'su: Sustem error' issue

== Seed from /home/git/gitlab/db/fixtures/production/001_admin.rb
2014-07-07T04:52:16Z 587 TID-27thk INFO: Sidekiq client with redis options {:url=>"redis://localhost:6379", :namespace=>"resque:gitlab"}

Administrator account created:

login.........admin@local.host
password......5iveL!fe
su: System error

I wish I knew how to get better info for you.

@sameersbn
Copy link
Owner

@auraltension looks like a selinux issue. try disabling selinux and retry.

@0x646e78
Copy link
Author

0x646e78 commented Jul 7, 2014

I see on moby/moby#6345 that people are having similar issues with 3.15. I will try a Linode 3.14 kernel this evening and report back.

Also, I have no selinux nor apparmor.

@0x646e78
Copy link
Author

0x646e78 commented Jul 7, 2014

I used an earlier kernel version and it all worked fine.

$ uname -a
Linux jake 3.14.5-x86_64-linode42 #1 SMP Thu Jun 5 15:22:13 EDT 2014 x86_64 GNU/Linux

@sameersbn
Copy link
Owner

@auraltension glad you got it resolved.

@sameersbn
Copy link
Owner

@auraltension closing.. please reopen if required

@0x646e78
Copy link
Author

Thanks for closing, I see Docker will be putting out a release which includes the fix for the 3.15 issues in early august.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants