Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

ci-release: no space on pi device #1889

Closed
BethGriggs opened this issue Aug 15, 2019 · 8 comments
Closed

ci-release: no space on pi device #1889

BethGriggs opened this issue Aug 15, 2019 · 8 comments

Comments

@BethGriggs
Copy link
Member

When running the builds for v8.16.1 I am hitting:

21:04:22 Building remotely on release-requireio_rvagg-debian9-armv6l_pi1p-1 (pi1-docker pi1-raspbian-stretch) in workspace /home/iojs/build/ws
...
21:04:23 java.nio.file.FileSystemException: /home/iojs/.jenkins/cache/jars/5C: No space left on device
21:04:23 	at sun.nio.fs.UnixException.translateToIOException(UnixException.java:91)
21:04:23 	at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:102)

https://ci-release.nodejs.org/job/iojs+release/nodes=pi1-docker/4633/console

@sam-github
Copy link
Contributor

I can't ssh into that machine:

build/hosts (build-10-with-devtoolset-s390x $%) % ssh release-requireio_rvagg-debian9-armv6l_pi1p-1                                                                          
zsh:1: parse error near `}'
ssh_exchange_identification: Connection closed by remote host
build/hosts (build-10-with-devtoolset-s390x $%) % grep -A 5 release-requireio_rvagg-debian9-armv6l_pi1p-1 ~/.ssh/config                                                      
Host release-requireio_rvagg-debian9-armv6l_pi1p-1
  HostName 192.168.2.40
  IdentityFile ~/.ssh/nodejs_build_release
  User pi
  ProxyCommand ssh -i {{ ansible_ssh_private_key_file }} -W %h:%p -p 2222 jump@vagg-arm.nodejs.org                                                                           

It looks like the .ssh/config has some ansible garbage in it, so I hacked it up, worked a bit better, but still wants a password:

 build/hosts (build-10-with-devtoolset-s390x $%) % ssh release-requireio_rvagg-debian9-armv6l_pi1p-1                    
The authenticity of host '[vagg-arm.nodejs.org]:2222 ([203.217.18.114]:2222)' can't be established.
ECDSA key fingerprint is SHA256:cQOh+H/IRjq/rFvpoqcgfFfneK7H2FRqo69rfzeS+Bs.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added '[vagg-arm.nodejs.org]:2222,[203.217.18.114]:2222' (ECDSA) to the list of known hosts.
jump@vagg-arm.nodejs.org's password: 

build/hosts (build-10-with-devtoolset-s390x $%) % grep -A 5 release-requireio_rvagg-debian9-armv6l_pi1p-1 ~/.ssh/config
Host release-requireio_rvagg-debian9-armv6l_pi1p-1 
  HostName 192.168.2.40
  IdentityFile ~/.ssh/nodejs_build_release
  User pi
  ProxyCommand ssh -i ~/.ssh/nodejs_build_release -W %h:%p -p 2222 jump@vagg-arm.nodejs.org

@sam-github
Copy link
Contributor

I tried directly sshing into jump, too, but either I have the wrong ssh command, or don't have access.

@mhdawson
Copy link
Member

I can get into the jump box, but trying to use as per the config files I just get:

user1@user1-lbuntu:~/.ssh$ ssh release-requireio_rvagg-debian9-armv6l_pi1p-1
channel 0: open failed: administratively prohibited: open failed
stdio forwarding failed
ssh_exchange_identification: Connection closed by remote host

@BethGriggs
Copy link
Member Author

The pi is now offline in ci-release Jenkins with:
Disk space is too low. Only 0.000GB left on /home/iojs/build

@mhdawson
Copy link
Member

mhdawson commented Aug 15, 2019

I've kind of run out of ideas. I can get on the jump box but it won't let me run ping to see if the machines are reachable, and ssh does not seem to be installed so I can't try to ssh from there to one of the machines either.

At this point I think we need to release for the rest of the platforms without ARMv6 and then wait for @rvagg to help fix the release machines.

@rvagg
Copy link
Member

rvagg commented Aug 16, 2019

Awake and on it, full disk and something weird with jump host, will address them both.
FYI half of the Pi's operate on one dedicated disk from the nfs host and the other half on another disk, so a full disk shouldn't take them all out of action—but will likely make the cluster a mess.

@rvagg
Copy link
Member

rvagg commented Aug 16, 2019

OK, that was a bit traumatic. Raspbian seem to have released a firmware earlier this month that prevented most of them from booting and a fix was out just recently but they didn't get that fix (because I wasn't aware of this problem) before I shut them all down .. bringing them back was a bit tedious and individual. Most of them are back now and I'm running some CI jobs to make sure they are operating properly.

WRT ssh access, I've updated the ProxyCommand in #1890 to something that should work (I updated the jump host recently and did some security modifications but didn't notice it stopped -W from working anymore since I don't use that, mostly). But I still don't know why {{ ansible_ssh_private_key_file }} ends up in there, it's been doing that for some time now and nobody's spent time trying to work out what broke.

@rvagg
Copy link
Member

rvagg commented Aug 16, 2019

@BethGriggs after a couple of reruns we got it warmed up and producing binaries again, you'll find a v8.16.1: linux-armv6l sitting in staging for you ready to go when you're around. Sorry for the dramas.

@rvagg rvagg closed this as completed Aug 16, 2019
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

4 participants