Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

SIGSEGV in Teleport on Arm7 #6642

Closed
BinaryPaean opened this issue Apr 28, 2021 · 12 comments · Fixed by #6711
Closed

SIGSEGV in Teleport on Arm7 #6642

BinaryPaean opened this issue Apr 28, 2021 · 12 comments · Fixed by #6711
Assignees
@xacrimon

Comments

@BinaryPaean
Copy link

I have a Teleport node+proxy+auth running in docker, with ports 3022, 3023, 3024, 3025, 3080 mapped in. Additionally, port 443 is proxied on the Docker host and forwarded to 443 on the container based on requested host name match.

The web interface works and I can use "tsh login" with no problems. Further, I can connect to its "own" node via web console or tsh. I have added two nodes that show up as expected under tsh ls or on the web interface. However, when I attempt to connect to either of the two separate nodes (One Raspberry Pi 3B+, One Pi 4) the teleport ssh client on that node has a SIGSEGV error, and the other end of the connection receives "ERROR: EOF". I am presumptively assuming this arm-related, but I will set up an x86 linux node to verify.

Node & Auth Server:
$tsh version
Teleport v6.1.2 git:v6.1.2-0-g23ab88c00 go1.15.5

Node config:

teleport:
  nodename: big-sister
  auth_token: "<redacted>"
  ca_pin: "<redacted>"
  advertise_ip: big-sister.baymaterials.local
  auth_servers:
   - teleport.baymaterials.com:3025
ssh_service:
    enabled: yes
    listen_addr: 0.0.0.0:3022
    commands:
      - name: hostname
        command: [hostname]
        period: 1m0s
      - name: machine
        command: [uname, -m]
        period: 1h0m0s
auth_service:
   enabled: false
proxy_service:
   enabled: false

Auth/server config:

teleport:
  nodename: teleport.baymaterials.com
  advertise_ip: 192.168.54.3
  data_dir: /var/lib/teleport
  log:
    output: stderr
    severity: INFO
  ca_pin: ""
auth_service:
  enabled: yes
  cluster_name: bmus-lan
ssh_service:
  # Do not advertise teleport.baymaterials.com here;
  # Docker internal network address is "correct" relative
  # to the proxy.
  enabled: yes
  labels:
    env: docker
  commands:
  - name: hostname
    command: [hostname]
    period: 1m0s
proxy_service:
  enabled: yes
  # listen_addr: 0.0.0.0:3023
  # tunnel_listen_addr: 0.0.0.0:3024
  # web_listen_addr: 0.0.0.0:3080
  https_keypairs: 
    - key_file: /var/lib/teleport_keys/privkey.pem
      cert_file: /var/lib/teleport_keys/fullchain.pem

Log from the node after a connection attempt:

Apr 27 19:20:54 big-sister systemd[1]: Started Teleport SSH Service.
Apr 27 19:20:55 big-sister teleport[20580]: [PROC:1]    INFO Connecting to the cluster bmus-lan with TLS client certificate. service/connect.go:132
Apr 27 19:20:55 big-sister teleport[20580]: [AUDIT:1]   INFO Creating directory /var/lib/teleport/log. service/service.go:1903
Apr 27 19:20:55 big-sister teleport[20580]: [AUDIT:1]   INFO Creating directory /var/lib/teleport/log/upload. service/service.go:1903
Apr 27 19:20:55 big-sister teleport[20580]: [AUDIT:1]   INFO Creating directory /var/lib/teleport/log/upload/sessions. service/service.go:1903
Apr 27 19:20:55 big-sister teleport[20580]: [AUDIT:1]   INFO Creating directory /var/lib/teleport/log/upload/sessions/default. service/service.go:1903
Apr 27 19:20:55 big-sister teleport[20580]: [AUDIT:1]   INFO Creating directory /var/lib/teleport/log. service/service.go:1903
Apr 27 19:20:55 big-sister teleport[20580]: [AUDIT:1]   INFO Creating directory /var/lib/teleport/log/upload. service/service.go:1903
Apr 27 19:20:55 big-sister teleport[20580]: [AUDIT:1]   INFO Creating directory /var/lib/teleport/log/upload/streaming. service/service.go:1903
Apr 27 19:20:55 big-sister teleport[20580]: [AUDIT:1]   INFO Creating directory /var/lib/teleport/log/upload/streaming/default. service/service.go:1903
Apr 27 19:20:55 big-sister teleport[20580]: [PROC:1]    INFO Service node is creating new listener on 0.0.0.0:3022. service/signals.go:213
Apr 27 19:20:55 big-sister teleport[20580]: [NODE:1]    INFO Service 6.1.2:v6.1.2-0-g23ab88c00 is starting on 0.0.0.0:3022 sqlite cache that will expire after connection to database is lost after 20h0m0s, will cache frequently accessed items for 2s. service/service.go:1774
Apr 27 19:20:55 big-sister teleport[20580]: [NODE:1]    INFO Service 6.1.2:v6.1.2-0-g23ab88c00 is starting on 0.0.0.0:3022. utils/cli.go:235
Apr 27 19:20:55 big-sister teleport[20580]: [NODE]         Service 6.1.2:v6.1.2-0-g23ab88c00 is starting on 0.0.0.0:3022.
Apr 27 19:20:55 big-sister teleport[20580]: [PROC:1]    INFO The new service has started successfully. Starting syncing rotation status with period 10m0s. service/connect.go:435
Apr 28 13:57:57 big-sister teleport[20580]: panic: runtime error: invalid memory address or nil pointer dereference
Apr 28 13:57:57 big-sister teleport[20580]:         panic: runtime error: invalid memory address or nil pointer dereference
Apr 28 13:57:57 big-sister teleport[20580]: [signal SIGSEGV: segmentation violation code=0x1 addr=0x0 pc=0x1f84c]
Apr 28 13:57:57 big-sister teleport[20580]: goroutine 312214 [running]:
Apr 28 13:57:57 big-sister teleport[20580]: runtime/internal/atomic.goXadd64(0x52060bc, 0x1, 0x0, 0x0, 0x0)
Apr 28 13:57:57 big-sister teleport[20580]:         /opt/go/src/runtime/internal/atomic/atomic_arm.go:103 +0x1c
Apr 28 13:57:57 big-sister teleport[20580]: go.uber.org/atomic.(*Int64).Add(...)
Apr 28 13:57:57 big-sister teleport[20580]:         /go/src/github.com/gravitational/teleport/vendor/go.uber.org/atomic/int64.go:50
Apr 28 13:57:57 big-sister teleport[20580]: go.uber.org/atomic.(*Int64).Inc(...)
Apr 28 13:57:57 big-sister teleport[20580]:         /go/src/github.com/gravitational/teleport/vendor/go.uber.org/atomic/int64.go:60
Apr 28 13:57:57 big-sister teleport[20580]: github.com/gravitational/teleport/lib/events.(*AuditWriter).EmitAuditEvent(0x5206000, 0x2433d38, 0x557ace0, 0x2447940, 0x52bc800, 0x0, 0x0)
Apr 28 13:57:57 big-sister teleport[20580]:         /go/src/github.com/gravitational/teleport/lib/events/auditwriter.go:270 +0x8c
Apr 28 13:57:57 big-sister teleport[20580]: github.com/gravitational/teleport/lib/srv.(*SessionRegistry).emitSessionLeaveEvent(0x557b200, 0x512d6c0)
Apr 28 13:57:57 big-sister teleport[20580]:         /go/src/github.com/gravitational/teleport/lib/srv/sess.go:274 +0x2a4
Apr 28 13:57:57 big-sister teleport[20580]: github.com/gravitational/teleport/lib/srv.(*SessionRegistry).leaveSession(0x557b200, 0x512d6c0, 0x0, 0x0)
Apr 28 13:57:57 big-sister teleport[20580]:         /go/src/github.com/gravitational/teleport/lib/srv/sess.go:303 +0x78
Apr 28 13:57:57 big-sister teleport[20580]: github.com/gravitational/teleport/lib/srv.(*party).Close.func1()
Apr 28 13:57:57 big-sister teleport[20580]:         /go/src/github.com/gravitational/teleport/lib/srv/sess.go:1418 +0xac
Apr 28 13:57:57 big-sister teleport[20580]: sync.(*Once).doSlow(0x512d724, 0x534f694)
Apr 28 13:57:57 big-sister teleport[20580]:         /opt/go/src/sync/once.go:66 +0xe4
Apr 28 13:57:57 big-sister teleport[20580]: sync.(*Once).Do(0x512d724, 0x534f694)
Apr 28 13:57:57 big-sister teleport[20580]:         /opt/go/src/sync/once.go:57 +0x44
Apr 28 13:57:57 big-sister teleport[20580]: github.com/gravitational/teleport/lib/srv.(*party).Close(0x512d6c0, 0x0, 0x0)
Apr 28 13:57:57 big-sister teleport[20580]:         /go/src/github.com/gravitational/teleport/lib/srv/sess.go:1416 +0x64
Apr 28 13:57:57 big-sister teleport[20580]: github.com/gravitational/teleport/lib/srv.closeAll(0x567e570, 0x6, 0x6, 0x6, 0x0)
Apr 28 13:57:57 big-sister teleport[20580]:         /go/src/github.com/gravitational/teleport/lib/srv/ctx.go:850 +0x68
Apr 28 13:57:57 big-sister teleport[20580]: github.com/gravitational/teleport/lib/srv.(*ServerContext).Close(0x56b6140, 0x0, 0x0)
Apr 28 13:57:57 big-sister teleport[20580]:         /go/src/github.com/gravitational/teleport/lib/srv/ctx.go:610 +0xac
Apr 28 13:57:57 big-sister teleport[20580]: panic(0x1d9c3b8, 0x3380420)
Apr 28 13:57:57 big-sister teleport[20580]:         /opt/go/src/runtime/panic.go:969 +0x158
Apr 28 13:57:57 big-sister teleport[20580]: runtime/internal/atomic.goXadd64(0x52060bc, 0x1, 0x0, 0x0, 0x0)
Apr 28 13:57:57 big-sister teleport[20580]:         /opt/go/src/runtime/internal/atomic/atomic_arm.go:103 +0x1c
Apr 28 13:57:57 big-sister teleport[20580]: go.uber.org/atomic.(*Int64).Add(...)
Apr 28 13:57:57 big-sister teleport[20580]:         /go/src/github.com/gravitational/teleport/vendor/go.uber.org/atomic/int64.go:50
Apr 28 13:57:57 big-sister teleport[20580]: go.uber.org/atomic.(*Int64).Inc(...)
Apr 28 13:57:57 big-sister teleport[20580]:         /go/src/github.com/gravitational/teleport/vendor/go.uber.org/atomic/int64.go:60
Apr 28 13:57:57 big-sister teleport[20580]: github.com/gravitational/teleport/lib/events.(*AuditWriter).EmitAuditEvent(0x5206000, 0x2433d38, 0x557ace0, 0x2447a60, 0x52c8900, 0x0, 0x0)
Apr 28 13:57:57 big-sister teleport[20580]:         /go/src/github.com/gravitational/teleport/lib/events/auditwriter.go:270 +0x8c
Apr 28 13:57:57 big-sister teleport[20580]: github.com/gravitational/teleport/lib/srv.(*session).startInteractive(0x5618200, 0x243ba00, 0x518aa00, 0x56b6140, 0x534fc64, 0x1)
Apr 28 13:57:57 big-sister teleport[20580]:         /go/src/github.com/gravitational/teleport/lib/srv/sess.go:773 +0x974
Apr 28 13:57:57 big-sister teleport[20580]: github.com/gravitational/teleport/lib/srv.(*SessionRegistry).OpenSession(0x557b200, 0x243ba00, 0x518aa00, 0x5322a00, 0x56b6140, 0x0, 0x0)
Apr 28 13:57:57 big-sister teleport[20580]:         /go/src/github.com/gravitational/teleport/lib/srv/sess.go:216 +0x284
Apr 28 13:57:57 big-sister teleport[20580]: github.com/gravitational/teleport/lib/srv.(*TermHandlers).HandleShell(0x5580230, 0x243ba00, 0x518aa00, 0x5322a00, 0x56b6140, 0x2, 0x2)
Apr 28 13:57:57 big-sister teleport[20580]:         /go/src/github.com/gravitational/teleport/lib/srv/termhandlers.go:113 +0xa8
Apr 28 13:57:57 big-sister teleport[20580]: github.com/gravitational/teleport/lib/srv/regular.(*Server).dispatch(0x546fd40, 0x243ba00, 0x518aa00, 0x5322a00, 0x56b6140, 0x0, 0x0)
Apr 28 13:57:57 big-sister teleport[20580]:         /go/src/github.com/gravitational/teleport/lib/srv/regular/sshserver.go:1303 +0x554
Apr 28 13:57:57 big-sister teleport[20580]: github.com/gravitational/teleport/lib/srv/regular.(*Server).handleSessionRequests(0x546fd40, 0x2433d38, 0x53494c0, 0x581dc20, 0x55314c0, 0x5, 0x0, 0x0, 0x54f5e98, 0x5, ...)
Apr 28 13:57:57 big-sister teleport[20580]:         /go/src/github.com/gravitational/teleport/lib/srv/regular/sshserver.go:1242 +0x524
Apr 28 13:57:57 big-sister teleport[20580]: github.com/gravitational/teleport/lib/srv/regular.(*Server).HandleNewChan.func1(0x546fd40, 0x2433d38, 0x53494c0, 0x581dc20, 0x55314c0, 0x5, 0x0, 0x0, 0x54f5e98, 0x5, ...)
Apr 28 13:57:57 big-sister teleport[20580]:         /go/src/github.com/gravitational/teleport/lib/srv/regular/sshserver.go:1022 +0x58
Apr 28 13:57:57 big-sister teleport[20580]: created by github.com/gravitational/teleport/lib/srv/regular.(*Server).HandleNewChan
Apr 28 13:57:57 big-sister teleport[20580]:         /go/src/github.com/gravitational/teleport/lib/srv/regular/sshserver.go:1021 +0xbd4
Apr 28 13:57:57 big-sister systemd[1]: teleport.service: Main process exited, code=exited, status=2/INVALIDARGUMENT
Apr 28 13:57:57 big-sister systemd[1]: teleport.service: Failed with result 'exit-code'.
Apr 28 13:57:57 big-sister systemd[1]: teleport.service: Service RestartSec=100ms expired, scheduling restart.
Apr 28 13:57:57 big-sister systemd[1]: teleport.service: Scheduled restart job, restart counter is at 11.
Apr 28 13:57:57 big-sister systemd[1]: Stopped Teleport SSH Service.
Apr 28 13:57:57 big-sister systemd[1]: Started Teleport SSH Service.
Apr 28 13:57:57 big-sister teleport[941]: INFO [PROC:1]    Connecting to the cluster bmus-lan with TLS client certificate. service/connect.go:132
Apr 28 13:57:58 big-sister teleport[941]: INFO [AUDIT:1]   Creating directory /var/lib/teleport/log. service/service.go:1903
Apr 28 13:57:58 big-sister teleport[941]: INFO [AUDIT:1]   Creating directory /var/lib/teleport/log/upload. service/service.go:1903
Apr 28 13:57:58 big-sister teleport[941]: INFO [AUDIT:1]   Creating directory /var/lib/teleport/log/upload/sessions. service/service.go:1903
Apr 28 13:57:58 big-sister teleport[941]: INFO [AUDIT:1]   Creating directory /var/lib/teleport/log/upload/sessions/default. service/service.go:1903
Apr 28 13:57:58 big-sister teleport[941]: INFO [AUDIT:1]   Creating directory /var/lib/teleport/log. service/service.go:1903
Apr 28 13:57:58 big-sister teleport[941]: INFO [AUDIT:1]   Creating directory /var/lib/teleport/log/upload. service/service.go:1903
Apr 28 13:57:58 big-sister teleport[941]: INFO [AUDIT:1]   Creating directory /var/lib/teleport/log/upload/streaming. service/service.go:1903
Apr 28 13:57:58 big-sister teleport[941]: INFO [AUDIT:1]   Creating directory /var/lib/teleport/log/upload/streaming/default. service/service.go:1903
Apr 28 13:57:58 big-sister teleport[941]: INFO [PROC:1]    Service node is creating new listener on 0.0.0.0:3022. service/signals.go:213
Apr 28 13:57:58 big-sister teleport[941]: INFO [NODE:1]    Service 6.1.2:v6.1.2-0-g23ab88c00 is starting on 0.0.0.0:3022 sqlite cache that will expire after connection to database is lost after 20h0m0s, will cache frequently accessed items for 2s. service/service.go:1774
Apr 28 13:57:58 big-sister teleport[941]: INFO [NODE:1]    Service 6.1.2:v6.1.2-0-g23ab88c00 is starting on 0.0.0.0:3022. utils/cli.go:235
Apr 28 13:57:58 big-sister teleport[941]: [NODE]         Service 6.1.2:v6.1.2-0-g23ab88c00 is starting on 0.0.0.0:3022.
Apr 28 13:57:58 big-sister teleport[941]: INFO [PROC:1]    The new service has started successfully. Starting syncing rotation status with period 10m0s. service/connect.go:435
@webvictim
Copy link
Contributor

webvictim commented Apr 29, 2021 •
edited
Loading

I think this is actually a known underlying issue and will be resolved by #6431

It should be out in Teleport 6.1.3 or 6.2, whichever comes first.

@BinaryPaean
Copy link
Author

Could be, and I hope so - since I'd love to see a fix soon!
However, #6431 seems to require specific timing to reproduce, and I was seeing this 100% of the time in my environment.

@BinaryPaean
Copy link
Author

I'm still seeing the same error in 6.1.3 after upgrading node and auth/proxy. I think this is identical to #6703.

@BinaryPaean
Copy link
Author

I noticed that the install instructions' advice for ARM:

curl https://get.gravitational.com/teleport-v6.1.0-linux-arm-bin.tar.gz.sha256

or manually edited to:

curl https://get.gravitational.com/teleport-v6.1.3-linux-arm-bin.tar.gz.sha256

Yields a version built by go 1.15.5, but the instructions for building from source indicate go 1.16 is required. Has anyone been able to test functionality on ARM with a go 1.16-built version yet? Is the teleport C.I. pipe out of date, or the instructions?

@webvictim
Copy link
Contributor

We've upgraded to Go 1.16 on master but 6.x versions are still using 1.15. Teleport 7.0 is likely to be the first version officially released using Go 1.16.

You can build from master yourself on Go 1.16 using the instructions in the README if you like.

@xacrimon
Copy link
Contributor

xacrimon commented May 3, 2021

Duplicate of #6703. Found the origin and currently testing a patch.

@BinaryPaean
Copy link
Author

v6.1.13 still blows up on ARM7, as does master even when built under Go 1.16.3. I can confirm that v6.0.0 does work on ARM7, however.

@xacrimon
Copy link
Contributor

xacrimon commented May 3, 2021

Hi @BinaryPaean, are you able to verify that the branch joel/armv7-atomic-align fixes this?

@xacrimon xacrimon mentioned this issue May 3, 2021
Merged
@BinaryPaean
Copy link
Author

Hi @BinaryPaean, are you able to verify that the branch joel/armv7-atomic-align fixes this?

No, that branch as of 50cv7cc1f did not resolve the crash-on-connect, when doing the below:

git pull && git checkout joel/armv7-atomic-align
make clean && make full
sudo cp ./build/teleport /usr/local/bin/
systemctl restart teleport

Should I clear out /usr/lib/teleport/* and try adding the node anew? (this was not necessary when downgrading from 6.1.3 to 6.0.0, however).

@xacrimon
Copy link
Contributor

xacrimon commented May 3, 2021 •
edited
Loading

@BinaryPaean Made a mistake before pushing and got the wrong commit, updated branch to commit b23520df5e27dd5751d32b278d35608bed61fe9a . Could you try again?

@BinaryPaean
Copy link
Author

@xacrimon Now on b23520df5e27 and I can confirm it works for me.

@xacrimon
Copy link
Contributor

xacrimon commented May 3, 2021

Hooray, we will try to issue a patch release soon.

@xacrimon xacrimon self-assigned this May 4, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@xacrimon xacrimon

Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

Align atomics to prevent segmentation faults on ARMv7 gravitational/teleport
3 participants
@BinaryPaean @webvictim @xacrimon