x/build: AIX builders fail with "unknown directive "Disconnected" "

[ayappanec]

AIX golang CI recently started failing with the below error.

--- FAIL: TestImportTestdata (0.05s)
    gcimporter_test.go:58: compile: /dev/null:1: unknown directive "Disconnected"
    gcimporter_test.go:59: go tool compile generics.go failed: exit status 1
--- FAIL: TestTypeNamingOrder (0.04s)
    gcimporter_test.go:58: compile: /dev/null:1: unknown directive "Disconnected"
    gcimporter_test.go:59: go tool compile g.go failed: exit status 1

https://build.golang.org/log/28dadf4b964f28d9137fb36e3f3181f03394faa7

Looks like there is some problem with the CI machine because our internal CI is working fine.
Any idea what could be the problem ? Any hints will be useful to fix that CI machine.

[cherrymui]

cc @golang/aix

[gabyhelp]

Related Issues and Documentation

• x/build: go/internal/gcimporter test fail consistently on AIX builder #68502 (closed)

_{(Emoji vote if this was helpful or unhelpful; more detailed feedback welcome in this discussion.)}

[cherrymui]

These tests also shows flaky but high frequency failures on the netbsd-arm-bsiegert builder, e.g. https://build.golang.org/log/7d88bbec36f25867219a11c16791e0c24ea622e7 . Not sure if they are separate builder issues or they are related.

cc @bsiegert

[pmur]

I was pinged by the AIX maintainers to take a look. Something on the VM is misbehaving, /dev/null is returning a single line of sshd logging. It claims an uptime of 1011 days. It's probably due for a reboot, so I've rebooted it.

@ayappanec I suspect this VM might be due for updates. Would you be able to investigate or verify it is up to date?

[bsiegert]

This may actually be caused by the Go test in some way. The other day, I noticed that /dev/null on the netbsd-arm builder had been overwritten with a file containing one line of text.

Could it be that something specifies /dev/null as output file and the program is misbehaving somehow?

[ayappanec]

I was pinged by the AIX maintainers to take a look. Something on the VM is misbehaving, /dev/null is returning a single line of sshd logging. It claims an uptime of 1011 days. It's probably due for a reboot, so I've rebooted it.

@ayappanec I suspect this VM might be due for updates. Would you be able to investigate or verify it is up to date?

Yes, the machine is due for updates. I will check on this.

[cherrymui]

It wouldn't be surprising if some tests write to /dev/null, but usually that should be fine just like it is common to do things like some shell command > /dev/null, which should just discard the output. Unless /dev/null somehow becomes a regular file on the machine? I don't know how that could happen, like running as root, deleting /dev/null and recreating it as a regular file?

[pmur]

It appeared something was recreating /dev/null as a regular file. When non-empty, it had the full contents of an sshd log message, without the syslog prefix. It seems unlikely to be the fault of the Go tooling or CI tooling.

I modified sshd_config and syslog to place sshd's logs into /var/log/messages last night. It still seems intact. If we start seeing builds pass again, I'll close this issue.

I think everything is running as root, so there are few guardrails. I don't have enough background with AIX to say what is clobbering the file, but sshd seems suspect.

[bsiegert]

I assume this happens if the software thinks it's extra clever by creating the file under a different name and using rename(2) to put it into place.