The worker requires a different binary architecture than what the builder sent it

[Skarlso]

@michelvocks Hi!

I thought the worker rebuilds the binary once it's trying to run it? I have a worker which is a linux machine, a raspberry pi model 4 running Raspbian GNU/Linux 10 (buster). That's a debian distro.

Once my worker picks something up, I'm getting this error:

2019-07-12T21:44:31.759+0100 [DEBUG] plugin: starting plugin: path=/home/pi/goprojects/gaia/gaia/pipelines/test-pi1_golang args=[]
2019-07-12T21:44:31.767+0100 [DEBUG] Gaia: cannot initialize the pipeline: error="fork/exec /home/pi/goprojects/gaia/gaia/pipelines/test-pi1_golang: exec format error

--- output ---
" pipeline="&{2 test-pi1 <nil> golang /home/pi/goprojects/gaia/gaia/pipelines/test-pi1_golang [0 129 36 190 78 75 92 36 44 108 200 119 162 183 171 156 173 222 183 173 155 223 64 133 135 114 20 36 198 104 197 98] [] 0001-01-01 00:00:00 +0000 UTC  false []  [] <nil>}"
2019-07-12T21:44:31.767+0100 [ERROR] Gaia: cannot get pipeline jobs: error="fork/exec /home/pi/goprojects/gaia/gaia/pipelines/test-pi1_golang: exec format error

--- output ---
" pipelinerun="unique_id:"c33ed8de-5c49-4549-bea5-634452ae6947" id:1 status:"scheduled" schedule_date:1562964261 pipeline_id:2 pipeline_name:"test-pi1_golang" pipeline_type:"golang" sha_sum:"\000\201$\276NK\\$,l\310w\242\267\253\234\255\336\267\255\233\337@\205\207r\024$\306h\305b" "
2019-07-12T21:44:31.892+0100 [ERROR] Gaia: failed to reschedule work at primary instance: error="rpc error: code = Unknown desc = worker is not registered"

Which clearly says that the binary is in an invalid format.

[Skarlso]

Also getting this warning:

2019-07-12T22:44:31.887+0200 [WARN ] Gaia: worker tries to update work but is not registered: metadata="map[:authority:[gaia-pipeline.io] content-type:[application/grpc] user-agent:[grpc-go/1.18.0]]"

Which I'm not sure I quite understand since the worker registered successfully.

[Skarlso]

At least workload sharing appears to be working. :D Once I started two jobs in quick succession they did pick them up individually. ( I have 2 of them ).

[michelvocks]

Currently, we do not support the build of a pipeline on the worker side. This is on my list since this is also wanted/needed for docker pipeline runs but for now it is simply not supported.

Also getting this warning:

2019-07-12T22:44:31.887+0200 [WARN ] Gaia: worker tries to update work but is not registered: metadata="map[:authority:[gaia-pipeline.io] content-type:[application/grpc] user-agent:[grpc-go/1.18.0]]"

Which I'm not sure I quite understand since the worker registered successfully.

This happens when a worker sends a request to the Gaia primary instance with an ID which is not in the database (e.g. is not registered). Maybe a process still running somewhere? 😄

[Skarlso]

Ah, I see. :D I didn't realise that, sorry. Should I take a look at it, or do you already have some thoughts on the matter? :)

Also, yeah no.. no idea why it was saying that. :( I'll have to do some more testing.

[michelvocks]

Ah, I see. :D I didn't realise that, sorry. Should I take a look at it, or do you already has some thoughts on the matter? :)

Feel free to pick that up if you want. My general idea was the following:
Worker tries to download and verify pipeline on worker -> verify step fails on worker -> worker tries to clone source repository if it is available -> worker builds pipeline and verifies it -> worker runs pipeline run.

What do you think?

[Skarlso]

Hmm...

Pros:
We already spent the resource of verifying and building the thing, so building it again is futile. We can just reuse what's already there. But that means you have send the binary to the worker. Which brings me to...

Con:
You have to send the binary to the worker which arguably is the larger cost of bandwidth. Usually if you are on a server, you don't care about spent CPU and memory. But what you care about is bandwidth. If you are on a local server, sure, no problem. But if there is anything in between usually that's an extra charge somewhere. And for large binaries that could be profound.

That said, the worker still has to download the thing from somewhere, which is github or a private git repo and then build it, which could be more. Gaia-s binary is like 22MB but the repo is 432MB since it includes the frontend...

Alright. :) I agree. Sorry for the rambling. :D I thought it through. :D

[michelvocks]

That said, the worker still has to download the thing from somewhere, which is github or a private git repo and then build it, which could be more. Gaia-s binary is like 22MB but the repo is 432MB since it includes the frontend...

Yep, that was the reason why I already implemented the download of pipelines. So workers currently detect that the plugin is not locally available and download them from the primary instance.

I think a single binary (which is usually machine code) is usually much smaller than the source code (with comments etc. included).

[Skarlso]

Yep. True. I'll do it. ;)

[Skarlso]

Done in #200.