Internal failure executing runner: fork/exec /bin/zsh: argument list too long

[jlewi]

I was trying to run the command

jq -r 'select(.traceId == "35cf4b28e26a572eac7bfcca6d9582c4") | .message' "${LASTLOG}"

In RunMe and I was getting the error

Internal failure executing runner: fork/exec /bin/zsh: argument list too long

The command ran fine in my MacOs terminal.

Here's the notebook.
https://gist.github.com/jlewi/4417f86d9357a2fbe7491330064837df

Resetting the session seemed to fix the problem.

[jlewi]

I was using a dev build with stateful/vscode-runme#1475

[sourishkrout]

@pastuxso when you have a moment, can you try reproducing this issue, please. Thank you.

[pastuxso]

@jlewi Could you please provide me a raw JSON file? The gist contains a truncated example.

By the way, I've been trying to use a large dataset (504.3MB) and I haven't replicated it yet.

Datasets: https://jsoneditoronline.org/indepth/datasets/json-file-example/

[jlewi]

I don't think this has anything to do with jq or JSON. I just hit this problem in a completely different notebook (see attached).
Here the problem occurred in the cell running docker images. I had just done the docker build.

I reset the session and it worked. Do we have any thoughts on whether the error is coming from the golang runner or the frontend?

The error seems similar to #625 which was fixed with the v2 runner. I confirmed I'm using the v2alpha1 runner.

In that issue I mentioned server logs but I can't remember how I looked at server logs. I think I was manually starting runme golang server and so specified the "--log" flag.

My suspicion is that this error is coming from the GoRunner

runme/internal/runner/shell.go

Line 52 in 004c66d

func (s *Shell) Run(ctx context.Context) error {

Would adding logs help debug this?

build_inflight_batcher.md

[jlewi]

I recently bumped up the scrollback option to 100000 and now its happening very frequently. It seems to happen after running commands that produce a lot of output.

[sourishkrout]

I believe I accidentally reproduced the problem. Can you confirm that you're running on runner v2alpha1 @jlewi? It appears this problem is only present when you run v2.

[sourishkrout]

Following up. It looks like as part of the v2 rewrite saving the last exec's output in $__ is no longer being size-checked:

https://github.com/stateful/runme/blob/main/internal/command/env_store.go#L40-L60

In v1 we won't let any single variable exceed a limit. Perhaps we should just port this behavior.

https://github.com/stateful/runme/blob/main/internal/runner/service.go#L491-L495

[jlewi]

Confirmed; I'm using v2alpha1.

I didn't know this functionality existed.
Is this the relevant documentation: https://docs.runme.dev/getting-started/features#piping?

I think the documentation is a bit confusing

you can pass a variable declared in a previous execution to another in the same file.

IUC, it looks like the variable "$" is set to the output of the previous execution. So if you want to post process the output in another cell you can do that by using the variable "$".

The documentation makes it sound like you can't explicitly set variables in one cell and reference them in another but I do that all the time. e.g. in one cell

cd ${REPOROOT}/app
export CONFIGFILE=${RUNDIR}/config.yaml

and then in other cells refer to ${CONFIGFILE}.

[sourishkrout]

IUC, it looks like the variable "$" is set to the output of the previous execution. So if you want to post process the output in another cell you can do that by using the variable "$".

Just confirming: 🤔 Does $__ not show up in the docs for you, as in the screenshot below? I'm asking because your text says $ versus $__ which confused me.

I believe what you're referring to with referencing ENV var is the section above Piping: https://docs.runme.dev/getting-started/features#environment-variable-prompts

[jlewi]

Sorry. The docs are rendered correctly. I did type $__ but I didn't escape it so it gets rendered differently in github markdown.

To illustrate, how, I was confused by the documentation about piping. I might suggest changing

By default, Runme prevents variables declared in one cell from being directly accessed in another cell. However, by using Piping, which is declared using $__, you can pass a variable declared in a previous execution to another in the same file.

To something like

By default, Runme prevents the output of one cell from being directly accessed in another cell. However, Piping allows you to use $__ to refer to the output of the most recently executed cell.

[sourishkrout]

To something like

By default, Runme prevents the output of one cell from being directly accessed in another cell. However, Piping allows you to use $__ to refer to the output of the most recently executed cell.

👍 used it verbatim here: stateful/docs.runme.dev@8653b04