Exit codes are not reported

[shanet]

Sessions opened with the session manager plugin always report an exit code of zero even when the last executed command within the session returns a non-zero value. For example:

$ aws ssm start-session --target [ec2 instance]
  Starting session with SessionId: [name]-0ac90007f8af508b9
  $ exit 1
  Exiting session with sessionId: [name]-0ac90007f8af508b9.

$ echo $?
  0

Compare this to SSH where the exit code is carried over:

$ ssh [host]
  [host]:~$ exit 1
  logout
  Connection to [host] closed.

$ echo $?
  1

Not reporting the exit code makes scripting extremely difficult since there's no way to tell if a given command was successful or not. Reporting the exit code of the last command run in the session would be more inline with other standard tools.

[noefroidevaux]

This issue has implication in the AWS Copilot CLI (see aws/copilot-cli#4317).

[gileshinchcliff]

Hi Guys, I don't suppose there was any progress on this, it seems like a pretty basic piece of functionality and fundamentally is blocking us from removing SSH from our estate. Anything we can do to help progress this?

[yuting-fan]

Hi @shanet ,

By running echo $? after the start-session call, it means it is getting the exit code of the start-session CLI command, which successfully completed. If you would like to get the exit code of each command executed in the session, you would need to do it within the session.

Cheers,
Yuting

[shanet]

@yuting-fan Yes, I understand what echo $? is returning. The point is that the behavior of the session-manager is not consistent with that of standard tooling such as OpenSSH. This makes scripting incredibly difficult with SSM as it's impossible to determine if the last command run within SSM was successful or not. See the example I provided above to illustrate this discrepancy: #59 (comment).

I don't believe this issue should be closed for the reason you've given: that the exit code can be checked within the session. The whole point is that in order to use SSM in a script you need to know if the remote command was successful or not and this is currently impossible.

[noefroidevaux]

Hi @yuting-fan,
Why closing this issue? I think it's important to follow the same behaviour as SSH. Especially since it has a implication on other AWS project: aws/copilot-cli#4317

[yuting-fan]

Hi @shanet , @noefroidevaux ,

Session Manager is not implemented on top of SSH, and the intent is to provide interactive shell-like access based on IAM permissions, instead of SSH protocol. The exit code of start-session command returns zero because the AWS CLI call succeeds - this is by current design, and not a bug.

I understand your use case, and I will reopen to capture it as a feature request internally. We have a list of backlog items and feature requests to prioritize. Please stay tuned and we will post here when there is any update on this request.

Cheers,
Yuting

[yuting-fan]

Additionally, there are some workarounds you could try out in order to get the exit code of a command execution with SSM:

1. Record exit code of the commands executed within the session in a file on the disk, and then retrieve exit code after session ends.
2. Enable CloudWatch or S3 logging for Session Manager to get session logs and determine the outcome of commands executed within the session.
3. If you are executing a single command in a session, you can also try to use RunCommand for single target. The RunCommand execution aims for single command execution only, but it can return the exit code of that single command in the command response.

[shanet]

For reference, my use case is the following: I have a deploy script that needs to run database migrations within an ECS container running on an EC2 host. This is a fully automated process, but the shell is still interactive in case it needs to be manually cancelled or something goes wrong and a rollback needs to be manually initiated. SSM is used for running all of the remote commands needed for the database migrations on the ECS container. There is some additional complexity in that an SSM shell is first opened with the EC2 host and then an additional shell within the relevant Docker container running on that host.

Record exit code of the commands executed within the session in a file on the disk, and then retrieve exit code after session ends.

How would this work within a script that executes a remote command on a host via SSM? I don't believe SSM has the ability to download a file from a host/container so there wouldn't be a way to automate this.

Enable CloudWatch or S3 logging for Session Manager to get session logs and determine the outcome of commands executed within the session.

Yes, this is the current workaround that I'm using. I have to wait for the session log to be present on S3 after the session ends, download the log, and parse its output for a failure message. That's many moving parts to just get the exit code of a process, requires additional S3 permissions, slows down the pipeline, and also relies on the failure text being present exactly as expected.

If you are executing a single command in a session, you can also try to use RunCommand for single target.

In my case, this won't work because the shell needs to be interactive so I'm using the AWS-StartInteractiveCommand document. If I remember correctly from when I first opened this issue, the RunCommand won't provide an interactive shell to run commands.

[RothAndrew]

Ugly hacky workaround:

aws ssm start-session --region foo --target bar --document-name AWS-StartInteractiveCommand \
  --parameters command='["<TheCommand> && echo \"EXITCODE: 0\""]' | tee /dev/tty | grep -q "EXITCODE: 0"

If the "inner" command succeeded, we'll echo EXITCODE: 0 that we'll then grep for afterward. tee /dev/tty is there so the output still prints to the console rather than getting swallowed up by grep.

The string can be whatever you want, it doesn't have to be EXITCODE: 0. Just make sure it is something unique.

[mendhak]

Would appreciate any attention on this one. Even using ssm send-command is quite cumbersome because of its async nature, it's a multi step process to submit the command, monitor and wait for it to finish, then read the output. start-session with a command is almost a drop-in for running commands on a remote server and would be very CI/CD friendly.

[arianvp]

If you pass separateOutputStream=true ssm will report the exit code. It's not great but it's something:

$ aws ssm start-session  --target i-06a15fca1c90edf73 --document-name AWS-StartNonInteractiveCommand --parameters 'command=echo hey,separateOutputStream=true'

Starting session with SessionId: arian-lyyzj7qdusihsvnizrhfc3k22e
hey
EXIT_CODE: 0

Exiting session with sessionId: arian-lyyzj7qdusihsvnizrhfc3k22e.