Session Manager plugin prompts to update even though it's on the latest version

[FraserThompson]

I am running Copilot in a Docker container on top of the official aws-cli Docker image.

When I try to do svc exec I get:

1.2.458.0

  Looks like the Session Manager plugin is using version .
Would you like to update it to the latest version 1.2.458.0? (y/N)

It's already at the latest version, so this feels like a bug (also the strange formatting of the message).

Also, if possible it would be great if there was an option to skip these update checks because it makes automated exec calls harder to do if there's a chance it might prompt for user input.

This is my Dockerfile in case it helps:

FROM amazon/aws-cli:2.11.2

RUN curl -Lo copilot https://github.com/aws/copilot-cli/releases/download/v1.26.0/copilot-linux && chmod +x copilot && mv copilot /usr/local/bin/copilot
RUN curl "https://s3.amazonaws.com/session-manager-downloads/plugin/latest/linux_64bit/session-manager-plugin.rpm" -o "session-manager-plugin.rpm" && yum install -y session-manager-plugin.rpm

ENTRYPOINT ["/usr/local/bin/copilot"]

[JakeBrown]

We're seeing this issue when running copilot svc exec inside our Copilot deployment pipeline.

Multiple team members are also able to replicate it on MacOS locally.

[JakeBrown]

The source code for session-manager-plugin v1.2.458.0 is not available, making it impossible to debug: aws/session-manager-plugin#64

[wstewarttennes]

I'm having what seems to be a similar issue where CodePipelines are failing during the TestCommands phase across all my projects using Copilot -- get a session-manager-plugin error

[wstewarttennes]

Also, if possible it would be great if there was an option to skip these update checks because it makes automated exec calls harder to do if there's a chance it might prompt for user input.

This is the issue for me -- @JakeBrown have you figured out a way to automate commands when this error pops up?

[KollaAdithya]

Hello folks! Sorry for the late response 🙇
This seems to be an issue with Session Manager plugin.

I have changed this line over here to set the command Stderr instead of Stdout something like this seems to fix the issue (will not prompt to update).
if err := s.runner.Run(ssmPluginBinaryName, []string{"--version"}, Stderr(&s.currentVersionBuffer))

Even though running below command to check for local ssm manager plugin version

$ session-manager-plugin --version
1.2.458.0

is fetching version 1.2.458.0, but I could not find the release of the version 1.2.458.0 and the latest GitHub release is Release 1.2.398.0 - 2022-10-13.

I will update soon once I find out whether if there is any bug introduced in session-manager-plugin version v1.2.458.0 (still not available on the Github) or they have changed the Output of command session-manager-plugin --version from os.Stdout to os.Stderr

[timothy-bailey-redbox]

Just ran into this same issue in our Copilot pipeline testing stage. Running a second svc exec command in the same command set will give an error about the session-manager-plugin:

[Container] 2023/03/15 11:43:03 Running command unbuffer ./copilot-linux svc exec -c "npm run test:one" -n app-web -e dev --yes
   ---[ Installs and runs command, all good ]---

[Container] 2023/03/15 11:43:33 Running command unbuffer ./copilot-linux svc exec -c "npm run test:two" -n app-web -e dev --yes
1.2.458.0
Loaded plugins: ovl, priorities
Examining /tmp/ssmplugin2312962203/session-manager-plugin.rpm: session-manager-plugin-1.2.458.0-1.x86_64
/tmp/ssmplugin2312962203/session-manager-plugin.rpm: does not update installed package.
Error: Nothing to do
✘ update ssm plugin: exit status 1

Was working fine yesterday with version 1.2.398.0.

[wstewarttennes]

Just ran into this same issue in our Copilot pipeline testing stage. Running a second svc exec command in the same command set will give an error about the session-manager-plugin:

[Container] 2023/03/15 11:43:03 Running command unbuffer ./copilot-linux svc exec -c "npm run test:one" -n app-web -e dev --yes
   ---[ Installs and runs command, all good ]---

[Container] 2023/03/15 11:43:33 Running command unbuffer ./copilot-linux svc exec -c "npm run test:two" -n app-web -e dev --yes
1.2.458.0
Loaded plugins: ovl, priorities
Examining /tmp/ssmplugin2312962203/session-manager-plugin.rpm: session-manager-plugin-1.2.458.0-1.x86_64
/tmp/ssmplugin2312962203/session-manager-plugin.rpm: does not update installed package.
Error: Nothing to do
✘ update ssm plugin: exit status 1

Was working fine yesterday with version 1.2.398.0.

This is the same full error I'm getting. Unfortunately quite urgent because my multi-stage pipelines can't get past the first stage.

[KollaAdithya]

A quick work around could be to run copilot svc exec --yes=false so that it will not prompt to update Session Manager Plugin. let me know if this work for you!

[wstewarttennes]

@KollaAdithya I'm running with --yes=true so that my commands respond yes to everything (command that is run has manual prompts), am I misunderstanding something? I'll try this out.

[wstewarttennes]

@KollaAdithya when running with --yes=false my pipeline still fails and i get this:

Execute "python manage.py migrate" in container ftf in task -------------------.
993 | ✘ execute command python manage.py migrate in container ftf: start session ecs-execute-command------------ using ssm plugin: start session: exec: "session-manager-plugin": executable file not found in $PATH

[KollaAdithya]

Hi@wstewarttennes !

Regarding --yes flag is to indicate Whether to update the Session Manager Plugin.

copilot svc exec --yes=false only works if have already installed session-manger-plugin.
Can you check once whether you already have session manager plugin installed.

[kruhly]

We have had success in our pipeline when we have multiple svc exec calls in our test_commands, if we use, --yes for the first (so the session manager plugin gets updated) and then --yes=false for subsequent calls.

If we --yes=false for all of them the session-manager-plugin is not found

"session-manager-plugin":
executable file not found in $PATH 

[FraserThompson]

Thanks! The --yes flag is a good workaround for me (and will prevent stuff like this breaking automated pipelines in the future too)

[KollaAdithya]

@kruhly, @FraserThompson
I am glad that you are able to deploy pipeline successfully 🥳

[TiE23]

Switching to using --yes fixed my issue, too.

I spent over four hours fruitlessly trying to fix my deployment (unfortunately, while I didn't originally configure it, I was the only one available at my company to figure it out). A long time before finally suspecting that the very benign-sounding message "Looks like the Session Manager plugin is using version 1.2.398.0." was a clue - it was the only thing I had to go on.

After adding --yes, I got to see the results of the plugin being updated.

I'm happy I found a solution, but this kinda derailed my day.

[KollaAdithya]

Hello folks!
We are really sorry for the inconvenience 🙇 . Currently SSM agent team is actively working on reverting this change which is the root cause of the bug.

After successfully reverting the changes and releasing a new version, we expect that there will be no further errors going forward. We will keep you informed about the progress of the change reversion.

[JakeBrown]

Also, if possible it would be great if there was an option to skip these update checks because it makes automated exec calls harder to do if there's a chance it might prompt for user input.

This is the issue for me -- @JakeBrown have you figured out a way to automate commands when this error pops up?

@kruhly suggestion (#4627 (comment)) is working for us.

Also, we are considering pinning the session-manager-plugin version to avoid any such issues moving forward. I think we can do this by manually installing it using yum (eg https://github.com/aws/copilot-cli/blob/2e10abfd2b388519a206fe79b64e2b7161626718/internal/pkg/exec/ssm_plugin_install_latest_binary_linux.go) and then setting --yes=false.

[wstewarttennes]

Working for me too, thank you @kruhly , wouldn't have thought to do that

[TiE23]

I have to laugh about it now. Because this was me for a few hours yesterday:

Thanks for being so responsive.

[KollaAdithya]

Hello folks!
A new version of the Session Manager Plugin (available at https://github.com/aws/session-manager-plugin/releases/tag/1.2.463.0) has been released, which fixes the bug.
Please feel free to reopen any issues with the Session Manager Plugin if anyone still experiences problems.