segfault caused by os.execvpe in windows

[matthewdeanmartin]

I'm using windows 11 and os.execvpe will cause a segfault. I happen to be using gitbash and pipx, but I get the same behavior if I don't use pipx.

After a bit of hacking, I got this to work, but with 2 second additional time cost over using aws directly. I see that 9 month ago execvpe was introduced probably to make things better for linux or mac users

I'm guessing you don't have access to a windows machine, you might want to consider a github action on a windows image to check to see if awslocal --help can run without segfault.

Another possibility is to use something like shellingham to detect the shell before doing things that the OS isn't expected to be able to do.

def run(cmd, env=None):
    """
    Replaces this process with the AWS CLI process, with the given command and environment
    """
    # if not env:
    #     env = {}
    # print("os.execvpe")
    # os.execvpe(cmd[0], cmd, env)
    if env is None:
        env = os.environ.copy()
    else:
        # Make sure we include the current environment to avoid missing essential variables
        full_env = os.environ.copy()
        full_env.update(env)

    print("subprocess.run")
    result = subprocess.run(cmd, env=full_env, text=True, capture_output=True)
    print(result.stdout)  # Print standard output
    print(result.stderr, file=sys.stderr)  # Print standard error to the error stream
    return result

When I use powershell or cmd, I get "ModuleNotFoundError: No module named 'boto3'", probably because the .bat file gets launched and doesn't actual activate the right environment, but I also got that even when installing to the system python (which is a no-no). I didn't pursue why the entry point scripts fail to get an activated environment, since I never use powershell or cmd.exe.

[MarcelStranak]

Hello @matthewdeanmartin,

I, too, utilize LocalStack on the Windows OS. I do not use gitbash and pipx, but in a normal powershell terminal or cmd terminal all works as expected.

Could you clarify why installing Python is discouraged? The AWS CLI tool is also offered as a Python pip package, and awslocal serves as a thin wrapper around it.

I recommend upgrading your WSL version to the latest version and using the endpoint URL http://localhost.localstack.cloud:4566 to potentially enhance connection speed.

Alternatively, you could use AWS CLI version 2 and configure a profile in your ~/.aws/credentials and ~/.aws/config files.
You can find the steps here https://docs.localstack.cloud/user-guide/integrations/aws-cli/#configuring-a-custom-profile.

I hope this helps with your setup.

[matthewdeanmartin]

Well I'm glad to hear it works on your machine.

I see you all have a merge request to add github Actions to test on other operating systems & shells, you might want to merge that and include windows in the build matrix, who knows, you might see a break since os.execvpe isn't expected to behave the same on windows as linux:

ref: https://stackoverflow.com/questions/67371606/how-to-replace-os-execvpe-on-windows-efficiently-if-the-child-process-is
https://stackoverflow.com/questions/7004687/os-exec-on-windows
python/cpython#101191

Could you clarify why installing Python is discouraged?

I mean people shouldn't be installing all sorts of things into your system python, the python that exists before the first venv is created. I have dozens of cli commands installed on my machine, if they all install to the same site-packages, then each succeeding app will uninstall the preferred versions of the previous. This is why virtual environments exist. Pipx exists to keep cli tools out of the current venv and in their own isolated venv & also out of the system python.

[alexrashed]

@matthewdeanmartin We actually do have a few windows machine, and haven't seen this report before, which makes it hard for us to act on it.
The pipeline is a very good point! The PR you referenced was just merged by @dominikschubert (#81), so it would be there as a basis to be extended with a Windows runner.
Do you think you might contribute to this open source project with a PR adding a windows runner reproducing your issue? :)

[pandrews-phillips]

Adding a +1 here for what it's worth. I also use windows 11 with git bash and get the segfault immediately. I haven't really been able to use PowerShell because it tends to interpret all my usual commands completely differently and throw strange errors unless I rewrite everything I was using successfully in git bash.

[alexrashed]

Thanks for the update, @pandrews-phillips!
It seems there might be an issue only when using git bash? Could you give us more some more information to make sure that we are targeting the right environment?

• Which exact version of Windows are you on?
• Could you post any kind of logs / screenshots / screencasts which shows how to reproduce the issue and what the output is?
• Which version of awscli-local are you using?
• Is the normal version of the aws CLI working for you?

We can then try to dig into this. I'm really curious what that would be, given that this package is really just a tiny wrapper around the official AWS CLI...
Thanks!

[pandrews-phillips]

Sure, @alexrashed .

Which exact version of Windows are you on?

Edition Windows 11 Pro
Version 23H2
Installed on ‎3/‎25/‎2024
OS build 22631.4317
Serial number PF3A5NSR
Experience Windows Feature Experience Pack 1000.22700.1041.0

Could you post any kind of logs / screenshots / screencasts which shows how to reproduce the issue and what the output is?

Unfortunately all the output I can get is:

$ awslocal
Segmentation fault

awslocal -h works and prints the usage information, anything else throws the segfault.

Which version of awscli-local are you using?

According to pip, 0.22.0

Is the normal version of the aws CLI working for you?

Yeah, the regular CLI is working fine from git bash.

Let me know if you need any other info or would like me to run something else for debugging. Thanks.

[MarcelStranak]

Hi @pandrews-phillips,

If possible, please provide us with the output of running the following commands.

awslocal --debug s3 mb s3://test
aws --debug s3 mb s3://test --endpoint-url http://localhost:4566

I would suggest looking at the similar reports on the internet. It seems like the problem is with AV or permissions blocking it.

• https://stackoverflow.com/questions/52615628/git-bash-segmentation-fault-problem-windows

• https://stackoverflow.com/questions/44569450/segmentation-fault-for-git-commit-command-windows

• Segmentation fault when using in integrated VS Code/Atom terminal git-for-windows/git#2242

• bash.exe is Segmentation fault  git-for-windows/git#1291

• https://github.com/git-for-windows/build-extra/blob/main/ReleaseNotes.md#known-issues

[matthewdeanmartin]

Antivirus is unlikely. The problem is execvpe The python standard lib docs claim it is supported on windows, but stackoverflow reports imply that execvpe is not actually cross platform.

I'll fork the library and publish a fix to pypi sometime today. It looks like in switching to subprocess you only lose interactive input from the user.

https://stackoverflow.com/questions/67371606/how-to-replace-os-execvpe-on-windows-efficiently-if-the-child-process-is

and this guy says none of the exec* functions will ever work on any windows machine except by chance

python/cpython#101191 (comment)

[alexrashed]

@matthewdeanmartin A PR contribution from a fork would definitely be great and would really help us get this over the line. 💯 We're happy to incorporate any suggested changes when we really can verify that we are tackling the issue at hand.

[pandrews-phillips]

@MarcelStranak I assume I don't need an active session or a real S3 bucket...
There's not much for the first command...

$ awslocal --debug s3 mb s3://test
Segmentation fault

And the 2nd command has real output.

$ aws --debug s3 mb s3://test --endpoint-url http://localhost:4566
2024-10-22 09:15:39,666 - MainThread - awscli.clidriver - DEBUG - CLI version: aws-cli/2.17.48 Python/3.11.9 Windows/10 exe/AMD64
2024-10-22 09:15:39,666 - MainThread - awscli.clidriver - DEBUG - Arguments entered to CLI: ['--debug', 's3', 'mb', 's3://test', '--endpoint-url', 'http://localhost:4566']

... A bunch of debug logging about loading JSON and looking for credentials and stack traces, I can copy it all in but since it worked I'm not sure the rest is relevant...

make_bucket failed: s3://test Unable to locate credentials

Skimming a few of those links that seems unlikely because in most of those cases the AV is causing all commands in git bash to segfault but in my case awslocal is the only command that errors. Git, gradle, pip, aws, python, jq, ls, act, ssh, etc, all work fine in the bash terminal.

[matthewdeanmartin]

Okay this now works on windows for git-bash, cmd.exe and powershell

https://pypi.org/project/awscli-local-win/

Recommended install

pipx install awscli-local-win

Run with

awslocalw --help

Comments

• Using a shebang and the scripts argument of setup.py/setuptools is asking for cross plat problems.
• I switch it to a module with entrypoint scripts. They just work.
• I think the .bat entry point is unnecessary when using a module with entrypoint scripts.
• Pipx is better than installing a global tool into your system python. The code depends on boto3 but is relying on localstack-client and/or awscli to provide it. Those scripts with the shebang appear to run the system python without activating any virtual environment or bypassing the current one.
• This says it supports py 2.6-3.9, but there is no tox file. I was about to create one, but there aren't any tests.
• The github action checks py 3.8-3.12. It only tests ubuntu & only bash.
• The make file depends on flake8 but it is not a declared development dependency.
• I don't know why the execvpe code replace some code that launches 2 threads to gather stdout. I lack the context to know what the developer was trying to solve, except that the Popen() solution works on windows and the other one does not.
• I switched away from setup.py because there is no native code being compiled here. setup.py should only be used when nothing else works, usually packages that need to compile native code.

Python gives people more ways to do things that are useful, reliable or cross platform. The scripts argument of setup.py is an example of that.

I appreciate the offer to create an PR, but now that I have a working solution, I'm good, thanks!

[MarcelStranak]

@pandrews-phillips,
Thanks for the details.

Does the pip version work with other terminals like Windows Terminal, CMD, PowerShell (pwsh)?
We tried to reproduce it on our Windows machines, but had no issues running the AWS CLI wrapper in any of the mentioned shells or Git Bash.

The logs you shared show that you are using AWS CLI v2. The wrapper is for AWS CLI v1. For AWS CLI v2, I would suggest setting your own custom profile to use as mentioned in https://docs.localstack.cloud/user-guide/integrations/aws-cli/#configuring-a-custom-profile.

2024-10-22 09:15:39,666 - MainThread - awscli.clidriver - DEBUG - CLI version: aws-cli/2.17.48 Python/3.11.9 Windows/10 exe/AMD64
make_bucket failed: s3://test Unable to locate credentials

You can verify the binary locations by using the which aws and which awslocal commands.

I would also propose trying out the venv in python set through Git Bash. Please have the LocalStack container running.

# Create a virtual environment in the .venv directory
python -m venv .venv
# Activate the virtual environment
source .venv/Scripts/activate
# List installed packages in the virtual environment
pip list
# Install awscli and awscli-local packages
pip install awscli awscli-local
# Create a new S3 bucket named 'test' using awscli-local with debug mode enabled
awslocal --debug s3 mb s3://test
# List installed packages again to confirm installation
pip list

After the run, it would look like this.

You should have a similar list of installed packages at the end.

Package           Version
----------------- -----------
awscli            1.35.11
awscli-local      0.22.0
boto3             1.35.45
botocore          1.35.45
colorama          0.4.6
docutils          0.16
jmespath          1.0.1
localstack-client 2.7
pip               24.2
pyasn1            0.6.1
python-dateutil   2.9.0.post0
PyYAML            6.0.2
rsa               4.7.2
s3transfer        0.10.3
six               1.16.0
urllib3           2.2.3

[pandrews-phillips]

@MarcelStranak

Yeah the system awslocal works fine in other terminals, I tried in PowerShell but I was trying to send to SQS at the time and the shell kept interpreting all the parameters and quoted strings differently, throwing other errors so it was faster to just use the regular CLI directly.

I think I'm using a Chocolatey-managed aws-cli with the pip awslocal:

$ which aws
/c/Program Files/Amazon/AWSCLIV2/aws
$ which awslocal
/c/Python312/Scripts/awslocal

Thanks, yeah running in the venv everything works as advertised:

$ awslocal --debug s3 mb s3://test
2024-10-23 09:53:25,897 - MainThread - awscli.clidriver - DEBUG - CLI version: aws-cli/1.35.12 Python/3.12.6 Windows/11 botocore/1.35.46
2024-10-23 09:53:25,897 - MainThread - awscli.clidriver - DEBUG - Arguments entered to CLI: ['--endpoint-url=http://localhost:4566', '--debug', 's3', 'mb', 's3://test']
...
make_bucket: test
(.venv)

So I suppose that points more to a problem with the calling mechanism and aws-cli v2 specifically?