Unexpected KeyError while running test suite inside GitLab Worker

[konstista]

Hello there, I'm looking for help.
Basically all of a sudden started getting unexpected KeyError while running test suite inside GitLab Worker.

It seems like no dependencies were updated, no extra code was recently added. Pipelines seem to fail even for the code that previously passed all tests. Interesting that everything works fine locally.

Error Trace

ImportError while loading conftest '.../tests/conftest.py'.
/usr/local/lib/python3.8/site-packages/ddtrace/internal/module.py:
[208](https://gitlab.com/.../jobs/...#L208): in _exec_module
    self.loader.exec_module(module)
tests/conftest.py:5: in <module>
    from service.main import app
...
...
...
service/utils.py:10: in <module>
    from jsonschema import validate
/usr/local/lib/python3.8/site-packages/ddtrace/internal/module.py:208: in _exec_module
    self.loader.exec_module(module)
/usr/local/lib/python3.8/site-packages/jsonschema/__init__.py:16: in <module>
    from jsonschema.validators import (
/usr/local/lib/python3.8/site-packages/ddtrace/internal/module.py:208: in _exec_module
    self.loader.exec_module(module)
/usr/local/lib/python3.8/site-packages/jsonschema/validators.py:589: in <module>
    meta_schema=SPECIFICATIONS.contents(
/usr/local/lib/python3.8/site-packages/referencing/_core.py:479: in contents
    return self._resources[uri.rstrip("#")].contents
E   KeyError: 'http://json-schema.org/draft-03/schema'

Error happens when importing validate() from jsonschema.
In all test cases the function is not being called, it fails on import.

Implementation

Implementation is fairly straightforward: try/except block wrapped in a function.

 try:
     validate(entity, schema)
     return True
 except Exception as e:
     return False

Dependencies

jsonschema==4.21.1
├── attrs [required: >=22.2.0, installed: 23.2.0]
├── jsonschema-specifications [required: >=2023.03.6, installed: 2023.12.1]
│   └── referencing [required: >=0.31.0, installed: 0.33.0]
│       ├── attrs [required: >=22.2.0, installed: 23.2.0]
│       └── rpds-py [required: >=0.7.0, installed: 0.18.0]
├── referencing [required: >=0.28.4, installed: 0.33.0]
│   ├── attrs [required: >=22.2.0, installed: 23.2.0]
│   └── rpds-py [required: >=0.7.0, installed: 0.18.0]
└── rpds-py [required: >=0.7.1, installed: 0.18.0]

[konstista]

The issue seems to be somewhere on my side, but I don't think something changed in the docker executor. The code itself is perfectly functional. Would be very appreciated for any help.

I can overcome the issue with moving import to function local scope. As the function is never called in the test suite - the import never happens. But I'm just curious about the behaviour.

[silvanocerza]

Same issue when running with Github runners. Locally it works without any issue.

[Julian]

There isn't enough info to be helpful I'm afraid. All passes here in CI in GitHub Actions without doing anything special. The error message points to you possibly doing something wrong to break normal installation of static (non-Python) files but hard to say what.

(Going to convert this to a discussion. If there's more info there's a small chance I can help you figure it out but no promises.)

[konstista]

Yeah, no worries and thanks for the fast response. We'll check some things on our end.

[konstista]

Okay, workaround with moving dependency to a local scope does not work. When code is deployed to AWS Lambda as docker image it fails on import with the same issue. Docker image used for deployment was updated 8 days ago, so its not clear why it started failing around 20 hours ago.

[konstista]

Ok, managed to solve the issue. The problem was in two latest images: AWS Lambda Python Image (released 9 days ago from this day) we used as deployment container and Debian Bullseye (released 1 day ago from this day) we used for running pytest tests. For both cases we could use old (outdated) versions of images and things would work.

But the better solution was to migrate from Python 3.8 to Python 3.9. Latest Python 3.9 images are fully compatible and seem to work without issues.

So @Julian it might make sense to remove Python 3.8 from a list of supported versions, since the package might not work without issues. Especially in CI environment, as normally images are not locked there.

[Julian]

It's Lambda that isn't really supported and generally has bad packaging behavior. 3.8 is definitely supported and definitely works fine in normal environments!

[konstista]

AWS Lambda container (default runtime) is launched on top of Amazon Linux 2 what it a somewhat similar of CenOS and Fedora. Its very highly likely that things would brake on latest RHEL-based distros sooner or later.

Many people pack app/services in lambda containers, some of them even don't care about runtime that are using. This approach is becoming more and more popular due to simplicity and scalability. For sure not many devs would use Python 3.8 as its end-of-life already this year, but hopefully they will find this discussion helpful.

[Julian]

I'm not telling you how to run your infrastructure or app, I'm just telling you I would estimate 90% of the packaging issues filed on this package (or in my experience often in general across the ecosystem) are caused by users using Lambda! It's Lambda that I don't support (which just means "I won't help you figure out how to get Lambda to do the right thing").

3.8 is supported until it's EOL'ed essentially, and again as I say as far as I know there are no 3.8-specific bugs or issues related to this library. If there were, I'd fix it, at least until 3.8 support is dropped.

[konstista]

Totally understandable and I get your pain. However just a friendly reminder that initial discussion was not about Lambdas but about some issues inside GitLab runners.

Anyway, thanks for your help and contribution!

[patrick91]

Getting this issue on GitHub actions as well: https://github.com/strawberry-graphql/strawberry/actions/runs/8758073573/job/24040028878?pr=3116

I might skip adding this dep on 3.8 (since it's optional), but if there's another workaround I'll be happy to use that :D