Stop using ec2 runners in the conda+linux workflow #1302
Conversation
joe-iddon
force-pushed
the
no-ec2
branch
11 times, most recently
from
61631f5
to
e6d5151
Compare
joe-iddon
changed the title
| @@ -46,6 +50,7 @@ def __init__(self, mongo_uri: str, name: Optional[str] = None, client: Optional[ | |||
| self.client = client or MongoClient(mongo_uri) | |||
| if not name: | |||
| while True: | |||
| logger.log(logging.INFO, "Searching for new name") | |||
There was a problem hiding this comment.
Minor but logger.info is more idiomatic isn't it? This API with the enum is for when you calculate the log level dynamically
There was a problem hiding this comment.
Understood, I'll change it
| @@ -21,6 +22,9 @@ | |||
| if TYPE_CHECKING: | |||
| from pymongo import MongoClient | |||
|
|
|||
| logger = logging.getLogger("Mongo Storage Fixture") | |||
| logging.basicConfig(level=logging.DEBUG) | |||
There was a problem hiding this comment.
Do we want this?
There was a problem hiding this comment.
I think having informative log messages is useful, but I agree having a default call to basicConfig to actually enable them will be frustrating for other users of this server fixture.
Therefore I'd suggest to remove the basicConfig, and then for debugging the fixture users can import the log object and call basicConfig themselves if they would like to. This is how pytest-server-fixtures does it.
It's very useful for working on the CI as only the locally managed mongo server can be recreated locally, whereas the CI_MONGO_HOST one from the GitHub service would be harder to set up.
There was a problem hiding this comment.
Makes sense. How about letting the log level be set with an env var so that it's easy to turn it on for CI debugging?
There was a problem hiding this comment.
That makes sense. I've used ARCTICDB_mongo_test_fixture_loglevel as the env var.
I've also made all the fixture logs to DEBUG level, and configure the root logger to be INFO since that's the default for all our other logs.
I've manually tested that this works:
- With
INFOjust get a few logs coming through from the locally managed mongo instance, but not from the fixture:
$ ARCTICDB_mongo_test_fixture_loglevel=INFO python -m pytest -vs tests/integration/arcticdb/test_arctic.py | head -n 20
============================= test session starts ==============================
[cut output]
tests/integration/arcticdb/test_arctic.py::test_library_creation_deletion[mongo-EncodingVersion.V1] 2024-02-14T11:34:57.088+0000 I CONTROL [main] Automatically disabling TLS 1.0, to force-enable TLS 1.0 specify --sslDisabledProtocols 'none'
2024-02-14T11:34:57.090+0000 W ASIO [main] No TransportLayer configured during NetworkInterface startup
[cut output]- With
DEBUGyou can see the test fixture logs:
$ ARCTICDB_mongo_test_fixture_loglevel=DEBUG python -m pytest -vs tests/integration/arcticdb/test_arctic.py | head -n 20
============================= test session starts ==============================
[ cut output]
tests/integration/arcticdb/test_arctic.py::test_library_creation_deletion[mongo-EncodingVersion.V1] DEBUG:Mongo Storage Fixture:Env var CI_MONGO_HOST not set, so will try localhost.
DEBUG:Mongo Storage Fixture:Could not connect to localhost, so falling back to managed instance.
About to run: ['mongod', '--port', '15099', '--dbpath', '/tmp/tmp10vn8sdrManagedMongoDBServer']
2024-02-14T11:41:24.576+0000 I CONTROL [main] Automatically disabling TLS 1.0, to force-enable TLS 1.0 specify --sslDisabledProtocols 'none'
2024-02-14T11:41:24.579+0000 W ASIO [main] No TransportLayer configured during NetworkInterface startup
[cut output]
Reference Issues/PRs
What does this implement or fix?
@poodlewars
Stop using the ec2 runners for the conda build and return to
ubuntu-latest.@joe-iddon
Also improve (fix) the mongo storage test fixture to be able to fall back on a locally managed server using the
mongodb-orgpackage. This only occurs after trying theCI_MONGO_HOSTenv var, andlocalhostaddresses first, since we'd usually expect the service which is defined using:to be running on one of these. But sometimes it is not yet running, so we must have
mongodb-orginstalled locally to fallback on.Any other comments?
-nparallelism option. This is because the S3 storage fixture receives too many requests. With the usual 4 workers, it works fine. I'm just noting that here; I've not made any changes to this workflow.-voption to pytest, so if there's a failing test we can see quickly rather than having to re-run with this option.Checklist
Checklist for code changes...