Add e2e test for remote logging #56191
Conversation
gopidesupavan
requested review from
amoghrajesh,
ashb and
kaxil
as code owners
|
Tests needs to be added, not sure its tricky part where to add is it in airflow-core or task-sdk 🤔 |
Neither, it's integration between the two |
hehe correct :) |
ashb
changed the title
gopidesupavan
requested review from
jedcunningham and
potiuk
as code owners
|
|
have added a full e2e integration test with local stack, |
|
I am not sure if you wanted me to create separate PR for testing update i am happy to do that, i have added it part of this because we are trying to fix the remote log issue, i feel its make sense to add . |
![github-advanced-security[bot]](https://avatars.githubusercontent.com/in/57789?s=60&v=4){kind=small}
gopidesupavan
force-pushed
the
fix-connection-handling
branch
2 times, most recently
from
a698cd0 to
2222c11
Compare
|
verified on real env: 
|
gopidesupavan
force-pushed
the
fix-connection-handling
branch
4 times, most recently
from
fadc410 to
89570d0
Compare
|
I think this is ready for review 2 tests are failing not related to this, am trying to look them. |
|
raised PR here for failure tests #56270 |
gopidesupavan
force-pushed
the
fix-connection-handling
branch
from
2b30a66 to
4558e68
Compare
|
Bump on this one @gopidesupavan :) If you can address the comments, we can release 3.1.1 sooner with it :) |
gopidesupavan
force-pushed
the
fix-connection-handling
branch
2 times, most recently
from
785544a to
280a597
Compare
gopidesupavan
changed the title
Thanks @gopidesupavan |
gopidesupavan
force-pushed
the
fix-connection-handling
branch
2 times, most recently
from
a202686 to
cb5c9dc
Compare
potiuk
left a comment
potiuk
left a comment
There was a problem hiding this comment.
Nice. I like the composed docker-compose file :)
potiuk
force-pushed
the
fix-connection-handling
branch
from
cb5c9dc to
92f5172
Compare
|
Rebased to account for fixed task-sdk integration tests (just in case). |
yeah :) |
|
Awesome, thanks @gopidesupavan |
* Prefetch remote log connection id for api server in order to read remote logs * fix docker compose file path * Fixup tests * Add test with mock_aws * Fixup test * Extend quick start docker with localstack * remove comment * add test connection * fix static checks * Add only e2e tests for remote logging
* Prefetch remote log connection id for api server in order to read remote logs * fix docker compose file path * Fixup tests * Add test with mock_aws * Fixup test * Extend quick start docker with localstack * remove comment * add test connection * fix static checks * Add only e2e tests for remote logging
* Prefetch remote log connection id for api server in order to read remote logs * fix docker compose file path * Fixup tests * Add test with mock_aws * Fixup test * Extend quick start docker with localstack * remove comment * add test connection * fix static checks * Add only e2e tests for remote logging
* Prefetch remote log connection id for api server in order to read remote logs * fix docker compose file path * Fixup tests * Add test with mock_aws * Fixup test * Extend quick start docker with localstack * remove comment * add test connection * fix static checks * Add only e2e tests for remote logging
4 participants
Adding Remote logging tests to E2E test.
Why:
Currently we dont have a way to test remote logging, all that we are doing so far mocking handlers and we have seen recent times remote logging broken many times.
This is to add remote logging e2e test that replicates real environment with AWS using localstack. This test is remote logging with S3. for other remote logging it can be extended later.
^ Add meaningful description above
Read the Pull Request Guidelines for more information.
In case of fundamental code changes, an Airflow Improvement Proposal (AIP) is needed.
In case of a new dependency, check compliance with the ASF 3rd Party License Policy.
In case of backwards incompatible changes please leave a note in a newsfragment file, named
{pr_number}.significant.rstor{issue_number}.significant.rst, in airflow-core/newsfragments.