Skip to content

Conversation

@ellisms
Copy link
Contributor

@ellisms ellisms commented Apr 20, 2025


Updated EMR operators and Sensors to inherit the base AWS classes.

Partially addresses #35278

@eladkal eladkal merged commit ef268cd into apache:main Apr 22, 2025
67 checks passed
@o-nikolas
Copy link
Contributor

This is now causing our example_emr_eks system test to fail:

[2025-04-22 21:45:13,663] {taskinstance.py:2457} ERROR - Task failed with exception
Traceback (most recent call last):
  File "/opt/airflow/airflow-core/src/airflow/models/taskinstance.py", line 2161, in _execute_callable
    return create_executable_runner(
  File "/opt/airflow/task-sdk/src/airflow/sdk/execution_time/callback_runner.py", line 81, in run
    return func(*args, **kwargs)
  File "/opt/airflow/task-sdk/src/airflow/sdk/bases/operator.py", line 408, in wrapper
    return func(self, *args, **kwargs)
  File "/opt/airflow/providers/amazon/src/airflow/providers/amazon/aws/sensors/emr.py", line 319, in execute
    super().execute(context=context)
  File "/opt/airflow/task-sdk/src/airflow/sdk/bases/operator.py", line 408, in wrapper
    return func(self, *args, **kwargs)
  File "/opt/airflow/task-sdk/src/airflow/sdk/bases/sensor.py", line 227, in execute
    raise e
  File "/opt/airflow/task-sdk/src/airflow/sdk/bases/sensor.py", line 207, in execute
    poke_return = self.poke(context)
  File "/opt/airflow/providers/amazon/src/airflow/providers/amazon/aws/sensors/emr.py", line 304, in poke
    state = self.hook.poll_query_status(
  File "/opt/airflow/providers/amazon/src/airflow/providers/amazon/aws/hooks/emr.py", line 516, in poll_query_status
    query_state = self.check_query_status(job_id)
  File "/opt/airflow/providers/amazon/src/airflow/providers/amazon/aws/hooks/emr.py", line 488, in check_query_status
    response = self.conn.describe_job_run(
  File "/usr/local/lib/python3.9/site-packages/botocore/client.py", line 569, in _api_call
    return self._make_api_call(operation_name, kwargs)
  File "/usr/local/lib/python3.9/site-packages/botocore/client.py", line 980, in _make_api_call
    request_dict = self._convert_to_request_dict(
  File "/usr/local/lib/python3.9/site-packages/botocore/client.py", line 1047, in _convert_to_request_dict
    request_dict = self._serializer.serialize_to_request(
  File "/usr/local/lib/python3.9/site-packages/botocore/validate.py", line 381, in serialize_to_request
    raise ParamValidationError(report=report.generate_report())
botocore.exceptions.ParamValidationError: Parameter validation failed:
Invalid type for parameter virtualClusterId, value: None, type: <class 'NoneType'>, valid types: <class 'str'>

@o-nikolas
Copy link
Contributor

Testing a fix now

prabhusneha pushed a commit to astronomer/airflow that referenced this pull request Apr 25, 2025
* EMR operators inherit AwsBaseOperator

* Update EMR sensors to inherit AwsBaseSensor

---------

Co-authored-by: mse139 <mse139@users.noereply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants