Fix/Cleanup MyPy unreachable code warnings Across Providers

[gopidesupavan]

Body

We recently bumped mypy to 1.16.1 part of this activity we have cleaned up initial mypy ignores where possible. Now we have another opportunity to fix some of unreachable warnings flagged by mypy.

Some notes before attempting to fix following:

1. Some of them are fixed by changing type definition
2. Few are might be unnecessary conditions can be removed.

Note: When addressing these issues, please thoroughly understand the codebase and potential implications of your changes to avoid unintended side effects.

Fix only what is possible from the following list and rest you can ignore.

---

Amazon @gopidesupavan

#53414

• providers/amazon/src/airflow/providers/amazon/aws/secrets/systems_manager.py:94: error: Statement is unreachable [unreachable]
  self.connections_prefix = connections_prefix
• providers/amazon/src/airflow/providers/amazon/aws/secrets/systems_manager.py:98: error: Statement is unreachable [unreachable]
  self.variables_prefix = variables_prefix
• providers/amazon/src/airflow/providers/amazon/aws/secrets/systems_manager.py:102: error: Statement is unreachable [unreachable]
  self.config_prefix = config_prefix
• providers/amazon/src/airflow/providers/amazon/aws/secrets/systems_manager.py:142: error: Statement is unreachable [unreachable]
  return None
• providers/amazon/src/airflow/providers/amazon/aws/secrets/systems_manager.py:154: error: Statement is unreachable [unreachable]
  return None
• providers/amazon/src/airflow/providers/amazon/aws/secrets/systems_manager.py:166: error: Statement is unreachable [unreachable]
  return None
• providers/amazon/src/airflow/providers/amazon/aws/secrets/secrets_manager.py:208: error: Statement is unreachable [unreachable]
  return None
• providers/amazon/src/airflow/providers/amazon/aws/secrets/secrets_manager.py:237: error: Statement is unreachable [unreachable]
  return None
• providers/amazon/src/airflow/providers/amazon/aws/secrets/secrets_manager.py:249: error: Statement is unreachable [unreachable]
  return None
• providers/amazon/src/airflow/providers/amazon/aws/hooks/eks.py:534: error: Statement is unreachable [unreachable]
  self.log.info("Retrieved list of %s %s.", len(name_collection)...
• providers/amazon/src/airflow/providers/amazon/aws/executors/aws_lambda/lambda_executor.py:234: error: Subclass of "str" and "ExecuteTask" cannot exist: would have incompatible method signatures [unreachable]
  if isinstance(command[0], ExecuteTask):
• providers/amazon/src/airflow/providers/amazon/aws/executors/aws_lambda/lambda_executor.py:235: error: Statement is unreachable [unreachable]
  workload = command[0]
• providers/amazon/src/airflow/providers/amazon/aws/executors/ecs/ecs_executor.py:496: error: Subclass of "str" and "ExecuteTask" cannot exist: would have incompatible method signatures [unreachable]
  if isinstance(command[0], ExecuteTask):
• providers/amazon/src/airflow/providers/amazon/aws/executors/ecs/ecs_executor.py:497: error: Statement is unreachable [unreachable]
  workload = command[0]
• providers/amazon/src/airflow/providers/amazon/aws/executors/batch/batch_executor.py:382: error: Subclass of "str" and "ExecuteTask" cannot exist: would have incompatible method signatures [unreachable]
  if isinstance(command[0], ExecuteTask):
• providers/amazon/src/airflow/providers/amazon/aws/executors/batch/batch_executor.py:383: error: Statement is unreachable [unreachable]
  workload = command[0]
• providers/amazon/src/airflow/providers/amazon/aws/transfers/sql_to_s3.py:167: error: Statement is unreachable [unreachable]
  df[col] = df[col].astype(str)
• providers/amazon/src/airflow/providers/amazon/aws/operators/athena.py:242: error: Subclass of "str" and "list[Any]" cannot exist: would have incompatible method signatures [unreachable]
  totalTasks=len(self.query) if isinstance(self.query, list) els...
• providers/amazon/src/airflow/providers/amazon/aws/operators/sagemaker.py:92: error: Subclass of "dict[Any, Any]" and "list[Any]" cannot exist: would have incompatible method signatures [unreachable]
  if isinstance(config, list):
• providers/amazon/src/airflow/providers/amazon/aws/operators/sagemaker.py:93: error: Statement is unreachable [unreachable]
  for sub_config in config:
• providers/amazon/src/airflow/providers/amazon/aws/operators/sagemaker.py:100: error: Subclass of "dict[Any, Any]" and "list[Any]" cannot exist: would have incompatible method signatures [unreachable]
  if isinstance(config, list):
• providers/amazon/src/airflow/providers/amazon/aws/operators/sagemaker.py:101: error: Statement is unreachable [unreachable]
  for sub_config in config:
• providers/amazon/src/airflow/providers/amazon/aws/operators/sagemaker.py:1012: error: Statement is unreachable [unreachable]
  description = {} # never executed but makes static checke...

Apache Beam @kyungjunleeme

• providers/apache/beam/src/airflow/providers/apache/beam/hooks/beam.py:186: error: Statement is unreachable [unreachable]
  log.info("Waiting for Apache Beam process to complete.")

Apache Cassandra @kyungjunleeme

• providers/apache/cassandra/src/airflow/providers/apache/cassandra/hooks/cassandra.py:127: error: Right operand of "and" is never evaluated [unreachable]
  if self.session and not self.session.is_shutdown:
• providers/apache/cassandra/src/airflow/providers/apache/cassandra/hooks/cassandra.py:128: error: Statement is unreachable [unreachable]
  return self.session

Apache Hive @kyungjunleeme

• providers/apache/hive/src/airflow/providers/apache/hive/sensors/named_hive_partition.py:59: error: Subclass of "list[str]" and "str" cannot exist: would have incompatible method signatures [unreachable]
  if isinstance(partition_names, str):

Apache Kafka @kyungjunleeme

• providers/apache/kafka/src/airflow/providers/apache/kafka/sensors/kafka.py:198: error: Statement is unreachable [unreachable]
  return event

Apache Pig @kyungjunleeme

• providers/apache/pig/src/airflow/providers/apache/pig/hooks/pig.py:106: error: Statement is unreachable [unreachable]
  if self.sub_process.poll() is None:

Atlassian @sunank200

• providers/atlassian/jira/src/airflow/providers/atlassian/jira/operators/jira.py:72: error: Statement is unreachable [unreachable]
  resource = self.get_jira_resource_method.execute(**con...

CNCF @sunank200

• providers/cncf/kubernetes/src/airflow/providers/cncf/kubernetes/executors/kubernetes_executor_utils.py:393: error: Subclass of "str" and "ExecuteTask" cannot exist: would have incompatible method signatures [unreachable]
  if isinstance(command[0], ExecuteTask):
• providers/cncf/kubernetes/src/airflow/providers/cncf/kubernetes/executors/kubernetes_executor_utils.py:394: error: Statement is unreachable [unreachable]
  workload = command[0]
• providers/cncf/kubernetes/src/airflow/providers/cncf/kubernetes/hooks/kubernetes.py:295: error: Statement is unreachable [unreachable]
  return self._is_in_cluster

Common compat

• providers/common/compat/src/airflow/providers/common/compat/openlineage/check.py:56: error: Right operand of "and" is never evaluated [unreachable]
  if callable(provider_min_version) and client_min_version is None:

Common SQL @sunank200

• providers/common/sql/src/airflow/providers/common/sql/sensors/sql.py:116: error: Statement is unreachable [unreachable]
  message = f"self.failure is present, but not callable ...
• providers/common/sql/src/airflow/providers/common/sql/sensors/sql.py:122: error: Statement is unreachable [unreachable]
  message = f"self.success is present, but not callable -> {...
• providers/common/sql/src/airflow/providers/common/sql/operators/generic_transfer.py:140: error: Statement is unreachable [unreachable]
  self.page_size = int(self.page_size)

Databricks @sunank200

• providers/databricks/src/airflow/providers/databricks/sensors/databricks_partition.py:205: error: Subclass of "str" and "list[Any]" cannot exist: would have incompatible method signatures [unreachable]
  if isinstance(partition_value, list):
• providers/databricks/src/airflow/providers/databricks/sensors/databricks_partition.py:206: error: Statement is unreachable [unreachable]
  output_list.append(f"""{partition_col} in {tup...
• providers/databricks/src/airflow/providers/databricks/sensors/databricks_partition.py:208: error: Subclass of "str" and "int" cannot exist: would have incompatible method signatures [unreachable]
  if isinstance(partition_value, (int, float, comple...
• providers/databricks/src/airflow/providers/databricks/sensors/databricks_partition.py:208: error: Subclass of "str" and "float" cannot exist: would have incompatible method signatures [unreachable]
  if isinstance(partition_value, (int, float, comple...
• providers/databricks/src/airflow/providers/databricks/sensors/databricks_partition.py:208: error: Subclass of "str" and "complex" cannot exist: would have incompatible method signatures [unreachable]
  if isinstance(partition_value, (int, float, comple...
• providers/databricks/src/airflow/providers/databricks/sensors/databricks_partition.py:209: error: Statement is unreachable [unreachable]
  output_list.append(

Docker @sunank200

• providers/docker/src/airflow/providers/docker/operators/docker.py:522: error: Statement is unreachable [unreachable]
  self.log.info("Not attempting to kill container as it ...
• providers/docker/src/airflow/providers/docker/operators/docker_swarm.py:204: error: Statement is unreachable [unreachable]
  self.log.info("Service started: %s", self.service)
• providers/docker/src/airflow/providers/docker/operators/docker_swarm.py:241: error: Statement is unreachable [unreachable]
  return self.cli.tasks(filters={"service": self.service["ID"]})...
• providers/docker/src/airflow/providers/docker/operators/docker_swarm.py:250: error: Statement is unreachable [unreachable]
  last_line_logged, last_timestamp = "", 0
• providers/docker/src/airflow/providers/docker/operators/docker_swarm.py:320: error: Statement is unreachable [unreachable]
  self.log.info("Removing docker service: %s", self.service[...

Edge3 @jscheffl

#53430

• providers/edge3/src/airflow/providers/edge3/cli/worker.py:193: error: Statement is unreachable [unreachable]
  parsed = urlparse(api_url)

Elasticsearch

• providers/elasticsearch/src/airflow/providers/elasticsearch/log/es_task_handler.py:511: error: Statement is unreachable [unreachable]
  self.closed = True

FAB @gopidesupavan

#53436

• providers/fab/src/airflow/providers/fab/auth_manager/security_manager/override.py:1570: error: Statement is unreachable [unreachable]
  try:

Google

• providers/google/src/airflow/providers/google/cloud/utils/bigquery.py:29: error: Statement is unreachable [unreachable]
  return None
• providers/google/src/airflow/providers/google/cloud/hooks/vertex_ai/ray.py:62: error: Statement is unreachable [unreachable]
  if isinstance(value, ScalarMapContainer):
• providers/google/src/airflow/providers/google/cloud/hooks/tasks.py:168: error: Statement is unreachable [unreachable]
  elif isinstance(task_queue, dict):
• providers/google/src/airflow/providers/google/cloud/hooks/bigquery.py:220: error: Subclass of "str" and "dict[Any, Any]" cannot exist: would have incompatible method signatures [unreachable]
  keyfile_content = keyfile_dict if isinstance(keyfile_dict, dict) e...
• providers/google/src/airflow/providers/google/cloud/hooks/cloud_sql.py:600: error: Subclass of "str" and "dict[Any, Any]" cannot exist: would have incompatible method signatures [unreachable]
  keyfile_content = keyfile_dict if isinstance(keyfile_dict, dict) e...
• providers/google/src/airflow/providers/google/cloud/secrets/secret_manager.py:159: error: Statement is unreachable [unreachable]
  return None
• providers/google/src/airflow/providers/google/cloud/secrets/secret_manager.py:171: error: Statement is unreachable [unreachable]
  return None
• providers/google/src/airflow/providers/google/cloud/secrets/secret_manager.py:183: error: Statement is unreachable [unreachable]
  return None
• providers/google/src/airflow/providers/google/cloud/transfers/facebook_ads_to_gcs.py:149: error: Statement is unreachable [unreachable]
  message = (
• providers/google/src/airflow/providers/google/cloud/transfers/cassandra_to_gcs.py:272: error: Statement is unreachable [unreachable]
  if isinstance(value, Decimal):
• providers/google/src/airflow/providers/google/cloud/triggers/gcs.py:198: error: Statement is unreachable [unreachable]
  res = {
• providers/google/src/airflow/providers/google/cloud/operators/vertex_ai/hyperparameter_tuning_job.py:275: error: Statement is unreachable [unreachable]
  return
• providers/google/src/airflow/providers/google/cloud/operators/cloud_composer.py:755: error: Statement is unreachable [unreachable]
  return
• providers/google/src/airflow/providers/google/cloud/operators/dataproc.py:1400: error: Statement is unreachable [unreachable]
  self.hook.cancel_job(project_id=self.project_id, job_id=se...
• providers/google/src/airflow/providers/google/cloud/operators/gcs.py:793: error: Statement is unreachable [unreachable]
  orig_start = pendulum.instance(orig_start)
• providers/google/src/airflow/providers/google/cloud/operators/gcs.py:796: error: Statement is unreachable [unreachable]
  orig_end = pendulum.instance(orig_end)
• providers/google/src/airflow/providers/google/cloud/operators/bigquery.py:281: error: Statement is unreachable [unreachable]
  self.project_id = hook.project_id
• providers/google/src/airflow/providers/google/cloud/operators/bigquery.py:436: error: Statement is unreachable [unreachable]
  self.project_id = hook.project_id
• providers/google/src/airflow/providers/google/cloud/operators/bigquery.py:603: error: Statement is unreachable [unreachable]
  self.project_id = hook.project_id
• providers/google/src/airflow/providers/google/cloud/operators/bigquery.py:2319: error: Subclass of "dict[str, Any]" and "str" cannot exist: would have incompatible method signatures [unreachable]
  if isinstance(self.configuration, str) and self.configuration....
• providers/google/src/airflow/providers/google/cloud/operators/bigquery.py:2319: error: Right operand of "and" is never evaluated [unreachable]
  if isinstance(self.configuration, str) and self.configuration.en...
• providers/google/src/airflow/providers/google/cloud/operators/bigquery.py:2320: error: Statement is unreachable [unreachable]
  with open(self.configuration) as file:
• providers/google/src/airflow/providers/google/cloud/operators/bigquery.py:2371: error: Statement is unreachable [unreachable]
  self.project_id = hook.project_id
• providers/google/src/airflow/providers/google/cloud/sensors/gcs.py:473: error: Statement is unreachable [unreachable]
  self.inactivity_seconds = (get_time() - self.last_activity...
• providers/google/tests/system/google/cloud/cloud_functions/example_functions.py:70: error: Statement is unreachable [unreachable]
  body["sourceRepository"] = {"url": SOURCE_REPOSITORY}
• providers/google/src/airflow/providers/google/marketing_platform/operators/campaign_manager.py:298: error: Subclass of "dict[str, Any]" and "str" cannot exist: would have incompatible method signatures [unreachable]
  if isinstance(self.report, str) and self.report.endswith(".jso...
• providers/google/src/airflow/providers/google/marketing_platform/operators/campaign_manager.py:298: error: Right operand of "and" is never evaluated [unreachable]
  if isinstance(self.report, str) and self.report.endswith(".jso...
• providers/google/src/airflow/providers/google/marketing_platform/operators/campaign_manager.py:299: error: Statement is unreachable [unreachable]
  with open(self.report) as file:
• providers/google/src/airflow/providers/google/marketing_platform/operators/display_video.py:98: error: Subclass of "dict[str, Any]" and "str" cannot exist: would have incompatible method signatures [unreachable]
  if isinstance(self.body, str) and self.body.endswith(".json"):
• providers/google/src/airflow/providers/google/marketing_platform/operators/display_video.py:98: error: Right operand of "and" is never evaluated [unreachable]
  if isinstance(self.body, str) and self.body.endswith(".json"):
• providers/google/src/airflow/providers/google/marketing_platform/operators/display_video.py:99: error: Statement is unreachable [unreachable]
  with open(self.body) as file:

HashiCorp Vault

• providers/hashicorp/src/airflow/providers/hashicorp/secrets/vault.py:129: error: Statement is unreachable [unreachable]
  self.connections_path = connections_path
• providers/hashicorp/src/airflow/providers/hashicorp/secrets/vault.py:133: error: Statement is unreachable [unreachable]
  self.variables_path = variables_path
• providers/hashicorp/src/airflow/providers/hashicorp/secrets/vault.py:137: error: Statement is unreachable [unreachable]
  self.config_path = config_path

JDBC @gopidesupavan

• providers/jdbc/src/airflow/providers/jdbc/hooks/jdbc.py:227: error: Statement is unreachable [unreachable]
  return False

Microsoft @fweilun

• providers/microsoft/azure/src/airflow/providers/microsoft/azure/secrets/key_vault.py:106: error: Statement is unreachable [unreachable]
  self.connections_prefix = connections_prefix
• providers/microsoft/azure/src/airflow/providers/microsoft/azure/secrets/key_vault.py:110: error: Statement is unreachable [unreachable]
  self.variables_prefix = variables_prefix
• providers/microsoft/azure/src/airflow/providers/microsoft/azure/secrets/key_vault.py:114: error: Statement is unreachable [unreachable]
  self.config_prefix = config_prefix
• providers/microsoft/azure/src/airflow/providers/microsoft/azure/secrets/key_vault.py:151: error: Statement is unreachable [unreachable]
  return None
• providers/microsoft/azure/src/airflow/providers/microsoft/azure/secrets/key_vault.py:163: error: Statement is unreachable [unreachable]
  return None
• providers/microsoft/azure/src/airflow/providers/microsoft/azure/secrets/key_vault.py:175: error: Statement is unreachable [unreachable]
  return None
• providers/microsoft/azure/src/airflow/providers/microsoft/azure/hooks/wasb.py:599: error: Statement is unreachable [unreachable]
  conn = await sync_to_async(self.get_connection)(self.conn_id)
• providers/microsoft/azure/src/airflow/providers/microsoft/azure/hooks/msgraph.py:463: error: Statement is unreachable [unreachable]
  request_information.content = data
• providers/microsoft/azure/src/airflow/providers/microsoft/azure/operators/synapse.py:129: error: Statement is unreachable [unreachable]
  self.hook.cancel_job_run(
• providers/microsoft/azure/src/airflow/providers/microsoft/azure/operators/msgraph.py:260: error: Statement is unreachable [unreachable]
  if append_result_as_list_if_absent:

OpenAI @mingdaoy

• providers/openai/src/airflow/providers/openai/hooks/openai.py:469: error: Statement is unreachable [unreachable]
  return
• providers/openai/src/airflow/providers/openai/triggers/openai.py:73: error: Statement is unreachable [unreachable]
  yield TriggerEvent(
• providers/openai/src/airflow/providers/openai/triggers/openai.py:89: error: Statement is unreachable [unreachable]
  yield TriggerEvent(
• providers/openai/src/airflow/providers/openai/triggers/openai.py:97: error: Statement is unreachable [unreachable]
  yield TriggerEvent(

OpenLineage @mingdaoy

• providers/openlineage/src/airflow/providers/openlineage/utils/selective_enable.py:52: error: Statement is unreachable [unreachable]
  enable_lineage(obj.operator)
• providers/openlineage/src/airflow/providers/openlineage/utils/selective_enable.py:69: error: Statement is unreachable [unreachable]
  disable_lineage(obj.operator)

OpenSearch @mingdaoy

• providers/opensearch/src/airflow/providers/opensearch/log/os_task_handler.py:253: error: Statement is unreachable [unreachable]
  self.closed = True

SFTP @jroachgolf84

• providers/sftp/src/airflow/providers/sftp/hooks/sftp.py:291: error: Subclass of "str" and "BytesIO" cannot exist: would have incompatible method signatures [unreachable]
  if isinstance(local_full_path, BytesIO):
• providers/sftp/src/airflow/providers/sftp/hooks/sftp.py:293: error: Statement is unreachable [unreachable]
  self.log.info("Using streaming download for %s", remot...
• providers/sftp/src/airflow/providers/sftp/hooks/sftp.py:323: error: Subclass of "str" and "BytesIO" cannot exist: would have incompatible method signatures [unreachable]
  if isinstance(local_full_path, BytesIO):
• providers/sftp/src/airflow/providers/sftp/hooks/sftp.py:324: error: Statement is unreachable [unreachable]
  conn.putfo(local_full_path, remote_full_path, confirm=...

Singularity

• providers/singularity/src/airflow/providers/singularity/operators/singularity.py:176: error: Statement is unreachable [unreachable]
  self.log.info("Stopping Singularity instance")

Snowflake @jroachgolf84

• providers/snowflake/src/airflow/providers/snowflake/utils/sql_api_generate_jwt.py:150: error: Subclass of "str" and "bytes" cannot exist: would have incompatible method signatures [unreachable]
  if isinstance(token, bytes):
• providers/snowflake/src/airflow/providers/snowflake/utils/sql_api_generate_jwt.py:151: error: Statement is unreachable [unreachable]
  token = token.decode("utf-8")

SQLite @jscheffl

#53433

• providers/sqlite/src/airflow/providers/sqlite/hooks/sqlite.py:51: error: Statement is unreachable [unreachable]
  airflow_conn.conn_type = self.conn_type

Standard @jscheffl

#53431

• providers/standard/src/airflow/providers/standard/utils/python_virtualenv.py:71: error: Statement is unreachable [unreachable]
  python_bin = sys.executable
• providers/standard/src/airflow/providers/standard/operators/branch.py:60: error: Statement is unreachable [unreachable]
  return
• providers/standard/src/airflow/providers/standard/sensors/date_time.py:102: error: Subclass of "str" and "datetime" cannot exist: would have incompatible method signatures [unreachable]
  if isinstance(self.target_time, datetime.datetime):
• providers/standard/src/airflow/providers/standard/sensors/date_time.py:103: error: Statement is unreachable [unreachable]
  return self.target_time

Teradata @fweilun

• providers/teradata/src/airflow/providers/teradata/triggers/teradata_compute_cluster.py:154: error: Subclass of "tuple[Any, ...]" and "str" cannot exist: would have incompatible method signatures [unreachable]
  if isinstance(result_set, list) and isinstance(result_set[0], str):
• providers/teradata/src/airflow/providers/teradata/triggers/teradata_compute_cluster.py:154: error: Subclass of "list[tuple[Any, ...]]" and "str" cannot exist: would have incompatible method signatures [unreachable]
  if isinstance(result_set, list) and isinstance(result_set[0], str):
• providers/teradata/src/airflow/providers/teradata/triggers/teradata_compute_cluster.py:155: error: Statement is unreachable [unreachable]
  status = str(result_set[0])

Yandex

• providers/yandex/src/airflow/providers/yandex/operators/dataproc.py:149: error: Statement is unreachable [unreachable]
  services = []

--

Follow the below steps

1. Build lates ci image with breeze ci-image build --python 3.10
2. export SHOW_UNUSED_MYPY_WARNINGS=true
3. export SHOW_UNREACHABLE_MYPY_WARNINGS=true
4. Run the ./scripts/ci/pre_commit/mypy_folder.py {Provider Name}

If the execution does not provide expected results or if you want to be fully certain, also run breeze down --cleanup-mypy-cache as caching of mypy influences results.

Committer

• I acknowledge that I am a maintainer/committer of the Apache Airflow project.

[kyungjunleeme]

I'd like to take on the Apache part and work on it. could you assign me? (if you ready....)

[gopidesupavan]

I'd like to take on the Apache part and work on it. could you assign me? (if you ready....)

Thanks @kyungjunleeme sure noted, will send out devlist after this merge #53398. then you can build image and follow the instructions

[gopidesupavan]

also adding example reference PR AWS provider. will update that here.

[sunank200]

I would like to work on Atlassian, CNCF, Common SQL, Databricks and Docker

[gopidesupavan]

I would like to work on Atlassian, CNCF, Common SQL, Databricks and Docker

thanks :) done

[jroachgolf84]

Can I grab SFTP and Snowflake?

[jroachgolf84]

I've gone ahead and followed the steps in this issue to run the ./scripts/ci/pre_commit/mypy_folder.py script for both the providers/sftp and providers/snowflake providers. However, both comeback with Success: no issues found in 46 source files. Is this to be expected?

cc: @jscheffl