Skip to content

Conversation

@nailo2c
Copy link
Contributor

@nailo2c nailo2c commented May 31, 2025

Closes: #47225

🐞 Problem

SlackAPIFileOperator looks up channel names with conversations.list (a Tier‑2 Slack API,
≈ 20 req/min). When a DAG fires many parallel tasks, these look‑ups quickly hit
HTTP 429 rate_limited errors and the tasks fail.

Reference: https://api.slack.com/apis/rate-limits

How to reproduce

from airflow import DAG
from airflow.providers.slack.operators.slack import SlackAPIFileOperator
from airflow.utils.task_group import TaskGroup

NUM_UPLOADS = 100

with DAG(
   dag_id="slack_file_upload_rl_repro",
   schedule=None,
   max_active_tasks=NUM_UPLOADS,
) as dag:
   with TaskGroup(group_id="bulk_uploads") as bulk:
       for i in range(NUM_UPLOADS):
           SlackAPIFileOperator(
               task_id=f"upload_{i}",
               slack_conn_id="slack_default",
               channels="#all-aaron-airflow-test",
               filename="/opt/airflow/RELEASE_NOTES.rst",
               initial_comment=f"File {i} 🚀"
           )

Running the DAG returns multiple SlackApiError: ratelimited exceptions.
Screenshot (before fix):
slack_api_rate_limit

✅ Fix

  • SlackHook._call_conversations_list() now retries up to 5 times when Slack responds
    with 429, respecting the Retry‑After header.
  • If the limit is still exceeded after 5 attempts, the hook raises
    AirflowException("Max retries reached for conversations.list").

This keeps the operator working without changing user DAGs.

▶️ Manual / end‑to‑end check

Verified in a local Breeze environment against a test Slack workspace:
截圖 2025-05-31 上午9 41 03

截圖 2025-05-31 上午9 41 46

Unit tests added

  • test_call_conversations_list_retries_then_succeeds
  • test_call_conversations_list_exceeds_max

Copy link
Member

@potiuk potiuk left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nice one.

@potiuk potiuk merged commit e037932 into apache:main May 31, 2025
66 checks passed
@potiuk
Copy link
Member

potiuk commented May 31, 2025

Thanks for all the screenshots etc.

Out of curiosity - was the fix and PR description AI-assisted ? I genuinly would love to know :D - nothing bad or good, just curious if it was (and with what if it was).

@nailo2c
Copy link
Contributor Author

nailo2c commented Jun 1, 2025

Yes, before I submit a PR or take over an issue, I usually discuss it with AI to ensure my understanding aligns with the issue.

I also use AI to check the code for style consistency with the community and to suggest improvements for my first draft of the PR description, making it clearer and reducing the review effort for the reviewer.

I think AI is a great tool to accelerate my efficiency.

sanederchik pushed a commit to sanederchik/airflow that referenced this pull request Jun 7, 2025
…pache#51265)

* add sleep interval in operator

* add retry when get conversations_list

* add unit test & function description
jose-lehmkuhl pushed a commit to jose-lehmkuhl/airflow that referenced this pull request Jul 11, 2025
…pache#51265)

* add sleep interval in operator

* add retry when get conversations_list

* add unit test & function description
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Rate-limited when using SlackAPIFileOperator with channel name

2 participants