ImapHook: allow limiting number of processed emails

[vaefremov95]

Description

Add an optional 'max_mails' parameter to ImapHook attachement methods (download_mail_attachments etc.) that allows limiting the number of latest emails being processed, while still retrieving all attachments
from those emails.

Use case/motivation

Problem

ImapHook currently has only two options for processing attachments from emails matching the given
mail_filter.

For example:

• the last 3 emails match the filter
• each email contains 2 attachments

Currently, users must either:

• download all 6 attachments, or
• use latest_only=True and receive only a single attachment

There is no way to get only latest email with more than 1 attachment.
There is no way to retrieve all attachments from the last N emails.
There is no way to limit the number of emails being processed other than latetest_only flag.

This becomes problematic when working with emails that contain multiple attachments.

Proposed solution

Introduce an optional max_mails parameter to ImapHook methods that retrieve
messages or attachments.

The parameter would limit the number of latest emails being processed,
while still returning all attachments from those emails.

The default value would be None, preserving the current behavior.

Conceptually, this can be implemented by modifying the internal _list_mail_ids_desc method as follows:

import itertools

def _list_mail_ids_desc(
    self,
    mail_filter: str,
    max_mails: int | None = None,
) -> Iterable[str]:
    if not self.mail_client:
        raise RuntimeError("The 'mail_client' should be initialized before!")

    _, data = self.mail_client.search(None, mail_filter)
    mail_ids = data[0].split()

    mail_ids_desc = reversed(mail_ids)

    if max_mails is not None:
        return itertools.islice(mail_ids_desc, max_mails)

    return mail_ids_desc

Of cource this parameter shoud aslo be added to attachemnt methods.

Example use case

hook.retrieve_mail_attachments(
    mail_filter="SINCE 01-Jan-2024",
    max_mails=2,
)

This would retrieve all attachments from the two most recent matching emails.

Related issues

Haven't found any

Are you willing to submit a PR?

• Yes I am willing to submit a PR!

Code of Conduct

• I agree to follow this project's Code of Conduct

[boring-cyborg]

Thanks for opening your first issue here! Be sure to follow the issue template! If you are willing to raise PR to address this issue please do so, no need to wait for approval.

[shahar1]

Welcome to Apache Airflow, feel free to raise a PR!

[sarth-akvaish]

Hi, I want to work on this, if this is not assigned

[Bucky789]

Hi, I’m Manthan. I’m preparing for GSoC and would love to work on this issue.
The proposed solution makes sense to me and I can implement it.
Could you please assign it to me?
Thanks!

[shahar1]

Hey, as the issue's OP expressed his will to open a PR - I'd kindly ask to give him the opportunity to do so.
If there's no activity within a week, then it will be assigned to the next commenter - and so on (please tag me once then to remind).
In the meanwhile, feel free to ask for assignment for other unassigned issues labeled as "good first issue", or assigned issues that have been inactive for too long.

[vaefremov95]

Hi, working on my PR. This is gonna be my first Issue/PR/contribution to airflow, so its going to take some time)

[vaefremov95]

@shahar1
Hello!
Changed my mind? no longer willing to open a PR

[SeifEddineMezned]

hi, can i work on this ?

[nishanthfx98]

Hi @shahar1,

Since the original author has mentioned they are no longer willing to open a PR, I would like to take up this issue and work on it.

My planned approach is as follows:

1. Extend the internal _list_mail_ids_desc method to accept an optional max_mails: int | None parameter, limiting the number of latest matching emails while preserving the current default behavior when None.

2. Propagate the max_mails parameter to relevant ImapHook public methods that retrieve emails or attachments (e.g., retrieve_mail_attachments, download_mail_attachments, etc.).

3. Ensure backward compatibility:

   • Default value will be None
   • Existing behavior remains unchanged when the parameter is not provided

4. Add unit tests to validate:

   • All attachments are retrieved from the last N emails
   • Behavior remains unchanged when max_mails is not set
   • Interaction with latest_only=True

5. Update docstrings and method signatures where applicable.

I’m comfortable working within the providers/imap codebase and following Airflow’s contribution and testing guidelines.

Please let me know if I can be assigned to this issue.

Thanks!