Meta injection

[rodrigondec]

Resume

• related to django-celery-results doesn't update state correctly with task.update_state() #37

The issue #37 points to save data using the arg meta from the method update_state, but the meta attribute is renamed to result when it is passed to self.backend.store_result.

As explained here in this comment #37 (comment).

https://github.com/celery/celery/blob/cb4c3256326e33f005c63d48520d0abb5e898151/celery/app/task.py#L982-L994

    def update_state(self, task_id=None, state=None, meta=None, **kwargs):
        """Update task state.

        Arguments:
            task_id (str): Id of the task to update.
                Defaults to the id of the current task.
            state (str): New state.
            meta (Dict): State meta-data.
        """
        if task_id is None:
            task_id = self.request.id
        self.backend.store_result(
            task_id, meta, state, request=self.request, **kwargs)

https://github.com/celery/celery/blob/cb4c3256326e33f005c63d48520d0abb5e898151/celery/backends/base.py#L513-L522

    def store_result(self, task_id, result, state,
                     traceback=None, request=None, **kwargs):
        """Update task state and result.

        if always_retry_backend_operation is activated, in the event of a recoverable exception,
        then retry operation with an exponential backoff until a limit has been reached.
        """
        result = self.encode_result(result, state)


        retries = 0

Problem

So the argued meta arg on store_result is not really arbitrary metadata. It's actually the result of the Task.

Solution

Use the TaskResult.meta to store arbitrary data.

How to inject?

I implemented the meta injection through the Task Celery Request. This way I don't mess around with too many implementations and function signatures.

[auvipy]

can you please check the CI failures?

[rodrigondec]

can you please check the CI failures?

Yes, I was awaiting the maintainer approval for the CI to run 😁

[auvipy]

anything else you want to add or improve in this PR?

[rodrigondec]

anything else you want to add or improve in this PR?

No, for me it's good to go 🚀

[rodrigondec]

There's a roadmap with this PR on a release? There's a expected date for the new version?

[auvipy]

if possible this week, if not in this month

[cdevacc1]

@rodrigondec @auvipy I still have this problem with the latest version (2.5.1)

self.update_state(state='RUNNING', meta={'current': order_counter, 'total': total_order})

[ericswpark]

@cdevacc1 I have the same problem, see issue #367. In your screenshot though, I think it might be because RUNNING is not one of the task states defined within the library. If you look at my screenshot here, you can see PROGRESS printed next to the task ID, but the actual task state field is set to - like yours.

[cdevacc1]

@ericswpark Yes. I looked through models.py, status must be in celery.states.ALL_STATES
['FAILURE', 'PENDING', 'RECEIVED', 'RETRY', 'REVOKED', 'STARTED', 'SUCCESS']

status = models.CharField(
        max_length=50, default=states.PENDING,
        choices=TASK_STATE_CHOICES,
        verbose_name=_('Task State'),
        help_text=_('Current state of the task being run'))

ShouId I remove choices from status field to make it work with custom states?

[ericswpark]

@cdevacc1 that would be great :)

[peterbaumert]

For me the example from the "injecting-metadata" file is not working at all. No matter what i do, the information i put in request.meta or the update_state function, the info is nowhere to be found :(

"Result Data" only shows the pid and hostname. ( i dont write that in there, so must come from somewhere else ^^ )

[ericswpark]

@cdevacc1 decided to tackle this issue today so I just made a PR from your comment. See ^