Supporting batched queue jobs for Feed Me imports #1569
Unanswered
jamesmacwhite
asked this question in
Ideas
Replies: 0 comments
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
-
It would be nice if Feed Me supported batched queue jobs, rather than just the standard queue jobs. The rationale would be for Feed Me queue jobs which can't use or the data source does not support pagination, because of this larger queue jobs may exceed the TTR. In some cases increasing the TTR is not always possible or the best move for performance.
E.g. On Craft Cloud there is a hard max execution time of 15 minutes for any queue jobs, so increasing the TTR will not be possible, this constraint may also be in other scenarios depending on the hosting environment.
Currently to ensure queue jobs stay below a maximum execution time, you have chunk the data by a defined batch size and then create a queue job per each batch, allowing each to run for the maximum allowed time, rather than as a single job, splitting the memory and time required.
It would be great if Feed Me provided a native batched job by extending craft\queue\BaseBatchedJob as an alternative to the standard queue job.
Maybe not all Feed Me queue jobs needs to be batched but perhaps controlled via the feed setting model?
Beta Was this translation helpful? Give feedback.
All reactions