Product export gets "Message is added to queue, wait to get your file soon"

[mesnitu]

Just updated magento from 2.3.1 to 2.3.2, all looked fine until I wanted to export a csv file for products, etc... Is this a new featured, or how can I simple download / export the products as it was doing a couple hours ago?
The only thing I get is "Message is added to queue, wait to get your file soon".

Preconditions (*)

1. version 2.3.2

Steps to reproduce (*)

1. update from 2.3.1 to 2.3.2
2. try to export csv files (products, etc)

Expected result (*)

1. Get the files

Actual result (*)

1. "Message is added to queue, wait to get your file soon"
2. a temporary file is created at var/ , like importexport_5d14dcb332d84
3. No error log
4. No job is added to the cron_schedule table

[m2-assistant]

Hi @mesnitu. Thank you for your report.
To help us process this issue please make sure that you provided the following information:

• Summary of the issue
• Information on your environment
• Steps to reproduce
• Expected and actual results

Please make sure that the issue is reproducible on the vanilla Magento instance following Steps to reproduce. To deploy vanilla Magento instance on our environment, please, add a comment to the issue:

@magento give me 2.3-develop instance - upcoming 2.3.x release

For more details, please, review the Magento Contributor Assistant documentation.

@mesnitu do you confirm that you were able to reproduce the issue on vanilla Magento instance following steps to reproduce?

• yes
• no

---

• Join Magento Community Engineering Slack and ask your questions in #github channel.

[m2-assistant]

Hi @engcom-Bravo. Thank you for working on this issue.
In order to make sure that issue has enough information and ready for development, please read and check the following instruction: 👇

• 1. Verify that issue has all the required information. (Preconditions, Steps to reproduce, Expected result, Actual result).

  Details

  If the issue has a valid description, the label Issue: Format is valid will be added to the issue automatically. Please, edit issue description if needed, until label Issue: Format is valid appears.

• 2. Verify that issue has a meaningful description and provides enough information to reproduce the issue. If the report is valid, add Issue: Clear Description label to the issue by yourself.

• 3. Add Component: XXXXX label(s) to the ticket, indicating the components it may be related to.

• 4. Verify that the issue is reproducible on 2.3-develop branch

  Details

  - Add the comment @magento give me 2.3-develop instance to deploy test instance on Magento infrastructure.
  - If the issue is reproducible on 2.3-develop branch, please, add the label Reproduced on 2.3.x.
  - If the issue is not reproducible, add your comment that issue is not reproducible and close the issue and stop verification process here!

• 5. Verify that the issue is reproducible on 2.2-develop branch.

  Details

  - Add the comment @magento give me 2.2-develop instance to deploy test instance on Magento infrastructure.
  - If the issue is reproducible on 2.2-develop branch, please add the label Reproduced on 2.2.x

• 6. Add label Issue: Confirmed once verification is complete.

• 7. Make sure that automatic system confirms that report has been added to the backlog.

[engcom-Bravo]

Hello @mesnitu. Thanks for applying. To get those files now You need Your cron jobs be running. Will You try please to execute php bin/magento cron:run.

[mesnitu]

ok, I'm still waiting for the cron to end ( 5 minutes now...) , so, meanwhile, how can I disable this? I didn't do anything, except the update. This csv export, is a operation that expects a immediate conclusion to get the work done.

[mesnitu]

Nothing.
If I clear the cron_schedule table, and try to export, I get the same message, but there's no schedule job or id created at that table. Not sure how this works...

[agbdxb]

I had the same issue.

1. Got a sample csv for import >>System>>Import>>select Products>>click "Download Sample File",
2. Created 10 records of data for my products and imported successfully >> System >> Import
3. Triggered the cron run on command line
4. Guess what! that triggered all the exports of my past multiple attempts which had no movements despite several cron runs on command line. I could successfully download the exported file on System >> Export panel

Hope this helps.

[mesnitu]

Still no luck here. I'm on windows 10.
I'm doing a upgrade from 1.9.2 to 2.3.1 , all went more or less ok. I've working on 2.3.1 for about a month. So I saw the update msg, and I thought from 2.3.1 to 3.2 it's a piece of cake.
@agbdxb following your first step I found a error realated to the update:

1 exception(s):
Exception #0 (Magento\Framework\Exception\LocalizedException): Please upgrade your database: Run "bin/magento setup:upgrade" from the Magento root directory.
The following modules are outdated:
Magento_InventoryExportStockApi schema: current version - none, required version - 1.0.0
Magento_InventoryExportStockApi data: current version - none, required version - 1.0.0
Magento_InventoryReservationCli schema: current version - none, required version - 1.0.0
Magento_InventoryReservationCli data: current version - none, required version - 1.0.0
Magento_InventoryExportStock schema: current version - none, required version - 1.0.0
Magento_InventoryExportStock data: current version - none, required version - 1.0.0

So I did the setup:upgrade
But I get no results doing the next steps. ( except the I can actually EXPORT the sample_data file )... funny, right?
So I looked up how to setup a cron schedule on windows, etc ( That on 2.3.1 had no need to do this stuff )
Now I have a cmd popup every 5 minutes, and the cron_schedule table keeps growing. I have 1208 entries and counting.

A lot of missed jobs are created : bulk_cleanup, sales_grid_order_shipment_async_insert, sales_send_order_shipment_emails.
And the table keeps growing.

But i guess the bottom line, is that this was not occurring in 2.3.1 and I don't know why is happening now and how to fix this.

[mesnitu]

Reverting to 2.3.1 there's no such issue.

[NightmareGrl]

Same issue. Any solution yet?

[engcom-Bravo]

Hi again @mesnitu. Here are our steps after vanilla instance installation of 2.3-develop version:

1. Installation + cron:run
   
2. Product created
   
3. Export
   

[mesnitu]

Hi
My solution for now was to revert to 2.3.1 where this issue doesn't occur.
I guess ( really guessing ) that this is related to the cron tasks on windows. If I do that cron:run all I get is that cron_table populate heavily every time a job is fired. It will never end. I'll tested later if I can update to 2.3.2 on a virtual machine or something.

[engcom-Bravo]

@mesnitu, please take a not of this Magento official page, in particular the point about Operating Systems. As You can see Magento supports only Linux. Taking into consideration all aforementioned in this ticket we are supposed to close it. But please feel free to reopen it if You find a way to reproduce it on a vanilla Magento instance running on officially supported server environment. (By the way we also tried to reproduce Your issue on 2.3.2 version, the one You reported about, but trhe result was the same - can not reproduce). Thanks for applying.

[gaurangpadhiyar007]

Reopen this issue.
Not an issue with Linux or operating system.

Reopen issue.
Still present in magento 2.3.2

[gaurangpadhiyar007]

Temporary solution for products export:

public function execute()
{
    if ($this->getRequest()->getPost(ExportModel::FILTER_ELEMENT_GROUP)) {
        try {
            $params = $this->getRequest()->getParams();
            $model = $this->_objectManager->create(\Magento\ImportExport\Model\Export::class);
            $model->setData($this->getRequest()->getParams());
            $this->sessionManager->writeClose();

            return $this->fileFactory->create(
                $model->getFileName(),
                $model->export(),
                \Magento\Framework\App\Filesystem\DirectoryList::VAR_DIR,
                $model->getContentType()
            );

            /** @var ExportInfoFactory $dataObject */
            $dataObject = $this->exportInfoFactory->create(
                $params['file_format'],
                $params['entity'],
                $params['export_filter']
            );

            $this->messagePublisher->publish('import_export.export', $dataObject);
            $this->messageManager->addSuccessMessage(
                __('Message is added to queue, wait to get your file soon')
            );
        } catch (\Exception $e) {
            $this->_objectManager->get(\Psr\Log\LoggerInterface::class)->critical($e);
            $this->messageManager->addError(__('Please correct the data sent value.'));
        }
    } else {
        $this->messageManager->addError(__('Please correct the data sent value.'));
    }
    /** @var \Magento\Backend\Model\View\Result\Redirect $resultRedirect */
    $resultRedirect = $this->resultFactory->create(ResultFactory::TYPE_REDIRECT);
    $resultRedirect->setPath('adminhtml/*/index');
    return $resultRedirect;
}

[jenn0pal]

Hello @mesnitu. Thanks for applying. To get those files now You need Your cron jobs be running. Will You try please to execute php bin/magento cron:run.

I am also experiencing this issue. I also tried manually running the cron task.
Still no job is created

[battamich]

Clean mag 2.3.2 cloud installation (using docker locally) using centos 7
I Installed and run cron jobs, created the product and tried the export but no file has been created for me to download.
I can spot files inside var folder such as 'importexport_{id}' but they are all empty

[hostep]

Can everybody who runs against this, check if the message queue consumer processes are running?
So it works like this:

• the cron searches for running consumer processes
• if it finds a running process, it won't do anything
• if it doesn't find a running process, it should spawn one
• a running consumer process is sitting there and checking every second or so if a new message was added to the queue
• when it finds a new message in the queue, it will start processing it

You can find such processes with a command like:
ps aux | grep [c]onsumer

It should output something like this on a vanilla 2.3.2 installation:

someuser 123  0.0  2.1 530072 175892 ?       S    Aug09   0:33 php /path/to/bin/magento queue:consumers:start product_action_attribute.update --pid-file-path=product_action_attribute.update-some-hostname.pid --max-messages=10000
someuser 124  0.0  2.1 530068 176016 ?       S    Aug09   0:33 php /path/to/bin/magento queue:consumers:start product_action_attribute.website.update --pid-file-path=product_action_attribute.website.update-some-hostname.pid --max-messages=10000
someuser 125  0.0  2.1 530060 175952 ?       S    Aug09   0:33 php /path/to/bin/magento queue:consumers:start exportProcessor --pid-file-path=exportProcessor-some-hostname.pid --max-messages=10000
someuser 126  0.0  2.1 530048 175840 ?       S    Aug09   0:33 php /path/to/bin/magento queue:consumers:start codegeneratorProcessor --pid-file-path=codegeneratorProcessor-some-hostname.pid --max-messages=10000

If you can not find running consumer processes even though the cron is running. Then for some reason those process aren't getting spawned.
I would then start to debug the Magento\MessageQueue\Model\Cron\ConsumersRunner::run method. To see why they aren't getting spawned.

If you think you spotted a bug after debugging, then please let us know!

Alternatively you can also manually execute the bin/magento queue:consumers:start command but first look at the help section and see how this works so you understand the different options you can pass it. Also devdocs has some info on these commands.
This can also be automated using supervisor for example instead of using the cron, here's the documention around that.

[bruce-gemberling]

I followed HOSTEP's advice and solved my queue problems

[himanshu8dev]

I am facing the same issue can anyone provide the solution for magento2.3.2

[battamich]

Looking at my cron_schedule table I see that the crons are running, even the consumers_runner with success status.

Alternatively you can also manually execute the bin/magento queue:consumers:start command but first look at the help section and see how this works so you understand the different options you can pass it. Also devdocs has some info on these commands.

I tried to manually execute the command as @hostep suggested:

php bin/magento queue:consumers:start exportProcessor

and the files were generated correctly! Thank you very much

[ArthurSCD]

After updating to 2.3.2, I started to experience issues that seem to be related to this as well. What I was doing was setting an attribute for multiple products at once. It would give me a message like so;
http://puu.sh/E5h3e/10b3c9a900.png

It never processed however. I found them sitting like mentioned above:
php bin/magento queue:consumers:list

product_action_attribute.website.update
exportProcessor
codegeneratorProcessor
async.operations.all

[Bhagyashri-Ranosys]

@TawabG,

you have to pass $filename instead of '$Path'. in below file.

Magento\ImportExport\Controller\Adminhtml\Export\File\Download.php

Before

public function execute()
{
if (empty($fileName = $this->getRequest()->getParam('filename'))) {
throw new LocalizedException(('Please provide export file name'));
}
try {
$path = 'export/' . $fileName;
$directory = $this->filesystem->getDirectoryRead(DirectoryList::VAR_DIR);
if ($directory->isFile($path)) {
return $this->fileFactory->create(
$path,
$directory->readFile($path),
DirectoryList::VAR_DIR,
'application/octet-stream'
);
}
} catch (LocalizedException | \Exception $exception) {
throw new LocalizedException(('There are no export file with such name %1', $fileName));
}
}
After
public function execute()
{
if (empty($fileName = $this->getRequest()->getParam('filename'))) {
throw new LocalizedException(('Please provide export file name'));
}
try {
$path = 'export/' . $fileName;
$directory = $this->filesystem->getDirectoryRead(DirectoryList::VAR_DIR);
if ($directory->isFile($path)) {
return $this->fileFactory->create(
$fileName,
$directory->readFile($path),
DirectoryList::VAR_DIR,
'application/octet-stream'
);
}
} catch (LocalizedException | \Exception $exception) {
throw new LocalizedException(('There are no export file with such name %1', $fileName));
}
}

[TawabG]

@TawabG,

you have to pass $filename instead of '$Path'. in below file.

Magento\ImportExport\Controller\Adminhtml\Export\File\Download.php

Before

public function execute()
{
if (empty($fileName = $this->getRequest()->getParam('filename'))) {
throw new LocalizedException(('Please provide export file name')); } try { $path = 'export/' . $fileName; $directory = $this->filesystem->getDirectoryRead(DirectoryList::VAR_DIR); if ($directory->isFile($path)) { return $this->fileFactory->create( $path, $directory->readFile($path), DirectoryList::VAR_DIR, 'application/octet-stream' ); } } catch (LocalizedException | \Exception $exception) { throw new LocalizedException(('There are no export file with such name %1', $fileName));
}
}
After
public function execute()
{
if (empty($fileName = $this->getRequest()->getParam('filename'))) {
throw new LocalizedException(('Please provide export file name')); } try { $path = 'export/' . $fileName; $directory = $this->filesystem->getDirectoryRead(DirectoryList::VAR_DIR); if ($directory->isFile($path)) { return $this->fileFactory->create( $fileName, $directory->readFile($path), DirectoryList::VAR_DIR, 'application/octet-stream' ); } } catch (LocalizedException | \Exception $exception) { throw new LocalizedException(('There are no export file with such name %1', $fileName));
}
}

Thanks for the reply! Your solution partly solved my problem.
Now, I get the notification "Message is added to queue, wait to get your file soon" then when I look at my export folder: a new catalog_product_xxx.CSV is created. However this file is only created when the command "php bin/magento queue:consumers:start exportProcessor" is running. Without this command no export will be done.

In addition, I noticed that the new export files are saved with '664' permissions.

[TawabG]

bin/magento queue:consumers:start

I can't imagine this being the only solution to this problem. This means I have to open CLI and run this command for EVERY export i'm doing!? That's ridiculous.

[seattletechworks]

@TawabG - check your cron processes.

Run "ps -ef | grep cron"

Delete any old processes which may be stuck / still running.

I had similar issue in my dev site. I ran this command -

php bin/magento queue:consumers:list

product_action_attribute.update
product_action_attribute.website.update
exportProcessor
codegeneratorProcessor
async.operations.all

Cron should be kicking off jobs for these consumers. The exportProcessor is the job which handles the export.

You should see similar process in your terminal -

Run "ps -ef | grep consumers"

php bin/magento queue:consumers:start exportProcessor --single-thread --max-messages=10000

[PsmIndia]

There any way to export product csv without run commands ? my magento ver 2.3.4
Running the command manually works as a temporary fix: php bin/magento queue:consumers:start exportProcessor

[lesliemcc]

hi i have no coding experience and i need to export my magento 2.3.2 stores customers over to shopify but when i try the following message appears and nothing happens.

[bruce-gemberling]

Can you SSH into your server? If so then we can help you out.

…

On Tue, Jul 21, 2020, 6:39 AM lesliemcc ***@***.***> wrote: hi i have no coding experience and i need to export my magento 2.3.2 stores customers over to shopify but when i try the following message appears and nothing happens. [image: lkjhbnm,lkjn] <https://user-images.githubusercontent.com/68594308/88050663-2ad9e780-cb4f-11ea-8e9b-ce5b34c50c69.JPG> — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#23450 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AADAT2FM5E4TE5RSKOOFIRTR4V47DANCNFSM4H35E6BQ> .

[lesliemcc]

I don’t I’m afraid as an independent team deals with it. Kind Regards, Leslie McCulloch, Miskie London LTD Phone: 07532725328 Email: lesliemcculloch@miskielondon.com

…

On 21 Jul 2020, at 14:49, Bruce Gemberling ***@***.***> wrote:  Can you SSH into your server? If so then we can help you out. On Tue, Jul 21, 2020, 6:39 AM lesliemcc ***@***.***> wrote: > hi i have no coding experience and i need to export my magento 2.3.2 > stores customers over to shopify but when i try the following message > appears and nothing happens. > [image: lkjhbnm,lkjn] > <https://user-images.githubusercontent.com/68594308/88050663-2ad9e780-cb4f-11ea-8e9b-ce5b34c50c69.JPG> > > — > You are receiving this because you commented. > Reply to this email directly, view it on GitHub > <#23450 (comment)>, > or unsubscribe > <https://github.com/notifications/unsubscribe-auth/AADAT2FM5E4TE5RSKOOFIRTR4V47DANCNFSM4H35E6BQ> > . > — You are receiving this because you commented. Reply to this email directly, view it on GitHub, or unsubscribe.

[adamlavery]

For some reason, Magento Core decided to make exports always asynchronous (run in the background) instead of optionally running big jobs in the background. Which means you have to wait for the job to run and check back on the export page regularly until you see your export file is there. It's a bit rubbish, but that's how it is now! If it never appears, it means your cron is not running (should see a notification about that). That's something you'll need to get your web developer to sort out.

[agbdxb]

Add a cron on the control panel of your webhost and set it to run the below command every 2 min.s (or the longest you are willing to wait). If you are unable to do it yourself, perhaps seek help from your webhost helpdesk.

php bin/magento queue:consumers:start exportProcessor

[HenKun]

Can everybody who runs against this, check if the message queue consumer processes are running?
So it works like this:

* the cron searches for running consumer processes

* if it finds a running process, it won't do anything

* if it doesn't find a running process, it should spawn one

* a running consumer process is sitting there and checking every second or so if a new message was added to the queue

* when it finds a new message in the queue, it will start processing it

You can find such processes with a command like:
ps aux | grep [c]onsumer

It should output something like this on a vanilla 2.3.2 installation:

someuser 123  0.0  2.1 530072 175892 ?       S    Aug09   0:33 php /path/to/bin/magento queue:consumers:start product_action_attribute.update --pid-file-path=product_action_attribute.update-some-hostname.pid --max-messages=10000
someuser 124  0.0  2.1 530068 176016 ?       S    Aug09   0:33 php /path/to/bin/magento queue:consumers:start product_action_attribute.website.update --pid-file-path=product_action_attribute.website.update-some-hostname.pid --max-messages=10000
someuser 125  0.0  2.1 530060 175952 ?       S    Aug09   0:33 php /path/to/bin/magento queue:consumers:start exportProcessor --pid-file-path=exportProcessor-some-hostname.pid --max-messages=10000
someuser 126  0.0  2.1 530048 175840 ?       S    Aug09   0:33 php /path/to/bin/magento queue:consumers:start codegeneratorProcessor --pid-file-path=codegeneratorProcessor-some-hostname.pid --max-messages=10000

If you can not find running consumer processes even though the cron is running. Then for some reason those process aren't getting spawned.
I would then start to debug the Magento\MessageQueue\Model\Cron\ConsumersRunner::run method. To see why they aren't getting spawned.

If you think you spotted a bug after debugging, then please let us know!

Alternatively you can also manually execute the bin/magento queue:consumers:start command but first look at the help section and see how this works so you understand the different options you can pass it. Also devdocs has some info on these commands.
This can also be automated using supervisor for example instead of using the cron, here's the documention around that.

@hostep I debugged in and found the following:

• Cron is running correctly and all settings are fetched correctly, also the empty array [] for cron_consumers_runner does work
• The problem lies in the locking.
• Although no processes for the consumers are spawned (anymore!?), MySQL (in my case MariaDB) still holds locks for all consumers. When the consumers are triggered by cron they are started as single-thread, hence, checking the lock. So the proccesses are not spawned. The locking and unlocking is done in StartConsumerCommand. It might be that there lies the issue, cause unlock is not called, if something goes wrong in $consumer->process($numberOfMessages);. I am not that good in PHP, in C# one needs to put the unlock in a finally statement, so it is safely unlocked, even when process throws. You said "a running consumer process is sitting there and checking every second or so if a new message was added to the queue", which would mean that $consumer->process($numberOfMessages); is not supposed to end? (<-- I get the impression this is true, when calling via cli). Under what circumstances does process return without exception, so that unlock will be called? Alternatively, if $consumer->process($numberOfMessages); would return immediately after spawning the process (leaving the process runnning), the lock would be of no use, so I might not understand the inner workings correctly.
• If you run the consumer in console, it is not single-threaded by default, hence, the lock is not checked. That's why it does work for most of the users, although it does not work via cron. If the lock is held, one can check by using bin/magento queue:consumers:start product_action_attribute.update --single-thread. An error Consumer with the same name is running is returnd if the lock is held.
• Even more unfortunately I can not release the lock manually from database. RELEASE_LOCK(md5(consumerName)); must be called from the thread which triggered the lock. RELEASE_ALL_LOCKS is not yet available in my MariaDB version, and I am not sure if this would remove locks triggered by other threads.

In a nutshell:

• 'cron_consumers_runner' => [ 'cron_run' => true, 'max_messages' => 50, 'consumers' => [ 'product_action_attribute.update', 'product_action_attribute.website.update' ] ], can be completely left out from the env.php, the default values are fetched correctly
• in my case the database held a lock, preventing the consumer proccess from being spawned, when called by cron
• I cannot remove the lock from database
• starting consumers from CLI does work, cause it does not use the --single-thread parameter by default, avoiding the check for the lock.

If it is not fully clear, what I wrote, please ask for clarification :D

[hostep]

@HenKun

I've had some issues in the past as well with locks in mysql getting stuck (not necessarily with queue consumers, it was with cronjobs I memory serves me right, but they both use the same locking mechanism). We've found that switching the lock provider from the database to the filesystem to be a lot more stable.
Here is the documentation to do that: https://devdocs.magento.com/guides/v2.3/install-gde/install/cli/install-cli-subcommands-lock.html#instgde-cli-lockconfig

I hope this can solve your issue!

[HenKun]

Thanks for the link! While this might solve the "stucking" issue for me, since I can interfere with the file system, I think there is a larger underlying problem.

1. The unlock is not guaranteed to be called (as far as I can see). At least it should be in a finally clause.
   1a. If running as single-thread and no max-messages parameter is given, process will never end gracefully (it's in a while(true) loop - ), hence, the lock cannot be freed. (Although I think the consumers are not supposed to be run without this parameter, it is still possible, even from cron if the default settings are changed)
   1b. If running as single-thread and max-messages parameter is given, it is run in a for loop, where exceptions can occur. There is even an exit(0); (although I do not understand when this is called ad-hoc). If an exception occurs, the lock is not freed.
2. The DB locking mechanism is the default, so as far as I can read from multiple GitHub issues and StackOverflow questions, more users seem to have problems with not starting bulk actions, that might be caused by stucked locking. Starting the consumers from CLI seems to solve the issue for many users, but at that moment they don't realize it just solves the issue for a very short period of time. (Like me when I noticed it last week)
3. Is the locking meta data copied, when databases are copied? E.g. when reapplying backup databases or when moving databases from stage to production? That might be another reason for locks being present but no processes being spawned.
4. Does using the file system lock have any performance drawbacks against using the db locking mechanism?
5. Would it be a possibility to not using locks but directly check for a matching running process? Again, I am not that much into PHP and processes, so this is just my naive idea.

[arvind-curotec]

Magento version 2.3.4-p2
All processes running correctly. Still the export not working.

Here are the consumer processes:

php bin/magento queue:consumers:list
product_action_attribute.update
product_action_attribute.website.update
exportProcessor
codegeneratorProcessor
async.operations.all

[HenKun]

Magento version 2.3.4-p2
All processes running correctly. Still the export not working.

Here are the consumer processes:

php bin/magento queue:consumers:list
product_action_attribute.update
product_action_attribute.website.update
exportProcessor
codegeneratorProcessor
async.operations.all

Listing the queue consumers does not mean the processes are running.
Run ps aux to see if they are really spawned.

[arvind-curotec]

@HenKun Please See the output below:

-bash-4.2$ ps aux | grep [c]onsumer
user     11883  0.0  1.0 487880 166556 ?       S    Sep08   1:12 /usr/bin/php httpdocs/bin/magento queue:consumers:start product_action_attribute.update --single-thread --max-messages=10000
user     11885  0.0  1.0 487880 164960 ?       S    Sep08   1:13 /usr/bin/php httpdocs/bin/magento queue:consumers:start product_action_attribute.website.update --single-thread --max-messages=10000
user     11889  0.0  1.0 487884 164348 ?       S    Sep08   1:13 /usr/bin/php httpdocs/bin/magento queue:consumers:start codegeneratorProcessor --single-thread --max-messages=10000
root     14218  1.0  1.0 591576 164296 pts/1   S    09:41   0:00 /usr/bin/php httpdocs/bin/magento queue:consumers:start exportProcessor --single-thread --max-messages=10000

[HenKun]

So obviously they are running. Just wanted to make you aware, that queue:consumers:list is no indicator of running processes.

[perfpcs]

Alot of us are having this bulk update problem and general messaging broker problems with M 2.4.0. Would be nice to get team Magento to fix this bad problem!! #29797

[Cesarorp]

@gaurangpadhiyar007

In what module can we run this?

public function execute()
{
if ($this->getRequest()->getPost(ExportModel::FILTER_ELEMENT_GROUP)) {
try {
$params = $this->getRequest()->getParams();
$model = $this->_objectManager->create(\Magento\ImportExport\Model\Export::class);
$model->setData($this->getRequest()->getParams());
$this->sessionManager->writeClose();

        return $this->fileFactory->create(
            $model->getFileName(),
            $model->export(),
            \Magento\Framework\App\Filesystem\DirectoryList::VAR_DIR,
            $model->getContentType()
        );

        /** @var ExportInfoFactory $dataObject */
        $dataObject = $this->exportInfoFactory->create(
            $params['file_format'],
            $params['entity'],
            $params['export_filter']
        );

        $this->messagePublisher->publish('import_export.export', $dataObject);
        $this->messageManager->addSuccessMessage(
            __('Message is added to queue, wait to get your file soon')
        );
    } catch (\Exception $e) {
        $this->_objectManager->get(\Psr\Log\LoggerInterface::class)->critical($e);
        $this->messageManager->addError(__('Please correct the data sent value.'));
    }
} else {
    $this->messageManager->addError(__('Please correct the data sent value.'));
}
/** @var \Magento\Backend\Model\View\Result\Redirect $resultRedirect */
$resultRedirect = $this->resultFactory->create(ResultFactory::TYPE_REDIRECT);
$resultRedirect->setPath('adminhtml/*/index');
return $resultRedirect;

}

[gaurangpadhiyar007]

@gaurangpadhiyar007

In what module can we run this?

public function execute()
{
if ($this->getRequest()->getPost(ExportModel::FILTER_ELEMENT_GROUP)) {
try {
$params = $this->getRequest()->getParams();
$model = $this->_objectManager->create(\Magento\ImportExport\Model\Export::class);
$model->setData($this->getRequest()->getParams());
$this->sessionManager->writeClose();

        return $this->fileFactory->create(
            $model->getFileName(),
            $model->export(),
            \Magento\Framework\App\Filesystem\DirectoryList::VAR_DIR,
            $model->getContentType()
        );

        /** @var ExportInfoFactory $dataObject */
        $dataObject = $this->exportInfoFactory->create(
            $params['file_format'],
            $params['entity'],
            $params['export_filter']
        );

        $this->messagePublisher->publish('import_export.export', $dataObject);
        $this->messageManager->addSuccessMessage(
            __('Message is added to queue, wait to get your file soon')
        );
    } catch (\Exception $e) {
        $this->_objectManager->get(\Psr\Log\LoggerInterface::class)->critical($e);
        $this->messageManager->addError(__('Please correct the data sent value.'));
    }
} else {
    $this->messageManager->addError(__('Please correct the data sent value.'));
}
/** @var \Magento\Backend\Model\View\Result\Redirect $resultRedirect */
$resultRedirect = $this->resultFactory->create(ResultFactory::TYPE_REDIRECT);
$resultRedirect->setPath('adminhtml/*/index');
return $resultRedirect;

}

This is file.

Magento\ImportExport\Controller\Adminhtml\Export\File\Download

[magecoders]

@hostep As per your suggestion, the consumer processes are running. But both bulk product attribute update and export products operation not working. Any ideas ?

Thanks

[hostep]

@magecoders: the status says 'in progress', so they should be running and your screenshot also shows that those particular consumers are running, so that's good!
But if it takes a really long time and you don't see any progress, then something is wrong somehow. You'll have to start to debug this somewhat I'm afraid. We personally haven't noticed problems like these yet on Magento 2.3.x

[hostep]

@HenKun: you made some good remarks before, could you open a new issue with your findings, because these might be important to get fixed. Thanks!

[magecoders]

@HenKun I agree with your points, I have debug into running consumer process and they stuck due to DB lock in my case.

[rou9e]

Hello there,

In our case, we can confirm hostep and HenKun answers :

• Everything was running fine from cron perspective
• ps aux | grep [c]onsumer would never show any consumer running
• MySQL (MariaDB) indeed seems to have lock issues because

bin/magento queue:consumers:start product_action_attribute.update works as expected but
bin/magento queue:consumers:start product_action_attribute.update --single-thread fails with the message : "An error Consumer with the same name is running"

Switching the lock configuration of Magento to this :

bin/magento setup:config:set --lock-provider="file";
bin/magento setup:config:set --lock-file-path="var/locks";

...seems to resolve our problem : Mass products updates and exports work as expected.

On the other hand, the related bulk actions (domain.com/admin/bulk/index/index/) were not updated accordingly (tablename magento_bulk). They stay with the status "Not Started" even if the task has been completed.

Here is how to fix it : #29797

Thank you very much for your help !

[rdeelstra]

Feature request: It would be easier to add a checkbox to generate it in the background yes/no on creation
If you have a small collection, through the filters or don't have 1000's of products, and would like to have it immediately, it can really be annoying and has a negative impact on the UX.

[amirmdp]

Hi I have a similar issue as soon as I select any option among Entity Type to export and click continue, the page returns with a notice "Please correct the data sent value." without showing any Entity Attributes. Please help I am new in magento.
environment: Plesk
PHP 7.4
Magento ver. 2.4.2