Cron not Reindexing Index

The indexes are not being reindexed with the standard cron:run cron job. This is particularly noticeable when making a change to a category and it not being updated on the front end. I can add a static block or description to a category and save it in the backend and the updates will not appear on the frontend until the indexers have been manually reindexed. This could possibly be related to #2855.

Preconditions

1. Magento 2.1 CE
2. CentOS6
3. PHP 5.6.24
4. MySQL 5.6
5. Shared hosting with Plesk control panel
6. Using flat categories and products

Steps to reproduce

1. Make a change to a categories description or add a static content block
2. Save changes
3. Go to category on frontend to see result

Expected result

1. Category should be updated with new content

Actual result

1. Category is not updated with new content

I have noticed that the debug.log has been filling up quite fast with lines like the following:

[2016-07-27 11:33:01] main.DEBUG: cache_invalidate:  {"method":"GET","url":"http:/","invalidateInfo":{"identifier":"DB_PDO_MYSQL_DDL_design_config_grid_flat_1"},"is_exception":false} []
[2016-07-27 11:33:01] main.DEBUG: cache_invalidate:  {"method":"GET","url":"http:/","invalidateInfo":{"identifier":"DB_PDO_MYSQL_DDL_design_config_grid_flat_2"},"is_exception":false} []
[2016-07-27 11:33:01] main.DEBUG: cache_invalidate:  {"method":"GET","url":"http:/","invalidateInfo":{"identifier":"DB_PDO_MYSQL_DDL_design_config_grid_flat_3"},"is_exception":false} []
[2016-07-27 11:33:01] main.DEBUG: cache_invalidate:  {"method":"GET","url":"http:/","invalidateInfo":{"identifier":"DB_PDO_MYSQL_DDL_design_config_grid_flat_4"},"is_exception":false} []
[2016-07-27 11:33:01] main.DEBUG: cache_invalidate:  {"method":"GET","url":"http:/","invalidateInfo":{"identifier":"DB_PDO_MYSQL_DDL_design_config_grid_flat_1"},"is_exception":false} []
[2016-07-27 11:33:01] main.DEBUG: cache_invalidate:  {"method":"GET","url":"http:/","invalidateInfo":{"identifier":"DB_PDO_MYSQL_DDL_design_config_grid_flat_2"},"is_exception":false} []
[2016-07-27 11:33:01] main.DEBUG: cache_invalidate:  {"method":"GET","url":"http:/","invalidateInfo":{"identifier":"DB_PDO_MYSQL_DDL_design_config_grid_flat_3"},"is_exception":false} []
[2016-07-27 11:33:01] main.DEBUG: cache_invalidate:  {"method":"GET","url":"http:/","invalidateInfo":{"identifier":"DB_PDO_MYSQL_DDL_design_config_grid_flat_4"},"is_exception":false} []
[2016-07-27 11:33:02] main.DEBUG: cache_invalidate:  {"method":"GET","url":"http:/","invalidateInfo":{"identifier":"DB_PDO_MYSQL_DDL_customer_grid_flat_1"},"is_exception":false} []
[2016-07-27 11:33:02] main.DEBUG: cache_invalidate:  {"method":"GET","url":"http:/","invalidateInfo":{"identifier":"DB_PDO_MYSQL_DDL_customer_grid_flat_2"},"is_exception":false} []
[2016-07-27 11:33:02] main.DEBUG: cache_invalidate:  {"method":"GET","url":"http:/","invalidateInfo":{"identifier":"DB_PDO_MYSQL_DDL_customer_grid_flat_3"},"is_exception":false} []
[2016-07-27 11:33:02] main.DEBUG: cache_invalidate:  {"method":"GET","url":"http:/","invalidateInfo":{"identifier":"DB_PDO_MYSQL_DDL_customer_grid_flat_4"},"is_exception":false} []
[2016-07-27 11:33:02] main.DEBUG: cache_invalidate:  {"method":"GET","url":"http:/","invalidateInfo":{"identifier":"DB_PDO_MYSQL_DDL_customer_grid_flat_1"},"is_exception":false} []
[2016-07-27 11:33:02] main.DEBUG: cache_invalidate:  {"method":"GET","url":"http:/","invalidateInfo":{"identifier":"DB_PDO_MYSQL_DDL_customer_grid_flat_2"},"is_exception":false} []
[2016-07-27 11:33:02] main.DEBUG: cache_invalidate:  {"method":"GET","url":"http:/","invalidateInfo":{"identifier":"DB_PDO_MYSQL_DDL_customer_grid_flat_3"},"is_exception":false} []
[2016-07-27 11:33:02] main.DEBUG: cache_invalidate:  {"method":"GET","url":"http:/","invalidateInfo":{"identifier":"DB_PDO_MYSQL_DDL_customer_grid_flat_4"},"is_exception":false} []
[2016-07-27 11:33:02] main.DEBUG: cache_invalidate:  {"method":"GET","url":"http:/","invalidateInfo":{"identifier":"DB_PDO_MYSQL_DDL_catalog_product_entity_tmp_indexer_1"},"is_exception":false} []
[2016-07-27 11:33:02] main.DEBUG: cache_invalidate:  {"method":"GET","url":"http:/","invalidateInfo":{"identifier":"DB_PDO_MYSQL_DDL_catalog_product_entity_tmp_indexer_2"},"is_exception":false} []
[2016-07-27 11:33:02] main.DEBUG: cache_invalidate:  {"method":"GET","url":"http:/","invalidateInfo":{"identifier":"DB_PDO_MYSQL_DDL_catalog_product_entity_tmp_indexer_3"},"is_exception":false} []
[2016-07-27 11:33:02] main.DEBUG: cache_invalidate:  {"method":"GET","url":"http:/","invalidateInfo":{"identifier":"DB_PDO_MYSQL_DDL_catalog_product_entity_tmp_indexer_4"},"is_exception":false} []
[2016-07-27 11:33:02] main.DEBUG: cache_invalidate:  {"method":"GET","url":"http:/","invalidateInfo":{"identifier":"DB_PDO_MYSQL_DDL_catalog_product_entity_decimal_tmp_indexer_1"},"is_exception":false} []
[2016-07-27 11:33:02] main.DEBUG: cache_invalidate:  {"method":"GET","url":"http:/","invalidateInfo":{"identifier":"DB_PDO_MYSQL_DDL_catalog_product_entity_decimal_tmp_indexer_2"},"is_exception":false} []
[2016-07-27 11:33:02] main.DEBUG: cache_invalidate:  {"method":"GET","url":"http:/","invalidateInfo":{"identifier":"DB_PDO_MYSQL_DDL_catalog_product_entity_decimal_tmp_indexer_3"},"is_exception":false} []
[2016-07-27 11:33:02] main.DEBUG: cache_invalidate:  {"method":"GET","url":"http:/","invalidateInfo":{"identifier":"DB_PDO_MYSQL_DDL_catalog_product_entity_decimal_tmp_indexer_4"},"is_exception":false} []
[2016-07-27 11:33:02] main.DEBUG: cache_invalidate:  {"method":"GET","url":"http:/","invalidateInfo":{"identifier":"DB_PDO_MYSQL_DDL_catalog_product_entity_text_tmp_indexer_1"},"is_exception":false} []
[2016-07-27 11:33:02] main.DEBUG: cache_invalidate:  {"method":"GET","url":"http:/","invalidateInfo":{"identifier":"DB_PDO_MYSQL_DDL_catalog_product_entity_text_tmp_indexer_2"},"is_exception":false} []
[2016-07-27 11:33:02] main.DEBUG: cache_invalidate:  {"method":"GET","url":"http:/","invalidateInfo":{"identifier":"DB_PDO_MYSQL_DDL_catalog_product_entity_text_tmp_indexer_3"},"is_exception":false} []
[2016-07-27 11:33:02] main.DEBUG: cache_invalidate:  {"method":"GET","url":"http:/","invalidateInfo":{"identifier":"DB_PDO_MYSQL_DDL_catalog_product_entity_text_tmp_indexer_4"},"is_exception":false} []
[2016-07-27 11:33:02] main.DEBUG: cache_invalidate:  {"method":"GET","url":"http:/","invalidateInfo":{"identifier":"DB_PDO_MYSQL_DDL_catalog_product_entity_varchar_tmp_indexer_1"},"is_exception":false} []
[2016-07-27 11:33:02] main.DEBUG: cache_invalidate:  {"method":"GET","url":"http:/","invalidateInfo":{"identifier":"DB_PDO_MYSQL_DDL_catalog_product_entity_varchar_tmp_indexer_2"},"is_exception":false} []
[2016-07-27 11:33:02] main.DEBUG: cache_invalidate:  {"method":"GET","url":"http:/","invalidateInfo":{"identifier":"DB_PDO_MYSQL_DDL_catalog_product_entity_varchar_tmp_indexer_3"},"is_exception":false} []
[2016-07-27 11:33:02] main.DEBUG: cache_invalidate:  {"method":"GET","url":"http:/","invalidateInfo":{"identifier":"DB_PDO_MYSQL_DDL_catalog_product_entity_varchar_tmp_indexer_4"},"is_exception":false} []
[2016-07-27 11:33:02] main.DEBUG: cache_invalidate:  {"method":"GET","url":"http:/","invalidateInfo":{"identifier":"DB_PDO_MYSQL_DDL_catalog_product_entity_int_tmp_indexer_1"},"is_exception":false} []
[2016-07-27 11:33:02] main.DEBUG: cache_invalidate:  {"method":"GET","url":"http:/","invalidateInfo":{"identifier":"DB_PDO_MYSQL_DDL_catalog_product_entity_int_tmp_indexer_2"},"is_exception":false} []
[2016-07-27 11:33:02] main.DEBUG: cache_invalidate:  {"method":"GET","url":"http:/","invalidateInfo":{"identifier":"DB_PDO_MYSQL_DDL_catalog_product_entity_int_tmp_indexer_3"},"is_exception":false} []
[2016-07-27 11:33:02] main.DEBUG: cache_invalidate:  {"method":"GET","url":"http:/","invalidateInfo":{"identifier":"DB_PDO_MYSQL_DDL_catalog_product_entity_int_tmp_indexer_4"},"is_exception":false} []
[2016-07-27 11:33:02] main.DEBUG: cache_invalidate:  {"method":"GET","url":"http:/","invalidateInfo":{"identifier":"DB_PDO_MYSQL_DDL_catalog_product_entity_int_tmp_indexer_value_1"},"is_exception":false} []
[2016-07-27 11:33:02] main.DEBUG: cache_invalidate:  {"method":"GET","url":"http:/","invalidateInfo":{"identifier":"DB_PDO_MYSQL_DDL_catalog_product_entity_int_tmp_indexer_value_2"},"is_exception":false} []
[2016-07-27 11:33:02] main.DEBUG: cache_invalidate:  {"method":"GET","url":"http:/","invalidateInfo":{"identifier":"DB_PDO_MYSQL_DDL_catalog_product_entity_int_tmp_indexer_value_3"},"is_exception":false} []
[2016-07-27 11:33:02] main.DEBUG: cache_invalidate:  {"method":"GET","url":"http:/","invalidateInfo":{"identifier":"DB_PDO_MYSQL_DDL_catalog_product_entity_int_tmp_indexer_value_4"},"is_exception":false} []
[2016-07-27 11:33:02] main.DEBUG: cache_invalidate:  {"method":"GET","url":"http:/","invalidateInfo":{"identifier":"DB_PDO_MYSQL_DDL_catalog_product_entity_datetime_tmp_indexer_1"},"is_exception":false} []
[2016-07-27 11:33:02] main.DEBUG: cache_invalidate:  {"method":"GET","url":"http:/","invalidateInfo":{"identifier":"DB_PDO_MYSQL_DDL_catalog_product_entity_datetime_tmp_indexer_2"},"is_exception":false} []
[2016-07-27 11:33:02] main.DEBUG: cache_invalidate:  {"method":"GET","url":"http:/","invalidateInfo":{"identifier":"DB_PDO_MYSQL_DDL_catalog_product_entity_datetime_tmp_indexer_3"},"is_exception":false} []
[2016-07-27 11:33:02] main.DEBUG: cache_invalidate:  {"method":"GET","url":"http:/","invalidateInfo":{"identifier":"DB_PDO_MYSQL_DDL_catalog_product_entity_datetime_tmp_indexer_4"},"is_exception":false} []
[2016-07-27 11:33:02] main.DEBUG: cache_invalidate:  {"method":"GET","url":"http:/","invalidateInfo":{"identifier":"DB_PDO_MYSQL_DDL_catalog_product_flat_1_tmp_indexer_1"},"is_exception":false} []
[2016-07-27 11:33:02] main.DEBUG: cache_invalidate:  {"method":"GET","url":"http:/","invalidateInfo":{"identifier":"DB_PDO_MYSQL_DDL_catalog_product_flat_1_tmp_indexer_2"},"is_exception":false} []
[2016-07-27 11:33:02] main.DEBUG: cache_invalidate:  {"method":"GET","url":"http:/","invalidateInfo":{"identifier":"DB_PDO_MYSQL_DDL_catalog_product_flat_1_tmp_indexer_3"},"is_exception":false} []
[2016-07-27 11:33:02] main.DEBUG: cache_invalidate:  {"method":"GET","url":"http:/","invalidateInfo":{"identifier":"DB_PDO_MYSQL_DDL_catalog_product_flat_1_tmp_indexer_4"},"is_exception":false} []
[2016-07-27 11:33:02] main.DEBUG: cache_invalidate:  {"method":"GET","url":"http:/","invalidateInfo":{"identifier":"DB_PDO_MYSQL_DDL_catalog_product_flat_1_tmp_indexer_1"},"is_exception":false} []
[2016-07-27 11:33:02] main.DEBUG: cache_invalidate:  {"method":"GET","url":"http:/","invalidateInfo":{"identifier":"DB_PDO_MYSQL_DDL_catalog_product_flat_1_tmp_indexer_2"},"is_exception":false} []
[2016-07-27 11:33:02] main.DEBUG: cache_invalidate:  {"method":"GET","url":"http:/","invalidateInfo":{"identifier":"DB_PDO_MYSQL_DDL_catalog_product_flat_1_tmp_indexer_3"},"is_exception":false} []
[2016-07-27 11:33:02] main.DEBUG: cache_invalidate:  {"method":"GET","url":"http:/","invalidateInfo":{"identifier":"DB_PDO_MYSQL_DDL_catalog_product_flat_1_tmp_indexer_4"},"is_exception":false} []
[2016-07-27 11:33:02] main.DEBUG: cache_invalidate:  {"method":"GET","url":"http:/","invalidateInfo":{"identifier":"DB_PDO_MYSQL_DDL_catalog_product_flat_1_drop_indexer_1"},"is_exception":false} []
[2016-07-27 11:33:02] main.DEBUG: cache_invalidate:  {"method":"GET","url":"http:/","invalidateInfo":{"identifier":"DB_PDO_MYSQL_DDL_catalog_product_flat_1_drop_indexer_2"},"is_exception":false} []
[2016-07-27 11:33:02] main.DEBUG: cache_invalidate:  {"method":"GET","url":"http:/","invalidateInfo":{"identifier":"DB_PDO_MYSQL_DDL_catalog_product_flat_1_drop_indexer_3"},"is_exception":false} []
[2016-07-27 11:33:02] main.DEBUG: cache_invalidate:  {"method":"GET","url":"http:/","invalidateInfo":{"identifier":"DB_PDO_MYSQL_DDL_catalog_product_flat_1_drop_indexer_4"},"is_exception":false} []

I have setup a cron job just for reindexing as a work around for now and it seems to be working. My cron jobs are setup as follows:

* * * * * /opt/plesk/php/5.6/bin/php -d memory_limit=1G -c /opt/plesk/php/5.6/etc/php.ini /magento2/bin/magento cron:run >> /magento2/var/log/magento.cron.log
* * * * * /opt/plesk/php/5.6/bin/php -d memory_limit=1G -c /opt/plesk/php/5.6/etc/php.ini /magento2/update/cron.php >> /magento2/var/log/update.cron.log
* * * * * /opt/plesk/php/5.6/bin/php -d memory_limit=1G -c /opt/plesk/php/5.6/etc/php.ini /magento2/bin/magento setup:cron:run >> /magento2/var/log/setup.cron.log
* * * * * /opt/plesk/php/5.6/bin/php -d memory_limit=1G -c /opt/plesk/php/5.6/etc/php.ini /magento2/bin/magento indexer:reindex

[hostep]

We also see this using CE 2.1.0 - PHP 7.0.8 - Mysql 5.7.13

When adding products, they don't appear on the frontend, until we manually run a bin/magento indexer:reindex every time.

Cronjobs are setup correctly and indexers are set to 'Update by Schedule'.

Sounds like a pretty important bug.

[grasdaggel]

We are experiencing the same problem on lower Mage2 versions (2.0.4 and 2.0.6). So far we haven't found a solution for that.

[nmallepally]

Yes, it is a important bug. I am experiencing the same issue as well, Magento 2.1 v, The cron is setup properly, but does not show products until it is re indexed.

[boldhedgehog]

Works for me if I set 'Use Separate Process' to 'No' for the index CRON group.

[hostep]

@boldhedgehog: tnx, but unfortunately it doesn't seem to fix our issue :(

Setting 'Use Separate Process' to 'No' for the index CRON group does not work for me either.

[boldhedgehog]

@hostep I had to wait a little and after 5-10 runs of cron:run Magento had added indexer tasks to cron_schedule:

schedule_id	job_code	status	messages	created_at	scheduled_at	executed_at	finished_at
1106	indexer_update_all_views	success	NULL	2016-07-29 15:34:45	2016-07-29 15:34:00	2016-07-29 15:34:47	2016-07-29 15:34:47
1105	indexer_reindex_all_invalid	success	NULL	2016-07-29 15:34:45	2016-07-29 15:34:00	2016-07-29 15:34:47	2016-07-29 15:34:47

[hostep]

@boldhedgehog: yes, I see them too, even with 'Use separate process' on 'Yes'. So this means the indexers are correctly executed through the cron, but something in the execution itself goes wrong I guess...

If I find some time in the coming days, I'll debug and try figure out what exactly goes wrong.

Tnx for the hints!

This is the same behaviour for me as well. The cron_schedule table says success for the status but the frontend does not update.

[boldhedgehog]

It has rebuilt my Elasticsearch index. This can be an index/caching issue, not a CRON/index issue you have now, in addition.

[ojhaujjwal]

I have the same problem in 2.1.0. I have setup the cron job nicely. I even tried to run cron:run manually more than once but the indexing was not done. However, running indexer:reindex did all the reindexing. So, I think that means a bug in cron:run which does not do the job of indexing.

[ojhaujjwal]

This seems a critical issue. Is there any workaround for fixing this until an official fix is done?

@ojhaujjwal I setup an additional cron job to run the indexer:reindex command for now. It fills up the debug.log file quite quickly when it's run though.

[ojhaujjwal]

@crtsl I noticed that if I add magento indexer:reindex to cron, the cache is invalidated everytime the command is run and I have to flush the page cache. This creates even more problems.