Introduces indexables verification cron

[thijsoo]

Context

• We want to add a system that verifies that your indexables are all in an up to date state. This means that we need to re-index all your non-timestamped indexables (terms,users,links) and check mismatched timestamps for posts.
• We want to do this only when some has has disabled and re-enabled the plugin.
• The solution is similar to the background indexing cron. It will reschedule itself after its done and there is still work to be done.
• It saves the progress in a set of options.

There are still some todo's left:

• Write test instructions.
• Finish 100% code coverage and test the infrastructure components via integration tests.

Summary

This PR can be summarized in the following changelog entry:

• Adds a background cron that makes sure all indexables are up to date after re-activating the plugin.

Relevant technical choices:

• This PR trys to follow the discussed onion architecture flow.

• Some notable new things I introduced here are the infrastructure folder and the port folder in the application.
  The infrastructure folder holds the connection to WordPress. This is why I placed my verification actions here. These call wp functions and run queries (via repositories)

• I used the port folder to describe the interface of the outdated post repository. This is then implemented in the infrastructure folder, since it then calls wp functions.

Test instructions

Test instructions for the acceptance test before the PR gets merged

This PR can be acceptance tested by following these steps:

• Make sure you only have free enabled. and make sure you have the WP Crontrol plugin installed.
• This is easier with a small site (so not that many posts and terms) this way you can see the results of the queries faster.

To make sure the tasks get scheduled when they are supposed to

• Disable free and make sure you are not installing it fresh.
• Enable free and go to /wp-admin/tools.php?page=crontrol_admin_manage_page
• Look for Hooks: ‘wpseo_indexable_verify_post_indexables’ and ‘wpseo_indexable_verify_non_timestamped_indexables
• Select them, in the bulk actions menu select delete and click on Apply
• Reset your options via the test helper (simulating a new install)
• Disable and enable free and make sure the crons were not created.
• Finish the FTC.
• Disable and reenable free and make sure the crons are created again.
• Add the following filter to disable indexable use:
• add_filter('Yoast\WP\SEO\should_index_indexables','__return_false');
• Run the cron in the WP cli with the command wp cron event run "wpseo_indexable_verify_post_indexables" and make sure it is unscheduled after 1 run.
• Remove the filter and add the following filter:
• add_filter('Yoast\WP\SEO\enable_cron_verification','__return_false');
• Run the other cron in the WP cli with the command wp cron event run "wpseo_indexable_verify_non_timestamped_indexables" and make sure it is unscheduled after 1 run.
• Remove all filters added.

Data prep

• Make sure you run the SEOO
• Run the following list of queries and choose 1 record and remember/ write down the ID’s and object_id’s. The ID is the id of the indexable and the object_id is the id of the post/page/category etc.

select * from wp_yoast_indexable where object_type = 'term' and object_sub_type = 'category';

select * from wp_yoast_indexable where object_type = 'term' and object_sub_type = 'post_tag';

select * from wp_yoast_indexable where object_type = 'post' and object_sub_type = 'page';

select * from wp_yoast_indexable where object_type = 'post' and object_sub_type = 'post';

select * from wp_yoast_indexable where object_type = 'post-type-archive';

• Disable free
• Go to each of the objects you choose and change something that is also represented in their indexable. for example the permalink or the title or if the post is published or not.

Verification and re-syncing the data

• Enable free and go to /wp-admin/tools.php?page=crontrol_admin_manage_page
• Look for Hooks: ‘wpseo_indexable_verify_post_indexables’ and ‘wpseo_indexable_verify_non_timestamped_indexables`
• Run the wpseo_indexable_verify_non_timestamped_indexables with the command wp cron event run "wpseo_indexable_verify_non_timestamped_indexables" cron until it is no longer scheduled. Now make sure the changes you made in the terms are also represented in your indexable.
• Run the wpseo_indexable_verify_post_indexables with the command wp cron event run "wpseo_indexable_verify_post_indexables" cron until it is no longer scheduled. Now make sure the changes you made in the posts/pages are also represented in your indexable.

Testing the filters

For both the post indexables and the non timestamped indexables (terms) there is a different filter to increase or decrease the amount of indexables done in each batch.

For the post indexables:
- Run select * from wp_yoast_indexable where object_type = 'post' and object_sub_type = 'post'; and choose 3 posts with their id's in a row (so 1,3,6 is fine as long as there is nog 2,4,5 in between).
- Disable free and change something in those posts that is represented in the indexable.
- Add the following filter add_filter('Yoast\WP\SEO\post_verify_indexing_limit_size',2);
- Run the cli command wp cron event run "wpseo_indexable_verify_post_indexables" and make sure only the first two post indexables are updated.

For non timestamped indexables:

• Run select * from wp_yoast_indexable where object_type = 'term' and object_sub_type = 'category'; and choose 3 categories with their id's in a row.
  • Disable free and change something in those posts that is represented in the indexable.
  • Add the following filter add_filter('Yoast\WP\SEO\no_timestamped_indexable_verify_limit_size',2);
  • Run the cli command wp cron event run "wpseo_indexable_verify_non_timestamped_indexables" and make sure only the first two post indexables are updated.

Relevant test scenarios

• Changes should be tested with the browser console open
• Changes should be tested on different posts/pages/taxonomies/custom post types/custom taxonomies
• Changes should be tested on different editors (Block/Classic/Elementor/other)
• Changes should be tested on different browsers
• Changes should be tested on multisite

Test instructions for QA when the code is in the RC

• QA should use the same steps as above.

QA can test this PR by following these steps:

Impact check

This PR affects the following parts of the plugin, which may require extra testing:

UI changes

• This PR changes the UI in the plugin. I have added the 'UI change' label to this PR.

Other environments

• This PR also affects Shopify. I have added a changelog entry starting with [shopify-seo], added test instructions for Shopify and attached the Shopify label to this PR.

Documentation

• I have written documentation for this change.

Quality assurance

• I have tested this code to the best of my abilities.
• During testing, I had activated all plugins that Yoast SEO provides integrations for.
• I have added unit tests to verify the code works as intended.
• If any part of the code is behind a feature flag, my test instructions also cover cases where the feature flag is switched off.
• I have written this PR in accordance with my team's definition of done.
• I have checked that the base branch is correctly set.

Innovation

• No innovation project is applicable for this PR.
• This PR falls under an innovation project. I have attached the innovation label.
• I have added my hours to the WBSO document.

Fixes #

[coveralls]

Pull Request Test Coverage Report for Build 6892519348

• 129 of 283 (45.58%) changed or added relevant lines in 25 files are covered.
• 652 unchanged lines in 2 files lost coverage.
• Overall coverage decreased (-0.08%) to 49.203%

---

Changes Missing Coverage	Covered Lines	Changed/Added Lines	%
inc/options/class-wpseo-option-wpseo.php	6	7	85.71%
src/indexables/application/next-verification-action-handler.php	3	4	75.0%
src/indexables/application/verification-cron-batch-handler.php	10	11	90.91%
src/indexables/domain/batch-size.php	3	4	75.0%
src/indexables/application/commands/verify-non-timestamp-indexables-command.php	1	3	33.33%
src/indexables/domain/abstract-indexables-command.php	2	4	50.0%
src/indexables/domain/exceptions/no-outdated-posts-found-exception.php	0	2	0.0%
src/indexables/domain/outdated-post-indexables-list.php	10	12	83.33%
src/indexables/user-interface/schedule-verification-cron-integration.php	0	2	0.0%
src/indexables/application/commands/verify-non-timestamp-indexables-command-handler.php	14	18	77.78%

Files with Coverage Reduction	New Missed Lines	%
src/repositories/indexable-repository.php	1	39.49%
src/generated/container.php	651	0.0%

Totals
Change from base Build 6744816013:	-0.08%
Covered Lines:	13368
Relevant Lines:	27169

---

💛 - Coveralls

[igorschoester]

CR 🏗️

Quite the PR you created here! 🙇

I think there might be more cases in the plugin deactivated hole though. I had to think of when the permalink structure changes.
Might be worth it to check our watchers for cases.
But could very well be done as a follow up of course 😉

I have not checked the tests and their coverage, because it is late and I'm not working tomorrow, but you are. So let's keep that in mind for the next CR round 😉

[pls78]

Shouldn't this be determine_next_verification_action instead?

[pls78]

Shouldn't this be determine_next_verification_action instead?