Duplicates on upload/sync

[grig-23]

Issue

In 8942863-zen, On upload/sync, we get duplicate products created, and these duplicates correspond to errors such as:

Product #3471 is excluded from sync as the product type "variation" is unsupported.

This results in a stub product in Square where 3471 and it's SKU is showing up in what I would call an orphaned state, , as well as a properly synced product, which contains all the variations, #3471 and #3472 etc.

429 rate limiting causes duplicates #108 but we have reset the batch sizes, and rate limiting errors are no longer occurring.

SOR - WooCommerce
Square has a clean DB

The merchant tried cleaning products from Square and re-syncing, only to see more duplicates.

p1730380630885499-slack-C01TYRCJ1QD

[nicdwilson]

I have imported the customers products into a test site, and can replicate the validation error.

Product #3471 is excluded from sync as the product type "variation" is unsupported.

However, my products are not syncing at all to the sandbox: I have no items in the library, WooCommerce is SOR.

I did find some reports that look very similar on the Square forums such as here

[qasumitbagthariya]

QA update:

I attempted to reproduce the issue on my end but was unable to do so. Here's what I tested:

• Platforms: SOR WooCommerce and Square
• Product: Variable
• WordPress Version: 6.7
• PHP Version: 8.0.30
• Web Server: Nginx 1.20.2
• Branch: trunk

[iamdharmesh]

Hi @grig-23,

Thank you for reporting the issue, and @nicdwilson, thanks for providing the additional details.

I’ve tested this across multiple scenarios but haven’t been able to reproduce the issue. However, I was able to replicate the error Product #XXXX is excluded from sync as the product type "variation" is unsupported by making manual changes to the database. Based on my findings, though, this doesn’t appear to be the root cause of the duplication issue.

I’m not sure what we might be missing when attempting to replicate the issue, but I wonder if it could be related to specific product data? 🤔

I have imported the customers products into a test site, and can replicate the validation error.

@nicdwilson, could you share the exported data of the product causing the issue so I can replicate it on my end and investigate further? Any additional details would also be appreciated.

Thank you!

[nicdwilson]

Hi @iamdharmesh - I can give you full access to the site and sandbox. Possibly that would be easier? It took a while to get the whole thing running :)

We have a thread here: p1730380630885499-slack-C01TYRCJ1QD

However, I was able to replicate the error Product #XXXX is excluded from sync as the product type "variation" is unsupported by making manual changes to the database.

I found that the variations were marked for sync, and removing that in the database removed the error. However, I agree: this is not the source of the duplication. But it is interesting that while my site UI showed over 7000 products (variations + products) to be synced, the merchant's site shows only products (3k products) but still throws the error.

My duplication looks a little different from the merchants. My productsare completely duplicated. There is no "stub"

On my item library, for example, I see
Token Item Name SKU
WYDQRV6TTPNK7HBLVYVKUZXF Flood Gate Controller 26712
ALX3OE7HZOXN5I2ZOAA7GXUK Flood Gate Controller 26712

While both appear in the logs, only the second ID appears in the DB. This site uses product ID as SKU.

[iamdharmesh]

Hi @iamdharmesh - I can give you full access to the site and sandbox. Possibly that would be easier? It took a while to get the whole thing running :)

Yes @nicdwilson, that would be very helpful. Thanks.

[iamdharmesh]

Thank you for the details, @nicdwilson. I can see duplicate items in my Square dashboard. I will investigate the issue and raise a PR soon. Thanks!

[jobthomas]

@iamdharmesh any updates for us on a fix?

[iamdharmesh]

Hi @jobthomas 👋 ,

I have raised a draft PR for this and am currently performing final testing. Once completed, the PR will go through the review and QA process before being released. Meanwhile, could you please ask the merchant to adjust the upsert limit and try syncing again? This will help resolve the issue temporarily until the fix is released.

add_filter( 'wc_square_sync_max_objects_per_upsert', function ( $limit ) {
	return 50;
} );

Thank you.

[nicdwilson]

@iamdharmesh Thanks for the PR! Unfortunately the changes to the upsert limits you mention above have in the past had limited success. While reducing 429 errors, it's tough to eliminate them (this is on Square, and they are working on this too), but even without 429 errors we continue to get duplication in some cases. Worth a look at #106 which is advice we received direct from Square while working with one of their top customers with this issue.

[iamdharmesh]

@iamdharmesh Thanks for the PR! Unfortunately the changes to the upsert limits you mention above have in the past had limited success.

Thanks for providing further information on this, @nicdwilson. Yes, I agree that the code shared above is not a complete fix and is more of a temporary workaround, as we currently don’t have any limits in the upsert_new_products process.

Based on my review, the duplication issue was occurring due to two main reasons:

1. 429 Error: This has been addressed in the PR by retrying the same request with the same idempotency key.
2. Successful product creation on Square's side but failure to save Square data, resulting in sending the same products again: This has been addressed in the PR by upserting batches within reasonable limits.

I’ve resolved both of these issues in the PR, which should fix the duplication problem. I am curious to see how well this will work on the merchant's site to help them resolve these duplication issues. Thank you!