Image import exclude-flag not set consistently

[mblarsen]

The 'exclude' flag is not set when running in create mode but only in update mode.

I tried:

1. Import first new product with create
2. Run same again in create mode—still no change
3. Only when running in update mode the exclude flag was set.

With create:

With update:

(Note: that order isn't set correctly as well)

Both views above are for a store view not the default.

I tried running several times with create deleting the products between each run. Sometimes the exclude flag would be set.

This is the input:

websites,store,sku,image,small_image,thumbnail,media_gallery
world,world_english,NY99-01-01-32,+http://192.168.40.40:8000/images/products/n8-08-06-2-1000pix-a4ca8-big.jpg,-http://192.168.40.40:8000/images/products/n8-08-06-2-1000pix-a4ca8-small.jpg,-http://192.168.40.40:8000/images/products/n8-08-06-2-1000pix-a4ca8-thumb.jpg,+http://192.168.40.40:8000/images/products/n8-08-06-3-1000pix-a2729-big.jpg;+http://192.168.40.40:8000/images/products/n8-08-06-1000pix-b69fb-big.jpg
world,world_english,NY99-01-01-33,+http://192.168.40.40:8000/images/products/n8-08-06-2-1000pix-a4ca8-big.jpg,-http://192.168.40.40:8000/images/products/n8-08-06-2-1000pix-a4ca8-small.jpg,-http://192.168.40.40:8000/images/products/n8-08-06-2-1000pix-a4ca8-thumb.jpg,+http://192.168.40.40:8000/images/products/n8-08-06-3-1000pix-a2729-big.jpg;+http://192.168.40.40:8000/images/products/n8-08-06-1000pix-b69fb-big.jpg
world,world_english,NY99-01-01-34,+http://192.168.40.40:8000/images/products/n8-08-06-2-1000pix-a4ca8-big.jpg,-http://192.168.40.40:8000/images/products/n8-08-06-2-1000pix-a4ca8-small.jpg,-http://192.168.40.40:8000/images/products/n8-08-06-2-1000pix-a4ca8-thumb.jpg,+http://192.168.40.40:8000/images/products/n8-08-06-3-1000pix-a2729-big.jpg;+http://192.168.40.40:8000/images/products/n8-08-06-1000pix-b69fb-big.jpg
world,world_english,NY99-04-BLAX-34,+http://192.168.40.40:8000/images/products/n8-04-01-2-1000pix-big.jpg,-http://192.168.40.40:8000/images/products/n8-04-01-2-1000pix-small.jpg,-http://192.168.40.40:8000/images/products/n8-04-01-2-1000pix-thumb.jpg,+http://192.168.40.40:8000/images/products/n8-04-01-3-1000pix-big.jpg;+http://192.168.40.40:8000/images/products/n8-04-01-1000pix-big.jpg

[mblarsen]

I've tested some more and it seems pretty random what gets set.

[mblarsen]

In addition:

I tried importing the same data-set multiple times. The main product (configurable) stabilizes after 2 imports but will never set the exclude flags for small and thumbnail for the default config values. For the stores it flip flops a few times and then eventually the exclude flags is set correctly and.

For the simple products (associated with the main product) I had to run the import 4-5 times before the exclude flags were constantly correctly set for the stores. The default config values never gets exclude flags set.

@dweeves hope this can give you some idea about what is going on. If there no outlook to more stability please let me know. I'll see if I can come up with a new image processing plugin.

[mblarsen]

5 hours later. It seems that magmi leaves duplicate rows in catalog_product_entity_media_gallery and or catalog_product_entity_media_gallery_value and when you visit the product page in the Magento Admin some kind of cleanup is happening which results in most of the duplicated data is removed entirely. So the aim should be to figure out why there is duplicate data.

I noticed that the duplication is reduced with each run. The two said tables shrink until the reach what is likely the "real" count of rows needed to express the state.

[mblarsen]

So two odd things.

1. catalog_product_entity_varchar only contains values for store 0 and 2 (image, small, thumb). I noticed that when creating images there are both storeid and targetsid and for store 1 those values are 0 and 1 respectively whereas for store 2 they are both 2. As far as I can see the reason why there are now rows for store 1 in catalog_product_entity_varchar is that the image process plugin doesn't handle the insertion of this part but lets it be handled by the underlying engine which uses storeid and not targetsid.
2. When running the second time rows for store 1 are created. Doh?!? What's going on?

(I don't know if this affects duplication)

Regarding what happens on viewing a product in Magento I've debugged a little and here is a bunch of findings:

SELECT * FROM catalog_product_entity_media_gallery mg
LEFT JOIN catalog_product_entity_media_gallery_value mgv ON mg.value_id=mgv.value_id
WHERE
    mg.entity_id = 92
    AND mgv.store_id IN (1,2)
ORDER BY mg.entity_id, attribute_id
;

/*

BEFORE

store 1
701 88  92  /n/1/n11-08-00-daisykids-1a-800pix-big.jpg  701 1   NULL    0   0
702 88  92  /n/1/n11-08-00-daisykids-1a-800pix-small.jpg    702 1   NULL    0   1
703 88  92  /n/1/n11-08-00-daisykids-1a-800pix-thumb.jpg    703 1   NULL    0   1
704 88  92  /n/1/n11-08-00-daisykids-2a-800pix-big.jpg  704 1   NULL    0   0

store 2
727 88  92  /n/1/n11-08-00-daisykids-1a-800pix-small.jpg    727 2   NULL    1   1
728 88  92  /n/1/n11-08-00-daisykids-1a-800pix-thumb.jpg    728 2   NULL    2   1
704 88  92  /n/1/n11-08-00-daisykids-2a-800pix-big.jpg  704 2   NULL    3   0
726 88  92  /n/1/n11-08-00-daisykids-1a-800pix-big.jpg  726 2   NULL    0   0

AFTER

store 1
703 88  92  /n/1/n11-08-00-daisykids-1a-800pix-thumb.jpg    703 1   NULL    0   1
704 88  92  /n/1/n11-08-00-daisykids-2a-800pix-big.jpg  704 1   NULL    0   0

store 2
704 88  92  /n/1/n11-08-00-daisykids-2a-800pix-big.jpg  704 2   NULL    3   0
726 88  92  /n/1/n11-08-00-daisykids-1a-800pix-big.jpg  726 2   NULL    0   0
727 88  92  /n/1/n11-08-00-daisykids-1a-800pix-small.jpg    727 2   NULL    1   1

*/


SELECT * FROM catalog_product_entity_media_gallery ORDER BY entity_id, value_id;

/*
BEFORE
701 88  92  /n/1/n11-08-00-daisykids-1a-800pix-big.jpg    x store = 1    --- deleted duplicate
702 88  92  /n/1/n11-08-00-daisykids-1a-800pix-small.jpg  x store = 1    --- deleted duplicate
703 88  92  /n/1/n11-08-00-daisykids-1a-800pix-thumb.jpg  x store = 1 
704 88  92  /n/1/n11-08-00-daisykids-2a-800pix-big.jpg    √ store = 1+2
726 88  92  /n/1/n11-08-00-daisykids-1a-800pix-big.jpg    √ store = 2
727 88  92  /n/1/n11-08-00-daisykids-1a-800pix-small.jpg  √ store = 2
728 88  92  /n/1/n11-08-00-daisykids-1a-800pix-thumb.jpg  √ store = 2    --- deleted duplicate

AFTER
703 88  92  /n/1/n11-08-00-daisykids-1a-800pix-thumb.jpg  √ store = 1
704 88  92  /n/1/n11-08-00-daisykids-2a-800pix-big.jpg    √ store = 1+2
726 88  92  /n/1/n11-08-00-daisykids-1a-800pix-big.jpg    √ store = 2
727 88  92  /n/1/n11-08-00-daisykids-1a-800pix-small.jpg  √ store = 2
*/

SELECT vc.entity_id, a.attribute_code, vc.attribute_id, vc.store_id, value FROM catalog_product_entity_varchar vc
JOIN eav_attribute a ON vc.attribute_id = a.attribute_id
WHERE a.attribute_id IN (85,86,87,88)
    AND entity_id IN (92)
ORDER BY vc.entity_id, vc.attribute_id, vc.store_id;

/*
92  image   85  0   /n/1/n11-08-00-daisykids-1a-800pix-big.jpg
92  image   85  2   /n/1/n11-08-00-daisykids-1a-800pix-big.jpg
92  small_image 86  0   /n/1/n11-08-00-daisykids-1a-800pix-small.jpg
92  small_image 86  2   /n/1/n11-08-00-daisykids-1a-800pix-small.jpg
92  thumbnail   87  0   /n/1/n11-08-00-daisykids-1a-800pix-thumb.jpg
92  thumbnail   87  2   /n/1/n11-08-00-daisykids-1a-800pix-thumb.jpg
*/

You can see that before doing anything magmi has created multiple rows to the same image. Some for store 1 and some for store 2 (remember store 1 and store 2 are not in same row).

After viewing the product page in Magento, thus involuntarily cleaning up, the duplicates are removed. I cannot spot any pattern in how they are removed, - both store 1 and store 2 specific values are removed. But naturally since they are removed so are the 'exclude' values.

[mblarsen]

I found what appears to be a bug or at least what is causing the duplication (and shrinking effect).

catalog_product_entity_varchar (image, small, thumb) is only created after handleVarcharAttribute() returns so naturally when getImage() tries to fetch an existing image in medai_gallery using refid (e.g. image, small, thumb) then the return image id will be null.

The code is:

if (... if image with matching refid is found use that ...)
else (... find just by attribute and entity id etc ...)

if (.. no image id was found still create it ...)  // this will be a duplicate since it may exist without refid

It should be:

if (... if image with matching refid is found use that ...)

if  (.. if not found using refid then find just by attribute and entity id etc ...)

if (.. no image id was found still create it ...)

Going about it this way removes duplication and Magneto behaves nicely. Not deleting duplicate images without sorting out relations.

I'll put this small change in a PR.

There is still one problem left. That is figuring out why catalog_product_entity_varchar rows are not created for store 1

websites, store
world,store1,sku1
world,store2,sku1

So in the above the first row results in store 0 (admin) not store 1 (my store1). Whereas store2 with id 2 is created as it should.