Undefined Offset: in Migration/Step/Eav/Data.php #813

ChameleonDevil · 2020-05-11T07:53:04Z

Before I continue here:

For reference, I created this topic around the same time I added this issue last year:
Why is the newer 2.3.5 Step/Eav/Data.php so much smaller #812

Edit 2021:

Please read the link above for the concern I raised:
Summary: Looking at the current commits (2021) for Step/Eav/Data.php it still has less code (alot old code removed.... and hard-coded quantity $this->progress->start(7); vs OLDER commits: $this->progress->start($this->getIterationsCount()); ) as previous commits I mentioned in that topic; I wondered then, and still do: is this not the reason why Step/Eav/Data.php processing is giving us so much issues? Why were so much attribute processing removed, why are the dynamic getIterationsCount() not used anymore?

Original topic:

Preconditions

Magento 1.7.0.2 database
Magento 2.3.5 database, PHP 7.3

Steps to reproduce

Setup and install a clean new version of Magento 2.3.5
Install Migration Data tool v 2.3.5 composer require magento/data-migration-tool:2.3.5
Run setup:upgrade
4.1 Make sure that var/migration* is deleted so that you have a new migration from start. (in particular the file migration-tool-progress.lock). I would have assumed --reset does the same, but seems not.
Run bin/magento migrate:data path/to/config.xml that has configuration info from your source Magento 1 database and Magento 2 destination.

Expected result

A proper clean migration with no exceptions/errors/halts.

Actual result

Halts immediately on the following error
1.

Notice: Undefined offset: 4 in /var/www/html/mag_new/vendor/magento/data-migration-tool/src/Migration/Step/Eav/Data.php on line 492

Inside src/Migration/Step/Eav/Data.php on line 492: It seems that it is attempting to update the record attribute_group_id with the populated entry inside mapAttributeGroupIdsSourceDest[id], but it is not available inside my data...

$record['attribute_group_id'] = $this->mapAttributeGroupIdsSourceDest[$record['attribute_group_id']];

attribute_group_id for $record is 4, but there seems to be a missing mapAttributeGroupIdsSourceDest[4] corresponding map source/destination

In fact there seems to be more missing indexes when I debugged

[Screenshot, logs]

_{[2020-05-11 07:11:45] main.ERROR: Notice: Undefined offset: 4 in /var/www/html/mag_new/vendor/magento/data-migration-tool/src/Migration/Step/Eav/Data.php on line 492

#0 /var/www/html/mag_new/vendor/magento/data-migration-tool/src/Migration/Step/Eav/Data.php(494): Magento\Framework\App\ErrorHandler->handler(8, 'Undefined offse...', '/var/www/html/m...', 492, Array)

#1 /var/www/html/mag_new/vendor/magento/data-migration-tool/src/Migration/Step/Eav/Data.php(183): Migration\Step\Eav\Data->migrateCustomEntityAttributes()

#2 /var/www/html/mag_new/vendor/magento/data-migration-tool/src/Migration/Mode/AbstractMode.php(82): Migration\Step\Eav\Data->perform()

#3 /var/www/html/mag_new/vendor/magento/data-migration-tool/src/Migration/Mode/Data.php(124): Migration\Mode\AbstractMode->runStage(Object(Migration\Step\Eav\Data), 'EAV Step', 'data migration')

#4 /var/www/html/mag_new/vendor/magento/data-migration-tool/src/Migration/Mode/Data.php(69): Migration\Mode\Data->runData(Array, 'EAV Step')

#5 /var/www/html/mag_new/vendor/magento/data-migration-tool/src/Migration/Console/MigrateDataCommand.php(59): Migration\Mode\Data->run()

#6 /var/www/html/mag_new/vendor/symfony/console/Command/Command.php(255): Migration\Console\MigrateDataCommand->execute(Object(Symfony\Component\Console\Input\ArgvInput), Object(Symfony\Component\Console\Output\ConsoleOutput))

#7 /var/www/html/mag_new/vendor/magento/framework/Interception/Interceptor.php(58): Symfony\Component\Console\Command\Command->run(Object(Symfony\Component\Console\Input\ArgvInput), Object(Symfony\Component\Console\Output\ConsoleOutput))

#8 /var/www/html/mag_new/vendor/magento/framework/Interception/Interceptor.php(138): Migration\Console\MigrateDataCommand\Interceptor->___callParent('run', Array)

#9 /var/www/html/mag_new/vendor/magento/framework/Interception/Interceptor.php(153): Migration\Console\MigrateDataCommand\Interceptor->Magento\Framework\Interception{closure}(Object(Symfony\Component\Console\Input\ArgvInput), Object(Symfony\Component\Console\Output\ConsoleOutput))

#10 /var/www/html/mag_new/generated/code/Migration/Console/MigrateDataCommand/Interceptor.php(104): Migration\Console\MigrateDataCommand\Interceptor->___callPlugins('run', Array, Array)

#11 /var/www/html/mag_new/vendor/symfony/console/Application.php(1001): Migration\Console\MigrateDataCommand\Interceptor->run(Object(Symfony\Component\Console\Input\ArgvInput), Object(Symfony\Component\Console\Output\ConsoleOutput))

#12 /var/www/html/mag_new/vendor/symfony/console/Application.php(271): Symfony\Component\Console\Application->doRunCommand(Object(Migration\Console\MigrateDataCommand\Interceptor), Object(Symfony\Component\Console\Input\ArgvInput), Object(Symfony\Component\Console\Output\ConsoleOutput))

#13 /var/www/html/mag_new/vendor/magento/framework/Console/Cli.php(115): Symfony\Component\Console\Application->doRun(Object(Symfony\Component\Console\Input\ArgvInput), Object(Symfony\Component\Console\Output\ConsoleOutput))

#14 /var/www/html/mag_new/vendor/symfony/console/Application.php(147): Magento\Framework\Console\Cli->doRun(Object(Symfony\Component\Console\Input\ArgvInput), Object(Symfony\Component\Console\Output\ConsoleOutput))

#15 /var/www/html/mag_new/bin/magento(23): Symfony\Component\Console\Application->run()

#16 {main} [] []}

Additional notes

** This is a clean installation of v2.3.5 Magento as described above, with no pending migration-tool-progress.lock inside var/
My possible related query [ - regarding a much smaller Eav/Data.php file](Why is the newer 2.3.5 Step/Eav/Data.php so much smaller #812)

Questions

Is the map code from the last few commits that should append array mapAttributeGroupIdsSourceDest with entries not missing some processing that used to be there? (See my related query above) (I have done previous migrations pre v.2.3 successfully using the same source Magento 1 database)
Is my issue that I need to map groups as well? Where do I do this?

The text was updated successfully, but these errors were encountered:

KZNcode · 2020-05-11T15:41:12Z

Having a very similar issue.

[2020-05-11 14:37:40][INFO][mode: data][stage: data migration][step: EAV Step]: started
100% [============================] Remaining Time: < 1 sec
In ErrorHandler.php line 61:
                                                                                                                                                                 
  Notice: Undefined offset: 18 in /home/public_html/vendor/magento/data-migration-tool/src/Migration/Step/Eav/Data.php on line 492

Destination: Clean installed Magento 2.3.5-p1; data-migration-tool 2.3.5;
Source: Magento 1.9.4.5

Previous magento versions import worked fine:

Destination: Clean installed Magento 2.3.4; data-migration-tool 2.3.4;
Source: Magento 1.9.4.4

ChameleonDevil · 2020-05-12T08:15:25Z

Having a very similar issue.
[2020-05-11 14:37:40][INFO][mode: data][stage: data migration][step: EAV Step]: started
100% [============================] Remaining Time: < 1 sec
In ErrorHandler.php line 61:
                                                                                                                                                                 
  Notice: Undefined offset: 18 in /home/public_html/vendor/magento/data-migration-tool/src/Migration/Step/Eav/Data.php on line 492
Destination: Clean installed Magento 2.3.5-p1; data-migration-tool 2.3.5;
Source: Magento 1.9.4.5

Previous magento versions import worked fine:

Destination: Clean installed Magento 2.3.4; data-migration-tool 2.3.4;
Source: Magento 1.9.4.4

This is most definitely the same issue, line 492 as explained above the array does not contain an item with index 18.

The method createMapProductAttributeGroupIds() populates mapAttributeGroupIdsSourceDest array but skips your group id 18. For me it was ID 4, where group 4 was '4-Meta Information', but the actual source database group id 4 is 'General Information', so '4-General Information'. There seems to be never a 'General Information' item in my migration, therefor mapAttributeGroupIdsSourceDest never contains an ID 4 when code on line 492 processes.

Potential fix - but read notes below
I have created a commit to check for non-existing key. There are two changes in migrateCustomEntityAttributes(). This essentially just makes sure to skip non-existing IDs. After applying this to my local installation and running migrate:data on a backed up new M2.3.5 database, all the other steps continued without issue - took about 40 minutes.

Please note:
I do not know which custom attributes we are now skipping on import with that commit, thats why my query of #812 still remains. I absolutely have no idea yet, and if the maintainers of this project could just explain that issue in detail.

victor-v-rad · 2020-05-13T12:28:06Z

'Undefined offset' error was fixed in ea9cc9f and merged into 2.3-develop branch

ChameleonDevil · 2020-05-18T07:36:33Z

Yeah I fixed this with a similar way on my branch. Thanks for update!

Since fixing this, I am now trying to figure out why my categories do not show,

Every single one of them returns error 404. Anyways topic for another day

yogeshkhasturi · 2020-05-26T13:55:29Z

@ChameleonDevil Yes same issue here. products data is empty and category mapping the is there but form frontend it says "NO PRODUCT"

Did you find any solution?

ChameleonDevil · 2020-05-26T14:12:23Z

@yogeshkhasturi

I do not know if you really have the same problem I had here, but in my description I commented this:

Potential fix - but read notes below
Go to that section above and read about the solution. I also believe that @victor-v-rad added a similar commit for this issue. Everything related to this issue is in my top comment, with my commit and @victor-v-rad replying later on with official commit.

This definitely fixed undefined index issues on this step, I have ran several migration items afterwards. Other failures, but not related to this issue anymore.

yogeshkhasturi · 2020-05-26T16:28:20Z

@ChameleonDevil

This definitely fixed undefined index issues on this step, I have ran several migration items afterwards. Other failures, but not related to this issue anymore.

Even my migration was completed after ea9cc9f this fix but something was affected with the attribute group in my database. the database available in the database but the relation of the product with attributes is not there, if you can help with debugging it would be appreciable.

Thanks,

ChameleonDevil · 2020-05-27T14:59:19Z

Hi @yogeshkhasturi

Sorry just saw your message now.

Glad to hear that it fixed your first issue (of many :P).

This eav_attribute_group issue that you mentioned as far as I remember only occurs when you try to run bin/magento migrate:data --reset again once a previous migration already migrated all the attributes. I seem to recall the migration tool confused with the IDs of the groups, so just restart from the beginning.

What I did for myself to test and restore migrations (I have alot of other issues still not working :( ) was sort of the following:

Install Magento 2.3.5 completely, setup modules etc, setup backend settings that you want. Once you have a working database before even migrating M1 data, make a backup of this M2 database state.... because you will most likely restart alot of times because of migration issues.
So create a full m2_full_install_before_migration.sql backup.
Create the same for Magento 1 database - until you are at the point where you can migrate without failures, you can restore this as well. NOTE: Delta migration is useless until you have successful migrate:data.
DROP M2 migrated database and restore from SQL in step 1.
VERY IMPORTANT: *Delete the var/migration ** entries in Magento 2 folder. These files are used by the tool it seems to keep track of where you stopped previously. So since the migration does not work 100% yet, just restart from the beginning. (this might also be why you have group issues). I always delete these files.
So a complete DROP/RECREATE, RESTORE M2 DB, setup:upgrade, migrate data from M1 copy (locally) takes around an hour in my case because I have done it so many times.
I would ignore migrate:delta for now if I were you, you will have to restart so many times. Get data working, make notes of issues that are specific to your instance, so that you could do things step by step. My current list before even getting to the point where I am able to run migrate:delta is roughly 15 things that I have to change manually in the database beforehand!

Good luck!

yogeshkhasturi · 2020-05-27T18:50:39Z

@ChameleonDevil Thank you for the guidance this step was really new and seems to be helpful.

*VERY IMPORTANT: Delete the var/migration ** entries in Magento 2 folder. These files are used by the tool it seems to keep track of where you stopped previously. So since the migration does not work 100% yet, just restart from the beginning. (this might also be why you have group issues). I always delete these files.

estebansmith-ris · 2020-06-08T03:49:09Z

@ChameleonDevil I'm having the same issue with the 404 did you fix this issue?

ChameleonDevil · 2020-06-08T08:00:07Z

@ChameleonDevil I'm having the same issue with the 404 did you fix this issue?

@estebansmith-ris , Are you referring to categories that I had issues with error 404? Then yes, but since you are in this Eav/Data.php issue thread, Im not sure what you are talking about - there are plenty of 404 issues in M2.

If it is related to Categories, I am almost certain this issue I created a solution for #815 I had was related to the 404s for categories as well. (It was about two weeks ago, and since then trying to solve so many extra issues, that I can't recall.)

jerry-shopcommerce · 2020-06-12T15:02:29Z

I had the same issue with a migration from magento 1.9.4.5 to 2.3.5 today.

This comment gave me the idea to change the name of the default product attribute set to "Default" in the magento1 database: #112 (comment)

So far it seems to fix my problem.

RCady · 2020-06-25T16:44:22Z

I am having this issue in a migration from 1.9.3.2 to 2.3.5-p1. However, I have used the develop branches fix for the 'undefined offset' error and the migration completes. However, one of my custom product attribute sets has no attribute groups assigned to it and I'm thinking that this fix is part of the reason. Is it possible that ignoring these errors is causing my attribute groups to not be imported properly?

ChameleonDevil · 2020-06-26T11:26:30Z

When you do not have that fix applied, the data migration immediately fails, so that is not an option. At the point that fix is applied in Step/Eav/Data/migrateCustomEntityAttributes() the current implementation was created as follows by maintainer:

`if (!isset($this->mapAttributeGroupIdsSourceDest[$record['attribute_group_id']])) {
            continue;
        }`

The processing of the mapAttributeGroupIdsSourceDest beforehand in code seems to not have any group Ids mapped - no idea why.

There is this new updated commit 4 days ago (I assume this is the one you have) : #af40def29438fb that added another new condition:

if (empty($customAttributeIds)) { return; }

I seem to recall having this issue as well at one point where $customAttributeIds are somehow empty. See the history of the commits of Migration/Step/Eav/Data.php here : commits history.

The original issue discussed here was fixed in #ea9cc9f21d3634dd3695fd88f9c869d9535cb6a6.

Maybe you should revert one step commit back from the newest one and see if it changes anything?

My concern with this whole Step/Eav/Data.php file content was discussed here where I noticed some strange things: Why is the newest Eav Data file so much smaller.

Go read there, I do not even know if the current Data.php file we have are accurate - alot of processing code got removed in the commits I mention there. Also -- why is this there $this->progress->start(7); when previous older commits had $this->progress->start($this->getIterationsCount()); - hardcoded quantity vs dynamic.

Finally I just want to say good luck with Eav migration - it is a nightmare when you later discover all your missing attribute data.
Hints: (Attribute sets vs attribute data), catalog_product_entity_int, catalog_product_super_link etc.

mhafizumer · 2020-07-15T10:13:26Z

'Undefined offset' error was fixed in ea9cc9f and merged into 2.3-develop branch

@victor-v-rad I have latest code,

if (!isset($this->mapAttributeGroupIdsSourceDest[$record['attribute_group_id']])) { continue; }

But Still giving me error

Notice: Undefined offset: 19 in /var/www/html/vendor/magento/data-migration-tool/src/Migration/Step/Eav/Data.php on line 286

What is the problem? even not found attribute_group_id is on `continue;

PascalBrouwers · 2020-09-18T09:12:42Z

I'm having the same issue, but then for category attributes. It seems that those attribute groups are never processed. Only products are being processed in createMapProductAttributeGroupIds().

hewgim · 2021-08-23T17:31:45Z

@victor-v-rad

We were also having the issue with:

Notice: Undefined offset: 7 in /var/www/html/mag_new/vendor/magento/data-migration-tool/src/Migration/Step/Eav/Data.php on line 492

We tried the fix mentioned above

if (!isset($this->mapAttributeGroupIdsSourceDest[$record['attribute_group_id']])) { continue; }

but this resulted in loss of data. Product names, descriptions etc didn't get migrated over for one of our attribute sets.

In the end we noticed that the attribute set with the missing data had originally been named "Default" and had been renamed to "Clothing". We renamed it back to "Default" and removed the code fix. When we reran the migration it completed error free. I suspect you have to have an attribute set named 'Default' as per the original Magento 1 installation.

Also if it helps anyone, we were migrating from 1.9.4.5 to 2.3.7-p1 but because there is no migration tool above 2.3.5 we used that instead and it seemed to work fine.

Hope that helps someone

Regards

Mark

ChameleonDevil · 2021-09-08T09:16:44Z

@victor-v-rad

We were also having the issue with:

Notice: Undefined offset: 7 in /var/www/html/mag_new/vendor/magento/data-migration-tool/src/Migration/Step/Eav/Data.php on line 492

We tried the fix mentioned above

if (!isset($this->mapAttributeGroupIdsSourceDest[$record['attribute_group_id']])) { continue; }

but this resulted in loss of data. Product names, descriptions etc didn't get migrated over for one of our attribute sets.

In the end we noticed that the attribute set with the missing data had originally been named "Default" and had been renamed to "Clothing". We renamed it back to "Default" and removed the code fix. When we reran the migration it completed error free. I suspect you have to have an attribute set named 'Default' as per the original Magento 1 installation.

Also if it helps anyone, we were migrating from 1.9.4.5 to 2.3.7-p1 but because there is no migration tool above 2.3.5 we used that instead and it seemed to work fine.

Hope that helps someone

Regards

Mark

@hewgim

Can you perhaps please send us the full Migration/Step/Eav/Data.php that you managed to successfully migrate with.

Current commits (including the latest 2.4-develop) also uses
if (!isset($this->mapAttributeGroupIdsSourceDest[$record['attribute_group_id']])) { continue; }

So, new migrations, even though it completes, does not really have entries in eav_entity_attribute that should attach eav_attribute_set with the different entities....

Before I continue, I am just putting this here for people skipping the top issue:

For reference, I created this topic around the same time I added this issue last year:
Why is the newer 2.3.5 Step/Eav/Data.php so much smaller #812

Please read the link above for the concern I raised:
Summary: Looking at the current commits (2021) for Step/Eav/Data.php it still has less code (alot old code removed.... and hard-coded quantity $this->progress->start(7); vs OLDER commits: $this->progress->start($this->getIterationsCount()); ) as previous commits I mentioned in that topic; I wondered then, and still do: is this not the reason why Step/Eav/Data.php processing is giving us so much issues? Why were so much attribute processing removed, why are the dynamic getIterationsCount() not used anymore?

@hewgim, can you please assist us here with your last reply:

It would be great if you clear up this for us:

In the end we noticed that the attribute set with the missing data had originally been named "Default" and had been renamed to "Clothing". We renamed it back to "Default" and removed the code fix. When we reran the migration it completed error free. I suspect you have to have an attribute set named 'Default' as per the original Magento 1 installation.

"Default" as attribute set has 8x Default as name:
(I am talking about the Magento 1 Source database):

EXPLAIN eav_attribute_set;

attribute_set_id
entity_type_id
attribute_set_name
sort_order

SELECT * FROM eav_attribute_set WHERE attribute_set_name = 'Default';

Each entity_type_id has its own Default attribute set, so there are 8x:

Results set:

1	1	Default	1
2	2	Default	1
3	3	Default	1
4	4	Default	1
5	5	Default	1
6	6	Default	1
7	7	Default	1
8	4	Default	1

When you mention '"Default" had been renamed to "Clothing"', do you mean your source Magento 1 database entry was renamed to Clothing or Magento 2 destination?

This is not clear from your explanation above,

If it's in the source database: then I think I am out of luck since they are already called Default for all entity_type_id entries; I also don't know what to do, as we have over 120x different custom attribute sets (product attribute sets) in addition to those 8x built-in ones...

Would you perhaps please give us a workflow of what you did in the source & destination database to prepare (before running migration).

KZNcode mentioned this issue May 11, 2020

Why is the newer 2.3.5 Step/Eav/Data.php so much smaller #812

Open

ChameleonDevil closed this as completed May 18, 2020

KZNcode mentioned this issue May 25, 2020

Undefined offset: error at [stage: data migration][step: EAV Step] for Migration from Magento 1.7.0.2 to Magento 2.3.5 #816

Open

ChameleonDevil reopened this May 27, 2020

p810 mentioned this issue Jun 12, 2020

Undefined index: 1-Default in Migration\Step\Eav\Data on line 624 #825

Closed

ChameleonDevil mentioned this issue Jun 15, 2020

Undefined offset: 11 in vendor/magento/data-migration-tool/src/Migration/Step/Eav/Data.php on line 651 #824

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Undefined Offset: in Migration/Step/Eav/Data.php #813

Undefined Offset: in Migration/Step/Eav/Data.php #813

ChameleonDevil commented May 11, 2020 •

edited

Loading

KZNcode commented May 11, 2020

ChameleonDevil commented May 12, 2020 •

edited

Loading

victor-v-rad commented May 13, 2020

ChameleonDevil commented May 18, 2020

yogeshkhasturi commented May 26, 2020

ChameleonDevil commented May 26, 2020 •

edited

Loading

yogeshkhasturi commented May 26, 2020

ChameleonDevil commented May 27, 2020

yogeshkhasturi commented May 27, 2020

estebansmith-ris commented Jun 8, 2020

ChameleonDevil commented Jun 8, 2020 •

edited

Loading

jerry-shopcommerce commented Jun 12, 2020 •

edited

Loading

RCady commented Jun 25, 2020

ChameleonDevil commented Jun 26, 2020 •

edited

Loading

mhafizumer commented Jul 15, 2020 •

edited

Loading

PascalBrouwers commented Sep 18, 2020

hewgim commented Aug 23, 2021

ChameleonDevil commented Sep 8, 2021 •

edited

Loading

Undefined Offset: in Migration/Step/Eav/Data.php #813

Undefined Offset: in Migration/Step/Eav/Data.php #813

Comments

ChameleonDevil commented May 11, 2020 • edited Loading

Before I continue here:

Edit 2021:

Original topic:

Preconditions

Steps to reproduce

Expected result

Actual result

Additional notes

Questions

KZNcode commented May 11, 2020

ChameleonDevil commented May 12, 2020 • edited Loading

victor-v-rad commented May 13, 2020

ChameleonDevil commented May 18, 2020

yogeshkhasturi commented May 26, 2020

ChameleonDevil commented May 26, 2020 • edited Loading

yogeshkhasturi commented May 26, 2020

ChameleonDevil commented May 27, 2020

yogeshkhasturi commented May 27, 2020

estebansmith-ris commented Jun 8, 2020

ChameleonDevil commented Jun 8, 2020 • edited Loading

jerry-shopcommerce commented Jun 12, 2020 • edited Loading

RCady commented Jun 25, 2020

ChameleonDevil commented Jun 26, 2020 • edited Loading

mhafizumer commented Jul 15, 2020 • edited Loading

PascalBrouwers commented Sep 18, 2020

hewgim commented Aug 23, 2021

ChameleonDevil commented Sep 8, 2021 • edited Loading

Before I continue, I am just putting this here for people skipping the top issue:

@hewgim, can you please assist us here with your last reply:

ChameleonDevil commented May 11, 2020 •

edited

Loading

ChameleonDevil commented May 12, 2020 •

edited

Loading

ChameleonDevil commented May 26, 2020 •

edited

Loading

ChameleonDevil commented Jun 8, 2020 •

edited

Loading

jerry-shopcommerce commented Jun 12, 2020 •

edited

Loading

ChameleonDevil commented Jun 26, 2020 •

edited

Loading

mhafizumer commented Jul 15, 2020 •

edited

Loading

ChameleonDevil commented Sep 8, 2021 •

edited

Loading