Migrate Drupal 9 update and fixes

[alxp]

migrate_7x_claw updates

401 errors with Fedora 3 using migrate_7x_claw
Update Migrate Source CSV to 8.x-3.2 #1460

What does this Pull Request do?

Update migrate_7x_claw to be Drupal 9 compatible.

Resolving several long-standing bugs.

Add original PID as a migrate variable, useful for migrate lookup calls when migrating e.g. into paragraph fields.

What's new?

A in-depth description of the changes made by this PR. Technical details and
possible side effects.

• .info.yml updates
• updated syntax of CSV migration to be compatible with current migrate plugin.
• • Changed authentication syntax to match what the download plugin expects.
• Does this change require documentation to be updated?

No

• Does this change add any new dependencies?

No

• Does this change require any other modifications to be made to the repository

No

• Could this change impact execution of existing code?

No

How should this be tested?

Get access to a Fedora 3 instance with solr, follow the documentation to configure the Islandora migrate settings, and run the Islandora Objects migrations.

In particular, where the files migration had be been failing reporting 401 errors, this should no longer happen

Interested parties

Tag (@ mention) interested parties or, if unsure, @Islandora-Devops/committers

[edlington]

I'm running isle-dc on an Ubuntu laptop. I want to play around with migrate_7x_claw on my isle-dc instance. But composer wouldn't install migrate_7x_claw on isle-dc because of version incompatibilities. A kind community member on Islandora Slack pointed me to this branch, which includes a PR that might enable me to install migrate_7x_claw on isle-dc. Short version: after including this in my composer.json and running composer -vvv require roblib/migrate_7x_claw the migrate_7x_claw module installed and enabling it worked on isle-dc! Hallelujah! Here's what I added to composer.json - credit again to the kind community member on Islandora Slack, who shared this with me.

"repositories": [
        {
            "type": "composer",
            "url": "https://packages.drupal.org/8"
        },
       {
           "type": "composer",
           "url": "https://asset-packagist.org"
       },
      {
            "type": "package",
            "package": {
            "name": "roblib/migrate_7x_claw",
              "version": "2.0",
                  "type": "drupal-module",
                  "source": {
                        "url": "https://github.com/roblib/migrate_7x_claw.git",
                        "type": "git",
                        "reference": "migrate-update"
                }

          }
        }
    ],

The install of migrate_7x_claw came with two modules. The other one is called Migrate 7x Claw Feature. There was a problem with it post install. So it installed without an issue. But when I tried to enable Migrate 7x Claw Feature I got a white screen of death with the message: "The website encountered an unexpected error. Please try again later." The error message in the logs is: Drupal\Core\Config\UnsupportedDataTypeConfigException: Invalid data type in config migrate_plus.migration.islandora_files, found in file modules/contrib/migrate_7x_claw/modules/islandora_migrate_7x_claw_feature/config/install/migrate_plus.migration.islandora_files.yml: Duplicate key "application/base64" detected at line 81 (near "application/base64: mme"). in Drupal\Core\Config\FileStorage->read() (line 118 of /var/www/drupal/web/core/lib/Drupal/Core/Config/FileStorage.php).

[RodBruce]

I'm running the Ansible playbook version on an Ubuntu 18.04 lxc container. I followed the same process as @edlington describes. The "composer require" and "drush enable" both worked for us. The problem we ran into was when we were trying to test a migration. It would fail with Class 'Islandora\Tuque\Guzzle\Client' not found. I found this closed issue (Islandora/documentation#954) and followed Islandora/documentation#954 (comment) and that got us past the error.

We do have the 'jonathangreen/tuque' code installed in our main Drupal vendor directory so I'm not sure why it isn't finding that.

[jordandukart]

Is the files migration meant to be nuked here as well?

[elizoller]

@jordandukart you want to take another pass at this one to try to get it in for the release?

[jordandukart]

My bad for not getting back to looking at this. My previous comment about the removal of files is wrong. Looks like the deduping is predicated due to the fact that static_map is only going to find the last value when it constructs the map as an array.

Technically, there could be a "change" of expectation here given that MIME types would return differently prior.

For example:

application/x-gzip: gz
application/x-gzip: gzip

Would now return gz as opposed to gzip for the extension of a file going in. Not sure how much of an actual issue this is but noting accordingly.

[alxp]

Closing due to being against out-of-date branch