Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

migrate_7x_claw exits batch prematurely if xpath query does not find a result in the first batch #1516

Open
nigelgbanks opened this issue May 21, 2020 · 0 comments
Assignees
Labels
Subject: Migration Concerning migration from Islandora 7 to Islandora 2.x.x Type: bug identifies a problem in the software with clear steps to reproduce

Comments

@nigelgbanks
Copy link
Member

There is an edge case wherein, the migration will not run if there is no matching results in the first batch of results. This is due to how the core/modules/migrate/src/MigrateExecutable.php prepares to perform the migration and how DataParserPluginBase deals with non-matching results. Before starting the migration MigrateExecutable rewinds the iterator and checks if it's valid. In our implementation DataParserPluginBase is used as the iterator, when told to rewind, it will do so, but it will iterate over its contents until it finds a the first valid result, since there isn't any guarantee of valid results for the URLs it's told to iterator over. If no valid result is found the iterator itself (DataParserPluginBase) will be considered invalid, as it has no concept of batches. Therefore MigrateExecutable will assume there is no content to migrate.

From MigrateExecutable:

try {
      $source->rewind();
    }
    catch (\Exception $e) {
      $this->message->display(
        $this->t('Migration failed with source plugin exception: @e', ['@e' => $e->getMessage()]), 'error');
      $this->migration->setStatus(MigrationInterface::STATUS_IDLE);
      return MigrationInterface::RESULT_FAILED;
    }
 // .... The check here will fail, even though additional batches exist.
 while ($source->valid()) {
      $row = $source->current();
      $this->sourceIdValues = $row->getSourceIdValues();

      try {
        $this->processRow($row);
        $save = TRUE;
      }
  // ....

Steps to reproduce:

  1. Create a migration (see the example below).
  2. Make sure that it will match against the given query, but will not be in the first 10 results of the query.
  3. Run the migration.
  4. Note that no taxonomy terms were created.
langcode: en
status: true
dependencies:
  enforced:
    module:
      - migrate_7x_claw
      - migrate_plus
      - islandora
id: islandora_untyped_name
class: null
field_plugin_method: null
cck_plugin_method: null
migration_tags: null
migration_group: islandora_7x
label: 'Islandora Untyped Name'
source:
  plugin: islandora
  solr_base_url: 'http://localhost:9999/solr'
  q: 'fedora_datastreams_ms:MODS'
  row_type: MODS
  fedora_base_url: 'http://localhost:8080/fedora'
  data_fetcher_plugin: http
  authentication:
    plugin: basic
    username: fedoraAdmin
    password: fedoraAdmin
  data_parser_plugin: authenticated_xml
  item_selector: '/mods:mods/mods:name[@type = "" or not(@type)]'
  constants:
    creator_uid: 1
  fields:
    -
      name: name
      label: Name
      selector: 'mods:namePart'
  ids:
    name:
      type: string
process:
  name: name
destination:
  plugin: 'entity:taxonomy_term'
  default_bundle: untyped_name
migration_dependencies:
  required: {  }
  optional: {  }
@nigelgbanks nigelgbanks self-assigned this May 21, 2020
@kstapelfeldt kstapelfeldt added Type: bug identifies a problem in the software with clear steps to reproduce Subject: Migration Concerning migration from Islandora 7 to Islandora 2.x.x and removed bug labels Sep 25, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Subject: Migration Concerning migration from Islandora 7 to Islandora 2.x.x Type: bug identifies a problem in the software with clear steps to reproduce
Projects
Development

No branches or pull requests

2 participants