Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Check character length of harvested resource urls #2485

Merged
merged 2 commits into from
May 10, 2018
Merged

Check character length of harvested resource urls #2485

Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
79f2472
Check character length of harvested resource urls
janette May 2, 2018
c1bde9f
Merge branch '7.x-1.x' into cdt-112
dafeder May 10, 2018
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
22 changes: 12 additions & 10 deletions modules/dkan/dkan_harvest/dkan_harvest.migrate.inc
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -149,8 +149,8 @@ class HarvestMigration extends MigrateDKAN {

// Disable the logging system of the migration module.
// The module logs every time the processImport() method is called but we
// want to log only once, when the full migration event finishes. If we don't
// do that then we are going to get one log record per batch chunk.
// want to log only once, when the full migration event finishes. If we
// don't do that then we are going to get one log record per batch chunk.
$this->logHistory = FALSE;

// Initialize results.
Expand All @@ -165,7 +165,7 @@ class HarvestMigration extends MigrateDKAN {
'datasets_created' => 0,
'datasets_updated' => 0,
'datasets_failed' => 0,
'datasets_orphaned' => 0
'datasets_orphaned' => 0,
);

// Add Field mappings.
Expand Down Expand Up @@ -210,7 +210,7 @@ class HarvestMigration extends MigrateDKAN {
'endtime' => $event_data['end_time'],
'initialHighwater' => $event_data['initial_highwater'],
'finalHighwater' => $event_data['final_highwater'],
'numprocessed' => $event_data['datasets_processed']
'numprocessed' => $event_data['datasets_processed'],
))
->execute();
}
Expand Down Expand Up @@ -313,8 +313,8 @@ class HarvestMigration extends MigrateDKAN {
// Check the license field.
// Convert URI values to license ID values from hook_license_subscribe(),
// change the value to match the expected key value.
// Clients will need to provide a custom hook_license_subscribe() function if
// they are using different urls than dkan: (i.e. opendatacommons.org or
// Clients will need to provide a custom hook_license_subscribe() function
// if they are using different urls than dkan: (i.e. opendatacommons.org or
// creativecommons.org rather than opendefinition.org)
if (isset($row->license) && $row->license != '') {
$licenses = dkan_dataset_content_types_license_subscribe();
Expand Down Expand Up @@ -1159,10 +1159,12 @@ class HarvestMigration extends MigrateDKAN {
return FALSE;
}

// Finally this URL is determined to be field_link_remote_file
// material.
// Make sure to use the effective URL.
$resource->url = $remoteFileInfo->getEffectiveUrl();
// If the URL is determined to be field_link_remote_file material,
// use the effective URL unless it is more than 255 characters.
// More than 255 will give the 'Data too long for column' error.
if (strlen($remoteFileInfo->getEffectiveUrl()) < 256) {
$resource->url = $remoteFileInfo->getEffectiveUrl();
}
$resource->url_type = 'file';
$format_detected = $remoteFileInfo->getExtension();
}
Expand Down