Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Post import status #3929

Merged
merged 9 commits into from
Apr 11, 2023
Merged

Post import status #3929

merged 9 commits into from
Apr 11, 2023

Conversation

kaise-lafrai
Copy link
Contributor

fixes [org/repo/issue#]

  • Test coverage exists
  • Documentation exists

QA Steps

  • Create a new dataset here /node/add/data with the following csv file: drugproductsinthemedicaiddrugrebateprogram3q-11072022_7_0_4.csv
  • Navigate to the Datastore Import Status page here /admin/dkan/datastore/status and confirm you see a new row for the new dataset and a new column at the end of the row called "Post Import" with the value "waiting".
  • Navigate here /admin/dkan/data-dictionary/settings and select "Disabled" in the Dictionary Mode and Save Configuration.
  • Run CRON with ddev drush cron, confirm no errors.
  • Navigate to the Datastore Import Status page here /admin/dkan/datastore/status and confirm you see the "Post Import" column and a value of "waiting Data-Dictionary Disabled".
  • Navigate here to create a new Data Dictionary /node/add/data?schema=data-dictionary, entered values below and save.
  • Identifier: Post-Import-Test
  • Title: Post Import Test
  • Dictionary Fields (Name): reactivation_date
  • Title: Reactivation Date
  • Type: datetime
  • Format: %m/%d/%Y %H:%M:%S %p
  • Navigate here /admin/dkan/data-dictionary/settings, change Dictionary Mode to "Sitewide", Enter "Post-Import-Test" in Sitewide Dictionary ID.
  • Navigate back to the dataset that was previously created. Change the Last Update time, and Save.
  • Navigate here /admin/dkan/datastore/status and confirm Post Import is set to "waiting"
  • Run CRON with ddev drush cron, confirm you see the following error outputed
    [error] SQLSTATE[HY000]: General error: 1411 Incorrect datetime value: '09/07/2017 12:00:00 AM' for function str_to_date: UPDATE "datastore_ada767d52d5d6bd643ca8fba32725402" SET "reactivation_date"=STR_TO_DATE(reactivation_date, :date_format); Array ( [:date_format] => %m/%d/%Y %H:%i:%s %p )
  • Navigate to the Datastore Import Status page here /admin/dkan/datastore/status and confirm you see the "Post Import" column and a value of:
    error SQLSTATE[HY000]: General error: 1411 Incorrect datetime value: '09/07/2017 12:00:00 AM' for function str_to_date: UPDATE "datastore_ada767d52d5d6bd643ca8fba32725402" SET "reactivation_date"=STR_TO_DATE(reactivation_date, :date_format); Array ( [:date_format] => %m/%d/%Y %H:%i:%s %p )
  • Navigate back to the data dictionary previously created and alter the format to the following: %m/%d/%Y %I:%M:%S %p
  • Navigate back to the dataset that was previously created. Change the Last Update time, and Save.
  • Navigate here /admin/dkan/datastore/status and confirm Post Import is set to "waiting"
  • Run CRON with ddev drush cron, confirm no errors.
  • Navigate to the Datastore Import Status page here /admin/dkan/datastore/status and confirm you see the "Post Import" column and a value of "done".
  • Navigate back to the dataset that was previously created. Change the Last Update time, and change the state to "DRAFT", and save.
  • Navigate here /admin/dkan/datastore/status and confirm a new row is created with a "revision" state of draft, confirm Post Import column displays "waiting". Confirm the original published row is still set to done.
  • Run CRON with ddev drush cron, confirm no errors.
  • Navigate to the Datastore Import Status page here /admin/dkan/datastore/status and confirm you see the "Post Import" column and a value of "done" for both the draft and published rows.
  • Run the following command ddev drush dkan:datastore:drop #resource_id replacing #resource_id with the resource_id of the distribution. Which can be found using ddev drush dkan:dataset-info.
  • Confirm you see the following messages:
  • [notice] Successfully dropped the datastore for resource cc13cf29e76aba01c8bff97789b6fe27
    [notice] Successfully removed the post import job status for resource cc13cf29e76aba01c8bff97789b6fe27
  • Navigate here /admin/dkan/datastore/status and confirm that the last revision row, which in our case is the draft row is now all set to "waiting" and nothing was altered for the published row.
  • Navigate back to the dataset that was previously created. Change the Last Update time and Save.
  • Run CRON with ddev drush cron, confirm no errors.
  • Navigate to the Datastore Import Status page here /admin/dkan/datastore/status and confirm you see the "Post Import" column and a value of "done" for both the draft and published rows.
  • Perform any additional testing you think is valuable.

… display errors on datastore import status page
…lso remove the related status row when the datastore is dropped, added new service function for removing rows from the dkan_post_import_job_status table, added additional key to the dkan_post_import_job_status table to store the resource_version as this will better handle revisions created for each dataset.
…us, adjusted css to handle color background display for the new post import status of NA
@kaise-lafrai kaise-lafrai requested a review from janette April 6, 2023 14:53
@janette janette merged commit 12c94d7 into 2.x Apr 11, 2023
@janette janette deleted the post-import-status branch April 25, 2023 04:14
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants