Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Switchover to using Registry API for generating context products json #1036

Merged
merged 7 commits into from
Oct 18, 2024

Conversation

jordanpadams
Copy link
Member

@jordanpadams jordanpadams commented Oct 17, 2024

🗒️ Summary

Query Registry API to generate the context products JSON file used for context product validation

⚙️ Test Data and/or Report

$ validate -u

PDS Validate Tool Report

Configuration:
   Version     3.6.0-SNAPSHOT
   Date        2024-10-17T21:50:32Z

Parameters:
   Targets                      []
   Severity Level               WARNING
   Recurse Directories          true
   File Filters Used            [*.xml, *.XML]
   Data Content Validation      on
   Product Level Validation     on
   Max Errors                   100000
   Registered Contexts File     /Users/jpadams/proj/pds/pdsen/workspace/validate/validate-3.6.0-SNAPSHOT/resources/registered_context_products.json



Product Level Validation Results

  PASS: /Users/jpadams/proj/pds/pdsen/workspace/validate/validate-3.6.0-SNAPSHOT/resources/registered_context_products.json
    Begin Fragment: file:/Users/jpadams/proj/pds/pdsen/workspace/validate/validate-3.6.0-SNAPSHOT/resources/registered_context_products.json
      INFO  [info.validation.general]   Successfully updated registered context products config file from PDS Search API.
      INFO  [info.validation.general]   4590 registered context products found.
    End Fragment: file:/Users/jpadams/proj/pds/pdsen/workspace/validate/validate-3.6.0-SNAPSHOT/resources/registered_context_products.json
        1 product validation(s) completed

Summary:

  1 product(s)
  0 error(s)
  0 warning(s)

  Product Validation Summary:
    1          product(s) passed
    0          product(s) failed
    0          product(s) skipped
    1          product(s) total

  Referential Integrity Check Summary:
    0          check(s) passed
    0          check(s) failed
    0          check(s) skipped
    0          check(s) total


End of Report
Completed execution in 38713 ms

# testing running validate against a product
PDS Validate Tool Report

Configuration:
   Version     3.6.0-SNAPSHOT
   Date        2024-10-17T21:46:01Z

Parameters:
   Targets                      [file:/Users/jpadams/proj/pds/pdsen/workspace/validate/src/test/resources/github597/document/spiceds_v001.xml]
   Severity Level               WARNING
   Recurse Directories          true
   File Filters Used            [*.xml, *.XML]
   Data Content Validation      on
   Product Level Validation     on
   Max Errors                   100000
   Registered Contexts File     /Users/jpadams/proj/pds/pdsen/workspace/validate/validate-3.6.0-SNAPSHOT/resources/registered_context_products.json



Product Level Validation Results

  PASS: file:/Users/jpadams/proj/pds/pdsen/workspace/validate/src/test/resources/github597/document/spiceds_v001.xml
      WARNING  [warning.label.context_ref_mismatch]   line 37: Context reference name mismatch. Value: 'Mars 2020 Perseverance Rover Mission' Expected one of: '[Mars 2020]'
      WARNING  [warning.label.context_ref_mismatch]   line 46: Context reference name mismatch. Value: 'Mars 2020 Perseverance Rover' Expected one of: '[Perseverance]'
        1 product validation(s) completed

Summary:

  1 product(s)
  0 error(s)
  2 warning(s)

  Product Validation Summary:
    1          product(s) passed
    0          product(s) failed
    0          product(s) skipped
    1          product(s) total

  Referential Integrity Check Summary:
    0          check(s) passed
    0          check(s) failed
    0          check(s) skipped
    0          check(s) total

  Message Types:
    2            warning.label.context_ref_mismatch

End of Report
Completed execution in 4617 ms

♻️ Related Issues

Resolves #675
Resolves #1034

Al Niessner and others added 5 commits August 30, 2023 13:48
Change the validate.properties to use the registry-api. Adjusted code to get all data then walk through the results as lists and maps of the JSON response object.
* updated to use `search-after`
* added `useDelimiter` to Scanner in order to keep all whitespace
* updated search query to ignore extraneous context products

resolves #675
@jordanpadams jordanpadams requested a review from a team as a code owner October 17, 2024 21:54
Copy link
Member

@nutjob4life nutjob4life left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reproduced just fine:

$ git checkout issue_675
branch 'issue_675' set up to track 'origin/issue_675'.
Switched to a new branch 'issue_675'
$ mvn -Dmaven.test.skip=true clean package
[INFO] Scanning for projects...
[INFO] Inspecting build with total of 1 modules...

[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time:  33.590 s
[INFO] Finished at: 2024-10-17T16:56:51-05:00
[INFO] ------------------------------------------------------------------------
66.594u 8.499s 0:36.63 204.9%   0+0k 0+0io 3201pf+0w
$ tar xzf target/validate-3.6.0-SNAPSHOT-bin.tar.gz
$ validate-3.6.0-SNAPSHOT/bin/validate -u

PDS Validate Tool Report

Configuration:
   Version     3.6.0-SNAPSHOT
   Date        2024-10-17T21:57:23Z

Parameters:
   Targets                      []
   Severity Level               WARNING
   Recurse Directories          true
   File Filters Used            [*.xml, *.XML]
   Data Content Validation      on
   Product Level Validation     on
   Max Errors                   100000
   Registered Contexts File     /Users/kelly/Documents/Clients/JPL/PDS/Development/nasa-pds/validate/validate-3.6.0-SNAPSHOT/resources/registered_context_products.json

Summary:

  1 product(s)
  0 error(s)
  0 warning(s)

  Product Validation Summary:
    1          product(s) passed
    0          product(s) failed
    0          product(s) skipped
    1          product(s) total

  Referential Integrity Check Summary:
    0          check(s) passed
    0          check(s) failed
    0          check(s) skipped
    0          check(s) total


End of Report
Completed execution in 55810 ms

Copy link
Contributor

@al-niessner al-niessner left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If this removes all solr (it should) then remove solr from pom please.

ditch solr
@al-niessner al-niessner dismissed their stale review October 17, 2024 23:17

I removed solr from pom

@jordanpadams jordanpadams merged commit a44c221 into main Oct 18, 2024
3 checks passed
@jordanpadams jordanpadams deleted the issue_675 branch October 18, 2024 19:36
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
3 participants