Fails with internal error Error: Could not load HED schema version "Latest" from remote repository

[Shotgunosine]

I'm trying to run the BIDS validator from a singularity container on a HPC system. I built a container from the latest version on DockerHub (1.6.20) with the following command:

 $PWD/bids_validator.simg docker://bids/validator

I've tried running it on a compute node on the HPC and I get the following error:

(/vf/users/ABCD_MBDU/dcan_abcd_dcm_bids_env) [nielsond@cn2878 notebooks]$ singularity run --cleanenv --no-home -B /data/AB
CD_MBDU/abcd_bids/test_download:/data /data/ABCD_MBDU/singularity_images/bids_validator.simg /data                        
bids-validator@1.6.2                                                                                                      
                                                                                                                          
        1: [ERR] Internal error. SOME VALIDATION STEPS MAY NOT HAVE OCCURRED (code: 0 - INTERNAL ERROR)                   
                .undefined                                                                                                
                        Evidence: Error: Could not load HED schema version "Latest" from remote repository - "Error: getad
drinfo EAI_AGAIN raw.githubusercontent.com".                                                                              
    at /src/node_modules/hed-validator/utils/schema.js:40:13                                                              
    at tryCallOne (/src/node_modules/promise/lib/core.js:37:12)                                                           
    at /src/node_modules/promise/lib/core.js:123:15                                                                       
    at flush (/src/node_modules/asap/raw.js:50:29)                                                                        
    at processTicksAndRejections (internal/process/task_queues.js:76:11)                                                  
                                                                                                                          
        Please visit https://neurostars.org/search?q=INTERNAL ERROR for existing conversations about this issue.          
                                                                                                                          
                                                                                                                          
        Summary:                 Available Tasks:                 Available Modalities:                                   
        13 Files, 13.49MB        Stop-Signal Task                 T1w                                                     
        1 - Subject              Monetary Incentive Delay         epi                                                     
        1 - Session              n-back                                                                                   
                                 Resting State (eyes open)                                                                
                                                                                                                          
                                                                                                                          
        If you have any questions, please post on https://neurostars.org/tags/bids.                                       
                                                                                              

Working backwards, I found that if I build an image from version 1.5.7 it runs successfully without this error, but fails with version 1.5.8 with the same error. My guess is that the validator is trying to request some resource from raw.githubusercontent.com and getting stymied by the proxy rules the HPC has in place on the compute nodes. Can anyone tell me what the request it's trying to make is? Does anyone have any advice on the needed proxy settings?

[sappelhoff]

Hey @Shotgunosine thanks for the report and sorry for taking so long to get back to you.

This seems due to the hed-validator library that we use to validate HED tags within BIDS. Specifically these lines seem to trigger the error:

https://github.com/hed-standard/hed-javascript/blob/e01dfebc436f5a0e902a7960a77a38b1423dc788/utils/schema.js#L31-L33

@happy5214 do you have any advise on this?

[happy5214]

https://raw.githubusercontent.com/hed-standard/hed-specification/master/hedxml/HEDLatest.xml is the file it's trying to load. I don't have much experience with proxy servers, but Stack Overflow says getaddrinfo EAI_AGAIN is a DNS timeout error, so it's unlikely to be related to the HED validator code itself, but rather a proxy issue.

[VisLab]

Dylan, To avoid the problem in your particular case, you can download the version of the HED schema you are using, say https://github.com/hed-standard/hed-specification/tree/master/hedxml/HED7.2.0.xml into your BIDS dataset source directory. In the dataset_description.json file put: "HEDVERSION": "HED7.2.0.xml"

…

On Tue, Mar 30, 2021 at 10:41 AM Alexander Jones ***@***.***> wrote: https://raw.githubusercontent.com/hed-standard/hed-specification/master/hedxml/HEDLatest.xml is the file it's trying to load. I don't have much experience with proxy servers, but Stack Overflow says getaddrinfo EAI_AGAIN is a DNS timeout error, so it's unlikely to be related to the HED validator code itself, but rather a proxy issue. — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub <https://github.com/bids-standard/bids-validator/issues/1205#issuecomment-810368920>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAJCJOTBQ7R3XT7ZL4AI6KDTGHWLDANCNFSM4YPKYS7A> .

[happy5214]

"HEDVERSION": "HED7.2.0.xml"

Not all-caps. Use this casing:

"HEDVersion": "HED7.2.0.xml"

[sappelhoff]

I am closing this, assuming that it truly is a proxy issue (and not a bids-validator issue), and assuming that the solution to download the HED schema directly worked.

Please feel free to re-open if this is a sticking issue that we can/should do something about from the bids-validator side.

[mgxd]

I think this has been a thorn in our side while implementing nipreps/fmriprep#2409 - I get the same internal error as the OP when running the bids-validator through the command line inside a docker container with --network=none, but then get

Unhandled rejection (
  reason: TypeError: Cannot read property 'split' of undefined
    at Minimatch.match (/usr/lib/node_modules/bids-validator/node_modules/minimatch/minimatch.js:717:9)
    at minimatch (/usr/lib/node_modules/bids-validator/node_modules/minimatch/minimatch.js:107:42)
    at Object.ignoredFile (/usr/lib/node_modules/bids-validator/utils/config.js:11:13)
    at Function.format (/usr/lib/node_modules/bids-validator/utils/issues/index.js:65:16)
    at Function.exceptionHandler (/usr/lib/node_modules/bids-validator/utils/issues/index.js:191:25)
    at /usr/lib/node_modules/bids-validator/validators/bids/fullTest.js:217:35
    at processTicksAndRejections (internal/process/task_queues.js:95:5)
).

when running the same command through Python's subprocess. FWIW, the bids-validator command I'm running can be found here https://github.com/bids-standard/bids-validator/issues/1233#issuecomment-848120435

Would it be reasonable to include or ask hed-validator to include the latest HED tags within the packaging to allow network-less validation?

[happy5214]

The only HED issue here is that it doesn't catch its own exception (which is probably poor code practice). My guess is that the issue is stemming from some issue objects' file fields not including relative paths (i.e. they're undefined), which is the source of the type error. It's only popping up here because this code path (an uncaught internal exception caught by the handler at https://github.com/bids-standard/bids-validator/blob/c60daacf3dbb4cea5a07d5274947c366cf4c0e3a/bids-validator/validators/bids/fullTest.js#L214) is almost never encountered.

As for the other request, the package already contains several versions of the schema for testing purposes, but you can't use them from that location. It may or may not be reasonable to post a new release for every schema release, depending on the future volume of schema releases. I'll discuss with @VisLab.

[VisLab]

Including HED latest seems reasonable. We will investigate and try to address the issue.

…

On Wed, Jun 9, 2021 at 7:50 PM Mathias Goncalves ***@***.***> wrote: I think this has been a thorn in our side while implementing nipreps/fmriprep#2409 <nipreps/fmriprep#2409> - I get the same internal error as the OP when running the bids-validator through the command line inside a docker container with --network=none, but then get Unhandled rejection ( reason: TypeError: Cannot read property 'split' of undefined at Minimatch.match (/usr/lib/node_modules/bids-validator/node_modules/minimatch/minimatch.js:717:9) at minimatch (/usr/lib/node_modules/bids-validator/node_modules/minimatch/minimatch.js:107:42) at Object.ignoredFile (/usr/lib/node_modules/bids-validator/utils/config.js:11:13) at Function.format (/usr/lib/node_modules/bids-validator/utils/issues/index.js:65:16) at Function.exceptionHandler (/usr/lib/node_modules/bids-validator/utils/issues/index.js:191:25) at /usr/lib/node_modules/bids-validator/validators/bids/fullTest.js:217:35 at processTicksAndRejections (internal/process/task_queues.js:95:5) ). when running the same command through Python's subprocess. FWIW, the bids-validator command I'm running can be found here #1233 (comment) <https://github.com/bids-standard/bids-validator/issues/1233#issuecomment-848120435> Would it be reasonable to include or ask hed-validator to include the latest HED tags <https://raw.githubusercontent.com/hed-standard/hed-specification/master/hedxml/HEDLatest.xml> within the packaging to allow network-less validation? — You are receiving this because you commented. Reply to this email directly, view it on GitHub <https://github.com/bids-standard/bids-validator/issues/1205#issuecomment-858197409>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAJCJOULY55J2M25LH7OLCDTSAD7FANCNFSM4YPKYS7A> .

[effigies]

I asked @rwblair, but it might be best to bring it up here: Would it be possible to delay the import of the HED validator until we know there are HED tags to validate? Or can we only tell if there are HED tags through calls to the HED validator?

And if the answer is that we need HED to detect HED tags, but we could delay the import, would it be possible to write a stub HED detector that we could use, so that we can effectively isolate HED?

[happy5214]

Strictly speaking, the code that detects whether any HED tags exist in the TSVs or sidecars doesn't require a schema to be loaded, since it's all on the BIDS side, but refactoring the code to move the schema import after the existence check would require massive changes to the module structure.

As for bundling the latest schema, that will likely start with the stable 8.0.0 schema release, which should be in a couple of weeks. Datasets really should be using fixed versions anyway rather than the default "Latest" to avoid unexpected future failures, and this change wouldn't solve that problem.

[happy5214]

I'm planning on tackling the schema import issue in the next few days. I'll also take the opportunity to split the rather monolithic main function into smaller blocks, which should please Code Climate.