diff --git a/CHANGELOG.md b/CHANGELOG.md index a8215100..790718bd 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -1,5 +1,11 @@ # nextflow-io/nf-schema: Changelog +# Version 2.1.0 + +## JSON schema fixes + +1. The `defs` keyword is now deprecated in favor of the `$defs` keyword. This to follow the JSON schema guidelines. We will continue supporting `defs` for backwards compatibility. + # Version 2.0.1 - Tsukemen ## Vulnerability fix diff --git a/docs/migration_guide.md b/docs/migration_guide.md index 836865fd..31161ae4 100644 --- a/docs/migration_guide.md +++ b/docs/migration_guide.md @@ -65,15 +65,15 @@ e 's/from "plugin\/nf-validation"/from "plugin\/nf-schema"/g' {} + If you aren't using any special features in your schemas, you can simply update your `nextflow_schema.json` file using the following command: ```bash -sed -i -e 's/http:\/\/json-schema.org\/draft-07\/schema/https:\/\/json-schema.org\/draft\/2020-12\/schema/g' -e 's/definitions/defs/g' nextflow_schema.json +sed -i -e 's/http:\/\/json-schema.org\/draft-07\/schema/https:\/\/json-schema.org\/draft\/2020-12\/schema/g' -e 's/definitions/$defs/g' nextflow_schema.json ``` -This will replace the old schema draft specification (`draft-07`) by the new one (`2020-12`), and the old keyword `definitions` by the new notation `defs`. +This will replace the old schema draft specification (`draft-07`) by the new one (`2020-12`), and the old keyword `definitions` by the new notation `$defs`. !!! note Repeat this command for every JSON schema used in your pipeline. e.g. for the default samplesheet schema in nf-core pipelines: - `bash sed -i -e 's/http:\/\/json-schema.org\/draft-07\/schema/https:\/\/json-schema.org\/draft\/2020-12\/schema/g' -e 's/definitions/defs/g' assets/schema_input.json` + `bash sed -i -e 's/http:\/\/json-schema.org\/draft-07\/schema/https:\/\/json-schema.org\/draft\/2020-12\/schema/g' -e 's/definitions/$defs/g' assets/schema_input.json` !!! warning diff --git a/docs/nextflow_schema/create_schema.md b/docs/nextflow_schema/create_schema.md index 5fd9ea57..d0acd517 100644 --- a/docs/nextflow_schema/create_schema.md +++ b/docs/nextflow_schema/create_schema.md @@ -51,7 +51,7 @@ nf-core schema build The current version of `nf-core` tools (v2.13.1) does not support the new schema draft used in `nf-schema`. Running this command after building the schema will convert the schema to the right draft: ```bash - sed -i -e 's/http:\/\/json-schema.org\/draft-07\/schema/https:\/\/json-schema.org\/draft\/2020-12\/schema/g' -e 's/definitions/defs/g' nextflow_schema.json + sed -i -e 's/http:\/\/json-schema.org\/draft-07\/schema/https:\/\/json-schema.org\/draft\/2020-12\/schema/g' -e 's/definitions/$defs/g' nextflow_schema.json ``` A new version of the nf-core schema builder will be available soon. Keep an eye out! diff --git a/docs/nextflow_schema/nextflow_schema_specification.md b/docs/nextflow_schema/nextflow_schema_specification.md index dd76685a..11a9486f 100644 --- a/docs/nextflow_schema/nextflow_schema_specification.md +++ b/docs/nextflow_schema/nextflow_schema_specification.md @@ -30,14 +30,14 @@ You can find more information about JSON Schema here: ## Definitions -A slightly strange use of a JSON schema standard that we use for Nextflow schema is `defs`. +A slightly strange use of a JSON schema standard that we use for Nextflow schema is `$defs`. JSON schema can group variables together in an `object`, but then the validation expects this structure to exist in the data that it is validating. In reality, we have a very long "flat" list of parameters, all at the top level of `params.foo`. -In order to give some structure to log outputs, documentation and so on, we group parameters into `defs`. +In order to give some structure to log outputs, documentation and so on, we group parameters into `$defs`. Each `def` is an object with a title, description and so on. -However, as they are under `defs` scope they are effectively ignored by the validation and so their nested nature is not a problem. +However, as they are under `$defs` scope they are effectively ignored by the validation and so their nested nature is not a problem. We then bring the contents of each definition object back to the "flat" top level for validation using a series of `allOf` statements at the end of the schema, which reference the specific definition keys. @@ -47,7 +47,7 @@ which reference the specific definition keys. "$schema": "https://json-schema.org/draft/2020-12/schema", "type": "object", // Definition groups - "defs": { // (1)! + "$defs": { // (1)! "my_group_of_params": { // (2)! "title": "A virtual grouping used for docs and pretty-printing", "type": "object", @@ -64,7 +64,7 @@ which reference the specific definition keys. }, // Contents of each definition group brought into main schema for validation "allOf": [ - { "$ref": "#/defs/my_group_of_params" } // (6)! + { "$ref": "#/$defs/my_group_of_params" } // (6)! ] } ``` @@ -77,7 +77,7 @@ which reference the specific definition keys. 5. Shortened here for the example, see below for full parameter specification. 6. A `$ref` line like this needs to be added for every definition group -Parameters can be described outside of the `defs` scope, in the regular JSON Schema top-level `properties` scope. +Parameters can be described outside of the `$defs` scope, in the regular JSON Schema top-level `properties` scope. However, they will be displayed as ungrouped in tools working off the schema. ## Nested parameters diff --git a/examples/paramsHelp/pipeline/nextflow_schema.json b/examples/paramsHelp/pipeline/nextflow_schema.json index 6096ceb9..62b5c396 100644 --- a/examples/paramsHelp/pipeline/nextflow_schema.json +++ b/examples/paramsHelp/pipeline/nextflow_schema.json @@ -4,7 +4,7 @@ "title": "nf-core/testpipeline pipeline parameters", "description": "this is a test", "type": "object", - "defs": { + "$defs": { "input_output_options": { "title": "Input/output options", "type": "object", @@ -33,7 +33,7 @@ }, "allOf": [ { - "$ref": "#/defs/input_output_options" + "$ref": "#/$defs/input_output_options" } ] } diff --git a/examples/paramsSummaryLog/pipeline/nextflow_schema.json b/examples/paramsSummaryLog/pipeline/nextflow_schema.json index 6096ceb9..62b5c396 100644 --- a/examples/paramsSummaryLog/pipeline/nextflow_schema.json +++ b/examples/paramsSummaryLog/pipeline/nextflow_schema.json @@ -4,7 +4,7 @@ "title": "nf-core/testpipeline pipeline parameters", "description": "this is a test", "type": "object", - "defs": { + "$defs": { "input_output_options": { "title": "Input/output options", "type": "object", @@ -33,7 +33,7 @@ }, "allOf": [ { - "$ref": "#/defs/input_output_options" + "$ref": "#/$defs/input_output_options" } ] } diff --git a/examples/paramsSummaryMap/pipeline/nextflow_schema.json b/examples/paramsSummaryMap/pipeline/nextflow_schema.json index 6096ceb9..62b5c396 100644 --- a/examples/paramsSummaryMap/pipeline/nextflow_schema.json +++ b/examples/paramsSummaryMap/pipeline/nextflow_schema.json @@ -4,7 +4,7 @@ "title": "nf-core/testpipeline pipeline parameters", "description": "this is a test", "type": "object", - "defs": { + "$defs": { "input_output_options": { "title": "Input/output options", "type": "object", @@ -33,7 +33,7 @@ }, "allOf": [ { - "$ref": "#/defs/input_output_options" + "$ref": "#/$defs/input_output_options" } ] } diff --git a/examples/validateParameters/pipeline/nextflow_schema.json b/examples/validateParameters/pipeline/nextflow_schema.json index c0df5201..623613ba 100644 --- a/examples/validateParameters/pipeline/nextflow_schema.json +++ b/examples/validateParameters/pipeline/nextflow_schema.json @@ -4,7 +4,7 @@ "title": "nf-core/testpipeline pipeline parameters", "description": "this is a test", "type": "object", - "defs": { + "$defs": { "input_output_options": { "title": "Input/output options", "type": "object", @@ -34,7 +34,7 @@ }, "allOf": [ { - "$ref": "#/defs/input_output_options" + "$ref": "#/$defs/input_output_options" } ] } diff --git a/examples/validationFailUnrecognisedParams/pipeline/nextflow_schema.json b/examples/validationFailUnrecognisedParams/pipeline/nextflow_schema.json index 6096ceb9..62b5c396 100644 --- a/examples/validationFailUnrecognisedParams/pipeline/nextflow_schema.json +++ b/examples/validationFailUnrecognisedParams/pipeline/nextflow_schema.json @@ -4,7 +4,7 @@ "title": "nf-core/testpipeline pipeline parameters", "description": "this is a test", "type": "object", - "defs": { + "$defs": { "input_output_options": { "title": "Input/output options", "type": "object", @@ -33,7 +33,7 @@ }, "allOf": [ { - "$ref": "#/defs/input_output_options" + "$ref": "#/$defs/input_output_options" } ] } diff --git a/examples/validationWarnUnrecognisedParams/pipeline/nextflow_schema.json b/examples/validationWarnUnrecognisedParams/pipeline/nextflow_schema.json index 6096ceb9..62b5c396 100644 --- a/examples/validationWarnUnrecognisedParams/pipeline/nextflow_schema.json +++ b/examples/validationWarnUnrecognisedParams/pipeline/nextflow_schema.json @@ -4,7 +4,7 @@ "title": "nf-core/testpipeline pipeline parameters", "description": "this is a test", "type": "object", - "defs": { + "$defs": { "input_output_options": { "title": "Input/output options", "type": "object", @@ -33,7 +33,7 @@ }, "allOf": [ { - "$ref": "#/defs/input_output_options" + "$ref": "#/$defs/input_output_options" } ] } diff --git a/parameters_meta_schema.json b/parameters_meta_schema.json index cf26192c..5662fd43 100644 --- a/parameters_meta_schema.json +++ b/parameters_meta_schema.json @@ -30,7 +30,7 @@ "type": "string", "const": "object" }, - "defs": { + "$defs": { "title": "Parameter groups", "type": "object", "patternProperties": { @@ -139,7 +139,7 @@ "properties": { "$ref": { "type": "string", - "pattern": "^#/defs/" + "pattern": "^#/$defs/" } } } diff --git a/plugins/nf-schema/src/main/nextflow/validation/SchemaValidator.groovy b/plugins/nf-schema/src/main/nextflow/validation/SchemaValidator.groovy index b7dd4b06..0351e32a 100644 --- a/plugins/nf-schema/src/main/nextflow/validation/SchemaValidator.groovy +++ b/plugins/nf-schema/src/main/nextflow/validation/SchemaValidator.groovy @@ -243,14 +243,15 @@ class SchemaValidator extends PluginExtensionPoint { // Check for nextflow core params and unexpected params def slurper = new JsonSlurper() def Map parsed = (Map) slurper.parse( Path.of(Utils.getSchemaPath(baseDir, schemaFilename)) ) - def Map schemaParams = (Map) parsed.get('defs') + // $defs is the adviced keyword for definitions. Keeping defs in for backwards compatibility + def Map schemaParams = (Map) (parsed.get('$defs') ?: parsed.get("defs")) def specifiedParamKeys = params.keySet() // Collect expected parameters from the schema def enumsTuple = collectEnums(schemaParams) def List expectedParams = (List) enumsTuple[0] + addExpectedParams() def Map enums = (Map) enumsTuple[1] - // Collect expected parameters from the schema when parameters are specified outside of "defs" + // Collect expected parameters from the schema when parameters are specified outside of "$defs" if (parsed.containsKey('properties')) { def enumsTupleTopLevel = collectEnums(['top_level': ['properties': parsed.get('properties')]]) expectedParams += (List) enumsTupleTopLevel[0] @@ -602,10 +603,11 @@ class SchemaValidator extends PluginExtensionPoint { private static LinkedHashMap paramsRead(Path json_schema) throws Exception { def slurper = new JsonSlurper() def Map schema = (Map) slurper.parse( json_schema ) - def Map schema_defs = (Map) schema.get('defs') + // $defs is the adviced keyword for definitions. Keeping defs in for backwards compatibility + def Map schema_defs = (Map) (schema.get('$defs') ?: schema.get("defs")) def Map schema_properties = (Map) schema.get('properties') /* Tree looks like this in nf-core schema - * defs <- this is what the first get('defs') gets us + * $defs <- this is what the first get('$defs') gets us group 1 title description @@ -623,7 +625,7 @@ class SchemaValidator extends PluginExtensionPoint { parameter 1 type description - * properties <- parameters can also be ungrouped, outside of defs + * properties <- parameters can also be ungrouped, outside of $defs parameter 1 type description diff --git a/plugins/nf-schema/src/resources/META-INF/MANIFEST.MF b/plugins/nf-schema/src/resources/META-INF/MANIFEST.MF index d2fcf8c7..8bf31df6 100644 --- a/plugins/nf-schema/src/resources/META-INF/MANIFEST.MF +++ b/plugins/nf-schema/src/resources/META-INF/MANIFEST.MF @@ -1,6 +1,6 @@ Manifest-Version: 1.0 Plugin-Id: nf-schema -Plugin-Version: 2.0.1 +Plugin-Version: 2.1.0 Plugin-Class: nextflow.validation.ValidationPlugin Plugin-Provider: nextflow Plugin-Requires: >=22.10.0 diff --git a/plugins/nf-schema/src/testResources/nextflow_schema.json b/plugins/nf-schema/src/testResources/nextflow_schema.json index 5bb7c6e9..7dcf103b 100644 --- a/plugins/nf-schema/src/testResources/nextflow_schema.json +++ b/plugins/nf-schema/src/testResources/nextflow_schema.json @@ -4,7 +4,7 @@ "title": "nf-core/testpipeline pipeline parameters", "description": "this is a test", "type": "object", - "defs": { + "$defs": { "input_output_options": { "title": "Input/output options", "type": "object", @@ -250,19 +250,19 @@ }, "allOf": [ { - "$ref": "#/defs/input_output_options" + "$ref": "#/$defs/input_output_options" }, { - "$ref": "#/defs/reference_genome_options" + "$ref": "#/$defs/reference_genome_options" }, { - "$ref": "#/defs/institutional_config_options" + "$ref": "#/$defs/institutional_config_options" }, { - "$ref": "#/defs/max_job_request_options" + "$ref": "#/$defs/max_job_request_options" }, { - "$ref": "#/defs/generic_options" + "$ref": "#/$defs/generic_options" } ] } diff --git a/plugins/nf-schema/src/testResources/nextflow_schema_draft7.json b/plugins/nf-schema/src/testResources/nextflow_schema_draft7.json index 006e9459..e4822a21 100644 --- a/plugins/nf-schema/src/testResources/nextflow_schema_draft7.json +++ b/plugins/nf-schema/src/testResources/nextflow_schema_draft7.json @@ -4,7 +4,7 @@ "title": "nf-core/testpipeline pipeline parameters", "description": "this is a test", "type": "object", - "defs": { + "$defs": { "file_patterns": { "title": "Input/output options", "type": "object", @@ -19,7 +19,7 @@ }, "allOf": [ { - "$ref": "#/defs/file_patterns" + "$ref": "#/$defs/file_patterns" } ] } diff --git a/plugins/nf-schema/src/testResources/nextflow_schema_file_path_pattern.json b/plugins/nf-schema/src/testResources/nextflow_schema_file_path_pattern.json index e0fc8a41..98cf94cf 100644 --- a/plugins/nf-schema/src/testResources/nextflow_schema_file_path_pattern.json +++ b/plugins/nf-schema/src/testResources/nextflow_schema_file_path_pattern.json @@ -4,7 +4,7 @@ "title": "nf-core/testpipeline pipeline parameters", "description": "this is a test", "type": "object", - "defs": { + "$defs": { "file_patterns": { "title": "Input/output options", "type": "object", @@ -19,7 +19,7 @@ }, "allOf": [ { - "$ref": "#/defs/file_patterns" + "$ref": "#/$defs/file_patterns" } ] } diff --git a/plugins/nf-schema/src/testResources/nextflow_schema_required_numerics.json b/plugins/nf-schema/src/testResources/nextflow_schema_required_numerics.json index cc272782..3fb2bb5e 100644 --- a/plugins/nf-schema/src/testResources/nextflow_schema_required_numerics.json +++ b/plugins/nf-schema/src/testResources/nextflow_schema_required_numerics.json @@ -4,7 +4,7 @@ "title": "nf-core/testpipeline pipeline parameters", "description": "this is a test", "type": "object", - "defs": { + "$defs": { "input_output_options": { "title": "Input/output options", "type": "object", @@ -53,10 +53,10 @@ }, "allOf": [ { - "$ref": "#/defs/input_output_options" + "$ref": "#/$defs/input_output_options" }, { - "$ref": "#/defs/numeric_options" + "$ref": "#/$defs/numeric_options" } ] } diff --git a/plugins/nf-schema/src/testResources/nextflow_schema_with_deeply_nested_samplesheet.json b/plugins/nf-schema/src/testResources/nextflow_schema_with_deeply_nested_samplesheet.json index 8377896e..fb29b17f 100644 --- a/plugins/nf-schema/src/testResources/nextflow_schema_with_deeply_nested_samplesheet.json +++ b/plugins/nf-schema/src/testResources/nextflow_schema_with_deeply_nested_samplesheet.json @@ -4,7 +4,7 @@ "title": "nf-core/testpipeline pipeline parameters", "description": "this is a test", "type": "object", - "defs": { + "$defs": { "input_output_options": { "title": "Input/output options", "type": "object", diff --git a/plugins/nf-schema/src/testResources/nextflow_schema_with_exists_false.json b/plugins/nf-schema/src/testResources/nextflow_schema_with_exists_false.json index 4c30cf97..c8f5cf09 100644 --- a/plugins/nf-schema/src/testResources/nextflow_schema_with_exists_false.json +++ b/plugins/nf-schema/src/testResources/nextflow_schema_with_exists_false.json @@ -4,7 +4,7 @@ "title": "nf-core/testpipeline pipeline parameters", "description": "this is a test", "type": "object", - "defs": { + "$defs": { "input_output_options": { "title": "Input/output options", "type": "object", @@ -21,7 +21,7 @@ }, "allOf": [ { - "$ref": "#/defs/input_output_options" + "$ref": "#/$defs/input_output_options" } ] } diff --git a/plugins/nf-schema/src/testResources/nextflow_schema_with_samplesheet.json b/plugins/nf-schema/src/testResources/nextflow_schema_with_samplesheet.json index f88e51ad..ea4b43cd 100644 --- a/plugins/nf-schema/src/testResources/nextflow_schema_with_samplesheet.json +++ b/plugins/nf-schema/src/testResources/nextflow_schema_with_samplesheet.json @@ -4,7 +4,7 @@ "title": "nf-core/testpipeline pipeline parameters", "description": "this is a test", "type": "object", - "defs": { + "$defs": { "input_output_options": { "title": "Input/output options", "type": "object", @@ -28,7 +28,7 @@ }, "allOf": [ { - "$ref": "#/defs/input_output_options" + "$ref": "#/$defs/input_output_options" } ] } diff --git a/plugins/nf-schema/src/testResources/nextflow_schema_with_samplesheet_converter.json b/plugins/nf-schema/src/testResources/nextflow_schema_with_samplesheet_converter.json index b411b160..0c788763 100644 --- a/plugins/nf-schema/src/testResources/nextflow_schema_with_samplesheet_converter.json +++ b/plugins/nf-schema/src/testResources/nextflow_schema_with_samplesheet_converter.json @@ -4,7 +4,7 @@ "title": "nf-core/testpipeline pipeline parameters", "description": "this is a test", "type": "object", - "defs": { + "$defs": { "input_output_options": { "title": "Input/output options", "type": "object", diff --git a/plugins/nf-schema/src/testResources/nextflow_schema_with_samplesheet_no_header.json b/plugins/nf-schema/src/testResources/nextflow_schema_with_samplesheet_no_header.json index c37c0b4a..11baca01 100644 --- a/plugins/nf-schema/src/testResources/nextflow_schema_with_samplesheet_no_header.json +++ b/plugins/nf-schema/src/testResources/nextflow_schema_with_samplesheet_no_header.json @@ -4,7 +4,7 @@ "title": "nf-core/testpipeline pipeline parameters", "description": "this is a test", "type": "object", - "defs": { + "$defs": { "input_output_options": { "title": "Input/output options", "type": "object", diff --git a/plugins/nf-schema/src/testResources/nextflow_schema_with_samplesheet_no_meta.json b/plugins/nf-schema/src/testResources/nextflow_schema_with_samplesheet_no_meta.json index 6c7a0220..2f12c160 100644 --- a/plugins/nf-schema/src/testResources/nextflow_schema_with_samplesheet_no_meta.json +++ b/plugins/nf-schema/src/testResources/nextflow_schema_with_samplesheet_no_meta.json @@ -4,7 +4,7 @@ "title": "nf-core/testpipeline pipeline parameters", "description": "this is a test", "type": "object", - "defs": { + "$defs": { "input_output_options": { "title": "Input/output options", "type": "object", diff --git a/plugins/nf-schema/src/testResources/nextflow_schema_with_samplesheet_uniqueEntries.json b/plugins/nf-schema/src/testResources/nextflow_schema_with_samplesheet_uniqueEntries.json index 1c113b8f..a47eb22b 100644 --- a/plugins/nf-schema/src/testResources/nextflow_schema_with_samplesheet_uniqueEntries.json +++ b/plugins/nf-schema/src/testResources/nextflow_schema_with_samplesheet_uniqueEntries.json @@ -4,7 +4,7 @@ "title": "nf-core/testpipeline pipeline parameters", "description": "this is a test", "type": "object", - "defs": { + "$defs": { "input_output_options": { "title": "Input/output options", "type": "object", @@ -28,7 +28,7 @@ }, "allOf": [ { - "$ref": "#/defs/input_output_options" + "$ref": "#/$defs/input_output_options" } ] }