Please submit pointers to your validator output formats

@effigies

Please checkout README in https://github.com/con/validation for the motivations etc.

BIDS (attn @effigies)
- Interaction of BIDS and HED for errors bids-standard/bids-validator#157 (comment) for overall schema/description
- validator version: deno: Report/record bids validator + hed validator in the bids-validator output bids-standard/bids-validator#10
- standard versions: How to get schema version? bids-standard/bids-validator#118
NWB (attn @rly; @bendichter for nwb-inspector)
- HDMF Error objects described at https://github.com/hdmf-dev/hdmf/blob/dev/src/hdmf/validate/errors.py#L19
- standard versions: [Feature]: Include version of NWB standard used in validation errors NeurodataWithoutBorders/pynwb#2062
HED (attn @VisLab)
- described below
- BIDS/HED validation records discussion: Interaction of BIDS and HED for errors bids-standard/bids-validator#157 (comment)
Zarr -- TODO
OME-Zarr -- TODO
SPARC's SDS (attn @tgbugs)
- below
- https://github.com/SciCrunch/sparc-curation/blob/master/sparcur/schemas.py
PsychDS (attn @bleonar5)
- originally inspired by BIDS validator but likely is not a fork of it: https://github.com/psych-ds/psychds-validator
- ...
...

Generic higher level (domain agnostic?) specifications

SARIF (brought up by @effigies): Static Analysis Results Interchange Format (SARIF)

The NWB validator can be called from the command line and as a Python function.

The CLI is described here. When validation passes, the output looks like:

Validating /Users/rly/Documents/NWB/scratch/test.nwb against cached namespace information using namespace 'core'.
 - no errors found.

with exit code 0.

When it fails, it looks like:

Validating /Users/rly/Documents/NWB/scratch/test.nwb against cached namespace information using namespace 'core'.
 - found the following errors:
SpatialSeries/data (processing/behavior/position/series_0/data): incorrect shape - expected '[[None], [None, 1], [None, 2], [None, 3]]', got '(27206, 6)'

with exit code 1.

The Python function is described here. Using the paths keyword argument returns a tuple (list of errors, exit code). Using the io keyword argument returns a list of errors.

FWIW, here is where we are mapping pynwb.validate output into our ValidationResult: https://github.com/dandi/dandi-cli/blob/HEAD/dandi/pynwb_utils.py#L359 so it looks like

                ValidationResult(
                    origin=ValidationOrigin(
                        name="pynwb",
                        version=pynwb.__version__,
                    ),
                    severity=Severity.WARNING,
                    id=f"pywnb.{error_output}",
                    scope=Scope.FILE,
                    path=Path(path),
                    message="Failed to validate.",
                )

but that is quite suboptimal since there is only error_output string without better decomposition into the "path" within the file etc. For nwb inspector outputs we seems to get some path within asset (file) and map it as well: https://github.com/dandi/dandi-cli/blob/HEAD/dandi/files/bases.py#L546

@yarikoptic

Thanks @yarikoptic . The pynwb validator errors are parseable. I tried to address this in dandi/dandi-cli#1513

The HED Python validator represents each issue as a dictionary and the error list is a list of these dictionaries. The error code is used to select a template for composing the error message from the dictionary. Most of our calling functions then get a printable issue string for these. The issue list could trivially be output as JSON but it maximizing usefulness would take some thought.

I guess the first step for us would be to define a JSON schema for the format of the issue list (which we currently don't have). We have a lot of fields, some of which correspond to yours and some don't.

@VisLab

Thank you @VisLab , of special interest would be those which don't ATM have anything corresponding. Would you be so kind to create a list/table of them?

I went over the hed-python code. There are single issue and context keys. The actual issue objects are nested a tree structure so that the context that applies to multiple issues can be easily printed once. This tree could be flattened.

Single issue keys

Key	Description
code	External HED error code from the specification-- to be mapped to web-page explanation
message	Formatted message after parameters are filled in Ex:'Invalid character "x08" at index 7'
severity	Numerical code 1 == error 10 is warning
index_in_tag	Position of start of problem in tag
index_in_tag_end	Position of end of problem in tag
source_tag	Pointer to an object with a lot of info including link to object in schema if available

Context keys:

Key	Description
ec_title	overall title for error report
ec_filename	file name this error applies to
ec_sidecarColumnName	sidecar column name
ec_sidecarKeyName	name of a categorical value
ec_row	row number in a tsv file
e_column	column number in tsv file
ec_line	line number in the file for which error is reported
ec_HedString	HED String in which tag appears
ec_section	Section of the HED schema in which error appears (for Schema validation)
ec_schema_tag	Schema tag in which error appears (for Schema validation)
ec_attribute	Schema attribute for which error appears (For schema validation) }

Sample tree structure of the issues:


{'children': [], 
('ec_sidecarColumnName', 'defs'): {
    'children': [], 
	('ec_sidecarKeyName', 'def1'): {
	    'children': [], 
		('ec_HedString', '(Definition/Apple, Definition/Banana, (Blue))'): {
		    'children': [
				{  'code': 'DEFINITION_INVALID', 
				   'message': "Too many tags found in definition for Apple.  Expected 1, found: ['Definition/Banana']", 
				   'severity': 1, 
				   'ec_sidecarColumnName': 'defs', 
				   'ec_sidecarKeyName': 'def1', 
				   'ec_HedString': <hed.models.hed_string.HedString object at 0x000002AFFF59B2E0>
				}, 
				{'code': 'TAG_GROUP_ERROR', 
				    'message': "Multiple top level tags found in a single group.  First one found: Definition/Apple. Remainder:['Definition/Banana']  Problem spans string indexes: 1, 17", 
					'severity': 1, 
					'source_tag': <hed.models.hed_tag.HedTag object at 0x000002AFFF4A2520>, 
					'ec_filename': '', 
					'ec_sidecarColumnName': 'defs', 
					'ec_sidecarKeyName': 'def1', 
					'ec_HedString': <hed.models.hed_string.HedString object at 0x000002AFFF54B7F0>, 
					'char_index': 1, 'char_index_end': 17
					}
				]
				}
				}, ...

The sparc validator is a bit weird in that except in cases where something breaks due to a bug in the validator itself we return the export as is or remove the malformed data (with a note in the validator that it has been removed). The validator output is embedded as errors sections within objects and a summary is lifted out, however that is not currently described in the linked schema below because we don't validate that part of the export, so I will updated the schema so that it is at least visible.

https://github.com/SciCrunch/sparc-curation/blob/master/sparcur/schemas.py

    "path_error_report": {
      "#/inputs/dataset_description_file": {
        "error_count": 6,
        "messages": [
          "'description' is a required property",
          "'name' is a required property",
          "'protocol_url_or_doi' is a required property"
        ]
      },
      "#/inputs/manifest_file/-1/checksums/-1": {
        "error_count": 1,
        "messages": [
          "{'type': 'checksum', ... 44 bytes later ... e3531d7671eab8911'} is not valid under any of the given schemas"
        ]
      },
      "#/inputs/submission_file/submission": {
        "error_count": 1,
        "messages": [
          "{'consortium_data_st ... 69 bytes later ... er': 'U19NS130608'} is not valid under any of the given schemas"
        ]
      },
      "#/meta/award_number": {
        "error_count": 1,
        "messages": [
          "'U19NS130608' does not match '^(OT2OD|OT3OD|U18|TR|U01)'"
        ]
      },
      "#/meta/techniques/-1": {
        "error_count": 2,
        "messages": [
          "'RNA -seq' is not a 'iri'",
          "'single-cell RNA sequencing' is not a 'iri'"
        ]
      }
    }

@VisLab


{'children': [], 
('ec_sidecarColumnName', 'defs'): {
    'children': [], 
	('ec_sidecarKeyName', 'def1'): {
	    'children': [], 
		('ec_HedString', '(Definition/Apple, Definition/Banana, (Blue))'): {
		    'children': [
				{  'code': 'DEFINITION_INVALID', 
				   'message': "Too many tags found in definition for Apple.  Expected 1, found: ['Definition/Banana']", 
				   'severity': 1, 
				   'ec_sidecarColumnName': 'defs', 
				   'ec_sidecarKeyName': 'def1', 
				   'ec_HedString': <hed.models.hed_string.HedString object at 0x000002AFFF59B2E0>
				}, 
				{'code': 'TAG_GROUP_ERROR', 
				    'message': "Multiple top level tags found in a single group.  First one found: Definition/Apple. Remainder:['Definition/Banana']  Problem spans string indexes: 1, 17", 
					'severity': 1, 
					'source_tag': <hed.models.hed_tag.HedTag object at 0x000002AFFF4A2520>, 
					'ec_filename': '', 
					'ec_sidecarColumnName': 'defs', 
					'ec_sidecarKeyName': 'def1', 
					'ec_HedString': <hed.models.hed_string.HedString object at 0x000002AFFF54B7F0>, 
					'char_index': 1, 'char_index_end': 17
					}
				]
				}
				}, ...

@VisLab I am not sure if I understand this correctly and need some confirmation/clarification. The following are statements that need to be confirmed or corrected and questions that need to be answered.

Individual errors, each represented as a dictionary, are always contained within a list which is the value of a "children" key.
The location of a list of errors within the nested structure specifies the context of the errors.
Why does each error have keys such as 'ec_sidecarColumnName' and 'ec_sidecarKeyName' which are context keys, and some of these context keys are a context key in the path of the location of the error while others are not?

Individual errors, each represented as a dictionary, are always contained within a list which is the value of a "children" key.

The basic format for a HED issue is a simple dictionary whose fields vary depending on the issue, but code, message, and severity are always there and for the purposes of integration, this might be sufficient. Here is an example (from above).

{
   'code': 'TAG_GROUP_ERROR', 
   'message': "Multiple top level tags found in a single group.  First one found: Definition/Apple. Remainder:['Definition/Banana'] Problem spans string indexes: 1, 17", 
   'severity': 1, 
   'source_tag': <hed.models.hed_tag.HedTag object at 0x000002AFFF4A2520>, 
   'ec_filename': '', 
   'ec_sidecarColumnName': 'defs', 
   'ec_sidecarKeyName': 'def1', 
   'ec_HedString': <hed.models.hed_string.HedString object at 0x000002AFFF54B7F0>, 
   'char_index': 1, 'char_index_end': 17
}

The location of a list of errors within the nested structure specifies the context of the errors.

Why does each error have keys such as 'ec_sidecarColumnName' and 'ec_sidecarKeyName' which are context keys, and some of these context keys are a context key in the path of the location of the error while others are not?

Alll of the validators take an optional ErrorHandler object. If this is used, then when validating a sidecar, for example, the ErrorHandler pushes the sidecar context before starting the validation and pops it when the validation has completed. This allows all of the errors generated from that sidecar to be grouped. The messages can be augmented by sidecar context information such as the column key for which the sidecar error occurs, etc.

The get_printable_issue_string takes care of that so that the error messages come out nicely formatted with all of the errors for that sidecar under a single title.

We are open to producing an "Extracted" version which just has a list of simple dictionaries and all of the context information is in the error message itself if this would be useful for Dandi.

@VisLab

@VisLab Thanks for the detailed explanation.

We are open to producing an "Extracted" version which just has a list of simple dictionaries and all of the context information is in the error message itself if this would be useful for Dandi.

An "extracted" version would be nice. To make the extracted version accessible to machines as well as to humans, I think the context information of the error should be included as well as the human readable message.

The jsonschema Python package provides an elaborate representation for validation errors. May be we can adopt some of its features? 🙂

The jsonschema Python package provides an elaborate representation for validation errors. May be we can adopt some of its features? 🙂

This is a good idea, but maybe not exactly that. I'd like to see a code, severity, message being required. The context being flexible.

The jsonschema Python package provides an elaborate representation for validation errors. May be we can adopt some of its features? 🙂

This is a good idea, but maybe not exactly that. I'd like to see a code, severity, message being required. The context being flexible.

Oh, I mean to use the ValidationError from jsonschema only as a reference. In fact, the context attribute in the ValidationError doesn't have the same meaning as context in HED.

What I like from jsonschema is that individual ValidatorError objects are yielded from iter_errors, and each object contains all the info about an the represented error.

Okay I added an iter_errors generator to the develop branch of hed-python. See: hed-standard/hed-python#1053

For OME-Zarr, https://github.com/ome-zarr-models/ome-zarr-models-py is (I think) the only software that exists to do full validation of OME-Zarr data. We don't output any machine readable validation information at the moment.

It's definitely something we'd be open to exploring, so if anyone has any pointers or wants to open an issue to request this feature that would be great!

Dear Standards folks. Following up on the

[Feature]: Include version of NWB standard used in validation errors NeurodataWithoutBorders/pynwb#2062
How to get schema version? bids-standard/bids-validator#118

and others, I am adding in original description pointers to issues or solutions on how/where versions of the standards are listed in the output of the validators. Also I think it is worth while to list also names and versions of validator software itself -- this way, when arriving to a validation output file it would be clear what validator (and version) was used and against which standards (versions) it was validated against. ATM in dandi-cli we need to duplicate the effort of sensing versions of the standards and the tools.

@yarikoptic

@yarikoptic Could you be a little more specific about how you envision this being done -- i.e. where the information would be stored?

I am talking about serialization of the validator output. E.g. output of bids-validator (in summarized form) and especially bids-validator --json (likely per each validation record as we are aiming to do in dandi-cli ATM) should contain those records. Here is e.g. an example of the record we are composing in dandi-cli from the outputs of different validators, and in this case from output from bids-validator (pardon complicated invocation, we are working on proper interface):

❯ python -c "from json import dumps; from pydantic import TypeAdapter; from pathlib import Path; from dandi.validate_types import ValidationResult; from dandi.bids_validator_deno import bids_validate; result_adapter = TypeAdapter(list[ValidationResult]); print(result_adapter.dump_json(bids_validate(Path('/home/yoh/proj/bids/bids-examples/2d_mb_pcasl')), indent=2, exclude_none=True).decode())" | jq '.[0]'
{
  "id": "BIDS.JSON_KEY_RECOMMENDED",
  "origin": {
    "type": "VALIDATION",
    "validator": "bids-validator-deno",
    "validator_version": "2.0.5",
    "standard": "BIDS",
    "standard_version": "1.0.4"
  },
  "scope": "file",
  "severity": 20,
  "dataset_path": "/home/yoh/proj/bids/bids-examples/2d_mb_pcasl",
  "message": "A JSON file is missing a key listed as recommended.\nsubCode: License\nissueMessage: Field description: The license for the dataset.\nThe use of license name abbreviations is RECOMMENDED for specifying a license\n(see [Licenses](SPEC_ROOT/appendices/licenses.md)).\nThe corresponding full license text MAY be specified in an additional\n`LICENSE` file.\n",
  "path": "/home/yoh/proj/bids/bids-examples/2d_mb_pcasl/dataset_description.json"
}

so here you could see that this record is clearly associated with BIDS. Later we will show examples where we have records for different validators in the same output.

The Psych-DS validator typically either outputs results in a formatted list all at once or in a more linear checklist-like format (where only the first error encountered in the checklist is displayed). It can also be formatted as a JSON, which looks like this:

{
    "valid": false,
    "issues": [
        {
            "key": "INVALID_SCHEMAORG_PROPERTY",
            "severity": "warning",
            "reason": "The schema.org ontology contains a fixed set of legal properties which can be applied to objects within the metadata. If schema.org is used as the only @context within your metadata, then all properties will be interpreted as schema.org properties. Using an invalid schema.org property is not considered an error in the psych-DS specification, but it should be understood that such usages result in the property in question not being interpretable by machines. If you are having trouble creating your dataset_description.json file, you can download a template by filling out the form at https://psych-ds.github.io/cedar-wizard-psychds/.\n",
            "requires": [
                "rules.files.common.core.dataset_description",
                "rules.files.tabular_data.data.Datafile"
            ],
            "files": [
                {
                    "name": "file_metadata.json",
                    "path": "/data/file_metadata.json",
                    "expanded": {},
                    "issueInfo": [],
                    "evidence": "This file contains one or more keys that use the schema.org namespace, but are not  official schema.org properties.\n                      According to the psych-DS specification, this is not an error, but be advised that these terms will not be\n                      machine-interpretable and do not function as linked data elements. These are the keys in question: [.key]"
                }
            ]
        },
        {
            "key": "INVALID_OBJECT_TYPE",
            "severity": "warning",
            "reason": "Properties in the schema.org ontology have selective restrictions on which types of objects can be used for their values. Including an object with a @type that does not match the selective restrictions of its property is not an error in psych-DS, but it will result in the object in question not being interpretable by machines. If you are having trouble creating your dataset_description.json file, you can download a template by filling out the form at https://psych-ds.github.io/cedar-wizard-psychds/.\n",
            "requires": [
                "rules.files.common.core.dataset_description",
                "rules.files.tabular_data.data.Datafile"
            ],
            "files": [
                {
                    "name": "study-bfi_data.json",
                    "path": "/data/raw_data/study-bfi_data.json",
                    "expanded": {},
                    "issueInfo": [],
                    "evidence": "This file contains one or more objects with types that do not match the selectional constraints of their keys.\n                        Each schema.org property (which take the form of keys in your metadata json) has a specific range of types\n                        that can be used as its value. Type constraints for a given property can be found by visiting their corresponding schema.org\n                        URL. All properties can take strings or URLS as objects, under the assumption that the string/URL represents a unique ID.\n                        Type selection errors occured at the following locations in your json structure: [.author,.creator]"
                },
                {
                    "name": "dataset_description.json",
                    "path": "/dataset_description.json",
                    "expanded": {},
                    "issueInfo": [],
                    "evidence": "This file contains one or more objects with types that do not match the selectional constraints of their keys.\n                        Each schema.org property (which take the form of keys in your metadata json) has a specific range of types\n                        that can be used as its value. Type constraints for a given property can be found by visiting their corresponding schema.org\n                        URL. All properties can take strings or URLS as objects, under the assumption that the string/URL represents a unique ID.\n                        Type selection errors occured at the following locations in your json structure: [.author,.creator]"
                }
            ]
        },
        {
            "key": "OBJECT_TYPE_MISSING",
            "severity": "warning",
            "reason": "For compliance with the schema.org ontology, all objects within the metadata (with a few exceptions) that appear as the value of a schema.org key/property must contain a \"@type\" key with a valid schema.org type as its value. If you are having trouble creating your dataset_description.json file, you can download a template by filling out the form at https://psych-ds.github.io/cedar-wizard-psychds/.\n",
            "requires": [
                "rules.files.common.core.dataset_description",
                "rules.files.tabular_data.data.Datafile"
            ],
            "files": [
                {
                    "name": "dataset_description.json",
                    "path": "/dataset_description.json",
                    "expanded": {},
                    "issueInfo": [],
                    "evidence": "This file contains one or more objects without a @type property. Make sure that any object that you include\n                      as the value of a schema.org property contains a valid schema.org @type, unless it is functioning as some kind of \n                      base type, such as Text or URL, containing a @value key. @type is optional, but not required on such objects.\n                      The following objects without @type were found: [.variableMeasured]"
                }
            ]
        },
        {
            "key": "UNKNOWN_NAMESPACE",
            "severity": "warning",
            "reason": "The psych-DS validator only has access to one external vocabulary, \"http://schema.org\"; any other reference to an external schema is permitted, but the validity of the terms used cannot be confirmed. If you are having trouble creating your dataset_description.json file, you can download a template by filling out the form at https://psych-ds.github.io/cedar-wizard-psychds/.\n",
            "requires": [
                "rules.files.common.core.dataset_description",
                "rules.files.tabular_data.data.Datafile"
            ],
            "files": [
                {
                    "name": "dataset_description.json",
                    "path": "/dataset_description.json",
                    "expanded": {},
                    "issueInfo": [],
                    "evidence": "This file contains one or more references to namespaces other than https://schema.org:\n                      [https://linkml/minValue]."
                }
            ]
        },
        {
            "key": "JSON_KEY_REQUIRED",
            "severity": "error",
            "reason": "The metadata object listed below is missing a required key.",
            "requires": [
                "rules.files.common.core.dataset_description",
                "rules.files.tabular_data.data.Datafile"
            ],
            "files": [
                {
                    "name": "study-bfi_data.csv",
                    "path": "/data/raw_data/study-bfi_data.csv",
                    "expanded": {},
                    "issueInfo": [],
                    "evidence": "metadata object missing fields: [name] as per schema.rules.compiled_metadata.CompiledMetadata. \n                    If these fields appear to be present in your metadata, then there may be an issue with your schema.org context"
                }
            ]
        },
        {
            "key": "MISSING_CHANGES_DOC",
            "severity": "warning",
            "reason": "It is recommended to include a 'CHANGES.md' or 'CHANGES.txt' file in the base directory",
            "requires": [],
            "files": []
        },
        {
            "key": "MISSING_MATERIALS_DIRECTORY",
            "severity": "warning",
            "reason": "It is recommended to include subdirectory named 'materials' in the base directory",
            "requires": [],
            "files": []
        },
        {
            "key": "MISSING_DOCUMENTATION_DIRECTORY",
            "severity": "warning",
            "reason": "It is recommended to include subdirectory named 'documentation' in the base directory",
            "requires": [],
            "files": []
        }
    ],
    "summary": {
        "totalFiles": 7,
        "size": 359541,
        "dataProcessed": false,
        "dataTypes": [],
        "suggestedColumns": [
            "A1",
            "A2",
            "A3",
            "A4",
            "A5",
            "C1",
            "C2",
            "C3",
            "C4",
            "C5",
            "E1",
            "E2",
            "E3",
            "E4",
            "E5",
            "N1",
            "N2",
            "N3",
            "N4",
            "N5",
            "O1",
            "O2",
            "O3",
            "O4",
            "O5",
            "gender",
            "education",
            "age"
        ]
    }
}

For an example of how the more linear issue output looks, you could try using the web validator

Most of the errors that we output can be found in the 'errors' section of the YAML schema model

FWIW, some today's development and demos to

promote visidata as navigator for validation records: What if all validation records were in .jsonl files?! bids-standard/bids-examples#548
WiP in dandi-cli to provide various grouppings and output serializations, with primary being json lines used in above bids-examples: ENH: Machine-readable validate output with store/reload dandi/dandi-cli#1822

One outside reference we might want to consider: Static Analysis Results Interchange Format (SARIF)

Navigation Menu

Repository navigation

Please submit pointers to your validator output formats #1

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Please submit pointers to your validator output formats #1

Description

Activity

rly commented on Oct 3, 2024

yarikoptic commented on Oct 3, 2024

rly commented on Oct 4, 2024

VisLab commented on Oct 4, 2024

yarikoptic commented on Oct 4, 2024

VisLab commented on Oct 4, 2024

tgbugs commented on Oct 7, 2024

candleindark commented on Feb 21, 2025

VisLab commented on Feb 21, 2025

candleindark commented on Feb 23, 2025

VisLab commented on Feb 24, 2025

candleindark commented on Feb 24, 2025

VisLab commented on Feb 26, 2025

dstansby commented on Mar 4, 2025

yarikoptic commented on Apr 4, 2025

VisLab commented on Apr 4, 2025

yarikoptic commented on Apr 8, 2025

bleonar5 commented on Apr 25, 2025

yarikoptic commented on Mar 20, 2026

effigies commented on Jul 20, 2026

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions