Is it possible to validate a stream instead of validating completely loaded object?

[anton-daneyko-ultramarin]

I would like to validate a large JSON without loading it into memory completely. Is there a way to do that? If not, what are the challenges implementing it?

[by-ae]

Not directly. jsonschema validates a Python object that is already in memory (validate() / iter_errors() take a decoded instance), and some keywords need the whole value at once anyway: uniqueItems, contains/minContains, maxItems, required on a big object, if/then, unevaluated*, and so on. A true streaming validator would need a different, event-driven design.

The common case is a huge array of records, and you can handle that by streaming the records with ijson and validating each one against the items subschema. If that subschema uses $ref, use evolve() on a validator built from the full schema so the refs still resolve against the root:

import ijson
from jsonschema import Draft202012Validator

schema = {
    "$defs": {"record": {"type": "object", "required": ["id"],
                         "properties": {"id": {"type": "integer"}, "price": {"type": "number"}}}},
    "type": "object",
    "properties": {"records": {"type": "array", "items": {"$ref": "#/$defs/record"}}},
}

root = Draft202012Validator(schema)
item_validator = root.evolve(schema=schema["properties"]["records"]["items"])

with open("big.json", "rb") as f:
    for i, record in enumerate(ijson.items(f, "records.item")):
        for error in item_validator.iter_errors(record):
            print(f"records[{i}]: {error.message}")

I tested this on a 15 MB file with 500,001 records, the last one bad (jsonschema 4.26, ijson 3.6). It printed:

records[500000]: 'id' is a required property
records[500000]: 'x' is not of type 'number'

Peak RSS was about 20 MB, against about 160 MB for json.load of the same file. Caveats: array-level keywords (uniqueItems, maxItems, contains) and anything outside the streamed path are not checked, so validate those separately (count items as you go, etc.). Also, ijson yields non-integer numbers as decimal.Decimal by default. jsonschema treats those as numbers, but multipleOf with a float divisor currently raises TypeError on a Decimal (see #1515), so pass use_float=True to ijson.items if you use it.

[anton-daneyko-ultramarin]

Thanks. That's exactly what I ended up doing back then.