Does lazy loading validate and/or convert fields into tag's type upon read?

[ketozhang]

I understand there is a special case with !ndarray tag gets converted to asdf.types.NDArrayType immediately. What happens to the other tag types. I can think of a few possibilities:

• Fields are immediately converted to the tag's model. Instantiation of the class is a form of validation.
• It's kept as a YAML-parsed dictionary. Validation only occurs if ASDF schema exists. Warning is raised if the extensions installed does not contain tag.
• YAML is not parsed at all. Validation does not occur. Grabbing deeply nested field parses YAML as needed.

[braingram]

Thanks for opening this discussion. Hopefully I didn't go overboard with my response :)

lazy_load only impacts loading ASDF blocks. This primarily impacts ndarray tags and NDArrayType where, when lazy_load=True the ASDF block won't be read from disk until the data in the NDArrayType is accessed for the first time. Here's an example:

import asdf
import numpy as np

arr = np.zeros(100)

asdf.AsdfFile({"arr": arr}).write_to("arr.asdf")

with asdf.open("arr.asdf", lazy_load=True) as af:
    print(af["arr"])
    print(af["arr"][0])
    print(af["arr"])

Running the above example gives the following output:

<array (unloaded) shape: [100] dtype: float64>
0.0
[0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0.
 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0.
 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0.
 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0.
 0. 0. 0. 0.]

Note the <array (unloaded) shape: [100] dtype: float64> printed before the array data was accessed. This allows asdf to open a file containing blocks (which map to arrays) quickly by waiting to load the blocks only when the array data is accessed.

Validation is a separate process that happens on the tagged YAML representation of the tree. For the above example, the call to asdf.open (using the default validate_on_read=True, set in the asdf config) will:

• read the YAML header of the ASDF file
• validate the YAML header (using schemas looked up based on the YAML tags)
• convert the YAML header to custom objects (again based on the YAML tags and what converters support those tags)

To add some comments to the possibilities you mentioned (to hopefully provide a bit more context and information about what's happening):

Fields are immediately converted to the tag's model. Instantiation of the class is a form of validation.

The conversion of a tagged YAML representation to a custom (python) object (deserialization) occurs during asdf.open. This might involve instantiation of a custom class but that's not considered validation (it still occurs even if validate_on_read=False). "validation" for ASDF is the checking of the tagged YAML representation against the schemas.

It's kept as a YAML-parsed dictionary. Validation only occurs if ASDF schema exists. Warning is raised if the extensions installed does not contain tag.

At the end of asdf.open the tree is fully deserialized (even if lazy_load=True) and the YAML representation is discarded (with one caveat below). If the AsdfFile is re-written a new YAML representation will be generated (by the converters in the extensions). Unknown tags are one exception. These will produce AsdfConversionWarning messages and be put in the tree as a Tagged object which largely captures the tagged YAML representation.

YAML is not parsed at all. Validation does not occur. Grabbing deeply nested field parses YAML as needed.

I don't think there is any combination of options currently that would produce this behavior. The closest match would be the "lazy tree" feature discussed here.