CNN model cannot be loaded with latest keras and tensorflow

[kushalkolar]

1. Operating System (Linux, MacOS, Windows): Debian 11
2. Hardware type (x86, ARM..) and RAM: x86_64, Ryzen 5900X
3. Python Version (e.g. 3.9): 3.11
4. Caiman version (e.g. 1.9.12): 1.10.1

Component evaluation fails with latest tensorflow & keras since it cannot load the CNN model file, cnn_model.json. This is with keras v3.2.1 which was released 3 days ago and tensorflow v2.16.1 released on March 7.

This does not occur with keras v2.12.0 and tensorflow v2.12.0 (I'm not sure what's the last version in which the models can be loaded). Rolling back to these versions also allows the model file to be loaded.

Here is the traceback in-place when you try to use evaluate_components in caiman:

File "[/home/kushal/repos/CaImAn/caiman/source_extraction/cnmf/estimates.py", line 1036](http://localhost:8888/lab/tree/repos/mesmerize-core/notebooks/repos/CaImAn/caiman/source_extraction/cnmf/estimates.py#line=1035), in evaluate_components
    estimate_components_quality_auto(imgs, self.A, self.C, self.b, self.f, self.YrA,
  File "[/home/kushal/repos/CaImAn/caiman/components_evaluation.py", line 608](http://localhost:8888/lab/tree/repos/mesmerize-core/notebooks/repos/CaImAn/caiman/components_evaluation.py#line=607), in estimate_components_quality_auto
    idx_components, idx_components_bad, cnn_values = select_components_from_metrics(A, dims, gSig, r_values, comp_SNR,
                                                     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "[/home/kushal/repos/CaImAn/caiman/components_evaluation.py", line 653](http://localhost:8888/lab/tree/repos/mesmerize-core/notebooks/repos/CaImAn/caiman/components_evaluation.py#line=652), in select_components_from_metrics
    predictions, _ = evaluate_components_CNN(A, dims, gSig)
                     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "[/home/kushal/repos/CaImAn/caiman/components_evaluation.py", line 294](http://localhost:8888/lab/tree/repos/mesmerize-core/notebooks/repos/CaImAn/caiman/components_evaluation.py#line=293), in evaluate_components_CNN
    loaded_model = model_from_json(loaded_model_json)
                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "[/home/kushal/repos/mesmerize-viz/venv/lib/python3.11/site-packages/keras/src/models/model.py", line 571](http://localhost:8888/lab/tree/repos/mesmerize-core/notebooks/repos/mesmerize-viz/venv/lib/python3.11/site-packages/keras/src/models/model.py#line=570), in model_from_json
    return serialization_lib.deserialize_keras_object(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "[/home/kushal/repos/mesmerize-viz/venv/lib/python3.11/site-packages/keras/src/saving/serialization_lib.py", line 687](http://localhost:8888/lab/tree/repos/mesmerize-core/notebooks/repos/mesmerize-viz/venv/lib/python3.11/site-packages/keras/src/saving/serialization_lib.py#line=686), in deserialize_keras_object
    cls = _retrieve_class_or_fn(
          ^^^^^^^^^^^^^^^^^^^^^^
  File "[/home/kushal/repos/mesmerize-viz/venv/lib/python3.11/site-packages/keras/src/saving/serialization_lib.py", line 805](http://localhost:8888/lab/tree/repos/mesmerize-core/notebooks/repos/mesmerize-viz/venv/lib/python3.11/site-packages/keras/src/saving/serialization_lib.py#line=804), in _retrieve_class_or_fn
    raise TypeError(
TypeError: Could not locate class 'Sequential'. Make sure custom classes are decorated with `@keras.saving.register_keras_serializable()`. Full object config: {'class_name': 'Sequential', 'config': {'name': 'sequential', 'layers': [{'class_name': 'Conv2D', 'config': {'name': 'conv2d_20', 'trainable': True, 'batch_input_shape': [None, 50, 50, 1], 'dtype': 'float32', 'filters': 32, 'kernel_size': [3, 3], 'strides': [1, 1], 'padding': 'valid', 'data_format': 'channels_last', 'dilation_rate': [1, 1], 'activation': 'relu', 'use_bias': True, 'kernel_initializer': {'class_name': 'VarianceScaling', 'config': {'scale': 1.0, 'mode': 'fan_avg', 'distribution': 'uniform', 'seed': None, 'dtype': 'float32'}}, 'bias_initializer': {'class_name': 'Zeros', 'config': {'dtype': 'float32'}}, 'kernel_regularizer': None, 'bias_regularizer': None, 'activity_regularizer': None, 'kernel_constraint': None, 'bias_constraint': None}}, {'class_name': 'Activation', 'config': {'name': 'activation_13', 'trainable': True, 'dtype': 'float32', 'activation': 'relu'}}, {'class_name': 'Conv2D', 'config': {'name': 'conv2d_21', 'trainable': True, 'dtype': 'float32', 'filters': 32, 'kernel_size': [3, 3], 'strides': [1, 1], 'padding': 'valid', 'data_format': 'channels_last', 'dilation_rate': [1, 1], 'activation': 'linear', 'use_bias': True, 'kernel_initializer': {'class_name': 'VarianceScaling', 'config': {'scale': 1.0, 'mode': 'fan_avg', 'distribution': 'uniform', 'seed': None, 'dtype': 'float32'}}, 'bias_initializer': {'class_name': 'Zeros', 'config': {'dtype': 'float32'}}, 'kernel_regularizer': None, 'bias_regularizer': None, 'activity_regularizer': None, 'kernel_constraint': None, 'bias_constraint': None}}, {'class_name': 'Activation', 'config': {'name': 'activation_14', 'trainable': True, 'dtype': 'float32', 'activation': 'relu'}}, {'class_name': 'MaxPooling2D', 'config': {'name': 'max_pooling2d_10', 'trainable': True, 'dtype': 'float32', 'pool_size': [2, 2], 'padding': 'valid', 'strides': [2, 2], 'data_format': 'channels_last'}}, {'class_name': 'Dropout', 'config': {'name': 'dropout_17', 'trainable': True, 'dtype': 'float32', 'rate': 0.25, 'noise_shape': None, 'seed': None}}, {'class_name': 'Conv2D', 'config': {'name': 'conv2d_22', 'trainable': True, 'dtype': 'float32', 'filters': 64, 'kernel_size': [3, 3], 'strides': [1, 1], 'padding': 'same', 'data_format': 'channels_last', 'dilation_rate': [1, 1], 'activation': 'linear', 'use_bias': True, 'kernel_initializer': {'class_name': 'VarianceScaling', 'config': {'scale': 1.0, 'mode': 'fan_avg', 'distribution': 'uniform', 'seed': None, 'dtype': 'float32'}}, 'bias_initializer': {'class_name': 'Zeros', 'config': {'dtype': 'float32'}}, 'kernel_regularizer': None, 'bias_regularizer': None, 'activity_regularizer': None, 'kernel_constraint': None, 'bias_constraint': None}}, {'class_name': 'Activation', 'config': {'name': 'activation_15', 'trainable': True, 'dtype': 'float32', 'activation': 'relu'}}, {'class_name': 'Conv2D', 'config': {'name': 'conv2d_23', 'trainable': True, 'dtype': 'float32', 'filters': 64, 'kernel_size': [3, 3], 'strides': [1, 1], 'padding': 'valid', 'data_format': 'channels_last', 'dilation_rate': [1, 1], 'activation': 'linear', 'use_bias': True, 'kernel_initializer': {'class_name': 'VarianceScaling', 'config': {'scale': 1.0, 'mode': 'fan_avg', 'distribution': 'uniform', 'seed': None, 'dtype': 'float32'}}, 'bias_initializer': {'class_name': 'Zeros', 'config': {'dtype': 'float32'}}, 'kernel_regularizer': None, 'bias_regularizer': None, 'activity_regularizer': None, 'kernel_constraint': None, 'bias_constraint': None}}, {'class_name': 'Activation', 'config': {'name': 'activation_16', 'trainable': True, 'dtype': 'float32', 'activation': 'relu'}}, {'class_name': 'MaxPooling2D', 'config': {'name': 'max_pooling2d_11', 'trainable': True, 'dtype': 'float32', 'pool_size': [2, 2], 'padding': 'valid', 'strides': [2, 2], 'data_format': 'channels_last'}}, {'class_name': 'Dropout', 'config': {'name': 'dropout_18', 'trainable': True, 'dtype': 'float32', 'rate': 0.25, 'noise_shape': None, 'seed': None}}, {'class_name': 'Flatten', 'config': {'name': 'flatten_8', 'trainable': True, 'dtype': 'float32', 'data_format': 'channels_last'}}, {'class_name': 'Dense', 'config': {'name': 'dense_15', 'trainable': True, 'dtype': 'float32', 'units': 512, 'activation': 'linear', 'use_bias': True, 'kernel_initializer': {'class_name': 'VarianceScaling', 'config': {'scale': 1.0, 'mode': 'fan_avg', 'distribution': 'uniform', 'seed': None, 'dtype': 'float32'}}, 'bias_initializer': {'class_name': 'Zeros', 'config': {'dtype': 'float32'}}, 'kernel_regularizer': None, 'bias_regularizer': None, 'activity_regularizer': None, 'kernel_constraint': None, 'bias_constraint': None}}, {'class_name': 'Activation', 'config': {'name': 'activation_17', 'trainable': True, 'dtype': 'float32', 'activation': 'relu'}}, {'class_name': 'Dropout', 'config': {'name': 'dropout_19', 'trainable': True, 'dtype': 'float32', 'rate': 0.5, 'noise_shape': None, 'seed': None}}, {'class_name': 'Dense', 'config': {'name': 'dense_16', 'trainable': True, 'dtype': 'float32', 'units': 2, 'activation': 'linear', 'use_bias': True, 'kernel_initializer': {'class_name': 'VarianceScaling', 'config': {'scale': 1.0, 'mode': 'fan_avg', 'distribution': 'uniform', 'seed': None, 'dtype': 'float32'}}, 'bias_initializer': {'class_name': 'Zeros', 'config': {'dtype': 'float32'}}, 'kernel_regularizer': None, 'bias_regularizer': None, 'activity_regularizer': None, 'kernel_constraint': None, 'bias_constraint': None}}, {'class_name': 'Activation', 'config': {'name': 'activation_18', 'trainable': True, 'dtype': 'float32', 'activation': 'softmax'}}]}, 'keras_version': '2.2.4-tf', 'backend': 'tensorflow'}

I think that the reason is because the latest keras/tensorflow can no longer load the model file. If I do:

with open("/home/kushal/caiman_data/model/cnn_model.json", "r") as f:
     model_from_json(f.read())

I get:

---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
Cell In[9], line 2
      1 with open("/home/kushal/caiman_data/model/cnn_model.json", "r") as f:
----> 2     model_from_json(f.read())

File ~/repos/mesmerize-viz/venv/lib/python3.11/site-packages/keras/src/models/model.py:571, in model_from_json(json_string, custom_objects)
    568 from keras.src.saving import serialization_lib
    570 model_config = json.loads(json_string)
--> 571 return serialization_lib.deserialize_keras_object(
    572     model_config, custom_objects=custom_objects
    573 )

File ~/repos/mesmerize-viz/venv/lib/python3.11/site-packages/keras/src/saving/serialization_lib.py:687, in deserialize_keras_object(config, custom_objects, safe_mode, **kwargs)
    684     if obj is not None:
    685         return obj
--> 687 cls = _retrieve_class_or_fn(
    688     class_name,
    689     registered_name,
    690     module,
    691     obj_type="class",
    692     full_config=config,
    693     custom_objects=custom_objects,
    694 )
    696 if isinstance(cls, types.FunctionType):
    697     return cls

File ~/repos/mesmerize-viz/venv/lib/python3.11/site-packages/keras/src/saving/serialization_lib.py:805, in _retrieve_class_or_fn(name, registered_name, module, obj_type, full_config, custom_objects)
    802     if obj is not None:
    803         return obj
--> 805 raise TypeError(
    806     f"Could not locate {obj_type} '{name}'. "
    807     "Make sure custom classes are decorated with "
    808     "`@keras.saving.register_keras_serializable()`. "
    809     f"Full object config: {full_config}"
    810 )

TypeError: Could not locate class 'Sequential'. Make sure custom classes are decorated with `@keras.saving.register_keras_serializable()`. Full object config: {'class_name': 'Sequential', 'config': {'name': 'sequential', 'layers': [{'class_name': 'Conv2D', 'config': {'name': 'conv2d_20', 'trainable': True, 'batch_input_shape': [None, 50, 50, 1], 'dtype': 'float32', 'filters': 32, 'kernel_size': [3, 3], 'strides': [1, 1], 'padding': 'valid', 'data_format': 'channels_last', 'dilation_rate': [1, 1], 'activation': 'relu', 'use_bias': True, 'kernel_initializer': {'class_name': 'VarianceScaling', 'config': {'scale': 1.0, 'mode': 'fan_avg', 'distribution': 'uniform', 'seed': None, 'dtype': 'float32'}}, 'bias_initializer': {'class_name': 'Zeros', 'config': {'dtype': 'float32'}}, 'kernel_regularizer': None, 'bias_regularizer': None, 'activity_regularizer': None, 'kernel_constraint': None, 'bias_constraint': None}}, {'class_name': 'Activation', 'config': {'name': 'activation_13', 'trainable': True, 'dtype': 'float32', 'activation': 'relu'}}, {'class_name': 'Conv2D', 'config': {'name': 'conv2d_21', 'trainable': True, 'dtype': 'float32', 'filters': 32, 'kernel_size': [3, 3], 'strides': [1, 1], 'padding': 'valid', 'data_format': 'channels_last', 'dilation_rate': [1, 1], 'activation': 'linear', 'use_bias': True, 'kernel_initializer': {'class_name': 'VarianceScaling', 'config': {'scale': 1.0, 'mode': 'fan_avg', 'distribution': 'uniform', 'seed': None, 'dtype': 'float32'}}, 'bias_initializer': {'class_name': 'Zeros', 'config': {'dtype': 'float32'}}, 'kernel_regularizer': None, 'bias_regularizer': None, 'activity_regularizer': None, 'kernel_constraint': None, 'bias_constraint': None}}, {'class_name': 'Activation', 'config': {'name': 'activation_14', 'trainable': True, 'dtype': 'float32', 'activation': 'relu'}}, {'class_name': 'MaxPooling2D', 'config': {'name': 'max_pooling2d_10', 'trainable': True, 'dtype': 'float32', 'pool_size': [2, 2], 'padding': 'valid', 'strides': [2, 2], 'data_format': 'channels_last'}}, {'class_name': 'Dropout', 'config': {'name': 'dropout_17', 'trainable': True, 'dtype': 'float32', 'rate': 0.25, 'noise_shape': None, 'seed': None}}, {'class_name': 'Conv2D', 'config': {'name': 'conv2d_22', 'trainable': True, 'dtype': 'float32', 'filters': 64, 'kernel_size': [3, 3], 'strides': [1, 1], 'padding': 'same', 'data_format': 'channels_last', 'dilation_rate': [1, 1], 'activation': 'linear', 'use_bias': True, 'kernel_initializer': {'class_name': 'VarianceScaling', 'config': {'scale': 1.0, 'mode': 'fan_avg', 'distribution': 'uniform', 'seed': None, 'dtype': 'float32'}}, 'bias_initializer': {'class_name': 'Zeros', 'config': {'dtype': 'float32'}}, 'kernel_regularizer': None, 'bias_regularizer': None, 'activity_regularizer': None, 'kernel_constraint': None, 'bias_constraint': None}}, {'class_name': 'Activation', 'config': {'name': 'activation_15', 'trainable': True, 'dtype': 'float32', 'activation': 'relu'}}, {'class_name': 'Conv2D', 'config': {'name': 'conv2d_23', 'trainable': True, 'dtype': 'float32', 'filters': 64, 'kernel_size': [3, 3], 'strides': [1, 1], 'padding': 'valid', 'data_format': 'channels_last', 'dilation_rate': [1, 1], 'activation': 'linear', 'use_bias': True, 'kernel_initializer': {'class_name': 'VarianceScaling', 'config': {'scale': 1.0, 'mode': 'fan_avg', 'distribution': 'uniform', 'seed': None, 'dtype': 'float32'}}, 'bias_initializer': {'class_name': 'Zeros', 'config': {'dtype': 'float32'}}, 'kernel_regularizer': None, 'bias_regularizer': None, 'activity_regularizer': None, 'kernel_constraint': None, 'bias_constraint': None}}, {'class_name': 'Activation', 'config': {'name': 'activation_16', 'trainable': True, 'dtype': 'float32', 'activation': 'relu'}}, {'class_name': 'MaxPooling2D', 'config': {'name': 'max_pooling2d_11', 'trainable': True, 'dtype': 'float32', 'pool_size': [2, 2], 'padding': 'valid', 'strides': [2, 2], 'data_format': 'channels_last'}}, {'class_name': 'Dropout', 'config': {'name': 'dropout_18', 'trainable': True, 'dtype': 'float32', 'rate': 0.25, 'noise_shape': None, 'seed': None}}, {'class_name': 'Flatten', 'config': {'name': 'flatten_8', 'trainable': True, 'dtype': 'float32', 'data_format': 'channels_last'}}, {'class_name': 'Dense', 'config': {'name': 'dense_15', 'trainable': True, 'dtype': 'float32', 'units': 512, 'activation': 'linear', 'use_bias': True, 'kernel_initializer': {'class_name': 'VarianceScaling', 'config': {'scale': 1.0, 'mode': 'fan_avg', 'distribution': 'uniform', 'seed': None, 'dtype': 'float32'}}, 'bias_initializer': {'class_name': 'Zeros', 'config': {'dtype': 'float32'}}, 'kernel_regularizer': None, 'bias_regularizer': None, 'activity_regularizer': None, 'kernel_constraint': None, 'bias_constraint': None}}, {'class_name': 'Activation', 'config': {'name': 'activation_17', 'trainable': True, 'dtype': 'float32', 'activation': 'relu'}}, {'class_name': 'Dropout', 'config': {'name': 'dropout_19', 'trainable': True, 'dtype': 'float32', 'rate': 0.5, 'noise_shape': None, 'seed': None}}, {'class_name': 'Dense', 'config': {'name': 'dense_16', 'trainable': True, 'dtype': 'float32', 'units': 2, 'activation': 'linear', 'use_bias': True, 'kernel_initializer': {'class_name': 'VarianceScaling', 'config': {'scale': 1.0, 'mode': 'fan_avg', 'distribution': 'uniform', 'seed': None, 'dtype': 'float32'}}, 'bias_initializer': {'class_name': 'Zeros', 'config': {'dtype': 'float32'}}, 'kernel_regularizer': None, 'bias_regularizer': None, 'activity_regularizer': None, 'kernel_constraint': None, 'bias_constraint': None}}, {'class_name': 'Activation', 'config': {'name': 'activation_18', 'trainable': True, 'dtype': 'float32', 'activation': 'softmax'}}]}, 'keras_version': '2.2.4-tf', 'backend': 'tensorflow'}

[pgunn]

Interesting; conda-forge has had packages of 3.x available for 3 months:

https://anaconda.org/conda-forge/keras/files

but I don't have any environments with them. It looks like conda packages of tensorflow pin keras down very precisely:

https://github.com/conda-forge/tensorflow-feedstock/blob/main/recipe/meta.yaml#L153C7-L153C12

How did you end up running into this? Is this a pure-pip install?

[ethanbb]

Just chiming in that I also ran into this on my latest Windows dev install, for which I installed all requirements except tensorflow using conda and then switched to pip. This seemed like the most hassle-free way given that tensorflow 2 for Windows isn't on conda-forge. I think the recommended method on other issues has been to install tf from the defaults channel instead on Windows, but I know mixing conda-forge and defaults is also strongly discouraged and I had some difficulty when I tried this method previously.

Will try downgrading keras and cross my fingers!

[pgunn]

Hello,
I still recommend mixing defaults and conda-forge to build a caiman environment; most of our users on Windows do it that way and it seems to work ok. We may not need it after we remove tensorflow from the codebase, but that'll take a bit.

[ethanbb]

Yeah, pip's failure to take the compatibility of previously-installed packages into account (in this case keras) is frustrating. Anyway, my attempt to fix this failed because the environment had Python 3.12 and there isn't an old enough version of tensorflow available (2.16+ is not compatible with keras version 2, apparently). Will try to stick to your recommendation in the future.

Edit: I confirmed that keras 3.1 does not work; I assume we need 2.x.

[ethanbb]

For future reference (or maybe updating documentation?) here are the steps that currently works for me to install tensorflow without pip on Windows:

1. Edit environment.yml file:

• comment out - tensorflow
• change - python <=3.12 to - python <=3.10 (later versions are not compatible with available tensorflow versions)

2. Run commands (from CaImAn dir):

mamba env create  -f environment.yml -n caiman
mamba activate caiman
mamba install --override-channels -c conda-forge -c defaults defaults::tensorflow
mamba install vs2019_win-64
pip install -e .

Also maybe worth noting, installing in WSL2 is also possible and simpler, but I found it to be intolerably slower (not sure why exactly but could be because I am loading/saving data from/to a NAS).

[pgunn]

I would expect usually the mamba env create step to get you a working caiman env with a working tensorflow. The only case where we've seen that not happen is with alternative builds of conda that don't ship the defaults channel as a default. I think I can probably get around that by editing environment.yml to add defaults in explicitly (if that works, you wouldn't need to do a separate tensorflow install - if you like you could try this). It may also be necessary to disable strict channel priority (those alternate distros often enable it); that's also something I can do in the environment.yml

I made a branch with these changes:
https://github.com/flatironinstitute/CaImAn/tree/dev-environment_rework

I'm hoping it will work correctly right out of the box, removing at least the need for the separate tensorflow install step and the need to edit the environment.yml

[ethanbb]

Ah OK. I have been using Miniforge (which excludes the defaults channel and strongly discourages using it), and I thought that's what you recommended too, but now I see that the install page actually links to Miniconda. I would probably have had an easier time if I started from a Miniconda or Anaconda install.

[pgunn]

No worries; it'd be good for people to be able to install caiman more easily with Miniforge too; I think this change will hopefully make that so. Let me know if you get a chance to try it (with miniforge).

One of the reasons I don't want to be limited to miniconda is that people install conda for a variety of software, and if someone has 4 environments they use for other software and they want to add a caiman environment, it'd be a bit antisocial for me to ever tell someone to remove their conda install and get a different one; it'd mess up all those other environments. Obviously that's less of a problem if caiman is the only reason they're using conda, but we can't depend on that being the case.

[pgunn]

My bad; apparently channel_priority is not settable in environment.yml; the example snippet I found for it was actually for conda configs. There are proposals to add it but they haven't happened yet.

[pgunn]

If conda/conda#8675 ever gets implemented, we'll have a way, but at least for now the best we can do is just add defaults and hope that strict channel priority doesn't mess us up too much (we can add instructions to disable it).

This is just for people building caiman anyhow, which is the minority of our users.