Where can I get started to convert internvl model to mlx format?

[hiima123]

https://github.com/OpenGVLab/InternVL

[100ZZ]

mlx_lm.convert --hf-path Mini-InternVL-Chat-2B-V1-5 --mlx-path Mini-InternVL-Chat-2B-V1-5-MLX
[INFO] Loading
ERROR:root:Model type internvl_chat not supported.
Traceback (most recent call last):
File "/opt/homebrew/Caskroom/miniconda/base/envs/mlx/lib/python3.11/site-packages/mlx_lm/utils.py", line 56, in _get_classes
arch = importlib.import_module(f"mlx_lm.models.{model_type}")
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/homebrew/Caskroom/miniconda/base/envs/mlx/lib/python3.11/importlib/init.py", line 126, in import_module
return _bootstrap._gcd_import(name[level:], package, level)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "", line 1204, in _gcd_import
File "", line 1176, in _find_and_load
File "", line 1140, in _find_and_load_unlocked
ModuleNotFoundError: No module named 'mlx_lm.models.internvl_chat'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/opt/homebrew/Caskroom/miniconda/base/envs/mlx/bin/mlx_lm.convert", line 8, in
sys.exit(main())
^^^^^^
File "/opt/homebrew/Caskroom/miniconda/base/envs/mlx/lib/python3.11/site-packages/mlx_lm/convert.py", line 58, in main
convert(**vars(args))
File "/opt/homebrew/Caskroom/miniconda/base/envs/mlx/lib/python3.11/site-packages/mlx_lm/utils.py", line 658, in convert
model, config, tokenizer = fetch_from_hub(model_path, lazy=True)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/homebrew/Caskroom/miniconda/base/envs/mlx/lib/python3.11/site-packages/mlx_lm/utils.py", line 464, in fetch_from_hub
model = load_model(model_path, lazy)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/homebrew/Caskroom/miniconda/base/envs/mlx/lib/python3.11/site-packages/mlx_lm/utils.py", line 392, in load_model
model_class, model_args_class = _get_classes(config=config)
^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/homebrew/Caskroom/miniconda/base/envs/mlx/lib/python3.11/site-packages/mlx_lm/utils.py", line 60, in _get_classes
raise ValueError(msg)
ValueError: Model type internvl_chat not supported.

[awni]

That's a multi-modal model, so it won't run in MLX LM. You need to manually write a model file that is capable of running it. For some examples of VLMs check-out MLX VLM by @Blaizzy.

To learn more about how to convert PyTorch models to MLX you can also look at some example models in this repo. I don't know of any good guides to help with that yet. Maybe that's something we should work on 🤔

[Blaizzy]

Thanks for the shout out @awni!

Hey @hiima123 and @100ZZ , we have InternVL on our roadmap.

Read more here:
Blaizzy/mlx-vlm#39

You can also send us a PR, and I will happiply review it. Here is an example:
Blaizzy/mlx-vlm#43

[Blaizzy]

To learn more about how to convert PyTorch models to MLX you can also look at some example models in this repo. I don't know of any good guides to help with that yet. Maybe that's something we should work on 🤔

Working on it...
I will have a series detailing how to convert VLM and LM to MLX soon.

[OKHand-Zy]

Hello~ I try to cover BELLE-2/Belle-distilwhisper-large-v2-zh model to MLX format . but i meet some error.
my setp .

1. git clone https://huggingface.co/BELLE-2/Belle-distilwhisper-large-v2-zh
2. python convert.py --torch-name-or-path ori-model/Belle-distilwhisper-large-v2-zh/pytorch_model.bin --mlx-path mlx_models/Belle-distilwhisper-large-v2-zh
3. so i meet this error

(whisper-p3.11) astl@ASTLs-Mac-mini whisper % python convert.py --torch-name-or-path ori-model/Belle-distilwhisper-large-v2-zh/pytorch_model.bin --mlx-path mlx_models/Belle-distilwhisper-large-v2-zh
[INFO] Loading
/Users/astl/RemoteFolder/mlx-examples/whisper/convert.py:142: FutureWarning: You are using `torch.load` with `weights_only=False` (the current default value), which uses the default pickle module implicitly. It is possible to construct malicious pickle data which will execute arbitrary code during unpickling (See https://github.com/pytorch/pytorch/blob/main/SECURITY.md#untrusted-models for more details). In a future release, the default value for `weights_only` will be flipped to `True`. This limits the functions that could be executed during unpickling. Arbitrary objects will no longer be allowed to be loaded via this mode unless they are explicitly allowlisted by the user via `torch.serialization.add_safe_globals`. We recommend you start setting `weights_only=True` for any use case where you don't have full control of the loaded file. Please open an issue on GitHub for any issues related to this experimental feature.
  checkpoint = torch.load(fp)
Traceback (most recent call last):
  File "/Users/astl/RemoteFolder/mlx-examples/whisper/convert.py", line 321, in <module>
    model = torch_to_mlx(load_torch_model(args.torch_name_or_path), dtype)
                         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/astl/RemoteFolder/mlx-examples/whisper/convert.py", line 142, in load_torch_model
    checkpoint = torch.load(fp)
                 ^^^^^^^^^^^^^^
  File "/opt/homebrew/Caskroom/miniconda/base/envs/whisper-p3.11/lib/python3.11/site-packages/torch/serialization.py", line 1097, in load
    return _load(
           ^^^^^^
  File "/opt/homebrew/Caskroom/miniconda/base/envs/whisper-p3.11/lib/python3.11/site-packages/torch/serialization.py", line 1525, in _load
    result = unpickler.load()
             ^^^^^^^^^^^^^^^^
  File "/opt/homebrew/Caskroom/miniconda/base/envs/whisper-p3.11/lib/python3.11/site-packages/torch/serialization.py", line 1492, in persistent_load
    typed_storage = load_tensor(dtype, nbytes, key, _maybe_decode_ascii(location))
                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/homebrew/Caskroom/miniconda/base/envs/whisper-p3.11/lib/python3.11/site-packages/torch/serialization.py", line 1466, in load_tensor
    wrap_storage=restore_location(storage, location),
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/homebrew/Caskroom/miniconda/base/envs/whisper-p3.11/lib/python3.11/site-packages/torch/serialization.py", line 414, in default_restore_location
    result = fn(storage, location)
             ^^^^^^^^^^^^^^^^^^^^^
  File "/opt/homebrew/Caskroom/miniconda/base/envs/whisper-p3.11/lib/python3.11/site-packages/torch/serialization.py", line 391, in _deserialize
    device = _validate_device(location, backend_name)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/homebrew/Caskroom/miniconda/base/envs/whisper-p3.11/lib/python3.11/site-packages/torch/serialization.py", line 364, in _validate_device
    raise RuntimeError(f'Attempting to deserialize object on a {backend_name.upper()} '
RuntimeError: Attempting to deserialize object on a CUDA device but torch.cuda.is_available() is False. If you are running on a CPU-only machine, please use torch.load with map_location=torch.device('cpu') to map your storages to the CPU.
(whisper-p3.11) astl@ASTLs-Mac-mini whisper %

how to solve this error? add some arg in CLI ?

update :
i try use mlx_lm.convert but same have error.
=> python -m mlx_lm.convert --hf-path BELLE-2/Belle-distilwhisper-large-v2-zh -q

(whisper-p3.11) astl@ASTLs-Mac-mini mlx-examples % python -m mlx_lm.convert --hf-path BELLE-2/Belle-distilwhisper-large-v2-zh -q
[INFO] Loading
Fetching 10 files: 100%|███████████████████████████████████████████████████████████████████████████| 10/10 [00:00<00:00, 60176.53it/s]
ERROR:root:No safetensors found in /Users/astl/.cache/huggingface/hub/models--BELLE-2--Belle-distilwhisper-large-v2-zh/snapshots/4fcdfe1df2ce31ebdbc11382f8e9f70c5975d192
Traceback (most recent call last):
  File "<frozen runpy>", line 198, in _run_module_as_main
  File "<frozen runpy>", line 88, in _run_code
  File "/opt/homebrew/Caskroom/miniconda/base/envs/whisper-p3.11/lib/python3.11/site-packages/mlx_lm/convert.py", line 62, in <module>
    main()
  File "/opt/homebrew/Caskroom/miniconda/base/envs/whisper-p3.11/lib/python3.11/site-packages/mlx_lm/convert.py", line 58, in main
    convert(**vars(args))
  File "/opt/homebrew/Caskroom/miniconda/base/envs/whisper-p3.11/lib/python3.11/site-packages/mlx_lm/utils.py", line 661, in convert
    model, config, tokenizer = fetch_from_hub(model_path, lazy=True)
                               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/homebrew/Caskroom/miniconda/base/envs/whisper-p3.11/lib/python3.11/site-packages/mlx_lm/utils.py", line 467, in fetch_from_hub
    model = load_model(model_path, lazy)
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/homebrew/Caskroom/miniconda/base/envs/whisper-p3.11/lib/python3.11/site-packages/mlx_lm/utils.py", line 389, in load_model
    raise FileNotFoundError(f"No safetensors found in {model_path}")
FileNotFoundError: No safetensors found in /Users/astl/.cache/huggingface/hub/models--BELLE-2--Belle-distilwhisper-large-v2-zh/snapshots/4fcdfe1df2ce31ebdbc11382f8e9f70c5975d192
(whisper-p3.11) astl@ASTLs-Mac-mini mlx-examples %

(whisper-p3.11) astl@ASTLs-Mac-mini mlx-examples % cd /Users/astl/.cache/huggingface/hub/models--BELLE-2--Belle-distilwhisper-large-v2-zh/snapshots/
(whisper-p3.11) astl@ASTLs-Mac-mini snapshots % ls
4fcdfe1df2ce31ebdbc11382f8e9f70c5975d192
(whisper-p3.11) astl@ASTLs-Mac-mini snapshots %

this mlx_lm.convert no safetensors file , it can't cover to mlx format?
and this project convert.py too?

[awni]

The MLX example doesn't support loading / converting that format of model. Even fixing the device issue would require some changes to our convert script and/or a model class with the appropriate names to match the Transformers WhisperModel.

[OKHand-Zy]

OK~ I know MLX example not support this model to cover.
When I use mlx_lm.convert, I encounter issues. Is it because it is not yet supported, or does mlx_lm.convert only work with models that have safetensors files?

[awni]

It doesn't have to do with safetensors or not. The model type is not yet supported.