core: Multimodal Embeddings proposal

[fzowl]

Thank you for contributing to LangChain!

• PR title: "package: description"
  • Where "package" is whichever of langchain, community, core, etc. is being modified. Use "docs: ..." for purely docs changes, "infra: ..." for CI changes.
  • Example: "community: add foobar LLM"

Description: this is a proposal on how to introduce multimodal embedding models to Langchain

• Lint and test: Run make format, make lint and make test from the root of the package(s) you've modified. See contribution guidelines for more: https://python.langchain.com/docs/contributing/

Additional guidelines:

• Make sure optional dependencies are imported within a function.
• Please do not add dependencies to pyproject.toml files (even optional ones) unless they are required for unit tests.
• Most PRs should not touch more than one package.
• Changes should be backwards compatible.
• If you are adding something to community, do not re-import it in langchain.

If no one reviews your PR within a few days, please @-mention one of baskaryan, efriis, eyurtsev, ccurme, vbarda, hwchase17.

[vercel]

The latest updates on your projects. Learn more about Vercel for Git ↗︎

Name	Status	Preview	Comments	Updated (UTC)
langchain	✅ Ready (Inspect)	Visit Preview	💬 Add feedback	Dec 19, 2024 11:38am

[efriis]

worth checking out #25356 as well!

[fzliu]

I think perhaps we can figure out a way to make this a bit more universal -- applying not just to different types of modalities (audio, video, etc) but also to interleaved data or possibly even multiple parallel "tracks" of data (e.g. few video frames plus audio of those frames, embedded together).

@efriis From the Voyage perspective, we would love to support interleaved inputs (texts + images) to embedding models. Let us know what makes the most sense.

[ccurme]

Thanks very much for the thought put into this. Closing in favor of #25356. @fzowl @fzliu would you mind providing your feedback there?