roadmap: llamacpp-engine to align with llama.cpp upstream

[dan-homebrew]

Goal

• Goal: Can we have a minimalist fork of llama.cpp as llamacpp-engine
  • cortex.cpp's desktop focus means Drogon's features are unused
  • We should contribute our vision and multimodal work upstream as a form of llama.cpp server
  • Very clear Engines abstraction (i.e. support OpenVino etc in the future)
• Goal: Contribute upwards to llama.cpp
  • Vision, multimodal
  • May not be possible if the vision, audio encoders are Python-runtime based

Can we consider refactoring llamacpp-engine to use the server implementation, and maintain a fork with our improvements to speech, vision etc? This is especially if we do a C++ implementation of whisperVQ in the future.

[vansangpfiev]

I agree that we should align with the llama.cpp upstream, but I have several concerns:

• Drogon is part of cortex.cpp, we have already removed it from llama-cpp engine. If we remove Drogon from cortex.cpp, we need to find a replacement, which will be costly.
• Repository Structure: Forking the server implementation will necessitate changes to our repository structure, since we currently use llama.cpp as a submodule.
• Our current version differs significantly from the upstream version, which will require considerable time for refactoring.

[vansangpfiev]

Task list:

• Fork llama.cpp and try to use llama.cpp server
• Split vision model flow with chat model flow
• llama.cpp server should be a new process
• Test with OpenAI API compatible features that Alex and James added
• CI
• Docs

Related tickets need to be tested and verified:

• feat: Support response_format and structured JSON responses. jan#3785
• feat: [support log prob like OpenAI API] cortex.llamacpp#262
• feat: Cortex API supports /embeddings #1102
• feat: [support return multiple choices] cortex.llamacpp#264
• feat: [support logit_bias for OpenAI API compatible] cortex.llamacpp#263
• roadmap: Cortex can support Reasoning Models with chat_templates #1758
• feat: [support stream_option for OpenAI API compatible] cortex.llamacpp#265

Approach 1: cortex.llamacpp spawns llama.cpp server as a new process

• pros: can directly use llama.cpp server binary
• cons: spawn a new process is expensive and hard to control the process lifetime

Approach 2: Build llama.cpp server as a library and load it into cortex.llamacpp process

• pros: llama.cpp server can be embedded into cortex.llamacpp
• cons: will need to apply a patch to build llama.cpp server as a library