Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

feat: support running multiple engines at the same time #891

Merged
merged 3 commits into from
Jul 18, 2024
Merged

feat: support running multiple engines at the same time #891

merged 3 commits into from
Jul 18, 2024

Conversation

vansangpfiev
Copy link
Contributor

Describe Your Changes

  • Currently we only allow to run one engine at a time. Because cortex.trt-llm and cortex.onnx don't support embeddings yet, therefore we need to run cortex.llamacpp for embeddings along side with cortex.trt-llm/cortex.onnx for chat completions.
  • This PR also includes a commit to refactor the codebase.

@vansangpfiev vansangpfiev requested a review from louis-jan July 18, 2024 06:54
@vansangpfiev vansangpfiev marked this pull request as ready for review July 18, 2024 06:54
louis-jan
louis-jan approved these changes Jul 18, 2024
View reviewed changes
Copy link
Contributor

@louis-jan louis-jan left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@vansangpfiev vansangpfiev merged commit fd7c40d into dev Jul 18, 2024
5 checks passed
@vansangpfiev vansangpfiev deleted the feat/multi-engines branch July 18, 2024 06:59
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@louis-jan louis-jan louis-jan approved these changes

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
2 participants
@vansangpfiev @louis-jan