Accept buffer in LLMPipeline ctor #1262
Conversation
github-actions
bot
added
category: LLM
| if (m_core) { | ||
| return m_core; | ||
| } | ||
| m_core = std::make_shared<ov::Core>(); |
There was a problem hiding this comment.
| @@ -260,6 +260,23 @@ void slice_matmul_statefull_model(std::shared_ptr<ov::Model> model) { | |||
| } | |||
| } | |||
|
|
|||
| template <typename T> | |||
There was a problem hiding this comment.
template function cannot be defined in .cpp file, how does it work?
|
|
||
| auto ov_tokenizer = core.read_model(tokenizer_model_str, tokenizer_weights_tensor); | ||
| auto ov_detokenize = core.read_model(detokenizer_model_str, detokenizer_weights_tensor); | ||
| *this = TokenizerImpl(std::make_pair(ov_tokenizer, ov_detokenize), properties); |
There was a problem hiding this comment.
instead of such hacks, we can check whether tokenizer_model_str and tokenizer_weights_tensor are not empty, then call read_model.
In this case we don' need TokenizerImpl(const std::pair<std::shared_ptr<ov::Model>, std::shared_ptr<ov::Model>>& models, const ov::AnyMap& properties)
| std::string model_path = models_path + "/openvino_model.xml"; | ||
| std::string weights_path = std::regex_replace(model_path, std::regex(".xml"), ".bin"); | ||
| std::ifstream model_file(model_path, std::ios::binary | std::ios::ate); | ||
| std::ifstream weights_file(weights_path, std::ios::binary | std::ios::ate); |
There was a problem hiding this comment.
let's create a dedicated samples, which demonstrates that models are loaded via some "decrypt_llm_model", "decrypt_tokenizer", "decrypt_detokenizer" functions? such functions just read files to str + Tensor and return them
Note, str + Tensor can be ONNX, PDPD or other model type
There was a problem hiding this comment.
This sample was added by mistake. Of course for reading from buffer i will crease a separate sample.
| Tokenizer( | ||
| std::string& tokenizer_model_str, | ||
| ov::Tensor& tokenizer_weights_tensor, | ||
| std::string& detokenizer_model_str, |
There was a problem hiding this comment.
| std::string& detokenizer_model_str, | |
| const std::string& detokenizer_model_str, |
github-actions
bot
added
category: continuous batching
Ticket: CVS-158144