Can I move artifacts to cuda or cpu after starting serve?

[ZeroAct]

I have only 8Gb gpu ram.
But, I want that the BentoService has multiple artifacts (over 8Gb total).

Is there any function bentoml provides?
Or do i have to implement this?

Thank you!

[parano]

Hi @ZeroAct - could you elaborate more on what you are trying to achieve? I'm not sure I understand your use case enough to answer the question. Is it related to dynamically loading and unloading models in a BentoServer?

[ZeroAct]

@parano
Is it related to dynamically loading and unloading models in a BentoServer?
Yes it is.

More specifically,
Gpu ram resource : 8 gb
model1 : 5 gb
model2 : 5 gb
model3 : 5 gb

I want to serve these three models(artifacts) with just one BentoService.
Is it possible to dynamically loading and unloading these models?

Thank you!