Support parallel inference?

[mmiumiu]

How can the model support parallel inference?

For a single model instance, even a small number of concurrent requests significantly increases response time. From what I've observed, the model only accepts one text input at a time and returns the corresponding audio response. Is it possible to make some simple changes to allow it to process multiple texts simultaneously without slowing down?

[prad-human-007]

You can run multiple instances of the model using Python's ProcessPoolExecutor.