🔶 Add asyncIO feature for optimization of batch_translate

[jiaulislam]

Description

At current version of deep-translator(1.10.1) batch_translate is doing a linear loop of same function of single translate function. So in that context request is creating new session every time for requesting url to translate. Which is causing translator slow. Also It's working synchronously as each time request is blocking so until that request is finished translator is waiting idle. It can be optimized with asyncio feature also adding a dependency of aiohttp package and further for duplicate request we can utilize lru_cache system. Theres still long way to go for every translator to add the asyncio.

Changes Made

Changes are going to be non-breaking. Maybe we could add different function for async versions. But new dependency should be added aside of request we need to add aiohttp for async http calls.

• Making new async version function as async_translate_batch for high level function (non-breaking change which adds functionality)
• Added only for Google Translator with updated tests [ Function Docs Will be added in future commits ]
• Added _async_translate as abstract method to translate a single text asynchronusly in base.py which should be implemented in every translator

Checklist:

• My code follows the style guidelines of this project
• I have performed a self-review of my code
• I have added tests that prove my fix is effective or that my feature works
• Make the asyncio aiohttp dependency optional
• Add asyncio for deepl translator with tests
• Add asyncio for libre translator with tests
• Add asyncio for linguee translator with tests
• Add asyncio for microsoft translator with tests
• Add asyncio for mymemory translator with tests
• Add asyncio for papago translator with tests
• Add asyncio for pons translator with tests
• Add asyncio for qcri translator with tests
• Add asyncio for yandex translator with tests

Proof Of Concept

Usage

from deep_translator.google import GoogleTranslator
import asyncio

def medium_batch_bengali():
    return [
        "আমি ভালো আছি। তুমি কেমন আছো?",
        "আমার স্কুলে বিশাল লাইব্রেরি আছে।",
        "আমরা বাংলাদেশের জন্য সেরা করে চেষ্টা করছি।",
        "আমরা দীর্ঘ সময় ধরে একসময় সবাই প্রত্যেকের সাথে যুক্ত থাকতে চাই।",
    ]

async def main():
    translator = GoogleTranslator(target="en")
    translated_texts = await translator.async_translate_batch(medium_batch_bengali())

if __name__== "__main__":
    asyncio.run(main())

[jiaulislam]

@nidhaloff , It would be great if you provide some of observations here.

[nidhaloff]

Thanks, I really appreciate this work. This is a great feature to add.
I will take a deeper look, but for the moment I only noticed the additional dependencies and I would like to make it optional by default. So it would only be usable if the user wants to. In that case users should install the async version using:
pip install deep-translator[async]

This can be done using extra_requires. You can check the docx or pdf extras to see how this can be done

[nidhaloff]

It would be better to make this optional, because otherwise this would introduce a new dependency and increase the size of the package. Users will not like this (myself included). Generally I'm against adding dependencies, but if its optional, then its ok, so IMO just make it optional and add an extra_requires: async for it

[jiaulislam]

Ok adding to checklist for the task.

[nidhaloff]

I think it would be better to not re-create an async_translate function. Instead maybe it would be better to add an async=False parameter to the normal translate function and then basically reuse the translate function with async=True inside your async_translate_batch function.

That way, you will not have to re-define another translate function, which will be 90% similar to the normal translate function and hence repeating yourself.

[nidhaloff]

You will need a try catch here when you make this depedency optional or maybe just add the import inside the function where this will be used. For an example, check out the docx or pypdf dependencies.

[nidhaloff]

Would be better if you make the cache_maxsize value in a separate global config.py file

[nidhaloff]

Can you add this custom exception to exceptions.py just to keep everything consistent? Something like a NotValidInputBatch exception

[nidhaloff]

I think you can get a ValueError here if translation_tasks is empty. You may want to add a check for that

[nidhaloff]

Since most of the code in this function is the same as the non-async translate, maybe you can find a way to make some parts reusable

[jiaulislam]

@nidhaloff I'll try to keep all the things in mind and fix it. Thanks for you detailed explanation.