Reducing the number of API calls on large batch translations

[NorthOC]

An API call is made for each item in translate_batch, even if the same item is repeated a hundred of times. This is slow. That is why my pull request aims to fix that:

1. When a new batch item is translated, it is stored in a dictionary of already_translated items. The key for the translation in this dictionary is the previously untranslated sentence.
2. The next item in the batch is then checked against the keys of already_translated items and, if an identical match is found, uses the existing translation.
3. If there is no identical match, an API call is made for the translation.

Even with hundreds of items in a dictionary, checking against its keys is significantly faster than an API call.

The best part is that the already_translated dictionary will be used for the next big batch because it is initialized with the BaseTranslator class. In short, this means that only unique items will require an API call, while repeated items will be translated swiftly.

example:

translator = LibreTranslator(source='auto', target='es')
batch = ['Hello world', 'I am a python', 'Hello world', 'Hello world', 'I am a python']

translated_batch = translator.translate_batch(batch)
# API call (slow)
# API call (slow)
# Already translated (fast)
# Already translated (fast)
# Already translated (fast)
# Batch translation finished

# translator._already_translated = {
#   'Hello world' : 'Hola mundo',
#   'I am a python' : 'Soy un pitón'
# }

[nidhaloff]

is it .keys or .keys()?
btw I think you can even remove the .keys since it checks keys by default

[jiaulislam]

@nidhaloff @NorthOC , Just a thought, wouldn't it be nice to use the python builtin lru_cache system here ?

[nidhaloff]

@jiaulislam Great idea, yes it would be nice

[NorthOC]

is it .keys or .keys()? btw I think you can even remove the .keys since it checks keys by default

Removed redundant keys method on dict when iterating.

@nidhaloff @NorthOC , Just a thought, wouldn't it be nice to use the python builtin lru_cache system here ?

No idea how caching works :/

[nidhaloff]

@jiaulislam @NorthOC Alright, I think it's ok to merge this.
Maybe @jiaulislam can take a look at caching in a separate PR, if you have time of course.

[jiaulislam]

@jiaulislam @NorthOC Alright, I think it's ok to merge this. Maybe @jiaulislam can take a look at caching in a separate PR, if you have time of course.

yes, certainly. I will do it after PR #202

[nidhaloff]

Why would you need this declaration here?

[NorthOC]

For context, I ran the deep-translator for days.

In my case, I ran into API errors due to LibreTranslator mirror server resets (to prevent bots from running 24/7).

I had already translated many batches, so instead of doing API calls again, when the server reset, I added the translated batches into already_translated and saved a shit ton of time (from 5 days of runtime for a full translation to less than 24 hours of runtime).

We're talking like hundreds of API calls per hour.

[nidhaloff]

Ok but is it really needed to pass already_translated in the constructor? wouldn't it be sufficient to hide this from the user and just define it inside the class?

[nidhaloff]

@NorthOC unit tests are failing. I also asked a question about passing already_translated as a parameter and if this is needed

[NorthOC]

@nidhaloff Moved already_translated into the function for batch translations. I ran pytest, seems to pass all checks.