Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[WJ-1191] Implement fallback locales #1669

Merged
merged 20 commits into from
Oct 27, 2023
Merged

[WJ-1191] Implement fallback locales #1669

merged 20 commits into from
Oct 27, 2023

Conversation

emmiegit
Copy link
Member

@emmiegit emmiegit commented Oct 27, 2023

This PR resolves WJ-1191, which calls for two different, but related features:

  • Being able to pass in a list of locales when translating. If a translation cannot be found for the first locale, the second is tried, then the third, etc., until a match is found.
  • Automatically trying generic forms of a locale. One issue we presently have is that, if you request a translation in the en_US locale, you won't actually get anything, even though we have lots of en translations. This is because all those translations are for en specifically, so because there are no exact matches which include the region US, the search fails. Instead, we try increasingly generic forms of the locale until we find a match, before moving onto the next fallback locale. The order we use is taken from below:

locale fallback order

Present locale, no fallback

Two fallback locales were specified, but neither was used.

{
  "locales": [
    "zh_Hans",
    "en_CA",
    "fr"
  ],
  "messages": {
    "about": {},
    "navigated-to": {
      "path": "/start"
    }
  }
}

yielded

{
  "about": "关于",
  "navigated-to": "导航至 ⁨/start⁩"
}

Missing locale, generic locale

The first locale was not present, so a fallback was used. However this too was not present, so increasingly general variants of this locale (en-US-valenciaen-USen) were checked until it found a match.

{
  "locales": [
    "hy",
    "en_US_valencia"
  ],
  "messages": {
    "about": {},
    "navigated-to": {
      "path": "/start"
    }
  }
}

yielded

{
  "about": "About",
  "navigated-to": "Navigated to ⁨/start⁩"
}

Mixed case

Each message is fetched separately, so which fallback locale is used depends on the particular message. This way partial translations of a language are supported. (For testing purposes the Vietnamese translation for login strings were removed.)

{
  "locales": [
    "vi",
    "zh_Hans",
    "en"
  ],
  "messages": {
    "login": {},
    "login.toast": {},
    "logout": {},
    "logout.toast": {}
  }
}

yielded

{
  "login": "登入",
  "login.toast": "您已登入。",
  "logout": "Đăng xuất",
  "logout.toast": "Bạn đã được đăng xuất."
}

No locales specified

A new error, NoLocaleSpecified (error code 4105) has been introduced for this case.

Splits up each step into a separate function:

1. translate()           - The overall call, takes a list of locales
                           and a message key, and returns the finished
                           translated string.

2. get_pattern_locales() - Goes through a list of locales and returns
                           first one which has a translation for this
                           path and attribute.

                           Important TODO item here: add support for
                           locale generalization.

3. get_pattern()         - Get a particular bundle and pattern from a
                           locale, path, and attribute.

4. get_message()         - Get a particular bundle and message from a
                           locale and a path.
@emmiegit emmiegit self-assigned this Oct 27, 2023
@codecov
Copy link

codecov bot commented Oct 27, 2023

Codecov Report

Merging #1669 (fb626c1) into develop (cf9a0bf) will decrease coverage by 0.06%.
Report is 15 commits behind head on develop.
The diff coverage is 23.33%.

Impacted file tree graph

@@             Coverage Diff             @@
##           develop    #1669      +/-   ##
===========================================
- Coverage    40.50%   40.45%   -0.06%     
===========================================
  Files          340      341       +1     
  Lines        10678    10744      +66     
===========================================
+ Hits          4325     4346      +21     
- Misses        6353     6398      +45     
Flag Coverage Δ *Carryforward flag
deepwell 2.10% <23.33%> (+0.37%) ⬆️
ftml 76.83% <ø> (ø) Carriedforward from 3d40343

*This pull request uses carry forward flags. Click here to find out more.

Files Coverage Δ
deepwell/src/services/error.rs 0.00% <0.00%> (ø)
deepwell/src/services/special_page/service.rs 0.00% <0.00%> (ø)
deepwell/src/locales/fallback.rs 84.00% <84.00%> (ø)
deepwell/src/endpoints/locale.rs 0.00% <0.00%> (ø)
deepwell/src/services/view/service.rs 0.00% <0.00%> (ø)
deepwell/src/locales/fluent.rs 0.00% <0.00%> (ø)

This reverts commit ba7ebef.

This massively inflates the diff and these unused imports are not
helpful for preludes and other mass-import context. We should find
a way to suppress the warning in this case.
@emmiegit emmiegit marked this pull request as ready for review October 27, 2023 05:48
@Zokhoi
Copy link
Contributor

Zokhoi commented Oct 27, 2023

How is the locale list ordered if some of the generic locales has already been specified in the list?
e.g.

{
  "locales": [
    "en-US",
    "zh-hans",
    "en"
  ],
  ...
}

Does it still try to insert en before zh-hans?

langids.push(langid);
}
langids
};
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Forgot this was used here, I should move the helper function below out somewhere public so it can get imported and used here too.

@emmiegit
Copy link
Member Author

emmiegit commented Oct 27, 2023

How is the locale list ordered if some of the generic locales has already been specified in the list? e.g.

{
  "locales": [
    "en-US",
    "zh-hans",
    "en"
  ],
  ...
}

Does it still try to insert en before zh-hans?

Yes, for each locale it tries all the generic forms before going to the next one. So it would check en before and after zh-hans.

@emmiegit
Copy link
Member Author

thanks @Zokhoi

@emmiegit emmiegit merged commit 08209e2 into develop Oct 27, 2023
8 checks passed
@emmiegit emmiegit deleted the WJ-1191-locale branch October 27, 2023 19:39
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants