Rewrite the i18n scripts

[obulat]

Fixes

Fixes #4979 by @sarayourfriend
Fixes #629 by @obulat
Fixes #5059 by @obulat
Related to #592 by @sarayourfriend

Description

This PR re-writes the i18n locale metadata and translation files generation.

Folder structure

The recent update in the Nuxt i18n has introduced a new top-level i18n folder. Now all i18n-related files are located there:

• i18n/data folder contains the source-of-truth original en.json5 file. The locale metadata (valid-locales.json and untranslated-locales.json) are saved there as well.
• i18n/locales folder is where all downloaded translations are saved.
• i18n/scripts folder contains the scripts that download locale metadata, translations, convert en.json5 to .pot file that is used for translations on https://translate.wordpress.org

Code modernization

Now that it's easy to run modules in Node, the scripts can be converted to .mjs and use import instead of require. This PR contains the conversion, as well as migration from axios to native fetch.

Flattening of the keys

To simplify finding static and dynamic keys in the source code, as well as the conversion from and to .pot files, the keys of the translation strings have been flattened. So now, en.json5 is a flat JSON file, where all keys are top-level. This required the update to the ESLint rule for the json5 keys.

Description of changed files (what to review)

The main entry point for the i18n scripts is now i18n/scripts/setup.mjs. This file reads the CLI options, and creates the necessary i18n files. All modes (English-only, production - downloading all locale metadata and translations, testing - copying of the local files from the test folder) are now combined in a single workflow.

The order of the steps for production was changed because the translated property for the locale metadata is now generated based on the number of keys in the locale's json file, so we need to have the translations before creating the list of available locales.

Previously, the steps were:

1. get the list of locales and their properties from the GlotPress GitHub repository
2. parse the translation statistics from the Openverse project page on https://translate.wordpress.org and add the translated property to the locales from step 1.
3. download all translations in a zip and convert the contained flat json files into nested json.
4. save the list of locales for which the translations were downloaded to valid-locales.json file, and the locales without translations to invalid-locales.json.

In this PR, step 2 was eliminated, and now the process is:

1. (previously, step 3) download all translations and save the non-empty json files. We no longer need to convert flat json files, and we don't need to save empty files as we are doing now.
2. (previously, step 1) get the list of locales and their properties from the GlotPress GitHub repository, and compute the translated percentage (comparing the number of keys in the original English en.json5 file and the translated json file.
3. (previously, step 4) save the list of locales for which the translations were downloaded to valid-locales.json file, and the locales without translations to invalid-locales.json.

The code for the new step 1 is located in i18n/scripts/translations.mjs, for the new step 2 and 3 - in i18n/scripts/metadata.mjs.

The code for generating the pot file was also simplified now that we don't need to handle nested json, and is now located in i18n/scripts/generate-pot.mjs.

All of the paths for files are now located in a single file, i18n/scripts/paths.mjs, so it will be easier to safely move a file and update its paths in a single place.

The i18n check script was removed from frontend/package.json because it wasn't working, and the library doesn't seem to be maintained.

Testing Instructions

Run the i18n scripts, deleting the locale data between the runs to ensure that existing data does not interfere.

• "i18n" - the main script, by default generates all the data needed for production. Can also be used with flags.
• "i18n:debug" - the same as i18n, but has more verbose logging, including the logging of keys/placeholders that have been modified to fix one of the linked GitHub issues
• "i18n:en" - only generates the en.json file and an empty valid-locales.json. This can be used for quickly running local environment, including with the --watch flag (that watches the en.json5 file and updates en.json when the original file changes)
• "i18n:test" - copies the files from test/locales into correct folders
• "i18n:no-get" - used to generate valid-locales.json file based on existing translation files, without downloading translations from GlotPress
• "i18n:generate-pot" - converts en.json5 file to pot file. The generated file should be the same as the one generated on main.

Checklist

• My pull request has a descriptive title (not a vague title likeUpdate index.md).
• My pull request targets the default branch of the repository (main) or a parent feature branch.
• My commit messages follow best practices.
• My code follows the established code style of the repository.
• I added or updated tests for the changes I made (if applicable).
• I added or updated documentation (if applicable).
• I tried running the project locally and verified that there are no visible errors.
• I ran the DAG documentation generator (ov just catalog/generate-docs for catalog
  PRs) or the media properties generator (ov just catalog/generate-docs media-props
  for the catalog or ov just api/generate-docs for the API) where applicable.

Developer Certificate of Origin

Developer Certificate of Origin

Developer Certificate of Origin
Version 1.1

Copyright (C) 2004, 2006 The Linux Foundation and its contributors.
1 Letterman Drive
Suite D4700
San Francisco, CA, 94129

Everyone is permitted to copy and distribute verbatim copies of this
license document, but changing it is not allowed.


Developer's Certificate of Origin 1.1

By making a contribution to this project, I certify that:

(a) The contribution was created in whole or in part by me and I
    have the right to submit it under the open source license
    indicated in the file; or

(b) The contribution is based upon previous work that, to the best
    of my knowledge, is covered under an appropriate open source
    license and I have the right under that license to submit that
    work with modifications, whether created in whole or in part
    by me, under the same open source license (unless I am
    permitted to submit under a different license), as indicated
    in the file; or

(c) The contribution was provided directly to me by some other
    person who certified (a), (b) or (c) and I have not modified
    it.

(d) I understand and agree that this project and the contribution
    are public and that a record of the contribution (including all
    personal information I submit with it, including my sign-off) is
    maintained indefinitely and may be redistributed consistent with
    this project or the open source license(s) involved.

[github-actions]

Latest k6 run output¹

     ✓ status was 200

     checks.........................: 100.00% ✓ 6400      ✗ 0   
     data_received..................: 1.5 GB  8.0 MB/s
     data_sent......................: 838 kB  4.5 kB/s
     http_req_blocked...............: avg=13.31µs  min=1.93µs  med=3.93µs   max=5.13ms   p(90)=5.58µs   p(95)=6.07µs  
     http_req_connecting............: avg=8.56µs   min=0s      med=0s       max=5.08ms   p(90)=0s       p(95)=0s      
     http_req_duration..............: avg=649.83ms min=69.87ms med=557.2ms  max=2.45s    p(90)=1.17s    p(95)=1.28s   
       { expected_response:true }...: avg=649.83ms min=69.87ms med=557.2ms  max=2.45s    p(90)=1.17s    p(95)=1.28s   
   ✓ http_req_failed................: 0.00%   ✓ 0         ✗ 6400
     http_req_receiving.............: avg=148.25µs min=45.6µs  med=114.95µs max=23.81ms  p(90)=183.62µs p(95)=223.05µs
     http_req_sending...............: avg=20.25µs  min=6.41µs  med=19.21µs  max=537.85µs p(90)=25.17µs  p(95)=27.5µs  
     http_req_tls_handshaking.......: avg=0s       min=0s      med=0s       max=0s       p(90)=0s       p(95)=0s      
     http_req_waiting...............: avg=649.66ms min=69.74ms med=557.06ms max=2.45s    p(90)=1.17s    p(95)=1.28s   
     http_reqs......................: 6400    34.230756/s
     iteration_duration.............: avg=3.47s    min=1.23s   med=3.01s    max=9s       p(90)=6.79s    p(95)=7.07s   
     iterations.....................: 1200    6.418267/s
     vus............................: 3       min=3       max=30
     vus_max........................: 30      min=30      max=30

Footnotes

1. This comment will automatically update with new output each time k6 runs for this PR ↩

[github-actions]

Full-stack documentation: https://docs.openverse.org/_preview/5177

Please note that GitHub pages takes a little time to deploy newly pushed code, if the links above don't work or you see old versions, wait 5 minutes and try again.

You can check the GitHub pages deployment action list to see the current status of the deployments.

Changed files 🔄:

• https://docs.openverse.org/_preview/5177/frontend/reference/i18n.html