Indexing updates for ISO 639-3 three-letter codes

[escowles]

CaMS is now including ISO 639-3 three-letter language codes where appropriate (sample record: https://catalog.princeton.edu/catalog/9930372423506421/). DSSG discussed and approved two changes to how these are indexed:

1. Display the more specific language names from ISO 639-3, e.g., the record above should display "Language: Wu Chinese".
2. Index both the more specific name and the more general/family name in the Language facet, so both "Chinese" and "Wu Chinese".

Implementation notes

• Add the specific language name in the 'language_facet' .
• Don't add it in the 'language_iana_s'

Checklist

• add ISO 639-3 gem
• see if we can contribute upstream to traject to handle ISO 639-3 codes in the marc_languages macro Add ISO 639-3 language codes traject/traject#293
• create our own logic for the show page field (i2069: Add a new field for specific languages on the Show page #2092)
• Display the field in Orangelight Display the language_name_display field in the show page orangelight#3454

[mzelesky]

See https://catalog.princeton.edu/catalog/9930372423506421/staff_view

The 041s that have the ISO 639-3 have a $2 of iso639-3

So traject can handle the 041s with $2 iso639-3 separately from 041s without that $2.

[christinach]

Add the specific language name in the 'language_facet' .

[sandbergja]

We aren't always guaranteed to be able to find the more general language name (the "macrolanguage" using ISO 639-3 terminology) given a certain ISO 639-3 code.

The provided example, wuu, is included in the ISO 639-3 macrolanguage table with a macrolanguage of zho (Chinese). In the example MARC record, the 008 contains the ISO 639-2B code chi (Chinese), so we could also generate the requested facet "Chinese" from that.

However, another example: nuz (Tlamacazapa Nahuatl) isn't mapped to a macrolanguage of Nahuatl, since no such macrolanguage exists in ISO 639-3 (it does in ISO-639-2 and ISO-639-1). It seems that we would want both "Tlamacazapa Nahuatl" and "Nahuatl" in the facets -- will CaMS catalogers always put in both the ISO 639-3 and 639-2B code in these records?

[sandbergja]

Some possible gems:

• https://github.com/bbenno/languages -- it does not currently include macrolanguages and has not yet had a version 1.0.0 release, but does have CI and a nice script for updating the code data. Update: it now includes macrolanguages.
• https://github.com/mlj/iso-codes -- on version 1.2.0, tests but no CI, does include macrolanguages

We could also forego using an additional gem, and use the debian iso-codes package or the tables directly from SIL.

[sandbergja]

Also noting that, while we currently use the language_facet field for both the facet and the show page, we'll have to use two separate fields (and do the appropriate orangelight work) in order to meet these requirements.

Also, for the facet display, do we want them to be presented as a list, or as a "pivot facet" like the call number facet?

[maxkadel]

Trying to understand this issue better - these descriptions of the relationship between the iso-639-2/iso-639-5 and iso-639-3 standards were helpful to me - https://www.loc.gov/standards/iso639-2/iso639jac.html and https://iso639-3.sil.org/about/relationships

It looks like Nahuatl is a "collective", rather than a "macrolanguage" (see https://iso639-3.sil.org/code/nah vs. https://iso639-3.sil.org/code/zho).

From https://www.loc.gov/standards/iso639-2/faq.html#18:

Macrolanguages are distinguished from language collections in that the individual languages that correspond to a macrolanguage must be very closely related, and there must be some domain in which only a single language identity is recognized.

So, I think that it is intentional that the different languages with "Nahuatl" in their name are not grouped together in the macrolanguages table (presumably because they are not linguistically closely related, although they share a name - I am not a linguist).

I think that in order to do the work of the future ticket of linking together Indigenous languages, we will need to create our own concordance, as we did with the "augment the subject" work for "Indigenous Studies".

I like the idea of making this a pivot facet.

Also, the there is only one (legit) app that currently depends on either of the gems listed above, but the https://github.com/bbenno/languages gem supports the homosaurus vocabulary, which we also use, so might be a cool project to support.

[sandbergja]

• Opened Add ISO 639-3 language codes traject/traject#293
  • If this is merged and released, we'll just need to upgrade traject in order to meet the facet requirement of this ticket
  • If not, we'll need to make our own custom logic for the facet
• Opened this PR upstream: i49: Add collective codes from ISO 639-5 bbenno/languages#50 -- not a prerequisite for this PR to be merged, but it would allow us to simplify the LanguageService