For AI agents: the complete documentation index is available at /tc39-atlas/en/llms.txt, the full documentation bundle is available at /tc39-atlas/en/llms-full.txt, and this page is available as Markdown at /tc39-atlas/en/proposals/year/pending/fix-9-2-3-lookupmatcher-algorithm.md.
TC39 Atlas
  • English

      • Fix 9.2.3 LookupMatcher algorithm S0

      Proposal details
      Proposal overview

      This proposal aims to fix the LookupMatcher algorithm in ECMA-402, which currently fails to correctly resolve locales like az-IR or zh-AU to their proper language/script variants. The issue is that the current algorithm truncates subtags too aggressively, causing incorrect fallbacks.

      Note

      The README below comes from the upstream repository and may contain outdated stage or status metadata. Use the proposal details above as the current source of truth.

      Update

      Stage 0

      Ecma-402 has a specification for Lookup Matching, described by LookupMatcher (9.2.3) and BestAvailableLocale (9.2.2). It's based on the Lookup algorithm described in RFC 4647 section 3.4. But, both Ecma-402 specification and RFC 4647 algorithm fail to perform such task in some cases. For example:

      • az-IR maps to az (default for az-Latn-AZ), not to the correct az-Arab-IR.
      • en-Latn-GB and en-Latn-IN map to en (default for en-Latn-US), not to the correct en-GB and en-IN respectively.
      • ha-CM and ha-SD map to ha (default for ha-Latn-NG), not the correct ha-Arab-*.
      • kk-CN, kk-IR, and kk-MN map to kk (default for kk-Cyrl-KZ), not the correct kk-Arab-*.
      • sr-ME, sr-RO, sr-RU, sr-TR map to sr (default for sr-Cyrl-RS), not to the correct sr-Latn-*.
      • uz-AF maps to uz (default for uz-Latn-UZ), not to the correct uz-Arab-AF.
      • zh-AU, zh-BN, zh-GB, zh-GF, zh-ID, zh-MO, zh-MY, zh-PA, zh-PF, zh-PH, zh-SR, zh-TH, zh-US, and zh-VN map to zh (default for zh-Hans-CN), not to the correct zh-Hant-*.

      Impact

      The wrong specification can be observed in today's implementations.

      // Both bugs bellow happen on latest Firefox and Chrome.

new Intl.NumberFormat("az").format(NaN); // "NaN" (as expected)
new Intl.NumberFormat("az-IR").format(NaN); // "NaN" (wrong, Arab expected)

new Intl.NumberFormat("en-IN").format(123456789); // "12,34,56,789" (as expected)
new Intl.NumberFormat("en-Latn-IN").format(123456789); // "123,456,789" (bypasses en-IN)

new Intl.DateTimeFormat("en-GB").format(new Date()); // "13/01/2015" (as expected)
new Intl.DateTimeFormat("en-Latn-GB").format(new Date()); // "1/13/2015" (bypasses en-GB)

new Intl.NumberFormat("ha-CM").format(NaN); // "NaN" (wrong, Arab expected)
new Intl.NumberFormat("ha-SD").format(NaN); // "NaN" (wrong, Arab expected)

new Intl.NumberFormat("kk-AF").format(NaN); // "NaN" (wrong, Arab expected)
new Intl.NumberFormat("kk-CN").format(NaN); // "NaN" (wrong, Arab expected)
new Intl.NumberFormat("kk-IR").format(NaN); // "NaN" (wrong, Arab expected)

new Intl.DateTimeFormat("sr", {month: "long"}).format(new Date()); // "фебруар" (as expected)
new Intl.DateTimeFormat("sr-Latn", {month: "long"}).format(new Date()); // "februar" (as expected)
new Intl.DateTimeFormat("sr-ME", {month: "long"}).format(new Date()); // "februar" (as expected)
new Intl.DateTimeFormat("sr-ME", {month: "long"}).format(new Date()); // "фебруар" (wrong, Latn expected)
new Intl.DateTimeFormat("sr-RO", {month: "long"}).format(new Date()); // "фебруар" (wrong, Latn expected)
new Intl.DateTimeFormat("sr-RU", {month: "long"}).format(new Date()); // "фебруар" (wrong, Latn expected)
new Intl.DateTimeFormat("sr-TR", {month: "long"}).format(new Date()); // "фебруар" (wrong, Latn expected)

new Intl.NumberFormat("uz").format(NaN); // "NaN" (as expected)
new Intl.NumberFormat("uz-AF").format(NaN); // "NaN" (wrong, Arab expected)

new Intl.NumberFormat("zh").format(NaN); // "NaN" (as expected)
new Intl.NumberFormat("zh-AU").format(NaN); // "NaN" (wrong, Hant expected)
new Intl.NumberFormat("zh-BN").format(NaN); // "NaN" (wrong, Hant expected)
new Intl.NumberFormat("zh-CN").format(NaN); // "NaN" (as expected)
new Intl.NumberFormat("zh-GB").format(NaN); // "NaN" (wrong, Hant expected)
new Intl.NumberFormat("zh-GF").format(NaN); // "NaN" (wrong, Hant expected)
new Intl.NumberFormat("zh-HK").format(NaN); // "非數值" (as expected)
new Intl.NumberFormat("zh-ID").format(NaN); // "NaN" (wrong, Hant expected)
new Intl.NumberFormat("zh-MO").format(NaN); // "NaN" (wrong, Hant expected)
new Intl.NumberFormat("zh-MO").format(NaN); // "NaN" (wrong, Hant expected)
new Intl.NumberFormat("zh-MY").format(NaN); // "NaN" (wrong, Hant expected)
new Intl.NumberFormat("zh-PA").format(NaN); // "NaN" (wrong, Hant expected)
new Intl.NumberFormat("zh-PF").format(NaN); // "NaN" (wrong, Hant expected)
new Intl.NumberFormat("zh-PH").format(NaN); // "NaN" (wrong, Hant expected)
new Intl.NumberFormat("zh-SR").format(NaN); // "NaN" (wrong, Hant expected)
new Intl.NumberFormat("zh-TH").format(NaN); // "NaN" (wrong, Hant expected)
new Intl.NumberFormat("zh-TW").format(NaN); // "非數值" (as expected)
new Intl.NumberFormat("zh-US").format(NaN); // "NaN" (wrong, Hant expected)
new Intl.NumberFormat("zh-VN").format(NaN); // "NaN" (wrong, Hant expected)

      Cause

      The algorithm specified is too simplistic. It basically instruct implementations to perform subtags truncation until it finds a locale, which is wrong. Related issues and more details can be found on:

      Fix

      According to Mark Davis, "the recommended methodology for Bundle lookup is to use Language Matching". Therefore, there are two options:

      1. Follow the Language Matching algorithm specified by the Unicode Technical Standard #35.
      2. Follow the quicker algorithm from cldrjs, which produces the same result as the Language Matching considering a score threshold of 100%.