Initialize the IgnoreInvalidPunycode flag when calling UTS 46

[hsivonen]

What is the issue with the URL Standard?

UTS 46 revision 31 added a IgnoreInvalidPunycode flag to its ToASCII and ToUnicode operations. The URL Standard should be explicit about the value of this flag when it calls into ToASCII or into ToUnicode.

[hsivonen]

AFAICT, the current behavior of Firefox and Safari would be consistent with setting this flag to false and Chrome’s behavior would be consistent with setting this flag to true.

Looking at how browsers comply with the existing spec, Safari seems to comply well, Firefox seems to comply except Firefox fails to enforce bidi rule on LTR labels in a bidi domain name (i.e. Firefox enforces the bidi rule on a per-label basis), and Chrome’s behavior seems hard to explain from the spec.

These observations would support setting IgnoreInvalidPunycode to false. However, I’m missing some context of why the IgnoreInvalidPunycode flag was introduced in UTS 46. The rationale says it enables an ASCII fast path, but UTS 46 still requires validating xn-- labels that decode successfully as Punycode, so the flag does not, AFAICT, enable an ASCII fast path in general (and the “industry practice” evidently doesn’t cover Firefox and Safari).

@markusicu, @macchiati, can you share more context for the motivation of IgnoreInvalidPunycode and how you’d expect the URL Standard to set the flag?

[macchiati]

I can't remember off the top of my head; would have to look back at the development notes.

…

---------- Forwarded message --------- From: Henri Sivonen ***@***.***> Date: Fri, Mar 1, 2024, 04:37 Subject: Re: [whatwg/url] Initialize the IgnoreInvalidPunycode flag when calling UTS 46 (Issue #821) To: whatwg/url ***@***.***> Cc: Mark Davis ***@***.***>, Mention ***@***.***> AFAICT, the current behavior of Firefox and Safari would be consistent with setting this flag to false and Chrome’s behavior would be consistent with setting this flag to true. Looking at how browsers comply with the existing spec, Safari seems to comply well, Firefox seems to comply except Firefox fails to enforce bidi rule on LTR labels in a bidi domain name (i.e. Firefox enforces the bidi rule on a per-label basis), and Chrome’s behavior seems hard to explain from the spec. These observations would support setting IgnoreInvalidPunycode to false. However, I’m missing some context of why the IgnoreInvalidPunycode flag was introduced in UTS 46. The rationale says it enables an ASCII fast path, but UTS 46 still requires validating xn-- labels that decode successfully as Punycode, so the flag does not, AFAICT, enable an ASCII fast path in general (and the “industry practice” evidently doesn’t cover Firefox and Safari). @markusicu <https://github.com/markusicu>, @macchiati <https://github.com/macchiati>, can you share more context for the motivation of IgnoreInvalidPunycode and how you’d expect the URL Standard to set the flag? — Reply to this email directly, view it on GitHub <#821 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ACJLEMCPDTNYLKLQTLNVWXLYWBY77AVCNFSM6AAAAABC3OVTROVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTSNZTGEYTMMJQHA> . You are receiving this because you were mentioned.Message ID: ***@***.***>

[annevk]

Yeah I don't understand this either. This was not part of our feedback to UTS46 last year (#744) and I would not want ASCII special casing of this sort.

[socram8888]

I've been trying to figure out why my domain was not working on FF but did on Chrome, and found about the IgnoreInvalidPunycode flag.

I'd encourage you to set it to true, as false will break domains that can be registered - see my xn--i29h.kz domain.

[annevk]

That domain also fails in Safari and in any conforming URL parser: https://jsdom.github.io/whatwg-url/#url=aHR0cHM6Ly94bi0taTI5aC5rei8=&base=YWJvdXQ6Ymxhbms=. There are certainly domains you can register or use as subdomain that won't end up working. It's not immediately clear to me that all of those necessarily should.

cc @markusicu

[socram8888]

@annevk That website you've just given me kinda proves why IgnoreInvalidPunycode should be true.

If an URL were to have a 15.1 character such as \U0002EBF0, my Firefox ESR 128.0 would be unable to process it - not even in the punycoded form! https://jsdom.github.io/whatwg-url/#url=aHR0cDovL3huLS04ZzBuLmNvbS8=&base=YWJvdXQ6Ymxhbms=

And even more, if you try to use 🪉, the harp emoji in 16.0, it will not work on neither: https://jsdom.github.io/whatwg-url/#url=aHR0cDovL3huLS1rMDloLmNvbS8=&base=YWJvdXQ6Ymxhbms=

Despite being actually valid according to IdnaMappingTable for 16.0.0:

1FA89         ; valid      ;      ; NV8    # 16.0 HARP

Why is that? Because the tr46 library @jsdom/whatwg-url uses implements UTS 46 with the IDNA table 15.1.0, while my Firefox ESR 128.0 supports only up to 15.0.0, with the latest being 16.0.0.

If IgnoreInvalidPunycode were true by default, as it is on Chrome, browsers would still prevent accessing via invalid or not-yet-supported Unicode characters that could introduce security problems due to homographic attacks and confusables, but would allow navigating just fine via the punycoded version.

In short, requiring software updates to use new DNS domains that are all valid to the basic RFC 1034 seems like a bad idea with no obvious benefits to me.