added decoded=False to hyperlink parser

[37b]

Added decoded=False to hyperlink.parse to mitigate errors when parsing URLs with non-ASCII chars.

See python-hyper/hyperlink#173

[damiencarol]

@37b the change broke a lot of thing endpoint related. it seems that it needs more modifications

[damiencarol]

FYI @kiblik

[37b]

I only pushed it into a PR so the tests could run and noticed that. I was going to take a look at fixes when I have time. Chris Fort Sent from my iPhone.

…

On Aug 27, 2021, at 4:12 AM, Damien Carol ***@***.***> wrote:  @37b the change broke a lot of thing endpoint related. it seems that it needs more modifications — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub, or unsubscribe.

[Maffooch]

Hi @37b how are things going on this?

[github-actions]

This pull request has conflicts, please resolve those before we can evaluate the pull request.

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[37b]

@kiblik have you looked at this? I am not as familiar with the downstream problems caused

[kiblik]

Well, I see no problem with this implementation. However, @37b, are you able to prepare unittests for this scenario?

[github-actions]

Conflicts have been resolved. A maintainer will review the pull request shortly.

[37b]

@kiblik @Maffooch I finally had time to look at this and found the downstream problem and put in a fix. I pushed the changes but still working on additional tests.

[37b]

This issue is likely not able to be resolved with current parsing methods.

From my limited understanding of URL encoding the character %F6 is a Windows-1252 encoding value. The character ö is %F6 with Windows-1252 encoding and %C3%B6 as UTF-8.

When requesting the URL https://host.com/BMF%20Veröffentlichungen\?WamReturn\=true\&host_header\=host the application/server responds with the proper URL encoded value https://host.com/BMF%20Ver%C3%83%C2%B6ffentlichungen?host_header=host">here</a>.</h2> in a 302 response.

I presume then that the test tool (in this case WhiteHat Sentinel) is converting the URL to Windows-1252 encoding while being processed for reporting.

Since this is somewhat of an edge case and a fix would need fundamental changes to the Endpoint validations/encapsulation I'll just close this out.