Add troubleshooting section for the date filter

[robbavey]

Relates: #16530

[lucabelluccini]

Q: Can it be changed by using the ENV vars like https://www.baeldung.com/linux/locale-environment-variables ?

[lucabelluccini]

This option needs to be protected with a "Only for JDK < 23"

[jsvd]

Logstash only supports 11, 17 and 21.

[lucabelluccini]

Thank you @robbavey for this!

Can you confirm those QA below are valid? If they are, I would like to write some suggestions on top of what you've drafted.

1. As a user, do I have to check something before upgrading Logstash or JDK? If yes, in which situations I should pay attention?

The change can affect any Logstash user using JDK 17 or more recent.

According to the support matrix, it's possible to use JDK 17 in 7.16.x onwards.

So the "might be affected" users are those matching all the following conditions:

• Using date parsing (using the date filter - or even ruby filters with Ruby date formats patterns) in Logstash pipelines
• Locale (in the date filter, or in env vars of the Logstash process) is different from en_US
• Running JDK >= 17 (respecting the supported Logstash versions in the support matrix)

2. As a user, what can I do to not be affected by those changes?

As usual, all pipelines should be validated & tested by the user.

If you plan to use Logstash with JDK versions 17 or more recent, you have to apply one of the following:

• (Only for JDK < 23) Switch the locale provider to -Djava.locale.providers=COMPAT,SPI. This will guarantee the behavior of the JVM will be the same as before this change.

• Force the locale via the locale env variables to en_US (this will affect all Logstash plugin & core). Note: if a locale is specified at date plugin level, it will override the locale env variables

• Force the locale via the date filter setting to en_US

• Introduce a filter prior the date parsing filters to replace Sep occurrences into Sept

[robbavey]

Thank you @robbavey for this!

Can you confirm those QA below are valid? If they are, I would like to write some suggestions on top of what you've drafted.

1. As a user, do I have to check something before upgrading Logstash or JDK? If yes, in which situations I should pay attention?

The change can affect any Logstash user using JDK 17 or more recent.

According to the support matrix, it's possible to use JDK 17 in 7.16.x onwards.

So the "might be affected" users are those matching all the following conditions:

• Using date parsing (using the date filter - or even ruby filters with Ruby date formats patterns) in Logstash pipelines
• Locale (in the date filter, or in env vars of the Logstash process) is different from en_US

This makes sense - do you think it makes sense to mention that we know about the changes in en_gb and en_au, but others might be affected?

• Running JDK >= 17 (respecting the supported Logstash versions in the support matrix)

It's actually JDK21 that has the issue - moving from JDK17 to JDK20 and above included a change to CLDR 42, which included the incompatible updates

2. As a user, what can I do to not be affected by those changes?

As usual, all pipelines should be validated & tested by the user.

If you plan to use Logstash with JDK versions 17 or more recent, you have to apply one of the following:

This should be "versions 20 or more recent"

• (Only for JDK < 23) Switch the locale provider to -Djava.locale.providers=COMPAT,SPI. This will guarantee the behavior of the JVM will be the same as before this change.
• Force the locale via the locale env variables to en_US (this will affect all Logstash plugin & core). Note: if a locale is specified at date plugin level, it will override the locale env variables

Or a locale that matches the data being sent into logstash

• Force the locale via the date filter setting to en_US
• Introduce a filter prior the date parsing filters to replace Sep occurrences into Sept

[jsvd]

Logstash only supports 11, 17 and 21.

[lucabelluccini]

This makes sense - do you think it makes sense to mention that we know about the changes in en_gb and en_au, but others might be affected?

Yes, we can say "we are aware that at least those are affected - other differences might also be present in other locales".

It's actually JDK21 that has the issue - moving from JDK17 to JDK20 and above included a change to CLDR 42, which included the incompatible updates

Ok - sorry for the misunderstanding - the problem is JDK >= 20. Anyway the support matrix only allows JDK 21 so we could mention JDK 20, but we also mention support matrix doesn't allow it.

think the rest of the text I've written should be amended with 17 -> 20/21.

Or a locale that matches the data being sent into Logstash

Yep, I can add it.

I've written a KB to dump the date formats, with the ability to switch of JDK version and pass the locale settings and default JDK locale provider preference at https://support.elastic.dev/knowledge/view/5028e60c. I wonder if it might be useful.