Releases for supported Spark major/minor versions

[slyons]

Hello!

I see that version 0.18.0 was released just over a week ago, but it seems that it was only released for Spark 3.2. Is there any way we can get it also released for 3.0 and 3.1 as well?

Thanks!

[nightscape]

Since 0.18.3 spark-excel should properly cross-publish for multiple Spark versions.

[rbharathkumar]

On a similar note, I see that the spark-excel version 0.18.x are available for spark 3.2.2 in the 3.2 series.
Is it ok to use the jar packaged for spark 3.2.2 with spark 3.2.1 or is it safe to stick to the 0.17 versions ?
Databricks 10.4 LTS version includes Apache Spark 3.2.1, Scala 2.12

Thank you for all the hard work..

[nightscape]

It's better to use the same major.minor version than to use a lower one.
There are binary incompatibilities between minor versions in Spark.

[rbharathkumar]

Thank you very much for your response. As per your suggestion, I will stick with the older 0.17.1 version.
Would it be possible to release a version for 3.2.1 maybe once in a series say once for the 0.18 and so on?
This is considering that the LTS Databricks runtime is supported till 2024.
Of course, if it is too much work or not enough users of that particular version, then it is better to ignore and let users use the supported spark version.

**Databricks Runtime Support Timelines: **
https://learn.microsoft.com/en-us/azure/databricks/release-notes/runtime/releases

Thank you once again for your kind response.

[nightscape]

@rbharathkumar that was a misunderstanding. I'm not recommending to use 0.17.1, but 3.2.2_0.18.5.

[rbharathkumar]

@nightscape My apologies for the confusion.
Let me try to clear it up...
I am using the Databricks Runtime 10.4 which is tied to the Spark version 3.2.1
Since you had mentioned that we should expect binary incompatibility between Spark versions, I had wrongly assumed that you meant for me to stick with the last available version for 3.2.1 which is 0.17.1.

So considering that I am using a runtime with Spark 3.2.1, should I use 0.17.1 or 0.18.5 ?

Apologies once again for the long winded questions....

[nightscape]

For Spark 3.2.1 I would recommend 3.2.2_0.18.5.
Spark versions can have incompatibilities in the major and minor versions, but rather seldomly in the patch version (the version format is major.minor.patch), so it shouldn't be an issue to use a library compiled with Spark 3.2.2 for Spark 3.2.1.

[rbharathkumar]

Thank you very much for your response.