[SPARK-49992][SQL] Default collation resolution for DDL and DML queries

[stefankandic]

What changes were proposed in this pull request?

This PR proposes not using session-level collation in DDL commands (create/alter view/table, add/replace columns).

Also, resolution of default collation should happen in the analyzer and not in the parser. However, due to how we are checking for default string type (using reference equals with StringType object) we cannot just replace this object with StringType("UTF8_BINARY") because they compare as equal so the tree node framework will just return the old plan. Because of this we have to perform this resolution twice, once by changing the StringType object into a TemporaryStringType and then back to StringType("UTF8_BINARY") which is not considered a default string type anymore.

Another thing is that the dependent rules ResolveInlineTables and CollationTypeCoercion are updated so that they don't execute if there are still unresolved string types in the plan.

Why are the changes needed?

The default collation for DDL commands should be tied to the object being created or altered (e.g., table, view, schema) rather than the session-level setting. Since object-level collations are not yet supported, we will assume the UTF8_BINARY collation by default for now.

Does this PR introduce any user-facing change?

No.

How was this patch tested?

Added new unit tests.

Was this patch authored or co-authored using generative AI tooling?

No.

[cloud-fan]

This is not only for DML but also for normal queries, right?

[stefankandic]

not sure what you mean by normal queries; DML includes SELECT, INSERT, UPDATE and DELETE. So essentialy anything this is not create/alter

[cloud-fan]

How can we replace StringType without collation in tableDesc for v1 command?

[stefankandic]

We will replace it in the parsed plan so the CreateTable will have the correct schema of the table

[cloud-fan]

this shouldn't extend StringType, otherwise StringType(id) == TemporaryStringType(id) and we may hit the TreeNode transformation issue again.

[stefankandic]

getTemporaryStringType fn makes sure we get a different collation id. I tried not extending StringType but it causes many issues, since for example we have to create a literal with new type and that would of course fail since it doesn't recognize it

[cloud-fan]

I think we can make the code a bit more unified. How about this

var hasTempStringType = boolean
plan.resolveOperators {
  case AddColumn =>
    var defaultStringType = getDefaultStringType(table) // Always return utf8 for now.
    if (defaultStringType == StringType) {
      hasTempStringType = true
      defaultStringType = TemporaryStringType...
    }
    do the replacement work
  case other DDL commands =>  ...
  case ctas | rtas | create_view | alter_view => ...
  case normal query => ...
}
if (hasTempStringType) {
  the same pattern match but replace TempStringType only.
}

This code is closer to the eventual code, as we need to determine the default collation with the table/schema/catalog metadata, which is not a global thing, must be determined when a certain command is matched.

[stefankandic]

I don't really like that approach, even if we didn't have to do the traversal twice I would rather decouple the logic for DDL and DML and not have it all mixed together in one match statement