Description

Fixes #515 .

In this PR, we are adding a way of building a columnStats Schema using the current column Transformers and the actual Schema of the Data. We want to:

• Ensure the fields are properly parsed from the JSON string.
• If a JSON string is specified, but the row is returned null, we assume the string is not following the correct syntax.

For that, I've added a QbeastColumnStats case class that contains the columnStatsSchema and the columnStatsRow. Also, a QbeastColumnStatsBuilder is needed to retrieve all the information given the parameters mentioned above.

case class QbeastColumnStats(columnStatsSchema: StructType, columnStatsRow: Row)


object QbeastColumnStatsBuilder {
  /**
   * Builds the QbeastColumnStats
   *
   * @param statsString
   *   the stats in a JSON string
   * @param columnTransformers
   *   the set of columnTransformers to build the Stats from
   * @param dataSchema
   *   the data schema to build the Stats from
   * @return
   */
  def build(
      statsString: String,
      columnTransformers: Seq[Transformer],
      dataSchema: StructType): QbeastColumnStats
}

Type of change

Bug fix.

Checklist:

Here is the list of things you should do before submitting this pull request:

• New feature / bug fix has been committed following the Contribution guide.
• Add logging to the code following the Contribution guide.
• Add comments to the code (make it easier for the community!).
• Change the documentation.
• Add tests.
• Your branch is updated to the main branch (dependent changes have been merged).

How Has This Been Tested? (Optional)

Testing different parsings on QbeastColumnStatsTestBuilder.