Fix nested union issue in avro schema conversion

[xinyuiscool]

The conversion of the union type between Beam schema and avro schema, e.g.:

{
  "type": "record",
  "name": "topLevelRecord",
  "fields": [
    {
      "name": "Entity",
      "type": [
        {
          "type": "record",
          "name": "MemberClient",
          "namespace": "topLevelRecord",
          "fields": [
            {
              "name": "member",
              "type": "string",
            }]
        },
        {
          "type": "record",
          "name": "OrganizationClient",
          "namespace": "topLevelRecord",
          "fields": [
            {
              "name": "organization",
              "type": "string",
            },
            {
              "name": "displayName",
              "type": [
                "null",
                "string"]
            }]
        }
      ]}]}

If we first convert it to Beam schema, and then back to avro, it will cause the following exception:

Caused by: org.apache.avro.AvroRuntimeException: Nested union

This is due to the following code path:

• For union avro type, when each field in the union is actually a real type (compared to [null, string] union), Beam will create a OneOfType type, and the sub types in the OneOfType will be the list of union inner types. Note during this conversion, Beam also marks each inner type to be nullable, since this is a union.
• In later part of the pipeline, our code tries to create the avro schema from beam schema. When dealing with OneOfType, since each of the sub type is nullable, Beam will create a inner avro union type of [null, subtype]. Then the outer type will be a union again. Here is why Avro throws nested union exception.

The fix is relatively simple: we remove the nullable flag from the sub types, since it's generated by Beam in the beginning, not from the original avro schema. After that, the avro union schema is created correctly. It's the same to the original schema.

---

Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily:

• Mention the appropriate issue in your description (for example: addresses #123), if applicable. This will automatically add a link to the pull request in the issue. If you would like the issue to automatically close on merging the pull request, comment fixes #<ISSUE NUMBER> instead.
• Update CHANGES.md with noteworthy changes.
• If this contribution is large, please file an Apache Individual Contributor License Agreement.

See the Contributor Guide for more tips on how to make review process smoother.

To check the build health, please visit https://github.com/apache/beam/blob/master/.test-infra/BUILD_STATUS.md

GitHub Actions Tests Status (on master branch)

See CI.md for more information about GitHub Actions CI.

[yananhao12]

Thanks for the quick fix!