Schema inference parameterized types

[reuvenlax]

Beam automatically infers schema for common Java types: POJOs, JavaBean, AutoValue, AVRO, among others. In addition for it's use with schema transforms, this also provides a simple, efficient way of using these types in PCollections without needing to construct a Coder. A frequent complaint has been that this inference did not work in the presence of generic type parameters. E.g. this means that while Beam can handle PCollections of this class:

@DefaultSchema(JavaFieldSchema.class)
class MyType {
String field1;
}
PCollection myTypes = read();

Schema inference would fail for this:

@DefaultSchema(JavaFieldSchema.class)
class MyType<T> {
T field1;
}
PCollection<MyType> myTypes = read();

This PR adds support for generic type parameters to schema inference for common types, addressing the above issue.

[github-actions]

Checks are failing. Will not request review until checks are succeeding. If you'd like to override that behavior, comment assign set of reviewers

[ahmedabu98]

Functionality looks sound and tests look great

Left some nits

There's few changes to public, non-internal, method signatures that are breaking though. Might make sense to overload these methods just in case (would also narrow down the scope of this PR)

[reuvenlax]

@ahmedabu98 Any more comments? Are you ok with marking these classes as Internal, or do you prefer adding overloads?

[ahmedabu98]

+1 to marking classes as @Internal

As for overloads (do you mean the ReflectUtils.get__Type() methods?), it'd be an improvement but doesn't need to block this PR

LGTM 👍🏽

[ahmedabu98]

I believe this change introduced a regression (see #32795)

Iceberg unit and integration tests are failing with the following error:

java.lang.IllegalArgumentException: unable to serialize SchemaCoder<Schema: Fields:
Field{...
Field{...

Tests are failing at HEAD, but passing when reverting this PR: #32802

[kennknowles]

Suspected culprit of breaking this one too https://github.com/apache/beam/actions/workflows/beam_PreCommit_Yaml_Xlang_Direct.yml?query=branch%3Amaster+is%3Acompleted

[reuvenlax]

The fix has been submitted - is that Yaml test still failing? Separately we should figure out why none of those tests triggered on my initial PR :(

…

On Wed, Oct 16, 2024 at 6:00 PM Kenn Knowles ***@***.***> wrote: Suspected culprit of breaking this one too https://github.com/apache/beam/actions/workflows/beam_PreCommit_Yaml_Xlang_Direct.yml?query=branch%3Amaster+is%3Acompleted — Reply to this email directly, view it on GitHub <#32757 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AFAYJVJWYRWOHOV74QY6PS3Z34D4HAVCNFSM6AAAAABPZOTUP2VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDIMJYGI3TCNJYGE> . You are receiving this because you modified the open/close state.Message ID: ***@***.***>

[stankiewicz]

I'm curious if prior work from #32081 had same issues. Can @tilgalas take a look as he explored this feature already and his PR was waiting in review queue for extended amount of time.

[ahmedabu98]

Created an issue for the failing YAML tests: #32832

[kennknowles]

The fix has been submitted - is that Yaml test still failing? Separately we should figure out why none of those tests triggered on my initial PR :(
…
On Wed, Oct 16, 2024 at 6:00 PM Kenn Knowles @.> wrote: Suspected culprit of breaking this one too https://github.com/apache/beam/actions/workflows/beam_PreCommit_Yaml_Xlang_Direct.yml?query=branch%3Amaster+is%3Acompleted — Reply to this email directly, view it on GitHub <#32757 (comment)>, or unsubscribe https://github.com/notifications/unsubscribe-auth/AFAYJVJWYRWOHOV74QY6PS3Z34D4HAVCNFSM6AAAAABPZOTUP2VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDIMJYGI3TCNJYGE . You are receiving this because you modified the open/close state.Message ID: @.>

Yea we could see if there is a natural precommit trigger that would make sense. It may be that these are far enough apart in the codebase that postcommit is where we expect to catch it anyhow. Then we would roll back and roll forward touching a trigger file in .github/trigger_files