Migrate lineage counters to bounded tries.

[robertwb]

Also cleans up the filesystem-specific logic.

---

Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily:

• Mention the appropriate issue in your description (for example: addresses #123), if applicable. This will automatically add a link to the pull request in the issue. If you would like the issue to automatically close on merging the pull request, comment fixes #<ISSUE NUMBER> instead.
• Update CHANGES.md with noteworthy changes.
• If this contribution is large, please file an Apache Individual Contributor License Agreement.

See the Contributor Guide for more tips on how to make review process smoother.

To check the build health, please visit https://github.com/apache/beam/blob/master/.test-infra/BUILD_STATUS.md

GitHub Actions Tests Status (on master branch)

See CI.md for more information about GitHub Actions CI or the workflows README to see a list of phrases to trigger workflows.

[robertwb]

R: @rohitsinha54

Depends on #33378.

[github-actions]

Stopping reviewer notifications for this pull request: review requested by someone other than the bot, ceding control. If you'd like to restart, comment assign set of reviewers

[robertwb]

Rebased on master now that upstream is merged.

[rohitsinha54]

Thank you.
Has this change tested on Dataflow runner to see how the service behaves once 2.62.0 start sending BoundedTrie as a metric which it does not know about? If not then, I can help with that if needed. Since on submission of this PR 2.62.0 jobs will start sending it.

One thing I am thinking is if we need stringSetDisabled kind of flag for BoundedTrie in python or not. We do not have stringSetDisabled for python as of now. I think it will be ok to go without it because

1. The race issue we saw for large GCS IO filepaths was java specific, given Python does a traditional blocking lock this should not happen here.
2. Python SDK usage is lower compared to Java but we have also not verified the very large GCSIO behavior at scale. This I can do this month.

[robertwb]

The UW gracefully ignores metric types it does not recognize. (This is why the string set changes could go in the SDK before they went into the service as well.) In addition, IIRC the PreCommit Python Integration (and others) run a couple of tests against dataflow. (I'll ensure all the tests pass before merging.)

I also don't think we need a stringSetDisabled equivalent.

[rohitsinha54]

SG. Oh yes UW part I remember now. IIRC we had to disable reporting StringSet for legacy worker though as tests started failing. I believe router will reject these with fail-close

https://source.corp.google.com/piper///depot/google3/cloud/dataflow/router/translation/v1beta3.cc;l=1361

[robertwb]

Yes. Python doesn't have to worry about the legacy worker anymore.

[codecov]

Codecov Report

Attention: Patch coverage is 94.11765% with 3 lines in your changes missing coverage. Please review.

Project coverage is 57.44%. Comparing base (9232cd8) to head (4a91890).
Report is 7 commits behind head on master.

Files with missing lines	Patch %	Lines
sdks/python/apache_beam/io/aws/s3filesystem.py	80.00%	1 Missing ⚠️
...thon/apache_beam/io/azure/blobstoragefilesystem.py	80.00%	1 Missing ⚠️
sdks/python/apache_beam/io/gcp/gcsfilesystem.py	80.00%	1 Missing ⚠️

Additional details and impacted files

@@             Coverage Diff              @@
##             master   #33381      +/-   ##
============================================
+ Coverage     57.37%   57.44%   +0.07%     
  Complexity     1475     1475              
============================================
  Files           973      979       +6     
  Lines        154905   155237     +332     
  Branches       1076     1076              
============================================
+ Hits          88879    89181     +302     
- Misses        63810    63840      +30     
  Partials       2216     2216              

Flag	Coverage Δ
python	81.30% <94.11%> (+0.04%)	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.