Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[Feature Request]: Add per-worker labelled/histogram metrics to BigQuery IO for the Dataflow Legacy Harness. #28903

Open
1 of 16 tasks
JayajP opened this issue Oct 9, 2023 · 1 comment
Open
1 of 16 tasks

[Feature Request]: Add per-worker labelled/histogram metrics to BigQuery IO for the Dataflow Legacy Harness. #28903

JayajP opened this issue Oct 9, 2023 · 1 comment
Assignees
@JayajP
Labels
dataflow new feature P2

Comments

@JayajP
Copy link
Contributor

JayajP commented Oct 9, 2023

What would you like to happen?

Get better observability on BigQuery IO Operations for user workers using the Java Legacy Harness.

  • Support a new metrics path from User Worker -> Windmill Worker by piggybacking metrics GetData heartbeats.
    • This path will be used to support per-user worker metrics in Dataflow
  • Instrument BigQueryIO's storage write API with metrics that track RPC response status, request latency, request size.
    • Latency/Request size metrics will be supported with histogram metrics.

Issue Priority

Priority: 2 (default / most feature requests should be filed as P2)

Issue Components

  • Component: Python SDK
  • Component: Java SDK
  • Component: Go SDK
  • Component: Typescript SDK
  • Component: IO connector
  • Component: Beam YAML
  • Component: Beam examples
  • Component: Beam playground
  • Component: Beam katas
  • Component: Website
  • Component: Spark Runner
  • Component: Flink Runner
  • Component: Samza Runner
  • Component: Twister2 Runner
  • Component: Hazelcast Jet Runner
  • Component: Google Cloud Dataflow Runner
@JayajP
Copy link
Contributor Author

JayajP commented Oct 9, 2023

.take-issue

JayajP added a commit to JayajP/beam that referenced this issue Oct 13, 2023
@JayajP
Implement java exponential histograms (apache#28903)
9664a02
Abacn pushed a commit that referenced this issue Oct 18, 2023
@JayajP
Add MetricsContainer::getPerWorker{Counter|Histogram} (#28903) (#28923)
130ad8b 
* Add MetricsContainer::getPerWorker{Counter|Histogram}

* Add comments to NoOp{Counter|Histogram}.
Abacn pushed a commit that referenced this issue Oct 18, 2023
@JayajP
Implement java exponential histograms (#28903) (#28995)
9aaf9c2 
* Implement java exponential histograms (#28903)

* Address comments

* Address comments
kkdoon pushed a commit to twitter-forks/beam that referenced this issue Oct 21, 2023
@JayajP
Implement java exponential histograms (apache#28903) (apache#28995)
10b8d10 
* Implement java exponential histograms (apache#28903)

* Address comments

* Address comments
johnjcasey pushed a commit that referenced this issue Nov 15, 2023
@JayajP
Implement a handful of perworkermetrics the bigquery sink (#28903). (#…
2012107 
…29098)

* Implement a handful of perworker bigquery sink metrics

* Spotless check

* Address comments and spotless fixes

* Spotless
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@JayajP JayajP

Labels
dataflow new feature P2
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
1 participant
@JayajP