Support remote write v2 by converting request #6330

SungJin1212 · 2024-11-11T06:13:58Z

This PR supports Prometheus remote write 2.0 by converting the v2 request to v1 at the API.

Which issue(s) this PR fixes:
Fixes #6324

Checklist

Tests updated
Documentation added
CHANGELOG.md updated - the order of entries should be [CHANGE], [FEATURE], [ENHANCEMENT], [BUGFIX]

yeya24 · 2024-11-14T21:16:53Z

Looks promising. Thanks!

FYI we have prometheus/client_golang#1658 which exports remote write handler. Not a blocker for this PR but we should keep it on our radar to switch to use the library

SungJin1212 · 2024-11-15T00:53:02Z

@yeya24
Thanks for letting me know.
Should we make the issue to track it?

alanprot · 2024-11-15T19:16:47Z

Maybe we can open a issue for someone give a try to use the client_golang handler even before it get merged so we can give feedback on the open PR. Changing that handler after is merged probably will be more difficult as it could potentially break all projects that are already using it.

SungJin1212 · 2024-11-15T22:28:59Z

@alanprot
I added a comment here: #6324

yeya24 · 2024-11-17T20:19:11Z

I took a breif look at prometheus/client_golang#1658. Left some comments there and we have some changes Cortex specific that might not make sense for Prometheus. I think we are ok to proceed with this PR first.

SungJin1212 · 2024-11-18T00:34:20Z

@yeya24
Thanks. I read it, and it would be good if we could reuse its functions!

yeya24 · 2024-11-19T05:36:35Z

pkg/util/push/push.go

+			}
+		case config.RemoteWriteProtoMsgV2:
+			var req writev2.Request
+			err := util.ParseProtoReader(ctx, r.Body, int(r.ContentLength), maxRecvMsgSize, &req, util.RawSnappy)


@alanprot @danielblando
I wonder if we want to introduce a feature flag to control the behavior for RW v2 request. We can either ignore the request or convert to v1 and in the future maybe just accept as is.

Let's add a feature flag for the purpose of rollout. If RW 2.0 conversion is enabled right away, then Ingesters need to be rolled out first because of the protocol change to return stats. If we want to rollout Ingester and Distributor the same time then things can go wrong without a feature flag.

yeya24 · 2024-11-24T18:14:13Z

pkg/ingester/ingester.go

-	return &cortexpb.WriteResponse{}, nil
+	writeResponse := &cortexpb.WriteResponse{
+		Samples:    int64(succeededSamplesCount),
+		Histograms: int64(succeededSamplesCount), //TODO(Sungjin1212): Should we count histogram?


How is this implemented in Prometheus. Why we don't count histogram here?

I left TODO since we are counting histograms just as we count the sample.
But, the Prometheus is counting native histogram https://github.com/prometheus/prometheus/blob/main/storage/remote/write_handler.go#L424.

How about starting to count histogram when we introduce PushV2?

I don't understand the concern here. There is nothing prevent us doing it. We should count native histograms

yes, I agree with counting native histograms by changing to Histograms: int64(nativeHistogramCount).
My concern is we are counting samples instead of native histograms
https://github.com/cortexproject/cortex/blob/master/pkg/ingester/ingester.go#L1269

I am fine to split but why it needs a separate PR? We can just add a new int64 variable to count succeeded histograms

Maybe some changes are needed like we are tracking ingestionRate by calculating succeededSamplesCount + ingestedMetadata.
We should change the calculation to sustain existing behavior to succeededSamplesCount + succeededHistogramCount + ingestedMetadata
Also, we can introduce new metrics like cortex_ingester_ingested_native_histograms_total and cortex_ingester_ingested_histograms_failures_total.
WDYT?

I am ok to add the new metrics. But they are not blocking this PR so can be done either now or after this change.

If we don't add new metrics just track succeeded histogram samples, it is a simple change.

Ok, I will make PR soon!

@yeya24
I make the PR addressing it! (#6370)

yeya24 · 2024-11-24T18:17:07Z

pkg/distributor/distributor.go

@@ -816,7 +824,7 @@ func (d *Distributor) doBatch(ctx context.Context, req *cortexpb.WriteRequest, s
 			}
 		}

-		return d.send(localCtx, ingester, timeseries, metadata, req.Source)
+		return d.send(localCtx, ingester, timeseries, metadata, req.Source, stats)


Do we need to sum the samples pulled from stats? Now I see we just overwrite stats for every request.

If so, isn't there a good chance the returned header value (X-Prometheus-Remote-Write-Samples-Written) would be multiple of samples in a write request?

Yeah. With replication factor it is expected to have more samples. I think this is fine.

yeya24 · 2024-11-24T18:18:50Z

pkg/util/push/push.go

+			}
+		case config.RemoteWriteProtoMsgV2:
+			var req writev2.Request
+			err := util.ParseProtoReader(ctx, r.Body, int(r.ContentLength), maxRecvMsgSize, &req, util.RawSnappy)


Let's add a feature flag for the purpose of rollout. If RW 2.0 conversion is enabled right away, then Ingesters need to be rolled out first because of the protocol change to return stats. If we want to rollout Ingester and Distributor the same time then things can go wrong without a feature flag.

SungJin1212 · 2024-11-27T02:02:30Z

@yeya24
I added -distributor.remote-write2-enabled flags to configure whether the Distributor can accept PRW2.0.
I added the TestIngesterRollingUpdate e2e test, where the Distributor can accept PRW2.0 and the Ingester uses the v1.18.1 image.
The result is PRW2.0 push is a success, but the response header (X-Prometheus-Remote-Write-xxx) values are all "0".
Is it expecting behavior?

yeya24 · 2024-12-03T02:08:39Z

The result is PRW2.0 push is a success, but the response header (X-Prometheus-Remote-Write-xxx) values are all "0".
Is it expecting behavior?

This doesn't sound like the right behavior. Is that what you got with this PR?

SungJin1212 · 2024-12-03T02:17:20Z

@yeya24
Yes, the test condition is that Ingester uses a v1.18.1 image and Distributor uses a PRW2.0-implemented one. The PRW2.0 push then gets that result.
If the Ingester and Distributor use the same images (PRW 2.0 implemented), we can get the expected response header.

yeya24 · 2024-12-03T03:48:40Z

Yes, the test condition is that Ingester uses a v1.18.1 image and Distributor uses a PRW2.0-implemented one. The PRW2.0 push then gets that result.

I see what you meant. Then it is expected to get that result if you use Ingester of old version and Distributor of new version.
That's why we introduce the PRW 2.0 feature flag in distributor to only enable PRW 2.0 request if backend Ingester is running the newer version.

SungJin1212 · 2024-12-03T06:50:49Z

@yeya24
Should I add comments to -distributor.remote-write2-enabled so that the user can do a rolling update of the Ingesters first and then update the Distributor afterward?

yeya24 · 2024-12-03T17:51:49Z

We can mention it in the flag description but I doubt users really look at it.
I prefer to create a dedicated doc/guide for users to migrate to Prometheus 3.0

SungJin1212 · 2024-12-04T00:55:48Z

@yeya24
Yes, the guide docs would be more good.

Signed-off-by: SungJin1212 <[email protected]>

pull-request-size bot added the size/XXL label Nov 11, 2024

dosubot bot added the component/distributor label Nov 11, 2024

SungJin1212 force-pushed the Add-remote-write-v2-api branch 6 times, most recently from 07bbbee to 83d0ba6 Compare November 11, 2024 11:25

SungJin1212 mentioned this pull request Nov 12, 2024

Prometheus Remote Write v2 Implementation #6324

Open

6 tasks

yeya24 reviewed Nov 19, 2024

View reviewed changes

yeya24 reviewed Nov 24, 2024

View reviewed changes

SungJin1212 force-pushed the Add-remote-write-v2-api branch from 83d0ba6 to 6ce5027 Compare November 27, 2024 01:57

SungJin1212 force-pushed the Add-remote-write-v2-api branch from 6ce5027 to 5344155 Compare November 28, 2024 09:28

Support remote write v2 by converting request

a9231e4

Signed-off-by: SungJin1212 <[email protected]>

SungJin1212 force-pushed the Add-remote-write-v2-api branch from 5344155 to a9231e4 Compare December 18, 2024 11:02

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Support remote write v2 by converting request #6330

Support remote write v2 by converting request #6330

SungJin1212 commented Nov 11, 2024 •

edited

Loading

yeya24 commented Nov 14, 2024

SungJin1212 commented Nov 15, 2024

alanprot commented Nov 15, 2024

SungJin1212 commented Nov 15, 2024

yeya24 commented Nov 17, 2024

SungJin1212 commented Nov 18, 2024

yeya24 Nov 19, 2024

yeya24 Nov 24, 2024

yeya24 Nov 24, 2024

SungJin1212 Nov 25, 2024

SungJin1212 Nov 25, 2024

yeya24 Nov 25, 2024

SungJin1212 Nov 25, 2024

yeya24 Nov 25, 2024

SungJin1212 Nov 25, 2024 •

edited

Loading

yeya24 Nov 25, 2024 •

edited

Loading

SungJin1212 Nov 25, 2024

SungJin1212 Nov 25, 2024

yeya24 Nov 24, 2024

SungJin1212 Nov 25, 2024

yeya24 Nov 25, 2024

yeya24 Nov 24, 2024

SungJin1212 commented Nov 27, 2024 •

edited

Loading

yeya24 commented Dec 3, 2024

SungJin1212 commented Dec 3, 2024 •

edited

Loading

yeya24 commented Dec 3, 2024

SungJin1212 commented Dec 3, 2024 •

edited

Loading

yeya24 commented Dec 3, 2024 •

edited

Loading

SungJin1212 commented Dec 4, 2024

Support remote write v2 by converting request #6330

Are you sure you want to change the base?

Support remote write v2 by converting request #6330

Conversation

SungJin1212 commented Nov 11, 2024 • edited Loading

yeya24 commented Nov 14, 2024

SungJin1212 commented Nov 15, 2024

alanprot commented Nov 15, 2024

SungJin1212 commented Nov 15, 2024

yeya24 commented Nov 17, 2024

SungJin1212 commented Nov 18, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

SungJin1212 Nov 25, 2024 • edited Loading

Choose a reason for hiding this comment

yeya24 Nov 25, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

SungJin1212 commented Nov 27, 2024 • edited Loading

yeya24 commented Dec 3, 2024

SungJin1212 commented Dec 3, 2024 • edited Loading

yeya24 commented Dec 3, 2024

SungJin1212 commented Dec 3, 2024 • edited Loading

yeya24 commented Dec 3, 2024 • edited Loading

SungJin1212 commented Dec 4, 2024

SungJin1212 commented Nov 11, 2024 •

edited

Loading

SungJin1212 Nov 25, 2024 •

edited

Loading

yeya24 Nov 25, 2024 •

edited

Loading

SungJin1212 commented Nov 27, 2024 •

edited

Loading

SungJin1212 commented Dec 3, 2024 •

edited

Loading

SungJin1212 commented Dec 3, 2024 •

edited

Loading

yeya24 commented Dec 3, 2024 •

edited

Loading