Add last_observed_time for raft messages #1276
Conversation
Signed-off-by: hhwyt <[email protected]>
|
|
proto/raft_serverpb.proto
Outdated
| @@ -29,6 +29,8 @@ message RaftMessage { | |||
| bytes extra_ctx = 11; | |||
|
|
|||
| disk_usage.DiskUsage disk_usage = 12; | |||
| // Used to measure the send wait duration. | |||
| uint64 last_observed_time = 13; | |||
There was a problem hiding this comment.
As BatchRaftMessage has one, why bother recording it here?
There was a problem hiding this comment.
@Connor1996 There are two steps:
- RaftStore sends a RaftMessage to the RaftClient, placing the RaftMessage into the RaftClient buffer. (in RaftClient::send.)
- The RaftClient batches RaftMessages into a BatchMessageBuffer and flushes them as BatchRaftMessages to gRPC.
The last_observed_time in RaftMessage is used to monitor the first step, where BatchRaftMessage has not yet been created.
There was a problem hiding this comment.
Looks like we don't need to add a new field in protobuf, as step 1 occurs within the process with no inter-network communication.
|
@Connor1996: adding LGTM is restricted to approvers and reviewers in OWNERS files. In response to this: Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. |
|
@v01dstar: adding LGTM is restricted to approvers and reviewers in OWNERS files. In response to this:
Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. |
|
@LykxSassinator: adding LGTM is restricted to approvers and reviewers in OWNERS files. In response to this:
Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. |
|
@overvenus PTAL |
|
/assign @overvenus |
|
/cc @overvenus PTAL, thx~ |
|
/assign @hhwyt |
|
/cc @overvenus I've removed the |
|
/cc @cfzjywxk PTAL, thx~ |
There was a problem hiding this comment.
Better to update the latency break down whole picture
like https://docs.google.com/presentation/d/1Jsha70eEGUOWQOGR5VtDkplGMZYMBMmVh67At6o_4lk/edit#slide=id.geb29d9636c_2_0 does,
and update related information in https://docs.pingcap.com/tidb/stable/latency-breakdown
|
/retest |
|
@hhwyt: Cannot trigger testing until a trusted user reviews the PR and leaves an In response to this:
Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. |
|
/retest |
|
/ok-to-test |
@cfzjywxk 👍🏻. I'll update it later. |
|
@cfzjywxk PTAL again, the LGTM label was removed by a misoperation. Thanks~ |
|
@v01dstar: adding LGTM is restricted to approvers and reviewers in OWNERS files. In response to this:
Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. |
|
[APPROVALNOTIFIER] This PR is APPROVED This pull-request has been approved by: cfzjywxk, Connor1996, glorv, LykxSassinator, overvenus, v01dstar The full list of commands accepted by this bot can be found here. The pull request process is described here
Needs approval from an approver in each of these files:
Approvers can indicate their approval by writing |
[LGTM Timeline notifier]Timeline:
|
close #17683 Add metrics for raft message send wait & receive delay duration ## Send wait duration This phase begins when the RaftStore thread sends the RaftMessage to the RaftClient and ends when the RaftMessage leaves the BatchRaftMessage buffer, just before being flushed to the gRPC client. Since this phase occurs entirely within the same node, we measure it using monotonic time. ## Receive delay duration Receive delay duration can also be called send duration. The name 'Receive delay duration' is used because this duration is reported by the receiver, making it more clearer. This phase begins after the send wait ends and continues as the message is sent over the network, ends when the target peer receives it. Since this phase spans across nodes, we measure it using physical time. To facilitate this, we’ve introduced a last_observed_time field in the BatchRaftMessage to record the physical times (please also review the related PR: pingcap/kvproto#1276). Although physical clock drift between nodes is a possibility, its impact on our measurement is limited. Our primary goal is to observe trends in duration changes rather than relying on absolute precision. NOTE: Metrics are only added for the batch_raft RPC, as the raft RPC is deprecated and no longer tracked. Signed-off-by: hhwyt <[email protected]>
…#17735) close tikv#17683 Add metrics for raft message send wait & receive delay duration ## Send wait duration This phase begins when the RaftStore thread sends the RaftMessage to the RaftClient and ends when the RaftMessage leaves the BatchRaftMessage buffer, just before being flushed to the gRPC client. Since this phase occurs entirely within the same node, we measure it using monotonic time. ## Receive delay duration Receive delay duration can also be called send duration. The name 'Receive delay duration' is used because this duration is reported by the receiver, making it more clearer. This phase begins after the send wait ends and continues as the message is sent over the network, ends when the target peer receives it. Since this phase spans across nodes, we measure it using physical time. To facilitate this, we’ve introduced a last_observed_time field in the BatchRaftMessage to record the physical times (please also review the related PR: pingcap/kvproto#1276). Although physical clock drift between nodes is a possibility, its impact on our measurement is limited. Our primary goal is to observe trends in duration changes rather than relying on absolute precision. NOTE: Metrics are only added for the batch_raft RPC, as the raft RPC is deprecated and no longer tracked. Signed-off-by: hhwyt <[email protected]> Signed-off-by: 3AceShowHand <[email protected]>
For RaftMessage send & send wait duration metric, we need the
last_observed_timefield for the BatchRaftMessage. More details see: tikv/tikv#17735.