Thriftpy Tracking

Thriftpy Tracking

When tracked client created, it will send a negotiation rpc call to server to check whether the server support tracking. If the server do not support tracking the client will behave like a normal client. If support the client will send a RequestHeader before every rpc call.

So both the client and server should work in a tracking or normal scenario.

Negotiation

Rpc call:

• method name: trace_method = "__thriftpy_tracing_method_name__"
• message type: TMessageType.CALL
• sequence id: 0
• call args struct: UpgradeArgs
• call reply struct: UpgradeReply
• should not oneway

Client send this rpc call and wait for response. If no exception raised set self._upgraded to True which means that the server support tracking.

If the client is upgraded, the client should send tracked info in the next rpc calls.

When received a rpc call, the server should check the api name and message type. If the api name is trace_method and message type is TMessageType.CALL, set self._upgraded to True which means that the server is switching to tracking mode. The server build a reply packet using call reply struct.

If the server is upgraded, it should extract RequestHeader first from the next incoming rpc requests.

Tracking

Data structures:

struct RequestHeader {
    1: string request_id
    2: string seq
}

struct RequestInfo {
    1: string request_id // used to identify a request
    2: string api // api name
    3: string seq // rpc call sequence
    4: string client // client name
    5: string server // server name
    6: bool status // request status
    7: i64 start // start timestamp
    8: i64 end // end timestamp
}

struct UpgradeReply {}
struct UpgradeArgs {}

Client use Tracker to build a RequestHeader and send it to tracked server.

Tracker should define three methods:

• gen_header, args: an empty RequestHeader instance, return: None. Used to generate a new request header.
• handle, args: RequestHeader received from client, return: None. Used to store the received header for transferring the header info.
• record, args: RequestInfo and exception, return: None. Used to record tracking info.

seq is the call sequence represented as a string of dot separated numbers like 1, 1.1, 1.1.1, 1.2, 1.2.1. When server handle a request header, it set two thread local variables header and counter. If it has nested rpc calls it should increase counter by one and generate a new seq by concatenating header.seq and counter.

When a rpc request finished client build a RequestInfo and use Tracker to record this info.

The flow:

                                              +-----------------------------------------------------+
          +------------------------------+    |  1. receive response                                |
          | 1. gen_header                |    |  2. collect header info:                            |
client    | 2. write header to transport |    |   request_id and seq from original generated header |
          | 3. remeber send_start time   |    |   client and server name from Tracker               |
          | 4. send rpc call             |    |   api, status, start, end are get from this client  |
          +-----------------------+------+    |  3. record                                          |
                                  |           +-----------------------------------------------------+
                         request  |                    ^
                                  |                    | response
                    +-------------V--------------------+----+
                    | 1. extract header info                |
                    | 2. handle                             |
server              | 3. read rpc call                      |
                    | 4. execute rpc call and send response |
                    +---------------------------------------+

Server uses Tracker to handle RequestHeader. It just stores this header and use it to generate the next headers for nested rpc calls. So it can transfer this request id to the next calls.