From 0467edb32b28c052f1d8ecb1b8d1d60375cb19af Mon Sep 17 00:00:00 2001 From: MyonKeminta Date: Wed, 31 Aug 2022 19:40:24 +0800 Subject: [PATCH 1/4] docs: Add design document about enhanced queueing for pessimistic lock contention Signed-off-by: MyonKeminta --- ...ueueing-for-pessimistic-lock-contention.md | 86 +++++++++++++++++++ 1 file changed, 86 insertions(+) create mode 100644 docs/design/2022-08-29-enhanced-queueing-for-pessimistic-lock-contention.md diff --git a/docs/design/2022-08-29-enhanced-queueing-for-pessimistic-lock-contention.md b/docs/design/2022-08-29-enhanced-queueing-for-pessimistic-lock-contention.md new file mode 100644 index 0000000000000..117ebf5bf0161 --- /dev/null +++ b/docs/design/2022-08-29-enhanced-queueing-for-pessimistic-lock-contention.md @@ -0,0 +1,86 @@ +# Enhanced Queueing for Pessimistic Lock Contention + +- Author(s): [MyonKeminta](http://github.com/MyonKeminta) +- Tracking Issue: https://github.com/tikv/tikv/issues/13298 + +## Abstract + +This proposes to make use of the new pessimistic lock waiting model as designed in [TiKV RFC #100](https://github.com/tikv/rfcs/pull/100), which is expected to reduce the tail latency problem in scenarios with frequent pessimistic lock conflicts. The design is currently only applicable for single key point-get locking. + +## Background + +As said in [TiKV RFC #100](https://github.com/tikv/rfcs/pull/100), our current implementation of pessimistic locks might be problematic in case there are frequent conflicts. Transactions waiting for the lock on the same key may be granted the lock in random order and may perform too many useless statement retries, which may lead to high tail latency. To solve the problem, we designed a new lock waiting model. We expect the new model can enhance the queueing behavior of concurrent conflicting pessimistic lock operations, so that the conflicting transactions can execute in more serialized order. It's also expected to reduce useless statement retries, which saves CPU cost and RPC calls. Due to the complexity of the implementation, as the first step, we will support the optimized model for pessimistic lock requests that affects only one key. + +## Design + +### Changes in TiKV side + +The majority part of the change is in TiKV side, which is explained in detail in [TiKV RFC #100](https://github.com/tikv/rfcs/pull/100). Briefly speaking, the differences from TiDB's perspective are (only applicable for pessimistic requests that locks only one key): + +- By specifying a special parameter, a pessimistic lock request is allowed to lock a key even there is write conflict, in which case TiKV can return the value and `commit_ts` of the latest version, namely `locked_with_conflict_ts`. The actual lock written down in TiKV will have its `for_update_ts` field equal to the latest `commit_ts` on the key. +- By specifying the parameter mentioned above, the pessimistic lock request is also allowed to continue locking the key after waiting for the lock of another transaction, instead of always reporting WriteConflict after being woken up. + +### Aggressive Locking + +When a key is locked with conflict, the current statement becomes executing at a different snapshot. However, the statement may have already read data in the expired snapshot (as specified by the `for_update_ts` of the statement). In this case, we have no choice but retry the current statement with a new `for_update_ts`. + +In our original implementation, when retrying a statement, the pessimistic locks that were already acquired will be rolled back, since it's possible that the keys we need to lock may change after retrying. However, we will choose a different way to handle this case with our new locking mechanism. We expect that in most cases, the keys we need to lock won't change after retrying at a newer snapshot. Therefore, we adopt this way, namely *aggressive locking*: + +- When performing statement retry, if there are already some keys locked during the previous attempt, do not roll them back immediately. We denote the set of the keys locked during previous attempt by $S_0$. +- After executing the statement again, it might be found that some keys needs to be locked. We denote the set of the keys need to be locked by $S$. +- Then, we will send requests to lock keys in $S - S_0$, and rollback keys in $S_0 - S$. Those keys that were already locked in the previous attempt don't need to be locked again. + +In most cases, we expect that $S_0 \subseteq S$, therefore rolling back the locks and acquire them again like the old implementation may be a waste. It's also possible in the original implementation that the lock is acquired by another transaction between rolling back the lock and acquiring the lock again, causing the statement retry useless. By keeping the lock until it's confirmed that the lock is not needed anymore, we avoid the problem, at the cost of causing more conflict when $S_0 - S$ is not empty. + +We plan to support this kind of behavior only for simple point-get queries for now. + +To support this behavior, we add some new methods to the `KVTxn` type in client-go: + +```go +// Start an aggressive locking session. Usually called when starting a DML statement. +func (*KVTxn) StartAggressiveLocking() +// Start (n+1)-th attempt of the statement (due to pessimistic retry); rollback unnecessary locks locked in (n-1)-th attempt. +func (*KVTxn) RetryAggressiveLocking() +// Rollback all pessimistic locks acquired during the pessimistic locking session, and exit aggressive locking state. +func (*KVTxn) CancelAggressiveLocking() +// Record keys locked in current (n-th) attempt to MemDB, rollback locks locked in previous (n-1)-th attempt, and exit aggressive locking state. +func (*KVTxn) DoneAggressiveLocking() +``` + +These functions will implement the behavior stated above, and the basic pattern of using these functions is like (pseudo code): + +```go +txn.StartAggressiveLocking() +for { + result := tryExecuteStatement() + switch result { + case PESSIMISTIC_RETRY: + txn.RetryAggressiveLocking() + continue; + case FAIL: + txn.CancelAggressiveLocking() + break; + case SUCCESS: { + txn.DoneAggressiveLocking() + break; + } +} +``` + +### Performance issue + +In the old implementation, when TiKV executes a scheduler command that releases a lock (e.g. `Commit` or `Rollback`), it wakes up the lock-waiting request after executing `process_write` of the command but before writing down the data to the Raft layer. At this time, releasing lock is actually not finished yet. However, in the new design of TiKV as stated in [TiKV RFC #100](https://github.com/tikv/rfcs/pull/100), the pessimistic lock request should be resumed after being woken up, and it can only return to TiDB after it successfully acquire the lock. This difference makes TiDB starts pessimistic retry later than before. In some scenarios, this may make the average latency higher than before, which may also results in lower QPS. + +It would be complicated to totally solve the problem, but we have a simpler idea to optimize it for some specific scenarios. When executing a statement and it performs lock-with-conflict, if the statement haven't performed any other read/write, it can actually continue executing, update the `for_update_ts` if necessary, and avoid retrying the whole statement. + +### Configurations + +We want to avoid introducing new configurations that user must know to make use of the new behavior, but it's a fact that the optimization is very complicated and there's known performance regression in a few scenarios. It would be risky if the new behavior is always enabled unconditionally. We can introduce a hidden system variable which can be used to disable the optimization in case there's any problem. + +#### `tidb_txn_enhanced_lock_queueing` + +Specifies whether the optimization stated above is enabled. + +- Scope: global or session +- Value: `0` or `1` +- Default: `1` for new clusters, `0` for clusters upgraded from old version From 8931b2126224758d0d3a255fc6b7c1b61b1e5edf Mon Sep 17 00:00:00 2001 From: MyonKeminta Date: Wed, 31 Aug 2022 19:43:48 +0800 Subject: [PATCH 2/4] refine expression --- ...2-08-29-enhanced-queueing-for-pessimistic-lock-contention.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/design/2022-08-29-enhanced-queueing-for-pessimistic-lock-contention.md b/docs/design/2022-08-29-enhanced-queueing-for-pessimistic-lock-contention.md index 117ebf5bf0161..dc91094842ea0 100644 --- a/docs/design/2022-08-29-enhanced-queueing-for-pessimistic-lock-contention.md +++ b/docs/design/2022-08-29-enhanced-queueing-for-pessimistic-lock-contention.md @@ -30,7 +30,7 @@ In our original implementation, when retrying a statement, the pessimistic locks - After executing the statement again, it might be found that some keys needs to be locked. We denote the set of the keys need to be locked by $S$. - Then, we will send requests to lock keys in $S - S_0$, and rollback keys in $S_0 - S$. Those keys that were already locked in the previous attempt don't need to be locked again. -In most cases, we expect that $S_0 \subseteq S$, therefore rolling back the locks and acquire them again like the old implementation may be a waste. It's also possible in the original implementation that the lock is acquired by another transaction between rolling back the lock and acquiring the lock again, causing the statement retry useless. By keeping the lock until it's confirmed that the lock is not needed anymore, we avoid the problem, at the cost of causing more conflict when $S_0 - S$ is not empty. +In most cases, we expect that $S_0 \subseteq S$, in which case rolling back the locks and acquire them again like the old implementation may be a waste. It's also possible in the original implementation that the lock is acquired by another transaction between rolling back the lock and acquiring the lock again, causing the statement retry useless. By keeping the lock until it's confirmed that the lock is not needed anymore, we avoid the problem, at the cost of causing more conflict when $S_0 - S$ is not empty. We plan to support this kind of behavior only for simple point-get queries for now. From 5d533e87ee6e9ba4d5816a6448c19089f7586bf7 Mon Sep 17 00:00:00 2001 From: MyonKeminta Date: Fri, 13 Jan 2023 18:17:05 +0800 Subject: [PATCH 3/4] Update the document to reflect the latest change Signed-off-by: MyonKeminta --- ...29-enhanced-queueing-for-pessimistic-lock-contention.md | 7 ++++--- 1 file changed, 4 insertions(+), 3 deletions(-) diff --git a/docs/design/2022-08-29-enhanced-queueing-for-pessimistic-lock-contention.md b/docs/design/2022-08-29-enhanced-queueing-for-pessimistic-lock-contention.md index dc91094842ea0..0047fa5335303 100644 --- a/docs/design/2022-08-29-enhanced-queueing-for-pessimistic-lock-contention.md +++ b/docs/design/2022-08-29-enhanced-queueing-for-pessimistic-lock-contention.md @@ -1,4 +1,4 @@ -# Enhanced Queueing for Pessimistic Lock Contention +# Aggressive Locking: Enhanced Queueing for Pessimistic Lock Contention - Author(s): [MyonKeminta](http://github.com/MyonKeminta) - Tracking Issue: https://github.com/tikv/tikv/issues/13298 @@ -47,7 +47,8 @@ func (*KVTxn) CancelAggressiveLocking() func (*KVTxn) DoneAggressiveLocking() ``` -These functions will implement the behavior stated above, and the basic pattern of using these functions is like (pseudo code): +These functions will implement the behavior stated above, and the basic pattern of using these functions is like +(pseudo code, the actual invocations of `(Retry|Cancel|Done)AggressiveLocking` are put in `OnStmtRetry`, `OnStmtRollback`, `OnStmtCommit` instead of the retry loop): ```go txn.StartAggressiveLocking() @@ -77,7 +78,7 @@ It would be complicated to totally solve the problem, but we have a simpler idea We want to avoid introducing new configurations that user must know to make use of the new behavior, but it's a fact that the optimization is very complicated and there's known performance regression in a few scenarios. It would be risky if the new behavior is always enabled unconditionally. We can introduce a hidden system variable which can be used to disable the optimization in case there's any problem. -#### `tidb_txn_enhanced_lock_queueing` +#### `tidb_pessimistic_txn_aggressive_locking` Specifies whether the optimization stated above is enabled. From e5b166bd9b093d23343d39b6933172791e470d09 Mon Sep 17 00:00:00 2001 From: MyonKeminta Date: Mon, 13 Mar 2023 15:04:00 +0800 Subject: [PATCH 4/4] renaming aggressive locking to fair locking Signed-off-by: MyonKeminta --- ...ueueing-for-pessimistic-lock-contention.md | 32 +++++++++---------- 1 file changed, 16 insertions(+), 16 deletions(-) diff --git a/docs/design/2022-08-29-enhanced-queueing-for-pessimistic-lock-contention.md b/docs/design/2022-08-29-enhanced-queueing-for-pessimistic-lock-contention.md index 0047fa5335303..1781213fcee61 100644 --- a/docs/design/2022-08-29-enhanced-queueing-for-pessimistic-lock-contention.md +++ b/docs/design/2022-08-29-enhanced-queueing-for-pessimistic-lock-contention.md @@ -1,4 +1,4 @@ -# Aggressive Locking: Enhanced Queueing for Pessimistic Lock Contention +# Fair Locking: Enhanced Queueing for Pessimistic Lock Contention - Author(s): [MyonKeminta](http://github.com/MyonKeminta) - Tracking Issue: https://github.com/tikv/tikv/issues/13298 @@ -20,11 +20,11 @@ The majority part of the change is in TiKV side, which is explained in detail in - By specifying a special parameter, a pessimistic lock request is allowed to lock a key even there is write conflict, in which case TiKV can return the value and `commit_ts` of the latest version, namely `locked_with_conflict_ts`. The actual lock written down in TiKV will have its `for_update_ts` field equal to the latest `commit_ts` on the key. - By specifying the parameter mentioned above, the pessimistic lock request is also allowed to continue locking the key after waiting for the lock of another transaction, instead of always reporting WriteConflict after being woken up. -### Aggressive Locking +### Fair Locking When a key is locked with conflict, the current statement becomes executing at a different snapshot. However, the statement may have already read data in the expired snapshot (as specified by the `for_update_ts` of the statement). In this case, we have no choice but retry the current statement with a new `for_update_ts`. -In our original implementation, when retrying a statement, the pessimistic locks that were already acquired will be rolled back, since it's possible that the keys we need to lock may change after retrying. However, we will choose a different way to handle this case with our new locking mechanism. We expect that in most cases, the keys we need to lock won't change after retrying at a newer snapshot. Therefore, we adopt this way, namely *aggressive locking*: +In our original implementation, when retrying a statement, the pessimistic locks that were already acquired will be rolled back, since it's possible that the keys we need to lock may change after retrying. However, we will choose a different way to handle this case with our new locking mechanism. We expect that in most cases, the keys we need to lock won't change after retrying at a newer snapshot. Therefore, we adopt this way, namely *fair locking*: - When performing statement retry, if there are already some keys locked during the previous attempt, do not roll them back immediately. We denote the set of the keys locked during previous attempt by $S_0$. - After executing the statement again, it might be found that some keys needs to be locked. We denote the set of the keys need to be locked by $S$. @@ -37,32 +37,32 @@ We plan to support this kind of behavior only for simple point-get queries for n To support this behavior, we add some new methods to the `KVTxn` type in client-go: ```go -// Start an aggressive locking session. Usually called when starting a DML statement. -func (*KVTxn) StartAggressiveLocking() +// Start an fair locking session. Usually called when starting a DML statement. +func (*KVTxn) StartFairLocking() // Start (n+1)-th attempt of the statement (due to pessimistic retry); rollback unnecessary locks locked in (n-1)-th attempt. -func (*KVTxn) RetryAggressiveLocking() -// Rollback all pessimistic locks acquired during the pessimistic locking session, and exit aggressive locking state. -func (*KVTxn) CancelAggressiveLocking() -// Record keys locked in current (n-th) attempt to MemDB, rollback locks locked in previous (n-1)-th attempt, and exit aggressive locking state. -func (*KVTxn) DoneAggressiveLocking() +func (*KVTxn) RetryFairLocking() +// Rollback all pessimistic locks acquired during the pessimistic locking session, and exit fair locking state. +func (*KVTxn) CancelFairLocking() +// Record keys locked in current (n-th) attempt to MemDB, rollback locks locked in previous (n-1)-th attempt, and exit fair locking state. +func (*KVTxn) DoneFairLocking() ``` These functions will implement the behavior stated above, and the basic pattern of using these functions is like -(pseudo code, the actual invocations of `(Retry|Cancel|Done)AggressiveLocking` are put in `OnStmtRetry`, `OnStmtRollback`, `OnStmtCommit` instead of the retry loop): +(pseudo code, the actual invocations of `(Retry|Cancel|Done)FairLocking` are put in `OnStmtRetry`, `OnStmtRollback`, `OnStmtCommit` instead of the retry loop): ```go -txn.StartAggressiveLocking() +txn.StartFairLocking() for { result := tryExecuteStatement() switch result { case PESSIMISTIC_RETRY: - txn.RetryAggressiveLocking() + txn.RetryFairLocking() continue; case FAIL: - txn.CancelAggressiveLocking() + txn.CancelFairLocking() break; case SUCCESS: { - txn.DoneAggressiveLocking() + txn.DoneFairLocking() break; } } @@ -78,7 +78,7 @@ It would be complicated to totally solve the problem, but we have a simpler idea We want to avoid introducing new configurations that user must know to make use of the new behavior, but it's a fact that the optimization is very complicated and there's known performance regression in a few scenarios. It would be risky if the new behavior is always enabled unconditionally. We can introduce a hidden system variable which can be used to disable the optimization in case there's any problem. -#### `tidb_pessimistic_txn_aggressive_locking` +#### `tidb_pessimistic_txn_fair_locking` Specifies whether the optimization stated above is enabled.