Merge remote-tracking branch 'upstream/master' into tunicate

CONTRIBUTING.md

@@ -115,7 +115,7 @@ Read our configuration guide to learn about various [configuration options](http
 
 This is a rough outline of what a contributor's workflow looks like:
 
-- Make sure what you want to contribute is already traced as an issue.
+- Make sure what you want to contribute is already traced as an issue (see below for linking issue).
   * We may discuss the problem and solution in the issue.
 - Create a Git branch from where you want to base your work. This is usually master.
 - Write code, add test cases, and commit your work (see below for message format).
@@ -150,45 +150,69 @@ The TiKV team actively develops and maintains a bunch of dependencies used in Ti
 
 See more in [TiKV Community](https://github.com/tikv/community).
 
-### Format of the commit message
+### Linking issues
+
+Code repositories in TiKV community require **ALL** the pull requests referring to its corresponding issues. In the pull request body, there **MUST** be one line starting with `Issue Number: ` and linking the relevant issues via the [keyword](https://docs.github.com/en/issues/tracking-your-work-with-issues/linking-a-pull-request-to-an-issue#linking-a-pull-request-to-an-issue-using-a-keyword), for example:
 
-We follow a rough convention for commit messages that is designed to answer two
-questions: what changed and why. The subject line should feature the what and
-the body of the commit should describe the why.
+If the pull request resolves the relevant issues, and you want GitHub to close these issues automatically after it merged into the default branch, you can use the syntax (`KEYWORD #ISSUE-NUMBER`) like this:
 
 ```
-engine/raftkv: add comment for variable declaration.
+Issue Number: close #123
+```
 
-Improve documentation.
+If the pull request links an issue but does not close it, you can use the keyword `ref` like this:
 
-Close #1234.
+```
+Issue Number: ref #456
 ```
 
-The format can be described more formally as follows:
+Multiple issues should use full syntax for each issue and separate by a comma, like:
 
 ```
-<subsystem>: <what changed>
-<BLANK LINE>
-<why this change was made>
-<BLANK LINE>
-<issue reference>
-<BLANK LINE>
-Signed-off-by: <Name> <email address>
+Issue Number: close #123, ref #456
+```
+
+For pull requests trying to close issues in a different repository, contributors need to first create an issue in the same repository and use this issue to track.
+
+If the pull request body does not provide the required content, the bot will add the `do-not-merge/needs-linked-issue` label to the pull request to prevent it from being merged.
+
+### Format of the commit message
+
+The bot we use will extract the pull request title as the one-line subject and messages inside the `commit-message` code block as commit message body. For example, a pull request with title `pkg: what's changed in this one package` and body containing:
+
+    ```commit-message
+    any multiple line commit messages that go into
+    the final commit message body
+
+    * fix something 1
+    * fix something 2
+    ```
+
+will get a final commit message:
+
+```
+pkg: what's changed in this one package (#12345)
+
+any multiple line commit messages that go into
+the final commit message body
+
+* fix something 1
+* fix something 2
 ```
 
-The first line is the subject and should be no longer than 50 characters, the other lines should be wrapped at 72 characters (see [this blog post](https://preslav.me/2015/02/21/what-s-with-the-50-72-rule/) for why).
+The first line is the subject (the pull request title) and should be no longer than 50 characters, the other lines should be wrapped at 72 characters (see [this blog post](https://preslav.me/2015/02/21/what-s-with-the-50-72-rule/) for why).
 
 If the change affects more than one subsystem, you can use comma to separate them like `util/codec,util/types:`.
 
 If the change affects many subsystems, you can use ```*``` instead, like ```*:```.
 
-The body of the commit message should describe why the change was made and at a high level, how the code works. It should also reference the issue for problem context.
+The body of the commit message should describe why the change was made and at a high level, how the code works.
 
 ### Signing off the Commit
 
 The project uses [DCO check](https://github.com/probot/dco#how-it-works) and the commit message must contain a `Signed-off-by` line for [Developer Certificate of Origin](https://developercertificate.org/).
 
-Use option `git commit -s` to sign off your commits.
+Use option `git commit -s` to sign off your commits. The bot will group and distinguish the signatures from all your commits in the pull request and append them to the final commit message body. 
 
 
 ### Testing AWS

Cargo.toml

@@ -112,7 +112,7 @@ pin-project = "1.0"
 pnet_datalink = "0.23"
 pprof = { version = "^0.6", default-features = false, features = ["flamegraph", "protobuf"] }
 protobuf = { version = "2.8", features = ["bytes"] }
-raft = { version = "0.6.0-alpha", default-features = false, features = ["protobuf-codec"] }
+raft = { version = "0.7.0", default-features = false, features = ["protobuf-codec"] }
 raftstore = { path = "components/raftstore", default-features = false }
 raft_log_engine = { path = "components/raft_log_engine", default-features = false }
 rand = "0.7.3"

Makefile

@@ -283,6 +283,7 @@ pre-clippy: unset-override
 
 clippy: pre-clippy
 	@./scripts/check-redact-log
+	@./scripts/check-docker-build
 	@./scripts/clippy-all
 
 pre-audit:

README.md

@@ -12,7 +12,8 @@ The design of TiKV ('Ti' stands for titanium) is inspired by some great distribu
 
 If you're interested in contributing to TiKV, or want to build it from source, see [CONTRIBUTING.md](./CONTRIBUTING.md).
 
-![cncf_logo](images/cncf.png)
+![cncf_logo](images/cncf.png#gh-light-mode-only)
+![cncf_logo](images/cncf-white.png#gh-dark-mode-only)
 
 TiKV is a graduated project of the [Cloud Native Computing Foundation](https://cncf.io/) (CNCF). If you are an organization that wants to help shape the evolution of technologies that are container-packaged, dynamically-scheduled and microservices-oriented, consider joining the CNCF. For details about who's involved and how TiKV plays a role, read the CNCF [announcement](https://www.cncf.io/announcements/2020/09/02/cloud-native-computing-foundation-announces-tikv-graduation/).
 

cmd/tikv-ctl/Cargo.toml

@@ -67,7 +67,7 @@ pd_client = { path = "../../components/pd_client", default-features = false }
 prometheus = { version = "0.13", features = ["nightly"] }
 promptly = "0.3.0"
 protobuf = { version = "2.8", features = ["bytes"] }
-raft = { version = "0.6.0-alpha", default-features = false, features = ["protobuf-codec"] }
+raft = { version = "0.7.0", default-features = false, features = ["protobuf-codec"] }
 raft_log_engine = { path = "../../components/raft_log_engine", default-features = false }
 raftstore = { path = "../../components/raftstore", default-features = false }
 rand = "0.8"

components/backup/Cargo.toml

@@ -47,7 +47,7 @@ lazy_static = "1.3"
 log_wrappers = { path = "../log_wrappers" }
 pd_client = { path = "../pd_client", default-features = false }
 prometheus = { version = "0.13", default-features = false, features = ["nightly"] }
-raft = { version = "0.6.0-alpha", default-features = false, features = ["protobuf-codec"] }
+raft = { version = "0.7.0", default-features = false, features = ["protobuf-codec"] }
 raftstore = { path = "../raftstore", default-features = false }
 security = { path = "../security", default-features = false }
 serde = "1.0"

components/backup/src/endpoint.rs

@@ -85,12 +85,6 @@ impl fmt::Debug for Task {
     }
 }
 
-#[derive(Clone)]
-struct LimitedStorage {
-    limiter: Limiter,
-    storage: Arc<dyn ExternalStorage>,
-}
-
 impl Task {
     /// Create a backup task based on the given backup request.
     pub fn new(
@@ -483,7 +477,7 @@ impl BackupRange {
         &self,
         engine: E,
         db: Arc<DB>,
-        storage: &LimitedStorage,
+        limiter: &Limiter,
         file_name: String,
         cf: CfNameWrap,
         compression_type: Option<SstCompressionType>,
@@ -495,7 +489,7 @@ impl BackupRange {
             db,
             &file_name,
             cf,
-            storage.limiter.clone(),
+            limiter.clone(),
             compression_type,
             compression_level,
             cipher,
@@ -793,7 +787,7 @@ impl<E: Engine, R: RegionInfoProvider + Clone + 'static> Endpoint<E, R> {
         request: Request,
         saver_tx: async_channel::Sender<InMemBackupFiles>,
         resp_tx: UnboundedSender<BackupResponse>,
-        backend: Arc<dyn ExternalStorage>,
+        _backend: Arc<dyn ExternalStorage>,
     ) {
         let start_ts = request.start_ts;
         let backup_ts = request.end_ts;
@@ -806,11 +800,6 @@ impl<E: Engine, R: RegionInfoProvider + Clone + 'static> Endpoint<E, R> {
         let limit = self.softlimit.limit();
 
         self.pool.borrow_mut().spawn(async move {
-            let storage = LimitedStorage {
-                limiter: request.limiter,
-                storage: backend,
-            };
-
             loop {
                 // when get the guard, release it until we finish scanning a batch, 
                 // because if we were suspended during scanning, 
@@ -868,7 +857,7 @@ impl<E: Engine, R: RegionInfoProvider + Clone + 'static> Endpoint<E, R> {
                             .backup_raw_kv_to_file(
                                 engine,
                                 db.clone(),
-                                &storage,
+                                &request.limiter,
                                 name,
                                 cf.into(),
                                 ct,
@@ -880,7 +869,7 @@ impl<E: Engine, R: RegionInfoProvider + Clone + 'static> Endpoint<E, R> {
                     } else {
                         let writer_builder = BackupWriterBuilder::new(
                             store_id,
-                            storage.limiter.clone(),
+                            request.limiter.clone(),
                             brange.region.clone(),
                             db.clone(),
                             ct,

components/cdc/Cargo.toml

@@ -31,7 +31,7 @@ grpcio = { version = "0.9",  default-features = false, features = ["openssl-vend
 keys = { path = "../keys" }
 kvproto = { git = "https://github.com/pingcap/kvproto.git" }
 pd_client = { path = "../pd_client", default-features = false }
-raft = { version = "0.6.0-alpha", default-features = false, features = ["protobuf-codec"] }
+raft = { version = "0.7.0", default-features = false, features = ["protobuf-codec"] }
 raftstore = { path = "../raftstore", default-features = false }
 resolved_ts = { path = "../resolved_ts", default-features = false }
 security = { path = "../security", default-features = false }
@@ -62,7 +62,7 @@ tempfile = "3.0"
 test_raftstore = { path = "../test_raftstore", default-features = false }
 test_util = { path = "../test_util", default-features = false }
 panic_hook = { path = "../panic_hook" }
-raft = { version = "0.6.0-alpha", default-features = false, features = ["protobuf-codec"] }
+raft = { version = "0.7.0", default-features = false, features = ["protobuf-codec"] }
 time = "0.1"
 criterion = "0.3"
 

components/cdc/src/channel.rs

@@ -387,12 +387,12 @@ impl<'a> Drain {
             });
             let (event_bytes, resolved_ts_bytes) = batcher.statistics();
             let resps = batcher.build();
-            let last_idx = resps.len() - 1;
+            let resps_len = resps.len();
             // Events are about to be sent, free pending events memory counter.
             memory_quota.free(bytes as _);
             for (i, e) in resps.into_iter().enumerate() {
                 // Buffer messages and flush them at once.
-                let write_flags = WriteFlags::default().buffer_hint(i != last_idx);
+                let write_flags = WriteFlags::default().buffer_hint(i + 1 != resps_len);
                 sink.feed((e, write_flags)).await?;
             }
             sink.flush().await?;

components/cdc/src/endpoint.rs

@@ -1673,6 +1673,7 @@ mod tests {
                 0,
                 "".to_owned(),
             )),
+            pending_remove: false,
         };
         store_meta.lock().unwrap().readers.insert(1, read_delegate);
         let (task_sched, task_rx) = dummy_scheduler();

components/cdc/tests/mod.rs

@@ -104,11 +104,13 @@ impl TestSuiteBuilder {
         }
     }
 
+    #[must_use]
     pub fn cluster(mut self, cluster: Cluster<ServerCluster>) -> TestSuiteBuilder {
         self.cluster = Some(cluster);
         self
     }
 
+    #[must_use]
     pub fn memory_quota(mut self, memory_quota: usize) -> TestSuiteBuilder {
         self.memory_quota = Some(memory_quota);
         self

components/encryption/src/file_dict_file.rs

@@ -548,14 +548,14 @@ mod tests {
             )
             .unwrap();
 
-            file_dict.insert(&"f1".to_owned(), &info1).unwrap();
-            file_dict.insert(&"f2".to_owned(), &info2).unwrap();
-            file_dict.insert(&"f3".to_owned(), &info3).unwrap();
+            file_dict.insert("f1", &info1).unwrap();
+            file_dict.insert("f2", &info2).unwrap();
+            file_dict.insert("f3", &info3).unwrap();
 
             file_dict.insert("f4", &info4).unwrap();
             file_dict.remove("f3").unwrap();
 
-            file_dict.remove(&"f2".to_owned()).unwrap();
+            file_dict.remove("f2").unwrap();
         }
         // Try open as v1 file. Should fail.
         {

components/encryption/src/io.rs

@@ -399,10 +399,7 @@ impl CrypterCore {
 
     fn reset_buffer(&mut self, size: usize) {
         // OCrypter require the output buffer to have block_size extra bytes, or it will panic.
-        self.buffer.reserve(size + self.block_size);
-        unsafe {
-            self.buffer.set_len(size + self.block_size);
-        }
+        self.buffer.resize(size + self.block_size, 0);
     }
 
     pub fn reset_crypter(&mut self, offset: u64) -> IoResult<()> {

components/engine_panic/Cargo.toml

@@ -11,5 +11,5 @@ tikv_alloc = { path = "../tikv_alloc" }
 # FIXME: Remove this dep from the engine_traits interface
 tikv_util = { path = "../tikv_util", default-features = false }
 kvproto = { git = "https://github.com/pingcap/kvproto.git" }
-raft = { version = "0.6.0-alpha", default-features = false, features = ["protobuf-codec"] }
+raft = { version = "0.7.0", default-features = false, features = ["protobuf-codec"] }
 txn_types = { path = "../txn_types", default-features = false }

components/engine_panic/src/raft_engine.rs

@@ -25,6 +25,10 @@ impl RaftEngineReadOnly for PanicEngine {
     ) -> Result<usize> {
         panic!()
     }
+
+    fn get_all_entries_to(&self, region_id: u64, buf: &mut Vec<Entry>) -> Result<()> {
+        panic!()
+    }
 }
 
 impl RaftEngine for PanicEngine {
@@ -55,6 +59,7 @@ impl RaftEngine for PanicEngine {
     fn clean(
         &self,
         raft_group_id: u64,
+        first_index: u64,
         state: &RaftLocalState,
         batch: &mut Self::LogBatch,
     ) -> Result<()> {

components/engine_rocks/Cargo.toml

@@ -45,7 +45,7 @@ tempfile = "3.0"
 serde = "1.0"
 serde_derive = "1.0"
 kvproto = { git = "https://github.com/pingcap/kvproto.git" }
-raft = { version = "0.6.0-alpha", default-features = false, features = ["protobuf-codec"] }
+raft = { version = "0.7.0", default-features = false, features = ["protobuf-codec"] }
 protobuf = "2"
 fail = "0.5"
 case_macros = { path = "../case_macros" }

components/engine_rocks/src/config.rs

@@ -199,7 +199,7 @@ pub mod compression_type_serde {
                     "disable" => DBCompressionType::Disable,
                     _ => {
                         return Err(E::invalid_value(
-                            Unexpected::Other(&"invalid compression type".to_string()),
+                            Unexpected::Other("invalid compression type"),
                             &self,
                         ));
                     }