Refactor and Query Discovery Mode

- Refactor the code to make it more readable and maintainable. Added
    a new module `proxysql` to handle all the proxysql related
    functionalities.
- Refactor the code to use the `query` module to handle all the
    query discovery related functionalities.

- Added Query Discovery configuration. Now users can specify the
    query discovery mode in the configuration file. The set of rules
    to discover the queries are defined in the README.md file.

Closes: #6

README.md

@@ -6,12 +6,12 @@ Unlock the full potential of your database integrating ReadySet and ProxySQL by
 This scheduler executes the following steps:
 
 1. Locks an in disk file (configured by `lock_file`) to avoid multiple instances of the scheduler to overlap their execution.
-2. Check if it can connect to Readyset and validate if `Snapshot Status` is `Completed`. In case it cannot connect or Readyset is still performing snapshot it adjust the server status to `SHUNNED` in ProxySQL.
-2. Queries the table `stats_mysql_query_digest` from ProxySQL and validates if each query is supported by Readyset
+2. If `mode=(All|HealthCheck)` -  Query `mysql_servers` and check all servers that have `comment='Readyset` (case insensitive) and `hostgroup=readyset_hostgroup`. For each server it checks if it can connect to Readyset and validate if `Snapshot Status` is `Completed`. In case it cannot connect or Readyset is still performing snapshot it adjust the server status to `SHUNNED` in ProxySQL.
+3. If `mode=(All|QueryDiscovery)` Query the table `stats_mysql_query_digest` finding queries executed at `source_hostgroup` by `readyset_user` and validates if each query is supported by Readyset. The rules to order queries are configured by [Query Discovery](#query-discovery) configurations. 
 3. If the query is supported it adds a cache in Readyset by executing `CREATE CACHE FROM __query__`.
 4. If `warmup_time` is NOT configure, a new query rule will be added redirecting this query to Readyset
 5. If `warmup_time` is configured, a new query rule will be added to mirror this query to Readyset. The query will still be redirected to the original hostgroup
-6. Once `warmup_time` seconds has elapsed since the query was mirrored, the query rule will be updated to redirect the qury to Readyset instead of mirroring.
+6. Once `warmup_time` seconds has elapsed since the query was mirrored, the query rule will be updated to redirect the query to Readyset instead of mirroring.
 
 
 
@@ -20,11 +20,13 @@ This scheduler executes the following steps:
 Assuming you have your ProxySQL already Configured you will need to create a new hostgroup and add Readyset to this hostgroup:
 
 ```
-INSERT INTO mysql_servers (hostgroup_id, hostname, port) VALUES (99, '127.0.0.1', 3307);
+INSERT INTO mysql_servers (hostgroup_id, hostname, port, comment) VALUES (99, '127.0.0.1', 3307, 'Readyset');
 LOAD MYSQL SERVERS TO RUNTIME;
 SAVE MYSQL SERVERS TO DISK;
 ```
 
+*NOTE*: It's required to add `Readyset` as a comment to the server to be able to identify it in the scheduler.
+
 To configure the scheduler to run execute:
 
 ```
@@ -40,13 +42,66 @@ Configure `/etc/readyset_proxysql_scheduler.cnf` as follow:
 * `proxysql_port` - (Required) - Proxysql admin port
 * `readyset_user` - (Required) - Readyset application user
 * `readyset_password` - (Required) - Readyset application password
-* `readyset_host` - (Required) - Readyset host
-* `readyset_port` - (Required) - Readyset port
 * `source_hostgroup` - (Required) - Hostgroup running your Read workload
 * `readyset_hostgroup` - (Required) - Hostgroup where Readyset is configure
 * `warmup_time` - (Optional) - Time in seconds to mirror a query supported before redirecting the query to Readyset (Default 0 - no mirror)
 * `lock_file` - (Optional) - Lock file to prevent two instances of the scheduler to run at the same time (Default '/etc/readyset_scheduler.lock')
+* `operation_mode` - (Optional) - Operation mode to run the scheduler. The options are described in [Operation Mode](#operation-mode) (Default All).
+* `number_of_queries` - (Optional) - Number of queries to cache in Readyset (Default 10).
+* `query_discovery_mode` / `query_discovery_min_execution` / `query_discovery_min_row_sent` - (Optional) - Query Discovery configurations. The options are described in [Query Discovery](#query-discovery) (Default CountStar / 0 / 0).
+
+
+# Query Discovery
+The Query Discovery is a set of configuration to find queries that are supported by Readyset. The configurations are defined by the following fields:
+
+* `query_discovery_mode`: (Optional) - Mode to discover queries to automatically cache in Readyset. The options are described in [Query Discovery Mode](#query-discovery-mode)   (Default CountStar).
+* `query_discovery_min_execution`: (Optional) - Minimum number of executions of a query to be considered a candidate to be cached (Default 0).
+* `query_discovery_min_row_sent`: (Optional) - Minimum number of rows sent by a query to be considered a candidate to be cached (Default 0).
+
+# Query Discovery Mode
+The Query Discovery Mode is a set of possible rules to discover queries to automatically cache in Readyset. The options are:
+
+1. `CountStar` - Total Number of Query Executions
+ * Formula: `total_executions = count_star`
+ * Description: This metric gives the total number of times the query has been executed. It is valuable for understanding how frequently the query runs. A high count_star value suggests that the query is executed often.
+
+2. `SumTime` - Total Time Spent Executing the Query
+ * Formula: `total_execution_time = sum_time`
+ * Description: This metric represents the total cumulative time spent executing the query across all its executions. It provides a clear understanding of how much processing time the query is consuming over time. A high total execution time can indicate that the query is either frequently executed or is time-intensive to process.
+
+3. `SumRowsSent` - Total Number of Rows Sent by the Query (sum_rows_sent)
+ * Formula: `total_rows_sent = sum_rows_sent`
+ * Description: This metric provides the total number of rows sent to the client across all executions of the query. It helps you understand the query’s output volume and the amount of data being transmitted. 
+
+4. `MeanTime` - Average Query Execution Time (Mean)
+ * Formula: `mean_time = sum_time / count_star`
+ * Description: The mean time gives you an idea of the typical performance of the query over all executions. It provides a central tendency of how long the query generally takes to execute.
+
+5. `ExecutionTimeDistance` - Time Distance Between Query Executions
+ * Formula: `execution_time_distance = max_time - min_time`
+ * Description: This shows the spread between the fastest and slowest executions of the query. A large range might indicate variability in system load, input sizes, or external factors affecting performance.
+
+6. `QueryThroughput` - Query Throughput
+ * Formula: `query_throughput = count_star / sum_time`
+ * Description: This shows how many queries are processed per unit of time. It’s useful for understanding system capacity and how efficiently the database is handling the queries.
+
+7. `WorstBestCase` - Worst Best-Case Query Performance
+ * Formula: `worst_case = min_time`
+ * Description: The min_time metric gives the fastest time the query was ever executed. It reflects the best-case performance scenario, which could indicate the query’s performance under optimal conditions.
+
+8. `WorstWorstCase` - Worst Worst-Case Query Performance
+ * Formula: `worst_case = max_time`
+ * Description: The max_time shows the slowest time the query was executed. This can indicate potential bottlenecks or edge cases where the query underperforms, which could be due to larger data sets, locks, or high server load.
+
+9. `DistanceMeanMax` - Distance Between Mean Time and Max Time (mean_time vs max_time)
+ * Formula: `distance_mean_max = max_time - mean_time`
+ * Description: The distance between the mean execution time and the maximum execution time provides insight into how much slower the worst-case execution is compared to the average. A large gap indicates significant variability in query performance, which could be caused by certain executions encountering performance bottlenecks, such as large datasets, locking, or high system load.
 
+# Operation Mode
+The Operation Mode is a set of possible rules to run the scheduler. The options are:
+* `All` - Run `HealthCheck` and `QueryDiscovery` operations.
+* `HealthCheck` - Run only the health check operation.
+* `QueryDiscovery` - Run only the query discovery operation.
 
 # Note
 Readyset support of MySQL and this scheduler are alpha quality, meaning they are not currently part of our test cycle. Run your own testing before plugging this to your production system.

build/test.cnf

@@ -6,7 +6,8 @@ readyset_user = 'root'
 readyset_password = 'noria'
 source_hostgroup = 1
 readyset_hostgroup = 2
-warmup_time = 60
+warmup_time = 5
 lock_file = '/tmp/readyset_scheduler.lock'
-operation_mode="All"
-number_of_queries=10
+operation_mode='All'
+number_of_queries=2
+query_discovery_mode='SumTime'

readyset_proxysql_scheduler.cnf

@@ -10,3 +10,4 @@ warmup_time = 60
 lock_file = '/tmp/readyset_scheduler.lock'
 operation_mode="All"
 number_of_queries=10
+query_discovery_mode='SumTime'

src/config.rs

@@ -14,7 +14,7 @@ pub enum OperationMode {
 
 impl From<String> for OperationMode {
     fn from(s: String) -> Self {
-        match s.as_str() {
+        match s.to_lowercase().as_str() {
             "health_check" => OperationMode::HealthCheck,
             "query_discovery" => OperationMode::QueryDiscovery,
             "all" => OperationMode::All,
@@ -33,6 +33,39 @@ impl Display for OperationMode {
     }
 }
 
+#[derive(serde::Deserialize, Clone, Copy, PartialEq, PartialOrd, Default)]
+pub enum QueryDiscoveryMode {
+    #[default]
+    CountStar,
+    SumTime,
+    SumRowsSent,
+    MeanTime,
+    ExecutionTimeDistance,
+    QueryThroughput,
+    WorstBestCase,
+    WorstWorstCase,
+    DistanceMeanMax,
+    External,
+}
+
+impl From<String> for QueryDiscoveryMode {
+    fn from(s: String) -> Self {
+        match s.to_lowercase().as_str() {
+            "count_star" => QueryDiscoveryMode::CountStar,
+            "sum_time" => QueryDiscoveryMode::SumTime,
+            "sum_rows_sent" => QueryDiscoveryMode::SumRowsSent,
+            "mean_time" => QueryDiscoveryMode::MeanTime,
+            "execution_time_distance" => QueryDiscoveryMode::ExecutionTimeDistance,
+            "query_throughput" => QueryDiscoveryMode::QueryThroughput,
+            "worst_best_case" => QueryDiscoveryMode::WorstBestCase,
+            "worst_worst_case" => QueryDiscoveryMode::WorstWorstCase,
+            "distance_mean_max" => QueryDiscoveryMode::DistanceMeanMax,
+            "external" => QueryDiscoveryMode::External,
+            _ => QueryDiscoveryMode::CountStar,
+        }
+    }
+}
+
 #[derive(serde::Deserialize, Clone)]
 pub struct Config {
     pub proxysql_user: String,
@@ -47,6 +80,9 @@ pub struct Config {
     pub lock_file: Option<String>,
     pub operation_mode: Option<OperationMode>,
     pub number_of_queries: u16,
+    pub query_discovery_mode: Option<QueryDiscoveryMode>,
+    pub query_discovery_min_execution: Option<u64>,
+    pub query_discovery_min_row_sent: Option<u64>,
 }
 
 pub fn read_config_file(path: &str) -> Result<String, std::io::Error> {

src/hosts.rs

@@ -1,4 +1,4 @@
-use crate::{config::Config, messages};
+use crate::{config::Config, queries::Query};
 use core::fmt;
 use mysql::{prelude::Queryable, Conn, OptsBuilder};
 
@@ -113,10 +113,19 @@ impl Host {
     /// # Returns
     ///
     /// The status of the host.
-    fn get_status(&self) -> HostStatus {
+    pub fn get_status(&self) -> HostStatus {
         self.status
     }
 
+    /// Changes the status of the host.
+    ///
+    /// # Arguments
+    ///
+    /// * `status` - The new status of the host.
+    pub fn change_status(&mut self, status: HostStatus) {
+        self.status = status;
+    }
+
     /// Checks if the host is online.
     ///
     /// # Returns
@@ -192,132 +201,18 @@ impl Host {
     /// # Returns
     ///
     /// true if the query was cached successfully, false otherwise.
-    pub fn cache_query(
-        &mut self,
-        digest_text: &String,
-        digest: &String,
-    ) -> Result<bool, mysql::Error> {
+    pub fn cache_query(&mut self, query: &Query) -> Result<bool, mysql::Error> {
         match &mut self.conn {
             None => return Ok(false),
             Some(conn) => {
-                conn.query_drop(format!("CREATE CACHE d_{} FROM {}", digest, digest_text))
-                    .expect("Failed to create readyset cache");
+                conn.query_drop(format!(
+                    "CREATE CACHE d_{} FROM {}",
+                    query.get_digest(),
+                    query.get_digest_text()
+                ))
+                .expect("Failed to create readyset cache");
             }
         }
         Ok(true)
     }
-
-    /// Changes the status of the host in the ProxySQL mysql_servers table.
-    /// The status is set to the given `status`.
-    pub fn change_status(
-        &mut self,
-        ps_conn: &mut Conn,
-        config: &Config,
-        status: HostStatus,
-    ) -> Result<bool, mysql::Error> {
-        let where_clause = format!(
-            "WHERE hostgroup_id = {} AND hostname = '{}' AND port = {}",
-            config.readyset_hostgroup,
-            self.get_hostname(),
-            self.get_port()
-        );
-        if self.status != status {
-            messages::print_info(
-                format!(
-                    "Server HG: {}, Host: {}, Port: {} is currently {}. Changing to {}",
-                    config.readyset_hostgroup,
-                    self.get_hostname(),
-                    self.get_port(),
-                    self.get_status(),
-                    status
-                )
-                .as_str(),
-            );
-            self.status = status;
-            ps_conn.query_drop(format!(
-                "UPDATE mysql_servers SET status = '{}' {}",
-                self.get_status(),
-                where_clause
-            ))?;
-            ps_conn.query_drop("LOAD MYSQL SERVERS TO RUNTIME")?;
-            ps_conn.query_drop("SAVE MYSQL SERVERS TO DISK")?;
-        }
-
-        Ok(true)
-    }
-}
-
-/// Represents a list of Readyset hosts
-pub struct Hosts {
-    hosts: Vec<Host>,
-}
-
-impl From<Vec<Host>> for Hosts {
-    fn from(hosts: Vec<Host>) -> Self {
-        Hosts { hosts }
-    }
-}
-
-impl Hosts {
-    /// Fetches the hosts from the ProxySQL mysql_servers table.
-    ///
-    /// # Arguments
-    ///
-    /// * `proxysql_conn` - The connection to the ProxySQL instance.
-    /// * `config` - The configuration object.
-    ///
-    /// # Returns
-    ///
-    /// A vector of `Host` instances.
-    pub fn new<'a>(proxysql_conn: &'a mut Conn, config: &'a Config) -> Self {
-        let query = format!(
-            "SELECT hostname, port, status, comment FROM mysql_servers WHERE hostgroup_id = {}",
-            config.readyset_hostgroup
-        );
-        let results: Vec<(String, u16, String, String)> = proxysql_conn.query(query).unwrap();
-        results
-            .into_iter()
-            .filter_map(|(hostname, port, status, comment)| {
-                if comment.to_lowercase().contains("readyset") {
-                    Some(Host::new(hostname, port, status, config))
-                } else {
-                    None
-                }
-            })
-            .collect::<Vec<Host>>()
-            .into()
-    }
-
-    /// Gets a mutable iterator over the hosts.
-    ///
-    /// # Returns
-    ///
-    /// A mutable iterator over the hosts.
-    pub fn iter_mut(&mut self) -> core::slice::IterMut<Host> {
-        self.hosts.iter_mut()
-    }
-
-    /// Mutate self by retaining only the hosts that are online.
-    pub fn retain_online(&mut self) {
-        self.hosts.retain(|host| host.is_online());
-    }
-
-    /// Checks if the hosts list is empty.
-    ///
-    /// # Returns
-    ///
-    /// true if the hosts list is empty, false otherwise.
-    pub fn is_empty(&self) -> bool {
-        self.hosts.is_empty()
-    }
-
-    /// Gets a mutable reference for the first host in the list.
-    ///
-    /// # Returns
-    ///
-    /// A reference to the first host in the list.
-    /// If the list is empty, the function returns None.
-    pub fn first_mut(&mut self) -> Option<&mut Host> {
-        self.hosts.first_mut()
-    }
 }