Replication Is Broken With Gramine

[harunkucuk5]

Hi, we observed that when we use MariaDB replication with Gramine it fails with the following error

Relay log read failure: Could not parse relay log event entry. The possible reasons are: the master's binary log is corrupted (you can check this by running 'mysqlbinlog' on the binary log), the slave's relay log is corrupted (you can check this by running 'mysqlbinlog' on the relay log), a network problem, or a bug in the master's or slave's MariaDB code. If you want to check the master's binary log or slave's relay log, you will be able to know their names by issuing 'SHOW SLAVE STATUS' on this slave

Without Gramine all is good, but with Gramine we have this error. This is not also related to the sgx because we get the same error with gramine-direct as well.

steps to reproduce the error.

Clone the repository

git pull https://github.com/enclaive/enclaive-docker-mariadb-sgx.git

Build docker image

docker build -t gramine-mariadb:v2 .

then create 2 instances with this docker image and create replication between them.

docker run -it --name master --security-opt seccomp=/root/docker_seccomp_aug_2022.json gramine-mariadb:v2
docker run -it --name replica --security-opt seccomp=/root/docker_seccomp_aug_2022.json gramine-mariadb:v2

CHANGE MASTER TO MASTER_HOST='<master-ip>', MASTER_USER='<user>',MASTER_PASSWORD='<password>',MASTER_USE_GTID = slave_pos;

Once the replication is started all is good. But some time later it fails with this error. You can just run the following sysbench test and reproduce this replication error

sysbench oltp_read_only --mysql-host=127.0.0.1 --tables=10 --table_size=50000  --mysql-port=3306 --mysql-user=<user>--mysql-password="<password>" --mysql-db=sbtest  prepare

replication status

MariaDB [sbtest]> show slave status \G
*************************** 1. row ***************************
                Slave_IO_State:
                   Master_Host: 172.17.0.2
                   Master_User: root
                   Master_Port: 3306
                 Connect_Retry: 60
               Master_Log_File: ON.000001
           Read_Master_Log_Pos: 7060656
                Relay_Log_File: mysqld-relay-bin.000002
                 Relay_Log_Pos: 5487680
         Relay_Master_Log_File: ON.000001
              Slave_IO_Running: No
             Slave_SQL_Running: No
               Replicate_Do_DB:
           Replicate_Ignore_DB:
            Replicate_Do_Table:
        Replicate_Ignore_Table:
       Replicate_Wild_Do_Table:
   Replicate_Wild_Ignore_Table:
                    Last_Errno: 1594
                    Last_Error: Relay log read failure: Could not parse relay log event entry. The possible reasons are: the master's binary log is corrupted (you can check this by running 'mysqlbinlog' on the binary log), the slave's relay log is corrupted (you can check this by running 'mysqlbinlog' on the relay log), a network problem, or a bug in the master's or slave's MariaDB code. If you want to check the master's binary log or slave's relay log, you will be able to know their names by issuing 'SHOW SLAVE STATUS' on this slave.
                  Skip_Counter: 0
           Exec_Master_Log_Pos: 5487388
               Relay_Log_Space: 7061258
               Until_Condition: None
                Until_Log_File:
                 Until_Log_Pos: 0
            Master_SSL_Allowed: No
            Master_SSL_CA_File:
            Master_SSL_CA_Path:
               Master_SSL_Cert:
             Master_SSL_Cipher:
                Master_SSL_Key:
         Seconds_Behind_Master: NULL
 Master_SSL_Verify_Server_Cert: No
                 Last_IO_Errno: 1595
                 Last_IO_Error: Relay log write failure: could not queue event from master
                Last_SQL_Errno: 1594
                Last_SQL_Error: Relay log read failure: Could not parse relay log event entry. The possible reasons are: the master's binary log is corrupted (you can check this by running 'mysqlbinlog' on the binary log), the slave's relay log is corrupted (you can check this by running 'mysqlbinlog' on the relay log), a network problem, or a bug in the master's or slave's MariaDB code. If you want to check the master's binary log or slave's relay log, you will be able to know their names by issuing 'SHOW SLAVE STATUS' on this slave.
   Replicate_Ignore_Server_Ids:
              Master_Server_Id: 1
                Master_SSL_Crl:
            Master_SSL_Crlpath:
                    Using_Gtid: Slave_Pos
                   Gtid_IO_Pos: 0-1-33
       Replicate_Do_Domain_Ids:
   Replicate_Ignore_Domain_Ids:
                 Parallel_Mode: optimistic
                     SQL_Delay: 0
           SQL_Remaining_Delay: NULL
       Slave_SQL_Running_State:
              Slave_DDL_Groups: 14
Slave_Non_Transactional_Groups: 0
    Slave_Transactional_Groups: 17

When we checked the binary logs in the server we verified that they are healthy but the relay logs on the replica is broken. However, if we do the same without gramine it works without any issues.

if we check the relay logs with the following command we see they are corrupted.

 show relaylog events IN 'mysqld-relay-bin.000002';
ERROR 1220 (HY000): Error when executing command SHOW BINLOG EVENTS: Wrong offset or I/O error



 show relaylog events IN 'mysqld-relay-bin.000002' limit 10,3;
+-------------------------+------+------------+-----------+-------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| Log_name                | Pos  | Event_type | Server_id | End_log_pos | Info                                                                                                                                                                                                                                                                                                                                                      |
+-------------------------+------+------------+-----------+-------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| mysqld-relay-bin.000002 |  961 | Query      |         1 |         756 | create database test                                                                                                                                                                                                                                                                                                                                      |
| mysqld-relay-bin.000002 | 1048 | Gtid       |         1 |         798 | GTID 0-1-4                                                                                                                                                                                                                                                                                                                                                |
| mysqld-relay-bin.000002 | 1090 | Query      |         1 |        1207 | use `test`; CREATE TABLE test_table (
    id INT AUTO_INCREMENT PRIMARY KEY,
    name VARCHAR(255) NOT NULL,
    email VARCHAR(255) NOT NULL,
    phone VARCHAR(20),
    address VARCHAR(255),
    city VARCHAR(100),
    state VARCHAR(100),
    postal_code VARCHAR(20),
    country VARCHAR(100),
    created_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP
) |
+-------------------------+------+------------+-----------+-------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
3 rows in set (0.000 sec)