Skip to content

A Python script designed to read the MySQL General Log file, save certain queries based on its configuration and optionally run them on other specified databases.

License

Notifications You must be signed in to change notification settings

WiddleWabbit/sqlgeneral-monitor

Folders and files

NameName
Last commit message
Last commit date

Latest commit

 

History

37 Commits
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 

Repository files navigation

                                       ---           README        ---

This script is given a configuration and a sql general log file. The general log is then processed and certain sql commands issued (based on configuration) are written to one or more export files. Depending on configuration these commands can be automatically run on another database as well.
It supports the automatic flush of a log file through the --flush-log option allowing you to move the log file, flush the mysql log through the mysqladmin flush-logs command and to then process the moved file. This prevents the log file becoming too large as the previously processed file is overridden each time this option is specified. This option also allows you to just aim the script with cron at the mysql general log and not worry about stopping or deleting it.

Use the --help option for for information on available options and how to use it.

The intention of the script is to allow certain database changes to be run on another database while other database changes are ignored, all on a per database standard. 
For example an e-commerce website can automatically transfer orders etc to a development version of the site while new products, configuration changes etc can be automatically added to the production site.

USE THE monitor.py FILE TO RUN THE SCRIPT

Developed for:

    -   MySQL 5.6
    -   Python 2.6.6

Requires:

    -   enum34 1.6.6 (backported)
    -   MySQLdb 1.2.5


                                        ---           TODO         ---

        -   Feature
                -   Silent Running and Verbose
                -   Per Database Ignore certain SQL Commands otherwise recorded (CREATE TEMPORARY TABLE etc)

        -   conf.py
                -   Create a splitConfig function to replace the multiple repeat code instances.

        -   Tidyup
                -   Add required permission values to readme
                -   Update Lists in readme


                                      ---     LIST VARIABLES NOTES   ---

SQLLOG LIST
name: sqllog

Type: List of Lists
Indexes: Enum Specified with COL enum class
  [COL.Line][COL.String][COL.Type][COL.QueryNo][COL.User][COL.Save][COL.Query]
Formats:
  [Integer][String][String][String][String][Integer][String]
Example:
  [
  [1]["Full SQL Connect Statment with Query Number Etc"]["Connect"]["1"]["deoem"][0]["SQL Connect Statement Only"]
  [2]["Full SQL Query Statment with Query Number Etc"]["Query"]["1"]["deoem"][1]["SQL Query Statement Only"]
  [3]["Full SQL Quit Statment with Query Number Etc"]["Quit"]["1"]["deoem"][0]["SQL Quit Statement Only"]
  ]

CONFIGURATION LIST
name: configuration 

Type: Dictionary of Lists
Indexes: Enum Specified with CONFIG enum class
  'DatabaseUser' : [CONFIG.Database, [CONFIG.IgnoreTables], CONFIG.Filter],[CONFIG.Pass],[CONFIG.Run]
Formats:
  [String][[String][List of Strings][Integer][String][List of Strings]]
Example:
  [
  'user1' : [database1, [list, of, tables, to, ignore], 1, Password1, [databases, to, run, queries]]
  'user2' : [database2, [list, of, tables, to, ignore], 0, Password2, [databases, to, run, queries]]
  'user3' : [database3, [list, of, tables, to, ignore], 0, Password3, [databases, to, run, queries]]
  ]

RECORDABLE QUERY LIST
name: recordable

Type: List
Formats: Strings

Example:
['INSERT', 'UPDATE', 'DROP']

About

A Python script designed to read the MySQL General Log file, save certain queries based on its configuration and optionally run them on other specified databases.

Topics

Resources

License

Stars

Watchers

Forks

Packages

No packages published

Languages