Feed: Planet MySQL
;
Author: Geoff Montee
;
A MariaDB Support customer recently asked how they could automatically drop old partitions after 6 months. MariaDB and MySQL do not have a mechanism to do this automatically out-of-the-box, but it is not too difficult to create a custom stored procedure and an event to call the procedure on the desired schedule. In this blog post, I will show one way to do that.
Partitioned table definition
For this demonstration, I’ll use a table definition based on one from MySQL’s documentation on range partitioning, with some minor changes:
DROP TABLE IF EXISTS db1.quarterly_report_status;
CREATE TABLE db1.quarterly_report_status (
report_id INT NOT NULL,
report_status VARCHAR(20) NOT NULL,
report_updated TIMESTAMP NOT NULL DEFAULT CURRENT_TIMESTAMP ON UPDATE CURRENT_TIMESTAMP
) ENGINE=InnoDB
PARTITION BY RANGE ( UNIX_TIMESTAMP(report_updated) ) (
PARTITION p201610 VALUES LESS THAN ( UNIX_TIMESTAMP('2016-11-01 00:00:00')),
PARTITION p201611 VALUES LESS THAN ( UNIX_TIMESTAMP('2016-12-01 00:00:00')),
PARTITION p201612 VALUES LESS THAN ( UNIX_TIMESTAMP('2017-01-01 00:00:00')),
PARTITION p201701 VALUES LESS THAN ( UNIX_TIMESTAMP('2017-02-01 00:00:00')),
PARTITION p201702 VALUES LESS THAN ( UNIX_TIMESTAMP('2017-03-01 00:00:00')),
PARTITION p201703 VALUES LESS THAN ( UNIX_TIMESTAMP('2017-04-01 00:00:00')),
PARTITION p201704 VALUES LESS THAN ( UNIX_TIMESTAMP('2017-05-01 00:00:00')),
PARTITION p201705 VALUES LESS THAN ( UNIX_TIMESTAMP('2017-06-01 00:00:00')),
PARTITION p201706 VALUES LESS THAN ( UNIX_TIMESTAMP('2017-07-01 00:00:00')),
PARTITION p201707 VALUES LESS THAN ( UNIX_TIMESTAMP('2017-08-01 00:00:00')),
PARTITION p201708 VALUES LESS THAN ( UNIX_TIMESTAMP('2017-09-01 00:00:00')),
PARTITION p_default VALUES LESS THAN (MAXVALUE)
);
The most significant change is that the partition naming scheme is based on the date. This will allow us to more easily determine which partitions to remove.
Stored procedure definition
The stored procedure itself contains some comments that explain what it does, so I will let the code speak for itself, for the most part:
DROP PROCEDURE IF EXISTS db1.drop_old_partitions; DELIMITER $$ CREATE PROCEDURE db1.drop_old_partitions(p_schema varchar(64), p_table varchar(64), p_months_to_keep int) LANGUAGE SQL NOT DETERMINISTIC SQL SECURITY INVOKER BEGIN DECLARE done INT DEFAULT FALSE; DECLARE current_partition_name varchar(64); -- We'll use this cursor later to get -- the list of partitions to drop. -- @last_partition_name_to_keep will be -- set later. DECLARE cur1 CURSOR FOR SELECT partition_name FROM information_schema.partitions WHERE TABLE_SCHEMA = p_schema AND TABLE_NAME = p_table AND PARTITION_NAME != 'p_default' AND PARTITION_NAMELet's try running the procedure as a test:
MariaDB [(none)]> SHOW CREATE TABLE db1.quarterly_report_status; +-------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Table | Create Table | +-------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | quarterly_report_status | CREATE TABLE `quarterly_report_status` ( `report_id` int(11) NOT NULL, `report_status` varchar(20) NOT NULL, `report_updated` timestamp NOT NULL DEFAULT CURRENT_TIMESTAMP ON UPDATE CURRENT_TIMESTAMP ) ENGINE=InnoDB DEFAULT CHARSET=latin1 /*!50100 PARTITION BY RANGE ( UNIX_TIMESTAMP(report_updated)) (PARTITION p201610 VALUES LESS THAN (1477972800) ENGINE = InnoDB, PARTITION p201611 VALUES LESS THAN (1480568400) ENGINE = InnoDB, PARTITION p201612 VALUES LESS THAN (1483246800) ENGINE = InnoDB, PARTITION p201701 VALUES LESS THAN (1485925200) ENGINE = InnoDB, PARTITION p201702 VALUES LESS THAN (1488344400) ENGINE = InnoDB, PARTITION p201703 VALUES LESS THAN (1491019200) ENGINE = InnoDB, PARTITION p201704 VALUES LESS THAN (1493611200) ENGINE = InnoDB, PARTITION p201705 VALUES LESS THAN (1496289600) ENGINE = InnoDB, PARTITION p201706 VALUES LESS THAN (1498881600) ENGINE = InnoDB, PARTITION p201707 VALUES LESS THAN (1501560000) ENGINE = InnoDB, PARTITION p201708 VALUES LESS THAN (1504238400) ENGINE = InnoDB, PARTITION p_default VALUES LESS THAN MAXVALUE ENGINE = InnoDB) */ | +-------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ 1 row in set (0.00 sec) MariaDB [(none)]> CALL db1.drop_old_partitions('db1', 'quarterly_report_status', 6); +--------------------------------------------------------------------------+ | CONCAT('Dropping all partitions before: ', @last_partition_name_to_keep) | +--------------------------------------------------------------------------+ | Dropping all partitions before: p201702 | +--------------------------------------------------------------------------+ 1 row in set (0.00 sec) +--------------------------------------------------------+ | CONCAT('Dropping partition: ', current_partition_name) | +--------------------------------------------------------+ | Dropping partition: p201610 | +--------------------------------------------------------+ 1 row in set (0.00 sec) +--------------------------------------------------------+ | CONCAT('Dropping partition: ', current_partition_name) | +--------------------------------------------------------+ | Dropping partition: p201611 | +--------------------------------------------------------+ 1 row in set (0.01 sec) +--------------------------------------------------------+ | CONCAT('Dropping partition: ', current_partition_name) | +--------------------------------------------------------+ | Dropping partition: p201612 | +--------------------------------------------------------+ 1 row in set (0.04 sec) +--------------------------------------------------------+ | CONCAT('Dropping partition: ', current_partition_name) | +--------------------------------------------------------+ | Dropping partition: p201701 | +--------------------------------------------------------+ 1 row in set (0.05 sec) Query OK, 0 rows affected (0.07 sec) MariaDB [(none)]> SHOW CREATE TABLE db1.quarterly_report_status; +-------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Table | Create Table | +-------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | quarterly_report_status | CREATE TABLE `quarterly_report_status` ( `report_id` int(11) NOT NULL, `report_status` varchar(20) NOT NULL, `report_updated` timestamp NOT NULL DEFAULT CURRENT_TIMESTAMP ON UPDATE CURRENT_TIMESTAMP ) ENGINE=InnoDB DEFAULT CHARSET=latin1 /*!50100 PARTITION BY RANGE ( UNIX_TIMESTAMP(report_updated)) (PARTITION p201702 VALUES LESS THAN (1488344400) ENGINE = InnoDB, PARTITION p201703 VALUES LESS THAN (1491019200) ENGINE = InnoDB, PARTITION p201704 VALUES LESS THAN (1493611200) ENGINE = InnoDB, PARTITION p201705 VALUES LESS THAN (1496289600) ENGINE = InnoDB, PARTITION p201706 VALUES LESS THAN (1498881600) ENGINE = InnoDB, PARTITION p201707 VALUES LESS THAN (1501560000) ENGINE = InnoDB, PARTITION p201708 VALUES LESS THAN (1504238400) ENGINE = InnoDB, PARTITION p_default VALUES LESS THAN MAXVALUE ENGINE = InnoDB) */ | +-------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ 1 row in set (0.00 sec)As you can see from the above output, several partitions were dropped.
Event definition
We want our stored procedure to run automatically every month, so we can use an event to do that.
Before testing the event, we need to do two things:
- We need to recreate the table with the original definition, so that it has all of the original partitions.
- We need to ensure that event_scheduler=ON is set, and if not, we need to set it.
MariaDB [(none)]> SHOW GLOBAL VARIABLES LIKE 'event_scheduler'; +-----------------+-------+ | Variable_name | Value | +-----------------+-------+ | event_scheduler | OFF | +-----------------+-------+ 1 row in set (0.00 sec) MariaDB [(none)]> SET GLOBAL event_scheduler=ON; Query OK, 0 rows affected (0.00 sec) MariaDB [(none)]> SHOW GLOBAL VARIABLES LIKE 'event_scheduler'; +-----------------+-------+ | Variable_name | Value | +-----------------+-------+ | event_scheduler | ON | +-----------------+-------+ 1 row in set (0.00 sec)
And then we can run the following:
CREATE EVENT db1.monthly_drop_old_partitions_event
ON SCHEDULE
EVERY 1 MONTH
STARTS NOW()
DO
CALL db1.drop_old_partitions('db1', 'quarterly_report_status', 6);
Since we specified STARTS NOW(), the event ran immediately. We can confirm this by looking at the table definition again:
MariaDB [(none)]> SHOW CREATE TABLE db1.quarterly_report_status; +-------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Table | Create Table | +-------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | quarterly_report_status | CREATE TABLE `quarterly_report_status` ( `report_id` int(11) NOT NULL, `report_status` varchar(20) NOT NULL, `report_updated` timestamp NOT NULL DEFAULT CURRENT_TIMESTAMP ON UPDATE CURRENT_TIMESTAMP ) ENGINE=InnoDB DEFAULT CHARSET=latin1 /*!50100 PARTITION BY RANGE ( UNIX_TIMESTAMP(report_updated)) (PARTITION p201702 VALUES LESS THAN (1488344400) ENGINE = InnoDB, PARTITION p201703 VALUES LESS THAN (1491019200) ENGINE = InnoDB, PARTITION p201704 VALUES LESS THAN (1493611200) ENGINE = InnoDB, PARTITION p201705 VALUES LESS THAN (1496289600) ENGINE = InnoDB, PARTITION p201706 VALUES LESS THAN (1498881600) ENGINE = InnoDB, PARTITION p201707 VALUES LESS THAN (1501560000) ENGINE = InnoDB, PARTITION p201708 VALUES LESS THAN (1504238400) ENGINE = InnoDB, PARTITION p_default VALUES LESS THAN MAXVALUE ENGINE = InnoDB) */ | +-------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ 1 row in set (0.00 sec)
Conclusion
Thanks to the flexibility of stored procedures and events, it is relatively easy to automatically drop old partitions in MySQL and MariaDB. Has anyone else implemented something like this?