Feed: Planet MySQL
;
Author: Geoff Montee
;
In a previous blog post, I showed how a DBA could configure MySQL or MariaDB to automatically drop old partitions. Some readers provided some feedback on some issues that they’ve run into while doing similar operations. Specifically:
- It can sometimes help to maintain an empty first partition when partitioning by dates, since partition pruning cannot always eliminate the first partition. Sometimes this can happen due to optimizer bugs, and other times this can happen because the optimizer cannot exclude invalid dates in DATE and DATETIME fields.
ALTER TABLE ... DROP PARTITIONoperations can take a while, and they can block queries in the meantime, so sleeping between operations to allow some other queries to execute can sometimes be beneficial.- It is also generally beneficial to automatically add new partitions.
In this blog post, I will discuss the first two items. I may discuss the third item in a future blog post.
Partitioned table definition
Our partitioned table is going to have a couple changes:
- It will contain an empty first partition called
p_first. - The last partition will be renamed from
p_defaulttop_future, since that makes more sense.
Here is our new definition:
DROP TABLE IF EXISTS db1.quarterly_report_status;
CREATE TABLE db1.quarterly_report_status (
report_id INT NOT NULL,
report_status VARCHAR(20) NOT NULL,
report_updated TIMESTAMP NOT NULL DEFAULT CURRENT_TIMESTAMP ON UPDATE CURRENT_TIMESTAMP
) ENGINE=InnoDB
PARTITION BY RANGE ( UNIX_TIMESTAMP(report_updated) ) (
PARTITION p_first VALUES LESS THAN ( UNIX_TIMESTAMP('2016-10-01 00:00:00')),
PARTITION p201610 VALUES LESS THAN ( UNIX_TIMESTAMP('2016-11-01 00:00:00')),
PARTITION p201611 VALUES LESS THAN ( UNIX_TIMESTAMP('2016-12-01 00:00:00')),
PARTITION p201612 VALUES LESS THAN ( UNIX_TIMESTAMP('2017-01-01 00:00:00')),
PARTITION p201701 VALUES LESS THAN ( UNIX_TIMESTAMP('2017-02-01 00:00:00')),
PARTITION p201702 VALUES LESS THAN ( UNIX_TIMESTAMP('2017-03-01 00:00:00')),
PARTITION p201703 VALUES LESS THAN ( UNIX_TIMESTAMP('2017-04-01 00:00:00')),
PARTITION p201704 VALUES LESS THAN ( UNIX_TIMESTAMP('2017-05-01 00:00:00')),
PARTITION p201705 VALUES LESS THAN ( UNIX_TIMESTAMP('2017-06-01 00:00:00')),
PARTITION p201706 VALUES LESS THAN ( UNIX_TIMESTAMP('2017-07-01 00:00:00')),
PARTITION p201707 VALUES LESS THAN ( UNIX_TIMESTAMP('2017-08-01 00:00:00')),
PARTITION p201708 VALUES LESS THAN ( UNIX_TIMESTAMP('2017-09-01 00:00:00')),
PARTITION p_future VALUES LESS THAN (MAXVALUE)
);
Stored procedure definition
Our stored procedure is also going to have a couple changes:
- It takes a new parameter called
p_seconds_to_sleepthat controls how many seconds the server sleeps betweenALTER TABLE ... DROP PARTITIONoperations. - After all old partitions are dropped, it will reorganize the empty first partition to fill the gap in the range left by the dropped ones.
Here is the new code (with comments inline):
DROP PROCEDURE IF EXISTS db1.drop_old_partitions;
DELIMITER $$
CREATE PROCEDURE db1.drop_old_partitions(p_schema varchar(64), p_table varchar(64), p_months_to_keep int, p_seconds_to_sleep int)
LANGUAGE SQL
NOT DETERMINISTIC
SQL SECURITY INVOKER
BEGIN
DECLARE done INT DEFAULT FALSE;
DECLARE current_partition_name varchar(64);
-- We'll use this cursor later to get
-- the list of partitions to drop.
-- @last_partition_name_to_keep will be
-- set later.
DECLARE cur1 CURSOR FOR
SELECT partition_name
FROM information_schema.partitions
WHERE TABLE_SCHEMA = p_schema
AND TABLE_NAME = p_table
AND PARTITION_NAME != 'p_first'
AND PARTITION_NAME != 'p_future'
AND PARTITION_NAME 0 THEN
SELECT CONCAT('Sleeping for ', p_seconds_to_sleep, ' seconds');
SELECT SLEEP(p_seconds_to_sleep);
END IF;
SELECT CONCAT('Dropping partition: ', current_partition_name);
-- First we build the ALTER TABLE query.
SET @schema = p_schema;
SET @table = p_table;
SET @partition = current_partition_name;
SET @q = 'SELECT CONCAT(''ALTER TABLE '', @schema, ''.'', @table, '' DROP PARTITION '', @partition) INTO @query';
PREPARE st FROM @q;
EXECUTE st;
DEALLOCATE PREPARE st;
-- And then we prepare and execute the ALTER TABLE query.
PREPARE st FROM @query;
EXECUTE st;
DEALLOCATE PREPARE st;
SET @first = FALSE;
END LOOP;
-- If no partitions were dropped, then we can also skip this.
IF ! @first THEN
-- Then we need to get the date of the new first partition.
-- We need the date in UNIX timestamp format.
SET @q = 'SELECT DATE_FORMAT(@last_month_to_keep, ''%Y-%m-01 00:00:00'') INTO @new_first_partition_date';
PREPARE st FROM @q;
EXECUTE st;
DEALLOCATE PREPARE st;
SELECT UNIX_TIMESTAMP(@new_first_partition_date) INTO @new_first_partition_ts;
-- We also need to get the date of the second partition
-- since the second partition is also needed for REORGANIZE PARTITION.
SET @q = 'SELECT DATE_ADD(@new_first_partition_date, INTERVAL 1 MONTH) INTO @second_partition_date';
PREPARE st FROM @q;
EXECUTE st;
DEALLOCATE PREPARE st;
SELECT UNIX_TIMESTAMP(@second_partition_date) INTO @second_partition_ts;
SELECT CONCAT('Reorganizing first and second partitions. first partition date = ', @new_first_partition_date, ', second partition date = ', @second_partition_date);
-- Then we build the ALTER TABLE query.
SET @schema = p_schema;
SET @table = p_table;
SET @q = 'SELECT CONCAT(''ALTER TABLE '', @schema, ''.'', @table, '' REORGANIZE PARTITION p_first, '', @last_partition_name_to_keep, '' INTO ( PARTITION p_first VALUES LESS THAN ( '', @new_first_partition_ts, '' ), PARTITION '', @last_partition_name_to_keep, '' VALUES LESS THAN ( '', @second_partition_ts, '' ) ) '') INTO @query';
PREPARE st FROM @q;
EXECUTE st;
DEALLOCATE PREPARE st;
-- And then we prepare and execute the ALTER TABLE query.
PREPARE st FROM @query;
EXECUTE st;
DEALLOCATE PREPARE st;
END IF;
END$$
DELIMITER ;
Let’s try running the new procedure:
MariaDB [db1]> SHOW CREATE TABLE db1.quarterly_report_status;
+-------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| Table | Create Table |
+-------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| quarterly_report_status | CREATE TABLE `quarterly_report_status` (
`report_id` int(11) NOT NULL,
`report_status` varchar(20) NOT NULL,
`report_updated` timestamp NOT NULL DEFAULT CURRENT_TIMESTAMP ON UPDATE CURRENT_TIMESTAMP
) ENGINE=InnoDB DEFAULT CHARSET=latin1
/*!50100 PARTITION BY RANGE ( UNIX_TIMESTAMP(report_updated))
(PARTITION p_first VALUES LESS THAN (1475294400) ENGINE = InnoDB,
PARTITION p201610 VALUES LESS THAN (1477972800) ENGINE = InnoDB,
PARTITION p201611 VALUES LESS THAN (1480568400) ENGINE = InnoDB,
PARTITION p201612 VALUES LESS THAN (1483246800) ENGINE = InnoDB,
PARTITION p201701 VALUES LESS THAN (1485925200) ENGINE = InnoDB,
PARTITION p201702 VALUES LESS THAN (1488344400) ENGINE = InnoDB,
PARTITION p201703 VALUES LESS THAN (1491019200) ENGINE = InnoDB,
PARTITION p201704 VALUES LESS THAN (1493611200) ENGINE = InnoDB,
PARTITION p201705 VALUES LESS THAN (1496289600) ENGINE = InnoDB,
PARTITION p201706 VALUES LESS THAN (1498881600) ENGINE = InnoDB,
PARTITION p201707 VALUES LESS THAN (1501560000) ENGINE = InnoDB,
PARTITION p201708 VALUES LESS THAN (1504238400) ENGINE = InnoDB,
PARTITION p_future VALUES LESS THAN MAXVALUE ENGINE = InnoDB) */ |
+-------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
1 row in set (0.00 sec)
MariaDB [db1]> CALL db1.drop_old_partitions('db1', 'quarterly_report_status', 6, 5);
+--------------------------------------------------------------------------+
| CONCAT('Dropping all partitions before: ', @last_partition_name_to_keep) |
+--------------------------------------------------------------------------+
| Dropping all partitions before: p201702 |
+--------------------------------------------------------------------------+
1 row in set (0.00 sec)
+--------------------------------------------------------+
| CONCAT('Dropping partition: ', current_partition_name) |
+--------------------------------------------------------+
| Dropping partition: p201610 |
+--------------------------------------------------------+
1 row in set (0.00 sec)
+---------------------------------------------------------+
| CONCAT('Sleeping for ', p_seconds_to_sleep, ' seconds') |
+---------------------------------------------------------+
| Sleeping for 5 seconds |
+---------------------------------------------------------+
1 row in set (0.02 sec)
+---------------------------+
| SLEEP(p_seconds_to_sleep) |
+---------------------------+
| 0 |
+---------------------------+
1 row in set (5.02 sec)
+--------------------------------------------------------+
| CONCAT('Dropping partition: ', current_partition_name) |
+--------------------------------------------------------+
| Dropping partition: p201611 |
+--------------------------------------------------------+
1 row in set (5.02 sec)
+---------------------------------------------------------+
| CONCAT('Sleeping for ', p_seconds_to_sleep, ' seconds') |
+---------------------------------------------------------+
| Sleeping for 5 seconds |
+---------------------------------------------------------+
1 row in set (5.03 sec)
+---------------------------+
| SLEEP(p_seconds_to_sleep) |
+---------------------------+
| 0 |
+---------------------------+
1 row in set (10.03 sec)
+--------------------------------------------------------+
| CONCAT('Dropping partition: ', current_partition_name) |
+--------------------------------------------------------+
| Dropping partition: p201612 |
+--------------------------------------------------------+
1 row in set (10.03 sec)
+---------------------------------------------------------+
| CONCAT('Sleeping for ', p_seconds_to_sleep, ' seconds') |
+---------------------------------------------------------+
| Sleeping for 5 seconds |
+---------------------------------------------------------+
1 row in set (10.05 sec)
+---------------------------+
| SLEEP(p_seconds_to_sleep) |
+---------------------------+
| 0 |
+---------------------------+
1 row in set (15.05 sec)
+--------------------------------------------------------+
| CONCAT('Dropping partition: ', current_partition_name) |
+--------------------------------------------------------+
| Dropping partition: p201701 |
+--------------------------------------------------------+
1 row in set (15.05 sec)
+--------------------------------------------------------------------------------------------------------------------------------------------------------------+
| CONCAT('Reorganizing first and second partitions. first partition date = ', @new_first_partition_date, ', second partition date = ', @second_partition_date) |
+--------------------------------------------------------------------------------------------------------------------------------------------------------------+
| Reorganizing first and second partitions. first partition date = 2017-02-01 00:00:00, second partition date = 2017-03-01 00:00:00 |
+--------------------------------------------------------------------------------------------------------------------------------------------------------------+
1 row in set (15.06 sec)
Query OK, 0 rows affected (15.11 sec)
MariaDB [db1]> SHOW CREATE TABLE db1.quarterly_report_status;
+-------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| Table | Create Table |
+-------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| quarterly_report_status | CREATE TABLE `quarterly_report_status` (
`report_id` int(11) NOT NULL,
`report_status` varchar(20) NOT NULL,
`report_updated` timestamp NOT NULL DEFAULT CURRENT_TIMESTAMP ON UPDATE CURRENT_TIMESTAMP
) ENGINE=InnoDB DEFAULT CHARSET=latin1
/*!50100 PARTITION BY RANGE ( UNIX_TIMESTAMP(report_updated))
(PARTITION p_first VALUES LESS THAN (1485925200) ENGINE = InnoDB,
PARTITION p201702 VALUES LESS THAN (1488344400) ENGINE = InnoDB,
PARTITION p201703 VALUES LESS THAN (1491019200) ENGINE = InnoDB,
PARTITION p201704 VALUES LESS THAN (1493611200) ENGINE = InnoDB,
PARTITION p201705 VALUES LESS THAN (1496289600) ENGINE = InnoDB,
PARTITION p201706 VALUES LESS THAN (1498881600) ENGINE = InnoDB,
PARTITION p201707 VALUES LESS THAN (1501560000) ENGINE = InnoDB,
PARTITION p201708 VALUES LESS THAN (1504238400) ENGINE = InnoDB,
PARTITION p_future VALUES LESS THAN MAXVALUE ENGINE = InnoDB) */ |
+-------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
1 row in set (0.01 sec)
We can see that our changes seem to be working as expected. In addition to old partitions being dropped, we can also see that p_first‘s date range was updated.
Event definition
Our event definition only needs one minor change: the addition of the new parameter for our stored procedure. This is fairly straight forward. If we want it to sleep for 5 seconds between operations, then the new definition would be:
DROP EVENT db1.monthly_drop_old_partitions_event;
CREATE EVENT db1.monthly_drop_old_partitions_event
ON SCHEDULE
EVERY 1 MONTH
STARTS NOW()
DO
CALL db1.drop_old_partitions('db1', 'quarterly_report_status', 6, 5);
Conclusion
I enjoy the challenge of writing interesting stored procedures like this, so I will probably eventually write a stored procedure that will also automatically add new partitions. In the mean time, I would love to hear if anyone else can think of any other improvements.