MySQL在线DDL工具之OAK


摘要:openarkkit工具包中oak-online-alter-table的使用

什么是OAK

openark工具包是一组由Python语言编写的用于MySQL的工具集。可以解决一些日常复杂或耗时的维护任务。
包括如下工具:
oak-apply-ri: apply referential integrity on two columns with parent-child relationship.
oak-block-account: block or release MySQL users accounts, disabling them or enabling them to login.
oak-chunk-update: perform long, non-blocking UPDATE/DELETE operation in auto managed small chunks.
oak-get-slave-lag: print slave replication lag and terminate with respective exit code.
oak-hook-general-log: hook up and filter general log entries based on entry type or execution plan criteria.
oak-kill-slow-queries: terminate long running queries.
oak-modify-charset: change the character set (and collation) of a textual column.
oak-online-alter-table: perform a non-blocking ALTER TABLE operation.
oak-prepare-shutdown: make for a fast and safe MySQL shutdown.
oak-purge-master-logs: purge master logs, depending on the state of replicating slaves.
oak-repeat-query: repeat query execution until some condition holds.
oak-security-audit: audit accounts, passwords, privileges and other security settings.
oak-show-limits: show AUTO_INCREMENT “free space”.
oak-show-replication-status: show how far behind are replicating slaves on a given master.

下载安装

下载:
[root@mysql1 src]# wget https://storage.googleapis.com/google-code-archive-downloads/v2/code.google.com/openarkkit/openark-kit-196.tar.gz
安装:
[root@mysql1 src]# tar -zxvf openark-kit-196.tar.gz
[root@mysql1 src]# cd openark-kit-196
[root@mysql1 openark-kit-196]# python setup.py install
查看安装好的系列工具集:
[root@mysql1 openark-kit-196]# ll /usr/local/bin/oak*
-rwxr-xr-x 1 root root 11020 Apr 20 10:27 /usr/local/bin/oak-apply-ri
-rwxr-xr-x 1 root root 10999 Apr 20 10:27 /usr/local/bin/oak-block-account
-rwxr-xr-x 1 root root 29490 Apr 20 10:27 /usr/local/bin/oak-chunk-update
-rwxr-xr-x 1 root root 5501 Apr 20 10:27 /usr/local/bin/oak-get-slave-lag
-rwxr-xr-x 1 root root 18181 Apr 20 10:27 /usr/local/bin/oak-hook-general-log
-rwxr-xr-x 1 root root 6314 Apr 20 10:27 /usr/local/bin/oak-kill-slow-queries
-rwxr-xr-x 1 root root 5630 Apr 20 10:27 /usr/local/bin/oak-modify-charset
-rwxr-xr-x 1 root root 39192 Apr 20 10:27 /usr/local/bin/oak-online-alter-table
-rwxr-xr-x 1 root root 7809 Apr 20 10:27 /usr/local/bin/oak-prepare-shutdown
-rwxr-xr-x 1 root root 15045 Apr 20 10:27 /usr/local/bin/oak-purge-master-logs
-rwxr-xr-x 1 root root 8371 Apr 20 10:27 /usr/local/bin/oak-repeat-query
-rwxr-xr-x 1 root root 19312 Apr 20 10:27 /usr/local/bin/oak-security-audit
-rwxr-xr-x 1 root root 5910 Apr 20 10:27 /usr/local/bin/oak-show-limits
-rwxr-xr-x 1 root root 8654 Apr 20 10:27 /usr/local/bin/oak-show-replication-status

操作流程

1、检查被操作的表是否符合如下条件:

  • 单列唯一索引(联合索引和联合主键是不可以的,会触发mysql的一个bug)
  • 没有外键
  • 没有触发器(有的话要先删除,做完DDL再加上)
    2、执行OAK命令
    3、数据一致性校验
    4、表切换
    5、删除触发器
    6、删除临时表

    在线DDL实操

    准备测试数据

    [root@mysql ~]# sysbench --test=/root/sysbench-0.5/sysbench/tests/db/insert.lua --oltp-table-size=1000000 --mysql-table-engine=innodb --mysql-user=root --mysql-passw
    ord=root123 --mysql-port=3306 --mysql-host=192.168.1.172 --mysql-db=jfedu --max-requests=0 --max-time=60 --oltp-tables-count=1 --report-interval=10 --num_threads=1 p
    repare
    sysbench 0.5: multi-threaded system evaluation benchmark
    Creating table 'sbtest1'...
    Inserting 1000000 records into 'sbtest1'
    Creating secondary indexes on 'sbtest1'...
    [root@mysql ~]#
    root@jfedu 01:20:08>show create table sbtest1;
    +---------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
    | Table | Create Table |
    +---------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
    | sbtest1 | CREATE TABLE `sbtest1` (
    `id` int(10) unsigned NOT NULL AUTO_INCREMENT,
    `k` int(10) unsigned NOT NULL DEFAULT '0',
    `c` char(120) NOT NULL DEFAULT '',
    `pad` char(60) NOT NULL DEFAULT '',
    PRIMARY KEY (`id`),
    KEY `k_1` (`k`)
    ) ENGINE=InnoDB AUTO_INCREMENT=1000001 DEFAULT CHARSET=utf8 MAX_ROWS=1000000 |
    +---------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
    1 row in set (0.02 sec)
    mysql> select count(*) from sbtest1;
    +----------+
    | count(*) |
    +----------+
    | 1060893 |
    +----------+
    1 row in set (0.16 sec)

检查是否符合在线DDL的条件

根据表结构可以看出是符合单列唯一索引,且无外键的。通过下面SQL查看结果显示是没触发器的:

root@jfedu 01:47:30>SELECT TRIGGER_SCHEMA,TRIGGER_NAME,EVENT_OBJECT_SCHEMA,EVENT_OBJECT_TABLE FROM information_schema.TRIGGERS WHERE event_object_schema= 'jfedu';
Empty set (0.03 sec)
root@jfedu 01:47:37>

执行OAK命令,给表加字段

[root@mysql1 ~]# oak-online-alter-table -u root --ask-pass -S /data/mysql/run/mysql.sock -d jfedu -t sbtest1 -g new_sbtest1 -a "add last_update_time timestamp NOT NULL DEFAULT CURRENT_TIMESTAMP ON UPDATE CURRENT_TIMESTAMP,add key last_update_time(last_update_time)" --sleep=300 --skip-delete-pass
OAK开跑后,开启一个会话向老表插入一条数据:
mysql> insert into sbtest1 values(20000000,2222222222,'22222222222','ccc');
Query OK, 1 row affected (0.02 sec)
OAK执行完毕后:
... ...
-- Copying range (1059000), (1060000), progress: 99%
-- + Will sleep for 0.3 seconds
-- - Reminder: altering jfedu.sbtest1: add last_update_time timestamp...
-- Copying range (1060000), (1060893), progress: 99%
-- + Will sleep for 0.3 seconds
-- Copying range 100% complete. Number of rows: 1060893
-- Ghost table creation completed. Note that triggers on jfedu.sbtest1 were not removed

可以看到只拷贝了1060893条数据,我们后来插入的数据有没有过去呢?

数据一致性校验

mysql> select count(*) from sbtest1 union all select count(*) from new_sbtest1;
+----------+
| count(*) |
+----------+
| 1060894 |
| 1060894 |
+----------+
2 rows in set (0.87 sec)
可以看到多了一条数据,说明我们后来插入的数据同步成功了。看看表结构:
mysql> desc sbtest1;
+-------+------------------+------+-----+---------+----------------+
| Field | Type | Null | Key | Default | Extra |
+-------+------------------+------+-----+---------+----------------+
| id | int(10) unsigned | NO | PRI | NULL | auto_increment |
| k | int(10) unsigned | NO | MUL | 0 | |
| c | char(120) | NO | | | |
| pad | char(60) | NO | | | |
+-------+------------------+------+-----+---------+----------------+
4 rows in set (0.00 sec)
mysql> desc new_sbtest1;
+------------------+------------------+------+-----+-------------------+-----------------------------+
| Field | Type | Null | Key | Default | Extra |
+------------------+------------------+------+-----+-------------------+-----------------------------+
| id | int(10) unsigned | NO | PRI | NULL | auto_increment |
| k | int(10) unsigned | NO | MUL | 0 | |
| c | char(120) | NO | | | |
| pad | char(60) | NO | | | |
| last_update_time | timestamp | NO | MUL | CURRENT_TIMESTAMP | on update CURRENT_TIMESTAMP |
+------------------+------------------+------+-----+-------------------+-----------------------------+
5 rows in set (0.00 sec)
可以看到我们加的字段已经出现在新表上面了。

验证一个疑问

用oak执行完ddl后,尚未rename的时候,在老表上做DML操作是否会同步到新表?
1、在老表插入数据

mysql> select count(*) from sbtest1 union all select count(*) from new_sbtest1;
+----------+
| count(*) |
+----------+
| 1060894 |
| 1060894 |
+----------+
2 rows in set (0.87 sec)
插入一条数据:
mysql> insert into sbtest1 values(20000001,3333333,'333333','ccc');
Query OK, 1 row affected (0.01 sec)
mysql> select count(*) from sbtest1 union all select count(*) from new_sbtest1;
+----------+
| count(*) |
+----------+
| 1060895 |
| 1060895 |
+----------+
2 rows in set (0.30 sec)
可以看到新表和旧表都新增了一条数据。
mysql> select sum(crc32(concat(ifnull(id,'NULL'),ifnull(k,'NULL')))) as sum from sbtest1 union all select sum(crc32(concat(ifnull(id,'NULL'),ifnull(k,'NULL')))) as sum from new_sbtest1;
+------------------+
| sum |
+------------------+
| 2278342522965511 |
| 2278342522965511 |
+------------------+
2 rows in set (0.67 sec)

2、在老表修改数据

root@jfedu 02:27:51>select * from sbtest1 limit 1;
+----+--------+-------------------------------------------------------------------------------------------------------------------------+-------------------------------------------------------------+
| id | k | c | pad |
+----+--------+-------------------------------------------------------------------------------------------------------------------------+-------------------------------------------------------------+
| 1 | 501885 | 08566691963-88624912351-16662227201-46648573979-64646226163-77505759394-75470094713-41097360717-15161106334-50535565977 | 63188288836-92351140030-06390587585-66802097351-49282961843 |
+----+--------+-------------------------------------------------------------------------------------------------------------------------+-------------------------------------------------------------+
1 row in set (0.01 sec)
root@jfedu 02:28:05>update sbtest1 set k=11111 where id=1;
Query OK, 1 row affected (0.01 sec)
Rows matched: 1 Changed: 1 Warnings: 0
root@jfedu 02:29:37>select * from sbtest1 where id=1;
+----+-------+-------------------------------------------------------------------------------------------------------------------------+-------------------------------------------------------------+
| id | k | c | pad |
+----+-------+-------------------------------------------------------------------------------------------------------------------------+-------------------------------------------------------------+
| 1 | 11111 | 08566691963-88624912351-16662227201-46648573979-64646226163-77505759394-75470094713-41097360717-15161106334-50535565977 | 63188288836-92351140030-06390587585-66802097351-49282961843 |
+----+-------+-------------------------------------------------------------------------------------------------------------------------+-------------------------------------------------------------+
1 row in set (0.00 sec)
root@jfedu 02:30:06>select * from new_sbtest1 where id=1;
+----+-------+-------------------------------------------------------------------------------------------------------------------------+-------------------------------------------------------------+---------------------+
| id | k | c | pad | last_update_time |
+----+-------+-------------------------------------------------------------------------------------------------------------------------+-------------------------------------------------------------+---------------------+
| 1 | 11111 | 08566691963-88624912351-16662227201-46648573979-64646226163-77505759394-75470094713-41097360717-15161106334-50535565977 | 63188288836-92351140030-06390587585-66802097351-49282961843 | 2017-04-20 14:28:47 |
+----+-------+-------------------------------------------------------------------------------------------------------------------------+-------------------------------------------------------------+---------------------+
1 row in set (0.00 sec)
root@jfedu 02:30:13>exit
Bye

可以看出在老表中的插入和更新操作都被同步到了新表。这是由OAK创建的三个触发器来实现的:

mysql> SELECT TRIGGER_SCHEMA,TRIGGER_NAME,EVENT_OBJECT_SCHEMA,EVENT_OBJECT_TABLE FROM information_schema.TRIGGERS WHERE event_object_schema= 'jfedu';
+----------------+----------------+---------------------+--------------------+
| TRIGGER_SCHEMA | TRIGGER_NAME | EVENT_OBJECT_SCHEMA | EVENT_OBJECT_TABLE |
+----------------+----------------+---------------------+--------------------+
| jfedu | sbtest1_AI_oak | jfedu | sbtest1 |
| jfedu | sbtest1_AU_oak | jfedu | sbtest1 |
| jfedu | sbtest1_AD_oak | jfedu | sbtest1 |
+----------------+----------------+---------------------+--------------------+
3 rows in set (0.00 sec)

看看这三个触发器的内容:

root@jfedu 02:26:24>show create trigger sbtest1_AI_oak;
+----------------+------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+----------------------+--------------------+
| Trigger | sql_mode | SQL Original Statement | character_set_client | collation_connection | Database Collation |
+----------------+------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+----------------------+--------------------+
| sbtest1_AI_oak | NO_ENGINE_SUBSTITUTION | CREATE DEFINER=`root`@`localhost` TRIGGER jfedu.sbtest1_AI_oak AFTER INSERT ON jfedu.sbtest1
FOR EACH ROW
REPLACE INTO jfedu.new_sbtest1 (`c`, `pad`, `k`, `id`) VALUES (NEW.`c`, NEW.`pad`, NEW.`k`, NEW.`id`) | utf8 | utf8_general_ci | utf8_general_ci |
+----------------+------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+----------------------+--------------------+
1 row in set (0.01 sec)
触发器sbtest1_AI_oak负责将老表的insert操作同步到新表
root@jfedu 02:26:41>show create trigger sbtest1_AU_oak;
+----------------+------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+----------------------+--------------------+
| Trigger | sql_mode | SQL Original Statement | character_set_client | collation_connection | Database Collation |
+----------------+------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+----------------------+--------------------+
| sbtest1_AU_oak | NO_ENGINE_SUBSTITUTION | CREATE DEFINER=`root`@`localhost` TRIGGER jfedu.sbtest1_AU_oak AFTER UPDATE ON jfedu.sbtest1
FOR EACH ROW
BEGIN
DELETE FROM jfedu.new_sbtest1 WHERE (id) = (OLD.id);
REPLACE INTO jfedu.new_sbtest1 (`c`, `pad`, `k`, `id`) VALUES (NEW.`c`, NEW.`pad`, NEW.`k`, NEW.`id`);
END | utf8 | utf8_general_ci | utf8_general_ci |
+----------------+------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+----------------------+--------------------+
1 row in set (0.00 sec)
触发器sbtest1_AU_oak负责将老表的update操作同步到新表
root@jfedu 02:27:22>show create trigger sbtest1_AD_oak;
+----------------+------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+----------------------+--------------------+
| Trigger | sql_mode | SQL Original Statement | character_set_client | collation_connection | Database Collation |
+----------------+------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+----------------------+--------------------+
| sbtest1_AD_oak | NO_ENGINE_SUBSTITUTION | CREATE DEFINER=`root`@`localhost` TRIGGER jfedu.sbtest1_AD_oak AFTER DELETE ON jfedu.sbtest1
FOR EACH ROW
DELETE FROM jfedu.new_sbtest1 WHERE (id) = (OLD.id) | utf8 | utf8_general_ci | utf8_general_ci |
+----------------+------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+----------------------+--------------------+
1 row in set (0.00 sec)
触发器sbtest1_AD_oak负责将老表的delete操作同步到新表

也就是说在我们rename切换表之前,对旧表的任何增、删、改操作都会同步到新表!

表切换

mysql> rename table sbtest1 to old_sbtest1,new_sbtest1 to sbtest1;
Query OK, 0 rows affected (0.09 sec)
mysql> drop trigger sbtest1_AI_oak;drop trigger sbtest1_AU_oak;drop trigger sbtest1_AD_oak;
Query OK, 0 rows affected (0.03 sec)
Query OK, 0 rows affected (0.01 sec)
Query OK, 0 rows affected (0.00 sec)
mysql> drop table old_sbtest1;
Query OK, 0 rows affected (0.08 sec)
mysql> desc sbtest1;
+------------------+------------------+------+-----+-------------------+-----------------------------+
| Field | Type | Null | Key | Default | Extra |
+------------------+------------------+------+-----+-------------------+-----------------------------+
| id | int(10) unsigned | NO | PRI | NULL | auto_increment |
| k | int(10) unsigned | NO | MUL | 0 | |
| c | char(120) | NO | | | |
| pad | char(60) | NO | | | |
| last_update_time | timestamp | NO | MUL | CURRENT_TIMESTAMP | on update CURRENT_TIMESTAMP |
+------------------+------------------+------+-----+-------------------+-----------------------------+
5 rows in set (0.01 sec)

在切换表之前,触发器一定不能删掉,否则数据就会不一致的!

遇到问题及解决

1、安装MySQLdb遇到的问题

[root@mysql1 openark-kit-196]# oak-online-alter-table --help
Traceback (most recent call last):
File "/usr/local/bin/oak-online-alter-table", line 21, in <module>
import MySQLdb
ImportError: No module named MySQLdb
[root@mysql1 openark-kit-196]# pip install MySQL-python
>>> import MySQLdb
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/usr/local/lib/python2.7/site-packages/MySQLdb/__init__.py", line 19, in <module>
import _mysql
ImportError: libperconaserverclient.so.18: cannot open shared object file: No such file or directory
>>> exit()
[root@mysql1 openark-kit-196]# find / -name "libperconaserverclient*"
没找到libperconaserverclient相关包,需要安装mysql-devel:
[root@mysql1 openark-kit-196]# yum -y install mysql-devel
[root@mysql1 openark-kit-196]# ln -s /opt/mysql/lib/libmysqlclient.so.18 /usr/lib64/libmysqlclient.so.18
[root@mysql1 openark-kit-196]# ln -s /usr/local/mysql/lib/libperconaserverclient_r.so.18 /usr/lib64
[root@mysql1 openark-kit-196]# ln -s /usr/local/mysql/lib/libperconaserverclient_r.so /usr/lib64
[root@mysql1 openark-kit-196]# ldconfig
[root@mysql1 openark-kit-196]# python
Python 2.7.10 (default, Jan 20 2016, 16:07:05)
[GCC 4.4.7 20120313 (Red Hat 4.4.7-16)] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import MySQLdb
>>> exit()
[root@mysql1 openark-kit-196]# oak-online-alter-table --help
Usage: oak-online-alter-table [options]
Options:
-h, --help show this help message and exit
-u USER, --user=USER MySQL user
-H HOST, --host=HOST MySQL host (default: localhost)
-p PASSWORD, --password=PASSWORD
MySQL password
--ask-pass Prompt for password
-P PORT, --port=PORT TCP/IP port (default: 3306)
-S SOCKET, --socket=SOCKET
MySQL socket file. Only applies when host is localhost
--defaults-file=DEFAULTS_FILE
Read from MySQL configuration file. Overrides all
other options
-d DATABASE, --database=DATABASE
Database name (required unless table is fully
qualified)
-t TABLE, --table=TABLE
Table to alter (optionally fully qualified)
-g GHOST, --ghost=GHOST
Table name to serve as ghost. This table will be
created and synchronized with the original table
-a ALTER_STATEMENT, --alter=ALTER_STATEMENT
Comma delimited ALTER statement details, excluding the
'ALTER TABLE t' itself
-c CHUNK_SIZE, --chunk-size=CHUNK_SIZE
Number of rows to act on in chunks. Default: 1000
-l, --lock-chunks Use LOCK TABLES for each chunk
-N, --skip-binlog Disable binary logging
-r MAX_LOCK_RETRIES, --max-lock-retries=MAX_LOCK_RETRIES
Maximum times to retry on deadlock or
lock_wait_timeout. (default: 10; 0 is unlimited)
--skip-delete-pass Do not execute the DELETE data pass
--sleep=SLEEP_MILLIS Number of milliseconds to sleep between chunks.
Default: 0
--sleep-ratio=SLEEP_RATIO
Ratio of sleep time to execution time. Default: 0
--cleanup Remove custom triggers, ghost table from possible
previous runs
-v, --verbose Print user friendly messages
-q, --quiet Quiet mode, do not verbose
[root@mysql1 openark-kit-196]#

2、用oak做Online DDL,如果rename那一步操作错了?不小心在从库上做rename了怎么恢复?(包含旧表删除和trigger删除)
可以使用oak在主库再做一遍,因为从库已经做过了因此要加一个参数–skip-binlog来关闭binlog写入。这样主库的操作就不会复制到从库,避免主从复制异常!

文章目录
  1. 1. 什么是OAK
  2. 2. 下载安装
  3. 3. 操作流程
  4. 4. 在线DDL实操
    1. 4.1. 准备测试数据
    2. 4.2. 检查是否符合在线DDL的条件
    3. 4.3. 执行OAK命令,给表加字段
    4. 4.4. 数据一致性校验
    5. 4.5. 验证一个疑问
    6. 4.6. 表切换
  5. 5. 遇到问题及解决
|