Friday, December 16, 2016

Importing tables with deep rows into Aurora - turning off extended-insert

I was having trouble setting up an Aurora instance as a slave of a physical server. I wanted to get  all the data into Aurora in preparation for a failover to Aurora. The problem is I kept getting an error when dumping and restoring all the data:

ERROR 1041 (HY000) at line 29836: Out of memory; check if mysqld or some other process uses all available memory; if not, you may have to use 'ulimit' to allow mysqld to use more memory or you can add more swap space

I got this error when using both MySQL dump and restore and MyDumper and MyLoader. I found the table which was the culprit and it has a structure like this:

CREATE TABLE `my_data` (
  `Token` varchar(50) NOT NULL,
  `Data` mediumtext NOT NULL,
  PRIMARY KEY (`Token`),
) ENGINE=InnoDB DEFAULT CHARSET=latin1;

In the Data column was a large amount of JSON data.

I had already restored the table structure on the target and just had to truncate the table from the last failed import.

I stopped replication on one of the slaves so that I would have static log position to start the Aurora instances once this was done. First I did a dump and restore of all the structure for all tables. Then dumped and restore data for all the non-problematic tables.

Then did a separate dump and restore just for this table

Dumped the table like this:

mysqldump --defaults-file='my.cnf' -h<my server> --no-create-info --no-create-db --skip-opt --extended-insert=FALSE --quick MyDB my_data > My_DB_my_data.sql

Restore like this:

mysql --defaults-file='my.cnf' -h aurname.cluster-blablabla.rds.amazonaws.com MyDB < My_DB_my_data.sql

By turning off the extended-insert, I was able to get the data to load but it was MUCH slower because it was doing one insert at a time. However, the inserts were very large and this was the only way I was able to get the data to restore.

No comments:

Post a Comment