I was having trouble setting up an Aurora instance as a slave of a physical server. I wanted to get all the data into Aurora in preparation for a failover to Aurora. The problem is I kept getting an error when dumping and restoring all the data:
ERROR 1041 (HY000) at line 29836: Out of memory; check if mysqld or some other process uses all available memory; if not, you may have to use 'ulimit' to allow mysqld to use more memory or you can add more swap space
I got this error when using both MySQL dump and restore and MyDumper and MyLoader. I found the table which was the culprit and it has a structure like this:
CREATE TABLE `my_data` (
`Token` varchar(50) NOT NULL,
`Data` mediumtext NOT NULL,
PRIMARY KEY (`Token`),
) ENGINE=InnoDB DEFAULT CHARSET=latin1;
In the Data column was a large amount of JSON data.
I had already restored the table structure on the target and just had to truncate the table from the last failed import.
I stopped replication on one of the slaves so that I would have static log position to start the Aurora instances once this was done. First I did a dump and restore of all the structure for all tables. Then dumped and restore data for all the non-problematic tables.
Then did a separate dump and restore just for this table
Dumped the table like this:
mysqldump --defaults-file='my.cnf' -h<my server> --no-create-info --no-create-db --skip-opt --extended-insert=FALSE --quick MyDB my_data > My_DB_my_data.sql
Restore like this:
mysql --defaults-file='my.cnf' -h aurname.cluster-blablabla.rds.amazonaws.com MyDB < My_DB_my_data.sql
By turning off the extended-insert, I was able to get the data to load but it was MUCH slower because it was doing one insert at a time. However, the inserts were very large and this was the only way I was able to get the data to restore.
No comments:
Post a Comment