I had heard a great deal about TokuDB the last couple years and I've been dying to get a good use case for it. Ever since Percona bought TokuDB and included it in their distribution, I've been excited to test it.
I have a table that stores millions of rows and hundreds of thousands to a million more get added every day. I thought it would be a good candidate for TokuDB. As a test, I installed Percona Server 5.6.25 and then installed the TokuDB plugins. Everything was working fine up to that point. My test VM has 4 GB of RAM and 4 CPU. I left all the default TokuDB settings and let it automatically take 2 GB RAM (50% of RAM) for the TokuDB buffer pool. I changed all the tables from InnoDB to TokuDB and then lowered the InnoDB buffer pool. I'm no expert in TokuDB tuning so I wasn't really sure what to change to make it perform better.
My process for loading data into the large table involves collecting stats from thousands of data sources and then importing all those data sources into thousands of tiny staging tables and then merging all that data into a single table. Every night when my job to import that data into the staging tables would kick off, the mysql test instance with TokuDB would use up all the RAM and then crash. When I would re-start the instance, the TokuDB recovery would take several minutes and then it would happen again. I didn't have a lot of time to figure out what was going so I switched all my tables back to InnoDB and re-ran the tests and didn't have any problems. I thought TokuDB would perform better "out of the box" and with the default settings but not so.
I've read several of Baron Schwartz's Blog entries and he praises InnoDB for how reliable it is and how well it works. At this point, I don't know what I need to do to make TokuDB work better but it appears that creating thousands of tiny tables (each with only a few hundred or a few thousand rows) that are all TokuDB storage engine isn't a good use case. It seems the frequent dropping and creating of thousands of tiny TokuDB tables might cause RAM problems. I wish I had more time to dig deeper.
No comments:
Post a Comment