Wednesday, July 27, 2016

Finding pesky utf8 data in your latin1 tables

For whatever reason, sometimes you get utf8 data written to a table/column that doesn't support it. It might get double encoded, turn into mojibake (garbled text), etc. These might break your application or cause weirdness. Finding these values can be a pain. Here is a way to do.

CREATE TABLE `test` (
  `id` int(10) NOT NULL AUTO_INCREMENT,
  `name` varchar(100) DEFAULT NULL,
  PRIMARY KEY (`id`)
) ENGINE=InnoDB DEFAULT CHARSET=latin1;

insert into `test` (name) values ('x'), (unhex('c2a3'));


SELECT
name,
length(`name`) as ln,
char_length(`name`) as cl,
length(CONVERT(`name` USING utf8)) as u_ln,
char_length(CONVERT(`name` USING utf8)) as u_cl,
hex(name) as hx,
CONVERT(`name` USING utf8) as u_nm,
hex(CONVERT(`name` USING utf8)) as h_nm
FROM test;

After running this query you will notice that the length is the same for latin characters when it counted using length and still the same after converting to utf8. However, for utf8 data you will notice that the length changes after being converted to utf8. This makes it easy to identify where you have utf8 data. 



With this understanding, you can add a WHERE clause to only show the rows where there are differences like this:

SELECT
name,
length(`name`) as ln,
char_length(`name`) as cl,
length(CONVERT(`name` USING utf8)) as u_ln,
char_length(CONVERT(`name` USING utf8)) as u_cl,
hex(name) as hx,
CONVERT(`name` USING utf8) as u_nm,
hex(CONVERT(`name` USING utf8)) as h_nm
FROM test
WHERE length(`name`) <> length(CONVERT(`name` USING utf8));


Wednesday, July 20, 2016

TokuDB vs InnoDB

I had heard a great deal about TokuDB the last couple years and I've been dying to get a good use case for it. Ever since Percona bought TokuDB and included it in their distribution, I've been excited to test it.

I have a table that stores millions of rows and hundreds of thousands to a million more get added every day. I thought it would be a good candidate for TokuDB. As a test, I installed Percona Server 5.6.25 and then installed the TokuDB plugins. Everything was working fine up to that point. My test VM has 4 GB of RAM and 4 CPU. I left all the default TokuDB settings and let it automatically take 2 GB RAM (50% of RAM) for the TokuDB buffer pool. I changed all the tables from InnoDB to TokuDB and then lowered the InnoDB buffer pool. I'm no expert in TokuDB tuning so I wasn't really sure what to change to make it perform better.

My process for loading data into the large table involves collecting stats from thousands of data sources and then importing all those data sources into thousands of tiny staging tables and then merging all that data into a single table. Every night when my job to import that data into the staging tables would kick off, the mysql test instance with TokuDB would use up all the RAM and then crash. When I would re-start the instance, the TokuDB recovery would take several minutes and then it would happen again. I didn't have a lot of time to figure out what was going so I switched all my tables back to InnoDB and re-ran the tests and didn't have any problems. I thought TokuDB would perform better "out of the box" and with the default settings but not so.

I've read several of Baron Schwartz's Blog entries and he praises InnoDB for how reliable it is and how well it works. At this point, I don't know what I need to do to make TokuDB work better but it appears that creating thousands of tiny tables (each with only a few hundred or a few thousand rows) that are all TokuDB storage engine isn't a good use case. It seems the frequent dropping and creating of thousands of tiny TokuDB tables might cause RAM problems. I wish I had more time to dig deeper.

Friday, July 15, 2016

Killing lots of linux processes from the command line

I run a lot of bash scripts from the crontab to automate database tasks. Sometimes my bash scrips get carried away and don't finish. When this happens I need to kill them. I use pkill.

How to kill processes from the command line that match a patters:

pkill -f my_pattern
Thank you to this post:
http://stackoverflow.com/questions/8987037/how-to-kill-all-processes-with-a-given-partial-name

Wednesday, July 6, 2016

Two ways to convert to utf8

I'm in the middle of a project to converts lots of legacy tables from latin1 to utf8. These are two ways of converting specific tables to utf8. These commands are the same for utf8mb4, just change the CHARACTER SET and COLLATE for utf8mb4.

Option 1:

       

-- Change the schema default to utf8

ALTER DATABASE databaseA CHARACTER SET utf8 COLLATE utf8_unicode_ci;



-- Convert the defaults for the table to utf8

ALTER TABLE tableA CHARACTER SET = utf8, COLLATE = utf8_unicode_ci;



-- Convert specific column(s) to utf8

ALTER TABLE tableA MODIFY value VARCHAR(40) CHARACTER SET utf8 COLLATE utf8_unicode_ci;



       
 

What is good about this option is it will preserve other character sets on the same table. Maybe you need to keep latin1 on a specific column or you want to have a case sensitive column that uses some special collation, doing the change like this will not touch those other columns.

Option 2:

       

-- Change the schema default to utf8

ALTER DATABASE databaseA CHARACTER SET utf8 COLLATE utf8_unicode_ci;



-- Convert the table and all columns in it to utf8

ALTER TABLE tableA CONVERT TO CHARACTER SET utf8 COLLATE utf8_unicode_ci;


       
 

This option converts all columns to utf8. This is nice because it will catch all columns so you don't have to do each one separately. However, if you wanted to keep a specific column a specific collation then you would want to use Option 1.


For an excellent primer on utf8 and MySQL this is a great source:

http://mysql.rjweb.org/doc.php/charcoll

Tuesday, June 28, 2016

MySQL IF EXISTS - example removing users

It wasn't until MySQL 5.7 where the CREATE USER IF NOT EXISTS and DROP USER IF EXISTS syntax became available. Previous to that you could use SELECT IF (EXISTS... commands to do the same thing.

Here is an example of using SELECT IF (EXISTS... to drop a user.

-- Step 1: First create two test users with different ip ranges:

CREATE USER 'test'@'%' IDENTIFIED BY 'blabla';
CREATE USER 'test'@'10.%' IDENTIFIED BY 'blabla';


-- Step 2: Next set the user you want to drop:

SET @user = 'test';
SELECT (SELECT host
FROM mysql.user WHERE user = @user LIMIT 1) into @ip_range;

-- Step 3: Confirm variables have values

SELECT concat('\'',@user,'\'@\'',@ip_range,'\'');

-- Step 4: Run the SELECT IF (EXISTS and then run the prepared statement

SELECT IF (EXISTS(
         SELECT DISTINCT user
         FROM mysql.user
         WHERE user = @user )
         ,concat('drop user \'',@user,'\'@\'',@ip_range,'\';')
         ,concat('select \'user does not exist: ',@user,'\';')) into @a;
     

PREPARE stmt1 FROM @a;
EXECUTE stmt1;
DEALLOCATE PREPARE stmt1;

-- Step 5: Check and see if there are additional users with the same name but different IP range, if so repeat

SELECT IF (EXISTS(
         SELECT DISTINCT user
         FROM mysql.user
         WHERE user = @user )
         ,concat('Another user called \'',@user,'\'',' still exists.')
         ,concat('User \'',@user,'\' no longer exists.')) into @b;


SELECT @b;


Here are some actual examples I've written:

-- Clean up monitor_user user which doesn't have host value as %
SET sql_log_bin = 0;
SET @user = 'monitor_user';
SELECT IFNULL((SELECT host
FROM mysql.user WHERE user = @user AND host <> '%' LIMIT 1),'HOST_DOES_NOT_EXISTS') into @ip_range;
SELECT concat('\'',@user,'\'@\'',@ip_range,'\'');
SELECT IF (EXISTS(
         SELECT DISTINCT user
         FROM mysql.user
         WHERE user = @user
         AND host = @ip_range)
         ,concat('drop user \'',@user,'\'@\'',@ip_range,'\';')
         ,concat('select \'user does not exist: ',@user,'@',@ip_range,'\';')) into @a;
         SELECT @a;
PREPARE stmt1 FROM @a;
EXECUTE stmt1;
DEALLOCATE PREPARE stmt1;
SET sql_log_bin = 1;

-- Clean up root user that has the host value which is the same as the hostname
SET sql_log_bin = 0;
SET @user = 'root';
SELECT IFNULL((SELECT host
FROM mysql.user WHERE user = @user AND host <> '%' AND host = @@hostname LIMIT 1),'HOST_DOES_NOT_EXISTS')  into @ip_range;
SELECT concat('\'',@user,'\'@\'',@ip_range,'\'');
SELECT IF (EXISTS(
         SELECT DISTINCT user
         FROM mysql.user
         WHERE user = @user
          AND host = @ip_range)
         ,concat('drop user \'',@user,'\'@\'',@ip_range,'\';')
         ,concat('select \'user does not exist: ',@user,'\';')) into @a;
         SELECT @a;
PREPARE stmt1 FROM @a;
EXECUTE stmt1;
DEALLOCATE PREPARE stmt1;
SET sql_log_bin = 1;

-- Remove root user that has host value from previous co-master
SET sql_log_bin = 0;

SELECT (SELECT host
FROM mysql.user
WHERE host NOT IN ('%','localhost','::1','127.0.0.1')
AND host LIKE CONCAT(LEFT(host, LENGTH(host) - 1),'%') LIMIT 1) into @old_co_master;
SELECT @old_co_master;
SET @user = 'root';
SELECT IFNULL((SELECT host
FROM mysql.user WHERE user = @user AND host <> '%' AND host = @old_co_master LIMIT 1),'HOST_DOES_NOT_EXISTS')  into @ip_range;
SELECT concat('\'',@user,'\'@\'',@ip_range,'\'');
SELECT IF (EXISTS(
         SELECT DISTINCT user
         FROM mysql.user
         WHERE user = @user
          AND host = @ip_range)
         ,concat('drop user \'',@user,'\'@\'',@ip_range,'\';')
         ,concat('select \'user does not exist: ',@user,'\';')) into @a;
         SELECT @a;
PREPARE stmt1 FROM @a;
EXECUTE stmt1;
DEALLOCATE PREPARE stmt1;

SET sql_log_bin = 1;


Thursday, June 23, 2016

Concatenate in bash the output of two commands without newline character

I'm working on a bash script and I needed to import some data that I've compressed. However, the tables could also have foreign key constraints that I want ignored. So as I'm importing the dump files, I add "SET FOREIGN_KEY_CHECKS=0" to the file like this:

       { echo "SET FOREIGN_KEY_CHECKS=0;" ; gunzip < my_dump_file.sql.gz ; } | mysql -u${USERNAME} -p${PASSWORD} -h${SERVER} ${DATABASE} 

Found a good example on stackoverflow which taught me how to do this:

Reference:

http://stackoverflow.com/questions/20871534/concatenate-in-bash-the-output-of-two-commands-without-newline-character

Question:

What I need:
Suppose I have two commands, A and B, each of which returns a single-line string (i.e., a string with no newline character, except possibly 1 at the very end). I need a command (or sequence of piped commands) C that concatenates the output of commands A and B on the same line and inserts 1 space character between them.

Answer:

You can use tr:
{ echo "The quick"; echo "brown fox"; } | tr "\n" " "
OR using sed:
{ echo "The quick"; echo "brown fox"; } | sed -e 'N;s/\n/ /'

OUTPUT:

The quick brown fox 




Tuesday, June 14, 2016

Exporting MySQL results from remote instance to local file system

Why does exporting to a CSV locally from a remote MySQL server have to be so hard?

If you are running MySQL on RDS or don't have access to the file system on the remote server you cannot use the recommend approach to create a CSV file. This would be the recommend way...

SELECT order_id,product_name,qty
FROM orders
WHERE foo = 'bar'
INTO OUTFILE '/tmp/orders.csv'
FIELDS TERMINATED BY ','
ENCLOSED BY '"'
LINES TERMINATED BY '\n';

I like all the creative ways people on this post came up with to get over this:

http://stackoverflow.com/questions/356578/how-to-output-mysql-query-results-in-csv-format

If you know your data doesn't have commas or tabs then many of these solutions will work just fine. Such as this Example 1. Example 2 seems to account for even having commas or tabs in your data. But extra quotations in the data might be problematic.

Example 1:
mysql --user=wibble --password wobble -B -e "select * from vehicle_categories;" | sed "s/'/\'/;s/\t/\",\"/g;s/^/\"/;s/$/\"/;s/\n//g" > vehicle_categories.csv

Regex Explanation:
  • s/// means substitute what's between the first // with what's between the second //
  • the "g" at the end is a modifier that means "all instance, not just first"
  • ^ (in this context) means beginning of line
  • $ (in this context) means end of line
So, putting it all together:
s/'/\'/          replace ' with \'
s/\t/\",\"/g     replace all \t (tab) with ","
s/^/\"/          at the beginning of the line place a "
s/$/\"/          at the end of the line place a "
s/\n//g          replace all \n (newline) with nothing

Example 2:

select concat_ws(',',
    concat('"', replace(field1, '"', '""'), '"'),
    concat('"', replace(field2, '"', '""'), '"'),
    concat('"', replace(field3, '"', '""'), '"'))

from your_table where etc;
Explanation:
  1. Replace " with "" in each field --> replace(field1, '"', '""')
  2. Surround each result in quotation marks --> concat('"', result1, '"')
  3. Place a comma between each quoted result --> concat_ws(',', quoted1, quoted2, ...)

I liked the Example 2 which I have listed above. I've created a script that dynamically generates that type of SQL and then I execute that dynamically generated SQL.

My script runs three commands like this:

# Generate the header
mysql -uUSERNAME -pPASSWORD -hHOSTNAME  -s -e "(SELECT GROUP_CONCAT(CONCAT('\"',COLUMN_NAME,'\"'),'' '' SEPARATOR ',')
        FROM INFORMATION_SCHEMA.COLUMNS c WHERE TABLE_NAME = 'MY_TABLE_NAME' AND TABLE_SCHEMA = 'MY_DATABASE_NAME');" > my_report.csv

# Create SQL that will be used to get the data
mysql -uUSERNAME -pPASSWORD -hHOSTNAME -s -e "SET SESSION group_concat_max_len = 1000000;
        (SELECT CONCAT('SELECT CONCAT_WS(\',\', ', GROUP_CONCAT('CONCAT(\'\"\', REPLACE(',column_name, ',\'\"\', \'\"\"\'), \'\"\')'), ') FROM MY_TABLE_NAME')
        FROM information_schema.columns c
        WHERE c.table_schema = MY_DATABASE_NAME
        AND c.TABLE_NAME = 'MY_TABLE_NAME');" > dynamically_generated_code.sql

# Run the above SQL to get the data
mysql -uUSERNAME -pPASSWORD -hHOSTNAME  -s MY_DATABASE_NAME < dynamically_generated_code.sql >> my_report.csv


Explanation:

  1. First query puts the header values for the CSV file and writes them to the CSV file
  2. Second query creates the SQL to pull the data so that it will be in comma separated value format.
  3. The third query executes the SQL that was generated from step 2 and pulls the data for the CSV file and appends it to the file created in step one.