Having given it much careful consideration, I’ve decided that the wisest way to survive the current economic problems is by blowing my savings on a year long holiday in Italy. Wait, did I say holiday? Not really. I’m still a Sun employee, and I’m still going to be active in the MySQL community. My dear support customers just won’t be seeing me around for a while.

I’m looking forward to having time to write more extensively about some of the cool things we’re doing, and what’s going on in the community at large. If there’s anything you’d like to hear about (either expanding on my previous posts, or a completely new topic), please let me know.

Another thing I’d like to do, if there are any interested parties, is to see how companies are using MySQL in their part of the world. So, if you don’t mind showing off what you’re doing and having me write a little bit about it, feel free to drop me a line. All of my current contact details can be found on my contact page. I’m going to be primarily based in Milan, but I’ll be looking to travel around the rest of Europe at some point, so I’d be more than happy to stop by and see you if the opportunity arises.

Even better, it’s fast. How fast? Well, the good people at Oki Labs, apart from having implemented several new GIS functions for MySQL, have done some benchmarking, and it’s looking good. If you’ll excuse the cliched comparison to Postgres, here are the response times (seconds) of MySQL GIS vs. PostGIS in Oki’s test:

Source: http://www.osgeo.jp/wordpress/wp-content/uploads/2008/11/foss4g2008_okumura.pdf

If you’re interested in checking it out, the source tree (regularly merged with MySQL 5.1) is available here. Have a look at Giuseppe’s guide to running a Bazaar export in MySQL Sandbox.

I, for one, am 100% behind this. I’ve always been a big fan of community foundations being a focus point for development efforts, they work well to bring everyone together, and to provide a sensible foundation to help avoid much of the uncertainty that seems to spring up around MySQL. I certainly hope that the ODA is able to do the same.

Though I do have one question, how does the ODA plan on handling competing members? If you have two companies offering the same service in the same market, which one will the ODA recommend? Monty specifically says that “all companies that are joining the Alliance should bring something to the table”, but it’s a bit difficult to bring something new when there are already several large players in the MySQL market.

I shall certainly be watching the progress of this alliance with great interest, it has the potential to turn the MySQL Community into a large driving force for development and change.

The press release is available here, Monty has written some interesting thoughts about it here.

mysql> SELECT * FROM foo;
ERROR 1030 (HY000): Got error 1 from storage engine

This error isn’t really helpful. The problem is, the Federated engine only checks that the remote table structure is correct when it initially connects. Once it has connected, no more checks. When you restart the server, you get a much more helpful message:

mysql SELECT * FROM foo;
ERROR 1431 (HY000): The foreign data source you are trying to reference does not exist. Data source error:  error: 1054  'Unknown column 'b' in 'field list''

Also, keep your eye on the FederatedX project. It’s still under development, but will hopefully upgrade the Federated engine to being useful again.

What does this have to do with MySQL? Well, over the past few weeks there have been a bunch of conspiracy theories bouncing around. There are various topics, but the two favourite at the moment happen to be Oracle’s plans for MySQL, and the licensing of the MySQL documentation. There has been a long history of conspiracies surrounding MySQL, from Oracle’s original purchase of InnoDB, to our decision to create the Enterprise edition of the server, through to our long and bumpy release cycle.

Now, don’t get me wrong. I’m not making any calls to stifle discussion, I’m a big fan of community input. I was a member of the community before I joined MySQL, and I like to think that I still am. But I would like it if we could at least think about conspiracy theories before posting about them. We’re all people here at MySQL, we have evenings and weekends and lives just like you. Some of us are crazy enough to do silly things like jumping out of aeroplanes. We’re not out to get you, and we’re certainly not planning on turning into some sort of faceless corporate stereotype. We’re here to do what we love, creating and supporting a really good product.

Oh, and how do you know this isn’t some corporate play to make us seem human? Well, it’s 9:30pm on a Sunday night here, I’m yet to find a company who could pay me well enough to be shilling for them. But MySQL happens to be a group of people I like enough to defend them on my own time.

Okay, so that’s a pretty strong statement, but I’m genuinely happy with it. 5.1 has fixed my main problem with 5.0, in that the EER diagram mode was horribly slow to render, now it’s all nice and smooth. The ability to easily visualise tables and their relationships makes design very simple.

In fact, I really only have one (minor) complaint, the ability to export without foreign keys would be nice. Sometimes you just don’t want to deal with the performance hit.

That’s about it. Go and download the OSS edition for free now, have a play around. Make it your Friday afternoon experiment. I promise you’ll like it.

shell> mysqldump -u root -p --databases db1 db2 ... > backup.sql

The problem is, what happens when you want to backup the permissions associated with these databases? Well, here are a few queries to help you out.

-- Grab the users with global permissions, 
-- with permissions to the databases you want, 
-- and tables/stored procedures in it.
mysql> SELECT u.* INTO OUTFILE '/tmp/user.txt'
	FIELDS TERMINATED BY ',' OPTIONALLY ENCLOSED BY '"' ESCAPED BY '\\'
	LINES TERMINATED BY '\n'
	FROM
		mysql.user u
	WHERE
		u.Select_priv='Y'
	UNION
	SELECT u.*
	FROM
		mysql.user u,
		mysql.db d
	WHERE
		d.Db IN('db1', 'db2', ...) AND
		d.User = u.user
	UNION
	SELECT u.*
	FROM
		mysql.user u,
		mysql.tables_priv t
	WHERE
		t.Db IN('db1', 'db2', ...) AND
		t.User = u.User
	UNION
	SELECT u.*
	FROM
		mysql.user u,
		mysql.procs_priv p
	WHERE
		p.Db IN('db1', 'db2', ...) AND
		p.User = u.User;
 
-- Now, grab the database permissions, and those of objects in the database.
mysql> SELECT * INTO OUTFILE '/tmp/db.txt'
	FIELDS TERMINATED BY ',' OPTIONALLY ENCLOSED BY '"' ESCAPED BY '\\'
	LINES TERMINATED BY '\n'
	FROM
		mysql.db
	WHERE
		Db IN('db1', 'db2', ...);
mysql> SELECT * INTO OUTFILE '/tmp/tables_priv.txt'
	FIELDS TERMINATED BY ',' OPTIONALLY ENCLOSED BY '"' ESCAPED BY '\\'
	LINES TERMINATED BY '\n'
	FROM
		mysql.tables_priv
	WHERE
		Db IN('db1', 'db2', ...);
mysql> SELECT * INTO OUTFILE '/tmp/procs_priv.txt'
	FIELDS TERMINATED BY ',' OPTIONALLY ENCLOSED BY '"' ESCAPED BY '\\'
	LINES TERMINATED BY '\n'
	FROM
		mysql.procs_priv
	WHERE
		Db IN('db1', 'db2', ...);

Then, re-loading the permissions onto the new server is simple:

mysql> LOAD DATA INFILE '/tmp/user.txt'
	FIELDS TERMINATED BY ',' OPTIONALLY ENCLOSED BY '"' ESCAPED BY '\\'
	LINES TERMINATED BY '\n'
	INTO TABLE mysql.user;
mysql> LOAD DATA INFILE '/tmp/db.txt'
	FIELDS TERMINATED BY ',' OPTIONALLY ENCLOSED BY '"' ESCAPED BY '\\'
	LINES TERMINATED BY '\n'
	INTO TABLE mysql.db;
mysql> LOAD DATA INFILE '/tmp/tables_priv.txt'
	FIELDS TERMINATED BY ',' OPTIONALLY ENCLOSED BY '"' ESCAPED BY '\\'
	LINES TERMINATED BY '\n'
	INTO TABLE mysql.tables_priv;
mysql> LOAD DATA INFILE '/tmp/procs_priv.txt'
	FIELDS TERMINATED BY ',' OPTIONALLY ENCLOSED BY '"' ESCAPED BY '\\'
	LINES TERMINATED BY '\n'
	INTO TABLE mysql.procs_priv;

All up, a few queries to account for everything, but pretty easy to include in your backup/restore process. For further development, you could put the database list in a variable, so that you only need to change it on one line, rather than 6.

Server Variables

For tuning InnoDB performance, your primary variable is innodb_buffer_pool_size. This is the chunk of memory that InnoDB uses for caching data, indexes and various pieces of information about your database. The bigger, the better. If you can cache all of your data in memory, you’ll see significant performance improvements.

For MyISAM, there is a similar buffer defined by key_buffer_size, though this is only used for indexes, not data. Again, the bigger, the better.

Other variables that are worth investigating for performance tuning are:

query_cache_size – This can be very useful if you have a small number of read queries that are repeated frequently, with no write queries in between. There have been problems with too large a query cache locking up the server, so you will need to experiment to find a value that’s right for you.

innodb_log_file_size – Don’t fall into the trap of setting this to be too large. A large InnoDB log file group is necessary if you have lots of large, concurrent transactions, but comes at the expense of slowing down InnoDB recover, in event of a crash.

sort_buffer_size – Another one that shouldn’t be set too large. Peter Zaitsev did some testing a while back showing that increasing sort_buffer_size can in fact reduce the speed of the query.

Server Hardware

There are a few solid recommendations for improving the performance of MySQL by upgrading your hardware:

• Use a 64-bit processor, operating system and MySQL binary. This will allow you to address lots of RAM. At this point in time, InnoDB does have issues scaling past 8 cores, so you don’t need to go out of your way to have lots of processors.
• Speaking of RAM, buy lots of it. Enough to fit all of your data and indexes, if you can.
• If you can’t fit all of your data into RAM, you’ll need fast disks, RAID if you can. Have multiple disks, so you can seperate your data files, OS files and log files onto different physical disks.

Query Tuning

Finally, though probably the most important, we look at tuning queries. In particular, we make sure that they’re using indexes, and they’re running quickly. To do so, turn on the Slow Query Log for a day, with log_queries_not_using_indexes enabled as well. Run the resulting log through mysqldumpslow, which will produce a summary of the log. This will help you prioritize which queries to tackle first. Then, you can use EXPLAIN to find out what they’re doing, and adjust your indexes accordingly.

Have fun!

There are a couple of bugs which you could run into, when quoting large numbers. First of all, Bug #34384. This is concerning quoting large INTs in the WHERE condition of an UPDATE or DELETE. It seems that this will cause a table scan, which is going to be slooooow on big tables.

Similarly, there is the more recently discovered Bug #43319. You can run into this if you quote large INTs in the IN clause of a SELECT … WHERE. For example:

mysql> EXPLAIN SELECT * FROM a WHERE a IN('9999999999999999999999')\G
*************************** 1. row ***************************
           id: 1
  select_type: SIMPLE
        TABLE: NULL
         type: NULL
possible_keys: NULL
          KEY: NULL
      key_len: NULL
          ref: NULL
         rows: NULL
        Extra: Impossible WHERE noticed after reading const TABLES
1 row IN SET (0.00 sec)
 
mysql> EXPLAIN SELECT * FROM a WHERE a IN('9999999999999999999999', '9999999999999999999999')\G
*************************** 1. row ***************************
           id: 1
  select_type: SIMPLE
        TABLE: a
         type: ALL
possible_keys: PRIMARY
          KEY: NULL
      key_len: NULL
          ref: NULL
         rows: 655360
        Extra: USING WHERE
1 row IN SET (0.00 sec)

Note that you only run into it when you quote multiple large numbers.

Anyway, the long and the short of this post is: if at all possible, don’t quote numbers. MySQL will love you for it.

For example, checking the master status:

mysql> SHOW MASTER STATUS;
+------------------+----------+--------------+------------------+
| File             | Position | Binlog_Do_DB | Binlog_Ignore_DB |
+------------------+----------+--------------+------------------+
| mysql-bin.000193 |     7061 |              |                  |
+------------------+----------+--------------+------------------+
1 row IN SET (0.00 sec)
 
mysql> SHOW MASTER STATUS\G
*************************** 1. row ***************************
            File: mysql-bin.000193
        Position: 7061
    Binlog_Do_DB:
Binlog_Ignore_DB:
1 row IN SET (0.00 sec)

Now try this for the much larger SHOW SLAVE STATUS. Or for the enormous SHOW ENGINE INNODB STATUS.

As you can see, this is a handy option to make your console output much easier to read.