HP DL360 G4 server

HP and its line of DL servers is very respected amongst IT engineers. It is (at least by my experience) a reliable class of servers, well built, easy to maintain and comes with an excellent server management software called Integrated Lights-Out (ILO). DLs SCSI and SAS storage subsystem is usually controlled by controllers called Smart Array, which can be integrated onto the motherboard or not. Fresh HP customer gets all the nifty RAID levels to play with and is usually satisfied. But what fresh customer usually DOES NOT KNOW is that Smart Array write performance really sucks if write cache is not enabled. And to enable it, fresh user needs to install a special module with attached battery called BBWC. So, HP, if you are accidentally reading this, please do notify your customers about such things.

HP DL380 G5 server

This is another server where I conducted the benchmark and, unlike DL360 G4, already came with BBWC installed. To simulate absence of write cache I disabled it with command line tool hpacucli (HP Array Configuration Utility Command Line Interface).

Benchmark metodology

For this benchmark I used two tools from sql-bench suite which heavily stress filesystem with lots of file creations and deletions. These tools are test-alter-table and test-create. The former test is faster and only gives rough figure. The later creates and deletes around 50,000 MyISAM tables which results in 150,000 files created and deleted. I executed both benchmarks first without and then with write cache enabled.
One of the machines (the DL380) was already in production, but it was benchmarked during the night when usage is negligible.

Test systems

HP DL360 G4

• Controller: Smart Array 6i
• Filesystem: ufs
• OS: FreeBSD 6.0
• MySQL: mysql-5.0.41-freebsd6.0-i386

HP DL380 G5

• Controller: Smart Array P400i
• ext3 filesystem
• OS: Slackware 12.2
• MySQL: mysql-5.0.77 compiled from source

Results on HP DL380 G5

Drive write cache DISABLED

Testing of ALTER TABLE
Time for insert (1000) 0 wallclock secs
( 0.02 usr 0.00 sys + 0.00 cusr 0.00 csys = 0.02 CPU)
Time for alter_table_add (100): 17 wallclock secs ( 0.00 usr 0.00 sys + 0.00 cusr 0.00 csys = 0.00 CPU)
Time for create_index (8): 2 wallclock secs ( 0.00 usr 0.00 sys + 0.00 cusr 0.00 csys = 0.00 CPU)
Time for drop_index (8): 2 wallclock secs ( 0.00 usr 0.00 sys + 0.00 cusr 0.00 csys = 0.00 CPU)
Time for alter_table_drop (91): 16 wallclock secs ( 0.01 usr 0.00 sys + 0.00 cusr 0.00 csys = 0.01 CPU)
Total time: 37 wallclock secs ( 0.03 usr 0.00 sys + 0.00 cusr 0.00 csys = 0.03 CPU)

Testing the speed of creating and dropping tables
Time for create_MANY_tables (10000): 253 wallclock secs ( 0.26 usr 0.06 sys + 0.00 cusr 0.00 csys = 0.32 CPU)
Time to select_group_when_MANY_tables (10000): 1 wallclock secs ( 0.09 usr 0.07 sys + 0.00 cusr 0.00 csys = 0.16 CPU)
Time for drop_table_when_MANY_tables (10000): 1 wallclock secs ( 0.09 usr 0.03 sys + 0.00 cusr 0.00 csys = 0.12 CPU)
Time for create+drop (10000): 259 wallclock secs ( 0.24 usr 0.18 sys + 0.00 cusr 0.00 csys = 0.42 CPU)
Time for create_key+drop (10000): 255 wallclock secs ( 0.41 usr 0.11 sys + 0.00 cusr 0.00 csys = 0.52 CPU)
Total time: 769 wallclock secs ( 1.09 usr 0.45 sys + 0.00 cusr 0.00 csys = 1.54 CPU)

Drive write cache ENABLED

Testing of ALTER TABLE
Time for insert (1000) 0 wallclock secs ( 0.02 usr 0.00 sys + 0.00 cusr 0.00 csys = 0.02 CPU)
Time for alter_table_add (100): 3 wallclock secs ( 0.01 usr 0.01 sys + 0.00 cusr 0.00 csys = 0.02 CPU)
Time for create_index (8): 1 wallclock secs ( 0.00 usr 0.00 sys + 0.00 cusr 0.00 csys = 0.00 CPU)
Time for drop_index (8): 0 wallclock secs ( 0.00 usr 0.00 sys + 0.00 cusr 0.00 csys = 0.00 CPU)
Time for alter_table_drop (91): 4 wallclock secs ( 0.01 usr 0.00 sys + 0.00 cusr 0.00 csys = 0.01 CPU)
Total time: 8 wallclock secs ( 0.04 usr 0.01 sys + 0.00 cusr 0.00 csys = 0.05 CPU)

Testing the speed of creating and dropping tables
Time for create_MANY_tables (10000): 9 wallclock secs ( 0.34 usr 0.06 sys + 0.00 cusr 0.00 csys = 0.40 CPU)
Time to select_group_when_MANY_tables (10000): 1 wallclock secs ( 0.06 usr 0.03 sys + 0.00 cusr 0.00 csys = 0.09 CPU)
Time for drop_table_when_MANY_tables (10000): 1 wallclock secs ( 0.10 usr 0.04 sys + 0.00 cusr 0.00 csys = 0.14 CPU)
Time for create+drop (10000): 9 wallclock secs ( 0.44 usr 0.10 sys + 0.00 cusr 0.00 csys = 0.54 CPU)
Time for create_key+drop (10000): 10 wallclock secs ( 0.35 usr 0.10 sys + 0.00 cusr 0.00 csys = 0.45 CPU)
Total time: 30 wallclock secs ( 1.29 usr 0.33 sys + 0.00 cusr 0.00 csys = 1.62 CPU)

Results on HP DL360 G4

Drive write cache DISABLED

Testing of ALTER TABLE
Time for insert (1000) 0 wallclock secs ( 0.02 usr 0.02 sys + 0.00 cusr 0.00 csys = 0.04 CPU)
Time for alter_table_add (100): 33 wallclock secs ( 0.01 usr 0.01 sys + 0.00 cusr 0.00 csys = 0.02 CPU)
Time for create_index (8): 4 wallclock secs ( 0.00 usr 0.00 sys + 0.00 cusr 0.00 csys = 0.00 CPU)
Time for drop_index (8): 3 wallclock secs ( 0.00 usr 0.00 sys + 0.00 cusr 0.00 csys = 0.00 CPU)
Time for alter_table_drop (91): 33 wallclock secs ( 0.01 usr 0.01 sys + 0.00 cusr 0.00 csys = 0.02 CPU)
Total time: 75 wallclock secs ( 0.04 usr 0.03 sys + 0.00 cusr 0.00 csys = 0.07 CPU)

Testing the speed of creating and dropping tables
Testing with 10000 tables and 10000 loop count
Time for create_MANY_tables (10000): 1035 wallclock secs ( 1.27 usr 0.25 sys + 0.00 cusr 0.00 csys = 1.52 CPU)
Time to select_group_when_MANY_tables (10000): 83 wallclock secs ( 0.63 usr 0.16 sys + 0.00 cusr 0.00 csys = 0.79 CPU)
Time for drop_table_when_MANY_tables (10000): 493 wallclock secs ( 0.50 usr 0.19 sys + 0.00 cusr 0.00 csys = 0.69 CPU)
Time for create+drop (10000): 958 wallclock secs ( 1.59 usr 0.38 sys + 0.00 cusr 0.00 csys = 1.97 CPU)
(NOTICE: Could not wait for this test to finish, because machine needed to get back in production,)
(thus I assume it to be around 900 seconds just to be on the safe side, probably would be more.)
Total time calculated: around 3400 seconds

Drive write cache ENABLED

Testing of ALTER TABLE
Time for insert (1000) 0 wallclock secs ( 0.02 usr 0.01 sys + 0.00 cusr 0.00 csys = 0.02 CPU)
Time for alter_table_add (100): 8 wallclock secs ( 0.02 usr 0.00 sys + 0.00 cusr 0.00 csys = 0.02 CPU)
Time for create_index (8): 1 wallclock secs ( 0.00 usr 0.00 sys + 0.00 cusr 0.00 csys = 0.00 CPU)
Time for drop_index (8): 1 wallclock secs ( 0.00 usr 0.00 sys + 0.00 cusr 0.00 csys = 0.00 CPU)
Time for alter_table_drop (91): 8 wallclock secs ( 0.01 usr 0.01 sys + 0.00 cusr 0.00 csys = 0.02 CPU)
Total time: 18 wallclock secs ( 0.05 usr 0.02 sys + 0.00 cusr 0.00 csys = 0.06 CPU)

Testing the speed of creating and dropping tables
Time for create_MANY_tables (10000): 104 wallclock secs ( 1.07 usr 0.24 sys + 0.00 cusr 0.00 csys = 1.31 CPU)
Time to select_group_when_MANY_tables (10000): 27 wallclock secs ( 0.48 usr 0.18 sys + 0.00 cusr 0.00 csys = 0.66 CPU)
Time for drop_table_when_MANY_tables (10000): 53 wallclock secs ( 0.32 usr 0.09 sys + 0.00 cusr 0.00 csys = 0.41 CPU)
Time for create+drop (10000): 143 wallclock secs ( 1.31 usr 0.30 sys + 0.00 cusr 0.00 csys = 1.62 CPU)
Time for create_key+drop (10000): 164 wallclock secs ( 1.56 usr 0.36 sys + 0.00 cusr 0.00 csys = 1.92 CPU)
Total time: 491 wallclock secs ( 4.74 usr 1.18 sys + 0.00 cusr 0.00 csys = 5.92 CPU)

Analysis

The test-alter-table results seem fine, slightly over 400% increase in performance. But what bothers me is the test-create difference. I expected the HP DL360 G4 to improve more and execute this test below 100 seconds barier, heck, actually I expected it below 50 seconds. It is true that this machine uses different operating- and filesystem. But 500 seconds still seems too much to me, especially when HP DL380 G5 excells at 30 seconds. If someone know the answer, please drop it in comments.

Conclusion

I believe this article has clearly shown why one must conduct even such synthetic tests before deploying the systems to production environment. Furthermore even before the “real-world benchmarks” are conducted. The phrase “real-world benchmark” signifies a comparative benchmark of certain application on an existing production systems and on the ones that are in testing phase. It often happens that hardware is not upgraded for quite some time, which means that new hardware is few generations younger than the existing one. The new one is far more powerful and one easily misses some not-so-innocent bottleneck if “real-world benchmark” displays certain improvement. Thus, as I believe, newer systems MUST perform better than older ones in every synthetic benchmark (if the systems are comparable, of course), and only then we can start conducting “real-world benchmarks”.

How did I discover this “issue”?

It happened to me back in the 2004 that I deployed such a HP server to collocation facility and only later discovered that it was performing worse than some old test machine lying under my desk. After couple of hours of googling I assumed that the lack of BBWC was our problem. I had to order it and then remove the server from collocation because I also wanted to upgrade all the firmwares, just in case. On top of that, I still had to figure out how to install ‘hpacucli’ on non-RedHat linux. After a long weekend the machine was back in production and never caused a single problem again.

Results
Total execution time is: 562 seconds

# run-all-tests
alter-table: Total time: 8 wallclock secs ( 0.02 usr 0.01 sys + 0.00 cusr 0.00 csys = 0.03 CPU)
ATIS: Total time: 2 wallclock secs ( 1.20 usr 0.09 sys + 0.00 cusr 0.00 csys = 1.29 CPU)
big-tables: Total time: 5 wallclock secs ( 2.45 usr 0.08 sys + 0.00 cusr 0.00 csys = 2.53 CPU)
connect: Total time: 50 wallclock secs (12.74 usr 4.50 sys + 0.00 cusr 0.00 csys = 17.24 CPU)
create: Total time: 31 wallclock secs ( 1.20 usr 0.44 sys + 0.00 cusr 0.00 csys = 1.64 CPU)
insert: Total time: 397 wallclock secs (97.95 usr 13.61 sys + 0.00 cusr 0.00 csys = 111.56 CPU)
select: Total time: 44 wallclock secs ( 8.71 usr 0.88 sys + 0.00 cusr 0.00 csys = 9.59 CPU)
transactions: Test skipped because the database doesn’t support transactions
wisconsin: Total time: 3 wallclock secs ( 0.91 usr 0.23 sys + 0.00 cusr 0.00 csys = 1.14 CPU)
TOTALS 562.00 123.77 19.82 143.59 3425950

System specification can be found here.

ReiserFS vs others
In the age of Linux kernel 2.4.x we used ReiserFS v3 as the filesystem of choice. With the available options of ReiserFS (journal, performance), ext2 (stable but slow) and ext3 (probably stable, but not so speedy as ReiserFS) the choice was obvious. I skipped few years then and this year again tried using ReiserFS with linux 2.6.29.1 but it turned out to be even slower than ext2 was in the old days. Googling around for an answer gave some hints that ReiserFS has an issue with someting called BIG_KERNEL_LOCK on 2.6 kernels. I didn’t really investigate further, but went down the ext3 way.

Comments on test-create
If test-create takes much more time than, say, 30-60 seconds, then you definitely have a problem with filesystem write performance. On HP DL360 and DL380 class of servers this correlates with the presence and activation of BBWC (Battery-Backed Write Cache enabler kit). Without BBWC and hence without write cache enabled, this test took more than 10 minutes to complete. Thus, if you are purchasing some new HP servers, be sure that you also order BBWCs.

Question about test-insert
Looking at the test times, this test-insert result is really standing out. Again, I do not have any other data to compare it to, but somewhere deep down in my memory I seem to remember that the total time for all the tests was around 300 seconds. This obviously means that this test-insert result is the bad guy here. Can someone comment on this result, or paste in the comments his own? Thanks.

Feedback
If you have any questions, recommendations or benchmark results to compare, do not hesitate to leave a comment.

Today I am starting with MySQL, the database of choice for many web developers.

For benchmarking MySQL performance, I primarily use three tools:

1. The MySQL benchmark suite,
2. Super Smack and
3. some custom applications for real world tests.

As it is commonly known, first two items from the list above are not what one might call “real world benchmark suites”. But they are very fine tools for discovering system bottlenecks and obvious configuration errors. Below follow benchmark commands and results with complete (relevant) setup description. Enjoy!

Results for select-key
Queries per second: 166541

cd /usr/share/smacks &&
super-smack select-key.smack 32 100000

Query Barrel Report for client smacker1
connect: max=34ms min=0ms avg= 14ms from 32 clients
Query_type num_queries max_time min_time q_per_s
select_index 6400000 0 0 166541.80

Results for update-key
Queries per second: 9225

cd /usr/share/smacks &&
super-smack update-select.smack 32 10000

Query Barrel Report for client smacker
connect: max=0ms min=0ms avg= 0ms from 32 clients
Query_type num_queries max_time min_time q_per_s
select_index 320000 2 0 9225.27
update_index 320000 2 0 9225.27

Hardware

• HP DL380 g5
• 2x Intel E5420 Xeon (4 cores @ 2.5GHz each)
• 8GB ECC RAM
• 2x 147 SAS 10k drives in hardware RAID-1 configuration
• Write cache enabled for Smart Array P400i

Software

• Slackware 12.2
• Ext3 filesystem
• Linux kernel version 2.4.29.1
• Super Smack 1.3, compiled from source
• MySQL version 5.0.77, compiled from source with configure line:
  

  ./configure –prefix=/usr/local/$PDIR \
  –with-charset=utf8 –with-collation=utf8_general_ci \
  –with-unix-socket-path=/var/mysql/run/mysql.sock \
  –with-mysqld-user=mysql \
  –with-openssl=/usr/local/openssl-0.9.8k

/etc/my.cnf

[client]
socket = /var/mysql/run/mysql.sock

[mysqld]
user = mysql
basedir = /usr/local/mysql
pid-file = /var/mysql/run/mysqld.pid
socket = /var/mysql/run/mysql.sock
datadir = /var/mysql/data
log-error = /var/mysql/log/mysql_error.log
log-slow-queries = /var/mysql/log/mysql_slow_query.log

ssl
ssl-cert = /etc/ssl/certs/_default.cert
ssl-key = /etc/ssl/certs/_default.key

key_buffer = 256M
max_allowed_packet = 32M
table_cache = 256
sort_buffer_size = 2M
read_buffer_size = 4M
read_rnd_buffer_size = 8M
myisam_sort_buffer_size = 64M
net_buffer_length = 2K
thread_stack = 128K
thread_cache_size = 16K
thread_concurrency = 8
query_cache_size = 32M
max_join_size = 4294967295

innodb_data_home_dir = /var/mysql/data
innodb_data_file_path = ibdata1:10M:autoextend
innodb_log_group_home_dir = /var/mysql/data
innodb_log_arch_dir = /var/mysql/data
innodb_buffer_pool_size = 16M
innodb_additional_mem_pool_size = 2M
innodb_log_file_size = 5M
innodb_log_buffer_size = 8M
innodb_flush_log_at_trx_commit = 1
innodb_lock_wait_timeout = 50

[mysqldump]
quick
max_allowed_packet = 32M

[mysql]
no-auto-rehash

[isamchk]
key_buffer = 128M
sort_buffer_size = 128M
read_buffer = 2M
write_buffer = 2M

[myisamchk]
key_buffer = 128M
sort_buffer_size = 128M
read_buffer = 2M
write_buffer = 2M

[mysqlhotcopy]
interactive-timeout

And also the results of sql-bench suite are here.

Feedback
If you have any questions, recommendations or benchmark results to compare, do not hesitate to leave a comment.