• SAN Best Practices
February 2nd, 2012

Finding Application Performance Bottlenecks – Queue Depths

Jim Bahn

It has been pretty well understood for quite some time, that when optimizing application performance in large enterprise environments, the biggest boosts come from the I/O subsystem – the SAN.  Application latency can come from CPUs and memory, but optimizing these may only buy you microseconds.  I/O is measured in milliseconds and that’s where your biggest bottlenecks are, and your biggest opportunities.  But “I/O subsystem” means much more than just disk seek time.  For example, over the past few years, we have noticed that customers very often set their HBA queue depths too high, and we’ve yet to find someone who sets them too low.  For an example of the effect of queue depth settings, check out our January 4, 2012 blog on the subject.  Since this is a “best practices” blog, and as we specialize in I/O monitoring and measurement, we wanted to share our thoughts.  Before you spend a wad of CAPEX on SSD memory or storage ports, or anything else for that matter, assume that your HBA queue depth settings are too high, and test for better settings.  And as we showed you in the January 4th blog, VirtualWisdom is a great way to see, in real time, the effect of your changes.

Since queue depth is per LUN, you have to take the server configuration into account.  The optimal queue depth for a server depends on the number of disks behind its LUNs, and the drive configuration.  For instance, Enterprise Flash Drive configurations require huge amounts of queued I/Os to get maximum throughput from the disk drives.  A LUN with 200 drives will need significantly more queued workload than a LUN with 5 drives to keep every drive busy.  It’s impossible to recommend a single queue depth policy for an entire environment, and each instance must be evaluated separately, but as stated above, the vast majority of customers we work with set them too high. It’s safe to assume that if you selectively lower your queue depths, you’ll reduce the I/O latency of many of your applications without spending a dime on additional hardware.