Tuning glusterfs for apache on EC2
The gluster installation described in a previous post is being used for a webserver cluster on Amazon EC2 using two storage bricks serving a whole bunch of “client” webservers. I tuned the system with “end-to-end” performance testing using a website load tester rather than worry about contrived disk-access tests. That, and helpful comments from various devs on the user list, lead to the following conclusions.
There’s a large collection of “performance translators” in gluster used for improving speed. Let’s have a look at the ones I didn’t use and why:
- performance/read-ahead – Probably useful if your server has physical disks as it will minimise disk seeks. But amazon EBS storage is no doubt a layered storage system with its own caching. So this translator doesn’t offer any speed increase and just gets in the way.
- performance/write-behind – Same issues as read-ahead. Plus this translator seems to have problems if you try to read a file quickly after writing it.
- performance/stat-prefetch – Pre-fetches and caches file stat information when a directory is read. Speeds up operations like ls -l but apache never needs that so it just gets in the way.
- performance/quick-read – Uses a feature of the gluster protocol so the whole of a (small) file can be fetched during the lookup phase so opens and reads are not needed. Also caches the file data. Unfortunately it has a memory-leak bug that may be fixed in v3.0.5. Until then it can’t really be used.
These are the performance filters I did use
- performance/io-cache – Caches read file data in 128K pages for 1-60 seconds. The page size and maximum cache timeout can be changed in the source. Should only be used in volumes where files are read much more often than they are written because the translator just invalidates a whole 128K page when any part of it is written. This is perfect for website pages though.
- performance/io-threads – Doesn’t fork extra processes, but does configure a thread pool that allows faster operations to leap-frog blocked ones.
The translator stack I came up with has this layout:
APACHE
|
performance/io-cache
|
performance/io-threads
|
cluster/replicate
|
protocol/client
| |
AMAZON NETWORK
| |
protocol/server
|
performance/io-threads
|
features/locks
|
storage/posix
|
ext3/xfs/whatever
| |
AMAZON EBS STORAGE
The philosophy is
- Only use the translators that you can prove actually provide a benefit. Translators are cheap but still get in the way. The gluster volgen command provides a good start for a general server but the volume config can be tweaked more for webservers.
- Caching first. It’s quick and should be serving most of the files.
- Lots of threads on the client side. Apache is multi-threaded and Amazon EC2 servers are multi-core. Anything we can do to help concurrency to the bricks is a good thing.
- Threads on the server side too. I’ve read some articles that say this is a waste. But, in my experience, a large rsync on one client for example can really hold up accesses made from other clients unless io-threads is configured on the server side too. Also, EBSs never “fail” but occasionally they do exhibit huge iowait spikes of 100s of ms. In these circumstances io-threads on the server side mean that a minimum of the clients are kept waiting.
- Don’t bother caching on the server side. The kernel will already be caching the filesystem underneath gluster.
The best tip though is to understand the whole architecture of your system and concentrate your optimisation efforts where they will have the most benefit. Seems obvious once it’s said, but it takes some out-of-the-box / holistic / whatever thinking to actually do it.
In the case of an Apache web service, moving from single-server nfs to replicated gluster initially caused pages to take an extra 500ms or much more to load! This was almost a disaster – glusterfs is tuned for big files rather than small… In this case the solution was simple: migrate all .htaccess files into
Updates:
Lee Simpson Says:
March 26th, 2010 at 1:07 am
Thanks for the interesting articles.
“performance/stat-prefetch” may be useful depending on what your apache is serving. If you are serving PHP pages which read the contents of folders (e.g wordpress/phpbb) then it could make a big difference. However, I’ve had big problems with “performance/stat-prefetch” and apache and have filed a bug here;
http://bugs.gluster.com/cgi-bin/bugzilla3/show_bug.cgi?id=762
Once those problems are cleared up in a future release it may be worth taking another look at performance/stat-prefetch