How to Speed up Drupal Forum Pages on a Busy Site
Drupal 5's forum module is OK. It lets you create multiple forums with taxonomy terms. Forum posts are true nodes so you get all the benefits of nodeness. And Drupal keeps track of which posts you've read. All that is great. But I've had some trouble scaling up a site that relies heavily on the forum module, even when using advcache. Investigating how the database was spending its time, I found lots of queries like this:
SELECT n.nid, n.title, n.sticky, l.comment_count, l.last_comment_timestamp FROM node n INNER JOIN node_comment_statistics l ON n.nid = l.nid INNER JOIN term_node r ON n.nid = r.nid AND r.tid = 123 WHERE n.status = 1 AND n.type = 'forum' ORDER BY n.sticky DESC, l.last_comment_timestamp desc
An EXPLAIN shows how nasty this is:
| id | select_type | table | type | possible_keys | key | key_len | ref | rows | Extra |
|---|---|---|---|---|---|---|---|---|---|
| 1 | SIMPLE | n | ref | PRIMARY,node_type,status,node_status_type,nid | node_status_type | 102 | const,const | 30679 | Using where; Using temporary; Using filesort |
| 1 | SIMPLE | l | eq_ref | PRIMARY | PRIMARY | 4 | drupal.n.nid | 1 | |
| 1 | SIMPLE | r | eq_ref | PRIMARY,nid,tid | PRIMARY | 8 | const,drupal.l.nid | 1 | Using where; Using index |
Searching Drupal's code turns up this query in theme_forum_topic_navigation(). It turns out the database is smoking, grinding away, eating up cycles generating previous and next links for forum topics. I thought about it for a minute, and realized that I don't think I've ever used those links. Which would you rather have? Blazing speed or a link to the next forum topic? Fortunately, the slow query is in a themable function, which means we can get our performance back with a few lines added to our theme's template.php:
// No previous/next links for forum topics.
function phptemplate_forum_topic_navigation($node) {
return '';
}In my case, server load dropped from 46 to 0.5. Sometimes it's the little things.
TCP Tuning for Busy Apache Webserver on CentOS5
Recently I was in a situation where a very busy webserver was not responding. Strangely, top showed plenty of CPU available. The server was essentially just sitting there. What do do?
Upon further investigation, it turned out that the network queue was saturated. So many incoming connections were being attempted that they were falling off the end. Some TCP tuning was in order. Fortunately the server was not memory-starved so allocating more memory to the network stack was not a problem. Here's what ended up in /etc/sysctl.conf and turned the server back into a faithful workhorse.
# Kernel tuning settings for CentOS5,
# busy webserver with lots of free memory.
# Big queue for the network device
net.core.netdev_max_backlog=30000
# Lots of local ports for connections
net.ipv4.tcp_max_tw_buckets=2000000
# Bump up send/receive buffer sizes
net.core.rmem_default=262141
net.core.wmem_default=262141
net.core.rmem_max=262141
net.core.wmem_max=262141
# Disable TCP selective acknowledgements
net.ipv4.tcp_sack=0
net.ipv4.tcp_dsack=0
# Decrease the amount of time we spend
# trying to maintain connections
net.ipv4.tcp_retries2=5
net.ipv4.tcp_fin_timeout=60
net.ipv4.tcp_keepalive_time=120
net.ipv4.tcp_keepalive_intvl=30
net.ipv4.tcp_keepalive_probes=3
# Increase the number of incoming connections
# that can queue up before dropping
net.core.somaxconn=256
# Increase option memory buffers
net.core.optmem_max=20480
There are plenty of other sysctl options to tune, but the above made the most difference.
And netstat -s is your friend.
Multistep Forms with Lookup Functions that Change Values
A number of people lately have been asking about how to change the values of form fields in multistep forms. So I wrote up an example.
Here we have a simple multistep form that lets you enter the name of a U.S. state, such as Iowa. Note the form has a default value.
Then, upon clicking the Lookup button, the value "Iowa" is replaced with a standardized state abbreviation, IA.
Upon submission both values are available to the submit function.
This example is not intended to do anything useful. It's just a little example that you can poke around with to learn how the sometimes confusing multistep form submission works in Drupal 5. You can download the module here.
FCKEditor toolbar will not appear - solution
When using FCKEditor as a WYSIWYG editor in Drupal, if you've double-checked all your settings and you can't get the formatting toolbar to appear, check to make sure that your template (e.g. page.tpl.php) is printing out the $closure variable. Thanks, latentdabbler.
Cannot install OS X Server Admin software on Leopard
I'd like to install Apple's Server Administration Tools for OS X 10.4 Tiger Server. But I'm running Leopard. The installer stops with the following helpful message: "You are not allowed to install the software on this disk for an unknown reason."
Apparently if you just copy the apps from a computer running Tiger to one running Leopard, it works fine.
Using Word 2007 Blog functionality to post to Drupal via the MetaWeblog API
First, I enabled the Blog API module on my Drupal site.
Then under Administer - Site Configuration - Blog APIs, I enabled the content type to which I want Word 2007 to publish to (just page in my case).
In Word 2007, I clicked on the round Office Button at the upper left of the screen and selected New.
I selected New blog post from the Microsoft Office Online template, then clicked the Create button.
I chose Other when asked about the provider:
.gif)
I chose the MetaWebLog API, which Drupal supports. The Blog Post URL being requested is the URL of your site plus xmlrpc.php. The username and password are the Drupal username and password.
I am not exploring image support at this time, so I clicked on Picture Options and chose None - Don't upload pictures.
I then typed in my blog post and clicked the friendly Publish button in the Word 2007 Blog Toolbar, and...happy happy, my post appeared on my blog.
Subversion server on OS X 10.5
Sonzea: Installing a Subversion server on Leopard.
Apache [client ::1] requests, broken pipes, virtual hosts and Drupal
I was puzzling over entries like this in Apache 2.2.3's error log:
[info] [client ::1] (32)Broken pipe: core_output_filter: writing data to the network
A check for which user agent was causing that revealed:
::1 - - [01/Dec/2007:04:08:53 +0000] "GET / HTTP/1.0" 200 85781 "-" "Apache/2.2.3 (CentOS) (internal dummy connection)"
So it's Apache's internal dummy connection throwing the errors. Oh, but what's this?
::1 - - [01/Dec/2007:04:08:53 +0000] "GET / HTTP/1.0" 200 85781 "-" "Apache/2.2.3 (CentOS) (internal dummy connection)"
Yikes! Here's what is happening.
1. Apache opens an internal dummy connection with a request for /.
2. On this server, / turns out to be a Drupal page.
3. Drupal spends time and effort putting together the page and sends it...
4. ...to the dummy connection that ignores it resulting in the broken pipe.
It turned out that in httpd.conf, an include statement which pulled in a virtual host configuration was being encountered before the regular virtual host configuration...making the included virtual host configuration the default host for all requests, including internal Apache requests.
I solved this by making sure the default page of the first virtual host is not a dynamic page.
Time Machine cannot do a full restore
I've been using Time Machine in OS X 10.5 Leopard on my Mac Pro to do nifty backups. It has come in handy a couple of times now, when I've been chasing problems down a rat hole (like, "why does ImageMagick work from the command line but not from an exec() call in PHP?") and I just want to give up and revert back to a previous state. Very nice.
Yesterday I got a new drive (WD2500YS) that has a nice 16MB cache on it, and I thought I'd swap in that drive as my boot drive. Normally I'd use SuperDuper! for this but it's not yet fully Leopard-compatible. So I thought, now's the time to do a full restore using Time Machine and see how it goes.
I booted from the Leopard DVD and selected Restore from Time Machine Backup. I was a bit nervous because the "time of last backup" shown by Time Machine was two hours earlier than it should have been, but this turned out to be a time zone issue; all my files were restored, including the ones saved just before I shut the machine down.
It took a while (over an hour) and booted up fine. As I've learned from my attempted Tiger-to-Leopard upgrades (do a clean install!) things are not always running smoothly under the hood. Sure enough, the console greeted me with
11/21/07 8:36:41 AM com.apple.launchd[1] (org.postfix.master) Failed to count the number of files in "/var/spool/postfix/maildrop": No such file or directory
Upon further examination, /var/spool is...empty! Taking a look at my previous drive, there are over 100 entries (some noted here). Why is this? Devin Lane has a nice post listing the exclusions Time Machine makes. Wait, I thought, maybe they are created on demand?
bash-3.2# mail foo@example.com
Subject: Hi
This is a test.
EOT
bash-3.2# postdrop: warning: mail_queue_enter: create file maildrop/229433.449: No such file or directory
postdrop: warning: mail_queue_enter: create file maildrop/230063.449: No such file or directory
postdrop: warning: mail_queue_enter: create file maildrop/230821.449: No such file or directory
Nope.
Conclusion: Time Machine is fine for protecting the mainstream data in your home directory, but is currently not capable of doing a full restore of a boot drive.
Innovation
There are few companies doing exciting things with hardware. Back in the golden era of computing (or was it the bronze age?) everyone would look to Applied Engineering, thinking "What will they think of next?"
A current company that fits that mold is WiebeTech.