A few months ago we launched an online community / social network at querdenker.de using our own software foronline communties. We have always been thinking that we already considered nearly everything which could break our system when beeing accessed by a lot of people concurrently. And we were right. What did break the site when more than 200 people accessed the site in the same minute, it has not been our software that collapsed but it has been the whole LAMP setup. So here is what we did to get the site up and running again:
Step 1: Optimize your software
First of all, optimize your software. No LAMP stack setup can repair the damage done by bad database queries, missing caching algorithms, bloated code and bad HTTP headers (client-side caching!). So there is no way to get around this step. When we were confronted with a huge load on the server, the first thing we did was optimizing all queries once again, disabling functions that were not necessary for the user (e.g. statistics).
Step 2: Split servers
The site I was talking about is hosted in the Amazon Cloud. So we used an EC2 instance for the web server, an RDS instance for our MySQL database and S3 storage for static images that were not submitted by the user (e.g. icons). This is an important step since you can reduce load to a third of what it was before. Especially if you consider that simple requests to static files are more likely to kill your LAMP setup than dynamic requests are.
Why is this?
When you use Apache MPM Prefork module, several Apache processes are started in background which then will handle the incoming requests. If your clients are required to load static files like images through your Apache setup it is likely that all processes are locked for simple static file requests rather than dynamic requests like PHP what they are really there for.
Step 3: Setup up MPM Prefork
Default Apache setup uses only one thread to handle all requests. Fine on localhost, but not fine on a production system. You should use MPM Prefork module installed. This module launches several Apache processes which then handle all incoming requests.
Choose the right number of processes
Each Apache process will consume a similar amount of memory on your server, depending on the Apache modules which are loaded (next step) and the tasks you'll execute. So you have to choose a number of processes which matches your available memory. If there is 1GB of memory available - not at all but for Apache, your OS will use memory too ;-) - and one of your Apache processes uses 20MB of memory, you can launch 50 processes equally. If you choose more, Apache will run in swap which results in huge memory issues.
Find out memory usage:
#ps -FA|grep apache2
To set the maximum number of processes launched open your Apache config (usually /etc/apache2/apache2.conf) and set MaxClients to the correct number.
Step 4: Disable Apache modules
Apache modules will use memory for each process that is launched and this is very expensive. The default Apache setup comes with more than 20 enabled modules which we did not need. You should disable everything that is not necessary for your site to run:
... and so on ...
This will save you memory and allows you to launch more parallel processes.
Step 4: Reduce KeepAliveTimeout
If a client connects to your web server it will keep the connection open for a specific number of seconds for further requests. During this time, the whole Apache process is locked for other requests. You can specify the number of keep-alive seconds in your Apache config using the directive KeepAliveTimeout. You should enable KeepAlive but reduce KeepAlive to a value of 2-3 seconds. Default is 30. We were also confrontend with a lot of processes running in Keep-Alive state when we monitored our server and this solved the issue and we were able to serve three times more visitors with the same server setup!
Step 5: Have a look at server-status!
Enable the mod_status module (#a2enmod status) and enable ExtendedStatus directive ("ExtendedStatus On") in /etc/apache2/mods-enabled/status.conf! Then have a look at /server-status which will tell you how many Apache processes are running and in which state (e.g. to find out if Keep-Alive is killing your setup) and which files are requested. Maybe you can find out which files to move to S3 storage using this technique.
What else to do?
If you are using PHP you should enable eAccelerator or something similar. You could use Memcached for sessions and caching algorithms and so on. There is a lot to do when optimizing your site for high traffic and it costs time. Nothing to do about that ;-)