Securing Your /tmp Partition with Cpanel/WHM Published: Nov 07, 2003
  • Rating


If you are reaching the limits of your server running Apache serving a lot of dynamic content, you can either spend thousands on new equipment or reduce bloat to increase your server capacity by anywhere from 2 to 10 times.

Optimize High-Traffic Servers
Focus: Linux, Apache 1.3+, [PHP], [MySQL]
Notes: Use at your own risk. If this has any errors, please let me know and I will correct them.

If you are reaching the limits of your server running Apache serving a lot of dynamic content, you can either spend thousands on new equipment or reduce bloat to increase your server capacity by anywhere from 2 to 10 times. This article concentrates on important and poorly-documented ways of increasing capacity without additional hardware.

There are a few common things that can cause server load problems, and a thousand uncommon. Let's focus on the common:

Drive Swapping - too many processes (or runaway processes) using too much RAM
CPU - poorly optimized DB queries, poorly optimized code, runaway processes
Network - hardware limits, moron attacks

Solutions: The Obvious

Briefly, and for completeness, here are the most obvious solutions:

Use "TOP" and "PS axu" to check for processes that are using too much CPU or RAM.
Use "netstat -anp | sort -u" to check for network problems.

Solutions: Apache's RAM Usage
First and most obvious, Apache processes use a ton a RAM. This minor issue becomes a major issue when you realize that after each process has done its job, the bloated process sits and spoon-feed data to the client, instead of moving on to bigger and better things. This is further compounded by a bit of essential info that should really be more common knowledge:

If you serve 100% static files with Apache, each httpd process will use around 2-3 megs of RAM.
If you serve 99% static files & 1% dynamic files with Apache, each httpd process will use from 3-20 megs of RAM (depending on your MOST complex dynamic page).

This occurs because a process grows to accommodate whatever it is serving, and NEVER decreases again unless that process happens to die. Quickly, unless you have very few dynamic pages and major traffic fluctuation, most of your httpd processes will take up an amount of RAM equal to the largest dynamic script on your system. A smart web server would deal with this automatically. As it is, you have a few options to manually improve RAM usage.

Reduce wasted processes by tweaking KeepAlive
This is a tradeoff. KeepAliveTimeout is the amount of time a process sits around doing nothing but taking up space. Those seconds add up in a HUGE way. But using KeepAlive can increase speed for both you and the client - disable KeepAlive and the serving of static files like images can be a lot slower. I think it's best to have KeepAlive on, and KeepAliveTimeout very low (like 1-2 seconds).

Limit total processes with MaxClients
If you use Apache to serve dynamic content, your simultaneous connections are severely limited. Exceed a certain number, and your system begins cannibalistic swapping, getting slower and slower until it dies. IMHO, a web server should automatically take steps to prevent this, but instead they seem to assume you have unlimited resources. Use trial & error to figure out how many Apache processes your server can handle, and set this value in MaxClients. Note: the Apache docs on this are misleading - if this limit is reached, clients are not "locked out", they are simply queued, and their access slows. Based on the value of MaxClients, you can estimate the values you need for StartServers, MinSpareServers, & MaxSpareServers.

Force processes to reset with MaxRequestsPerChild
Forcing your processes to die after a while makes them start over with low RAM usage, and this can reduce total memory usage in many situations. The less dynamic content you have, the more useful this will be. This is a game of catch-up, with your dynamic files constantly increasing total RAM usage, and restarting processes constantly reducing it. Experiment with MaxRequestsPerChild - even values as low as 20 may work well. But don't set it too low, because creating new processes does have overhead. You can figure out the best settings under load by examining "ps axu --sort:rss". A word of warning, using this is a bit like using heroin. The results can be impressive, but are NOT consistent - if the only way you can keep your server running is by tweaking this, you will eventually run into trouble. That being said, by tweaking MaxRequestsPerChild you may be able to increase MaxClients as much as 50%.

[pagebreak title='Tweaking Apache Continued']

Apache Further Tweaking
For mixed purpose sites (say image galleries, download sites, etc.), you can often improve performance by running two different apache daemons on the same server. For example, we recently compiled apache to just serve up images (gifs,jpegs,png etc). This way for a site that has thousands of stock photos. We put both the main apache and the image apache on the same server and noticed a drop in load and ram usage. Consider a page had about 20-50 image calls -- the were all off-loaded to the stripped down apache, which could run 3x more servers with the same ram usage than the regular apache on the server.

Finally, think outside the box: replace or supplement Apache

Use a 2nd server
You can use a tiny, lightning fast server to handle static documents & images, and pass any more complicated requests on to Apache on the same machine. This way Apache won't tie up its multi-megabyte processes serving simple streams of bytes. You can have Apache only get used, for example, when a php script needs to be executed. Good options for this are:

TUX / "Red Hat Content Accelerator" -
kHTTPd -
thttpd -

Try lingerd
Lingerd takes over the job of feeding bytes to the client after Apache has fetched the document, but requires kernel modification. Sounds pretty good, haven't tried it. lingerd -

Use a proxy cache
A proxy cache can keep a duplicate copy of everything it gets from Apache, and serve the copy instead of bothering Apache with it. This has the benefit of also being able to cache dynamically generated pages, but it does add a bit of bloat.

[pagebreak title='Replace Apache Completely']

Replace Apache completely
If you don't need all the features of Apache, simply replace it with something more scalable. Currently, the best options appear to be servers that use a non-blocking I/O technology and connect to all clients with the same process. That's right - only ONE process. The best include:

thttpd -
Caudium -
Roxen -
Zeus ($$) -

Solutions: PHP's CPU & RAM Usage
Compiling PHP scripts is usually more expensive than running them. So why not use a simple tool that keeps them precompiled? I highly recommend Turck MMCache. Alternatives include PHP Accelerator, APC, & Zend Accelerator. You will see a speed increase of 2x-10x, simple as that. I have no stats on the RAM improvement at this time.

Solutions: Optimize Database Queries

This is covered in detail everywhere, so just keep in mind a few important notes: One bad query statement running often can bring your site to its knees. Two or three bad query statements don't perform much different than one. In other words, if you optimize one query you may not see any server-wide speed improvement. If you find & optimize ALL your bad queries you may suddenly see a 5x server speed improvement. The log-slow-queries feature of MySQL can be very helpful.

How to log slow queries:

# vi /etc/rc.d/init.d/mysqld

Find this line:

change it to:
SAFE_MYSQLD_OPTIONS="--defaults-file=/etc/my.cnf --log-slow-queries=/var/log/slow-queries.log"

As you can see, we added the option of logging all slow queries to /var/log/slow-queries.log
Close and save mysqld. Shift + Z + Z

touch /var/log/slow-queries.log
chmod 644 /var/log/slow-queries.log

restart mysql
service myslqd restart
mysqld will log all slow queries to this file.

These sites contain additional, more well known methods for optimization.

Tuning Apache and PHP for Speed on Unix -
Getting maximum performance from MySQL -
System Tuning Info for Linux Servers -
mod_perl Performance Tuning (applies outside perl) -

Once again, if this has any errors or important omissions, please let me know and I will correct them.
If you experience a capacity increase on your server after trying the optimizations, let me know!

Article written by spagmoid additions by: albo,huck on the ev1 forums.

  • Rating


Related Articles

Comments (39)

  • Gravatar - Thomas Dawson
    Thomas Dawson 13:19, November 12, 2003
    mke2fs command is not recognized in my ssh. What do I do?
  • Gravatar - Ramprage
    Ramprage 15:22, November 14, 2003
    Use /sbin/mke2fs instead
  • Gravatar - kel
    kel 22:58, November 15, 2003
    How to increase the /tmp space?

    I got this error:

    Warning: Unknown(): write failed: No space left on device (28) in Unknown on line 0

    Warning: Unknown(): Failed to write session data (files). Please verify that the current setting of session.save_path is correct (/tmp) in Unknown on line 0

    Please help as soon.

  • Gravatar - Don Holeman
    Don Holeman 22:18, November 16, 2003
    I just followed these directions, pretty much flawlessly. Got one warning about it not being a block special device when I went to create the partition.

  • Gravatar - simoa
    simoa 02:10, November 17, 2003
    I run a chkrootkit and found this after I secured the /tmp partition
    Checking `aliens'...

    is this normal or I have to worry about?? PLease reply as soon thanks
  • Gravatar - Ramprage
    Ramprage 14:35, November 18, 2003
    The chkrootkit aliens found message is normal when creating a tmp from a file, no worries.
  • Gravatar - mazen
    mazen 23:42, January 6, 2004
    I make the step by step
    when I type mount command I get this message

    root@server [/]# mount -o loop,noexec,nosuid,rw /dev/tmpMnt /tmp
    mount: Could not find any loop device, and, according to /proc/devices,
    this kernel does not know about the loop device.
    (If so, then recompile or `insmod loop.o'.)
  • Gravatar - cruz
    cruz 20:37, January 9, 2004
    3 things.
    Does this break cpanel in its normal use and in updates?
    If I want to revert back to the way it was before doing this modification, how is it done?
    Do I have to reboot the server once this I implement the changes?
  • Gravatar - Mo
    Mo 12:42, January 16, 2004
    use: cp -pR /tmp /tmp_backup
    instead of : cp -R /tmp /tmp_backup

    add -p to preserve file permitions and ownership.
  • Gravatar - Shanlar
    Shanlar 09:50, April 3, 2004
    ok i made my tmp to big, how do i reverse this process? tmp keeps saying device is busy and wont umount...
  • Gravatar - Gary
    Gary 12:15, May 16, 2004
    remember though, this wont save you for losers who run scripts by perl php etc....those commands can still work in /tmp
  • Gravatar - Rona
    Rona 11:32, May 22, 2004
    Can anybody tell me now how to unsecure /tmp because when i install DigiChat it's give me this error
    root@server [/home]# sh ./Install_DigiChat.bin -i console /home
    Preparing to install...
    Extracting the JRE from the installer archive...
    Unpacking the JRE...
    Extracting the installation resources from the installer archive...
    Configuring the installer for this system's environment...

    Launching installer...

    ./Install_DigiChat.bin: /tmp/install.dir.9155/Linux/resource/jre/bin/java: /bin/sh: bad interpreter: Permission denied

    Hope to get help ... :)
  • Gravatar - ca
    ca 18:14, May 23, 2004
    I suppose you should take it out of fstab, do a server reboot and create a tmp directory again.
  • Gravatar - Michael Curtis
    Michael Curtis 16:39, May 24, 2004
    Instructions worked perfect for me other than that one warning about a block device.
    One tip, any path which exists and is owned by 'nobody'... create the directory under /tmp and make a symlink... this can be used to make any path noexec, anywhere on the server... just make sure /tmp has enough space ;)
  • Gravatar - C0NIk
    C0NIk 20:32, June 9, 2004
    well there is a better way for secure the tmp

    root@server [/]# cd scripts ; ./securetmp

    thats all and it well secure the /var/tmp as well

  • Gravatar - GIGI
    GIGI 04:38, June 17, 2004
    look this<br />
    ------------<br />
    root@localhost [~]# sh ./Install_DigiChat.bin -i console<br />
    Preparing to install...<br />
    tail: `-1' option is obsolete; use `-n 1'<br />
    Try `tail --help' for more information.<br />
    ./Install_DigiChat.bin: line 329: [: `)' expected, found -z<br />
    WARNING! The amount of /tmp disk space required and/or available<br />
    could not be determined. The installation will be attempted anyway.<br />
    Extracting the JRE from the installer archive...<br />
    Unpacking the JRE...<br />
    Extracting the installation resources from the installer archive...<br />
    Configuring the installer for this system's environment...<br />
    <br />
    Launching installer...<br />
    <br />
    ./Install_DigiChat.bin: /tmp/install.dir.3158/Linux/resource/jre/bin/java: /bin/sh: bad interpreter: Permission denied<br />
    -------<br />
    pliz help me....
  • Gravatar - Robert Greenwood
    Robert Greenwood 12:14, July 9, 2004
    You will need to increase the size of the mount
  • Gravatar - Foow
    Foow 18:15, August 23, 2004
    can you tell me why you set the /tmp partition to ext2 and not ext3?
  • Gravatar - Mark
    Mark 10:18, October 21, 2004
    i already run /scripts/securetmp from cpanel, but it seem that /tmp still not secured, can i use this tutorial while i already run /scripts/securetmp?
  • Gravatar - JLChafardet
    JLChafardet 08:10, November 3, 2004
    have any one tried this over Plesk Based webservers?<br />
    <br />
    if yes, please let me know if there is any change.<br />
    <br />
    regards,<br />
    <br />
  • Gravatar - Sotek
    Sotek 15:58, November 10, 2004
    change this:<br />
    <br />
    code:<br />
    chmod 0777 /tmp<br />
    <br />
    and put this:<br />
    <br />
    chmod 1777 /tmp<br />
    <br />
    beacuse you need to set the bit who lets everyone write on /tmp
  • Gravatar - kaveh
    kaveh 16:29, December 17, 2004
    hi<br />
    how can i resize the partition ? <br />
    97% full !!!!
  • Gravatar - Big AL
    Big AL 03:11, February 25, 2005
    you can't resize a partition, just move things to /home/tmp and link /tmp to /home/tmp via ln -s /home/tmp /tmp
  • Gravatar - Mateus
    Mateus 20:45, March 7, 2005
    Hi, thanks for this tuto..<br />
    But, i already have this partition <br />
    LABEL=/tmp /tmp ext3 defaults 1 2<br />
    <br />
    my server is in theplanet and is the default mount for them.. how can i modify this ? thank you!
  • Gravatar - Lechoad
    Lechoad 07:11, April 19, 2005
    how do you undo this?
  • Gravatar - netkinetics
    netkinetics 20:19, July 21, 2005
    Change defaults to say noexec,nosuid if you already have a seperate /tmp partition.<br />
    <br />
    Then umount / mount it again. <br />
    <br />
    Also do the same with the /dev/shm mount.<br />
    <br />
    Don't forget to secure /usr/local/apache/proxy, its 777 and owned by nobdoy after every apache build on cpanel servers. Change it to 0400 and owned by root.root , or safely remove it. If you use mod_proxy you should setup another loop device like in this tutorial and mount it to /usr/local/apache/proxy to run your proxy safely. <br />
    <br />
  • Gravatar - rayan
    rayan 12:14, July 25, 2005
    hi<br />
    ist same steps for directadmin control panel?<br />
  • Gravatar - Steve
    Steve 18:22, July 26, 2005
    Yeah it should be the same Rayan, it's not control panel reliant.<br />
  • Gravatar - David
    David 21:55, July 31, 2005
    Can i add the "noexec" just by changing the fstab?
  • Gravatar - complex
    complex 05:02, December 25, 2005
    Just in case anyone wants to do this with FreeBSD, see this page (example 16-5 or 16-7):<br />
    <br />
  • Gravatar - Kapil
    Kapil 20:23, April 1, 2006
    If you have cpanel for freebsd, then just run the script /tmp/securetmp<br />
    <br />
  • Gravatar - Mateus
    Mateus 05:18, May 16, 2006
    Securetmp from cpanel does not work well, it will not protect you /tmp dir, and, this won't protect your tmp from executing Perl Scripts!.. :/
  • Gravatar - M.S.
    M.S. 03:54, June 6, 2006
    Would you whiners stop bitching and pick up a manual? This guy was kind enough to offer you a solution. So STFU. As for the perl/php errors, just edit your php.ini and make sure safe mode is on, sql safe mode is on, and make sure that url_fopen and file uploads are off. you'll be good to go.
  • Gravatar - xkasi
    xkasi 14:27, June 28, 2006
    To increase the /tmp size just redo the tutorial with a bigger count and an different filename.<br />
    e.g: <br />
    dd if=/dev/zero of=tmpMnt2 bs=1024 count=2000000<br />
    <br />
  • Gravatar - Vijay
    Vijay 13:13, October 13, 2006
    Hi,<br />
    <br />
    /scripts/securetmp<br />
    Would you like to secure /tmp & /var/tmp at boot time? (y/n) y<br />
    Would you like to secure /tmp & /var/tmp now? (y/n) y<br />
    Securing /tmp & /var/tmp<br />
    /tmp is already secure<br />
    /var/tmp is already secure<br />
    Process Complete.<br />
    <br />
    But fstab does not show noexec,<br />
    # cat /etc/fstab<br />
    # This file is edited by fstab-sync - see 'man fstab-sync' for details<br />
    LABEL=/tmp /tmp ext3 defaults 1 2<br />
    <br />
    I request others to verify whether following method is correct for CPanel 10.8. ?<br />
    <br />
    1) Because cpanel installed with /tmp saperate partition. - we do not need to change/recreate partition & copy.<br />
    2) modify /etc/fstab to following.<br />
    LABEL=/tmp /tmp ext3 loop,noexec,nosuid,rw 0 0<br />
    3) reboot server.<br />
  • Gravatar - steve
    steve 23:40, January 4, 2007
    Use this to create a 512 MB /tmp parition<br />
    <br />
    dd if=/dev/zero of=tmpMnt bs=1024 count=2097152
  • Gravatar - Daniel
    Daniel 16:56, March 8, 2008
    Which way increase space folder / tmp? <br />
    I created folder / tmp with 100MB. <br />
    Now folder / tmp fills fast. <br />
    I need to increase 2GB. <br />
    Someone can help me?
  • Gravatar - Rick
    Rick 03:51, June 7, 2008
    I just wanted to say thanks for this tutorial! My /tmp partition was created too small, and my /home partition is huge, so I used your tutorial to mount a new tmp partition inside my /home directory. Worked like a charm, thanks a bunch! :)
  • Gravatar - guy
    guy 00:41, September 10, 2008
    so does this just create another tmp folder in /dev rather than /var/tmpDSK or something? how does everything forward to this folder rather than the old just /tmp? I don't quite understand, thanks.

Add Your Thoughts is a hosting directory, not a web host.

Copyright © 1998-2017