UbuntuSolutions

I’ve been wanting to try one one those Virtual Private Server hosting solutions for some time. Today was rainy day, I consequently spent the day on it. I started my search on that railshostinginfo.com and resolved to start my investigation with railsplayground.com. That company offers the cheapest credible deal and has a good reputation on the rails-talk mailing list. The price is $15 per month with a 12 months commitment for 256MB memory (up to 512MB spikes, what that means exactly I don’t know), 10GB disk space, 100GB. bandwitdh.

Getting started: 5 minutes after I entered my CC number, the server was up and running.

I went for the Ubuntu 8.04 server since this is distro I know well.

First impressions:

• I logged in with ssh, the server is very responsive, not the slightest delay when typing.
• Nice web console that allows you to track the status of your server.
• As specified, 10GB disk is available.
• The Ubuntu setup only has the bare essentials, no mysql, no apache, no compiler, no ruby.
• The internet connection appears to be lightning fast.

Memory The ‘free’ command before I started any daemon reports: root@myvps me # free

total used free shared buffers cached Mem: 524288 17860 506428 0 0 0 -/+ buffers/cache: 17860 506428 Swap: 0 0 0

That means 512MB available, but don’t think about relying on swap space.

Setup:

• apt is setup with only the main repository without access to updates, you’ll need to add the universe to /etc/apt/sources.list:

deb http://us.archive.ubuntu.com/ubuntu/ hardy-updates main restricted deb http://us.archive.ubuntu.com/ubuntu/ hardy universe

• “apt-get update” to get your local cache up-to-date
• “apt-get -u upgrade” to upgrade everything

Packages to install with “apt-get install ”:

• mysql-server
• ruby
• build-essential
• wget
• rdoc
• ruby1.8-dev
• libmagick10
• libmagick9-dev
• irb
• libfcgi-ruby1.8
• libmysqlclient15-dev
• libopenssl-ruby

As for the rubygems, I prefer to install it from rubygems website rather than from apt-get. That allows for OS upgrades that don’t destroy the already installed rubygems. After untarring the package, simply run “ruby setup.rb”. Note that it will fail unless rdoc has been setup previously.

After that, install the needed gems:

• rails
• gruff
• mongrel
• willpaginate
• mysql
• rmagick

Database Mysql is necessary but I soon as it’s started, almost 140MB of memory are gone. I thought that maybe I didn’t need innodb for now and I uncommented “skip-innodb” in /etc/mysql/my.cnf. Restarted mysql, memory usage is now down to 55MB. Yay!

Web server I naturally started with Apache that I intended to run with mod_fcgid. But, wow, right after setup with all the default options, Apache takes 250MB. It’s a no go.

Mongrel is a good replacement except that it only gives you one process listening for client requests. On the other hand, the memory footprint is less than 40MB.

I decide to give a try to Lighttpd that I have never used before. That’s a winner, the instructions at http://wiki.rubyonrails.org/rails/pages/Lighttpd are a breeze and it uses just ~25MB per lighttpd process. I don’t need more than 2.

Overall I’m very satisfied, everything is running in less that 115 MB, that leaves 385MB of memory and 9G of disk to spare. My fear that the 256MB VSP plans were just teasers was not justified. I originally just bought one month but I think I’m going to convert that to an annual subscription.

Recently, I bought myself a nice little Acer Aspire One laptop. I swiftly got rid of the dull Fedora variant installed on it and replaced it with Ubuntu 8.04 and then 8.10. I had my share of troubles with the wireless card on the machine, especially with 8.10 and WPA2 networks, must as of now, it all works.

The AAO doesn't have a lot of system resources. 512MB memory and just 8GB of disk. This disk is actually an SSD drive, but it's the slow kind, really really slow especially when writing. As a result the laptop regularly comes to a standstill for a few seconds, with the disk light on solid, before reemerging. Annoying.

Here's what hdparm has to say about the disk:

me@aao # hdparm -i /dev/sda1
/dev/sda1:
Model=SSDPAMM0008G1EA                         , FwRev=Ver2.I0H, SerialNo=CVCP8236821U5
Config={ HardSect NotMFM Fixed DTR>10Mbs }
RawCHS=15636/16/63, TrkSize=32256, SectSize=512, ECCbytes=4
BuffType=DualPort, BuffSize=1kB, MaxMultSect=1, MultSect=?0?
CurCHS=15636/16/63, CurSects=15761088, LBA=yes, LBAsects=15761088
IORDY=yes, tPIO={min:120,w/IORDY:120}, tDMA={min:120,rec:120}
PIO modes:  pio0 pio1 pio2 pio3 pio4
DMA modes:  mdma0 mdma1 mdma2
UDMA modes: udma0 udma1 udma2 udma3 *udma4
AdvancedPM=no
Drive conforms to: Unspecified:  ATA/ATAPI-4,5

Firefox is the application where it happens the most often. Firefox loads a page, it appears to be done, but it then freezes for a few seconds with that damn disk light on.

Why is that? Let's see. First let's make sure that it's Firefox using the disk. The great "pidstat" tool, part of the sysstat package is your friend. Start it up with the -d option to let it monitor disk activity and reports every X seconds which application is reading or writing to disk:

pidstat -d 5

There are always a few applications doing some IO at any point in time. Firefox, pdflush keep showing up in the list, but it's normally less than 4KB in or out per sec and that's fine.

Now load some web pages and keep an eye on the output of pidstat.

16:26:25          PID   kB_rd/s   kB_wr/s kB_ccwr/s  Command
16:26:30         6026      0.80      6.40      2.40  firefox

16:26:30          PID   kB_rd/s   kB_wr/s kB_ccwr/s  Command
16:26:35          181      0.00      2.40      0.00  pdflush
16:26:35         6026      5.60     41.60      0.80  firefox

16:26:35          PID   kB_rd/s   kB_wr/s kB_ccwr/s  Command
16:26:40          181      0.00      5.60      0.00  pdflush
16:26:40         6026     51.20      8.80      0.00  firefox

16:26:40          PID   kB_rd/s   kB_wr/s kB_ccwr/s  Command
16:26:45          181      0.80      1.60      0.00  pdflush
16:26:45         4540      0.00      2.40      0.00  syslogd
16:26:45         5441      0.00      0.80      0.00  wpa_supplicant
16:26:45         6026      3.20    107.20      1.60  firefox

Here you go, the page finishes loading and firefox starts writing to disk making the system unresponsive for a few seconds. It doesn't happen on every page and I noticed the high reading rates are not as bad as the lower write rates. 30KS/s write for 3 seconds is enough to make you feel the pain.

Now, let's find out what Firefox is writing. For that task, strace is what you need. First find the firefox pid. Once you have it, attacch to firefox using strace. strace will pring all the system calls the application makes including IO. What I want to know is what file is opened after a page finishes loading and how much data is written to it. Since there is a crazy amount of noise around the interesting data, I have to filter out of strace all the system calls I don't care about.

strace -p 6026 2>&1 | egrep -v "gettimeofday|poll|EAGAIN|itimer|sigaction|recvmsg|socket|bind|sendto|futex|select"

I try to start that command right when a page finishes loading and the disk light turns solid green and turn it off right after it goes black again.

What I see during those few seconds are a lot of writes to a couple of file descriptors (29 and 44):

write(44, "\r\r\374\0\t\t\244\0\16\302\16,\r\217\fy\v\304\v$\nu\n"..., 4096) = 4096
write(29, "\n\0\0\0\235\3o\0\0040\4E\4Z\4o\4\204\4\231\4\256\4\303"..., 4096) = 4096

Now to find what those file descriptors are:

me@aao # lsof -p 6026 | egrep " 29| 44"
firefox 6026 me   29uw  REG        8,1  1892352 467957 /home/me/.mozilla/firefox/kuz6twft.default/places.sqlite
firefox 6026 me   44u   REG        8,1    94904 467854 /home/me/.mozilla/firefox/kuz6twft.default/places.sqlite-journal

Firefox is writing to 2 sqlite database files. The places database is described at Mozilla Dev Center.
Places is the database where all of your history and bookmarks go. The history can grow very big thus getting the database to several MB or even several tens of MB. It seems every link you ever download, of any kind, text, js, images, everything goes in. That's a lot if inserts and as the database grows, each insert takes more time.

You can't turn off using that database AFAIK, nor limit the number of entries that goes in it. There are other databases that Firefox writes into and that also slow things down: the anti-phishing one and the cookies one. You can, although I don't recommend it, turn off the anti-phishing one: go to about:config, search for safebrowsing.enabled to change it to false.

That leaves you with few options to improve the situation. Here are the options sorted by order of effectiveness:

• change history expiration setting: Go to about:config and reduce browser.history_expire_days
• compact the database from times to times. When Firefox is off:

  sqlite3 "path to database" 'VACUUM;'

• move the mozilla files including the databases to a memory partition. Create a tmpfs partition at /home/yourself/.mozilla. Make a copy of the directory you have on disk at the moment. Create a script to start firefox. This script will start by copying the on disk .mozilla to the on memory .mozilla. Then it starts firefox. When firefox shuts down, the script copies the daata from meemory to disk.

Note that I haven't tried that last option as I'm afraid it would need too much memory on a computer where this resource is already scarce.

 

But the first option that requires reducing "reduce browser.history_expire_days" is simple and works quite well. I set it to 7, that drastically reduces the size of the sqlite database and delays displaying web pages are now rarely above 2 seconds.

 

The acpi bug in /etc/acpi/start.d/90-hdparm.sh that prevented acpi from updating the disks energy saving settings and described here was finally fixed today in ubuntu 8.10 1/15/2009.

The patch is modifies 90-hdparm.sh the following way:

15c15,17
>   if [ "$LMT_CONTROL_HD_POWERMGMT" != 0 ] \
>      && [ -e /var/run/laptop-mode-tools/enabled ]
>   then

The patch was described the following way:

Version 0.114-0intrepid1: * {ac,battery,resume,start}.d/90-hdparm.sh: don't just check whether laptop-mode is configured to control the drives, also check whether laptop-mode itself is *enabled*. Finally closes LP: #59695.

Launchpad bug 59695 was open in 2006 and is the infamous "High frequency of load/unload cycles on some hard disks may shorten lifetime".

Ironically, after this patch, when going into battery mode, acpi will set the APM value (hdparm -B) for your drives to 128. That's a little bit aggressive. On my "Hitachi Travelstar 5K160 series HTS541616J9SA00", that means a spindown after less than 10 seconds of inactivity. And despite the relatime option, the laptop being idle the disk is accessed about once every 10 seconds (firefox, wpa_supplicant seem to be the culprits but that's a subject for another post). As a consequence, the disk spins down and then up at that interval.

Below are the numbers of load cycles increasing as APM is set to 128 (hdparm -B 128 /dev/sda6). 3 new cycles in 22 seconds.

me@laptop ~ $ date && sudo smartctl -a /dev/sda6 | grep 193
Thu Jan 15 20:15:20 PST 2009
193 Load_Cycle_Count        0x0012   084   084   000    Old_age   Always       -       160535
me@laptop ~ $ date && sudo smartctl -a /dev/sda6 | grep 193
Thu Jan 15 20:15:42 PST 2009
193 Load_Cycle_Count        0x0012   084   084   000    Old_age   Always       -       160538

My solution is to edit /etc/acpi/{ac,battery,resume,start}.d/90-hdparm.conf and change 128 into 200:

31c33,35
>         hdparm -B 200 $dev

That's enough, the Load_Cycle_Count doesn't move anymore.

This morning I started mplayer on a mp3 file but got nothing out of it. It hanged and eventually timed out with this error:

[AO_ALSA] alsa-lib: pulse.c:272:(pulse_connect) PulseAudio: Unable to connect: Timeout
[AO_ALSA] Playback open error: Connection refused
Could not open/initialize audio device -> no sound.
Audio: no sound
Video: no video
Exiting... (End of file)

This comes after the latest Intrepid updates (1/4/2009) that includes new pulse audio libs.

The fix is simple:

1. Find the pulseaudio process id (ps -eaf | grep pulseaudio)
2. kill it (-9 is necessary, it won't die with a softer kill)
3. start mplayer again and all will be well.

Got to assume the intrepid update put pulseaudio in an unstable state. It probably did not stop the pulseaudio daemon before replacing the old libs by the new ones...