Outage Debriefing

There was a break-in this past Saturday on the machine that hosted CocoaDevCentral, Tree House Ideas, this site, and others. I first noticed something was up when I saw about 1000 bounces for spam advertising Hotmail Brazil. In fact, the whole purpose for the break-in seemed to be to send spam.

I did some digging around and found that the recently-discovered phpbb vunerability played a role. I didn't even know phpbb was still installed on the machine, but that's how the firewall was effectively circumvented. The box that was compromised has been decommissioned and replaced by a new FreeBSD server.

A large portion of the time involved in bringing everything back up was due to migrating the existing environment from Linux to FreeBSD.

Fair warning: the rest of this will probably be relevant to only to unixians.


All of the servers we (myself and a friend I've worked with for 10 years) have used up to this point have run Red Hat Linux. This wasn't the result of intensive analysis, but was simply a low-cost solution. However, maintaining a Red Hat installation is not what I consider straightforward.

The primary tool one has for software management on Red Hat is RPM. In the versions I've used, the biggest challenge with RPM is the dependency tree. For example, if you want to upgrade PHP, you need to personally upgrade all of the packages that it depends on, as well as all of their dependencies, and so on. This is interesting work only if your goal is to be a full-time sysadmin. :)

You also need to find the RPMs that specifically apply to your version of your distribution. So if you have Red Hat 7, it's potentially quite hard to find an RPM that will install PHP5 correctly on your system. At some point, Red Hat introduced a service to manage this for you, but I wasn't able to get it to work consistently. I believe there was also a subscription fee.


My solution to all of this was to basically ignore RPM and compile software by hand. This takes time, but is at least a known quantity.

Support for 'standard' Red Hat Linux was dropped last April, so there was no easy way to get necessary patches. So we found ourselves at a crossroad following the break in. The current counterpart for us was Red Hat Enterprise Linux ES, which appears to run at least $350 per machine. But even the processing of buying it is complicated. We'd be paying $1100 for something we didn't really want.

There are dozens of other linux distributions available, but it's hard to weed out the good ones. Everyone has their favorite, and recommendations sometimes come from a perspective of politics rather than getting something that works. Fedora Core -- which is to Red Hat as Darwin is to Apple -- was an option too, though it wasn't clear to me how closely it matched what we needed.

Bottom line: I just didn't have time to play around with each distribution to find the right one. The sheer quantity of linux distributions produces a constant background noise. For this and a variety of other reasons, I went with FreeBSD instead. Even given the fact it took several days to transition everything over, I'm satisfied with the decision.


The FreeBSD 5.3 installer is not nearly as polished as that of Red Hat 9, however. The former is a character-mode program that actually asked me to consider the geometry of a hard disk and LBA addressing for the first time in a long time. It turns out the disk geometry issue was largely a cosmetic bug, though I couldn't help but feel like it was MS-DOS all over again. By contrast, Red Hat's installer is a GUI app that handles these details quite well.

Configuring FreeBSD after installation is not anywhere near as easy as Mac OS X, but much more consistent and clear than what I've seen with Red Hat over the years. My experience with SunOS likely contributes to this opinion, though I've spent more time with Linux overall. I find FreeBSD ports and packages a substantial improvement over Red Hat RPMs. There's also a much stronger theme of solid security policies and tools throughout BSD.

I'll probably be writing about my experiences more, but so far it seems that FreeBSD is much more in the spirit of what I'm looking for in a server platform.

Comments Temporarily Disabled

I had to temporarily disable comments due to spam. I'll re-enable them soon.