Wednesday, November 28, 2007

Excursions With Find, Xargs, and Perl

It's a common sysadmin task to want to change permissions on all the files and subdirectories under a top-level directory. You could just use the '-R' switch to chmod, but what if your files and directories need different permissions? One scenario that comes up is with shared directories - you have a directory tree that has to be writable by users in a specific group. To do this you want to set the group ID bit on all the directories, so that files created by an individual user are always writable by the entire group (this is numeric permission mode 2775). We want regular files to just have permissions 664.


So we first need a way to differentiate files and directories - one easy way is with the find command, which as a bonus will also recurse into subdirectories for us. Here's our first crack at a solution - let's assume we have changed to the top-level directory we are interested in already:

find . -type f -exec chmod 664 {} \; find . -type d -exec chmod 2775 {} \;
A word of warning - don't try something like "find . -type f | chmod 664 *" - chmod will ignore its standard input and change the permissions on all the files in the current directory. This is easily fixable by just re-running chmod, but it would be a disaster if you were trying to delete only certain files or directories. Anyway, in the command above, the "-type f" and "-type d" output just files and just directories, respectively. The "-exec" will execute the given command on each file or directory produced by find. The special construct "{}" is a placeholder for the current argument, as output by find. These commands will work, but they are very slow on large directory trees, since the chmod is operating on one file or directory at a time. We could try to improve the speed by feeding the entire output of find to chmod:

chmod 664 $(find . -type f) chmod 2775 $(find . -type d)


These last two commands will work fine until we have more than a few dozen files or directories in total - if we do, we'll get the error "/bin/chmod: Argument list too long". That's a cue that we should be using xargs, a very useful command that will submit its input in manageable chunks to the specified command. Here is our next try:

find . -type f | xargs chmod 664 find . -type d | xargs chmod 2775
This is better - the errors about the command line being too long will go away, and this will work, most of the time. But what happens if we have directories or filenames with spaces, quotes or other special characters in them? This comes up quite a bit when you have transferred files from Windows filesystems - the end result will be that xargs will mangle its input, and the command will fail with an error like xargs: unmatched single quote; by default quotes are special to xargs unless you use the -0 option. The error leads us in the right direction, the solution is to use a couple of options to find and xargs that go together: -print0 and -0.

-print0: Find option that prints the full filename to standard output, terminated by a null character instead of a newline.

-0: Xargs option that says input is terminated by a null character, rather than a newline, and all characters are treated literally (even quotes or backslashes).

Here is our final attempt with find and xargs:

find . -type f -print0 | xargs -0 chmod 664 find . -type d -print0 | xargs -0 chmod 2775
This will work for us all the time, no matter what special characters comprise file or directory names.


There are some versions of find that don't support the "-print0" switch. On these systems, you may be able to use a Perl solution:

perl -MFile::Find -e 'find sub { -f && chmod 0664, $_; \ -d && chmod 02775, $_ },"."'
The find procedure exported by the the File::Find module takes two arguments - a callback subroutine and a list of directories. It will recursively descend into the list of supplied directories (in this case just the current directory "."), and run the callback subroutine on each file or directory found. The subroutine in this case is an anonymous one, given by "sub { -f && chmod 0664, $_; -d && chmod 02775, $_ }". It first tests whether the current argument is a regular file, if it is it performs the required "chmod 664". It then tests whether the current argument is a directory, and as you might expect, performs the required "chmod 2775". The variable "$_" represents the current argument, in this case whatever the current file or directory name is. Note also that the numeric permissions must always have a leading zero so that the Perl interpreter knows they are octal numbers.

This solution has the advantage of working on any Unix system that has Perl installed, since File::Find is a core Perl module.

I was curious about how fast each solution ran, here are the timings on a directory tree with 9105 files and 370 directories: time find . -type f -exec chmod 664 {} \; real 0m15.687s user 0m5.676s sys 0m9.877s time find . -type f -print0 | xargs -0 chmod 664 real 0m0.132s user 0m0.036s sys 0m0.080s time perl -MFile::Find -e 'find sub { -f && chmod 0664, $_; },"."' real 0m0.151s user 0m0.080s sys 0m0.056s time perl -MFile::Find -e 'find sub { -f && chmod 0664, $_; \ -d && chmod 02775, $_ },"."' real 0m0.160s user 0m0.064s sys 0m0.076s

The Perl solution was surprisingly fast, very much comparable to the xargs solution. When you consider that the last Perl solution timed tests for both files and directories at once, it is faster than running two xargs commands in a row.

Friday, November 23, 2007

Article Roundup

Some humor from - Bourne shell server pages. Classic:

The basic idea behind all server page technologies is this: rather than writing code that generates an HTML document on-the-fly by writing it out as a series of print statements, you start with a "skeleton" HTML document and embed the code right inside it. Voila! Instead of having a tangled, unreadable, unmaintainable mess of HTML embedded in source code, you have a tangled, unreadable, unmaintainable mess of source code embedded in HTML.

Bourne Shell Server Pages are ordinary ASCII text files, with the special extension .shit, which denotes "Shell-Interpreted Template." The result of invoking the page compiler on a .shit file, is, naturally, a shell script.

and yet...the minimalist in me thinks this might be a good idea...

Didier Stevens wanted to see if people would click on an ad that offered to infect them with a virus. Short version, they did.

Mark Pilgrim expresses his frustrations with Amazon's new ebook reader and DRM.

More humor - what if Gmail had been designed by Microsoft?.

Finally, you can run multiple HTTPS sites off of one IP address with OpenSSL and TLS extensions. You can also do this with mod_gnutls.

Saturday, November 17, 2007

Great Firefox Extension - It's All Text!

I just came across a great Firefox extension called "It's All Text!". Any HTML textarea you see while browsing gets a little edit button on the bottom right corner - clicking it launches your favorite editor (the frst time you use it, it brings you to the preferences screen). For me, that's GNU Emacs.

To use it with Emacs, just add (server-start) to your .emacs and use /usr/bin/emacsclient as your editor in the preferences dialog. Now when you click on the 'edit' button, you'll get a new, empty Emacs buffer to type in. When you are done, type C-x # to close the buffer and get back to the browser. You'll see the contents of the Emacs buffer in the text window. Made a mistake? Clicking 'edit' a second or subsequent time will copy whatever is in the textarea into your editor once again.

Friday, November 16, 2007

Article Roundup

This Code Goes to Eleven asks if adding namespaces to PHP can save it. That question presupposes that PHP is in need of saving - for better or worse, I think PHP is far too widely used at this point to be in danger of extinction. But yes, the lack of proper namespaces in PHP is a royal pain for anything outside of a trivial script.

You have to worry when Bruce Schneier wonders if the NSA put a backdoor in the new PRNG standard.

Gene Simmons is an idiot. Perhaps he should speak to one of these gentlemen. They seem to be doing quite well, despite the evil music downloaders.

Emacs fans who still like to read printed manuals, the GNU Emacs manual for the latest version 22 is finally out in paperback.

TechRepublic talks about alternative Linux desktops. has a good series of articles on desktop FreeBSD.

Thursday, November 15, 2007

Do You Run X on Linux or Unix Servers?

I very infrequently install X11/Xorg on any servers, unless I'm doing an install for a client and they ask for it. My most common server install is a base installation of Debian stable that weighs in at about 300MB. I always thought there was no need for a graphical display on a server, for the standard reasons:

  • The X server uses resources better devoted to key server processes
  • There are security implications to having the additional libraries and binaries on a system
  • The command line is much more efficient when you need to get something done

Of course, you can leave out the X server, and just install the needed X clients. SSH works great with its built-in X forwarding. But you still have a potential security problem to deal with on the server itself - local privilege escalation from an insecure X binary, for example.

It seems things have been changing lately. Memory and CPU are more plentiful, so resources are not as much of a concern as they were even five years ago. Default installs from the commercial Linux vendors install a full-blown graphical desktop, as much as they still offer the choice of a minimal installation. Security will always be an issue, but SELinux and AppArmor ease the concerns for buffer overflows and privilege escalation. And there are some useful graphical tools with features that would be hard to replicate from a shell - Red Hat's virtual machine manager comes to mind. I still refuse to install X on servers, mainly because I'm habituated to years of shell use (hell, even on my desktops I spend a disproportionate amount of time in a terminal or Emacs buffer). There just seems to be less reason not to install X these days, apart from personal preference.

So I'm wondering, do you install X on your servers, or recommend it for your clients or employer? If so, why?

Tuesday, November 13, 2007

Can a small business afford not to run Linux?

There's an interesting article at ITWire on whether or not a small business afford not to run Linux. The conclusion of the author is that small business should be running Linux, both on the desktop and server. One part of the article caught my eye:

I copped some flack from the Windows crowd for some comments in the prequel to this story in which I expressed my dismay at how slow my highly configured computer ran under Windows Home Server...Apparently, this was my fault according to those who serve Redmond. I should have configured and optimised my computer correctly, chosen my security package more wisely, so I was just an idiot and a dumbass who didn't know what he was talking about...Believe it or not, like most people who use computers for work, I don't have time to fiddle around to optimise my computers and network.

This is understandable, small-business owners don't have time to waste tweaking server and desktop settings to get something usable. They want something that works out-of-the-box. Next comes this: come when I partitioned my disk and installed a dual boot Ubuntu 7.10 system without any special tweaking, only then, when I had Linux up and running, did my computer give me the sort of performance I expected from the hardware?

This was the surprising part - for years, Windows advocates have picked on Linux for the need to configure and tweak it - largely true. It's only been the last year or so that Linux distributions like Ubuntu and Fedora have garnered enough hardware and video support to make installation and configuration pretty painless. Witness the automatic printer configuration in Ubuntu 7.10, for example, or the automatic X-configuration that happens now under Xorg.

Monday, November 12, 2007

Wasting TIme With Web 2.0? Say it Isn't So...

Boy, is this ever true. Of course, you don't need Google and Web 2.0 to waste time. Those that like to organize can spend hours doing just that, whether it be digital or paper-based organization (the funny thing is, the standard response to "I don't have enough time!" is usually to get organized). Those of us who are geeks can spend days "organizing" our Linux desktops just so...or fine-tuning our Emacs/Vim configuration to "be more efficient"... or blogging...which reminds me, I better get back to work.

Sunday, November 11, 2007

Article Roundup

I'm not sure why the fact that Hushmail is giving user's data to the Feds is surprising to anyone. First of all, they are complying with a court order. Secondly, if you're using HushMail's servers and trusting their java applets, don't expect too much. If you really want security, send mails through MixMaster only after encrypting them manually from your laptop replete with encrypted swap and drive partitions, while sitting inside a Faraday cage. brings us Basic presentations with LaTeX Beamer.

From Red Hat Magazine, splitting tar acrchives on the fly.

John Wiegley has written a great article on using Emacs as a day-planner with org-mode.

Free Operating Systems: Plenty of Choices Here, People

Apparently, there is some disagreement about whether or not Gobuntu is a 'free enough' operating system. I often wonder about these disputes, there are plenty of truly free operating systems for the taking. Debian without the non-free or contrib repositories would be quite free enough for even the most ardent Free Software advocate. Same with Fedora, which has steadfastly refused to ship with support for proprietary audio or video formats, for example. Likewise for OpenBSD, with it's principled stance on binary firmware and free software in general. If you're that angry about it, why not devote your time and energy to operating systems that already fit the bill?

Thursday, November 08, 2007

Using cURL for FTP over SSL file transfers

I recently helped a client work through some errors while trying to transfer a file over a secure FTP connection (FTP over SSL) with cURL. If you haven't used curl, it is a great tool that lends itself to scripted data transfers quite nicely. I'll quote from the curl website:
curl is a command line tool for transferring files with URL syntax, supporting FTP, FTPS, HTTP, HTTPS, SCP, SFTP, TFTP, TELNET, DICT, LDAP, LDAPS and FILE. curl supports SSL certificates, HTTP POST, HTTP PUT, FTP uploading, HTTP form based upload, proxies, cookies, user+password authentication (Basic, Digest, NTLM, Negotiate, kerberos...), file transfer resume, proxy tunneling and a busload of other useful tricks.

Anyway, using curl with FTP over SSL is usually done something like this: curl -3 -v --cacert /etc/ssl/certs/cert.pem \ --ftp-ssl -T "/file/to/upload/file.txt" \ Let's go over these options:
  • -3: Force the use of SSL v3.
  • -v: Gives verbose debugging output. Lines starting with ">" mean data sent by curl. Lines starting with "<" show data received by curl. Lines starting with "*" display additional information presented by curl.
  • --cacert: Specifies which file contains the SSL certificate(s) used to verify the server. This file must be in PEM format.
  • --ftp-ssl: Try to use SSL or TLS for the FTP connection. If the server does not support SSL/TLS, curl will fallback to unencrypted FTP.
  • -T: Specifies a file to upload

The last part of the command line is simply a way to specify the username, password, host and port all in one shot.

How FTP Works

Before I get to the problem, I need to explain a bit about how FTP works. FTP operates in one of two modes - active or passive. In active mode, the client connects to the server on a control port (usually TCP port 21), then starts listening on a random high port and sends this port number back to the server. The server then connects back to the client on the specified port (usually the server's source TCP port is 20). Active mode isn't used much or even recommended anymore, since the reverse connection from the server to the client is frequently blocked, and can be a security risk if not handled properly by intervening firewalls. Contrast this with passive mode, in which the client makes an initial connection to the server on the control port, then waits for the server to send an IP address and port number. The client connects to the specified IP address and port and then sends the data. From a firewall's perspective, this is much nicer, since the control and data connections are in the same direction and the ports are well-defined. Most FTP clients now default to passive mode, curl included.

The problem

Now, a problem can arise when the server sends back the IP address from a passive mode request. If the server is not configured properly, it will send back it's own host IP address, which is almost always a private IP address and different from the address the client connected to. Usually a firewall or router is doing Network Address Translation (NAT) to map requests from the server's public IP address to the server's internal IP address. When the client gets this IP address from the server, it is trying to connect to a non-routable IP address and the connection times out. How do you know when this problem has manifested itself? Take a look at this partial debug output from curl:

... > PASV < 227 Entering Passive Mode (172,19,2,90,41,20) * Trying
Here the client has sent the PASV command, which asks the server for a passive data connection. The server returns a string of six decimal numbers, representing the IP address (first four digits) and port (last two digits). Here the IP address is - a non-routable IP address as per RFC 1918. When the client tries to connect to this address, it will fail.

The solution...sort of

In 1998 RFC 2428 was released, which specified 'Extended Passive Mode', specifically meant to address this problem. In extended passive mode, only the port is returned to the client, the client assumes the IP address of the server has not changed. The problem with this solution is that many FTP servers still do not support extended passive mode. If you try, you will see something like this:
> EPSV * Connect data stream passively < 500 'EPSV': command not understood. * disabling EPSV usage > PASV < 227 Entering Passive Mode (172,19,2,90,41,20) * Trying

...and we're back to the same problem again.

The Real Solution

Curl has a neat solution to this problem, requiring two additional options. The first is --disable-epsv, which prevents curl from sending the EPSV command - it will just default to standard passive mode. The second is --ftp-skip-pasv-ip, which tells curl to ignore the IP address returned by the server, and to connect back to the server IP address specified in the command line. Let's put it all together:
curl -3 -v --cacert /etc/ssl/certs/cert.pem \ --disable-epsv --ftp-skip-pasv-ip \ --ftp-ssl -T "/file/to/upload/file.txt" \
If this succeeds, you'll see something like this:

* SSL certificate verify ok. ... < 226- Transfer complete - acknowledgment message is pending. < 226 Transfer complete. > QUIT < 221 Goodbye.
The final 226 Transfer complete is the sign that the file was transferred to the server successfully.

Tuesday, November 06, 2007

Happy Birthday VAX!

From Yahoo! news, the VMS operating system just turned 30 years old. Amazing that there are so many VAXen still in use today:
Gareth Williams, associate director of the Smithsonian Astrophysical Observatory Minor Planet Center since 1990, has been tracking the 400,000 orbits of known asteroids and comets in the solar system using a cluster of 12 VAXes, from offices on the Harvard University campus. The Deutsche Börse stock exchange in Frankfurt runs on VMS. The Australian Stock Exchange runs on it. The train system in Ireland, Irish Rail, runs on it, as does the Amsterdam police department. The U.S. Postal Service runs its mail sorters on OpenVMS, and uses it to ship 112,000 packages a day. It has "a very loyal installed base of customers," says Ann McQuaid, general manager of OpenVMS at HP, who shows no signs of wanting to give it up.

I haven't sat in front of a VAX terminal in years; the last time was in the late eighties when I was a CS student at UMASS, Amherst. It was a VAX 11-780, which I did C programming on. I still recall the VAX lab being reserved for junior and senior-year students only, as it was light-years ahead of the horrific Cyber mainframe freshman CS and Engineering students were subjected to.

Sunday, November 04, 2007

BusyBox Developers settle GPL Lawsuit against Monsoon Multimedia

You had to know this was going to happen, although it would have been nice if the GPL were finally proven in court. Perhaps the fact that it has not speaks to the thought that went into the GPL's development over the years. One thing I've noticed about these lawsuits - they are rare. Most cases are settled without an actual legal filing. It's safe to say that free software authors want to write code, not file lawsuits, so I'm guessing the number of frivolous GPL lawsuits is close to non-existent. Clearly the defendant's attorneys are always recommending a settlement rather than a (likely) loss in court.

Update: You can read some interesting commentary on this case at the following blogs:

Sunday Morning Humor

Some light reading, thanks to the alt.sysadmin.recovery Manpage Collection.

Saturday, November 03, 2007

Two Useful Firefox Extensions

I came across Customize Google the other day, a great Firefox extension that improves your Google experience, especially if you are concerned about privacy. Some highlights - it links to competitors and to the Wayback Machine in search results, anonymizes your Google user ID, removes click-tracking, and forces Google apps to use https URL's.

I've also been using the Better Gmail extension for a while, which duplicates some of CustomizeGoogle's Gmail features, plus lots more.

Friday, November 02, 2007

FlickOff: Escaping the Clutches of Web 2.0

A nice article from the latest Linux Gazette on escaping Flickr. The article details how to have pictures from a camera phone auto-posted to a web gallery. I can relate, I had used Yahoo photos for a few years - Yahoo discontinued it earlier this year and provided a migration path to Flickr. I took it, but later deleted my account after being nagged to pay for their pro account. In the end I just set up Gallery on my own web server.

Wednesday, October 24, 2007

Comments on "Switching From OS X to Ubuntu"

Here's an interesting rant by someone who Switched From OS X to Ubuntu and got homesick. Read through the comments - there are replacements for all of his missing applications. This is someone who was used to doing things one way, and never put the effort into learning something new, or looking for alternatives (that's not necessarily bad, just an observation). I've said it before, Linux will make an attractive Desktop when the OEM agreements are broken and you can walk into your local PC chain and buy Linux pre-installed, giving people a chance to get used to it. Otherwise, it's an uphill battle.

Tuesday, October 23, 2007

Serious Businesses Don't Use Ubuntu Because the Release Names are Silly

Ubuntu has an identity crisis, because the development names of its releases are silly. Sheesh. As one commenter points out, take a look at some other silly names. Also, to be on the safe side, you better not use any Intel hardware. Stay way from Apple, their codenames are just too silly for serious IT executives. Hopefully this is just satire and not a drunken rant by some columnist about to miss a deadline (or worse, a sober one).

Tuesday, October 16, 2007

Article Roundup

OK, I have no sympathy for these people. I mean, seriously, why are you using Hotmail for mailing lists?

It's about time the US patent system had an injection of sanity, especially given the recent visit from the patent trolls.

Mark Shuttleworth talks about Ubuntu 7.10, set to be released tomorrow.

John Wiegley brings us neat tricks with iptables

Here are ten funny quotes by Linus Torvalds.

More humor: I love reading the old BOFH episodes.

Sunday, October 14, 2007

BSDTalk - Richard Stallman Interview

BSDTalk has posted a nicely done interview with Richard Stallman. Well worth a listen. While you are there, check out the archived podcasts going back almost two years.

Friday, October 12, 2007

'Insecure by Default'? Well, Yes and No...

Azerblog mentions that Linux distros are insecure by default, specifically that you can boot into single user mode by editing the GRUB kernel line to get to a root shell. Actually, this doesn't work without the existing root password on Fedora, Debian, and Ubuntu (not sure about other Unices or Linux distros). Just before entering single-user mode, you will get a console message "Give root password for maintenance (or type control-D for normal startup)". To actually get a minimal root shell without a password, you need to add init=/bin/bash to the 'kernel' line in the GRUB edit shell. You could also boot from a live CD, or just take out the hard drive and mount it in another computer (all good methods of recovering a lost root password, BTW).

I guess the moral is that if a determined cracker gets physical access to your server, they can pretty much do what they want. I suppose to make things very difficult, you could 1) enable the system's BIOS password, 2) enable the GRUB password, and 3) use encrypted swap and filesystems. All of these would be a real pain if you don't have remote console on a CoLo'd or hosted server, since you would need someone physically present every time the server rebooted or lost power. I imagine that's why many of these security measures are not enabled by default.

Wednesday, October 10, 2007

Comments on 'Security without firewalls'

Debian Administration has an article up about the usefulness of firewalls. Are they really necessary? If you consider a firewall as just a non-stateful, layer-3 packet filter, then I would agree they are not very useful. However, modern firewalls can do all sorts of useful filtering that can protect a public application from compromise - things like stateful fragment reassembly, packet normalization, and rate limiting come to mind. Outbound filtering can also be useful, in the event of an internal compromise, or just as a spam-buster (only allowing outbound SMTP traffic to a mail relay with authentication).

This reminded me of an article I read some time ago by Abe Singer in the Usenix magazine ;login: about life without firewalls at the San Diego Supercomputer Center (SDSC) (PDF). Basically, they do the following:

  • They have a centralized configuration management system; they use only hardened 'reference systems' on any public networks
  • They have implemented aggressive patching policies
  • They enforce a strict policy on strong authentication

How well does this setup work? According to, pretty well. The SDSC has seen one compromise in six years without a firewall, and that one compromise would not have been stopped by a firewall, even if they had one.

Tuesday, October 09, 2007

The Next Leap for Linux - New York Times

The New York Times published a halfway-decent look at Linux from a consumer standpoint several days ago. First, the article is noteworthy in that it mentions software freedom, something very rare in mainstream media considerations of Linux:

But why would anyone want to use Linux, an open-source operating system, to run a PC? “For a lot of people,” said Jim Zemlin, executive director of the Linux Foundation, “Linux is a political idea — an idea of freedom. They don’t want to be tied to Microsoft or Apple. They want choice. To them it’s a greater cause.”

They go on to mention Dell's entry into the consumer PC and laptop market with Ubuntu, then this:

One challenge for Linux users is finding media players that work with encrypted music and DVDs. Ubuntu comes with a movie player, but it is not automatically configured to play copy-protected commercial DVDs. To watch a movie, the Linux user must install necessary codecs, or decoders.

This is wrong on two counts - one, that finding codecs is hard. Yes, it used to be, but isn't any longer. Feisty Fawn, the version of Ubuntu shipping on Dell systems, will automatically download needed codecs for most multimedia formats after some click-through warnings. Second, Windows itself doesn't come with all needed codecs, either - they are downloaded for you in a similar fashion. As for encrypted DVD's, last I checked, Windows will not play encrypted DVD's out-of-the-box either, you need to by commercial DVD viewing software for this (I'm not including the usual spate of cripple-ware on most new Windows PC's, since they are time- and/or feature-limited). At least on Ubuntu, there are clear instructions for dealing with restricted multimedia formats, none of which involve the consumer spending money (similar pages exist for Debian and Fedora, too). Decent iTunes support is perhaps the only thing lacking from Linux these days. Banshee, mentioned in the article, apparently comes close.

Sunday, October 07, 2007

Article Roundup

Wired talks to Rob Malda, the creator of Slashdot on its 10th anniversary.

There's a good series of articles at on desktop FreeBSD.

Those of us still content using mutt to read email will enjoy Daniel Webb's article on The Ultimate Email Setup. brings us how to speed up your Linux desktop.

If you're looking for a more secure alternative to FTP, head over to HowtoForge and read Chrooted SFTP With MySecureShell On Debian Etch.

For those that missed it, read DEranged Security's article on Tor snooping. This was the result (ouch). Remember, Tor only ensures anonymity as long as the content of your traffic stream doesn't give your identity (or more) away. Bruce Schneier has some insightful commentary on the matter as usual.

Friday, October 05, 2007

Updating Your Debian or Ubuntu Desktop Safely

Bruce Byfield wrote an article about the dangers of automatic updates at I agree with him that updates are often dangerous, however, I will say that after many years of updating systems running Debian "stable", I have not encountered any problems afterwards. After all, it's Debian's policy not to include any non-security updates in their stable branch. I don't take any chances, and run the updates manually (see my previous post on Remotely Administering Groups of Servers With Dsh and SSH for a way to run updates on many boxes at once), but still no problems. The one, big weakness of this policy over the years was that it excluded updates for packages that did need them - like Snort or ClamAV. With Debian volatile, this is no longer an issue.

My Ubuntu desktop, however, is another story. I tend not to update it unless I feel it is absolutely necessary - it usually isn't. With no services open to the world, security vulnerabilities will tend to be exposed via web or email, so important updates usually revolve around Firefox (I use mutt for email).

There's a couple of good ways on Debian or Ubuntu desktops to make updates a bit safer: You can select individual updates with the graphical update manager, or use apt-listchanges from a shell prompt. Using the update manager, you can not only select individual updates, you can also display detailed changelogs for the new version, so you can make an informed decision on whether or not to upgrade. Just select a package and click on the "Description of update" arrow.

Ubuntu Updates

Using the command line, let's say you just wanted to upgrade the 'tar' utility. Apt-listchanges gives us a similar functionality to the graphical update manager. First we have to install and configure it. The 'dpkg-reconfigure' step, below, will ask you a bunch of questions - the important ones are to have apt-listchanges display both changelogs and news, and to ask for confirmation before proceeding.

sudo apt-get update sudo apt-get install apt-listchanges sudo dpkg-reconfigure apt-listchanges
My /etc/apt/listchanges.conf is the following, this gets auto-generated after the 'sudo dpkg-reconfigure apt-listchanges' step, but you can certainly edit it by hand:

[apt] frontend=pager email_address=doug confirm=1 save_seen=/var/lib/apt/listchanges.db which=both
Now, after apt downloads the updated tar package, it will display a changelog, and ask for confirmation before updating:

doug@dev:~$ sudo apt-get install tar Reading package lists... Done Building dependency tree Reading state information... Done The following packages will be upgraded: tar 1 upgraded, 0 newly installed, 0 to remove and 112 not upgraded. Need to get 0B/322kB of archives. After unpacking 0B of additional disk space will be used. Reading changelogs... Done ... tar (1.16-2ubuntu0.1) feisty-security; urgency=low * SECURITY UPDATE: directory traversal with malicious tar files. * src/names.c: adjust dot dot checking, patched inline. * References CVE-2007-4131 ... apt-listchanges: Do you want to continue? [Y/n]?

UK Law Demands Decryption Keys for Your Data

The UK has a new law that forces criminal suspects to hand over encryption keys or face jail time. The penalty for refusing is up to five years for terrorism-related investigations or up to two years for other types of criminal investigations. I was immediately struck by the futility of this law - people who are criminally innocent and but use encryption to protect their privacy will surely acquiesce and have their privacy violated. Hardcore criminals will likely take the guaranteed sentence rather than expose what they have been up to, which in many cases would carry a much stiffer penalty.

Perhaps the real hope is that this law gives law enforcement a bit of leverage when trying to elicit confessions or cooperation. In the end, it will just drive legitimate data encryption services out of the UK (the article notes the law doesn't apply to data outside of or in transit through the UK), or force users to some form of security through obscurity. If they don't know it's encrypted, they can't ask for the keys, right?

The US went through a similar debate in the '90s over key escrow; thankfully it never came to pass. The FBI now realizes it is easier to circumvent the encryption entirely.

Thursday, October 04, 2007

Comments on "Why I’m staying with Debian"

Bruce Byfield has a post up in which he tells us why he is staying with Debian. I tend to agree, Debian is still my favorite Linux distro, primarily for its packaging system. Bruce mentions packaging, but also talks about Debian's repositories and unique community.

There are several comments along the lines of "Well, you could replace 'Debian' with any distribution name, and what you said would still be true". I don't agree - although most of the big distros now have large communities and decent packaging systems (Fedora/Yum, e.g.), they still don't compare to Debian in either respect, nor can you ignore the commercial factor. I don't have to worry about the motives of Debian, they tend to include features that make for happy users and sysadmins first. Red Hat, for example, uses Fedora as a testing ground for RHEL. They are up-front about that, but the motive is still there. I do agree that your perspective matters - if I used nothing but desktop Linux, I would probably choose Ubuntu as a favorite. Debian wins overall in my opinion for making my life as a sysadmin and power-user easy. It's a distribution for generalists.

Thursday, September 27, 2007

GNU Screen

I use GNU screen a lot. Nothing beats it for keeping SSH connections open to multiple servers, and it has some killer features if you spend a lot of time at a shell prompt. There is a very good introductory article up at Red Hat Magazine, a guide to GNU Screen. I particularly like their customization of the status line.

One customization I've had in my .screenrc for some time is to replace the normal command-key prefix (ctrl-a) with a single key (backquote), it's much faster: escape ``
If you ever need to type a single backquote (like when you are editing a shell script), you just have to hit the backquote key twice in a row.

I would only add to the article that on Debian or Ubuntu, install screen with apt-get install screen.

Article Roundup

From HowtoForge, Speeding Up Perl Scripts With SpeedyCGI On Debian Etch (a great site, BTW). Simpler and with fewer features than mod_perl, but good for running legacy Perl CGI scripts faster.

Here is a presentation on the new regex engine in coming in Perl 5.10.

From C. Titus Brown, a blog post on writing Python code that doesn't suck. Most of it applies to any language. This interested me because as a Perl coder, I'm constantly dealing with the tired "Perl code is soooo hard to read...but Python, on the other hand...". Really, you can write sucky code in any language. gives us Implementing Quotas to Restrict Disk Space Usage.

Apparently, OpenOffice is a hit on WalMart PC's.

The Simple Dollar gives us 30 Essential Pieces Of Free (and Open) Software for Windows. Missing were Cygwin and WinSCP - the first things I install on any Windows box I get stuck at.

Tuesday, September 25, 2007

What it Means to be a Hacker

A decent article for a change about what it means to be a hacker. You don't usually see commentary like this from the main-stream-media, to them, "hacker" is usually synonymous with the teenage script-kiddie. This about sums it up:

By focusing on the bad apples, Priest says, Madigan was glossing over DefCon's true spirit: smart people getting together to mess around with technology.

"Middle America thinks we're stealing your social security numbers, raping your children and breaking into your bank account," he says. "The reality is, we are the ultimate explorers. We see a technology, and we want to know how it works."

I've been to DefCon once (0x0c), so I can definitely back-up that sentiment. While there was the occasional network nuisance, they were not given any attention. More of a "if you know what you are doing, you won't be bothered by these idiots".

Snowed by SCO

Dan Lyons of admits he was wrong about SCO. If only we could get all journalists to be this honest...

The truth, as is often the case, is far less exciting than the conspiracy theorists would like to believe. It is simply this: I got it wrong. The nerds got it right.

SCO is road kill. Its lawsuit long ago ceased to represent any threat to Linux. That operating system has become far too successful to be dislodged. Someday soon the SCO lawsuits will go away, and I will never have to write another article about SCO ever again. I can't wait.

Saturday, September 22, 2007

Why Linux Hasn't "Made It" to a Desktop Near You

There is a column over at titled 13 reasons why Linux won't make it to a desktop near you. The author's main premise is that Linux will never truly infiltrate the consumer desktop because Linux isn't a "normal" product in the sense that you can easily market and brand it, and even if it were, it is far too complex and there are too many choices for consumers. I think he's way off-base. I've talked about this issue before. What's killing desktop Linux is Microsoft's lock on the OEM market.

Here's a quote from the article:
Even basic things like partitioning, windows managers, file managers and software update processes are not standardized across our shortlist of user-friendly Linux distros. To varying degrees, you will strike problems getting Linux set up correctly if your PC has an LCD screen that is large or wide, or if you have a fancy graphics card (NVIDIA or ATI) or you want to set up WI-FI or play video clips out of the box.

And if you're installing Linux on the same hard drive as Windows XP, you'll need to create a new partition or two. That's a knee trembler for simple users, a leap of faith of the white knuckle kind. It's a good idea to make full backups before you do this, yet the process can be quite straightforward. For example, Ubuntu offers to shrink your Windows partition to your chosen size and to create the additional partitions you need automatically.

It's not that it's hard, just that it's unfamiliar. Linux doesn't know about C, D and E drives and Windows will show up as sda1/dev or hda1/dev in the partitioning table. What's missing is a simple explanation of these basics, and none of the Linux desktops provide that. You're traveling in a foreign country and you have trouble reading the road signs, and there's no helpful traffic cop to be found. It spoils your trip.

Comparing the ease of use of Windows with Linux by saying "Linux is too difficult to install" misses the point - that few users ever have to install Windows. Their PCs come pre-loaded with the operating system. You could replace "Linux" with "Windows" in the above excerpt with the terms reversed and it would still make sense, viewed from the eyes of a non-Windows user.

Similarly, the issue of "too much choice" is a meaningless. Another quote:

On closer inspection, you find that there are 500 versions of the product. When you try to understand the subtle differences between them, you become confused. Your enthusiasm starts to flag.

If say, Ubuntu Linux came pre-installed on consumer laptops, the issue of choice is now "which model laptop do I buy?". Yes, you might have different laptop manufacturers offering different distributions of Linux, but most consumers won't use that fact to decide which laptop to buy. They will primarily look at the hardware support or the company's reputation, not the technical particulars.

My point above about replacing "Linux" with "Windows" above pertains to most of the article, really. Those of us who have been comfortably using Linux desktops for years read articles like this and immediately see the problem - the articles are always written from the point of view of a Windows user. It's an example of confirmation bias - you have a belief that Linux will never make inroads into the consumer desktop, so you look for theories that affirm this belief ("Linux is too complicated", "The support sucks", "Who can install Linux, anyway?"), and ignore that fact that "YourOS" suffers from the same problems.

There are some good points in the article, but they don't really impact Linux's future on the desktop. For example:

When you discover that some of the designers have made deals with their biggest competitor, the last drop of your enthusiasm drains away.

Obviously a poke at Novell, but I don't see Novell's deal with Microsoft impacting the OEM market for consumer PCs. Novell is after the business desktop.

In the end, I still contend that Microsoft's OEM agreements and monopolistic practices are what is preventing desktop Linux from taking hold. This is the simplest explanation, and I'm not sure it will change anytime soon. The Dell/Ubuntu offering is a good start, but you won't see these laptops in stores, and they are not linked from Dell's main site that the average consumer is likely to buy from.

Don't Mess With Your Sysadmin

A funny reminder not to mess with your sysadmin. Reminds me of the BOFH stories.

Thursday, September 13, 2007

Counting Words in Files With HTML Markup

I write blog posts with HTML markup, and I sometimes want to get a fairly accurate word count of my posts. By accurate I mean that HTML tags themselves as well as quoted values are not counted as words. There are a lots of utilities and scripts that do word counting, from the venerable Unix 'wc' to an elisp subroutine in the FSF's An Introduction to Programming in Emacs Lisp. The ones I looked at all suffered from the same problem - they counted markup as 'words'. If there was some way to strip out or ignore markup, the various methods of word counting would work.

First I tried a few ready-made utilities. The Unix text-mode browser lynx has a 'dump' option that will output formatted text content from a given html file (lynx -dump -nolist foo.html), however, it outputs formatted text, and some of the formatting markup is itself counted as a word by the 'wc' utility. w3m is similar in its output, so has the same problems. I found a Debian package called unhtml that seemed to do what I wanted, but after experimenting a bit with it, I found that it could not handle multiple opening and closing tags on the same line (it counted them as one tag, meaning any real words in that line were skipped). Thinking I might have to write my own utility, I set out to not reinvent the wheel and did a CPAN search - and had success on the first hit. After a few tests I found that HTML::Strip did indeed handle multiple tags on a line as well as HTML comments and values properly.

The next step was to write a wrapper around HTML::Strip for command line use. After a bit of hacking, I came up with From the script header:
Script that strips HTML tags from text. It uses HTML::Strip to do the real work; this is a wrapper around that module that allows you to specify command line arguments - standard input/output is assumed if no args are given. If only one arg is given, it is assumed to be the input pathname.

Requires HTML::Strip (perl -MCPAN -e 'install HTML::Strip' as root on any Unix-based OS will work).

Examples (the following have equivalent results): < foo.html > foo.txt foo.html > foo.txt foo.html foo.txt

I also needed a way to integrate this into Emacs, here is an elisp snippet you can put in your .emacs (don't forget to modify the path to the script):
(defun word-count nil "Count words in region" (interactive) (shell-command-on-region (point) (mark) "/home/dmaxwell/bin/ | wc -w")) (global-set-key "\C-c=" 'word-count)

As a bonus, it also handles XML and SGML properly. To use it while editing, just type C-x= to get a word count of the current region (use C-xh to make the region the entire buffer), minus HTML tags.

Tuesday, August 28, 2007

Article Roundup

Dan Meissler has written a good tutorial on tcpdump. gives us Pain-free disk space management with LVM.

Red Hat Magazine has a couple of good tips on getting a progress report from 'dd' and changing lots of passwords at once.

Five Ubuntu community servers were compromised.

Anyone stuck on Windows can now have the option of using a tabbed terminal emulator.

Thursday, July 26, 2007

Article Roundup

A bit of nostalgia, and a cool community of *nix hackers in the truest sense - The SDF public access Unix system celebrated it's 20th anniversary last month. SDF runs on a cluster of NetBSD/Alpha machines, maintains a text-mode BBS and one of the few remaining gopher servers (users have access to their own gopher-space). Basic shell accounts are free, but you can upgrade your experience with some minimal dues.

Speaking of Gopher, Cameron Kaiser gives us Down the Gopher Hole, some history and links to still-current Gopher sites (Firefox handles them best).

A fascinating article on anti-forensics, how criminals and black-hats are foiling standard computer forensic techniques. Law Enforcement is already changing their tactics.

Keeping the flame-wars alive. How a Vim User Converts to Emacs.

A Windows guy tries Ubuntu.

Monday, June 04, 2007

Article Roundup

Dan Martin brings us Things I can do in Linux that I can't do on Windows.

I agree with this one - Greylisting is Bad. I've used greylisting on production servers, the loss of "instant" communication is the biggest pain, especially when it comes to business email.

Why I haven't bought a commercial PC in years. I buy used or bare-bones systems and install the OS myself.

A very nice Bash FAQ, more like a Bash cookbook.

An interesting screenshot of Donald Knuth's desktop.

Don Hopkins brings us a lesser-known essay by RMS, on the AI Lab and Lisp-machine wars.

Thursday, May 24, 2007

Dell Now Selling Ubuntu Pre-Installed

This is good news - from Dell's Linux-announce mailing list: from: Matt Domsch to: date: May 24, 2007 4:42 PM subject: Dell Desktops and Notebooks with Ubuntu 7.04 now available They're here! Today we're unveiling our three consumer systems -- the XPS 410n and Dimension E520n desktops and the Inspiron E1505n notebook -- with the Ubuntu 7.04 Linux distribution factory installed. Available today in the U.S., the systems target the Linux enthusiast community and are a direct result of your feedback on IdeaStorm[1]. It's true... you will be able to customize and purchase Dell's Ubuntu systems at Just as you asked, the base price for each system is competitively priced and fully configured. Hardware support is available through normal Dell support channels, along with the option of selecting Ubuntu-specific support through Canonical, and a variety of software support options are available. has all the details about the configurations and the support options. And check out today's post on Direct2Dell[2] to watch a vlog from Dell's Linux Engineering team sharing their perspectives on today's announcement. Be sure to stop by and tell us what you think on Direct2Dell[3]. [1] [2] [3] -- Matt Domsch Software Architect Dell Linux Solutions & Linux on Dell mailing lists @

Sunday, April 08, 2007

Debian GNU/Linux 4.0 (Etch) Released

Finally, Debian Etch has been released!

  • Graphical installer (type 'installgui' at the installer prompt)
  • Support for encrypted partitions at install time
  • SELinux support (not enabled by default)
  • Secure APT
  • Fully automated installs via preseeding
  • Debian volatile is now an official service (for those who use snort, clamav, or any other packages that need frequent updates)
  • The usual major desktop software: KDE 3.5, GNOME 2.14, Xfce 4.4, X.Org 7.1, 2.0.4a

Release notes
Debian 4.0/Etch downloads via bittorrent

Saturday, March 10, 2007

Article Roundup

Dale Dougherty from O'reilly wonders if we are winning the war on spam.

The state of California introduces legislation to require open document formats.

Coding Horror gives us tips on reducing your website's bandwidth usage.

Red Hat Magazine gives us some handy sysadmin tips and Understanding Red Hat Daemons.

Tuesday, February 27, 2007

Microsoft Improving the Enduser Experience

I still haven't figured it out. Those who know me know I am a GNU/Linux advocate, so when I see news on how Microsoft is working on making their end users experience better, I have my doubts on who Microsoft is really looking out for. Ok, I’ll give Microsoft the benefit of doubt, I'm all for a better user experience. Tell me, how is it Microsoft is making the end user experience better? Are they improving the performance of the new Vista release? Maybe making their software a little more secure? No? What’s that you say? Microsoft has a new classification of user? Hmm… ok… what’s the new classification? “Maybe a pirate.” Awesome! However... how does that make my experience using a piece of software better? I would think that being labeled ‘maybe a pirate’ would make me feel that Microsoft ‘maybe doesn’t get it.’

This new user 'experience' helps user of Microsoft Windows by having a dialog box appear indicating your new status of 'maybe' being a software pirate, and gives you the option to help correct the problem, which is, according to Microsoft, usually attributed to a system or network error. How does Microsoft help you, other than labeling you as 'maybe a pirate'?

Microsoft would have instantly labeled you as a pirate of their software if there was a glitch with the validation of your install of Microsoft using the Genuine Advantage service. Instead of instantly labeling you as a pirate Microsoft will now display a dialog box indicating your new status of 'maybe being a pirate' and gives you an opportunity to click through to help diagnose why the Genuine Advantage Notification wasn't able to validate you as a rightful user (remember, you don't own the software that you purchased, you simply own the right to use the software as Microsoft deems you should use it). Granted, you can also either ignore or suppress the messages so you don't see them anymore, which the majority of users will select to do. After all who wants to see themselves labeled as 'maybe a pirate' every time they turn on their computer? However, what is the probability of Microsoft changing it's mind on the people that suppress the warning as being relabeled as a pirate instead of 'maybe' being a pirate/

Don't get me wrong… if you're going to use a Microsoft product, I think you should pay for it. If instead you decide you don't want to pay for the privilege of using Microsoft’s products how Microsoft (not you) decides, there are plenty of alternatives that treat you, the end user, with the respect that you deserve.

Creating Large (>2GB) DVD Backups Under Linux

I recently created a large tgz backup archive (4.2GB) and wanted to burn it to DVD. No problem, I thought, standard DVD's hold 4.7GB. But the Gnome desktop CD/DVD Creator kept silently failing. I jumped down to command line, and tried to manually make an iso image with mkisofs. I kept getting this error:
mkisofs: Value too large for defined data type. File backup.tgz is too large - ignoring
After some digging, I found that mkisofs won't handle files greater than 2GB. There is a workaround, however - growisofs can burn files directly to a device, skipping the ISO9660 filesystem creation step (I'm assuming here that /dev/dvd is a symlink to your real DVD burner. If not, the real device name could be /dev/hdc or /dev/scd0, for example. A 'dmesg | grep -i dvd' should tell you what device to use):

growisofs -dvd-compat -Z /dev/dvd=backup.tgz
This worked fine, however now the files on the DVD have to be accessed directly, as if they were on a tape device (you can't mount the DVD, since there is no filesystem on it). Here is how you can test the DVD you just created, if it works you should see the list of files in the tar archive displayed on standard output:

tar tzvf /dev/dvd

Wednesday, February 21, 2007

Article Roundup

The Simple Dollar gives us Live Free: Seven Pieces Of Open Source Software That Transformed My Life.

A couple of good PostgreSQL articles, PostgreSQL for MySQL users and Postgres for the Win!.

Jim Sampson gives up on Linux after trying for 10 years. I have to say, I've been using Linux for more than 10 years, and never thought of giving it up. He does have a point about Evolution/Exchange connector simply not working...but I say get rid of Exchange, not Evolution.

Geeks who are crossword puzzle fans will have fun with this, from the latest Linux Gazette.

Ah, nothing like a good flame war.

HP15C Goodness

At UMASS Amherst in the late 80's it was pretty common for Engineering and CompSci students to buy HP calculators for their math and physics classes. My freshman year, I bought an HP15C and promptly got hooked on RPN, doomed to forever hate calculators that limited the user to "algebraic entry". So I was happy to resurrect my HP calculator with a new set of batteries (Energizer 357 silver oxide batteries were the cheapest I found at about $3 each). The last set lasted literally years, but finally died last year and I'd been putting off getting new ones. You've got to love a piece of hardware that lasts for over 20 years with just battery changes. Oh, and for those that used to love RPN, but don't have an old HP laying around, Emacs has a wonderful Calc mode.