Saturday, May 06, 2017

Well, this was Unexpected

Nine years ago I stopped updating this blog and redirected it to a self-hosted wordpress site, which has been offline for just about as long. Some of the more popular tutorials I eventually converted and put on a static site, where they still remain. My blogger account then slipped from memory imagine my surprise when I happened upon my account settings and saw all my old posts from 2006-2007. I merely had to remove the defunct site redirect and the old blog was back online. Why do this? Well, mainly for historical reasons, I find it interesting reading the commentary and following the (working) links to other old blogs about what was happening in the FOSS community back then. No doubt there are even a few still-useful tips buried in here somewhere, but I expect most will not be unless you maintain some very old Linux or Unix systems.

Monday, February 18, 2008

Blogging Hiatus & Random Thoughts

I've been taking a break from blogging lately (as you can see from my last post). Blogs are interesting things. I'm not sure blogging itself was a very new or even ingenious idea; I think advances in search, coupled with ideas like tagging and the accessibility of a pure browser-based content medium made the idea of blogging more appealing (You mean someone might actually read what I write?). I had a PHP-Nuke-based pseudo-blog back in 2000-2001; I used it as a way to distribute software and howtos. There were certainly many online journals prior to that, they just weren't called blogs.

From a purely utilitarian standpoint, Usenet is a much nicer medium for conversations (Blogger's posting model is particularly bad). However, if you want to get noticed and perhaps make some money from what you write, blogs are it. Blogging has also changed the political landscape forever. I came across this decent comparison of blogs and Usenet recently (noted with some irony that I'm linking to a blog and not a Usenet post).

Anyway, just some musings, I'll have some more posts up shortly...

Wednesday, November 28, 2007

Excursions With Find, Xargs, and Perl

It's a common sysadmin task to want to change permissions on all the files and subdirectories under a top-level directory. You could just use the '-R' switch to chmod, but what if your files and directories need different permissions? One scenario that comes up is with shared directories - you have a directory tree that has to be writable by users in a specific group. To do this you want to set the group ID bit on all the directories, so that files created by an individual user are always writable by the entire group (this is numeric permission mode 2775). We want regular files to just have permissions 664.


So we first need a way to differentiate files and directories - one easy way is with the find command, which as a bonus will also recurse into subdirectories for us. Here's our first crack at a solution - let's assume we have changed to the top-level directory we are interested in already:

find . -type f -exec chmod 664 {} \; find . -type d -exec chmod 2775 {} \;
A word of warning - don't try something like "find . -type f | chmod 664 *" - chmod will ignore its standard input and change the permissions on all the files in the current directory. This is easily fixable by just re-running chmod, but it would be a disaster if you were trying to delete only certain files or directories. Anyway, in the command above, the "-type f" and "-type d" output just files and just directories, respectively. The "-exec" will execute the given command on each file or directory produced by find. The special construct "{}" is a placeholder for the current argument, as output by find. These commands will work, but they are very slow on large directory trees, since the chmod is operating on one file or directory at a time. We could try to improve the speed by feeding the entire output of find to chmod:

chmod 664 $(find . -type f) chmod 2775 $(find . -type d)


These last two commands will work fine until we have more than a few dozen files or directories in total - if we do, we'll get the error "/bin/chmod: Argument list too long". That's a cue that we should be using xargs, a very useful command that will submit its input in manageable chunks to the specified command. Here is our next try:

find . -type f | xargs chmod 664 find . -type d | xargs chmod 2775
This is better - the errors about the command line being too long will go away, and this will work, most of the time. But what happens if we have directories or filenames with spaces, quotes or other special characters in them? This comes up quite a bit when you have transferred files from Windows filesystems - the end result will be that xargs will mangle its input, and the command will fail with an error like xargs: unmatched single quote; by default quotes are special to xargs unless you use the -0 option. The error leads us in the right direction, the solution is to use a couple of options to find and xargs that go together: -print0 and -0.

-print0: Find option that prints the full filename to standard output, terminated by a null character instead of a newline.

-0: Xargs option that says input is terminated by a null character, rather than a newline, and all characters are treated literally (even quotes or backslashes).

Here is our final attempt with find and xargs:

find . -type f -print0 | xargs -0 chmod 664 find . -type d -print0 | xargs -0 chmod 2775
This will work for us all the time, no matter what special characters comprise file or directory names.


There are some versions of find that don't support the "-print0" switch. On these systems, you may be able to use a Perl solution:

perl -MFile::Find -e 'find sub { -f && chmod 0664, $_; \ -d && chmod 02775, $_ },"."'
The find procedure exported by the the File::Find module takes two arguments - a callback subroutine and a list of directories. It will recursively descend into the list of supplied directories (in this case just the current directory "."), and run the callback subroutine on each file or directory found. The subroutine in this case is an anonymous one, given by "sub { -f && chmod 0664, $_; -d && chmod 02775, $_ }". It first tests whether the current argument is a regular file, if it is it performs the required "chmod 664". It then tests whether the current argument is a directory, and as you might expect, performs the required "chmod 2775". The variable "$_" represents the current argument, in this case whatever the current file or directory name is. Note also that the numeric permissions must always have a leading zero so that the Perl interpreter knows they are octal numbers.

This solution has the advantage of working on any Unix system that has Perl installed, since File::Find is a core Perl module.

I was curious about how fast each solution ran, here are the timings on a directory tree with 9105 files and 370 directories: time find . -type f -exec chmod 664 {} \; real 0m15.687s user 0m5.676s sys 0m9.877s time find . -type f -print0 | xargs -0 chmod 664 real 0m0.132s user 0m0.036s sys 0m0.080s time perl -MFile::Find -e 'find sub { -f && chmod 0664, $_; },"."' real 0m0.151s user 0m0.080s sys 0m0.056s time perl -MFile::Find -e 'find sub { -f && chmod 0664, $_; \ -d && chmod 02775, $_ },"."' real 0m0.160s user 0m0.064s sys 0m0.076s

The Perl solution was surprisingly fast, very much comparable to the xargs solution. When you consider that the last Perl solution timed tests for both files and directories at once, it is faster than running two xargs commands in a row.

Friday, November 23, 2007

Article Roundup

Some humor from - Bourne shell server pages. Classic:

The basic idea behind all server page technologies is this: rather than writing code that generates an HTML document on-the-fly by writing it out as a series of print statements, you start with a "skeleton" HTML document and embed the code right inside it. Voila! Instead of having a tangled, unreadable, unmaintainable mess of HTML embedded in source code, you have a tangled, unreadable, unmaintainable mess of source code embedded in HTML.

Bourne Shell Server Pages are ordinary ASCII text files, with the special extension .shit, which denotes "Shell-Interpreted Template." The result of invoking the page compiler on a .shit file, is, naturally, a shell script.

and yet...the minimalist in me thinks this might be a good idea...

Didier Stevens wanted to see if people would click on an ad that offered to infect them with a virus. Short version, they did.

Mark Pilgrim expresses his frustrations with Amazon's new ebook reader and DRM.

More humor - what if Gmail had been designed by Microsoft?.

Finally, you can run multiple HTTPS sites off of one IP address with OpenSSL and TLS extensions. You can also do this with mod_gnutls.

Saturday, November 17, 2007

Great Firefox Extension - It's All Text!

I just came across a great Firefox extension called "It's All Text!". Any HTML textarea you see while browsing gets a little edit button on the bottom right corner - clicking it launches your favorite editor (the frst time you use it, it brings you to the preferences screen). For me, that's GNU Emacs.

To use it with Emacs, just add (server-start) to your .emacs and use /usr/bin/emacsclient as your editor in the preferences dialog. Now when you click on the 'edit' button, you'll get a new, empty Emacs buffer to type in. When you are done, type C-x # to close the buffer and get back to the browser. You'll see the contents of the Emacs buffer in the text window. Made a mistake? Clicking 'edit' a second or subsequent time will copy whatever is in the textarea into your editor once again.

Friday, November 16, 2007

Article Roundup

This Code Goes to Eleven asks if adding namespaces to PHP can save it. That question presupposes that PHP is in need of saving - for better or worse, I think PHP is far too widely used at this point to be in danger of extinction. But yes, the lack of proper namespaces in PHP is a royal pain for anything outside of a trivial script.

You have to worry when Bruce Schneier wonders if the NSA put a backdoor in the new PRNG standard.

Gene Simmons is an idiot. Perhaps he should speak to one of these gentlemen. They seem to be doing quite well, despite the evil music downloaders.

Emacs fans who still like to read printed manuals, the GNU Emacs manual for the latest version 22 is finally out in paperback.

TechRepublic talks about alternative Linux desktops. has a good series of articles on desktop FreeBSD.

Thursday, November 15, 2007

Do You Run X on Linux or Unix Servers?

I very infrequently install X11/Xorg on any servers, unless I'm doing an install for a client and they ask for it. My most common server install is a base installation of Debian stable that weighs in at about 300MB. I always thought there was no need for a graphical display on a server, for the standard reasons:

  • The X server uses resources better devoted to key server processes
  • There are security implications to having the additional libraries and binaries on a system
  • The command line is much more efficient when you need to get something done

Of course, you can leave out the X server, and just install the needed X clients. SSH works great with its built-in X forwarding. But you still have a potential security problem to deal with on the server itself - local privilege escalation from an insecure X binary, for example.

It seems things have been changing lately. Memory and CPU are more plentiful, so resources are not as much of a concern as they were even five years ago. Default installs from the commercial Linux vendors install a full-blown graphical desktop, as much as they still offer the choice of a minimal installation. Security will always be an issue, but SELinux and AppArmor ease the concerns for buffer overflows and privilege escalation. And there are some useful graphical tools with features that would be hard to replicate from a shell - Red Hat's virtual machine manager comes to mind. I still refuse to install X on servers, mainly because I'm habituated to years of shell use (hell, even on my desktops I spend a disproportionate amount of time in a terminal or Emacs buffer). There just seems to be less reason not to install X these days, apart from personal preference.

So I'm wondering, do you install X on your servers, or recommend it for your clients or employer? If so, why?