Sunday, March 12, 2006

Remotely Administering Groups of Servers With Dsh and SSH

This article details using Dsh (the distributed/dancer's shell) and SSH to remotely administer groups of machines.


Dsh is a tool that will allow you to use SSH to run commands or spawn login shells on multiple machines at once. When properly used with ssh-agent, dsh can run unattended commands on many different hosts at once. I assume you have a working su and sudo setup. Just in case, here is a tutorial on configuring and using sudo.

Installing Dsh

On a Debian GNU/Linux-based system, you can install dsh quite simply with apt-get install dsh.

If you aren't using a Debian system, or are on another Unix-based system, you can always install from the dsh sources using the instructions at the dsh home page.

Configuring Dsh

The main dsh configuration file is /etc/dsh/dsh.conf (or $HOME/.dsh/dsh.conf). Here is a working sample (note that this file is installed for you when you install the Debian dsh package):

#default configuration file for dsh. #suppled as part of dancer's shell verbose = 0 remoteshell = ssh showmachinenames = 0 waitshell=0 #remoteshellopt=... # default config file end.

Note: In the default config file supplied with Debian Sarge or Ubuntu Hoary, you will have to change the remoteshell option from rsh to ssh, and the waitshell option from 1 to 0. You can also just cut-n-paste the version listed above.

Other Dsh Configuration Files

There are several other files located in /etc/dsh. Here are their descriptions:

  • /etc/dsh/machines.list: A list of all hostnames or IP addresses, one per line. These are the remote servers you wish to run commands on.
  • /etc/dsh/group/all: This is really a symlink to ../machines.list. You can create this with the command cd /etc/dsh/group && ln -s ../machines.list all
  • /etc/dsh/group/servers: A chosen subset of all the workstation IP's or hostnames in machines.list, one per line. We will actually use the word servers as part of our dsh commands. You can actually choose any name for this file, and can have multiple group files.

Dsh Command Line Switches

The following are the most useful options to dsh:

Switch Description
-a All machines
-g servers Use the group servers
-c Use concurrent connections
-w Wait for one machine to finish before moving onto next
-v Verbose output
-M Show machine name, useful with -c

Using Dsh - An Example

Setting Up SSH and the SSH Authentication Agent

In this example, we will use dsh to update and install any packages available from an apt-get update and apt-get dist-upgrade. Since we need root access on each remote host, we will have to create a public/private key pair on the machine we are launching dsh from, under the root account (using ssh-keygen -t dsa). Next, append the public key you just created (this will be in /root/.ssh/ by default) to each remote host's /root/.ssh/authorized_keys files. Make sure the remote authorized_keys files all have a permission mode of 0600. Once this is done, you should be able to run the remote commands on all the configured remote machines without being prompted for a passphrase. Here are the steps, in detail:

  • Login to your own workstation, use su to become root
  • If you don't already have a public/private SSH keypair for the root user, run ssh-keygen -t dsa as root. Use the default locations/filenames, and be sure to use a secure passphrase
  • Append /root/.ssh/ to the /root/.ssh/authorized_keys file of the other servers you wish to run commands on
  • Make sure the remote authorized_keys files have permissions of 0600 by running chmod 0600 /root/.ssh/authorized_keys on each remote box
  • Exit the root shell you entered with su, above. If you are running in console mode (versus X, which always runs under an ssh-agent in Debian and most other GNU/Linux distributions), type ssh-agent bash (or whatever your preferred shell is) at the shell prompt. This will start another shell under its own ssh-agent process that you can access from the newly spawned shell
  • Type sudo ssh-add, this will add your root private key to the local authentication agent
  • Verify that the identity is added with ssh-add -l

Running Dsh

Now we are ready to actually run dsh. Test dsh access to the workstations you specified in one of your dsh config file groups sudo dsh -M -w -g servers -- 'w'.

You may be prompted to accept the remote host key if this is the first time you are logging in to the remote host(s). We have used the -w switch to ensure we process one host at a time, so that we can answer y to the host key question at each machine in turn. You will not be prompted for a passphrase, since you have cached it in the your local SSH authentication agent.

Assuming the access works (and you should have the output of the w command on all three hosts sent to your console or xterm), let's try running apt-get update and apt-get dist-upgrade on each host. This time, you will not be prompted to accept each remote host key:

sudo dsh -M -g servers -c -- 'apt-get update'

sudo dsh -M -g servers -c -- 'apt-get -y dist-upgrade'

You should see the output of the commands mixed together, as they are all running concurrently. The -M switch helps in this regard, since it prepends the machine name to each output line. When you are done, all machines in the host group serversshould be updated. Because the dist-upgrade may occasionally prompt for your input (despite the -y), or even abort with an error, you may wish to run it first with -u and -s, coupled with the dsh -w switch, to show which packages will be upgraded, while simulating what will happen if the apt-get command is actually run (sudo dsh -M -g servers -w -- 'apt-get -u -s dist-upgrade'. If you want to only download the packages to be updated, but not actually install them, use the -d switch to apt-get.

Sample Dsh Session

In this example, the /etc/dsh/group/servers file looks like this:

dmaxwell@lab0:~$ sudo ssh-add Could not open a connection to your authentication agent. dmaxwell@lab0:~$ ssh-agent bash dmaxwell@lab0:~$ sudo ssh-add Enter passphrase for /root/.ssh/id_dsa: Identity added: /root/.ssh/id_dsa (/root/.ssh/id_dsa) dmaxwell@lab0:~$ ssh-add -l 1024 dd:a9:8f:bd:e8:9e:7e:5d:cc:10:9f:e4:a4:c2:52:22 /root/.ssh/id_dsa (DSA) dmaxwell@lab0:~$ sudo dsh -M -w -g lab -- 'w' 12:58:51 up 239 days, 3:28, 1 user, load average: 0.00, 0.00, 0.00 USER TTY FROM LOGIN@ IDLE JCPU PCPU WHAT dmaxwell pts/0 lab0.turinglabs 03Mar05 16:11m 0.00s 0.00s -bash 12:59:09 up 324 days, 18:07, 1 user, load average: 0.00, 0.00, 0.00 USER TTY FROM LOGIN@ IDLE JCPU PCPU WHAT dmaxwell pts/0 lab0.turinglabs 15Mar05 16:12m 0.06s 0.06s -bash 12:57:06 up 234 days, 3:10, 1 user, load average: 0.00, 0.00, 0.00 USER TTY FROM LOGIN@ IDLE JCPU PCPU WHAT dmaxwell pts/1 lab0.turinglabs 27Apr05 15:47m 0.01s 0.01s -bash dmaxwell@lab0:~$ exit Technorati Tags: , , , ,


Sebas said...

Really nice and useful, I was using things like:

for f in 1 2 3 4 5 6 7 ; do
ssh root@$f apt-get install libqt-perl

But this is much better.

Giantlaser said...

Consider Cluster SSH. Convenient, if you don't mind using xterms. Konsole can also copy input in one tab to all other tabs in the same instance, but Cluster SSH is more flexible for this.

Steven said...

Thanks for the walkthrough (setting up keys, machine.list, groups, etc.), it was really easy to set up! Dsh is going to be super-useful for me... I am trying to build and administer a simple computing cluster