Wednesday, January 18, 2006

DD over Netcat for a Cheap Ghost Alternative

Recently I have seen several people ask “How can I clone my hardrive over Ethernet” on several forums that I read. There are several commercial options, such as Nortons Ghost, but I prefer the much cheaper solution of using the GNU DD utility (part of the fileutils package) and redirecting the results through Netcat (the GNU version can be found here). While it is not exactly the fastest solution out there, it is a decent enough of a solution, and I typically am never in a situation where I need to clone quickly through a network.

For the uninitiated, let me explain what these two tools do and how this whole trick plays out. First lets look at DD. From the DD Man page, DD will “copy and convert files”. Now you may be saying “So what”, but this description does not really do DD justice. DD is developed in such a way that is particularly suited to working with block devices, so it has a number of options for conversions. DD can also display the contents of those files to the standard output, or receive input via the standard input. Hard disk devices are usually accessible device files, usually under /dev, under *nix systems. So if I wanted to copy the contents of /dev/hda (usually the first IDE hard disk under Linux) to /tmp/harddisk_image, I could run the following command:

dd if=/dev/hda of=/tmp/harddisk_image

If I omit the “of=” option, it will stream onto the standard output device.  For more information about the options that DD supports, consult the man page. Alternatively, I found this article about using DD for forensic analysis.

Netcat is a tool that is called “the hackers swiss army knife”. That too is a description that I feel does not do this tool justice. The feature I will look at here is its ability to create a basic dummy connection. Basically we will create a simple listening socket on the target, and connect to it on the source system. This linux.com article describes this feature far better than I could, so I will not try to reinvent the wheel. Using the description provided by this article, if I wanted to copy a file from one system to another over a network, I would do something like this:

And on the target system first run
nc –l –p 7000 > file.txt

On the source system I would run:
cat file.txt > nc 192.168.1.120 7000 –q 10

This will do a basic copy of files from one system to another. Now, as I mentioned above, devices in *nix are files. So we are going to use this same principle to copy the contents of 1 hard drive one a machine to another, except instead of redirecting the standard inputs/outputs, we will pipe them to/from DD. To do so I would run the following commands:

On the target system I would run:
nc –l –p 7000 | dd of=/dev/hda

On the source system I would run:
dd if=/dev/hda | nc 192.168.1.120 7000 –q 10

Wait several hours and your good to go. You just saved yourself money on Ghost. There are other options here as well. If you’re worried about bandwidth, you can pipe the streams through gzip to compress the data transfer. So to do this, you would run the following commands:

On the target system:
nc –l –p 7000 | gzip –dfc | dd of=/dev/had

And on the source system:
dd if=/dev/hda | gzip  -cf | nc 192.168.1.120 7000 –q 10

In my opinion this has several advantages. If I have a central server, I can easily make disk images that I can use to remotely re-image a machine in the event of an issue. This works great in Standard Operating Environments, or environments where downtime needs to be cut down to a minimum and other fail over techniques have failed. In order to accomplish this, I would run something like this to create the disk image:

dd if=/dev/had | gzip –cf | dd of=source-192_168_1_120.img (or .zip, or whatever compressed format your comfortable with, such as tarball)

Now, when I need to re-image a machine, I can have the user either put in a special floppy disk or CD with a small image that simply boots and automatically runs the command to image to the target system. On the source system, since I already have a disk image ready to roll, I can run this command to send directly to the target system:

dd if= source-192_168_1_120.img | nc 192.168.1.120 7000 –q 10

I have been using this technique for several years now, and it works great. There are some caveats, of course. First, the target hard disk size must greater than or equal to the source disk size. If the disk is greater, you will have left over space on the drive, since this copy the partition tables exactly as they are. You can use a utility like Partition Magic, or I found this article explaining how to do this with NTFS using Knoppix. I am fairly certain that qtparted will also work on ext2/3 partitions as well. I will experiment with partition resizing and follow up with it. However, partition resizing is done at your own risk, and I would advise trying in a QA environment prior to using in any production environment. Also, if you create a disk image, you have to take into account system updates and patches. On a *nix system, you can mount the image and chroot to apply updates, however if you are using this on a Windows systems, this is not an option.

9 comments:

Anonymous said...

I have tried this method with rhel 4.xx and I cant seem to get it to work. I boot with the linux rescue mode on the target system and the dd through netcat seems to work ok but The system wont boot. after I have "cloned the drives and I do a fdisk -l /dev/sda it shows the same partitions as on the source drive.

any ideas why it is not booting?

thanks

John Ward said...

Have you tried mounting the partitions? Have you tried doing grub/lilo? Can you give me some more details, such as the method you used to boot both systems (source and target)?

Edgunz said...

I have used this successfully to copy a machine directly to another machine without using gzip. When I tried to use gzip to create an image file, the connection resets during the process. This happens at random sizes, there doesn't seem to be any threshold that I cross, I know I have plenty of disk space for the image file. Any ideas on what would cause it to fail? I am using the same ports and the same servers during the process.

Anonymous said...
This comment has been removed by a blog administrator.
Anonymous said...

Netcat v1.10

Download windows version directly from:
http://www.loranbase.com/idx/13/034/article/Netcat-v110.html

Anonymous said...

To the comment above where the new machine won't boot:

Modern Linux boxes need to have an new initrd made for the new machine.

Look here for more help:

http://www.linuxquestions.org/questions/fedora-installation-39/make-new-initrd-image-451904/

Maximo Migliari said...

More recent versions of the netcat command (nc) will not allow the -p and -l options to be used at the same time, so instead of:

nc -l -p 9000 | dd of=/dev/sda


you would type:

nc -l 9000 | dd of=/dev/sda


If you are using nc with dd to transfer an image of a partition from one machine to the other, one of the problems is that dd and netcat won't show you a progress bar of the operation. One solution to this is to install pipe viewer http://www.ivarch.com/programs/pv.shtml by Andrew Wood. It then allows you to pipe the netcat command to the pipe viewer, allowing you to view the progress of the entire operation and for debugging.

target machine:


nc -l 9000 | pv | dd of=/dev/sda


source machine:


dd if=/dev/sda | nc 192.168.0.12 90000

Anonymous said...

Use dd conv=noerror,sync otherwise you'll have problems with bad sectors on your source disk.

Rémi Bouhl said...

Hum.

You wrote your tutorial using the char "–" (hexa code 80 93) instead of "-" (hexa code 2D E2).

I've copied that directly from my browser to my CLI, and got this error:
–l: forward host lookup failed: Unknown host
Things work better with the good dash.

I don't know where the change came from, but thank it may be helpfull to notice it.