Big Day for BIRT

Yesterday was quite a day for the BIRT community. BIRT 2.0 is finally out, with quite a few new enhancements. Also, thanks to the folks over at EclipseZone, I was pointed to this article announcing broad support by Actuate, IBM, Pentaho, Scapa Technologies and Zend for BIRT. Interesting news, and congratulations to the BIRT team for all their hard work.

EclipseZone Article Published

My article introducing BIRT has been posted to the EclipseZone website. I am pretty psyched about this since this is the first article I have written for another site. I am surprised to see this article posted on JavaLobby also. It’s good to see that the work I put into it come into fruition, despite the many ups and downs like losing my awesome freelance editor.

DD over Netcat for a Cheap Ghost Alternative

Recently I have seen several people ask “How can I clone my hardrive over Ethernet” on several forums that I read. There are several commercial options, such as Nortons Ghost, but I prefer the much cheaper solution of using the GNU DD utility (part of the fileutils package) and redirecting the results through Netcat (the GNU version can be found here). While it is not exactly the fastest solution out there, it is a decent enough of a solution, and I typically am never in a situation where I need to clone quickly through a network.

For the uninitiated, let me explain what these two tools do and how this whole trick plays out. First lets look at DD. From the DD Man page, DD will “copy and convert files”. Now you may be saying “So what”, but this description does not really do DD justice. DD is developed in such a way that is particularly suited to working with block devices, so it has a number of options for conversions. DD can also display the contents of those files to the standard output, or receive input via the standard input. Hard disk devices are usually accessible device files, usually under /dev, under *nix systems. So if I wanted to copy the contents of /dev/hda (usually the first IDE hard disk under Linux) to /tmp/harddisk_image, I could run the following command:

dd if=/dev/hda of=/tmp/harddisk_image

If I omit the “of=” option, it will stream onto the standard output device.  For more information about the options that DD supports, consult the man page. Alternatively, I found this article about using DD for forensic analysis.

Netcat is a tool that is called “the hackers swiss army knife”. That too is a description that I feel does not do this tool justice. The feature I will look at here is its ability to create a basic dummy connection. Basically we will create a simple listening socket on the target, and connect to it on the source system. This linux.com article describes this feature far better than I could, so I will not try to reinvent the wheel. Using the description provided by this article, if I wanted to copy a file from one system to another over a network, I would do something like this:

And on the target system first run
nc –l –p 7000 > file.txt

On the source system I would run:
cat file.txt > nc 192.168.1.120 7000 –q 10

This will do a basic copy of files from one system to another. Now, as I mentioned above, devices in *nix are files. So we are going to use this same principle to copy the contents of 1 hard drive one a machine to another, except instead of redirecting the standard inputs/outputs, we will pipe them to/from DD. To do so I would run the following commands:

On the target system I would run:
nc –l –p 7000 | dd of=/dev/hda

On the source system I would run:
dd if=/dev/hda | nc 192.168.1.120 7000 –q 10

Wait several hours and your good to go. You just saved yourself money on Ghost. There are other options here as well. If you’re worried about bandwidth, you can pipe the streams through gzip to compress the data transfer. So to do this, you would run the following commands:

On the target system:
nc –l –p 7000 | gzip –dfc | dd of=/dev/had

And on the source system:
dd if=/dev/hda | gzip  -cf | nc 192.168.1.120 7000 –q 10

In my opinion this has several advantages. If I have a central server, I can easily make disk images that I can use to remotely re-image a machine in the event of an issue. This works great in Standard Operating Environments, or environments where downtime needs to be cut down to a minimum and other fail over techniques have failed. In order to accomplish this, I would run something like this to create the disk image:

dd if=/dev/had | gzip –cf | dd of=source-192_168_1_120.img (or .zip, or whatever compressed format your comfortable with, such as tarball)

Now, when I need to re-image a machine, I can have the user either put in a special floppy disk or CD with a small image that simply boots and automatically runs the command to image to the target system. On the source system, since I already have a disk image ready to roll, I can run this command to send directly to the target system:

dd if= source-192_168_1_120.img | nc 192.168.1.120 7000 –q 10

I have been using this technique for several years now, and it works great. There are some caveats, of course. First, the target hard disk size must greater than or equal to the source disk size. If the disk is greater, you will have left over space on the drive, since this copy the partition tables exactly as they are. You can use a utility like Partition Magic, or I found this article explaining how to do this with NTFS using Knoppix. I am fairly certain that qtparted will also work on ext2/3 partitions as well. I will experiment with partition resizing and follow up with it. However, partition resizing is done at your own risk, and I would advise trying in a QA environment prior to using in any production environment. Also, if you create a disk image, you have to take into account system updates and patches. On a *nix system, you can mount the image and chroot to apply updates, however if you are using this on a Windows systems, this is not an option.

Socket Programming Under Cygwin

While I have talked about how great Cygwin is before, one of the strengths that really makes it stand out is that it adds another layer of compatibility for source written for GNU based systems to Windows (now if only things could go the other way as easily), which includes the networking components. If you are familiar with any of the Unix-like operating systems TCP/IP stacks, you should have no problem writing network aware programs under Cygwin. I demonstrated this with the Scenario of the infiltration of a Red Hat 6.2 where I compiled an exploit written for *nix type systems under Windows and had it successfully run. A full tutorial on sockets programming is outside of the scope of this article. However, there are plenty of good resources on the net, a few of which are listed below:

http://www.scit.wlv.ac.uk/~jphb/comms/sockets.html
http://www.cs.cf.ac.uk/Dave/C/node28.html#SECTION002800000000000000000
http://www.cs.rpi.edu/courses/sysprog/sockets/sock.html
http://www.ecst.csuchico.edu/~chafey/prog/sockets/sinfo1.html

There are also several other useful *nix facilities ported over through Cygwin, such as IPC methods like shared memory and semaphores. In this article I will demonstrate writing a simple client/server echo program demonstrating how to do socket programming for Windows under Cygwin. This will show just basics like how to create a socket, connect to a socket, and read/write data to and from a socket. I will follow up with a later article on using shared memory under Cygwin (which provides two methods, one is an external IPC program, the other is a Windows service).

First lets look at the client program. The client program will take three arguments from the command line, the host IP, the port, and a single 1-word message to pass to the server. It will then do the appropriate conversions, find the host, create the sockets, and connect. Then it will send the message to the host and receive the echo. Pretty simple. Below is the code.

//needed for out networking functionality
#include <stdio.h>
#include <sys/types.h>
#include <sys/socket.h>
#include <netinet/in.h>
#include <netdb.h>

int main(int argc, char *argv[])
{
     //for the socket file descriptor, portnumber, and return status
     int sockfd, portno, n;
     //buffer for the messages
     char buffer[255];

     //server information and structure for host info returned by gethostbyname
     struct sockaddr_in serv_addr;
         struct hostent *server;

     //I hate using a union for this purpose, but I had issues with copying and casting
     union sock
     {
          struct     sockaddr s;
          struct  sockaddr_in i;
     } sock;
     
     //not exactly the best way to validate commandline input, but it will do for demonstration
     if (argc < 4)
     {
          printf("Please provide correct arguments");
          exit(1);
     }
     
     //Get the port number from the command line
     portno = atoi(argv[2]);
     
     //set up the socket file descriptor
         sockfd = socket(AF_INET, SOCK_STREAM, 0);
         if (sockfd < 0)
             exit(1);
     
     //get server from argument passed into the command line
     server = gethostbyname(argv[1]);
     if (server == NULL) {
             fprintf(stderr,"ERROR, no such host\n");
             exit(1);
         }
     
     //Clear out the servers info
     bzero((char *) &serv_addr, sizeof(serv_addr));
     
     //Set to a network socket instead of a Unix Domain
     serv_addr.sin_family = AF_INET;

     //byte copy the address retrieved from server  into the
     //server addr structure
     bcopy((char *)server->h_addr, (char *)&serv_addr.sin_addr.s_addr, server->h_length);
     serv_addr.sin_port = htons(portno);

     //Copy the serv_addr into the union
     bcopy(&serv_addr, &sock.i, sizeof(struct sockaddr_in));

     //Connect to the server
     if ( connect(sockfd, &sock.s, sizeof(struct sockaddr)) < 0)
     {
          printf("Error with connection");
             exit(1);
     }
     
     //Show the user what they put intot he command line
     printf("Message typed in is: %s \n", argv[3]);
     
     //write to the server
     write(sockfd, argv[3], sizeof(buffer));

     //Read data back from server
     read(sockfd, buffer, sizeof(buffer));
     
     //Show the user what was recieved from the server
     printf("Recieved from server: %s\n", buffer);

     //Close the socket
     close(sockfd);

     return 0;
}


Few things to note about the above code. First is the Union. I hate unions. But in this case I couldn’t avoid it. I had issues copying the structures for some reason, and a union seemed to work well in this situation. In normal circumstances I try to avoid unions since they make reading code difficult. Also, I used bcopy instead of memcpy.

The server code is not much different. It simple sets up its listening sockets, receives the message, and returns a greeting with the IP address of the connection and the message sent. Below is the code for the server.

//needed for out networking functionality
#include <sys/types.h>
#include <sys/socket.h>
#include <netinet/in.h>
#include <arpa/inet.h>
#include <stdio.h>

const int LISTEN_PORT_NUMER = 7000;
const int MAX_STRING_SIZE = 255;

int main()
{
     //sockets for the created socket and connections
     int sockfd, newsockfd;
     //length of the client addr
     socklen_t clilen;
     //structures for various information about the client and server
          struct sockaddr_in serv_addr, cli_addr;
     //the message storing strings
     char incoming_message[MAX_STRING_SIZE], temp_string[MAX_STRING_SIZE];

     //Create Socket, if fail, exit from program
     sockfd = socket(AF_INET, SOCK_STREAM, 0);
     if (sockfd < 0)
     {
          printf("Error building Socket FD");
          exit(1);
     }
     
     //Set up our server information
          serv_addr.sin_family = AF_INET;
          serv_addr.sin_addr.s_addr = INADDR_ANY;
    
          //Assign the port number from assigned port above
          serv_addr.sin_port = htons(LISTEN_PORT_NUMER);
     
     //bind socket
     if (bind(sockfd, (struct sockaddr *) &serv_addr, sizeof(serv_addr)) < 0)
     {
          printf("Error in Binding");
                   exit(1);
     }
     
     //Set up the socket to listen
        if (listen(sockfd, 2) < 0)
     {
          printf("Error listening");
          exit (1);
     };
     
     //infinite loop
     while(1)
     {
          //set the clilen variable prior to passing to accept.
          //per the accept man page (explining why I am doing this)
          //it should initially contain the size of the structure pointed
          //to by addr; on return it will contain the actual length (in bytes)
          //of the address returned. This would fail if clilen was null
          clilen = sizeof(cli_addr);
          printf("Ready to accept connections\n");
          newsockfd = accept(sockfd, (struct sockaddr *) &cli_addr, &clilen );

          if (newsockfd < 0)
          {
               printf("Error accepting connection");
                       exit(1);
          }

          //read incoming message
          read(newsockfd, incoming_message, sizeof(incoming_message));

          //create the message to return
          sprintf(temp_string, "Hello %s, you said %s", inet_ntoa(cli_addr.sin_addr), incoming_message);
                    
          //write the return message
          if (write(newsockfd, temp_string, sizeof(incoming_message)) < 0)
          {
               printf("Error writing out");
               exit(1);
          }
          
          //close the connection
          close (newsockfd);
     }

     //This never executes
     close(sockfd);
     
     return 0;
}

One pitfall I ran into using the accept function was that the clilen variable needed to be have the size of cli_addr prior to calling the function. Without that, the accept connect would fail, which I was clued in to this by the accept MAN pages. While the above is not as robust as it could be, it demonstrates how to make Below is a transcript of the client connecting to the server through the DOS prompt:

C:\ >client 127.0.0.1 7000 testing
Message typed in is: testing
Recieved from server: Hello 127.0.0.1, you said testing

Update: Apparently I forgot to include the command line switches for compiling. There is really nothing special about these programs. In my experience, the only time I have had to include special switches is when I’ve used IPC under Cygwin, which I do intend to cover in a future article. Anyway, here are the commands I used to compile:

gcc client.c -o client
gcc server.c –o server

Multiple Row Selection in Excel VBA

I have a real short article today to address a couple of Excel questions that have come my way. I am researching a good topic, and I didn’t want to go without posting anything in the mean time. Below is a simple VBA Excel macro that will do three things. First it will prompt the user to select the whole active sheet, then prompt the user to select multiple rows, then it will select the C10 cell, and activate the row for that ActiveCell, and finally will select a single cell and remove all rows preceding it, starting from row 1.

Sub Selections()
    'prompt user to let them know you will select the whole sheet
    MsgBox "Ready to select whole sheet?", vbOKOnly
    
    'Select whole sheet
    Cells.Select
        
    'prompt user to let them know you are going to select multiple rows
    MsgBox "Ready to select rows 2 through 5?", vbOKOnly
    
    'Select multiple rows
    Range("2:2", "5:5").Select
    
    'prompt user to let them know you are going to select multiple rows and delete
    MsgBox "Select row of current activecell", vbOKOnly
    
    'Select some cell
    Range("C10").Select
              
    'Select row of current active cell
    Range(ActiveCell.Row & ":" & ActiveCell.Row).Select
              
    'prompt user to let them know you are going to select multiple rows and delete
    MsgBox "Delete rows 1 up to active cell?", vbOKOnly
              
    'Select some cell
    Range("C10").Select
              
    'Delete all rows from row 1 up to the currently selected row
    Range("1:1", ActiveCell.Row & ":" & ActiveCell.Row).Delete
End Sub

I also would like to thank Bonnie Taylor for the help she has provided me over the past several months with getting started on this blog. Her patience with my incredibly bad grammar grants her nothing less than sainthood in my eyes, and the educational aspect went both ways. Her help was greatly appreciated, and she will be missed. I wish her well on her future endeavors.

Back from Vacation

Now that my vacation is officially over and all the bar top debauchery of New Years is behind me, I am ready to hit the ground running. While I was slacking away, Richard Bejtlich was hard at work on a portable Sguil instance to run under the VMWare Player as indicated with the below links:

New Sguil VM with Client
Sguil VM Now at VMTN Community Site

And he has been hard at work on a Sguil install script for FreeBSD.

In BIRT news, the Release candidate for version 2 is available. I haven’t previewed it, but I intend to. Jason Weathersby was pretty tight lipped on the new features, all he told me so far is they are “cool”, so I look forward to checking them out.