BIRT 2.0 Installation

With the announcement of BIRT Release 2, I have been very excited to give it a run through. I finally got a chance to install it, and here is my experience upgrading from the previous version of BIRT to the version 2.0.

The first step is to get the additional prerequisite packages for BIRT. I already have the base Eclipse packages needed as outlined in the EclipseZone article. Additionally BIRT 2.0 will require Apache Axis, iText, prototype.js, and the BIRT 2.0 Framework. Apache Axis is the first of the prerequisite packages I retrieved. Fortunately only 6 files are needed for BIRT, so I open the file in WinZip, and sort by file type. I select all the “Executable JAR files” and extract to a temporary directory.



I will only keep the following 6 files:
axis.jar axis-ant.jar commons-discovery-0.2.jar jaxrpc.jar saaj.jar wsdl4j-1.5.1.jar

I extract the ZIP file to C:\Download\Jars_for_BIRT_2\. The 6 files were found under the following directory:
C:\download\Jars_for_BIRT_2\axis-1_2_1\lib

iText. was saved to C:\ Download\Jars_for_BIRT_2\iText

And finally I downloaded prototype.js and saved it to C:\ Download\Jars_for_BIRT_2\.

I downloaded the BIRT 2.0 Framework from here.

Now that I have the components downloaded, I need to remove my original BIRT installation before I begin installing 2.0. The first thing I do is locate my Eclipse folder, and delete all subfolders starting with org.eclipse.birt.



For grins, I also run “eclipse –clean” to clear out any old garbage relating to the old version of BIRT.

I do get an error message warning me about not being able to load my workbench (which was BIRT the last time I ran Eclipse), but it can be ignored. From there I close Eclipse.

Next I open up my Zip file for the BIRT 2.0 Framework and extract to the folder containing Eclipse, with full directory options on.



Now that the directory structure is created for BIRT, I copy the 6 files from Apache to the C:\eclipse\eclipse\plugins\org.eclipse.birt.report.viewer_2.0.0\birt\WEB-INF\lib



Then I copy iText over



And finally, I copy over prototype.js to the appropriate BIRT directory



Once done, I go into the Eclipse folder, and launch Eclipse with the –clean switch again.



I am stoked that I do not get the workspace error again, and I can open the Employees example I used for the EclipseZone article. The only thing I need to do is to load my JDBC driver for Oracle again. These steps are outlined in the EclipseZone article and in Sguil Incident Reporting using BIRT articles, so I will not repeat them here. (Note: I ran into a bug here while writing this article. If you go into manage drivers, cancel out, then go back into manage drivers, the dialog will not pop up. I did have a BIRT 1 report opened at the time which was expecting the JDBC driver, so no telling if that is the catalyst for this bug. Annoying, but not critical).

Once added, I was successfully able to run the Employee test report under BIRT. I will play around with the new features in 2.0 and report back what I find.

Business Intelligence Role in NSM

Since I have been on a roll with the whole BI topic lately, I’ve decided to continue this trend, at least for a little while longer, and apply it to NSM. I have covered this before.

As it stands now, Sguils primary focus is providing facilities for the network security analyst, and it succeeds in that role admirably. In fact, I would go as far as to say it is the best tool for any serious network security monitoring operation. I consider Sguil to be the only tool that really facilitates the analyst’s portion of the NSM methodology as outlined by Richard Bejtlich, with capabilities such as session data, packet data, and full transcripts. But from a business perspective, the Sguil setup has great BI potential for NSM operations. In fact, I would say that the BI potential that NSM provides is one of its strongest, yet most overlooked, elements.

Consider this, the tools that Sguil is built on top of already provide the ability to collect data, and it already logs this data into a somewhat meaningful schema for further data analysis. Data is already stored in such a way that is can be categorized, grouped, and counted, thus providing the basis for real BI opportunities. In my opinion, this is one of the strongest points that separate NSM from other security practices. By being able to work with data, reports can be generated to help managers and stakeholders make informed decisions in regards to information security policies. If you are running a full Sguil setup, and data is collected that indicates a particular Internet facing host is frequently the target of attacks, then managers can make a decision regarding the necessity of this machines role to the companies infrastructure. The numbers will show that the machine is poses a potential problem, and management can address this and direct the IT staff. If the machine has outlived its usefulness, then it merely provides an avenue of attack, and management can decide to take this system offline. I myself have seen this in action with some of the clients that BATC had during my days working as a security analyst.

BI/MIS built on top of NSM data will allow managers to make informed decisions about network security, regardless of their understanding of security principles (although knowledge in this area are definitely beneficial). It doesn’t take an expert to understand the severity of a compromise, and with numbers to support it, the proof is in the pudding (sorry, I’ve been hearing that a lot lately, I’ve been looking for an excuse to use it). So while Sguils capabilities provide for an excellent platform to serve as a transactional system, I believe BIRT is the platform to serve as MIS in this setup, as I have stated previously in other articles.

It has been previously stated that IT staff should be given more authority to enforce security policies. I cannot stress how much I disagree with this belief. The extent of authority I would give IT staff is to remove machines that violate security policy from a network. Anything else is the responsibility of management and HR. IT staffs that I have encountered in the past are usually too busy to play network police, and they do not have proper “people skills” or “emotional intelligence”. Granting them that authority opens up companies to potential lawsuits, there is a reason why management has to undergo training for these sorts of tasks. The best way for IT to address these kinds of issues is to help facilitate better BI based off of NSM data.

Big Day for BIRT

Yesterday was quite a day for the BIRT community. BIRT 2.0 is finally out, with quite a few new enhancements. Also, thanks to the folks over at EclipseZone, I was pointed to this article announcing broad support by Actuate, IBM, Pentaho, Scapa Technologies and Zend for BIRT. Interesting news, and congratulations to the BIRT team for all their hard work.

EclipseZone Article Published

My article introducing BIRT has been posted to the EclipseZone website. I am pretty psyched about this since this is the first article I have written for another site. I am surprised to see this article posted on JavaLobby also. It’s good to see that the work I put into it come into fruition, despite the many ups and downs like losing my awesome freelance editor.

DD over Netcat for a Cheap Ghost Alternative

Recently I have seen several people ask “How can I clone my hardrive over Ethernet” on several forums that I read. There are several commercial options, such as Nortons Ghost, but I prefer the much cheaper solution of using the GNU DD utility (part of the fileutils package) and redirecting the results through Netcat (the GNU version can be found here). While it is not exactly the fastest solution out there, it is a decent enough of a solution, and I typically am never in a situation where I need to clone quickly through a network.

For the uninitiated, let me explain what these two tools do and how this whole trick plays out. First lets look at DD. From the DD Man page, DD will “copy and convert files”. Now you may be saying “So what”, but this description does not really do DD justice. DD is developed in such a way that is particularly suited to working with block devices, so it has a number of options for conversions. DD can also display the contents of those files to the standard output, or receive input via the standard input. Hard disk devices are usually accessible device files, usually under /dev, under *nix systems. So if I wanted to copy the contents of /dev/hda (usually the first IDE hard disk under Linux) to /tmp/harddisk_image, I could run the following command:

dd if=/dev/hda of=/tmp/harddisk_image

If I omit the “of=” option, it will stream onto the standard output device.  For more information about the options that DD supports, consult the man page. Alternatively, I found this article about using DD for forensic analysis.

Netcat is a tool that is called “the hackers swiss army knife”. That too is a description that I feel does not do this tool justice. The feature I will look at here is its ability to create a basic dummy connection. Basically we will create a simple listening socket on the target, and connect to it on the source system. This linux.com article describes this feature far better than I could, so I will not try to reinvent the wheel. Using the description provided by this article, if I wanted to copy a file from one system to another over a network, I would do something like this:

And on the target system first run
nc –l –p 7000 > file.txt

On the source system I would run:
cat file.txt > nc 192.168.1.120 7000 –q 10

This will do a basic copy of files from one system to another. Now, as I mentioned above, devices in *nix are files. So we are going to use this same principle to copy the contents of 1 hard drive one a machine to another, except instead of redirecting the standard inputs/outputs, we will pipe them to/from DD. To do so I would run the following commands:

On the target system I would run:
nc –l –p 7000 | dd of=/dev/hda

On the source system I would run:
dd if=/dev/hda | nc 192.168.1.120 7000 –q 10

Wait several hours and your good to go. You just saved yourself money on Ghost. There are other options here as well. If you’re worried about bandwidth, you can pipe the streams through gzip to compress the data transfer. So to do this, you would run the following commands:

On the target system:
nc –l –p 7000 | gzip –dfc | dd of=/dev/had

And on the source system:
dd if=/dev/hda | gzip  -cf | nc 192.168.1.120 7000 –q 10

In my opinion this has several advantages. If I have a central server, I can easily make disk images that I can use to remotely re-image a machine in the event of an issue. This works great in Standard Operating Environments, or environments where downtime needs to be cut down to a minimum and other fail over techniques have failed. In order to accomplish this, I would run something like this to create the disk image:

dd if=/dev/had | gzip –cf | dd of=source-192_168_1_120.img (or .zip, or whatever compressed format your comfortable with, such as tarball)

Now, when I need to re-image a machine, I can have the user either put in a special floppy disk or CD with a small image that simply boots and automatically runs the command to image to the target system. On the source system, since I already have a disk image ready to roll, I can run this command to send directly to the target system:

dd if= source-192_168_1_120.img | nc 192.168.1.120 7000 –q 10

I have been using this technique for several years now, and it works great. There are some caveats, of course. First, the target hard disk size must greater than or equal to the source disk size. If the disk is greater, you will have left over space on the drive, since this copy the partition tables exactly as they are. You can use a utility like Partition Magic, or I found this article explaining how to do this with NTFS using Knoppix. I am fairly certain that qtparted will also work on ext2/3 partitions as well. I will experiment with partition resizing and follow up with it. However, partition resizing is done at your own risk, and I would advise trying in a QA environment prior to using in any production environment. Also, if you create a disk image, you have to take into account system updates and patches. On a *nix system, you can mount the image and chroot to apply updates, however if you are using this on a Windows systems, this is not an option.

Socket Programming Under Cygwin

While I have talked about how great Cygwin is before, one of the strengths that really makes it stand out is that it adds another layer of compatibility for source written for GNU based systems to Windows (now if only things could go the other way as easily), which includes the networking components. If you are familiar with any of the Unix-like operating systems TCP/IP stacks, you should have no problem writing network aware programs under Cygwin. I demonstrated this with the Scenario of the infiltration of a Red Hat 6.2 where I compiled an exploit written for *nix type systems under Windows and had it successfully run. A full tutorial on sockets programming is outside of the scope of this article. However, there are plenty of good resources on the net, a few of which are listed below:

http://www.scit.wlv.ac.uk/~jphb/comms/sockets.html
http://www.cs.cf.ac.uk/Dave/C/node28.html#SECTION002800000000000000000
http://www.cs.rpi.edu/courses/sysprog/sockets/sock.html
http://www.ecst.csuchico.edu/~chafey/prog/sockets/sinfo1.html

There are also several other useful *nix facilities ported over through Cygwin, such as IPC methods like shared memory and semaphores. In this article I will demonstrate writing a simple client/server echo program demonstrating how to do socket programming for Windows under Cygwin. This will show just basics like how to create a socket, connect to a socket, and read/write data to and from a socket. I will follow up with a later article on using shared memory under Cygwin (which provides two methods, one is an external IPC program, the other is a Windows service).

First lets look at the client program. The client program will take three arguments from the command line, the host IP, the port, and a single 1-word message to pass to the server. It will then do the appropriate conversions, find the host, create the sockets, and connect. Then it will send the message to the host and receive the echo. Pretty simple. Below is the code.

//needed for out networking functionality
#include <stdio.h>
#include <sys/types.h>
#include <sys/socket.h>
#include <netinet/in.h>
#include <netdb.h>

int main(int argc, char *argv[])
{
     //for the socket file descriptor, portnumber, and return status
     int sockfd, portno, n;
     //buffer for the messages
     char buffer[255];

     //server information and structure for host info returned by gethostbyname
     struct sockaddr_in serv_addr;
         struct hostent *server;

     //I hate using a union for this purpose, but I had issues with copying and casting
     union sock
     {
          struct     sockaddr s;
          struct  sockaddr_in i;
     } sock;
     
     //not exactly the best way to validate commandline input, but it will do for demonstration
     if (argc < 4)
     {
          printf("Please provide correct arguments");
          exit(1);
     }
     
     //Get the port number from the command line
     portno = atoi(argv[2]);
     
     //set up the socket file descriptor
         sockfd = socket(AF_INET, SOCK_STREAM, 0);
         if (sockfd < 0)
             exit(1);
     
     //get server from argument passed into the command line
     server = gethostbyname(argv[1]);
     if (server == NULL) {
             fprintf(stderr,"ERROR, no such host\n");
             exit(1);
         }
     
     //Clear out the servers info
     bzero((char *) &serv_addr, sizeof(serv_addr));
     
     //Set to a network socket instead of a Unix Domain
     serv_addr.sin_family = AF_INET;

     //byte copy the address retrieved from server  into the
     //server addr structure
     bcopy((char *)server->h_addr, (char *)&serv_addr.sin_addr.s_addr, server->h_length);
     serv_addr.sin_port = htons(portno);

     //Copy the serv_addr into the union
     bcopy(&serv_addr, &sock.i, sizeof(struct sockaddr_in));

     //Connect to the server
     if ( connect(sockfd, &sock.s, sizeof(struct sockaddr)) < 0)
     {
          printf("Error with connection");
             exit(1);
     }
     
     //Show the user what they put intot he command line
     printf("Message typed in is: %s \n", argv[3]);
     
     //write to the server
     write(sockfd, argv[3], sizeof(buffer));

     //Read data back from server
     read(sockfd, buffer, sizeof(buffer));
     
     //Show the user what was recieved from the server
     printf("Recieved from server: %s\n", buffer);

     //Close the socket
     close(sockfd);

     return 0;
}


Few things to note about the above code. First is the Union. I hate unions. But in this case I couldn’t avoid it. I had issues copying the structures for some reason, and a union seemed to work well in this situation. In normal circumstances I try to avoid unions since they make reading code difficult. Also, I used bcopy instead of memcpy.

The server code is not much different. It simple sets up its listening sockets, receives the message, and returns a greeting with the IP address of the connection and the message sent. Below is the code for the server.

//needed for out networking functionality
#include <sys/types.h>
#include <sys/socket.h>
#include <netinet/in.h>
#include <arpa/inet.h>
#include <stdio.h>

const int LISTEN_PORT_NUMER = 7000;
const int MAX_STRING_SIZE = 255;

int main()
{
     //sockets for the created socket and connections
     int sockfd, newsockfd;
     //length of the client addr
     socklen_t clilen;
     //structures for various information about the client and server
          struct sockaddr_in serv_addr, cli_addr;
     //the message storing strings
     char incoming_message[MAX_STRING_SIZE], temp_string[MAX_STRING_SIZE];

     //Create Socket, if fail, exit from program
     sockfd = socket(AF_INET, SOCK_STREAM, 0);
     if (sockfd < 0)
     {
          printf("Error building Socket FD");
          exit(1);
     }
     
     //Set up our server information
          serv_addr.sin_family = AF_INET;
          serv_addr.sin_addr.s_addr = INADDR_ANY;
    
          //Assign the port number from assigned port above
          serv_addr.sin_port = htons(LISTEN_PORT_NUMER);
     
     //bind socket
     if (bind(sockfd, (struct sockaddr *) &serv_addr, sizeof(serv_addr)) < 0)
     {
          printf("Error in Binding");
                   exit(1);
     }
     
     //Set up the socket to listen
        if (listen(sockfd, 2) < 0)
     {
          printf("Error listening");
          exit (1);
     };
     
     //infinite loop
     while(1)
     {
          //set the clilen variable prior to passing to accept.
          //per the accept man page (explining why I am doing this)
          //it should initially contain the size of the structure pointed
          //to by addr; on return it will contain the actual length (in bytes)
          //of the address returned. This would fail if clilen was null
          clilen = sizeof(cli_addr);
          printf("Ready to accept connections\n");
          newsockfd = accept(sockfd, (struct sockaddr *) &cli_addr, &clilen );

          if (newsockfd < 0)
          {
               printf("Error accepting connection");
                       exit(1);
          }

          //read incoming message
          read(newsockfd, incoming_message, sizeof(incoming_message));

          //create the message to return
          sprintf(temp_string, "Hello %s, you said %s", inet_ntoa(cli_addr.sin_addr), incoming_message);
                    
          //write the return message
          if (write(newsockfd, temp_string, sizeof(incoming_message)) < 0)
          {
               printf("Error writing out");
               exit(1);
          }
          
          //close the connection
          close (newsockfd);
     }

     //This never executes
     close(sockfd);
     
     return 0;
}

One pitfall I ran into using the accept function was that the clilen variable needed to be have the size of cli_addr prior to calling the function. Without that, the accept connect would fail, which I was clued in to this by the accept MAN pages. While the above is not as robust as it could be, it demonstrates how to make Below is a transcript of the client connecting to the server through the DOS prompt:

C:\ >client 127.0.0.1 7000 testing
Message typed in is: testing
Recieved from server: Hello 127.0.0.1, you said testing

Update: Apparently I forgot to include the command line switches for compiling. There is really nothing special about these programs. In my experience, the only time I have had to include special switches is when I’ve used IPC under Cygwin, which I do intend to cover in a future article. Anyway, here are the commands I used to compile:

gcc client.c -o client
gcc server.c –o server

Multiple Row Selection in Excel VBA

I have a real short article today to address a couple of Excel questions that have come my way. I am researching a good topic, and I didn’t want to go without posting anything in the mean time. Below is a simple VBA Excel macro that will do three things. First it will prompt the user to select the whole active sheet, then prompt the user to select multiple rows, then it will select the C10 cell, and activate the row for that ActiveCell, and finally will select a single cell and remove all rows preceding it, starting from row 1.

Sub Selections()
    'prompt user to let them know you will select the whole sheet
    MsgBox "Ready to select whole sheet?", vbOKOnly
    
    'Select whole sheet
    Cells.Select
        
    'prompt user to let them know you are going to select multiple rows
    MsgBox "Ready to select rows 2 through 5?", vbOKOnly
    
    'Select multiple rows
    Range("2:2", "5:5").Select
    
    'prompt user to let them know you are going to select multiple rows and delete
    MsgBox "Select row of current activecell", vbOKOnly
    
    'Select some cell
    Range("C10").Select
              
    'Select row of current active cell
    Range(ActiveCell.Row & ":" & ActiveCell.Row).Select
              
    'prompt user to let them know you are going to select multiple rows and delete
    MsgBox "Delete rows 1 up to active cell?", vbOKOnly
              
    'Select some cell
    Range("C10").Select
              
    'Delete all rows from row 1 up to the currently selected row
    Range("1:1", ActiveCell.Row & ":" & ActiveCell.Row).Delete
End Sub

I also would like to thank Bonnie Taylor for the help she has provided me over the past several months with getting started on this blog. Her patience with my incredibly bad grammar grants her nothing less than sainthood in my eyes, and the educational aspect went both ways. Her help was greatly appreciated, and she will be missed. I wish her well on her future endeavors.

Back from Vacation

Now that my vacation is officially over and all the bar top debauchery of New Years is behind me, I am ready to hit the ground running. While I was slacking away, Richard Bejtlich was hard at work on a portable Sguil instance to run under the VMWare Player as indicated with the below links:

New Sguil VM with Client
Sguil VM Now at VMTN Community Site

And he has been hard at work on a Sguil install script for FreeBSD.

In BIRT news, the Release candidate for version 2 is available. I haven’t previewed it, but I intend to. Jason Weathersby was pretty tight lipped on the new features, all he told me so far is they are “cool”, so I look forward to checking them out.