Games: Ninja Gaiden Dragon Sword

I was really skeptical about this game at first. I had read plenty of pre-reviews about this game and using the stylus to "slash" the opponents on the screen. I just imagined some geeky kid on an airplane waving his hands widley, shopping at the tiny DS screen with a katana shapped stylus, poking the passengers around him in the eyes. And given the fact that Ninja Gaiden on the XBox is second only the GTA in the need for an ESRB rating, I was a little worried about the delivery on family friendly Nintendos consoles.

Now that I've gotten a hands on with it, I can say those fears are completely unfounded. NG:DS is a completely engrossing handheld experience right from the get go. Gone are the days and stories of Ryu chasing Jaquio around looking for statues and saving Irine from the clutches of certain death. NG:DS takes up from the original Xbox story line, some 6 months after the "Dark Dragon Blade" incident. While I haven't gotten enough into the story, I did run around quite a bit to make sure the control scheme wasn't nearly as wonky as I had feared. Holding the DS at 90 degrees, book style, is a bit awkward, but only because I've spent the past 20 years of my life holding controllers ALA the original NES control scheme. Moving the character around is done via the stylus, as is the attacks. How is this done? Why, by pointing where you want to move to, writing/slashing the stylus over the opponent to attack, tapping the opponent to throw ninja stars, and slashing up to jump. So the attacks are actually pretty intuitive, despite my original fears.

There are a few hiccups however. There re times where to want to jump, and the chacter runs right into the coming onslaught of enemies instead. Thats kind of annoying, but I can chalk that up to my own ineptness with the control scheme. Blocking is a bit of a pain. Any button will cause you to block. But since I am a righty, I hold the stylus with my right-hand, the DS is in a right-hand orientation, which puts the buttons on the right-screen. So, as you can imagine, having to use my left hand to push a button in a right-handed orientation leaves me a little short-circuited due to my lack of ambidextrous abilities. Fortunately, what I lack in coordination I more than make up for in hand size, so reaching over to push buttons to block isn't as bad as I make it seem.

While I haven't done enough story line to really pass judgment on it, I did find the game bordering on "not able to put down" fun from running around and slashing. While this game isn't quite suitable for my game time during meals while traveling due to it's more interactive nature, it is definitely suitable for the evenings at the hotel while traveling and on long plane rides so that I can be the uncoordinated geeky kid poking peoples eyes out.

Sguil: Sguil 0.7 is out

It's been a while since we've seen an update with Sguil. And although with most Open source projects, that usually an indication that a project is dead. However, frequenters of the Sguil IRC channel #snort-gui on irc.freenode.net know that Sguil is far from being dead. In fact, they just released version 0.7, even though I am about 3 days late in noting it. To be perfectly honest, I haven't messes with Sguil since versions 0.5 - 0.6, however necessity breeds use, and I have need of this awesome tool. And a new release version plus a need REALLY gives me a good excuse to revisit the tool the brought me to working with BIRT.

BIRT: Building Chart with Chart Engine API based off Report Document

I have been sitting on this example for a few weeks now, and have been waiting to put it out until after EclipseCon so I can absorb any ideas from the Charting API presentation that Jason Weathersby over at BirtWorld did. Now that is done and gone, I have decided to put this one out into the wild.

The original purpose of this was to be able to extract a chart design out of a BIRT report, and render it using the BIRT Chart Engine API’s. What I found out the hard way is that data set and data are not linked to Charts in BIRT outside of the BIRT render engine. The Chart Engine API, as it turns out, is completely decoupled from the BIRT Report Engine API, so axis to data mappings are not accessible in the Chart Engine API and the actual data binding is done through some Java-Fu wizardry at runtime. So, I salvaged what I could of the idea.

The following example will demonstrate a couple of different things. First, it will open a BIRT Report, and pull the Chart definition out of the report. It will then read the XML Representation of the chart to determine what columns in the report are mapped to which Axis. It then runs the report, and uses a Data Extraction task to pull the data and create an appropriate Chart Engine API Series definition. It then renders a chart to a PNG file.

There are a few things to keep in mind in this example. First, it assumes that there is a 2 column data set defined in the report, with column 1 containing what is the X axis definitions, and column 2 containing the values. It then calculates the aggregates based on those. Second, it assumes a chart definition is defined in the report. In a real world scenario, you would NEVER render a chart like this, but I am a glutton for punishment and needed a good way to learn how the Chart Engine API works. You will notice that I commented out a good chunk of code that defines the look and feel of the chart. I kept these commented out as a reference, but they aren’t necessary in this example since the visual attributes for the chart were defined in the source report design.

I did learn one really cool thing about Chart Engine API. There is a whole slew of example Chart Engine API code available in a BIRT install by default, by going to Window/Show View/Other/Report and Chart Design/Chart Examples. This will open a new view, just select any the examples listed and click on the Open button on the right hand side to see example code on how to build those charts.

Anyway, here is my example:

package com.test;

import java.io.File;
import java.io.StringBufferInputStream;
import java.util.ArrayList;
import java.util.HashMap;
import java.util.Iterator;
import java.util.Locale;
import java.util.Map;

import javax.xml.xpath.XPath;
import javax.xml.xpath.XPathConstants;
import javax.xml.xpath.XPathExpressionException;
import javax.xml.xpath.XPathFactory;

import org.eclipse.birt.chart.api.ChartEngine;
import org.eclipse.birt.chart.device.IDeviceRenderer;
import org.eclipse.birt.chart.device.image.PngRendererImpl;
import org.eclipse.birt.chart.exception.ChartException;
import org.eclipse.birt.chart.factory.GeneratedChartState;
import org.eclipse.birt.chart.factory.Generator;
import org.eclipse.birt.chart.factory.RunTimeContext;
import org.eclipse.birt.chart.model.Chart;
import org.eclipse.birt.chart.model.ChartWithoutAxes;
import org.eclipse.birt.chart.model.attribute.Bounds;
import org.eclipse.birt.chart.model.attribute.impl.BoundsImpl;
import org.eclipse.birt.chart.model.component.Series;
import org.eclipse.birt.chart.model.component.impl.SeriesImpl;
import org.eclipse.birt.chart.model.data.NumberDataSet;
import org.eclipse.birt.chart.model.data.SeriesDefinition;
import org.eclipse.birt.chart.model.data.TextDataSet;
import org.eclipse.birt.chart.model.data.impl.NumberDataSetImpl;
import org.eclipse.birt.chart.model.data.impl.SeriesDefinitionImpl;
import org.eclipse.birt.chart.model.data.impl.TextDataSetImpl;
import org.eclipse.birt.chart.model.type.PieSeries;
import org.eclipse.birt.chart.model.type.impl.PieSeriesImpl;
import org.eclipse.birt.core.exception.BirtException;
import org.eclipse.birt.core.framework.Platform;
import org.eclipse.birt.core.framework.PlatformConfig;
import org.eclipse.birt.data.engine.core.DataException;
import org.eclipse.birt.report.engine.api.EngineConfig;
import org.eclipse.birt.report.engine.api.EngineException;
import org.eclipse.birt.report.engine.api.IDataExtractionTask;
import org.eclipse.birt.report.engine.api.IDataIterator;
import org.eclipse.birt.report.engine.api.IExtractionResults;
import org.eclipse.birt.report.engine.api.IReportDocument;
import org.eclipse.birt.report.engine.api.IReportEngine;
import org.eclipse.birt.report.engine.api.IReportEngineFactory;
import org.eclipse.birt.report.engine.api.IReportRunnable;
import org.eclipse.birt.report.engine.api.IResultSetItem;
import org.eclipse.birt.report.engine.api.IRunTask;
import org.eclipse.birt.report.model.api.ExtendedItemHandle;
import org.eclipse.birt.report.model.api.ReportDesignHandle;
import org.eclipse.birt.report.model.api.extension.ExtendedElementException;
import org.w3c.dom.DOMException;
import org.xml.sax.InputSource;

import com.ibm.icu.util.ULocale;
import com.sun.org.apache.xml.internal.dtm.ref.DTMNodeList;

public class RenderChart {
    private static String BIRT_HOME = "C:/birt-runtime-2_2_1_1/ReportEngine";

    /**
     * Creates the X series for the Chart
     * @param uniqueMap
     * @return
     */
    public static TextDataSet createCategoriesDataSet(Map uniqueMap)
    {
        String[] categories = new String[uniqueMap.keySet().size()];
        int x = 0;
        for (Iterator keyIt = uniqueMap.keySet().iterator(); keyIt.hasNext();)
        {
            categories[x] = (String)keyIt.next();
            x++;
        }
        return TextDataSetImpl.create(categories);
    }

    /**
     * Creates the Y Series for a chart
     * @param uniqueMap
     * @return
     */
    public static NumberDataSet createValueDataSet(Map uniqueMap)
    {
        double[] values = new double[uniqueMap.values().size()];
        int x = 0;
        for (Iterator valIt = uniqueMap.values().iterator(); valIt.hasNext();)
        {
            values[x] = Math.round(((Double)valIt.next()).doubleValue());
            x++;
        }
        
        return NumberDataSetImpl.create(values);
    }
    
    /**
     * Will open a report design and return the Document instance
     * @param reportName
     * @return
     */
    public static IReportDocument executeReport(String reportName) throws EngineException {
        // create a new report engine factory
        IReportEngineFactory factory = (IReportEngineFactory) Platform
                .createFactoryObject(IReportEngineFactory.EXTENSION_REPORT_ENGINE_FACTORY);

        // create a new report engine
        EngineConfig engineConfig = new EngineConfig();
        engineConfig.setBIRTHome(BIRT_HOME); // will replace with
                                                // configuration file
        IReportEngine engine = factory.createReportEngine(engineConfig);

        // create the report task and open the report design file
        IReportRunnable design = engine.openReportDesign(reportName);
        IRunTask runTask = engine.createRunTask(design);

        // use the default locale
        runTask.setLocale(Locale.getDefault());

        // run and close the report run task
        File newTempFile = new File("C:/TEMP/birtRenderTemp",
                "test.rptDocument");
        String tempFileLocation = newTempFile.getAbsolutePath();

        // delete the temp file, we just needed the name and path
        newTempFile.delete();

        runTask.run(tempFileLocation);
        runTask.close();

        IReportDocument ird = engine
                .openReportDocument("C:/TEMP/birtRenderTemp/test.rptDocument");

        return ird;
    }
    
    /**
     * Creates an input source from the XMLProperties string
     * @param xmlProperties
     * @return
     */
    public static InputSource getInputSourceFromString(String xmlProperties)
    {
        StringBufferInputStream is = new StringBufferInputStream(
                xmlProperties);
        
        return new InputSource(is);
    }
    
    /**
     * Using a data extraction task, this will build a simple Map of columns and rows
     * This method assumes a 1-1 relation between what will be the x value and y value
     * @param det
     * @param ySeries
     * @param xSeries
     * @return
     * @throws BirtException
     */
    public static Map getValueMap(IDataExtractionTask det, String ySeries, String xSeries) throws BirtException
    {
        //extract the results from the task
        IExtractionResults iExtractResults = det.extract();
        IDataIterator iData = null;

        //hashmap to return
        Map uniqueMap = new HashMap();

        //if we have results, process them, otherwise, don't
        if (iExtractResults != null) {
            iData = iExtractResults.nextResultIterator();

            // iterate through the results
            if (iData != null) {
                //loop while there is still data in out iterator
                while (iData.next()) {
                    Object objColumn1;
                    Object objColumn2;
                    try {
                        objColumn1 = iData.getValue(ySeries);
                    } catch (DataException e) {
                        objColumn1 = new String("");
                    }
                    try {
                        objColumn2 = iData.getValue(xSeries);
                    } catch (DataException e) {
                        objColumn2 = new String("");
                    }

                    Double newNumber = (Double) objColumn1;
                    if (uniqueMap.keySet().contains(objColumn2)) {
                        newNumber += (Double) uniqueMap
                                .get(objColumn2);
                    }
                    uniqueMap.put(objColumn2, newNumber);
                }
                iData.close();
            }
        }

        //close the data extraction task and return our value map
        det.close();
        return uniqueMap;
    }
    
    /**
     * Extract the XML resources from the extended item handle, used to get the XML
     * representation of the chart from a report
     * 
     * @param eih
     * @return
     */
    public static String getXMLResources(ExtendedItemHandle eih)
    {
        return (String) eih.getProperty("xmlRepresentation");
    }
    
    /**
     * Retrieves the name of the X Series column from a chart XMLRepresentation
     * 
     * @param xmlProperties
     * @return
     * @throws XPathExpressionException
     * @throws DOMException
     */
    public static String getXSeries(String xmlProperties) throws XPathExpressionException, DOMException
    {
        InputSource inputSource = getInputSourceFromString(xmlProperties);
        XPath xpath = XPathFactory.newInstance().newXPath();

        DTMNodeList nodeList = (DTMNodeList) xpath
                .evaluate(
                        "//SeriesDefinitions/Series/DataDefinition/Definition",
                        inputSource, XPathConstants.NODESET);

        String xSeries = nodeList.item(1).getTextContent();

        //we found the series definitions, now remove the expression and just return
        //the key 
        return xSeries.replace("row[\"", "").replace("\"]", "");
    }
    
    /**    
     * Retrieves the name of the Y Series column from a chart XMLRepresentation
     * @param xmlProperties
     * @return
     * @throws XPathExpressionException
     * @throws DOMException
     */
    public static String getYSeries(String xmlProperties) throws XPathExpressionException, DOMException
    {    
        InputSource inputSource = getInputSourceFromString(xmlProperties);
        XPath xpath = XPathFactory.newInstance().newXPath();

        DTMNodeList nodeList = (DTMNodeList) xpath
                .evaluate(
                        "//SeriesDefinitions/Series/DataDefinition/Definition",
                        inputSource, XPathConstants.NODESET);

        String ySeries = nodeList.item(0).getTextContent();
        
        return ySeries.replace("row[\"", "").replace("\"]", "");
    }
        
    /**
     * Will render a Pie chart with given categories and values. Chart needs to be created beforehand, so pull from document first.
     * @param cwa
     * @param categories
     * @param values
     * @throws ChartException
     */
    public static void renderPieChart(ChartWithoutAxes cwa, TextDataSet categories, NumberDataSet values) throws ChartException
    {
        // Create the png renderer
        IDeviceRenderer idr = new PngRendererImpl();

        //create new run time context
        RunTimeContext rtc = new RunTimeContext();
        rtc.setULocale(ULocale.getDefault());

        //create a new generator
        final Generator gr = Generator.instance();
        GeneratedChartState gcs = null;
        
        //clear any existing series definitions since we created out own
        cwa.getSeriesDefinitions().clear();
        
        // Plot the chart...
        /* 
         * Note: I commented this stuff out since I already designed my chart
         * look and feel in the BIRT report designer, and all of that will
         * already be set in my chart opened from a BIRT Report
        cwa.setSeriesThickness(25);
        cwa.getBlock().setBackground(ColorDefinitionImpl.WHITE());
        Plot p = cwa.getPlot();
        cwa.setDimension(ChartDimension.TWO_DIMENSIONAL_LITERAL);
        
        p.getClientArea().setBackground(null);
        p.getClientArea().getOutline().setVisible(true);
        p.getOutline().setVisible(true);

        Legend lg = cwa.getLegend();
        lg.getText().getFont().setSize(16);
        lg.setBackground(null);
        lg.getOutline().setVisible(true);
        
        // Title
        //cwa.getTitle( ).getLabel( ).getCaption( ).setValue( "Pie Chart" );//$NON-NLS-1$
        cwa.getTitle( ).getOutline( ).setVisible( true );
        */
        
        //define base series
        Series seCategory = SeriesImpl.create();
        seCategory.setDataSet(categories);

        SeriesDefinition sd = SeriesDefinitionImpl.create();
        sd.getSeriesPalette().shift(0);
        sd.getSeries().add(seCategory);
        cwa.getSeriesDefinitions().add(sd);
        
        //define pie seies
        PieSeries categorySeries = (PieSeries) PieSeriesImpl
                .create();
        categorySeries.setDataSet(values);
        categorySeries.setSeriesIdentifier("Territories");

        SeriesDefinition sdValues = SeriesDefinitionImpl.create();
        sdValues.getQuery().setDefinition("Census.Territories");//$NON-NLS-1$
        sdValues.getSeries().add(categorySeries);
        sdValues.getSeries().add(categorySeries);
        sd.getSeriesDefinitions().add(sdValues);
                
        // Set the chart size
        Bounds bo = BoundsImpl.create(0, 0, 350, 275);
        
        //Now build the chart
        gcs = gr.build(idr.getDisplayServer(), (Chart)cwa, bo, null, rtc,
                null);

        // Specify the file to write to.
        idr.setProperty(IDeviceRenderer.FILE_IDENTIFIER,
                        "test.png"); //$NON-NLS-1$

        // generate the chart
        gr.render(idr, gcs);
    }

    /**
     * Main: The delegator of out program. Starts the BIRT Platform, gets chart, extracts values, and renders chart
     * @param args
     */
    public static void main(String[] args) {
        try {
            //start up the platform
            //note: needed to add STANDALONE = true, otherwise the chart engine
            //would not work. 
            PlatformConfig platformConfig = new PlatformConfig();
            platformConfig.setBIRTHome(BIRT_HOME);
            //standalone platform
            platformConfig.setProperty("STANDALONE", "true");
            ChartEngine.instance(platformConfig);
            Platform.startup(platformConfig);

            // create a new report engine factory
            IReportEngineFactory factory = (IReportEngineFactory) Platform
                    .createFactoryObject(IReportEngineFactory.EXTENSION_REPORT_ENGINE_FACTORY);

            // create a new report engine
            EngineConfig engineConfig = new EngineConfig();
            engineConfig.setBIRTHome(BIRT_HOME);
            IReportEngine engine = factory.createReportEngine(engineConfig);

            //open the given report and create a new Data Extraction task to get data from run report
            IReportDocument ird = executeReport("C:/contracts/GWTBirt/BIRTGwt/src/reports/Charts/TerritorySalesPieChart.rptdesign");
            IDataExtractionTask det = engine.createDataExtractionTask(ird);

            //get the report runnable from the document so we can grab the chart information from the report
            IReportRunnable r = ird.getReportRunnable();
            ReportDesignHandle rh = (ReportDesignHandle) r.getDesignHandle();
            
            //for each element in the report (assuming only charts), go through and grab info, then render to PNG
            for (Iterator i = rh.getBody().getContents().iterator(); i
                    .hasNext();) {
                Object o = i.next();

                //make sure this is an extended item handle
                if (o instanceof ExtendedItemHandle) {
                    ExtendedItemHandle eih = (ExtendedItemHandle) o;

                    // read in the XML Representation for getting the data
                    // definitions from the chart, get the values of the X and Y
                    // axis
                    String xSeries = getXSeries(getXMLResources(eih));
                    String ySeries = getYSeries(getXMLResources(eih));

                    //Look into using serializer to grab from a report design.
                    //alternative of doing this, although a little uglier
                    //ChartEngine.instance().getSerializer().fromXml(arg0, arg1)
                    
                    ChartWithoutAxes cwa = (ChartWithoutAxes) eih
                            .getReportItem().getProperty("chart.instance");
                    
                    // Get list of result sets
                    ArrayList resultSetList = (ArrayList) det
                            .getResultSetList();
                    
                    //we know out data is in the first result set
                    IResultSetItem resultItem = (IResultSetItem) resultSetList
                            .get(0);
                    String dispName = resultItem.getResultSetName();
                    
                    //tell the data extraction task to use the first result set
                    det.selectResultSet(dispName);

                    // retrieves the dataset with column/values as unique to map
                    // to Pie Chart
                    Map uniqueMap = getValueMap(det, ySeries, xSeries);

                    //crete the category, or X series, and values, or y series
                    TextDataSet categoryValues = createCategoriesDataSet(uniqueMap);
                    NumberDataSet seriesOneValues = createValueDataSet(uniqueMap);

                    //now that we have data and our chart, render to image
                    renderPieChart(cwa, categoryValues, seriesOneValues);
                }
            }
        } catch (ExtendedElementException e) {
            e.printStackTrace();
        } catch (XPathExpressionException e) {
            e.printStackTrace();
        } catch (DOMException e) {
            e.printStackTrace();
        } catch (EngineException e) {
            e.printStackTrace();
        } catch (ChartException e) {
            e.printStackTrace();
        } catch (BirtException e) {
            e.printStackTrace();
        }
    
        //shutdown the platform, we are done
        Platform.shutdown();
    }
}

Book: Excerpt Available Online

My Publisher has posted an article with an excerpt from my book on building a simple report. Check it out here: http://www.packtpub.com/article/creating-a-simple-report-using-birt

EclipseCon: The BIRT Theme Song

Scott Rosenbaum was nice enough to post Pierre Tessier singing the BIRT theme song and molesting my rubber duck... SFW

http://birtworld.blogspot.com/2008/03/fun-at-eclipsecon.html

Games: Super Smash Bros. Brawl

One of the great things about having free time again is that I have time to do thing I want to do, not things I have to do. So I spent the better part of this evening and early morning playing the new Super Smash Bros. for the Wii. After an annoying 2 hour wait in line with smelly gamers, I managed to score a shelf copy after one of the local gaming stores lost my reservation. I'm not bitter, I'm just glad I scored it.

So what do I think? The game is great. The control schema is simple and flexible, allowing you to use either the Wii controller and Nunchuck, the Sidways Wii Controller, the Classic Controller Add-on or the Gamecube controller. I must admit to being a little disappointed by the control schema using the wii controller, I was expecting some really cool motions and such for controls, but no such luck. The game instead uses the joystick part of the nunchuck and the various buttons for the incredibly simple punch, special moves, and block control schema. The simplistic scheme is what allows it to use any of the controller types, saving would be party hosts the expense of having to shell out 30 bucks for 4 Wii controllers.

So I had a group over and we spent the better part of 6 hours playing. The end result, we didn't jump over to Mario Party at all the whole night, staying engrossed in the combative competition, with my main character Donkey Kong taking the cake for the most wins, with my wifes character of Peach coming in second. The returning characters have all the same moves that they had in previous games, with DK returning with his Monkey Temper Tantrum ground slap, his "Donkey Punch", and the windmill spin. Mario has his ever annoying coin punch, spin, and stupid fireballs. Pikachu of course remains a pain with cheap lightning moves and such. Pretty much what you would expect in terms of a Smash Bros game.

The levels are a nice change. Unlike in Melee, Brawl has its own new levels, plus the addition of the levels in Melee. Noticeably absent, at least where we are at with only a few unlockables, are the original Smash Bros levels. Which brings up one of the only gripes I do have with the game. When I sit down with a party type game like Smash Bros or Mario Party, the last thing I really want is to have to unlock secret characters and levels. One of the big things we were looking forward to was Snake from Metal Gear, only to find out he was an unlockable. So, we will bare through it and try to unlock him. Also notably absent was the rumored original Smash Bros virtual console release. Although I am not disappointed by that, because this version has so much replay value.

So much to take in, and all we have done so far is play the multiplayer mode. I have yet to try and online gaming since I am selective as to who I will play with, and I have not tried solo play yet. The end result is the wait was worth it, and the game is incredibly addicting. I am sure there have been some nayser game reviews, part of the reason I quit reading game reviews by professional reviews is due to some ridiculous, quasi notion of "artistic quality" that they seem to judge games by, totally ignoring important aspects such as fun and entertainment, so to hell with those reviews. This game was worth putting up with smelly gamers and lame Star Trek jokes for two hours.

Update: I missed one important thing, you can create your own stages.... nuff said

ETL: Practical Example of Data Transformation Using Kettle

I’ve written about Kettle before. I personally think it is a great tool, and its easy to tell that this was written by someone who works with annoying data formats on a consistent basis. Well, Kettle is now known as the Pentaho Data Integration Project, and its now up to version 3. For brevity’s sake, I will just refer to it as Kettle going forward. Of course, none of this changes the functionality of the tool. In this article I am going to show a practical application of how I have been using Kettle to assist in the generation and transformation of annoying data formats.

I recently had to work with a feed from an entertainment vendor who distributes DVD, music CDs, and such. To the stores that use these files, they provide a rather confusing set of flat, tab delimited files, in a very un-userfriendly format. Since the store I was working for has a predefined format they want vendor files in to work with their search and navigation backend, we needed to transform these files to that format. This is where Kettle comes in.

The first file I need to deal with from them is their Products file. The file contains roughly 28 fields with various numbers that make no sense to me what so ever. The only fields I need to concern myself with are the Product ID, Product Name, Internal ID (used for mapping with the other files), a Category ID, the Artist, Price, and Availability Date. The other fields I can ignore. Since I am dealing with Categories I also need their Category file. Categories are going to be handled in a special way. I don’t need this in my actual file, but in a separate file, or files rather, that will be appended to a separate set of files only once. And the final file I need out of their set is their Attributes file, which will contain information about products such as if the product is Widescreen, Dubbed, Subtitled, etc. These are handled in a special way in the search backend, so I just need to provide them.

Figure 1. Conceptual Model of Data

What I need to do is transform this data into the format in Figure 2.

Figure 2. The Data Feed Format

The Artist field will go into Search Field 1, and the first two attributes I come across will go into search fields 2 and 3. Everything else will be a simple 1 to 1 mapping.

To get started, the first thing I need to do is start Kettle.exe, and create a new transformation. If you are using a repository, great, otherwise, choose No Repository at the startup screen. When you are in Kettle, go to File/New/Transformation.

Figure 3. New Transform

With the new file created, drag over 3 new Text File Inputs, located under the Core Objects/Input section, to the transformation area.

Figure 4. New Text Inputs

With the three text file inputs in the transformation, I need to set them up to read my data files. The first thing is to add the file to the input file list. Do this by clicking on Browse, then when you select your file, click on add. With the file selected, I now need to set up the delimiting information. These files are all tab delimited, with no text qualifiers (meaning no quotes around Strings), and no header row. So I click on the content and set the appropriate options. Since this is a tab delimited file, I need to click on the Insert Tab button to add in the tab.

Figure 5. Delimeter Options

Now I need to select the fields. To do that, since I have my file, all I need to do is click on the Get Fields button under the Fields tab. For developments sake, I will just name the fields that I need, and leave the field names alone for the remaining fields. Also, since I am just using these fields as description fields, I change the Date fields back to Strings.

Figure 6. Field selection

That’s it for Products, I do the same for the Categories and Attributes Tables. Now that the text inputs are set up, I need to do the transformations. The hardest part will be to denormalize and join the attributes into my input stream to feed into my output text file. The first step I need to take care of is sorting my data for the field row delimeter. So, I drag over a Sort Rows object from under the Transformation section. I need to connect the Attribute text data source to the Sort Rows object in order to edit it correctly. In order to do the connection, I need to hold down the Shift key on my keyboard, and drag my mouse from the Attibutes object to the Sort Rows object. This will indicate to the transformation that a “hop” in steps needs to occur between these two objects. Now, I edit my sort to sort based on the Product ID.

Figure 7. Sort Row Options

Next, I drag over a Select Values object and connect the Sort Rows object to it. In the Select/Alter tab, I click on Get Fields and leave everything default. Since I wont be using the non-named fields, I go over to the Remove tab, and select those fields.

Figure 8. Remove Fields

That was the easy part. Now I need to denormalize the data. What I want to do is have the first 3 attributes for each product to show up in consecutive columns. I tried using the Denormlizer here, with no success. So I ended up using the row flattener. The way the row flattener works is you define a single field that will contain the consecutive data. You then define additional columns. The flattener will then copy to each column in the order it receives data. So for example, lets say you have the following data defined in a field in your incoming data stream:

-Wide Screen

-Sub Titled

-Spanish

And in your row flattener, you defined the following target fields:

-FieldOne

-FieldTwo

-FieldThree

The flattener would assign the values like so:

-FieldOne = Wide screen

-FieldTwo = Sub Titles

-FieldThree = Spanish

It also seems that once unique values have been exhausted, it will just finish filling out the columns with the last unique value it encountered. For my purposes this is just fine. I define my field flattener with my values for Attribute Name in the following figure.

Figure 9. Field Flattener

So, with this final step in my transformation, my data stream for attribute will look something like:

-ProductID

-InternalID

-AttributeName

-Search_Field_1

-Search_Field_2

-Search_Field_3

-Search_Field_4

Now, I need to join it to my Products data stream. To do this I pull over a Join object. I need to join on my Internal ID or my Product ID since they both uniquely identify my product. So I will set that as the join. I also need to set this join type to Left Outer Join, since I need Products to show up even if there are no Attributes to join with. I also will set Products as my Primary feed with Attributes as my secondary feed. It is important that when you do the connections in the Transformation editor that you connect both the Products Text Input and the Row Flattener object.

Figure 10. Join

The final thing I need to do to my transformation is modify the Product ID. Since this is for a test feed, I need the product ID to be unique. I will do this with Java Script. I also need to modify the attribute field to remove any pipes in the data field, since my output file needs to be a pipe delimted text file. So, I will drag over a Java Script component, and in Javascript I will write the appropriate code to add in increments of 10 million to the product ID (which I will modify in sequential runs), and use the string replace method to replace any pipes.

Figure 11. Javascript Code.

The final part of this is to output my text file. I need to set my file name for output, delimiting options, and fields I will use in my Output File object.

Figure 12. Output Text Filename

Figure 13. Output Text Delimeter

Figure 14. Output text fields

You will notice in Figure 13 I have Append set. This will append every run of the transformation to the end of the output text file. I also have Price in twice since the Display Price and Actual Price are the same.

Now, the final part of this transformation is the Categories. The only thing I need special for Categories is a hard coded entry for my Super Category mapping in the internal search appliance for my categories in the vendor data feed. I accomplish this through a JavaScript component between the Categories Text Input and the Output files. I won’t show the Output files for Categories since it isn’t relevant to my main transformation with the Attributes and Products. The final figure shows my finished Transformation.

Figure 15. Finished Transformation

Now, I can create my dummy feed and add to it at the click of a button. Of course, this is applicable to actual production data transformations as well. Kettle does run a little slower than a hand coded solution, but not enough to rule it out due to the amount off time it saves in development and the amount of time saved in modifying it when necessary.

Wii: Error Code 209552 After System Update

So I got my Wii back online after a long period of neglect in that area. Since the wife and I have been playing a lot more, I decided it was time to get it back online. I'm also waiting to see if they ever release the original Super Smash Bros on the Virtual Console (no luck so far).

So, I get my Wii online, do the system update, and lo and behold, I can't connect to the Wii Store Channel. I kept getting an error 209552. I am not sure what caused this, since I have also recently changed wireless routers, but the dang thing wouldn't connect after the system update (had enough connectivity beforehand to at least get a system update).

To fix this, I ended up having to change the broadcast channel on my wireless router from 6 to 1. Once I changed that, everything worked like a charm again.

Book: Press Release

Design and create reports quickly with the Eclipse-based Business Intelligence and Reporting Tools system using new book

Packt is pleased to announce a new book on Business Intelligence and Reporting Tools (BIRT) that provides understanding and structure in a fast paced, task driven and tutorial style. Practical Data Analysis and Reporting with BIRT focuses on the most visible and familiar product built with the BIRT framework, which is the BIRT Report Designer.

BIRT, which stands for Business Intelligence and Reporting Tools, is an Eclipse-based open source software project that provides reporting and business intelligence capabilities for rich client and web applications, especially those based on Java and J2EE. BIRT is in fact a collection of development tools and technologies used for developing reports utilizing the BIRT runtime framework component on an application server. BIRT has two main components: a visual report designer within the Eclipse IDE for creating BIRT Reports, and a runtime component for generating reports that can be deployed to any Java environment.

This book has a fast-paced, task-driven, tutorial style, which provides understanding and structure, not just lists of steps to follow. The focus is on the most visible and familiar product built with the BIRT framework, which is the BIRT Report Designer. The BIRT Report Designer is an Eclipse plug-in that utilizes BIRT technologies to allow users to design reports in the BIRT document format. Also covered is the BIRT charting engine, which lets you add charts to your application.

Java developers who want to get reporting as quickly as possible will find this book useful. The book is published by Packt and is available now. For more information, please visit: http://www.packtpub.com/practical-data-analysis-reporting-with-birt/book