Announcing the Next Generation of Netezza Connectivity for Information Server
Better, Stronger and a whole lot Faster !!!!
I pleased to announce that the we have just released our next generation connectivity for IBM Netezza. This is a fantastic evolution to what was already a compelling integration provided by the original Netezza Enterprise Stage. InfoSphere Information Server and Netezza Appliances now work together better than ever to deliver break-through analytics .... yes, a little Six-Million Dollar Man like.... better, stronger and a whole lot faster !!!!
The Netezza Connector provides shared metadata and native connectivity across all data integration (both server and parallel), data quality and data profiling use cases. It provides a rich set of out-of-the-box features for creating tables with different distribution key options, support of executing UDX functions and Netezza statistics collection.
What I find the most cool are the automation features which have been extended to the Netezza Connector. These capabilities allow developers to solve complex problems with simple selections. They include:
- Multi-action modes that automatically attempt a secondary operation, including “Update then insert” and “Delete then insert”, if the first operations results in an error
- Action columns that allow a specific data value in the data integration stream to drive the operation (insert, update, delete) on the database
- Generated Netezza specific SQL for both the table actions and the creation of database objects based on the data fields being passed through the data integration job.
- Dynamic SQL Execution using job parameters or through a SQL file, executing the sql in a single node or multiple node.
- Partitioned reads through scaling out multiple read requests and automatically merging those results.
- Staging Table Management with choice of loading directly into target table or staging table, and automatically promotion from staging table to the target table
Balanced Optimization allows job logic to be executed in full or in part on the source or target database. The benefits of this flexibility are obvious, most specifically in cases where data integration tasks require homogenous processing (i.e. the source and target for the logic is the same database). In these scenarios, processing a set of logic directly on the database can have tremendous payback.
The new Netezza Connector makes it possible to leverage the features of our DataStage Balanced Optimization product. After following the same job design as he would for any other activity, the user chooses which pieces of job logic they would like performed on the database and dynamically choose which job to run based on appliance and ETL server’s load utilization. The product makes it simple to make this choice by offering intuitive selections like "push data reduction processing to the database". The unique Information Server approach allows these capabilities to be used in-flight with the Information Server engine - for the unique transformations and quality elements it provides – all without ever landing data to disk and while maintaining parallel processing throughout. This combination makes it possible to leverage massive scalability at whatever step necessary to achieve business objectives asap.
Faster .... oh yeah ....
In bringing this next generation to life, the development team spent considerable time in optimizing the code to maximize throughput rates and then prove them out through some grueling benchmark tests. At peak configuration, the connector for Netezza load and unload achieved performance throughput in excess of 2 TB/hr!!! If you are interested in that specifics of that benchmark, please drop me a note at email@example.com. I'd be happy to share a view on the commodity hardware we used to achieve that.
If you are an existing Information Server and Netezza customer, you might be interested in knowing how that stacks up against the original Netezza Enterprise stage which had also used native connectivity features. Thanks to the optimization efforts in the new connector, our lab test have shown a double-digit performance gain in reading and loading information. So, if you are already using Information Server with Netezza the new stage should also prove of great value to you.