Talend Open Studio 3.0

If any software market deserved to be shaken up by open source alternatives, it's enterprise data integration. Commercial, enterprise-grade integration tools - typically cobbled together from M&A and legacy patchworks - are notoriously unwieldy and impose an arduous learning curve. Complexity frequently stalls deployments by months, and aftermarket consulting can add hundreds of thousands of dollars to the TCO.

Enter Talend to this land of stodgy giants. Talend hits all the highlights one would look for in traditional integration platforms: batch delivery, transforms, ETL (extract, transform, and load), data governance, and a strong set of connectivity adapters. At the same time it keeps pace with important trends with such features as change data capture, metadata support, federated views, and SOA-based access to data services. Talend is capable of scaling from small departmental file migrations to large-scale enterprise warehousing projects.

It may not yet surpass the master data management and messaging transform prowess of IBM Information Server, or the legacy and business-to-business domain expertise found in Informatica PowerCenter. But they offer substantial cost savings compared to these commercial counterparts, and their ability to shortcut complexity makes them additionally hard to resist.

Talend has developed a holistic integration platform from the ground up in a very short time. If the company continues on its current trajectory, it could do for data integration what open source has already accomplished for servers and databases.

New features in Version 3 go a long way toward bolstering enterprise viability. In addition to a native SAP connector (extract and sync), developers will appreciate component search, an ecosystem overview of projects, change impact analysis, and drag-and-drop metadata.

Perhaps most important, Talend has added change data capture (specifically, via slowly changing dimensions). Change data capture enables real-time updates that significantly reduce the size of data transfers - an increasingly important efficiency measure for data sets that have grown so large, there's no longer enough time to complete batch runs in the overnight hours.

What I really like about Talend is its code-generating approach - a practice that fell by the wayside in favour of higher-level, user-friendly tools built around a centralised, proprietary engine. Although the proprietary "black boxes" often help streamline development, they can also lead to processing bottlenecks and scalability issues.

By contrast, Talend jobs can be packaged up and deployed anywhere a Java Virtual Machine or Perl interpreter can reside. Jobs can also be embedded direct into your Java apps or even encapsulated as REST/SOAP web services via easy export.

Not that Talend is suitable to every enterprise project. It's light on the connectors to mainframes and minis that you'll find in commercial products such as ETI Solution V6, a comparable code-generating solution that can output native code in Java as well as Cobol, C/C++, and SAP.

Open source competitor Pentaho Data Integration (Kettle), despite taking a black-box approach, does offer good control over distributed processing, as well as integration into a more elaborate set of tools for BI and EAI. Nevertheless, I prefer Talend; it's better developed and more extensible than Kettle, and it offers superb data governance.

Integrator's studio

Deploying the pieces of Talend Open Studio - namely Job Designer, Business Modeler, and the repository manager - is straightforward. I installed to a Windows Server 2003 platform with Sun JVM and ActiveState Perl, and was quickly off and running.

The Business Modeler component - a nice touch for Talend - is a piece of the puzzle often omitted even at the commercial level. The Business Modeler provides a palette of components that allow non-technical analysts to build a view of the system and its workflows, without ever touching a drop of Java. The result gets turned over to developers, who flesh out the details using the Job Modeler, an Eclipse-based IDE and debugger.

The Job Modeler will put any Eclipse-seasoned developer at ease with its own palette of drag-and-drop components. It also provides access to the central repository, which holds all of your organisation's business models, job designs, metadata, documentation, and connection-specific information.

The latest version of Job Modeler adds collapsible subroutines for easier navigation. Other niceties include quick tabbing between graphical layout and code, a job scheduling interface (that puts a GUI on the Unix crontab command), and a thumbnail overview for easy navigation of large document layouts.

I liked the tMap component for defining my transforms and data routings. Although it was reminiscent of an old switchboard with wires strewn about, it was ultimately fast and effective. An Automap option saves time setting up initial connections.

The Job Modeler IDE's graphical SQL editor and test facility, called SQLBuilder, helps with SQL chores. Talend generates native SQL code for every supported database, no additional effort required. XSLT and XPath are in tow for XML processing. And a good set of orchestration components makes long-running and staged processing a possibility.

Onboard debugging offers step-by-step trace and variable inspection, with real-time stats and trace data viewable directly from the layout. Other niceties, like auto- generation of HTML documentation, sweeten the offering.

You need to be able to trust the accuracy of your data, not just push it around. Talend has data governance covered with good provisions for data quality and profiling. Data conformity and consistency, beyond de-duplication, is achieved using filters such as search-and-replace, interval- and fuzzy- matching, and schema-based transformation. The profiler adds metrics on data quality - tracked and assessed over time - and graphically depicts stats and performance summaries for quick isolation of data in need of scrubbing.

I was impressed by Talend's rich set of components for third-party products, too. Support ranges from the higher end of OLAP cubes and Microsoft AX Server, down to QuickBooks and Google Apps. Even open BI solutions, including Jaspersoft and SpagoBI, as well as CRM apps, including Salesforce.com, Sugar, and Centric CRM, are supported.