Tag Archive for: ETL

How to Create Batch Data Mapping Projects


A common requirement in data processing is batch data mapping, especially in the context of data transformation and integration. It involves converting data in batches rather than processing individual data points one at a time. Batch data mapping is often required in data integration scenarios where input from multiple sources needs to be aligned or transformed together. Two common scenarios are “batch to batch” and “batch to one.”

In our latest series of MapForce demo videos, we explore these common data mapping challenges.

Decorative image: yellow piping in a factory setting

BATCH TO BATCH DATA MAPPING

Batch to batch data mapping is helpful in scenarios where you have data updates or changes coming in batches, and you need to synchronize or transform these batches together. This could be to convert them to a different format, perform some type of sort or calculation, or a combination of these.

In this demo, we create a data mapping project that reads files from a directory and uses wildcards to set up a mapping that will process data from multiple files at once. Then, we explore another option for defining batch conversion using dynamic file names supplied by the mapping. This demo also shows how to add calculations and comments to your data mapping projects.

While this video highlights a batch to batch transformation of JSON files to XML files, MapForce supports conversion and transformation for any combination of XML, JSON, PDF, database, text, Protobuf, Excel, XBRL, and so on, for advanced data integration and ETL processes.

BATCH TO ONE DATA TRANSFORMATION

Batch to one data transformation is another common requirement, for example, when you want to merge or combine multiple files into a single consolidated document and perform some data transformation, conversion, or calculation operations in between.

This example also explores multiple approaches to defining the batch process, which will be applicable in different scenarios:

  1. Specifying a collection of files in the same directory using a wildcard
  2. Selecting batch files based on a list of file names stored in a different directory

This demo also shows how to sort the data merged from multiple files before writing it to the target.

After watching these quick demos, you can download a free trial of MapForce to try batch mapping, conversion, and transformation for yourself.

MORE MAPFORCE DEMOS

If you liked these videos, check out the rest of the MapForce demo series, which covers everything from mapping XML and JSON to databases to configuring data processing functions and extracting data from PDF documents.

Tags: , , , , ,

Extract Data for PDF Mapping


MapForce, Altova’s award-winning data mapping tool, includes support for PDF input in data integration and ETL workflows. The MapForce PDF Extractor makes it easy to define rules for extracting PDF data in a structured format to make it available for mapping to other popular formats like Excel, XML, JSON, databases, and more.

Let’s take a look at how it works.

Read more…
Tags: , , , ,

How to Create a Chained Data Transformation


Data mapping plays a vital role in modern data-driven organizations, enabling efficient data management and integration. Altova MapForce is a powerful, graphical data mapping tool that supports endless data transformation scenarios, including one-to-one, one-to-many, many-to-one, and chained data conversion.

While there are applications for each of these approaches, chained data mapping is especially helpful for complex data processing tasks where multiple stages of data manipulation are required. Here’s a look at the benefits of a chained data conversion approach – and a video of how MapForce makes the process easy and straightforward.

Decorative image: three business people working on laptops
Read more…
Tags: , , , ,

How to Convert Data in MapForce [Video]


Altova MapForce offers a powerful, yet easy-to-use, approach to converting and transforming data. Whether you need to write XML to a database, convert JSON to EDI, or map Excel to multiple different data formats, MapForce has you covered.

From simple one-to-one conversions to complex ETL scenarios, the MapForce approach is to represent data structures as graphical components. To associate fields, drag and drop connecting lines. A comprehensive library of data filters and functions is available for transforming data before writing it to the target.

Decorative image showing a person working on a laptop with data formats supported by Altova MapForce in the background

We have recently revamped our series of Introduction to MapForce videos. Each short how-to gives a demo of a common MapForce scenario.  

Start at the beginning to learn how MapForce works:

And follow along to learn:

You can follow along with the examples in these how-to videos by downloading a free, 30-day trial of MapForce. Check back for new MapForce videos, which are added often.

Tags: , , , , , , ,

How to View Workflow Automation Reports on FlowForce Server


Altova FlowForce Server is a high-performance workflow engine for automating enterprise-level data processing, data integration, and ETL jobs.

FlowForce Server includes a helpful web interface for managing and monitoring all aspects of data processing jobs, including in-depth logging functionality and a complete visual dashboard. Let’s take a look at how you can take advantage of FlowForce charts and statistics to monitor the progress of FlowForce Server jobs, as well as performance of the server itself, in great detail.

Article about FlowForce Server
Read more…
Tags: , , , ,

MapForce Supports SQL Merge When It’s the Right Tool for the Job


Large database tables can easily contain a million, even hundreds of millions of rows of data. Database administrators and others charged with maintaining such large datasets are always concerned about execution time for ETL (Extract, Transform, and Load) operations, updates, and other SQL queries. To make these operations more efficient, some — but not all — database vendors implemented a SQL merge statement to insert or update rows of an existing table as a single bulk-insert statement rather than requiring individual statements for each row.

Altova MapForce automatically supports SQL merge when it is available for the target database. Let’s look at an example.

shutterstock_66084286

Read more…

Tags: , , ,

Altova MissionKit 2012 Released Today


Fall is bringing cooler temps, shorter days, and beautiful foliage to us here in the Eastern US and parts near and far. As we celebrate the changing season with fresh apple cider and warm sweaters, we’ve also been working hard to deliver Altova Software Version 2012. This year, Altova’s fall release brings support for HTML5 and CSS3, enhanced functionality for ETL applications, new Java-friendly options, Model Driven Architecture, and more, to the MissionKit software tool suite. clip_image001 Let’s take a look at the new features in detail here, starting with tools that help developers and designers take advantage of HTML5 to create more sophisticated Web pages and apps.

HTML5 and CSS3

The latest version of the lingua franca for publishing content on the Web includes numerous improvements, from enhanced handling of multimedia content to increased interoperability. As Web developers transition to HTML5, they’ll need an intelligent HTML editor that can provide context-sensitive editing guidance and other time-saving features. XMLSpy 2012 adds support for HTML5 in its HTML editor with entry helpers and drop down menus that offer valid choices based on your cursor location, and an integrated Browser View. It’s also lightning fast to test your edits in multiple browsers directly from the HTML editor. Other features such as integrated project management and support for popular version/source control systems – as well as SharePoint® Server support – make XMLSpy a powerful tool for HTML development. clip_image002 For web designers and developers looking to create HTML5 pages from XML, database, or even XBRL content using a graphical, drag and drop design tool, StyleVision 2012 also supports HTML5 for creating stylesheets and reports. Both XMLSpy 2012 and StyleVision 2012 also support CSS3 styles to complement the new HTML5 functionality.

Enhanced functionality for ETL

The MapForce 2012 data mapping tool now supports streaming reading of files, a crucial feature for ETL (Extract/Transform/Load) applications. When executing data mapping projects, the built-in MapForce engine can now read extremely large XML, CSV, and FLF files and create correspondingly large output streams. MapForce can even theoretically read an entire relational database in a single pass and generate an XML or other output file at once. This new support makes MapForce a highly effective, lightweight, and scalable tool for ETL. It’s also uniquely affordable, whether purchased as part of the MissionKit tool suite or individually.

New options for Java developers

The Altova MissionKit is designed to support users in their preferred development environment, whatever that may be. To that end we’ve added some new Java-friendly options, including support for JDBC database drivers in all database-enabled products. clip_image003 We’ve also introduced a completely redesigned, native Java API for automating the functionality of Altova MissionKit tools in custom applications. This revamped Java API joins the existing COM API, and the products also ship with code samples in various programming language to get you started using the APIs right away.

Model Driven Architecture (MDA)

In response to requests from our UModel customers, we’ve added support for MDA in Version 2012 of our UML modeling tool. Utilizing a Model Driven Architecture approach to software engineering in UModel provides two primary advantages:

  • During the design phase, developers do not need to be concerned with the details and variations between software languages
  • An existing UModel project can be transformed from one source code language to another. For instance, a UML model for a C# application can become a Java or Visual Basic project

clip_image004 Model Transformation can even be applied to projects that are reverse engineered from existing source code. For instance, an existing Java application can be reverse-engineered by UModel to create a UML model that can be transformed to generate C# classes.

Upgrade info

Check out the rest of the features added in the MissionKit 2012. This new version is free to download and install for customers with Support and Maintenance. If you’re not already a customer, you can download a free, fully functional 30-day trial.

Tags: , , , , , ,