MapForce Server Archives

Tag Archive for: MapForce Server

Transforming and Converting Protobuf

February 10, 2022/in Data Integration, Database, JSON, protocol buffers, XML /by David McGahey

MapForce supports mapping protocol buffers (Protobuf) to and from other structured data formats as mapping sources or targets. In the constant quest for more efficient ways to transfer, manipulate, and manage large structured data sets, Google has created a language- and platform-neutral data format similar to XML, but smaller, faster, and simpler than even JSON data. Tools are available to generate and work with Protobuf using Java, Python, C++, C#, Ruby, and other programming languages.

The structure of any Protobuf message is defined in a .proto file that defines each field name and value type. Altova MapForce lets users drop these .proto files into a data mapping as a source or target along with any other data, including XML, JSON, relational databases, Excel, flat files, REST and SOAP web services, and others. .proto files versions 2 and 3 are supported.

A MapForce data mapping creates compatibility between existing XML, JSON, database or legacy data formats and new applications leveraging the efficiency of Protobuf.

Transitioning Data Mapping Projects from Development through Testing and Production

November 29, 2019/in Data Integration, Database mapping, Servers, Software /by David McGahey

Data mapping projects often mirror software development efforts with distinct phases for design, testing, and deployment. This is especially true for ETL (Extract Transform Load) projects when repeated data mapping execution is required as new data becomes available, and the stakes increase higher with large data sets. The Altova MissionKit and Server Software products provide Global Resources to define configurations for each project phase and smoothly transition between them.

Let’s take a look at an example based on a MapForce data mapping from a source file to a database.

Handle HTTP Errors During Automated Data Integration

July 29, 2019/in Data Integration, JSON, Tools /by David McGahey

Data analysts and other professionals often need to generate real-time data through automated execution of data mappings that request Web services and save the results. During automated execution it’s important to gracefully handle any unexpected HTTP error rather than terminate the integration task.

In an earlier post we discussed conditional processing of a REST Web service response to handle HTTP errors, where separate output files were generated for a normal response and an error. Now let’s look at a revised mapping solution for the airport status example to generate a single mapping result file that contains either the requested airport status or a description of the error.

Handling HTTP Errors in Web Service Data Mappings

June 19, 2019/in Data Integration, Tools /by David McGahey

Data integration projects that include information from external Web services may be vulnerable to HTTP errors when retrieving remote data. When data mappings run under automated control it’s especially important to detect and report errors even if errors only occur very rarely.

A MapForce data mapping can include Web service calls and output the result directly to a file or database, or combine it with other inputs for further processing. Regardless of the final output, an HTTP Web service error encountered in a REST Web service request puts the mapping at risk.

MapForce includes features for handling HTTP errors instead of simply aborting execution of a mapping. Developers can configure the body of a REST Web service call to handle and report exceptions based on the HTTP status code returned.

Let’s look at an example.

Job Distribution on FlowForce Server

November 28, 2018/in Data Integration, Servers, XBRL, XML /by Erin Cavanaugh

FlowForce Server is Altova’s high-performance engine for automating workflows of XML processing, data integration, report generation, and more. It integrates with other Altova server software products to automate their functions, such as executing complex data integration processes, including ETL projects, designed in MapForce; running StyleVision report generation jobs; or validating XML, XBRL, or JSON files with RaptorXML Server.

Starting with Version 2019, FlowForce Server offers new options for distributed execution and load balancing to improve availability and performance. Let’s take a look at how configuring multiple FlowForce Servers to run as a cluster can help improve data throughput and provide redundancy.

Job distribution for high availability on FlowForce Server

MapForce Server Accelerator Edition Achieves a New Level of Data Transformation Performance

May 23, 2017/in Data Integration, Software, Tools /by David McGahey

MapForce Server automates recurring execution of data mappings and transformations designed and tested using Altova MapForce. Every day, MapForce Server is employed in business communication, financial reporting, database ETL, and many other applications to transform critical data between any of XML, JSON, database, EDI, XBRL, flat file, CSV, Excel, and/or Web service formats.

Now, MapForce Server Accelerator Edition offers even faster throughput for high-performance server platforms.

Use Join to Integrate Data in Any Format

April 18, 2017/in Data Integration, Tools /by David McGahey

Join is a powerful SQL operation implemented across most database types and familiar to database users. Join is typically used to select and combine information from multiple database tables.

Altova MapForce includes a join component for data mapping that works like a SQL join for database tables and extends data integration functionality by empowering users to join data trees of any data format. Anyone familiar with join operations for database tables will find the MapForce join component especially intuitive. A join operation in MapForce can even combine two different data formats and produce output in a new format altogether.

A Typical MapForce Server Use Case

March 2, 2017/in Data Integration, JSON, Tools /by David McGahey

Envision a manufacturing company that controls costs by exploiting a just-in-time assembly process with a very low supply of parts inventory on hand. New customer orders are logged in a sales database and at the end of every day the components needed to assemble that day’s sales are tabulated.

The IT department runs a SQL query to identify the required parts and transforms the list into a purchase order in JSON format to be transmitted to the supply chain.

Sound familiar? Our recent blog series on JSON tools and JSON data mapping were based on this real-life scenario. In this post we describe a MapForce Server use case that automates the repetitive task of generating each day’s purchase order.

Faster EDI Data Translation

March 7, 2016/in Data Integration, EDI /by David McGahey

Electronic Data Interchange (EDI) standards accelerate commerce worldwide by allowing companies and other organizations – even those in different regions, using different languages and currencies – to send and receive unambiguous information. Some EDI communication protocols and message formats still in wide use today were developed over 30 years ago, when telecommunication systems were slower and data storage was more expensive.

119647393_700px

Automating Data Integration Workflows – Download the E-Book

August 19, 2014/in Applications, Data Integration /by David McGahey

We’ve published an e-book titled Automating Data Integration Workflows with Altova FlowForce Server that is now available for free download from the Altova Web site.

Here is a brief description: FlowForce Server is a highly-customizable platform to automate data transformations defined by MapForce data mappings, report and document generation, and other tasks on dedicated servers, virtual machines, or workstations scaled for the scope of the project. FlowForce Server empowers data architects, analysts, and other IT professionals to efficiently complete enterprise-level data integration tasks.

Automating Data Integration Workflows with Altova FlowForce Server provides an overview of the Altova FlowForce Server platform through a series of real-world data integration tasks and examples.

Automating Data Integration Workflows with Altova FlowForce Server

The new e-book collects updated information from the Altova Blog, the Altova Web site, and some all-new material, organized in one location to help new FlowForce Server users get started and to suggest new applications and efficiencies for experienced users.

Installing Altova Server Software in the Cloud

July 8, 2014/in Software /by David McGahey

Cloud-first is becoming the new normal. At recent events we have frequently been asked about using Altova Server tools in the cloud. The answer is definitely, go for it. The installation is easy. In fact, we use Altova Server Software products ourselves for an internal reporting application, installed on local virtualized servers and on an AWS cloud instance. The charts below were generated by StyleVision Server running in the cloud to quickly communicate information about changes in dynamic data.

StyleVision Server is based on the built-in report and document generation engine developed for StyleVision and renders .SPS stylesheets originally designed in StyleVision, including features like a rich variety of charts to visually represent data.

In this post we will walk through the installation of FlowForce Server, MapForce Server, StyleVision Server, and RaptorXML Server for a complete data integration solution in the cloud.

Deploy Data Mappings and Report Designs for Automated Processing

March 4, 2014/in Data Integration, XSL /by David McGahey

Deploying data mappings created in MapForce and report designs created in StyleVision for automated processing by Altova server products is straightforward and quick.

The File menu in MapForce includes two options to optimize, preprocess, and deploy data mappings for MapForce Server and FlowForce Server. Preprocessing enables faster performance and reduced memory footprint for most data mappings. MapForce menu option to deploy to FlowForce Server Read more…

Tags: FlowForce Server, MapForce, MapForce Server, StyleVision, StyleVision Server

Filesystem Commands and More Wizardry with FlowForce Server Built-in Functions

August 13, 2013/in Data Integration /by David McGahey

In our earlier posts on FlowForce Server, we described jobs that execute MapForce Server for data transformations, StyleVision Server for report and document rendering, and RaptorXML Server for XML processing. In each scenario we also used built-in system functions to copy move, or delete files, as is often required in real-world workflows.

95662684_700px

In this post we’ll take a look at more of the built-in functions automatically installed in the FlowForce Server system container. You can use these commands as execution steps to automate the file housekeeping so often required in enterprise production.
Read more…

Tags: FlowForce Server, MapForce Server, RaptorXML, StyleVision Server

Taming Bad Input Data with FlowForce Server

June 18, 2013/1 Comment/in Data Integration /by David McGahey

Whenever you accept data from an outside source you risk encountering errors. We have blogged about this phenomenon in the past in Expect the Unexpected – Altova MissionKit Solves a Number Format Mystery and in the series of posts on Processing the Groupon API.

Bad data in an input file can cause the data transformation step of a FlowForce Server job to fail. When a FlowForce Server Job fails, further execution steps will not be performed. FlowForce Server is designed this way to prevent an error in one job step from cascading into a series of additional invalid results. Happily, FlowForce Server also includes features to help you recover from errors and keep production flowing.

In this post we will further extend the data mapping and report rendering job described in Customizing a FlowForce Server Job to gracefully handle bad data in an input file.

Come see us at TechEd in New Orleans!

May 29, 2013/in Events /by Cindy Neely

Altova will be exhibiting in New Orleans, LA from June 3-6 at Microsoft TechEd in booth #429. Come on down to the Morial Convention Center and speak with our experts about our new line of cross-platform server software products: FlowForce Server, MapForce Server and StyleVision Server. If you are attending TechEd, stop by our booth to see first-hand how these new products offer high-speed automation for projects designed using familiar Altova MissionKit developer tools. While you are at our booth mention this blog post to receive a special giveaway. We can’t wait to see you next week in New Orleans!

Tags: Altova events, FlowForce Server, MapForce Server, StyleVision Server, Tech Ed, tradeshows

Customizing a FlowForce Server Job

May 21, 2013/1 Comment/in Data Integration /by David McGahey

In our earlier post titled Automate Data Mapping and Transformation with FlowForce Server, we created a job called SimpleMapAndTransform to automate data mapping with MapForce Server and creation of html reports by StyleVision Server. After the FlowForce Server job ran several times, we have accumulated many output files in the same folder we use to process input files, as well as temporary intermediate files in the workFiles folder, as seen in the image below.

FlowForce Server job execution log and working folders

In this post we will enhance the job to create more orderly results and remove unneeded temporary files.
Read more…

Tags: data mapping, FlowForce Server, MapForce Server, reporting tool, StyleVision Server

Automate Data Mapping and Transformation with FlowForce Server

May 8, 2013/1 Comment/in Data Integration /by David McGahey

Altova FlowForce Server, launched on April 29, 2013, includes numerous enhancements over earlier beta releases and one of the most exciting new features is the implementation of StyleVision Server. Now a FlowForce Server job can automate a complete data transformation workflow by executing MapForce Server for data mapping and pipelining results to StyleVision Server to render a variety of output formats.This post describes a straightforward FlowForce Server mapping and transformation job as illustrated in these messages from the FlowForce Server Log, with the most recent step at the top of the list:

FlowForce Server Job Log showing MapForce Server and StyleVision Server job steps

We’ll start with the GPS log files created by a digital camera. We wrote about these files last January in the post titled Process Multiple Input Files in a Single Data Mapping. We’ll use the mapping output with a StyleVision SPS stylesheet adapted from the XPath Enhances XML Reports post to produce a time and elevation report for each file.A FlowForce Server For-each job step repeats based on the result of an expression. We can use For-each to build a list of files in a folder, then repeat one or more steps for each file. Here is how it looks in the job configuration page:

FlowForce Server data mapping job step

The line labeled Execute function defines the mapping to be used by MapForce Server, and the input parameter {file} refers to each file in the list C:CameraGPSexample*.LOG.We can add an execution step to instruct StyleVision Server to perform the transformation:

FlowForce Server transformation job step

The data mapping creates output files by adding .gpx to the name of the input file, and now we can define the transformation input using the {file} variable with the new file suffix. We chose to create .html output, but we could just as easily create other formats for a multi-channel publishing implementation.The transformation working directory is the location where StyleVision Server unpacks the contents of the .pxf file containing the stylesheet, XML Schema, and other needed components. Using a dedicated working folder will keep the workflow more organized.We want to allow network users to drop new .LOG files into the C:CameraGPSexample folder and we want run the FlowForce Server job on a regular schedule, but we don’t want to process the same files over and over. We can define one more job step to move the processed file to a different location:

FlowForce Server file move job step

The complete FlowForce Server job is a series of three steps that loops for each .LOG file found in the folder. We can set up a repeating trigger for the workweek or any other appropriate schedule:

FlowForce Server calendar-based job trigger

Here is a portion of a .LOG file created by the camera that is an example of one input file:

.csv input file example

We can drop this file into the C:CameraGPSexample folder, where it will be processed based on the FlowForce Server job trigger:

Workflow folder structure

When the timer triggers execution of the FlowForce Server job, the Web interface Job Log page displays this series of messages for the complete job:

FlowForce Server job log for complete job execution

The contents of the C:CameraGPSexample folder now look like this:

Completed work files after FlowForce Server job execution

We can examine the 121130.LOG.gpx file in XMLSpy:

XML file created by MapForce Server job step

And we can open the .html file in any Web browser:

.html output from StyleVision Server job step

In future blog posts we will enhance this FlowForce Server example to illustrate jobs with error handling and more complete cleanup of working files.FlowForce Server is available for Windows, Linux, and soon for Mac OS platforms. To get started yourself, click here to download a free trial!

Tags: data mapping, FlowForce Server, MapForce Server, reporting tool, StyleVision Server, XML charts

New Server Products for Data Transformation

April 29, 2013/in Data Integration /by David McGahey

Altova Server Products On the heels of our RaptorXML announcement earlier this month, we are very excited to tell you about the availability of three more new products that round out the Altova server software product family.

These high-performance server products are optimized for multi-CPU servers and are highly complementary when used together for defining and managing information pipelines. Cross-platform support is immediately available for Windows and Linux, with MacOS support expected just around the corner in May. Supported languages include English, German, Spanish, and Japanese.

Let’s take a look at each of these new products in detail.

FlowForce Server

After receiving valuable customer feedback during the beta period, we have announced general availability of FlowForce Server, a powerful new tool for managing multi-step, enterprise-level data aggregation, processing, and reporting tasks. The straight-forward FlowForce Web browser interface makes it easy to configure and monitor jobs, whether they require a single step, or multiple cascading steps starting with data aggregation and conversion, and completing with multi-channel report generation.

FlowForce Server Job Log

A flexible and highly customizable system of job triggers lets you schedule FlowForce Server job execution based on each job’s unique requirements. Jobs can be triggered by date and time, by the arrival of files in hot folders, or by http command – and you can define multiple triggers per job. To take full advantage of server resources and meet the demands of busy data transformation workflows, multiple jobs – even multiple instances of the same job – can run simultaneously on FlowForce Server.

FlowForce Server Job Triggers

FlowForce Server provides strong security measures through user, role, job credential, and permission definitions that enable multiple simultaneous jobs while preventing unauthorized access to restricted data repositories or system resources.

So, what kind of jobs can you set up on FlowForce Server? Most customers will want to use it in conjunction with MapForce Server and/or StyleVision Server for high-performance automation of data integration and multichannel publishing projects that doesn’t require writing or compiling any source code.

FlowForce Server enterprise workflow

MapForce Server

You may already be familiar with MapForce, Altova’s desktop developer software for aggregating, mapping, and converting data. We describe it as an any-to-any data mapping tool because it lets you move between any combination of XML, SQL database, flat file, EDI, Excel, XBRL, and Web services data. With the introduction of MapForce Server, it’s now easy to fully automate those transformations that have been defined in MapForce.

MapForce Server performs data transformations based on preprocessed and optimized data mappings stored in MapForce Server Execution files prepared by MapForce and uploaded over a network. Preprocessing enables faster performance and reduced memory footprint for most data mappings.

When operating under the management of FlowForce Server, MapForce Server is a powerful transformation engine suitable for the high-volume, quick turn-around demands of today’s leading edge data centers. The standalone version is suitable in a department level or other small environment that does not require the extensive multi-tasking, job scheduling, and security features of FlowForce Server.

StyleVision Server

StyleVision Server provides high-performance automation of report and document generation tasks based on templates designed in Altova StyleVision. A graphical XSLT stylesheet and report design tool, StyleVision helps you create one design to render XML and XBRL data in Web and print formats including HTML, Word, RTF, and PDF, simultaneously. Using StyleVision Server, you can generate reports from XML and XBRL data on a recurring basis faster than ever before.

Like MapForce Server, StyleVision Server is an important option for document production by FlowForce Server, and the standalone version can be run from the command line and is suitable in smaller deployments that may not require all the features of FlowForce Server.

This new line of Altova server products represents a huge leap forward in effective automation of essential business processes, whether they require a single step, or multiple cascading steps starting with data aggregation and conversion via MapForce Server and completing with multi-channel report generation via StyleVision Server. These high performance workflow servers help customers design information pipelines quickly and easily, and then deploy them in a flexible way, utilizing the power and speed afforded by multi CPU servers. We’re excited to offer this new family of server products to our customers developing data integration and reporting projects using Altova MissionKit developer tools, as well as welcoming new customers who need a multi-platform, lightning fast orchestration engine.

Download a free fully-functional 30-day trial!

As with all Altova tools, we invite you to try these new server products before making a purchasing decision. Download a free, 30-day trial now!

Tags: FlowForce Server, MapForce Server, StyleVision Server