PDF documents are used at many stages of modern business workflows, often serving as the format of choice for invoices, reports, legal contracts, and other critical documents. While PDFs are ideal for preserving content integrity and a particular visual layout, their structure makes automated data extraction challenging. For organizations engaged in data integration and ETL, unlocking information contained in PDFs is a necessity—and this is where the MapForce PDF Extractor comes in.
The MapForce PDF Extractor includes multiple tools for visually defining extraction rules to map PDF data to other formats. One that is particularly useful for zeroing in on specific content is text search. Here’s how it works – including a video demo.
The Altova Server Platform is comprised of the complete family of Altova’s high performance server software for automating data processing and data integration workflows. These cross-platform server software products allow for flexible installation either on premises or in any private or public cloud infrastructure.
For customers utilizing the Microsoft Azure cloud, we’ve created a convenient, free VM template with the Altova Server Platform pre-installed for easy deployment, available on the Azure Marketplace.
We’ve reported previously on support for node functions that simplify mapping structured data by eliminating need to copy-paste a function multiple times into a mapping. Repeating the same function unnecessarily clutters the mapping layout and makes the data mapping more difficult to understand or revise.
MapForce also includes additional filters are available for defining node functions. These parameters allow developers to apply functions and default values to specific nodes based on custom-defined criteria. For example, you can apply a node function based on node metadata such as the node name, node length, precision of the node’s data type, customized node annotations, and more.
Let’s look at a mapping with enhanced node functions.
A common requirement in data processing is batch data mapping, especially in the context of data transformation and integration. It involves converting data in batches rather than processing individual data points one at a time. Batch data mapping is often required in data integration scenarios where input from multiple sources needs to be aligned or transformed together. Two common scenarios are “batch to batch” and “batch to one.”
In our latest series of MapForce demo videos, we explore these common data mapping challenges.
BATCH TO BATCH DATA MAPPING
Batch to batch data mapping is helpful in scenarios where you have data updates or changes coming in batches, and you need to synchronize or transform these batches together. This could be to convert them to a different format, perform some type of sort or calculation, or a combination of these.
In this demo, we create a data mapping project that reads files from a directory and uses wildcards to set up a mapping that will process data from multiple files at once. Then, we explore another option for defining batch conversion using dynamic file names supplied by the mapping. This demo also shows how to add calculations and comments to your data mapping projects.
While this video highlights a batch to batch transformation of JSON files to XML files, MapForce supports conversion and transformation for any combination of XML, JSON, PDF, database, text, Protobuf, Excel, XBRL, and so on, for advanced data integration and ETL processes.
BATCH TO ONE DATA TRANSFORMATION
Batch to one data transformation is another common requirement, for example, when you want to merge or combine multiple files into a single consolidated document and perform some data transformation, conversion, or calculation operations in between.
This example also explores multiple approaches to defining the batch process, which will be applicable in different scenarios:
Specifying a collection of files in the same directory using a wildcard
Selecting batch files based on a list of file names stored in a different directory
This demo also shows how to sort the data merged from multiple files before writing it to the target.
After watching these quick demos, you can download a free trial of MapForce to try batch mapping, conversion, and transformation for yourself.
MORE MAPFORCE DEMOS
If you liked these videos, check out the rest of the MapForce demo series, which covers everything from mapping XML and JSON to databases to configuring data processing functions and extracting data from PDF documents.
MapForce, Altova’s award-winning data mapping tool, includes support for PDF input in data integration and ETL workflows. The MapForce PDF Extractor makes it easy to define rules for extracting PDF data in a structured format to make it available for mapping to other popular formats like Excel, XML, JSON, databases, and more.
Version 2024 of Altova Software introduces brand new AI Assistants in multiple products as well as long-awaited support for PDF data integration in MapForce. Other features include Markdown editing support, split output preview for business report creation, support for new XBRL standards, and much more.
One of the most common examples of AI in our everyday lives is facial recognition. Facial recognition is the process of identifying or verifying a person’s identity based on their face. Facial recognition is used in many applications, such as unlocking our phones with FaceID, tagging our friends on social media platforms like Facebook, and checking in at airports or hotels with biometric scanners. Facial recognition can make our lives more convenient and secure, but it can also raise some privacy and ethical concerns. For instance, how can we ensure that our facial data is not misused or stolen by hackers or malicious actors? How can we prevent facial recognition from being used for surveillance or discrimination? How can we ensure that facial recognition is accurate and fair, and does not have any biases or errors?
The paragraph above was generated by ChatGPT in response to my request to describe the benefits and risks of artificial intelligence and include a real-life example. It’s interesting that ChatGPT chose FaceID as the example, since FaceID is simply one variation of image analysis and AI-powered image classification offers potential to automate many real-world tasks.
One common use-case is a product catalog, wherein a company manages product information provided by many different manufacturers. A product loaded into that database may have a name that does not necessarily include a precise description of the item. For instance, wellington is a boot, fedora is a hat, a mongoose is a bicycle, and a yellow watermelon shiny needlefish is a fishing lure. We can make use of AI-powered image classification using the Microsoft Azure Cognitive Services Computer Vision API to address this problem. The Computer Vision Service takes the image data or URL as its input and returns information about the content. One service generates image classification tags based on a training set of recognizable objects, living beings, scenery, and actions that the Azure AI has been trained on. These tags allow us to categorize products in the database accordingly and may even correspond to search terms a user might provide to find products in the catalog.
Automated sentiment analysis of text, such as user reviews, has historically been a challenge. Because of the myriad intricacies of natural language, systems faced difficulties in analyzing context and nuances. This required an inordinate amount of manual work to overcome.
One of the many useful capabilities of modern AI systems that are based on large language models (LLMs) such as OpenAI’s GPT-4 is that they are very good at sentiment analysis of natural text inputs. We can use that capability to build a very efficient database solution in MapForce that, for example, goes through all the new incoming records in a support database and automatically determines whether a particular support request or other customer feedback is positive, negative, constitutes a bug report, or should be considered as a feature request.
Data mapping plays a vital role in modern data-driven organizations, enabling efficient data management and integration. Altova MapForce is a powerful, graphical data mapping tool that supports endless data transformation scenarios, including one-to-one, one-to-many, many-to-one, and chained data conversion.
While there are applications for each of these approaches, chained data mapping is especially helpful for complex data processing tasks where multiple stages of data manipulation are required. Here’s a look at the benefits of a chained data conversion approach – and a video of how MapForce makes the process easy and straightforward.
Altova MapForce offers a powerful, yet easy-to-use, approach to converting and transforming data. Whether you need to write XML to a database, convert JSON to EDI, or map Excel to multiple different data formats, MapForce has you covered.
From simple one-to-one conversions to complex ETL scenarios, the MapForce approach is to represent data structures as graphical components. To associate fields, drag and drop connecting lines. A comprehensive library of data filters and functions is available for transforming data before writing it to the target.
We have recently revamped our series of Introduction to MapForce videos. Each short how-to gives a demo of a common MapForce scenario.
Start at the beginning to learn how MapForce works:
You can follow along with the examples in these how-to videos by downloading a free, 30-day trial of MapForce. Check back for new MapForce videos, which are added often.
EDI (electronic data interchange) messages are used in a variety of industries for transmitting business information such as invoices, shipping notices, and purchase orders—which were traditionally recorded on paper—electronically. EDI transactions are structured according to standards that describe the format of each message. Adherence to a standard format enables the messages to be transmitted electronically between the computer systems of trading partners without human interaction.
Different industries take advantage of specialized EDI formats that are relevant to their business, and one such popular format is VDA EDI, which is used by the German automotive industry.
To work with VDA messages efficiently, companies often need to transform them to other formats, for instance, for storage in a backend database, or convert them to other EDI message formats for compatibility among systems. Altova MapForce makes this easy, with support for VDA as well as several other popular EDI standards. Let’s see how it works.
The latest release of Altova software includes ongoing enhancements for graphical XML and JSON editing in XMLSpy Grid View, new tools for working with XPath/XQuery, support for integrating VDA EDI data in MapForce, and much more.
Here’s a quick overview of all the new functionality being introduced.
Software developers and other data professionals often need to transform data from one format to another. These transformations can be simple one-to-one conversions or may require more complex manipulation. For instance, relationships must be generated when importing flat CSV files into a database, or source data may need to be split for the target, as in full name vs. first, middle, last, and optional suffix. Validating data transformation is critical to prevent data loss or corruption.
In an earlier post on Web service data integration, we combined a string value for GMT time with a numeric offset in seconds to generate the local time for weather forecasts. We created a user function that performed all the steps required to complete this operation. MapForce includes a powerful interactive data mapping debugger that can easily trace and validate this transformation. Let’s take a look at how it works.
Long-time XMLSpy and MapForce customers may remember the fun drawings and depictions from Altova’s early ads and logos. Those dark spy images have given way to brighter colors and imagery over the years – but at the same time, developers have gravitated towards dark mode in their applications. Now, dark mode is finally available for XMLSpy and MapForce!
This fun option is being released alongside important support for additional standards and databases, a new tool for managing schemas across the product line, and a brand new product for building ESEF XBRL reports in Excel.
Electronic Data Interchange (EDI) has proven to be a durable business-to-business communication technology in use today with history dating to the 1960s and even earlier. Efficient EDI data encoding reduced transaction payload size and improved data transfer speeds at a time when messages were sent over teletype at speeds equivalent to a 300 baud modem. EDI standards bodies promote wide acceptance among enterprises, and systematic evolution of new EDI formats extends support across industries. EDI improves profitability and is a dominant format for e-commerce data exchange.
Despite the advantages, EDI files are barely human readable and need to be translated and mapped for compatibility with other business technologies. Altova MapForce is a graphical EDI mapping tool with native support for all major business data formats in use today, including XML, JSON, databases, flat files, Excel, and Web services, as well as the EDIFACT, X12, HL7, NCPDP SCRIPT, IDoc, PADIS, and SWIFT EDI transaction sets. MapForce can even automatically convert EDI to XML without the need to specify a target XML Schema or perform any manual mapping.
CSV files are a quick and convenient way to record structured data in a generic format. Because CSV files are so easy to create, multiple similar versions of very large CSV files can quickly proliferate. Often it becomes necessary to compare CSV files to find the desired version. In an ETL scenario, a data analyst may want to compare a CSV file to a database table for validation or to update data.
DiffDog, the unique XML-aware diff / merge tool from Altova, supports CSV as a native file format for comparison and can compare and selectively merge data CSV to CSV, or between a CSV file and database table. Let’s look at an example.
MapForce supports mapping protocol buffers (Protobuf) to and from other structured data formats as mapping sources or targets. In the constant quest for more efficient ways to transfer, manipulate, and manage large structured data sets, Google has created a language- and platform-neutral data format similar to XML, but smaller, faster, and simpler than even JSON data. Tools are available to generate and work with Protobuf using Java, Python, C++, C#, Ruby, and other programming languages.
The structure of any Protobuf message is defined in a .proto file that defines each field name and value type. Altova MapForce lets users drop these .proto files into a data mapping as a source or target along with any other data, including XML, JSON, relational databases, Excel, flat files, REST and SOAP web services, and others. .proto files versions 2 and 3 are supported.
A MapForce data mapping creates compatibility between existing XML, JSON, database or legacy data formats and new applications leveraging the efficiency of Protobuf.
Excel began life as a simple spreadsheet tool. Over time, support for rich text styling options, built-in charts, and copy and paste formatting features has led many enterprises to create reports in Excel documents. This can cause difficulty when data changes and existing documents need to be manually updated for distribution to a wide audience in the familiar report style.
Altova MapForce, the award-winning, graphical data mapping tool for any-to-any conversion and integration, supports Excel data mapping to convert data to existing Excel documents while preserving styles and formulas in the original.
This feature lets you write directly to nicely formatted Excel files to update data at runtime: any designated worksheets, rows, and cells from the specified file will be replaced with data from the mapping and all formatting in the existing file will be preserved as-is. To protect functionality in the existing spreadsheet, cells with formulas are not overwritten.
Let’s look at an example of how to map Excel data.
Web service data integration with MapForce is a popular and proven strategy to capture timely information for analysis or generation of user-friendly reports. In an earlier post we demonstrated API data mapping in 5-day weather forecasts for busy cargo shipping ports by reading Web service data in JSON format and mapping to richly formatted Excel spreadsheets. The weather API we used is hosted by OpenWeather, a provider of historical, current, and weather forecast data.
But integrating data from any API is not a set-it-and-forget-it task. When you build a solution based on external data, you have to react quickly when the data structure changes. Since our original integration project OpenWeather revised the data delivered by their API. The API now includes wind gust predictions in a JSON property separate from wind speed. Since wind gusts are suspected as a cause of the recent Suez Canal blockage the new data is very relevant to our application! Fortunately, both the MapForce data mapping and the Excel spreadsheet are easily revised to add new data.
Altova FlowForce Server is a high-performance workflow engine for automating enterprise-level data processing, data integration, and ETL jobs.
FlowForce Server includes a helpful web interface for managing and monitoring all aspects of data processing jobs, including in-depth logging functionality and a complete visual dashboard. Let’s take a look at how you can take advantage of FlowForce charts and statistics to monitor the progress of FlowForce Server jobs, as well as performance of the server itself, in great detail.
XSLT3 adds trigonometry and other advanced math functions, new formatting functions, functions to collect environment variables, and more, extending XSLT and XSLT2 XML transformation standards. Data analysts and other data professionals can apply XSLT3 functions to solve XML data mapping and integration challenges that require complex mathematical computations. Let’s look at some MapForce examples of data mapping with XSLT3 math functions using trigonometry and other complex math expressions.
Binary objects – BLOBs — can be cumbersome to manage in databases. In an earlier post we described a MapForce data mapping to insert binary objects into a database with generated metadata to identify the BLOBs later. The companion challenge in data mapping binary objects is to extract binary data and save it in a comprehensible form faithful to the original.
Binary objects are difficult to manage in databases. They are large, their content is not human readable, and they can contain bytes of data easily misinterpreted as control characters. Even the data type name for binary large objects – BLOB – reflects most database managers’ dislike of them. Before relational databases, the definition of a blob was “something undefined or amorphous.”
Altova MapForce, the award-winning, graphical data mapping tool for any-to-any conversion and integration, includes features for effortlessly data mapping binary objects to or from all popular relational databases. Data such as images, PDF files, video files, or any other binary data can be mapped. Let’s look at an example.
In a previous post we wrote that every data integration and reporting task needs to start with a clear understanding of the source data. Using grid view in XMLSpy, the industry-leading XML and JSON editor, we analyzed JSON data for 5-day weather forecasts retrieved from a Web service.
Continuing with our earlier scenario, we’ll use MapForce, the award-winning, graphical data mapping tool for any-to-any conversion and integration, to map the forecasts for a series of major cargo shipping ports into nicely formatted Excel documents. We’ll want to highlight any predicted high winds or heavy rainfall that could cause delays by interfering with cranes loading and unloading containers, or slowing ships entering and exiting the harbors.
Altova MissionKit tools offer numerous ways to connect to, query, and integrate data from disparate sources. With multiple product releases each year, we’re constantly working to deliver increased power and efficiency for data integration, while adding features requested by customers. This includes ongoing updates to built-in support for all major SQL databases across the product line.
Let’s take a look at some of the recently added tools and
enhancements.
It’s a common requirement to convert non-standard or legacy text files to or from structured data formats like XML, JSON, and relational databases. However, many times legacy text files are not in a format that can be readily processed by data mapping tools, especially when they have a complex and unique structure that does not consistently fit into CSV or fixed-length field patterns. Moreover, sometimes you need to extract only portions of useful data from a legacy text file.
MapForce, Altova’s any-to-any data conversion tool, includes a unique utility called FlexText that makes it easy to visually define templates for parsing text files and making them accessible to the data mapping tool.
See how FlexText works in our video tutorial.
The example files referenced in the video are available here and you can try FlexText with a free, 30-day trial of MapForce.
The JSON data format continues to evolve as an open standard as it is creatively applied to new data interchange requirements. JSON Lines, defined at http://jsonlines.org/, is a convenient text format for storing structured data where each record is a single line and a valid JSON object. JSON Lines handles tabular data and clearly identifies data types without ambiguity. This allows records to be processed one at a time, which makes the format very useful for exporting and sending data.
Altova MapForce supports data mapping JSON Lines as either a data source or target. Let’s look at a mapping project to extract records from a database table and map to a JSON Lines file for output.
A recent mandate from the OECD called on large, multinational companies to report financials annually for each country in which they do business to their local tax authority. The OECD specified that this detailed Country by Country (CbC) Report be filed in an XML document according to their reporting schema. But for tax departments that work largely in Excel or other accounting software, this presented a significant stumbling block – and companies found themselves scrambling to meet the requirements by the deadline.
What was needed is a way to automatically generate valid, properly formatted CbC XML reports based on existing data. Altova created the Country by Country Reporting Solution to do just that, either based on manually entered data or figures imported directly from Excel. Let’s take a look at how it works.
Data mapping projects often mirror software development
efforts with distinct phases for design, testing, and deployment. This is
especially true for ETL (Extract Transform Load) projects when repeated data
mapping execution is required as new data becomes available, and the stakes
increase higher with large data sets. The Altova MissionKit and Server Software
products provide Global Resources to define configurations for each project
phase and smoothly transition between them.
Let’s take a look at an example based on a MapForce data mapping from a source file to a database.
Data analysts and other professionals often need to generate
real-time data through automated execution of data mappings that request Web
services and save the results. During automated execution it’s important to
gracefully handle any unexpected HTTP error rather than terminate the
integration task.
In an earlier post we discussed conditional processing of a REST Web service response to handle HTTP errors, where separate output files were generated for a normal response and an error. Now let’s look at a revised mapping solution for the airport status example to generate a single mapping result file that contains either the requested airport status or a description of the error.
Data integration projects that include information from
external Web services may be vulnerable to HTTP errors when retrieving remote
data. When data mappings run under automated control it’s especially important
to detect and report errors even if errors only occur very rarely.
A MapForce data mapping can include Web service calls and output
the result directly to a file or database, or combine it with other inputs for
further processing. Regardless of the final output, an HTTP Web service error
encountered in a REST Web service request puts the mapping at risk.
MapForce includes features for handling HTTP errors instead
of simply aborting execution of a mapping. Developers can configure the body of
a REST Web service call to handle and report exceptions based on the HTTP
status code returned.
Critical business processes depend on reliable data and database administrators and other data analysts want to be confident in the integrity of information stored in database tables. During automated ETL (Extract Transform Load) operations or other database import tasks, invalid data might be encountered that jeopardizes success of the procedure. Altova MapForce includes database exception handling to roll back the affected data when an error occurs and optionally proceed with the rest of a database mapping.
For instance, an error in a single record need not prevent execution of a mapping from continuing, such as when certain database constraints prevent the mapping from inserting or updating invalid data.
Database administrators and other data professionals often
want to maintain a record of changes in critical databases, especially when updates
are made by automated scripts or other operations. Database tracing lets
administrators track critical changes or anomalies, and help recover from
errors. Altova MapForce supports database tracing for all popular relational
databases to log the changes made by a data mapping project to the
database when the mapping runs.
When tracing is enabled, events such as database insert or
update actions, or errors, are logged in an XML file that you can later analyze or process further in an automated way.
Database tracing can be enabled at the database component, table, stored procedure, or database field level. You can choose to trace all messages or only errors, or you can disable tracing completely.
In addition to tracing errors that occur during the execution of a mapping to a target database, MapForce also enables database transaction handling to roll back the affected part of the database data when an error occurs, then optionally proceed with the rest of the mapping.
From updated support for standards and relational databases
to several new data mapping options, Altova’s Version 2019 Release 3 product
line addresses developer requests and ups the ante with innovative new
functionality. Let’s take a look at the highlights.
FlowForce Server is Altova’s high-performance engine for automating workflows of XML processing, data integration, report generation, and more. It integrates with other Altova server software products to automate their functions, such as executing complex data integration processes, including ETL projects, designed in MapForce; running StyleVision report generation jobs; or validating XML, XBRL, or JSON files with RaptorXML Server.
Starting with Version 2019, FlowForce Server offers new options for distributed execution and load balancing to improve availability and performance. Let’s take a look at how configuring multiple FlowForce Servers to run as a cluster can help improve data throughput and provide redundancy.
MapForce node functions simplify mapping hierarchical data such as XML nodes or CSV, JSON, EDI, or database fields by permitting users to define a data processing function at the node level and apply it recursively to all descendant items.
Similarly, default values can also be assigned to nodes and automatically applied to descendants.
Defaults and node functions are particularly useful when a data mapping and transformation task requires the same processing logic for multiple descendant items in a structure, for example:
Replace null values with some other value, recursively for all descendant items
Replace a specific value (for example, “N/A”) with some other value recursively for all descendant items
Replace all database null values when reading from a database table
Trim all trailing spaces for all values from a source database
Append a custom prefix or suffix to all values written to a target file or database
Formatting of output values
And many others
Defaults and node functions simplify mapping hierarchical data by eliminating need to copy-paste the same function multiple times into a mapping. Repeating the same function unnecessarily clutters the mapping layout and makes it more difficult to understand or revise.
EDI (Electronic Data Interchange) standards allow participants with different roles in an industry to communicate clearly and rapidly, and date back to the earliest implementations of electronic communication in the 1950s, long before modern business technologies such as ERP, CRM, and many others. Yet even today, EDI standards continue to evolve to support new requirements and opportunities.
SCRIPT is the state of the art EDI standard developed by the National Council for Prescription Drug Programs (NCPDP) for electronically transmitting medical prescriptions, also known as ePrescribing (eRX) in the United States.
MapForce Server automates recurring execution of data mappings and transformations designed and tested using Altova MapForce. Every day, MapForce Server is employed in business communication, financial reporting, database ETL, and many other applications to transform critical data between any of XML, JSON, database, EDI, XBRL, flat file, CSV, Excel, and/or Web service formats.
Now, MapForce Server Accelerator Edition offers even faster throughput for high-performance server platforms.
Join is a powerful SQL operation implemented across most database types and familiar to database users. Join is typically used to select and combine information from multiple database tables.
Altova MapForce includes a join component for data mapping that works like a SQL join for database tables and extends data integration functionality by empowering users to join data trees of any data format. Anyone familiar with join operations for database tables will find the MapForce join component especially intuitive. A join operation in MapForce can even combine two different data formats and produce output in a new format altogether.
Envision a manufacturing company that controls costs by exploiting a just-in-time assembly process with a very low supply of parts inventory on hand. New customer orders are logged in a sales database and at the end of every day the components needed to assemble that day’s sales are tabulated.
The IT department runs a SQL query to identify the required parts and transforms the list into a purchase order in JSON format to be transmitted to the supply chain.
Sound familiar? Our recent blog series on JSON tools and JSON data mapping were based on this real-life scenario. In this post we describe a MapForce Server use case that automates the repetitive task of generating each day’s purchase order.
JSON is a popular format for transferring data between systems thanks to its simple markup, small footprint, and heritage based on the JavaScript programming language. MapForce supports JSON as both an input and output format for JSON data mapping and transformation. For instance, MapForce can extract information from any popular database and produce a JSON file ready for transfer. The Requirement: Here is an example of a typical need for JSON data mapping: A manufacturing company controls costs by exploiting a just-in-time assembly process with very little parts inventory on hand. New customer orders are logged in a sales database, and at the end of every day the components needed to assemble that day’s sales are tabulated via a query into the database. The required parts will be ordered from suppliers via a purchase order transferred in JSON format.
Any computer industry standard that promotes reliability and clear communication between independent enterprises will have a long life. EDI (Electronic Data Interchange) originated in the 1960s to enable automated transactions between corporate computer systems. EDI remains in widespread use today and continues to evolve to meet modern requirements, under the administration of the UN/EDIFACT and ANSI standards bodies.
Altova MapForce supports EDI data mapping between EDI messages and XML, JSON, relational databases, flat files, Excel, or other data formats to bridge between commonly used information interchange and in-house technologies.
MapForce includes support for the latest EDIFACT versions 2015B and 2016A including the new VERMAS message. Mapping and translating EDIFACT messages to other usable data types for transfer, storage, and management is a common business requirement solved by MapForce.
Large database tables can easily contain a million, even hundreds of millions of rows of data. Database administrators and others charged with maintaining such large datasets are always concerned about execution time for ETL (Extract, Transform, and Load) operations, updates, and other SQL queries. To make these operations more efficient, some — but not all — database vendors implemented a SQL merge statement to insert or update rows of an existing table as a single bulk-insert statement rather than requiring individual statements for each row.
Altova MapForce automatically supports SQL merge when it is available for the target database. Let’s look at an example.
There are situations, especially when encountering loosely structured data, where you may want to map and transform structural components of a data stream along with content. MapForce can dynamically access node names of XML elements, attributes, or text file columns such as the contents of CSV files, to target components.
Dynamic access to node names allows creation on the fly of target elements and attributes whose names do not need to be known beforehand or specifically identified in the data mapping. This feature lets you create much more generic, flexible, and reusable mappings that require less manual intervention if data models evolve.
Electronic Data Interchange (EDI) standards accelerate commerce worldwide by allowing companies and other organizations – even those in different regions, using different languages and currencies – to send and receive unambiguous information. Some EDI communication protocols and message formats still in wide use today were developed over 30 years ago, when telecommunication systems were slower and data storage was more expensive.
The usefulness of the XBRL standard reaches far beyond SEC requirements for filing financial statements. Organizations such as XBRL.US and XBRL International are working to develop XBRL taxonomies and accompanying software solutions for countless other practical applications where standardizing data submission results in increased accuracy and productivity for all involved – for report filing, data analysis, and beyond.
One such project is the Work in Progress (WIP) Taxonomy created by XBRL.US for the surety industry. The new taxonomy helps save time and increase accuracy for report submission, and at the same time enables new opportunities for data analysis and decision making.
MapForce 2016 Release 2 includes expanded functionality for Web Services data mapping, providing robust support for REST Web services. MapForce accepts XML or JSON as the Web service response, allows definition of parameters, and supports custom HTTP headers. Users may define the Web service interface manually or by importing settings from a WADL file or a URL. Manual definition of REST Web Service Settings lets developers create settings based on a template URL. This is a convenient step when developers test and refine REST calls in a Web browser window, since the URL can be copied from the browser to become the template.
Google recently announced their next generation of managed MySQL offerings on Cloud SQL, so we wanted to take it for a spin and create a cloud-based SQL database that we could then utilize as the back-end for mobile apps, or even for advanced data analytics from our desktop.
According to Google, the two principal goals of the second generation of Cloud SQL were better performance and scalability per dollar. It seems that they succeeded in these goals: the second generation Cloud SQL is more than seven times faster than the first. And it scales to 10TB of data, 15,000 IOPS, and 104GB of RAM per instance — well beyond the first generation. So it looks like the ideal, scalable cloud-based database back-end for mobile apps.
Altova MapForce includes powerful mapping components that correspond to design patterns for data transformation requirements. Analyzing a data mapping challenge up front and following a few straightforward guidelines can uncover data mapping patterns that help simplify creation of the mapping design and lead to an optimal solution. The MapForce Examples project provides sample mapping files and data sets that illustrate many common data mapping patterns. Reviewing these examples and executing them with the MapForce Built-in Execution Engine is another good way to help select the best pattern for your own project.
MapForce 2016 introduces a revolutionary data mapping debugger that lets developers working on data integration projects examine data mapping output step by step to diagnose and perfect projects of any complexity. The MapForce data mapping debugger gives users deep insight into the exact inner workings of data integration and ETL projects in a way that was never before possible.
The MapForce data mapping debugger supports breakpoints and conditional breakpoints, and includes multiple manual stepping options to manually debug a data mapping or continue execution after a breakpoint is reached, allowing users to see as much detail as they need. Read more…
