Handle HTTP Errors During Automated Data Integration

Data analysts and other professionals often need to generate real-time data through automated execution of data mappings that request Web services and save the results. During automated execution it’s important to gracefully handle any unexpected HTTP error rather than terminate the integration task.

In an earlier post we discussed conditional processing of a REST Web service response to handle HTTP errors, where separate output files were generated for a normal response and an error. Now let’s look at a revised mapping solution for the airport status example to generate a single mapping result file that contains either the requested airport status or a description of the error.

One strategy to generate a single result would be to treat each output file created in our earlier MapForce mapping as an intermediate component, then map them into a final result that accommodates both possible outcomes.

However, a simpler solution is to directly map a normal response and an error result into one output component.

After publishing our earlier post, we discovered the FAA airport status Web service returns an error if the request has an empty field for the airport code. This generates an HTTP 404 error. Unfortunately, the Web service result for the error is delivered in HTML format instead of JSON, despite the specification in the request header:

Specifying Web service GET parameters in the data mapping.

We can avoid parsing an HTML error by mapping directly from the Status code returned in the response:

Handle HTTP errors by mapping the status code from the response

First, we need to add an item to the JSON schema for the target component to hold a message in case of any error. This is easily accomplished using the XMLSpy Graphical JSON Schema Editor:

In the Details helper window Occurrence is set to Optional because we will only generate this item in case of an HTTP error.

Now we can build a custom error message based on any HTTP status greater than 200:

Handle HTTP errors by mapping a custom error message into the result

The concat function above includes the Status code and the airport code to create a full explanation of any error.

Here is a complete view of the revised mapping tracing the path of the Status code to the output component:

Handle http errors with alternate paths for normal and error results

We can edit the airport code constant at the top left of the mapping, then click the Output button below the main mapping window to test various input values. When a valid airport code is included in the GET request to the Web service, the result is the JSON file we expect:

Handle http errors -- viewing a normal result of the mapping

We can generate an HTTP error by sending an empty string in the constant for the airport code. This generates an error response file in JSON format:

Handle http errors with a message in the result

Note that our mapping still includes separate response bodies for HTTP status codes 200-299 and for codes 300-599. This ensures that only valid JSON results are mapped for output. Error results received in HTML format are not mapped. They are simply discarded.

Handle http errors -- example of an error result provided by the Web service in HTML format

Automate the Data Mapping and Handle HTTP Errors

In our earlier post we described an automation scenario where we want to check and record the status of 50 airports on a regular schedule. To accomplish this we will need to call the Web service and map the result for each individual airport code.

The original example mapping is already designed with automation in mind. MapForce Server is a cross-platform server tool that automates execution of data mappings designed in MapForce. The component at the top left of the mapping following the string constant is an input parameter that can be provided at run-time for execution by MapForce Server.

Input parameter component for automated mapping execution

The string ‘PHX’ in the screen shot is only used during direct execution in MapForce.

FlowForce Server is a highly-customizable workflow engine for efficient automation of enterprise-level data integration tasks. A FlowForce Server job can execute the mapping using MapForce Server once for each required airport code and repeat on a regular schedule. The automated job will handle HTTP errors by producing result files that contain either the latest status or a description of the error for each airport code.

If you’re new to MapForce, check out this video overview for an introduction to its any-to-any data conversion and integration features. You can try MapForce, MapForce Server, and FlowForce Server for yourself with a free, 30-day trial.