Handling HTTP Errors in Web Service Data Mappings

Data integration projects that include information from external Web services may be vulnerable to HTTP errors when retrieving remote data. When data mappings run under automated control it’s especially important to detect and report errors even if errors only occur very rarely.

A MapForce data mapping can include Web service calls and output the result directly to a file or database, or combine it with other inputs for further processing. Regardless of the final output, an HTTP Web service error encountered in a REST Web service request puts the mapping at risk.

MapForce includes features for handling HTTP errors instead of simply aborting execution of a mapping. Developers can configure the body of a REST Web service call to handle and report exceptions based on the HTTP status code returned.

Let’s look at an example.

The sample data mappings installed with MapForce include a real-world REST Web service mapping called AirportStatus.mfd. This mapping calls a Web service sponsored by the Federal Aviation Administration that reports information about operation delays at major US airports. The mapping input is a three-letter airport code which is provided to the Web service, and the result is returned as a JSON file. The mapping and example result are shown below.

In the case of any HTTP error, the mapping execution would fail.

The MapForce Help file includes an explanation of conditional handling of a REST Web service response. We could simply paste the portion for handling HTTP errors onto the existing mapping, but we would have no way to validate the result in case of an HTTP error. We can’t force the FAA Web service to generate an HTTP error for testing!

Instead, we can use the Web service at httpstat.us to generate various HTTP status codes and descriptions for testing in a separate data mapping. For instance, the URL https://httpstat.us/200 in a browser window returns “200 OK.” The URL https://httpstat.us/400 returns “400 Bad Request,” and so on for all defined status codes up to 524.

We can implement a REST request for this service in MapForce by simply opening a new mapping file and clicking the Insert Web Service button in the toolbar:

This opens the Web Service Call Settings dialog where we can manually configure the settings. We will use a parameter in place of any specific status code so we can easily test alternate values.

Here is the mapping with a constant to supply the code parameter:

By default, a Web service function only supports normal HTTP response values in the range of 200-299. We can add support for other codes by right-clicking the Body element in the Response to open the Response Structure dialog and extend the range of accepted status codes:

We can map the result of the REST request to a simple text file, using the decode-mime-entity function. The Web service does not supply the character set value in the response header, so we’ll connect the constant value utf-8 to the decoding function.

Now we can use the Output button at the bottom of the MapForce mapping pane to execute the mapping and examine the result:

 We can change the input constant in the mapping pane and execute again to see the difference:

We can use a concat function to combine the result with other information to create a more user-friendly status message:

Once we test the mapping with a variety of possible HTTP status codes and we’re satisfied with the results, we can add HTTP error handling to the existing AirportStatus.mfd mapping via a copy-paste operation. First will create a new response body in the AirportStatus mapping to handle response codes above the normal range. We can right-click the Body item in the response side of the Web service to add a new Body node.

This adds the new body and opens the Response Structure dialog, where we will specify processing for any HTTP Status greater than 299:

Now we can copy the mapping components of our error-handling definition from the test mapping, paste into the AirportStatus mapping, and connect the new Response body:

The mapping is now defined with multiple outputs. During manual execution directly in MapForce, the user would choose one output or the other. The value of the execution path for handling HTTP errors becomes clear when the mapping is intended for execution under automated control by MapForce Server or in a regularly scheduled FlowForce Server job. For instance, an automated data mapping could report the status of 50 different airports every half hour.

During automated execution all data mapping output paths are processed. To ensure the web service is called only once for each input, an intermediate component would catch the Web service result before mapping to the final target. If any HTTP error occurs, the date, time, and the exact HTTP Status code will be recorded.

Handling HTTP errors in a REST Web Services data mapping is like a seat belt or a fire extinguisher – something very good to have but you hope you won’t need to use it!

If you’re new to MapForce, check out this video overview for an introduction. You can try MapForce yourself with a free, 30-day trial.