Web services Archives

Tag Archive for: Web services

API Data Mapping

April 30, 2021/in Data Integration, Tools /by David McGahey

Web service data integration with MapForce is a popular and proven strategy to capture timely information for analysis or generation of user-friendly reports. In an earlier post we demonstrated API data mapping in 5-day weather forecasts for busy cargo shipping ports by reading Web service data in JSON format and mapping to richly formatted Excel spreadsheets. The weather API we used is hosted by OpenWeather, a provider of historical, current, and weather forecast data.

But integrating data from any API is not a set-it-and-forget-it task. When you build a solution based on external data, you have to react quickly when the data structure changes. Since our original integration project OpenWeather revised the data delivered by their API. The API now includes wind gust predictions in a JSON property separate from wind speed. Since wind gusts are suspected as a cause of the recent Suez Canal blockage the new data is very relevant to our application! Fortunately, both the MapForce data mapping and the Excel spreadsheet are easily revised to add new data.

An Easy Way to Test HTTP Requests During Development

January 20, 2021/in Development, JSON, XML /by Erin Cavanaugh

Web and web services developers often need to send HTTP requests – whether for testing APIs, testing REST and SOAP web services, or managing web sites.

XMLSpy makes it easy to send and receive HTTP requests directly in the XML and JSON editor during development with its HTTP Window and WADL/WSDL Import Wizard, a great time-saving tool for debugging web services.

Web Service Data Integration

May 11, 2020/in Data Integration, JSON, Tools /by David McGahey

In a previous post we wrote that every data integration and reporting task needs to start with a clear understanding of the source data. Using grid view in XMLSpy, the industry-leading XML and JSON editor, we analyzed JSON data for 5-day weather forecasts retrieved from a Web service.

Continuing with our earlier scenario, we’ll use MapForce, the award-winning, graphical data mapping tool for any-to-any conversion and integration, to map the forecasts for a series of major cargo shipping ports into nicely formatted Excel documents. We’ll want to highlight any predicted high winds or heavy rainfall that could cause delays by interfering with cranes loading and unloading containers, or slowing ships entering and exiting the harbors.

Handle HTTP Errors During Automated Data Integration

July 29, 2019/in Data Integration, JSON, Tools /by David McGahey

Data analysts and other professionals often need to generate real-time data through automated execution of data mappings that request Web services and save the results. During automated execution it’s important to gracefully handle any unexpected HTTP error rather than terminate the integration task.

In an earlier post we discussed conditional processing of a REST Web service response to handle HTTP errors, where separate output files were generated for a normal response and an error. Now let’s look at a revised mapping solution for the airport status example to generate a single mapping result file that contains either the requested airport status or a description of the error.

Integrate Maps into Mobile Apps

July 17, 2019/in App Development, Mobile, mobile development /by David McGahey

Developers can create highly-customized location-based apps by leveraging geolocation functionality in mobile devices. Now MobileTogether, the cross-platform, low-code mobile development tool from Altova, supports deep integration of maps into applications for all popular mobile platforms.

Developers can integrate maps into mobile apps, add dedicated markers, and define custom actions based on user clicks on the map. For instance, an enterprise might want an app to include a map of all branch office locations, then display the current inventory when an office is clicked.

In a previous post we described a mobile app designed to let users check the status of major US airports by selecting an airport code from a combo box. An alternate technique would be to replace the 47 combo-box entries with pins on a map.

Let’s look at this example.

Web Service Error Handling in Mobile Apps

July 10, 2019/in Development, mobile development, Software /by David McGahey

Mobile phones bring a world of information to our fingertips, but functionality of even the best-designed mobile apps can be impacted by Web service errors that occur when communicating with external servers. Intermittent cell phone service in remote locations can also degrade app performance when looking up data.

MobileTogether, the low-code, cross-platform mobile app development tool from Altova, includes features that let developers gracefully handle Web service errors in mobile apps to avoid burdening end-users with unexpected app interruptions or cryptic error messages.

In an earlier post we explained HTTP error handling in a MapForce data mapping. Now we’ll look at the same Web service in a mobile app and describe error handling in MobileTogether.

Handling HTTP Errors in Web Service Data Mappings

June 19, 2019/in Data Integration, Tools /by David McGahey

Data integration projects that include information from external Web services may be vulnerable to HTTP errors when retrieving remote data. When data mappings run under automated control it’s especially important to detect and report errors even if errors only occur very rarely.

A MapForce data mapping can include Web service calls and output the result directly to a file or database, or combine it with other inputs for further processing. Regardless of the final output, an HTTP Web service error encountered in a REST Web service request puts the mapping at risk.

MapForce includes features for handling HTTP errors instead of simply aborting execution of a mapping. Developers can configure the body of a REST Web service call to handle and report exceptions based on the HTTP status code returned.

Let’s look at an example.

Web Service as a Look-Up Table to Refine GPS Data

June 20, 2012/1 Comment/in Data Integration /by David McGahey

Elevation data recorded by GPS devices is notoriously inaccurate, especially in hilly terrain like the Russian River Valley example from our earlier post.

The final elevation track plotted from the Russian River Valley GPX file is suspicious for several reasons. First, the graph shows we descended almost 50 feet below sea level. That’s hard to believe, since we were travelling along the bank of the river, only about 10 miles from the Pacific Ocean.

Altova StyleVision Chart Secondly, we were headed mostly west, following the river downstream, but the track shows a predominantly uphill trend.

We can evaluate the recorded GPS elevation data by comparing it to information available from the United States Geological Survey (USGS). The USGS operates a Web service that accepts latitude and longitude coordinates, and returns elevation data measured by NASA and assessed for accuracy based on over 13,000 control points in the continental United States.

Using the elevation Web service in an Altova MapForce mapping will let us extract each point from the GPX file, send the coordinates to the USGS Web service, and build a new GPX file with corrected elevation data.

Processing the Groupon API with Altova MapForce

August 2, 2011/2 Comments/in Data Integration, JSON /by David McGahey

We often think of a data integration project as a translation from one singular data input file to some other data set, but Altova MapForce lets you greatly expand the concept of an input file. For instance, the MergeMultipleFiles.mfd example installed with MapForce illustrates how you can use a filename with wildcard characters to merge multiple input files into a single output. A MapForce mapping input doesn’t even need to be a physical file – it can be a URL that returns predictable structured data, like the APIs for popular Web sites like Groupon and many others.

In this blog post we’ll describe how to use Altova tools to retrieve, filter, analyze, and present data available from a Web-based API, using Groupon as an example. If you want to follow along yourself, you will first need to visit http://www.groupon.com/pages/api to request your personal Groupon API client key. The Problem: All Deals Are Local The Groupon Web site and email subscriptions are great for finding deals in your local neighborhood, but what if you’re looking for a deal to use on an upcoming vacation, or for a gift for friends or family across the country? Sure, you could enter each location manually at the Groupon Web page, but that’s so last century. Let’s use the Altova MissionKit to automate things. The Groupon API offers two URL queries that return data in .json or .xml formats: the first returns a list of all Groupon localities (called divisions), and the second returns current deals information for one named division. If we want to see all the deals for more than one division, we need to resolve multiple URLs and aggregate the data into a single result. Yes, MapForce can do that! First We Need a Schema The Groupon API documentation describes the elements that will be returned by our requests, but doesn’t provide an XML Schema. That’s okay, we can use MapForce to generate one. All we have to do is open a new mapping design and choose Insert XML Schema/File, then click the Switch to URL button. Now we can enter the URL to retrieve the Groupon divisions list: When we click the Open button MapForce offers to generate the schema: When we click Yes, the File / Save dialog opens. I saved the schema as divisions.xsd, and the mapping with the new XML Schema inserted looks like this: And the Properties dialog for the XML Schema component automatically contains the API /divisions URL as the Input XML File: Check the Work We want to filter the Groupon divisions data to build a list of id names to use for deal queries for each locality. But before we go any further, now might be a good time to apply the text file trick from the Quick Solution for Complicated Functions blog post to look at the id values. When we insert the text file and connect the divisions and id schema elements, the mapping looks like this: We connected the division element to Rows in the text file in order to generate a new row in the text file for each unique division, so that Field1 in each row will hold the id. Clicking the Output button now generates this result: All we need to do is apply the concat string function to build the list of /deal URLs for all division IDs. The next step in the mapping looks like this: Rolling the cursor over the constant connected to value1 of the concat function displays its full definition: When we click the Output button to execute the mapping, the Output file now looks like this: As a further review, we can open the generated XML Schema in XMLSpy and display it in graphical Schema View: So far we have:

built a MapForce mapping that queries the Groupon API for all divisions
extracted the division id fields
and built a list of URLs for API queries to get the deals in each division

In the next post in this series we will process the list of deal queries as the input for a new mapping component and filter the output for some interesting information. Find out for yourself how easy it is to apply MapForce to convert data from a Web API! Download a free 30-day trial of MapForce.
Editor’s Note: Our original series on mapping data from the Groupon API ran in three parts you can see by clicking the links here: Part 1 of Processing the Groupon API with Altova MapForce describes how to create dynamic input by collecting data from multiple URLs. Processing the Groupon API with MapForce – Part 2 describes how we filtered data from the API and defined the output to extract only the most interesting details. Processing the Groupon API – Part 3 describes formatting the output as a single HTML document optimized for desktop and mobile devices, and reviews ways to automate repeat execution.

Tags: Altova XMLSpy, data integration, data mapping, MapForce, MissionKit, Web services, XML Schema

Case Study: Equifax

August 21, 2008/2 Comments/in /by Niki Devgood

Check out the case study below to learn how leading US credit reporting entity Equifax® built an advanced SOAP interface for their identity verification and authentication Web service.

Overview

Equifax is a leading credit reporting entity and provider of analytical and decision support tools. Their real-time authentication system, eIDverifier, offers government and businesses personalized online security measures that help protect them against fraud and comply with federal legislation. The eIDverifier process is used within e-commerce and other online applications to authenticate users’ identities based on their answers to personalized questions drawn from Equifax’s extensive data stores. The authentication process consists of five steps:

Integrity Check – eIDverifier standardizes and screens applicant-provided information to test for data inconsistencies and irregularities.
Pattern Recognition – A pattern recognition algorithm is conducted on each transaction. For example, a velocity parameter determines the number of times an applicant has applied for authentication in a specific time frame.
Identity Validation – To confirm an identity’s legitimacy, eIDverifier uses a “waterfall” approach in gathering validation information from multiple data sources. This means that if the identity cannot be validated with the first data source, eIDverifier will proceed to the next data source until the identity is validated.
Interactive Query – eIDverifier presents multiple-choice questions to the applicant based upon “shared secret” information that should only be known to the applicant and Equifax. The question sets are customizable to meet individual risk thresholds.
Decision Logic / Output Assessment – There are two output components to eIDverifier – an assessment score and reason codes. The assessment score indicates the likelihood of an applicant presenting fraudulent information, while reason codes provide important details on questionable information and highlight any discrepancies between the consumer’s application information and Equifax data sources.

eIDverifier relies on the SOAP protocol to send messages defining these interactions back and forth between the client interface and the Equifax servers. Third party institutions license the eIDverifier SOAP interface for use within their online application processes, enabling them to integrate its functionality and access information contained in Equifax’s databases.Equifax uses the XMLSpy XML Schema editor to graphically design the XSDs that serve as the foundation for their SOAP interface.

The Challenge

Equifax needed a sophisticated tool for designing the XML Schemas that would define the data types for their Web service, as well as a mechanism for creating the WSDL documents that would describe the interface as a whole. As a Java shop, Equifax needed a solution that would be compatible with their other development tools, and that would work seamlessly with the Eclipse IDE. Though there are plenty of Java tools available that have the capacity for XML Schema development, XMLSpy presented the most attractive option for schema design because of its comprehensive graphical design and editing options.The Equifax development team took a further step to simplify their Web services creation, using XML Beans and the Codehaus XFire/CXF Java SOAP framework to auto-generate WSDL from their XML Schemas.

The Solution

eIDverifier relies on a variety of different technologies to bring identity verification and authentication to its clients. XMLSpy provides the following benefits:XML Schema

XML Schema is used to express the structure of the data, as well as the individual elements and attributes that it is comprised of. Because a large portion of the data relies on end-user input in the form of address, phone number, driver’s license number, etc., it is vital that this information is in a format that can be digested by the system.Using XMLSpy’s graphical XML Schema editor, the Equifax development team was able to easily visualize and maintain the structure of their XML Schema. A portion of the schema that was created appears below:

SOAP interface

This data type definition provides the syntax, and dictates the structure, for the data that is transmitted by the eIDverifier Web service.

XMLSpy’s unique graphical XML Schema editor allowed the Equifax development team to create and maintain a complex schema definition without writing any code manually. They were also able to automatically generate human-readable documentation that can be used to present the architecture for review at any time in the development process, and that describes each element and attribute in detail.

WSDL

The processes executed by eIDverifier are described by a WSDL document that incorporates the XML Schema to provide information about data types, functions, and other interface details to the client – defining and dictating the actions taken by the client application to send and retrieve information between the end-user and the Equifax servers. The Equifax team chose to autogenerate a WSDL document using the Codehaus XFire/CXF framework. The XML Schema was used as the basis for an XMLBeans implementation, which was then compiled as a Java service class. Once the eIDverifier service was exposed, XFire automatically generated a WSDL – the WSDL is shown below in the XMLSpy graphical WSDL editor.

This WSDL serves as the basis for the eIDverifier application, defining the ports and messages that make up the communication infrastructure of the Web service.

The Results

The eIDverifier SOAP interface allows external applications to access Equifax’s backend data stores, exposing it as a Web service and enabling them to retrieve secure information without jeopardizing the integrity of the Equifax mainframe. Utilizing WSDL and SOAP, and surrounded by Java architecture, eIDverifier is able to confirm user identity by returning a set of multiple choice questions based on the secure data maintained by Equifax. SOAP interface XMLSpy enabled the Equifax team to quickly and easily create a graphical schema representation and the matching documentation to serve as the basis for the Web service. It also allowed the development team to focus on their Java code, rather than the intricacies of XML Schema and WSDL design. The Altova MissionKit provides numerous tools for advanced Web services development, from the graphical XML Schema and WSDL editing discussed here, to SOAP debugging, and even graphical Web services generation and data mapping. Download a free trial to check it out for yourself.

Tags: MissionKit, SOAP, Web services, WSDL, XML Schema, XMLSpy