Database mapping Archives

AI-based Database Image Classification with Altova MapForce

July 31, 2023/in Data Integration, Database mapping, Tools /by David McGahey

One of the most common examples of AI in our everyday lives is facial recognition. Facial recognition is the process of identifying or verifying a person’s identity based on their face. Facial recognition is used in many applications, such as unlocking our phones with FaceID, tagging our friends on social media platforms like Facebook, and checking in at airports or hotels with biometric scanners. Facial recognition can make our lives more convenient and secure, but it can also raise some privacy and ethical concerns. For instance, how can we ensure that our facial data is not misused or stolen by hackers or malicious actors? How can we prevent facial recognition from being used for surveillance or discrimination? How can we ensure that facial recognition is accurate and fair, and does not have any biases or errors?

The paragraph above was generated by ChatGPT in response to my request to describe the benefits and risks of artificial intelligence and include a real-life example. It’s interesting that ChatGPT chose FaceID as the example, since FaceID is simply one variation of image analysis and AI-powered image classification offers potential to automate many real-world tasks.

One common use-case is a product catalog, wherein a company manages product information provided by many different manufacturers. A product loaded into that database may have a name that does not necessarily include a precise description of the item. For instance, wellington is a boot, fedora is a hat, a mongoose is a bicycle, and a yellow watermelon shiny needlefish is a fishing lure. We can make use of AI-powered image classification using the Microsoft Azure Cognitive Services Computer Vision API to address this problem. The Computer Vision Service takes the image data or URL as its input and returns information about the content. One service generates image classification tags based on a training set of recognizable objects, living beings, scenery, and actions that the Azure AI has been trained on. These tags allow us to categorize products in the database accordingly and may even correspond to search terms a user might provide to find products in the catalog.

decorative image depicting an AI "brain"

Data Mapping NoSQL Databases

November 26, 2021/in Database, Database mapping, JSON /by David McGahey

NoSQL databases are non-tabular databases that store data differently than traditional databases made up of relational tables. Two of the most popular NoSQL databases, MongoDB and Apache CouchDB, store data as collections of BSON (binary JSON) and JSON documents. These databases leverage flexible JSON schemas and scale easily with large amounts of data and high user loads.

Altova MapForce has long supported data mapping all popular relational databases and now also includes native support for data mapping NoSQL databases. MapForce includes functionality for inserting, extracting, filtering, and ordering NoSQL data. Let’s look at an example.

Data Mapping Binary Objects – Part 2

October 30, 2020/in Data Integration, Database mapping, SQL, Tools /by David McGahey

Binary objects – BLOBs — can be cumbersome to manage in databases. In an earlier post we described a MapForce data mapping to insert binary objects into a database with generated metadata to identify the BLOBs later. The companion challenge in data mapping binary objects is to extract binary data and save it in a comprehensible form faithful to the original.

Let’s look at how that’s done.

Data Mapping Binary Objects

October 21, 2020/1 Comment/in Data Integration, Database mapping, SQL, Tools /by David McGahey

Binary objects are difficult to manage in databases. They are large, their content is not human readable, and they can contain bytes of data easily misinterpreted as control characters. Even the data type name for binary large objects – BLOB – reflects most database managers’ dislike of them. Before relational databases, the definition of a blob was “something undefined or amorphous.”

Altova MapForce, the award-winning, graphical data mapping tool for any-to-any conversion and integration, includes features for effortlessly data mapping binary objects to or from all popular relational databases. Data such as images, PDF files, video files, or any other binary data can be mapped. Let’s look at an example.

Tags: binary objects, data mapping

Data Mapping JSON Lines

February 24, 2020/in Data Integration, Database mapping, JSON /by David McGahey

The JSON data format continues to evolve as an open standard as it is creatively applied to new data interchange requirements. JSON Lines, defined at http://jsonlines.org/, is a convenient text format for storing structured data where each record is a single line and a valid JSON object. JSON Lines handles tabular data and clearly identifies data types without ambiguity. This allows records to be processed one at a time, which makes the format very useful for exporting and sending data.

Altova MapForce supports data mapping JSON Lines as either a data source or target. Let’s look at a mapping project to extract records from a database table and map to a JSON Lines file for output.

Transitioning Data Mapping Projects from Development through Testing and Production

November 29, 2019/in Data Integration, Database mapping, Servers, Software /by David McGahey

Data mapping projects often mirror software development efforts with distinct phases for design, testing, and deployment. This is especially true for ETL (Extract Transform Load) projects when repeated data mapping execution is required as new data becomes available, and the stakes increase higher with large data sets. The Altova MissionKit and Server Software products provide Global Resources to define configurations for each project phase and smoothly transition between them.

Let’s take a look at an example based on a MapForce data mapping from a source file to a database.

Database Mapping with Database Exception Handling

June 5, 2019/in Data Integration, Database, Database mapping /by David McGahey

Critical business processes depend on reliable data and database administrators and other data analysts want to be confident in the integrity of information stored in database tables. During automated ETL (Extract Transform Load) operations or other database import tasks, invalid data might be encountered that jeopardizes success of the procedure. Altova MapForce includes database exception handling to roll back the affected data when an error occurs and optionally proceed with the rest of a database mapping.

For instance, an error in a single record need not prevent execution of a mapping from continuing, such as when certain database constraints prevent the mapping from inserting or updating invalid data.

Database Tracing to Log Changes Made by a Data Mapping Project

May 1, 2019/in Data Integration, Database, Database mapping, Database tracing /by David McGahey

Database administrators and other data professionals often want to maintain a record of changes in critical databases, especially when updates are made by automated scripts or other operations. Database tracing lets administrators track critical changes or anomalies, and help recover from errors. Altova MapForce supports database tracing for all popular relational databases to log the changes made by a data mapping project to the database when the mapping runs.

When tracing is enabled, events such as database insert or update actions, or errors, are logged in an XML file that you can later analyze or process further in an automated way.

Database tracing can be enabled at the database component, table, stored procedure, or database field level. You can choose to trace all messages or only errors, or you can disable tracing completely.

In addition to tracing errors that occur during the execution of a mapping to a target database, MapForce also enables database transaction handling to roll back the affected part of the database data when an error occurs, then optionally proceed with the rest of the mapping.