Database Archives

Text Search for Precise PDF Data Extraction

December 3, 2024/in Data Integration, Database, ETL /by Erin Cavanaugh

PDF documents are used at many stages of modern business workflows, often serving as the format of choice for invoices, reports, legal contracts, and other critical documents. While PDFs are ideal for preserving content integrity and a particular visual layout, their structure makes automated data extraction challenging. For organizations engaged in data integration and ETL, unlocking information contained in PDFs is a necessity—and this is where the MapForce PDF Extractor comes in.

The MapForce PDF Extractor includes multiple tools for visually defining extraction rules to map PDF data to other formats. One that is particularly useful for zeroing in on specific content is text search. Here’s how it works – including a video demo.

cartoon image of computer monitor with PDF charts peeling off the screen

Tags: MapForce, PDF, PDF Extractor

AI Tools for Instant App Creation

February 6, 2024/in AI, App Development, Database, No Code /by Erin Cavanaugh

Visual, no code tools like Altova RecordsManager have revolutionized the field of data-centric app creation, making it faster and more accessible. But now imagine expressing your database vision in a single sentence, and having it created automatically – including not just the database structure and tables, but forms and reports as well. That’s exactly what the new AI Assistant in RecordsManager does.

With a single AI prompt, users of all skill levels can turn their ideas into functional database solutions without any coding or database design expertise required. RecordsManager lets you skip the manual work of database design so you can focus more on the higher-level aspects of your project.

Let’s see how it works.

AI-Ready Database Tool

December 8, 2023/in AI, Database /by Erin Cavanaugh

AI is a great productivity booster for IT projects, and working with databases is one area where AI is really making inroads for improving efficiency. By leveraging AI in database tools, DBAs and database developers of any skill level can save time and effort with AI-generated SQL scripts and sample data, for instance, as well as query optimization and troubleshooting.

Altova offers an integrated AI Assistant in DatabaseSpy to help with SQL script creation, data modeling, SQL and error explanations, and even SQL pretty-printing. This makes the multi-database tool, which supports all major databases in a single UI, even more useful.

Let’s take a look at how it works.

Tags: AI, AI SQL tools, DatabaseSpy, SQL editor

AI-based Database Image Classification with Altova MapForce

July 31, 2023/in Data Integration, Database mapping, Tools /by David McGahey

One of the most common examples of AI in our everyday lives is facial recognition. Facial recognition is the process of identifying or verifying a person’s identity based on their face. Facial recognition is used in many applications, such as unlocking our phones with FaceID, tagging our friends on social media platforms like Facebook, and checking in at airports or hotels with biometric scanners. Facial recognition can make our lives more convenient and secure, but it can also raise some privacy and ethical concerns. For instance, how can we ensure that our facial data is not misused or stolen by hackers or malicious actors? How can we prevent facial recognition from being used for surveillance or discrimination? How can we ensure that facial recognition is accurate and fair, and does not have any biases or errors?

The paragraph above was generated by ChatGPT in response to my request to describe the benefits and risks of artificial intelligence and include a real-life example. It’s interesting that ChatGPT chose FaceID as the example, since FaceID is simply one variation of image analysis and AI-powered image classification offers potential to automate many real-world tasks.

One common use-case is a product catalog, wherein a company manages product information provided by many different manufacturers. A product loaded into that database may have a name that does not necessarily include a precise description of the item. For instance, wellington is a boot, fedora is a hat, a mongoose is a bicycle, and a yellow watermelon shiny needlefish is a fishing lure. We can make use of AI-powered image classification using the Microsoft Azure Cognitive Services Computer Vision API to address this problem. The Computer Vision Service takes the image data or URL as its input and returns information about the content. One service generates image classification tags based on a training set of recognizable objects, living beings, scenery, and actions that the Azure AI has been trained on. These tags allow us to categorize products in the database accordingly and may even correspond to search terms a user might provide to find products in the catalog.

decorative image depicting an AI "brain"

AI-based support request sentiment analysis using MapForce and GPT-4

July 17, 2023/in Data Integration, Database, ETL /by Alexander Falk

Automated sentiment analysis of text, such as user reviews, has historically been a challenge. Because of the myriad intricacies of natural language, systems faced difficulties in analyzing context and nuances. This required an inordinate amount of manual work to overcome.

One of the many useful capabilities of modern AI systems that are based on large language models (LLMs) such as OpenAI’s GPT-4 is that they are very good at sentiment analysis of natural text inputs. We can use that capability to build a very efficient database solution in MapForce that, for example, goes through all the new incoming records in a support database and automatically determines whether a particular support request or other customer feedback is positive, negative, constitutes a bug report, or should be considered as a feature request.

Decorative lead photo depicting machine learning for sentiment analysis

Building Apps with an Intelligent Database Wizard

March 20, 2023/in App Development, Database, Development, Low Code /by Erin Cavanaugh

Data-driven solutions like database and enterprise apps rely on connection to, and interaction with, backend databases. Backend relational databases, however, store data in tables that reflect complex data relationships. This provides numerous advantages for effective data management and data integrity but can make it difficult to access and work with the data stored therein in new ways. App developers need to have a comprehensive understanding of database design principles and the SQL query language just to get started.

In contrast, real world data relationships most often represent parent-child relationships or even deeper hierarchical structure. As such, working with hierarchical data where relationships can be visualized in a tree structure can be much simpler and more flexible, leading to faster development. This approach is also more accessible to developers without extensive SQL expertise.

To make building apps that connect to the backend relational databases that are ubiquitous in today’s enterprise easier, faster, and available to a wider range of developers, Altova MobileTogether takes an entirely unique approach. Its visual Database Wizard lets developers easily build a query that returns hierarchical data, work with that data in the app, and then easily save the data back in hierarchical form, letting MobileTogether take care of normalizing the data and writing it back to the corresponding linked tables. Let’s take a look at how it works.

Role-based Access Control in Enterprise Apps

March 1, 2023/in App Development, Database, No Code /by Erin Cavanaugh

Enterprise database apps are increasing in prevalence due to their advantages for enabling access to—and easy management of—the ever-growing amount of critical data business users need to work with on a day-to-day basis. Unlike other types of business productivity apps, database apps must include measures for managing different levels of user access to maintain the security and integrity of the enterprise data they expose.

This can include managing read-only and editing access rights or restrictions on access to certain types of data. While it is essential to ensure that only authorized personnel have access to confidential data, levels of permissions often vary throughout an organization. Apps built using Altova RecordsManager include comprehensive tools for managing role-based access to database data that can reflect these complicated relationships that exist within an organization.

Let’s take a look at how RecordsManager makes it easy for app administrators to manage complex role-based permissions with visual tools.

Workers looking together at a tablet with an office in the background

Getting Started with Altova RecordsManager

July 13, 2022/in App Development, Database, No Code /by David McGahey

RecordsManager is a new tool from Altova to build business database solutions in record time using a powerful visual design interface. RecordsManager is a free, pre-built MobileTogether solution that is automatically available when you install MobileTogether Designer. The pre-built solution includes sample data sets, and the MobileTogether Simulator previews execution of the database solution right inside the free to use MobileTogether Designer. Getting started with Altova RecordsManager is just one click away when you launch the Designer. Soon you will be building your own custom database apps without needing backend development or manual coding.

Build No-Code Database Apps with RecordsManager

June 8, 2022/in App Development, Database, No Code /by Erin Cavanaugh

We are excited to announce availability of a new product in the Altova app development framework: RecordsManager.

Altova RecordsManager offers a completely visual, no-code interface for quickly creating custom database apps. RecordsManager is perfect for any app that handles data in records: think contract management, a customer database, an invoicing system, a database of local attractions or collections – the sky is the limit.

Your RecordsManager app will automatically be available on desktop devices as well as on mobile using native iOS and Android apps and provides tons of features that make it easy for end-users. Let’s see how it works.

Promotional image announcing RecordsManager

How to Compare CSV Files or Compare a CSV File to a Database Table

May 4, 2022/in Data Integration, Database, Tools /by David McGahey

CSV files are a quick and convenient way to record structured data in a generic format. Because CSV files are so easy to create, multiple similar versions of very large CSV files can quickly proliferate. Often it becomes necessary to compare CSV files to find the desired version. In an ETL scenario, a data analyst may want to compare a CSV file to a database table for validation or to update data.

DiffDog, the unique XML-aware diff / merge tool from Altova, supports CSV as a native file format for comparison and can compare and selectively merge data CSV to CSV, or between a CSV file and database table. Let’s look at an example.

Transforming and Converting Protobuf

February 10, 2022/in Data Integration, Database, JSON, protocol buffers, XML /by David McGahey

MapForce supports mapping protocol buffers (Protobuf) to and from other structured data formats as mapping sources or targets. In the constant quest for more efficient ways to transfer, manipulate, and manage large structured data sets, Google has created a language- and platform-neutral data format similar to XML, but smaller, faster, and simpler than even JSON data. Tools are available to generate and work with Protobuf using Java, Python, C++, C#, Ruby, and other programming languages.

The structure of any Protobuf message is defined in a .proto file that defines each field name and value type. Altova MapForce lets users drop these .proto files into a data mapping as a source or target along with any other data, including XML, JSON, relational databases, Excel, flat files, REST and SOAP web services, and others. .proto files versions 2 and 3 are supported.

A MapForce data mapping creates compatibility between existing XML, JSON, database or legacy data formats and new applications leveraging the efficiency of Protobuf.

Data Mapping NoSQL Databases

November 26, 2021/in Database, Database mapping, JSON /by David McGahey

NoSQL databases are non-tabular databases that store data differently than traditional databases made up of relational tables. Two of the most popular NoSQL databases, MongoDB and Apache CouchDB, store data as collections of BSON (binary JSON) and JSON documents. These databases leverage flexible JSON schemas and scale easily with large amounts of data and high user loads.

Altova MapForce has long supported data mapping all popular relational databases and now also includes native support for data mapping NoSQL databases. MapForce includes functionality for inserting, extracting, filtering, and ordering NoSQL data. Let’s look at an example.

NoSQL Database Support and More in Version 2022

October 27, 2021/in Database, Software, XBRL /by Erin Cavanaugh

Altova Software Version 2022 is now available, with exciting new support for mapping and converting NoSQL databases in MapForce, pure text report output in StyleVision, and Windows 11 across the product line. The release also adds support for the exciting new OIM standard from XBRL International.

Here’s a look at the highlights.

Data Mapping Binary Objects – Part 2

October 30, 2020/in Data Integration, Database mapping, SQL, Tools /by David McGahey

Binary objects – BLOBs — can be cumbersome to manage in databases. In an earlier post we described a MapForce data mapping to insert binary objects into a database with generated metadata to identify the BLOBs later. The companion challenge in data mapping binary objects is to extract binary data and save it in a comprehensible form faithful to the original.

Let’s look at how that’s done.

Data Mapping Binary Objects

October 21, 2020/1 Comment/in Data Integration, Database mapping, SQL, Tools /by David McGahey

Binary objects are difficult to manage in databases. They are large, their content is not human readable, and they can contain bytes of data easily misinterpreted as control characters. Even the data type name for binary large objects – BLOB – reflects most database managers’ dislike of them. Before relational databases, the definition of a blob was “something undefined or amorphous.”

Altova MapForce, the award-winning, graphical data mapping tool for any-to-any conversion and integration, includes features for effortlessly data mapping binary objects to or from all popular relational databases. Data such as images, PDF files, video files, or any other binary data can be mapped. Let’s look at an example.

Tags: binary objects, data mapping

New Data Integration Tools

March 25, 2020/in Data Integration, Database, EDI /by Erin Cavanaugh

Altova MissionKit tools offer numerous ways to connect to, query, and integrate data from disparate sources. With multiple product releases each year, we’re constantly working to deliver increased power and efficiency for data integration, while adding features requested by customers. This includes ongoing updates to built-in support for all major SQL databases across the product line.

Let’s take a look at some of the recently added tools and enhancements.

New data integration tools in Altova's release

Data Mapping JSON Lines

February 24, 2020/in Data Integration, Database mapping, JSON /by David McGahey

The JSON data format continues to evolve as an open standard as it is creatively applied to new data interchange requirements. JSON Lines, defined at http://jsonlines.org/, is a convenient text format for storing structured data where each record is a single line and a valid JSON object. JSON Lines handles tabular data and clearly identifies data types without ambiguity. This allows records to be processed one at a time, which makes the format very useful for exporting and sending data.

Altova MapForce supports data mapping JSON Lines as either a data source or target. Let’s look at a mapping project to extract records from a database table and map to a JSON Lines file for output.

Transitioning Data Mapping Projects from Development through Testing and Production

November 29, 2019/in Data Integration, Database mapping, Servers, Software /by David McGahey

Data mapping projects often mirror software development efforts with distinct phases for design, testing, and deployment. This is especially true for ETL (Extract Transform Load) projects when repeated data mapping execution is required as new data becomes available, and the stakes increase higher with large data sets. The Altova MissionKit and Server Software products provide Global Resources to define configurations for each project phase and smoothly transition between them.

Let’s take a look at an example based on a MapForce data mapping from a source file to a database.

Database Mapping with Database Exception Handling

June 5, 2019/in Data Integration, Database, Database mapping /by David McGahey

Critical business processes depend on reliable data and database administrators and other data analysts want to be confident in the integrity of information stored in database tables. During automated ETL (Extract Transform Load) operations or other database import tasks, invalid data might be encountered that jeopardizes success of the procedure. Altova MapForce includes database exception handling to roll back the affected data when an error occurs and optionally proceed with the rest of a database mapping.

For instance, an error in a single record need not prevent execution of a mapping from continuing, such as when certain database constraints prevent the mapping from inserting or updating invalid data.

Database Tracing to Log Changes Made by a Data Mapping Project

May 1, 2019/in Data Integration, Database, Database mapping, Database tracing /by David McGahey

Database administrators and other data professionals often want to maintain a record of changes in critical databases, especially when updates are made by automated scripts or other operations. Database tracing lets administrators track critical changes or anomalies, and help recover from errors. Altova MapForce supports database tracing for all popular relational databases to log the changes made by a data mapping project to the database when the mapping runs.

When tracing is enabled, events such as database insert or update actions, or errors, are logged in an XML file that you can later analyze or process further in an automated way.

Database tracing can be enabled at the database component, table, stored procedure, or database field level. You can choose to trace all messages or only errors, or you can disable tracing completely.

In addition to tracing errors that occur during the execution of a mapping to a target database, MapForce also enables database transaction handling to roll back the affected part of the database data when an error occurs, then optionally proceed with the rest of the mapping.

Exploring an Unfamiliar Database with DatabaseSpy

July 12, 2017/1 Comment/in Database, Software, Tools /by David McGahey

Software developers working on a new app, data professionals in a variety of enterprises, and even database administrators often encounter unfamiliar databases and need a database tool to quickly explore tables and relationships.

Altova DatabaseSpy is a unique multi-database query, design, and comparison tool with a graphical database design editor that empowers users exploring an unfamiliar database to quickly visualize tables, relationships, and even datatype definitions that may be unique among database types.

Tags: database tool, DatabaseSpy

Update to Altova’s Database Tool Adds Important New Features

April 24, 2017/in Database, Tools /by David McGahey

DatabaseSpy is the unique database tool that supports all major databases and facilitates database query, design, structure comparison, table content editing and comparison, and even generates elegant charts from query results.

The recent update of DatabaseSpy to version 2017 Release 3 adds several new features, including the ability to automatically generate a complete DDL script for any database schema.