In-memory data storage for IS applications
In October 2017 Software AG launched a new big data product, Terracotta DB. Terracotta DB is a unified data platform for operational and analytical work. It has an improved Ehcache API and a brand new Store API for in-memory storage and analytics. Terracotta DB combines the power and convenience of a NoSQL data store with the performance of Software AG’s fastest-yet in-memory data grid technology. At the same time, the webMethods adapter team released a new adapter for Terracotta DB’s Store API, providing easy drag-and-drop, structured access to read, write and query Terracotta DB from webMethods Integration Server (IS). With a flexible data model, in-memory indexing and purpose-built fast restartability, access to Terracotta DB in-memory storage is a powerful addition to the toolkit of IS developers.
| Issue 2, 2018 |  Download PDF |
Terracotta DB Adapter: A bridge to the webMethods world
webMethods developers are used to rapid development of integrated applications using FLOW logic. Key benefits that accelerate the development process include using a simple drag-and-drop approach, visual mapping, easy integration via structured IS doc types and managed connectivity. Terracotta has traditionally exposed Java® APIs and required explicit connection management. The Terracotta DB adapter introduces a new paradigm. It is an Adapter RunTime (ART)-based adapter that uses a Java API to interact with the Terracotta DB Store API. It provides insert, update, delete and query functions in a structured drag-and-drop adapter UI. And the connections can be created and managed within the adapter itself.
Simple and consistent: Connecting to Terracotta DB
Adapter connections are standard ART-based connections that are managed by IS. Users configure the Terracotta DB URI in the administration page of the adapter. IS provides automatic connection pooling. Adapter clients are designed to provide robust and tunable reconnect behavior.
Fig. 1: Adapter Connection Architecture
As usual, connections can be edited, deleted, viewed, copied, and enabled or disabled. Figure 1 shows how adapter clients connect to Terracotta while leveraging centralized connection management in the adapter runtime.
A new model: Understanding data storage in Terracotta DB
SQL databases enable a highly structured way of interacting with the data and the ability to introspect a data model that is static. In Terracotta DB, data is organized into flexible collections called datasets. There are no predefined schemas or table types; each row of cells, or record, can have different data but be part of the same dataset.
Fig. 2: Person Dataset Example
Similar to Terracotta BigMemory, data can be stored both off-heap and on disk. Terracotta DB is deployed in stripes of active and mirror servers which can be clustered together, enabling users to transparently query data across multiple stripes.
Figure 2 shows a person dataset example. The data may have come from different sources—for example, a form on the internet, a structured internal database, or a support ticket—but the unstructured nature of the dataset makes it easy to collect and centralize related information without expensive and complex data transformation. The schema-less design of records in the Terracotta Store is a key difference between classic relational databases and Terracotta DB. The power of this model becomes evident in use cases where data of the same type is being collected across many different appliances or devices and delivered in different formats. Being able to handle the high volumes of dynamic data is a core value delivered by Terracotta DB.
New patterns: Interacting with Terracotta DB
The Terracotta DB adapter offers a rich set of mechanisms for interacting with dynamic, NoSQL-style datasets.
- User-specified - The user defines the cells to insert or delete into the dataset, providing the cell name and cell type.
- IS doctype - The cell definitions are automatically parsed from the fields in an IS document. The IS document is referenced in the dataset definition. This pattern enables easy movement of data from one system to another with FLOW providing decision logic.
- Data sampling - Cell definitions (cell name and type) are sampled from the dataset of a Terracotta Store based on a row limit provided by the user. The adapter returns definitions of all the cell types found in the sampled rows. Being able to easily query the most common record formats is a useful tool for interacting with a data model which is inherently unpredictable.
Users can create insert, update, query and delete services in the Terracotta DB Adapter. One unique function of the update service is the ability to “upsert." Because all rows are not the same, if an update operation is applied to more than one row, it’s possible the cell may not exist in that row. The “upsert” operation enables creation of a new cell or updating of an existing cell. Similarly a “delete” capability can delete a cell definition from a set of rows if and only if that cell exists in the row.
The adapter’s query service queries on a given key. Users can filter query results based on cell name using standard relational operators (=, <>) and then connect those filter parameters using operands (AND, OR) to form more complex queries.
Frequently asked questions
- Is Terracotta Big Memory being replaced by Terracotta DB? No. These are two different products with different use cases.
- What happens to my current Terracotta license? It will continue to apply to Terracotta BigMemory. Terracotta DB is a separately licensed product.
- Will the Terracotta DB Adapter work with my current Terracotta BigMemory installation? No, the Terracotta DB adapter only works with Terracotta DB.
- Can I run Integration Server in a cluster with Terracotta BigMemory and connect to Terracotta DB for my applications? Yes, you can run IS in clustered mode with Terracotta BigMemory, and set up a Terracotta DB adapter to connect to Terracotta DB.
How to find out more
A free trial of Terracotta DB is available here: https://www.softwareag.com/us/products/terracotta/datafabric/terracotta_db/default.html