BigMemory: In-Memory Data Management for the Enterprise

Knowledge base
, ,
1

 

Issue 3, 2012 Download pdf


The age of Big Data is here. Exploding data volumes from many different sources and formats is overwhelming "the ability of typical database software tools to capture, store, manage, and analyze"1 it. With BigMemory, you can max out the biggest servers on the market by moving terabytes of high-value data out of slow, expensive databases and mainframes and into memory where applications can use it effectively. 
 

Cheap RAM + Big Data = Business Power 
Exploding data creates business opportunities and problems that prevailing technology platforms cannot address. To handle the volume and scale of data, as much data as possible must be moved out of slow, disk-based storage into high-speed memory where it can be addressed in real-time.

Luckily, the big data explosion is concurrent with a steep decline in the price of RAM that has made abundant memory available on commodity hardware. Tens of gigabytes of server RAM are now common and hundreds of gigabytes may be had for a few thousand dollars. There is every indication that the trend will continue, making terabytes of RAM cost-effective on commodity hardware in the near future.
 
However, because memory is volatile, getting data into memory and keeping it available and consistent requires a comprehensive in-memory data management solution. Software AG’s flagship data management product, BigMemory, provides enterprises with fast, predictable, reliable and inexpensive access to huge amounts of data in memory. It does this by providing a powerful and cost-effective way to access data where it's needed, when it's needed and how it's needed.
 

How It Works 
BigMemory is packaged in several ways, depending on the needs of the application. In its simplest form, as shown in Figure 1, BigMemory is an in-memory Java data store with a backing disk store for restarts. It uses a simple put/get/search API that goes into applications with the addition of a Java Archive (JAR) library and a few lines of configuration.

 
By using vastly more memory in a single JVM, BigMemory consolidates the number of JVMs and the amount of server hardware applications needed - often by a factor of five to ten fewer servers. By storing more data in memory locally, BigMemory reduces reliance on backend databases thus maximizing hardware utilization.
FIGURE 1: BigMemory Architecture

Understanding the Performance Gains
Data systems traditionally use a tiered store organization to automatically move data between faster, high-cost storage mechanisms to slower, low-cost (and, generally, much larger) ones. BigMemory uses a similar tiered store organization, as shown in Figure 2, to automatically move data between the different tiers as needed. The top two tiers––the Java Virtual Machine (JVM) heap memory and the in-process, off-heap BigMemory store––use the RAM on the application server host. Since application server hardware typically ships with tens of gigabytes of RAM and may be inexpensively fitted with hundreds of gigabytes or more, BigMemory efficiently stores terabytes of data in RAM where your application can most readily use it.

FIGURE 2: Tiered Memory Store
 
 

Scale Up And Out 
BigMemory's tiered store organization keeps data where applications need it for fast, predictable access precisely when it's needed. And, because local memory is fast and increasingly cheap and abundant, BigMemory keeps as much data locally as available RAM permits.

The highly available server array scales out to tens of terabytes of in-memory storage and delivers continuous uptime. A high-speed disk store ensures the durability of data kept in volatile memory and the ability to quickly return to steady state operation when applications are restarted.  Configurable consistency guarantees ensure that data in memory across multiple servers is up-to-date.

 

Snaps into Any Application 
BigMemory's data access API––the de-facto Java standard Ehcache interface––combines the simple get/put methods of a key/value store with powerful query, search and analysis capabilities to give applications unprecedented access to data that is otherwise locked away in slow, expensive databases.

Data in BigMemory is stored as plain Java objects. This simplifies the programming model and enables applications to efficiently use BigMemory data without the overhead of the object-relational mapping transformation required when accessing relational databases. Once data is in BigMemory, it stays in the format most readily used by the application.
 
BigMemory also works well in heterogeneous technology environments. Deployed as an in-memory data service, clients across the enterprise––for example, Java, .NET and web applications––use a technology-agnostic message-oriented middleware (MOM) interface to access BigMemory.
 
 

Conclusion 

BigMemory makes existing applications orders of magnitude faster and more affordable.  But, because terabyte in-memory scale is unprecedented, BigMemory also changes what applications can do. Terabytes of data available at microsecond speed make it possible to extract value from that data in new ways and enables entirely new applications that weren't possible before.

 
Learn more about BigMemory in my detailed article on the Tech Community Wiki 
 
Join the Software AG Tech Community for full access to additional documentation, demos, and discussions forums about BigMemory and Enterprise Ehcache at Tech Community.SoftwareAG.com/terracotta
 
 

Congratulations Terracotta!

BigMemory wins DataWeek's 2012 Big Data Technology Top Innovator award! 

http://www.dataweek.co/index/winners