In-Memory Data Grid with Apache Ignite
Apache Ignite® is an in-memory data grid that accelerates and scales your databases, services, and APIs. It supports key-value and ANSI SQL APIs, ACID transactions, co-located compute, and machine learning libraries required for real-time applications.
An in-memory data grid deployment is a read-through/write-through caching strategy, in which the application layer treats the data grid as the primary data store. The application layer writes to and reads from Ignite. Ignite ensures that any underlying database stays updated and consistent with the in-memory data.
As an in-memory data grid, Ignite provides all essential APIs needed to simplify its adoption. The APIs include distributed key-value and ANSI SQL queries, ACID transactions, co-located computations, and machine learning models. While key-value and SQL calls let you request, join, and group distributed data sets, the compute and machine learning components help to eliminate data shuffling over the network, thus, boosting compute and data-intensive calculations.
Ignite is capable of storing data both in memory and on disk with two options for data persistence -- you can persist changes in an external database or let Ignite keep data in its native persistence. Let's review both of these options.
Ignite and External Databases
Ignite can improve the performance and scalability of any external database such as RDBMS, NoSQL or Hadoop, by sliding in as an in-memory cache between the application and the database layer. When an application writes data to the cache, Ignite automatically writes-through or writes-behind all data modifications to the underlying external store. Ignite also performs ACID transactions where it coordinates and commits a transaction across the cluster as well as the database.
Additionally, Ignite can be deployed as a shared and unified in-memory layer that stores data sets originating from disjointed databases. Your applications can consume all the data from Ignite as a single store while Ignite can keep the original databases in sync whenever in-memory data gets updated.
However, there are some limitations if an external database is used as a persistence layer for Ignite deployments. For instance, if you run Ignite SQL or scan queries, you need to ensure that all the data has been preloaded to the in-memory cluster. Note that Ignite SQL or scan queries can read data from disk only if it is stored in the native persistence.
Ignite Native Persistence
Ignite native persistence is a distributed ACID and SQL-compliant disk store that transparently integrates with Ignite in-memory layer. When the native persistence is enabled, Ignite stores both data and indexes on disk and eliminates the time-consuming cache warm-up step. Since the native persistence always keeps a full copy of data on disk, you are free to cache a subset of records in memory. If a required data record is missing in memory, then Ignite reads it from the disk automatically regardless of the API you use -- be it SQL, key-value, or scan queries.