Memory-Centric Storage

Apache Ignite is based on distributed memory-centric architecture that combines the performance and scale of in-memory computing together with the disk durability and strong consistency in one system.

The main difference between the memory-centric approach and the traditional disk-centric approach is that the memory is treated as a fully functional storage, not just as a caching layer, like most databases do. For example, Apache Ignite can function in a pure in-memory mode, in which case it can be treated as an In-Memory Database (IMDB) and In-Memory Data Grid (IMDG) in one.

On the other hand, when persistence is turned on, Ignite begins to function as a memory-centric system where most of the processing happens in memory, but the data and indexes get persisted to disk. The main difference here from the traditional disk-centric RDBMS or NoSQL system is that Ignite is strongly consistent, horizontally scalable, and supports both SQL and key-value processing APIs.

The disk-centric systems, like RDBMS or NoSQL, generally utilize the classic client-server approach, where the data is brought from the server to the client side where it gets processed and then is usually discarded. This approach does not scale well as moving the data over the network is the most expensive operation in a distributed system.

A much more scalable approach is collocated processing that reverses the flow by bringing the computations to the servers where the data actually resides. This approach allows you to execute advanced logic or distributed SQL with JOINs exactly where the data is stored avoiding expensive serialization and network trips.

Depending on the configuration, Ignite can either partition or replicate data across its memory-centric storage. Unlike REPLICATED mode, where data is fully replicated across all nodes in the cluster, in PARTITIONED mode Ignite will equally split the data across multiple cluster nodes, allowing for staring TBs of data both in memory and on disk.

Ignite also allows to configure multiple backup copies to guarantee data resiliency in case of failures.

Regardless of which replication scheme is used, Ignite guarantees data consistency across all cluster members.

The persistence is totally optional in Ignite that allows using the cluster in the memory-only mode where all the data and indexes are stored solely in RAM. In this scenario, you can achieve the maximum performance possible because the data is never written to disk. To prevent possible data loss when a single cluster node fails, it is recommended to configure a number of backup copies (aka. replication factor) appropriately.

There are two ways to enable persistence in Ignite. The first approach is to use its own distributed, ACID, and SQL-compliant persistence that transparently and efficiently integrates with overall memory architecture.

With the the native persistence enabled, Ignite always stores a superset of data on disk, and as much as possible in RAM. For example, if there are 100 entries and RAM has the capacity to store only 20, then all 100 will be stored on disk and only 20 will be cached in RAM for better performance.

The second approach to have the persistence enabled, is to deploy Ignite above an existing 3rd party database such as RDBMS, Apache Cassandra or MongoDB. This mode is usually used to accelerate the underlying database by storing a copy of the data in memory in Ignite. Ignite supports both read-through and write-through modes that ensure data consistency and keeps Ignite and the database in sync. Refer to Ignite persistence page to compare Ignite persistence vs 3rd Party Persistence modes.