In-Memory Compute Grid

Ignite In-Memory Compute Grid allows executing distributed computations in a parallel fashion to gain high performance, low latency, and linear scalability. Ignite compute grid provides a set of simple APIs that allow users distribute computations and data processing across multiple computers in the cluster.

Distributed parallel processing is based on the ability to take any computation and execute it on any set of cluster nodes and return the results back.

Videos:

Code Examples:
                                Ignite ignite = Ignition.ignite();

                                // Print out hello message on all cluster nodes.
                                ignite.compute().broadcast(() -> System.out.println("Hello Node!"));
                            
                                Collection<IgniteCallable<Integer>> calls = new ArrayList<>();

                                // Iterate through all words in the sentence and create callable jobs.
                                for (String word : "How Many Characters".split(" "))
                                    calls.add(word::length);

                                // Execute collection of callables on the Ignite cluster.
                                Collection<Integer> res = ignite.compute().call(calls);

                                // Add all the word lengths received from cluster nodes.
                                int total = res.stream().mapToInt(Integer::intValue).sum();
                            
                                IgniteCompute compute  = ignite.compute();

                                // Execute closure on all cluster nodes.
                                Collection<Integer> res = ignite.compute().apply(
                                    String::length,
                                    Arrays.asList("How Many Characters".split(" "))
                                );

                                // Add all the word lengths received from cluster nodes.
                                int total = res.stream().mapToInt(Integer::intValue).sum();
GitHub Examples:

Also see Java 7 examples and Java 8 examples available on GitHub.

Compute Grid Features

Feature Description
Distributed Closure Execution

Ignite compute grid allows to broadcast and load-balance any closure within the cluster or a cluster group, including Java 8 lambdas, as well as plain Java runnables and callables.

Docs for this Feature
ForkJoin Processing

ComputeTask is Ignite abstraction for the in-memory ForkJoin, paradigm, which is a light-weight form of MapReduce. Pure MapReduce was never built for performance and only works well when dealing with off-line batch oriented processing (e.g. Hadoop MapReduce).

However, when computing on data that resides in-memory, real-time low latencies and high throughput usually take the highest priority. Also, simplicity of the API becomes very important as well. With that in mind, Ignite introduced the ComputeTask API, which is Ignite ForkJoin (or a light-weight MapReduce) implementation.

Docs for this Feature
Clustered ExecutorService

Ignite provides a cluster-enabled implementation of standard JDK ExecutorService and automatically executes all the computations in load-balanced fashion within the cluster. Your computations also become fault-tolerant and are guaranteed to execute as long as there is at least one node left. You can think of it as a distributed cluster-enabled thread pool.

Docs for this Feature
Collocation of Compute & Data

Collocation of computations with data allow for minimizing data serialization within the network and can significantly improve performance and scalability of your application. Whenever possible, you should always make best effort to collocate your computations with the cluster nodes caching the data that needs to be processed.

Ignite provides various ways to collocate compute with data either automatically or manually as needed.

Docs for this Feature
Fault Tolerance

Ignite supports automatic job failover. In case of a node crash or any other error, jobs are automatically transferred to other available nodes for re-execution. The pluggable FailoverSpi is responsible for handling the selection of a new node for the execution of a failed job.

At Least Once Guarantee - Ignite guarantees that as long as there is at least one node standing, no job will ever be lost.

Docs for this Feature
Load Balancing

Load balancing component balances job distribution among cluster nodes. In Ignite load balancing is achieved via a pluggable LoadBalancingSpi which controls load on all nodes and makes sure that every node in the cluster is equally loaded.

In homogeneous environments with homogeneous tasks load balancing is achieved by random or round-robin policies. However, in many other use cases, especially under uneven load, more complex adaptive load-balancing policies are provided.

Docs for this Feature
Job Checkpointing

Checkpointing is supported via a pluggable CheckpointSpi. It provides ability to save an intermediate job state. Checkpointing can be useful when long running jobs need to store some intermediate state to protect from node failures. Then on restart of a failed node, a job would load the saved checkpoint and continue from where it left off.

Docs for this Feature
Job Scheduling

Pluggable CollisionSpi provides a fine-grained control over how the jobs are scheduled once they arrive to a specific cluster node for execution. Various policies are supported, including FIFO, Priority, or even job stealing.

Docs for this Feature