High-Performance Computing With Apache Ignite
High-performance computing (HPC) is the ability to process data and perform complex calculations at high speeds. Using Apache Ignite® as a high-performance compute cluster, you can turn a group of commodity machines or a cloud environment into a distributed supercomputer of interconnected Ignite nodes. Ignite enables speed and scale by processing records in memory and reducing network utilization with APIs for data and compute-intensive calculations. Those APIs implement the MapReduce paradigm and allow you to run arbitrary tasks across the cluster of nodes.
Co-located Processing
Ignite uses the notion of co-located processing to guide HPC workloads implementations in distributed in-memory environments. Co-located processing increases the performance of your complex calculations by running them straight on the Ignite cluster nodes. These calculations are done only on local data sets available on the nodes, thus avoiding data shuffling over the network and resulting in orders of magnitude increase in performance.
To exploit the co-located processing in practice, first, you need to co-locate data by storing related
records on the same cluster node. As an example of related or co-located data, consider your bank account
and transactions posted to it. Once you set accountID as an affinity key for the
Transactions table, you'll instruct Ignite to store all transactions for your
accountId on the same node that keeps the record of your account in the
Accounts table. Now let's say a payment processing system sends a compute task that
verifies previous transactions of your account. Since the data is co-located, Ignite will execute this
task directly on the node that stores your account record with all completed transactions and finish the
verification locally on that machine instead of pulling all the transactions back to the application
over the network. This method of executing a task on the node where the data resides provides
exceptionally high performance.The effect is even more significant when the system needs to process
millions of transactions per second, verifying billions of previously completed payments.
Compute APIs
Ignite provides compute APIs (also known as compute grid) for creating and scheduling custom tasks of arbitrary complexity. The APIs implement MapReduce paradigm and are presently available for Java, C#, and C++.