Apache IGNITE 2.5

Apache Ignite was always appreciated by its users for two primary things it delivers - scalability and performance. Throughout the lifetime many distributed systems tend to do performance optimizations from a release to release while making scalability related improvements just a couple of times. It's not because the scalability is of no interest. Usually, scalability requirements are set and solved once by a distributed system and don't require significant additional interventions by engineers.

However, Apache Ignite grew to the point when the community decided to revisit its discovery subsystem that influences how well and far Ignite scales out. The goal was pretty clear - Ignite has to scale to 1000s of nodes as good as it scales to 100s now.

It took many months to get the task implemented. So, please join me in welcoming Apache Ignite 2.5 that now can be scaled easily to 1000s of nodes and goes with other exciting capabilities. Read this article which outlines major achievements of the release.

Features and Improvements

implement getCurrentCoordinator method in IgniteMXBean [#IGNITE-8430]

Add new metrics for data storage [#IGNITE-8078]

Implement new JMX metrics for transactions [#IGNITE-8077]

Add a GA Grid example that solves 'Knapsack Problem' [#IGNITE-8041]

Add Ignite nightly builds references to the site [#IGNITE-8040]

Ship SQL scripts with same database in Ignite [#IGNITE-7884]

ODBC: Implement connection failover [#IGNITE-7811]

LSQR: Sparse Equations and Least Squares for Lin Regression [#IGNITE-7438]

Partition based dataset implementation [#IGNITE-7437]

Thin client Java API - data grid API [#IGNITE-7421]

Apache Ignite 2.5 RPM and DEB packages [#IGNITE-7108]

Spark Data Frame Support. Strategy to convert complete query to Ignite SQL [#IGNITE-7077]

SQL: implement COPY command for efficient data loading [#IGNITE-6917]

Inform user about missed nodes from BLT during a cluster activation [#IGNITE-6906]

Adding GA Grid to Apache Ignite ML module [#IGNITE-6899]

ODBC: Add secure connection support [#IGNITE-6810]

SVM for Apache Ignite ML module [#IGNITE-6585]

Add clearNodeLocalMap() to IgniteMXBean [#IGNITE-6133]

Introduce pluggable string encoder/decoder [#IGNITE-5655]

JDBC: fix unordered stream compatibility check [#IGNITE-8498]

Move GA Grid description to ML site page [#IGNITE-8349]

Cache management and server node authorisation [#IGNITE-8221]

Debug why REST return null as sessionToken when authentication enabled [#IGNITE-8202]

Refactor REST API for authentication [#IGNITE-8201]

control.sh Failed with NPE in case of adding not online node in base line [#IGNITE-8159]

Fix suite [Prepare Vote #1] .Net & C++ [#IGNITE-8150]

JDBC thin driver: use semicolon as parameter's delimiter [#IGNITE-8148]

SQL: NPE during key/value pair validation [#IGNITE-8147]

Web Console: Fix change detection logic on configuration [#IGNITE-8142]

Missing SQL-DDL Authorization [#IGNITE-8135]

Java thin client: throw handshake exception on connect phase [#IGNITE-8097]

Web Console: Refactor components for generated code preview [#IGNITE-8095]

Java Thin Client Authentication [#IGNITE-8076]

.NET thin client: Add username/password to handshake [#IGNITE-8042]

.NET: Add "authenticationEnabled" flag to IgniteConfiguration [#IGNITE-8034]

Authentication: add to REST support of new authentication [#IGNITE-8002]

Enhance performance of the thin JDBC streaming mode [#IGNITE-7999]

DML: improve deadlock handling [#IGNITE-7984]

NPE in TxRollbackOnTimeoutNearCacheTest.testRandomMixedTxConfigurations [#IGNITE-7983]

Web Console: Refactor query notebooks to dedicated screen [#IGNITE-7979]

Web Console: Optimize on-focus-out directive [#IGNITE-7958]

SQL: Add upload benchmarks for sql streamer feature [#IGNITE-7950]

Web Console: Refactor post validation on sign in / sign up [#IGNITE-7949]

Visor CMD: Support cache.lostPartitions() and ignite.resetLostPartitions() [#IGNITE-7940]

Enable either swap space or persistence but not both [#IGNITE-7902]

Add tests for using TreeMap/TreeSet in cache key [#IGNITE-7881]

Ignite Platform .NET Long Running: Stable failed with execution timeout [#IGNITE-7878]

JDBC thin driver: set default socket buffer sizes to greater value [#IGNITE-7860]

Java thin client: add username/password authentication support [#IGNITE-7854]

ODBC: Support username/password authentication [#IGNITE-7852]

REST: Add support to get values inserted via API or SQL [#IGNITE-7803]

All critical system workers health should be covered by IgniteFailureProcessor [#IGNITE-7772]

Ignite Cache 6: testRandomMixedTxConfigurations failed probably after jsr166 removal [#IGNITE-7770]

Ignite CPP tests win32 failure [#IGNITE-7763]

SQL COPY: rename "batch_size" to "packet_size" [#IGNITE-7737]

Enable Ignite Update Notifier tests [#IGNITE-7695]

Move shared memory suite (IpcSharedMemoryCrashDetectionSelfTest) to Ignite Basic 2 [#IGNITE-7690]

Incorrect AllocationRate counting [#IGNITE-7685]

Broken javadoc in partitioned dataset [#IGNITE-7643]

Fix links for Javadoc [#IGNITE-7625]

Deprecate CacheLocalStore annotation [#IGNITE-7588]

CLI AUTH: add user authentication to JDBC thin driver [#IGNITE-7558]

CLI AUTH: implement SQL commands to users management [#IGNITE-7557]

SQL: Create data load benchmarks [#IGNITE-7531]

CLI AUTH: implement user credential storage and default admin user creation [#IGNITE-7439]

Get rid of LongAdder8, ConcurrentHashMap8, etc [#IGNITE-7386]

Add guard to prevent concurrent usage the JDBC thin connection [#IGNITE-7343]

Make use of plain java.util.Base64 instead of reflective alternatives [#IGNITE-7312]

JDBC thin driver: introduce streaming mode [#IGNITE-7253]

DiscoverySpi based on Apache ZooKeeper [#IGNITE-7222]

Reconsider WAL archive strategy [#IGNITE-7017]

Add ability to disable WAL for ceratin caches in runtime [#IGNITE-6411]

Native facility to control excessive GC pauses [#IGNITE-6171]

Integrate communication with coordinator in tx protocol [#IGNITE-5932]

Web Console: New Configuration Screen [#IGNITE-5466]

Eviction Rate memory metric to be implemented [#IGNITE-5265]

Test suites improvements [#IGNITE-4445]

Refactor direct usage of Angular API [#IGNITE-4091]

Implement support for optional key type in REST HTTP get command [#IGNITE-3345]

Add support for scan near cache to VisorQueryTask [#IGNITE-2093]

Peer deployment does not work for continuous query transformers [#IGNITE-8528]

Long running transaction JMX [#IGNITE-8402]

Assertion error during simultaneous auto-activation and manual activation [#IGNITE-8323]

Node could be stopped due to a valid termination of exchange worker [#IGNITE-8303]

PDS Direct IO flaky failure by timeout: 10% testPageRecoveryAfterFileCorruption [#IGNITE-8302]

Web console: make "Beta" ribbon less obtrusive [#IGNITE-8294]

Web console: place sign up inputs in two columns [#IGNITE-8287]

Add utilities to check and display cache info [#IGNITE-8277]

TxRecoveryStoreEnabledTest.testPessimistic fails on TC [#IGNITE-8256]

ML package cleanup for 2.5 release [#IGNITE-8232]

Print out information about configured failure handler [#IGNITE-8228]

Thousands of warning messages per second in log files [#IGNITE-8226]

Add exchange latch state to diagnostic messages [#IGNITE-8218]

Minor javadoc improvement (missing group for new package) [#IGNITE-8216]

Print out information on how many nodes left until auto-activation [#IGNITE-8192]

Print out information when cluster is not activated [#IGNITE-8191]

Print out an information message when local node is not in baseline [#IGNITE-8190]

Integrate gradient descent linear regression with partition based dataset [#IGNITE-8176]

Need to have better control on the size of SQL on heap cache [#IGNITE-8139]

Web console: A little re-design of Checkpointing section [#IGNITE-8132]

Add fail recovery mechanism to tracking pages [#IGNITE-8114]

Ability to terminate system workers by JMX for test purposes [#IGNITE-8101]

Allow changing of tx rollback timeout on exchange in runtime [#IGNITE-8074]

Add tests for failure handlers [#IGNITE-8071]

IgniteOutOfMemoryException should be handled accordingly to provided failure handler [#IGNITE-8069]

Add ability to properly wait for transaction finish in case of PRIMARY_SYNC cache mode [#IGNITE-8062]

Integrate decision tree with partition based dataset [#IGNITE-8059]

Clear error message needed when using a non-existing column name for CREATE TABLE primary key [#IGNITE-8052]

Avoid unnecessary unswap(...) call in GridCacheMapEntry.initialValue(...) [#IGNITE-8018]

Disable WAL during initial preloading [#IGNITE-8017]

Web console: Add a link to the SQL documentation in Notebook title [#IGNITE-8008]

Web console: move cluster configuration form templates [#IGNITE-7996]

Integrate MLP with partition based dataset [#IGNITE-7990]

Wrong arguments for ConcurrentHashMap [#IGNITE-7931]

Add transaction debugging support in JMX [#IGNITE-7915]

Add transaction debugging support in control.sh [#IGNITE-7914]

Improve transaction debugging support [#IGNITE-7910]

Java code examples are needed for Spark Data Frames [#IGNITE-7909]

Add example of using LSQR with data normalization [#IGNITE-7897]

Web console: revise unit tests [#IGNITE-7895]

Web console: extract new design collapsible panels into component [#IGNITE-7894]

ODBC: Add support for SQL_ATTR_LOGIN_TIMEOUT [#IGNITE-7888]

Improve code style in GA part [#IGNITE-7877]

Implement 2-phase waiting for partition release [#IGNITE-7871]

Dynamic start cache by stored cache data [#IGNITE-7869]

Need to provide method WAL manager for return serialize version [#IGNITE-7865]

Control utility: Add confirm on dangerous operations [#IGNITE-7864]

Update flatten-plugin to 1.0.1 version [#IGNITE-7862]

Throw Exceptions instead of AssertionErrors when reading from corrupted persistence [#IGNITE-7831]

Unify and improve Apache Ignite and Web Console Dockerfiles [#IGNITE-7821]

Web console: replace Roboto slab font with Roboto regular [#IGNITE-7810]

WAL manage getAndReserveWalFiles should not throw exception if segments not found [#IGNITE-7747]

Add usage of Roles for Web Console E2E tests [#IGNITE-7729]

REST: expand parameters list of GetOrCreateCache command [#IGNITE-7725]

IgnitePdsCheckpointSimulationWithRealCpDisabledTest generates too many strings [#IGNITE-7722]

Include cache name to rebalancing finish message [#IGNITE-7713]

Add an ability to globally enable 'lazy' flag for SQL queries [#IGNITE-7712]

Adopt KNN classification to the new Dataset from dataset package [#IGNITE-7702]

BinaryMetadata exchange should not be triggered if metadata was not updated [#IGNITE-7699]

Page read during replacement should be outside of segment write lock [#IGNITE-7698]

New node joining via ZookeeperDiscoverySpi should print out its ZooKeeper sessionId [#IGNITE-7693]

Add affinity version to snapshot plugin stub [#IGNITE-7676]

ContinuousQueryWithTransformer example [#IGNITE-7652]

Web Console: Rework signin page [#IGNITE-7650]

Unify code in test which clean up persistence directories [#IGNITE-7626]

Web Console: Refactor mongo models creation [#IGNITE-7612]

Web Console: Refactor profile page to component [#IGNITE-7610]

Write removed dirty page during replacement without holding segment write lock [#IGNITE-7606]

SQL COPY: add code examples [#IGNITE-7586]

Product version increment script [#IGNITE-7568]

SQL COPY command: implement encoding option [#IGNITE-7535]

Throttle writting threads according fsync progress and checkpoint writting speed instead of region fill [#IGNITE-7533]

Add new assertions to FilePageStore and provide page content if read page is broken [#IGNITE-7521]

add support for authentication parameters to control.sh utility [#IGNITE-7485]

Improve VerifyBackupPartitionsTask to calculate partition hashes in multiple threads [#IGNITE-7475]

Verify partition update counters and sizes on partition map exchange [#IGNITE-7467]

Username/password authentication for thin clients [#IGNITE-7436]

Enable Ignite cache events per cache [#IGNITE-7346]

Introduce DEV_ONLY marker to IgniteLogger [#IGNITE-7284]

No need to initialize BinaryMetadata working directory for client mode [#IGNITE-7252]

JDBC: support FQDN to multiple IPs during connection establishment [#IGNITE-7192]

JDBC: set socket buffer to 64k by default [#IGNITE-7191]

ODBC: add possibility to configure more than one address in connection string [#IGNITE-7180]

Web Agent: Implement support for comma-delimited list of node URIs [#IGNITE-7119]

Web console: implement mechanism to manage E2E tests environment [#IGNITE-7064]

Add an ability to provide multiple connection addresses for thin JDBC driver [#IGNITE-7029]

Change Ignite Logger configuration on the fly [#IGNITE-6946]

Thread Starvation monitoring [#IGNITE-6940]

OOM should be covered by failure handling [#IGNITE-6892]

Proper behavior on Persistence errors [#IGNITE-6891]

General way for handling Ignite failures (NodeInvalidator should be replaced with IgniteFailureProcessor) [#IGNITE-6890]

Stop all nodes after test by default [#IGNITE-6842]

Configurable rollback for long running transactions before partition exchange [#IGNITE-6827]

Webconsole: Upgrade to Webpack 4 [#IGNITE-6816]

Release process automation [#IGNITE-6763]

JDBC thin: support SSL connection to Ignite node [#IGNITE-6625]

Review default JVM options for better performance [#IGNITE-6521]

Remove redundant parameter of GridFutureAdapter::unregisterWaiter() [#IGNITE-6186]

Logging Ignite configuration at startup [#IGNITE-5798]

Web Console: Create New Design Concept [#IGNITE-4773]

Fixed

ML module compilation failure under JDK 9 [#IGNITE-8452]

Sql query may see intermediate results of topology changes and perform mapping incorrectly [#IGNITE-8405]

ODBC: Can not establish SSL connection to remote host [#IGNITE-8394]

WAL historical rebalance is not able to process cache.remove() updates [#IGNITE-8390]

Deadlock in IgnitePdsAtomicCacheRebalancingTest [#IGNITE-8358]

NPE on start node with enabled authentication [#IGNITE-8355]

Memory leaks on restart Ignite node with enabled persistence at ThreadLocal [#IGNITE-8347]

Compressor thread may miss notification on stop [#IGNITE-8325]

Ignite Cache Restarts 1 suite hangs with assertion error [#IGNITE-8324]

Page corruption during the rebalancing cache [#IGNITE-8320]

.NET: CacheAbstractTransactionalTest.TestTxRollbackOnly failure [#IGNITE-8312]

Web Console: NullPointException in agent in case of self-signed certificates [#IGNITE-8248]

SQL: CREATE TABLE doesn't take backups from template [#IGNITE-8230]

Example DbH2ServerStartup produces a huge annoying output [#IGNITE-8217]

Recovery after crash sometimes leads to starting from beginning absolute wal segment index [#IGNITE-8167]

Nodes fail to connect each other when SSL is enabled [#IGNITE-8153]

Partition state restored from WAL may be lost if no checkpoints are done [#IGNITE-8122]

DirectIO artifact is not published in the maven repository [#IGNITE-8112]

GridCacheWriteBehindStore.Flusher thread uses the wrong transformation from milliseconds to nanoseconds [#IGNITE-8110]

Partition update counter can be incremented twice in case of transaction commit and near cache enabled [#IGNITE-8080]

Reset wal segment idx [#IGNITE-8066]

Exception during checkpoint concurrent changes in topology [#IGNITE-8053]

Limit the number of operation cycles in B+Tree [#IGNITE-8049]

Dynamic indexes are not stored to cache data on node join [#IGNITE-8048]

IgniteQueryGenerator.getOptions() method should properly handle empty list of parameters [#IGNITE-8044]

Result of GridTestUtils.runMultiThreadedAsync has a bug in cancel() implementation [#IGNITE-8025]

Implicit transactions may not finish properly on unstable topology [#IGNITE-8000]

Affinity may be not calculated properly in case of merged exchanges with client nodes [#IGNITE-7987]

[Test failed] IgnitePersistentStoreCacheGroupsTest.testClusterRestartCachesWithH2Indexes fails on TC [#IGNITE-7976]

Flaky failure of IgniteTopologyValidatorGridSplitCacheTest.testTopologyValidatorWithCacheGroup [#IGNITE-7969]

Read a key from metastorage may hang on node restart [#IGNITE-7964]

More cases of suppressed exceptions in IsolatedUpdater [#IGNITE-7962]

Not all OWNING partitions saved in PartitionAllocationMap during checkpoint [#IGNITE-7947]

IgniteCacheClientQueryReplicatedNodeRestartSelfTest#testRestarts can hang on TC [#IGNITE-7946]

Disconnected client node tries to send JOB_CANCEL message [#IGNITE-7944]

Handle IO errors while checkpointing, rework [#IGNITE-7919]

ComputeTaskFuture.get() throws incorrect exception if ComputeTask.result() throws IgniteException [#IGNITE-7904]

IgniteCachePartitionLossPolicySelfTest is flaky on TC [#IGNITE-7898]

Node start with corrupted pds hangs indefinitely [#IGNITE-7890]

Enum values not shown correctly in Webconsole [#IGNITE-7880]

SQL query with group by and distinct in subquery produces JdbcSQLException [#IGNITE-7879]

.NET: linq query throws "Hexadecimal string with odd number of characters" exception [#IGNITE-7851]

ExchangeFuture misses onBaselineTopologyChanged callback when forceReassignment is false [#IGNITE-7836]

Marshaller mappings are not saved to disk on joining nodes [#IGNITE-7794]

Changing baseline topology on cluster may have error in control.sh utility [#IGNITE-7786]

Missing Google Cloud libraries at binary release [#IGNITE-7774]

walSegmentSize can be negative in config [#IGNITE-7765]

Streamer fails if IgniteUuid is indexed [#IGNITE-7756]

Potentially crash during write cp-***-start.bin can lead to the impossibility of recovering [#IGNITE-7755]

Pages Write Throttle mode doesn't protect from checkpoint buffer overflow [#IGNITE-7751]

IgniteRDDSpec. Failing tests [#IGNITE-7727]

C++: LocalSize cache functions [#IGNITE-7682]

AssertionError/NPE on "CREATE SCHEMA" [#IGNITE-7663]

transformer instance is not properly transfers into ScanQuery request [#IGNITE-7649]

NullPointerException in publicApiActiveState [#IGNITE-7639]

NPE in IgniteCacheOffheapManagerImpl$CacheDataStoreImpl.updateIgfsMetrics() [#IGNITE-7632]

SqlQuery hangs indefinitely with additional not registered in baseline node [#IGNITE-7628]

GridClusterStateProcessor#publicApiActiveState returns false when baseline change is in progress [#IGNITE-7577]

IgniteOfflineBaselineNodeFullApiSelfTest.testSize fails [#IGNITE-7573]

Index rebuild future is created too late [#IGNITE-7569]

Sequential checkpoints cause overwrite of already cleaned & freed offheap page [#IGNITE-7540]

Affinity assignment isn't recalculated if PRIMARY node isn't OWNER [#IGNITE-7514]

Weird FillFactor metric fluctuation [#IGNITE-7489]

Queries: transformers are not executed when paging the result [#IGNITE-7483]

InstanceNotFoundException appears in the log when a cache is closed and MBeans are disabled [#IGNITE-7398]

Perform right cleanup for MLPGroupUpdateTrainer [#IGNITE-7375]

ODBC: Third party libraries truncate any inserted varlen data to ColumnSize [#IGNITE-7362]

DatabaseMetaData getTables error [#IGNITE-7335]

AssertionError Entry cannot become obsolete when DHT local candidate is added in deadlock detection suite [#IGNITE-7324]

Daemon-mode Ignite node should not open&listen client port (10800) [#IGNITE-7263]

IgniteReflectionFactory does not handle primitive data types [#IGNITE-7193]

Semaphore Stuck when no acquirers to assign permit [#IGNITE-7090]

NPE when WAL path and WAL archive path are the same [#IGNITE-7074]

Test written in scala doesn't executed on TC [#IGNITE-7042]

Cluster can not survive after IgniteOOM [#IGNITE-7019]

Activation on bigger topology with enabled persistence doesn't work as expected [#IGNITE-6998]

Web console: Prepare Web Console package with simple deploy [#IGNITE-6920]

Lack of context information upon serializing and marshalling (writeObject and writeFields) [#IGNITE-6860]

Cassandra cache store does not clean prepared statements cache when remove old cassandra session [#IGNITE-6853]

"Unexpected exception during cache update" via NullPointerException thrown using TouchedExpiryPolicy [#IGNITE-6815]

Select query on a client with unused field throws an exception [#IGNITE-6692]

BinaryObjectImpl.writeFieldByOrder method does not support TIME [#IGNITE-6666]

Incorrect time units of average transaction commit/rollback duration cache metrics [#IGNITE-6630]

Test IgniteTxRemoveTimeoutObjectsTest has flaky fails [#IGNITE-6557]

Partition eviction prevents exchange from completion [#IGNITE-6113]

Web console: Implement persistent store in demo mode [#IGNITE-6094]

Null value have appear in the entry processor, but the entry is existing [#IGNITE-6083]

[Test failed] GridCachePartitionedDataStructuresFailoverSelfTest.testSemaphoreNonFailoverSafe [#IGNITE-6005]

[Test Failed] IgnitePartitionedCountDownLatchSelfTest.testLatchMultinode1 [#IGNITE-5978]

[Test Failed] GridCachePartitionedDataStructuresFailoverSelfTest.testSemaphoreSingleNodeFailure [#IGNITE-5975]

Ignite Continuous Query (Queries 3): IgniteCacheContinuousQueryClientReconnectTest fails [#IGNITE-5955]

Ignite WebSessions: Flaky failure for WebSessionSelfTest.testClientReconnectRequest() and subclasses [#IGNITE-5946]

Method stopGrid(name) doesn't work in multiJvm mode [#IGNITE-5910]

ScanQuery transformer applies to first results page only [#IGNITE-5804]

Incorrect mapping of smallint cassandra type [#IGNITE-5779]

Failures in GridTcpCommunicationSpiRecoverySelfTest [#IGNITE-5504]

Replicated cache reads load balancing [#IGNITE-5357]

IgniteRDD data frame does not handle object fields [#IGNITE-3077]

Cache instance is closed when client disconnects [#IGNITE-2766]

NPE at RendezvousAffinityFunction [#IGNITE-2539]

.NET: Features and Improvements

.NET: DML update via LINQ [#IGNITE-5298]

.NET: Add getRebalanceClearingPartitionsLeft JMX metric [#IGNITE-7773]

.NET: Build scripts: stop build when one of the steps fails [#IGNITE-7566]

.NET: Service proxy as dynamic object [#IGNITE-7561]

.NET: Thin client: SSL [#IGNITE-7329]

.NET: QueryMetrics [#IGNITE-6681]

.NET: Configure SSL without Spring [#IGNITE-3111]

Apache IGNITE 2.5

Features and Improvements

Fixed

.NET: Features and Improvements

.NET: Fixed