Apache Sandesha2 Architecture Guide

Introduction

Sandesha2 gives reliable messaging capabilities to Axis2. From the point of view of the Axis2 engine, Sandesha2 is a module. When this module is engaged to a service, clients have the option of invoking it in a reliable manner. In the client side Sandesha2 module can be used to interact with existing reliable Web services.

According to the Web service-ReliableMessaging (WS-RM) specification which is implemented by Sandesha2, reliable communication happens between two endpoints. These endpoints are called the RM Source (RMS) and the RM Destination (RMD). Before communication, RMS and RMD perform a message exchange to create a relationship called a Sequence between them. A sequence is always identified by a unique Sequence Identifier.

Each message of a sequence is numbered, starting from one. In Sandesha2 the maximum number of messages a sequence can support is 2 64 (size of long data type). Of course practically this may be limited by the memory available for your system . The message number is used by the destination to support additional delivery assurances. This will be explained later in this tutorial.

The reliability is obtained basically using acknowledgements. RMS is required to send each message one or more times to the RMD. RMD sends back acknowledgements to notify the successful reception of messages. After receiving an acknowledgement for a certain message RMS can stop the retransmission of that message.

When all messages of a certain sequence have been successfully transmitted to RMD, RMS sends a TerminateSequence message. If RMD receives this message it can free any resources allocated for this sequence. Otherwise resource de-allocation will happen based on a timeout.

Following diagram explains operation of the RMS and the RMD.

WS-RM Model

Sandesha2 supports two reliable messaging specifications. It fully supports WS-ReliableMessaging February 2005 specification which was created by collaborative efforts of several companies. Later this specification was submitted to OASIS and currently being standardized under the OASIS WS-RX technical committee. Sandesha2 supports up to the revision CD 3 of the specification being developed under this technical committee.

Architecture

Architecture

Sandesha2 components are used in a completely symmetric manner, in the server side and client as shown in the diagram above. Lets just consider a single side for this discussion.

Handlers

Sandesha2 adds three handlers to the execution chain of Axis2. Two of these handlers are added to a special user phase called 'RMPhase' of in and out flows. The other handler is added to the predispatch phase of the inFlow. These handlers and their functions are given below.

Storage

SandeshaInHandler

This is added to the RMPhase of the inFlow. Since RMPhase is a user phase, this handler will only be invoked for messages that are aimed at RM enabled service. This handler will identify the type of this message. The type can be an application message (a message that has to be delivered to the service) or a RM control message. Sandesha2 has a special set of classes called message processors which are capable of processing each type of message. Depending on the type, the message is send through the 'processInMessage' method of the message processor which will do the further processing of it.

SandeshaOutHandler

This handler is responsible for doing the basic outFlow processing. This will first generate an ID called the Internal Sequence ID which is used to identify the sequence this message should belongs to. All the messages having the same Internal Sequence ID will be sent within a single sequence. An Internal Sequence ID will have a corresponding Sequence ID which would be obtained after the Create Sequence message exchange. In the client side the Internal Sequence ID is the combination of the wsa:To address and a special value given by the client called Sequence Key. In the server side the Internal Sequence ID is a derivation of the Sequence ID value of the messages of the incoming sequence.

Before sending the message through other handlers the SandeshaOutHandler will send it through the 'processOutMessage' method of the respective message processor.

SandeshaGlobalInHandler

This handler is added to the predispatch phase of the inFlow. Since this is a global phase, this handler will be called for each and every message that comes to the Axis2 system. To maximize performance, the very first function of this handler is to identify whether the current message can be processed by it. It checks whether the message is intended for a RM enabled service, and if so, check the message type to further verify whether it should be processed globally. This handler was placed to perform functions that should be done before the instance dispatching level of Axis2.

Some of these functions are given below:

  • Detecting duplicate messages and dropping them.
  • Detecting faults that occur due to RM control messages and reporting them.
  • Answering to acknowledgement requests of the dropped application messages.

Sender

Sender is responsible for transmission and retransmission of messages. The Sender is a separate thread that keeps running all the time. At each iteration Sender checks whether there is any messages to be sent. If there is any, it is sent to the destination. Sender also identifies messages that has to be retransmitted and keep re-sending them until a maximum limit decided by Sandesha2 policies is exceeded.

Message Processors

Sandesha2 have a set of classes called message processors, each implementing the MessageProcessor interface. Each message processor is responsible for processing a certain type of message. For example, CreateSequenceProcessor will process CreateSequence messages and AcknowledgementProcessor will process Acknowledgement messages. The message processor interface defines two methods for processing incoming messages and outgoing messages. (namely 'processInMessage' and 'processOutMessage')

InOrderInvoker

InOrderInvoker is another separate thread that is started by the Sandesha2 system. This is started only if Sadesha2 has been configured to support in-order delivery assurance. InOrderInvoker makes sure that it invokes messages of a sequence only in the order of message numbers.

Storage Framework

Sandesha2 storage framework is one of the most important parts of the Sandesha2 system. This was designed to support the RM message exchange while being independent of the storage implementation used. The storage framework defines a set of interfaces and abstract classes that can be implemented by a particular storage implementation. Sandesha2 system comes with an in-memory storage implementation. There can be other implementations based on different databases and persistence mechanisms.

Following diagram gives a brief view of the Sandesha2 storage framework.

Storage

Storage framework defines several beans that extend the RMBean abstract class. They are given below:

  1. RMSBean (fields - internalSequenceID, createSequenceMsgID, sequenceID, createSequenceMsgStoreKey, referenceMsgStoreKey, securityTokenData)
  2. SenderBean (fields - messageContextRefKey, internalSequenceID, messageNumber, messageID, messageType, send, resent, sentCount,timeToSend)
  3. RMDBean (fields - sequenceID, nextMsgToProcess, pollingMode, referenceMessageKey)
  4. InvokerBean (fields - invoked,messageContextRefKey, sequenceID, msgNo)
  5. SequencePropertyBean (fields - sequencePropertyKey, name, value)

There are five bean manager interfaces corresponding to each of above beans.They are as follows:

  1. RMSBeanMgr
  2. InvokerBeanMgr
  3. RMDBeanMgr
  4. SenderBeanMgr
  5. SequencePropertyBeanMgr

Sandesha2 also defines a StorageManager interface that defines methods to create each of these bean managers and to create a Transaction object which should implement the Transaction interface. Transaction interface defines commit and rollback methods.

Collectively each Sandesha2 storage implementation should have following classes:

  1. An implementation of the StorageManager interface.
  2. Implementations of five Bean Manager interfaces.
  3. An implementation of a Transaction interface.

These classes can be packed as a jar archive and added to the classpath. The name of the StorageManager implementation class has to be mentioned in Sandesha2 policy configurations. This will be picked up after a restart of the the Axis2 engine.

Delivery Assurances

Sandesha2 can provide an in-order exactly-once delivery assurance. The ordering (in-order) is optional. You can disable it using Sandesha2 policy configurations. The ordering is done using the InOrderInvoker thread that was introduced earlier.

If ordering (in-order) is enabled, SandeshaInHandler pauses the execution of an incoming application message. As a result of this, the message will not go through rest of the handler chain in the first invocation. Note that it also starts the InOrderInvoker thread if it is stopped. This thread goes through the paused messages and resume each of them in the order of message numbers.

If in-order invocation is not enabled the SandeshaInHandler will not pause the messages and they will go in their full execution path in one go.

The delivery assurance to be used depends on your requirements. If you want the invocation to be as fast as possible, and you do not care about ordering, disable in order invocation. But if you want message to be invoked in the order they were sent by the client, you have to enable it. There could be a considerable performance improvements if this feature is disabled. Specially if majority of the messages come out of order.

In the current implementation, each message (identified by sequenceID and message number) will be invoked only once. So exactly once delivery assurance is guaranteed. You cannot ask Sandesha2 to invoke the same message more than once.

Example Scenario

This part explains how Sandesha2 framework works internally for the most common RM scenario, which is the sending of a couple of Ping messages from a client to the server. We will mainly look at how Sandesha2 uses its storage to do the RM message exchange correctly. While going through the following, keep the RM Beans and their fields which were mentioned earlier, in mind.

Client Side

  • Client does the first fireAndForget method invocation of a serviceClient after setting necessary properties.
  • Message reaches the SandeshaOutHandler which detects it as an application message. The processing is delegated to the processOutMessage method of the Application Message Processor.
  • Application Message Processor generates the Internal Sequence ID as explained earlier. It understands that this is a new sequence and generates a Create Sequence Message for it. The Application Message gets paused.
  • SandeshaOutHandler adds an entry to the CreateSequence bean manager representing the newly created Create Sequence message. This entry has three properties.The sequenceID property is initially null. The createSeqMsgID is the message ID of the created CreateSequence message. The internalSequenceID property gets the generated Internal Sequence ID value.
  • SandeshaOutHandler adds two entries to the SenderBeanManager. One which has the send property to 'false' represents the application message, other which has the send property to 'true' represents the CreateSequence message. The Sender thread sends (and retransmits) only the CreateSequence message.
  • After some time the client side would receive a Create Sequence Response message from the server. The SandeshaInHandler delegates the processing to the CreateSequenceResponse message processor. It finds the correct CreateSequence manager entry using the createSequenceMessageID property (which is in the relatesTo entry of the response message).
  • Client updates the sequenceID property of the CreateSequence bean manager entry. Also the send value of the application message entries are set to 'true'. The sender starts transmitting and retransmitting application messages.
  • When the client receives acknowledgements for the messages it send, they are delivered to the Acknowledgement Processor which removes the corresponding application message entries from the Sender bean manager.
  • If an acknowledgement says that all the sent messages (up to last message) was successfully received, the Acknowledgement Processor creates a Terminate Sequence message and adds a corresponding entry to the Sender bean manager.

Server Side

  • Server receives a CreateSequence message. It generates a new sequence ID and creates a new Create Sequence Response message containing this ID.
  • Server adds an entry to the NextMessage Bean Manager. The initial value for nextMessageToInvoke property is 1.
  • Server adds an entry to the SenderBeanManager (of server side) representing the application message. The send value is'true'. The CreateSequenceResponse message is sent by the Sender.
  • After some time the server receives an application message. The server side SandeshaInHandler delegates this to the ApplicationMessageProcessor which creates an acknowledgement and sends it. If in-order invocation is enabled, an entry is added to the InvokerBeanManager representing this new application message.

    Lets assume that the message number of this message is 2.

  • The InOrderInvoker which keeps looking at the InvokerBeanManager entries sees that there are entries to be invoked.
  • The InOrderInvoker checks the entry of the NextMessageBeanManager of the relevant sequence and sees that it is 1. But since only message number 2 is present in the invokerBeanManager entries, the invocation is not done.
  • After some time, application message 1 also comes. Now the Invoker sees this entry and invokes the message. It also updates the nextMessageToInvoke property of NextMessageBeanManagerto 2. The invoker again checks whether the new entry for the NextMessageToInvoke (2) is present in the InvokerBeanManager entries. Since this is present it is also invoked. The value is again updated (to 3) but no invocation is done since an entry is not found.
  • Some time later the server may receive a TerminateSequence message. It can partly remove the resources allocated for the sequence. The other part of resources (which is required by the InOrderInvoker) is removed after the invocation of the last message.