Distributable J2EE Web Applications
A Container Provider's View of the current Servlet Specification.

Prerequisites.

The Servlet Specification - distilled:

For a successful outcome the implementors of both Container and Containee need to be agreed on exactly what behaviour is expected of each other. For a really deep understanding of the contract they will need to know why it is as it is (TODO - This paper will provide such a view, from both sides).

The Specification mandates the following behaviour for distributable Servlets:

Non-Distributable Servlets

Single Threaded Servlets

Multi-Threaded Servlets

Distributable State

HttpSession Migration

In a J2EE (Version 1.4) environment (e.g. in a web container embedded in an application server), the set of supported types for HttpSession attributes is as follows, although web containers are free to extend this set (J2EE.6.4): (Note that using an extended type would impact your webapp's portability).

• java.io.Serializable
• javax.ejb.EJBObject,
• javax.ejb.EJBHome
• javax.ejb.EJBLocalObject
• javax.ejb.EJBLocalHome
• javax.transaction.UserTransaction (TODO ??)
• "a javax.naming.Context object for the java:comp/env context" (TODO)

Breaking this contract through use of an unagreed type will result in the container throwing an IllegalArgumentException upon its introduction to the HttpSession, since the container must maintain the migratability of this resource (SRV.7.7.2).

Migration Implementation

HttpSessionActivationListener

HttpSession Affinity

• Multiple instances of a distributable webapp will be running in multiple different JVMs within our proposed cluster
• A client browser may throw multiple concurrent requests for the same session at this cluster
• The spirit of the specification and performance requirements call for such a grouping of requests to be processed concurrently, rather than serially,

The Servlet Specification states:

"All requests that are part of a session must be handled by one Java Virtual Machine (JVM) at a time." (SRV.7.7.2).

The intention of this statement is to resolve such concurrency issues. It prunes the tree of possible implementations substantially, insisting that all concurrent requests for a particular session are delivered to the same node.

Delivering requests for the same session to the same node is known variously as 'session affinity', 'sticky sessions', persistent sessions' etc., depending on your container's vendor. The specification is trading complexity in the web-container tier for complexity in the load-balancer tier. This added requirement will impact the latency of this tier, in that the load-balancer will generally need to parse the uri or headers of each http request travelling through it (in a non-encrypted form) in order to extract the target session id. However, the reduction of potentially awkward concurrency issues/race conditions in the web-container tier is a gain considered worth this sacrifice.

It is worth noting that, since we have now introduced a requirement for the load-balancer tier to have knowledge of the location of httpsessions within the web-container tier, the ability to 'migrate' these objects may, therefore, require a certain amount of coordination between the two tiers.

Background Threads

The previous requirement reduces our problem from race conditions between distributed objects in different JVMs, to a situation where we simply have to manage coordination between multiple threads in the same JVM. The purpose of this coordination is to ensure that access to container managed resources that are available to multiple concurrent application space threads is properly synchronised.

Whilst the container has implicit knowledge about any thread, executing application code, for the lifecycle of which it is responsible (i.e. request threads), it has no control over any thread that is entirely managed by application code - Background thread. Such threads might execute across request boundaries, accessing otherwise predictably dormant resources that might otherwise be passivated or migrated elsewhere.

Fortunately, the specification also recommends that references to container-managed objects should not be given to threads that have been created by an application (SRV.2.3.3.3, SRV.S.17) and whose lifecycle is not entirely bounded by that of a request thread. The container is encouraged to generate warnings if this should occur. Application developers should understand that recommendations such as this become all the more important when working in a distributed environment.

We shall take "container-managed objects" to include any object that has been placed into an httpsession. By virtue of this placement, its lifecycle is now the responsibility of the encompassing session and ultimately therefore of the container. This is a useful constraint, since the container-provider may now prove that, provided that there are no request threads active for a session within the container, entirely thread-safe access may be made not only to an httpsession but also to its attributes, although their locking scheme, being application space components, is completely unknown to the container-provider.

HttpSession Events

Finally, given that HttpSessions are the only type to be distributed and that they should only ever be in one JVM at one time, it should come as no surprise that ServletContext and HttpSession events are not propagated outside the JVM in which they were raised (SRV.10.7) as this would result in container owned objects becoming active in a JVM through which no relevant request thread was passing.

Is this adequate ?

The specification has done a reasonably good job of outlining our area of interest. Before implementing a container, however, there are a number of issues that we still need to address.

Catastrophic failure

TODO - Looking at what this specification actually says about distributable webapps, it can be seen immediately that it seems to reliably outline a mechanism for the controlled shutdown of a node and the attendant migration of it's sessions to [an]other node[s], or persistant storage.

The ability to migrate sessions on controlled shutdown is useful functionality (maintenance will be one of the main reasons behind the occurrence of session migration), but it does not go far enough for many enterprise-level users, who require a solution capable of transparent recovery, without data loss, even in the case of a node's catastrophic failure. If a node is simply switched off, thus having no chance to perform a shutdown sequence, then volatile state will simply be lost. It is too late to call HttpSessionActivationListener.willPassivate() where necessary and serialise all user state to a safe place! Container implementors must ask themselves the question - 'What, within the bounds of the current specification, can we do to mitigate this event?'.

Session Backup - When

The answer to the concern of lost data is to frequently ship backup copies off-node, so that in the case of its catastrophic failure, we have a fallback position. The freshness of our backup data depends directly on the frequency of this process. This frequency is bounded by resource concerns and the contract between container and containee, as discussed above.

Let us examine some of the possibilities:

• Immediate - As soon as a change is made to a session, it is backed up.
  • Most Accurate - This policy constrains our window of data-loss as much as is reasonably possible.
  • Most Expensive - This accuracy has a cost. Every write to a session object will result in expensive back up code being triggered.
  • Synchronisation Issues/Ref vs Val semantics - TODO
• Request - All changes to a session are backed up at the end of each relevant request.
  • Less Accurate - This is less accurate than the 'Immediate' policy described above, since a failure halfway through a request thread would result in the loss of all changes that it had made to it's session.
  • Less Expensive - Since backups are only done at the end of each request, they will be fewer than 'Immediate' mode, resulting in much less expense.
  • Inconsistency - Assuming that the session is somehow in a 'consistent' state at the end of each request is misguided, since multiple requests may overlap. Although, it may be, with the benefit of knowledge of the application at one's disposal, that this problem can be safely discounted.
  • Synchronisation Issues/Ref vs Val semantics - TODO - This policy suffers from the same issues, in this respect as 'Immediate'.
• Request Group - All changes to a session are backed up as soon as all associated active requests have been processed.
  • Less Accurate - This policy will be less accurate still, if requests for the same session overlap within the container.
  • Less Expensive - For the above reason this will lead to it being still less expensive.
  • Consistency - This policy guarantees that what it backs up is a consistent view of the session's contents. No request is only half processed.
  • Thread-Safe - The real win for this policy is that, with no contract between containee and container, other than that described in the Servlet Specification, the container may access session attributes for backup, in the knowledge that no other application code is concurrently modifying them.
• WebApplication - Backup occurs in line with web application lifecycle. i.e. Not until the web application is stop()-ed by it's container.
  • This policy gives no protection against catastrophic failure, but is fine for maintenance-only scenarios.
  • There is no associated runtime overhead.
  • There are no consistency or synchronisation issues. All request and background threads will have terminated before the session is backed up.
• Timed - 'dirty' sessions are backed up at regular intervals, orthogonal to the lifecycles of requests, web applications etc...
  • This might conceivably be useful to overlay on top of e.g. the 'Request' policy if your request threads took a long time to run, or the 'RequestGroup' policy if you expected long periods without backing up because of many overlapping requests for he same session.
  • A backup policy that worked in the manner would be of conveniently tunable accuracy and impact.
  • However such a policy would suffer from all the consistency and synchronisation issues common to 'Immediate' and 'Request' approaches.

REFACTORING HAS GOT TO HERE...

Session Backup - At what granularity ?

• whole session
  • synchronisation issues - must lock more objects (TODO)
• single attribute
  • more complex
  • less contention
  • Object Identity issues - TODO - same object in different attributes not preserved ?
• batched attributes
  • even more complex
  • multiple changes to same attribute may be collapsed
  • Object Identity - as above - TODO

(TODO - requests do not have transactional semantics)

(TODO - if a single request reset an attribute a number of times, immediate xfer would be expensive, batching would also be expensive since each reset would involve a serialisation of which only the last would be useful (or can we leave this til the last moment?))

What can we do?

• whatever we do it must not break spec compliance/portability (so we cannot e.g. extend APIs).
• insist on the no background thread rule - unless distribution is only done on webapp.stop()
• agree an explicit session attribute synchronisation contract - synchronized(Object){...} - why implicit rule is not enough.

Reference vs Value Based Semantics

It is useful to introduce the distinction between reference and value based semantics at this point.

Given the following Servlet code snippet:

    Foo foo1=new Foo();
    session.setAttribute("foo", foo1);
    Foo foo2=session.getAttribute("foo");
    

Which of these assertions (assuming that Foo.equals() is well implemented) would you expect to be true?

•     foo1==foo2;
  	

•     foo1.equals(foo2);
  	

If you expect foo1==foo2 then you are expecting reference-based semantics.

If you are expecting reference-based semantics you might well write code such as this in order to avoid unnecessary de/rehashes:

    Point p=new Point(0,0);
    session.setAttribute("point", p);
    p.setX(100);
    p.setY(100);
    

and then might expect that :

    ((Point)session.getAttribute("point")).getX()==100;
    

Using value based-semantics, out of these three (TODO) assertions, only the second of the equality tests would succeed.

Every parameter passed to and from a value based API must be assumed to be copied from an original, since it may have come across the wire from another address space.

For this reason, when you start dealing with (possibly) remote objects in a distributed scenario, you generally shift your semantics from reference to value. (c.f. Remote EJB APIs)

Unfortunately, the Servlet Specification, whilst clearly mandating that every session attribute must be of a type that the container knows how to move from VM to VM omits to mention that a possible impact of doing this is an important shift in semantics. This is exacerbated by the fact that, unlike EJBs, which have been designed specifically for distributed use, the httpsession API does not change (c.f. Local/Remote) according to the semantic that is required, which is simply a single deployment option. This encourages developers to believe that they can make a webapp that has been written for Local use, into a fully functional distributed component, simply by adding the relevant tag to the web.xml. All attendant problems are delegated, by spec and developer, to the unfortunate container provider.

Thus the container provider must make a choice here

• continue to support reference-based semantics in which case migration may only occur when there are no active threads for a session and there is an implicit contract between container and containee that objects deriving from a session will not have their references compared to objects deriving from elsewhere, whose lifecycles may span across such periods.
• make an explicit new contract that states that all interaction with session attributes is by value and comparisons should only be made in this way. The full ramifications of this choice should become apparent as we progress further in this paper.

Object Identity, Object Streams and Synchronisation

TODO - I guess Object Identity can only be preserved within a single Object tree ? so attribute-based distribution will not recognise the same object shared between different attributes How can we guarantee, unless we know that no other threads are running, the synchronisation of values as we stream them out of the container ?

Unfortunately, the specification requires that every session object carries a 'LastAccessedTime' value. Which is updated every time the session is retrieved by an application thread for reading or writing. Thus any request requiring stateful interaction within the webapp will have the side effect of writing a change to the session. Taken literally these changes can be very expensive in a distributable scenario as a naive implementation will require each such change to be exported to another vm in case of catastrophic node failure.

DISCUSS

GC GRANULARITY ETC.

The spec has one last curve ball for us to face.

HttpSessionActivationListener:

If a session attribute implements the HttpSessionActivationListener interface, the spec requires the container to call it's willPassivate() method just before and it's didActivate() method just after 'migration'. Since we are no longer implementing straightforward migration from one node to another, but rather some form of multi-copy synchronisation protocol, it is, at first hard to see how these two different paradigms can be mapped to a single model that resolves the problem of when notification should take place.

How a container provider plays this ball really depends upon the perceived intention of these notifications. The spec implies that they are there so that an attribute which contains expensive non-serializable resources has a chance to release and reacquire them at opportune moments - basically a lifecycle for such attributes. (TODO - confirm)

The implication of this is that willPassivate() must always be called before the attribute is serialised (i.e. shipped off-node), however, this now leaves said attribute in a passivated state, unsuitable for continued use within the session, so before control flow is returned to the webapp, didActivate() must also be called to put this attribute back into normal service.

In effect, each mutation can be seen as the mini-migration of a single attribute off and back onto the same node - whilst the container retains a copy of its serialised form to ship off-node as an emergency backup.

Concurrency

Concurrency is a major issue. It can be divided into two smaller problems:

Concurrency between threads in different processes.

Concurrency between threads in the same process.

If you take the decision that your design will allow concurrent requests within the same session in different vms, you will need a strategy for ensuring that all vms have a consistent view of each session.

This can be achieved by making the session a single remote object, which all nodes make use of. Since there is only one copy of the session there are no consistency issues, provided that it has sufficient synchronisation within itself. However, since the session is a remote object it will have value-based semantics. If you get the same attribute value from it twice, unless you have a caching layer, they will be 'equal' but not '='. If you have a caching layer you will then have to concern yourself with the complexities of ensuring that items in your cache are invalidated on time, which just brings you full circle back to the consistency problem. I call this solution 'shared store'. Finally, since all interactions are with a remote object, this solution tends to be slow.

Alternatively, your design might choose to have multiple objects (one in each address space) all representing the same session. As one is changed it notifies the others of the change, so that they can apply it to themselves, so that all these objects maintain a state consistent with each other. This is generally know as '[in-vm] replication'. Implementors of this design need to consider the following issues.

1. Race conditions between concurrent changes to the same session occurring on different nodes. (TODO - spec avoids this issue).

CONSIDER THAT WHEN SPEC SAYS 'AT THE SAME TIME' IT IS TALKING IN TERMS OF REQUEST LIFETIME. - I.E. AFFINITY (AT LEAST FOR OVERLAPPING REQUEST GROUPS) IS MANDATORY

2. If upon change, you replicate more than just that change (i.e. each instance of a session has it's state completely replaced, rather than just the attribute which changed on some remote node), you will find that your session objects have inconsistent semantics, since sometimes when you get an attribute from a session it's reference will be the same as the last time, sometimes it won't, although your application may never actually change this attribute - since change to another attribute may have caused the entire session to have been replaced with a fresh copy from across the wire.

This first issue can be entirely avoided through the use of 'affinity' or 'sticky' or 'persistent' sessions. Nomenclature depends on your vendor. All amount to basically the same thing. A load-balancer that supports this feature will ensure, preferably by tracking the presence of the JSESSIONID cookie and jessionid path parameter, that all requests pertaining to the same session will be delivered to the same host:port combination. Thus, we can see that there never will be concurrent threads contending for the same session in different JVMs/WebContainers since all relevant requests will be routed to the same one. Affinity has one further important benefit. Many webapps may be deployed in complex environments where a lot of transparent caching occurs below them. Without affinity, requests will be delivered to a number of different nodes, all of which will have to populate such caches with objects that will only be reused if another request for the same webapp/session is directed to them. With affinity, requests will always be processed on the same node, so the cache is only populated once and then subsequently reused. This detail will increase cache hits and may have a dramatic effect on performance and resource consumption.

TODO - WHAT IS HAPPENING HERE ??

The second issue can be resolved with standard Java(tm) synchronisation primitives or libraries, provided that all code involved is in container-space. Problems arise when webapp code calls container code and vice versa. This is exacerbated by the spec's insistence that an HttpSession should allow access by multiple concurrent threads and the fact that the webapp may still be holding and modifying references to attribute values. Ultimately the container provider will have to specify some contract which he expects the webapp to abide by. A sensible one might be that any thread mutating an attribute should take it's Object-level lock for the duration, so that behind the scenes reads, such as those involved in serialisation etc can synchronise on the same lock and be assured of a consistent view of the object. (CHECK TO SEE WHETHER default read/writeObject are synchronised). This however may be seen as burdensome by the webapp developer who may have their own locking strategy for an attribute type which is compromised by this contract...

DISCUSS PROS AND CONS OF SHARED-STORE VS REPLICATION

other items...

does anything else other than session need to be distributed ?

- security info

- application level data (as opposed to user level)

- etc

store and replication mechanisms... - going to far.

other thoughts ?

reference semantics sacrificed if session is temporarily passiviated - although hopefully no-one (what about background thread?) is holding a reference to us...

replication with affinity and change-by-delta best solution because it preserves reference-semantics as far as possible - consider...

replication is faster than shared-store because 'getAttribute' is not a remote call. Effectively, with replication, each replicant IS a shared store which processes requests locally.

TODO - Survey existing impls:

• TC 4.x and 5.x
• Jetty
• JBoss
• Apache/mod_jk
• simple TC recommended 'C' lb
• mod_proxy solution
• mod_backhand
TODO additional requirements when operating in a J2ee environment

Further Reading:

• SRV.7 Sessions
• SRV.7.6 Last Accessed Times
• SRV.15.1.7 HttpSession
• SRV.15.1.8 HttpSessionActivationListener
• SRV.15.1.9 HttpSessionAttributeListener
• SRV.15.1.10 HttpSessionBindingEvent
• SRV.15.1.11 HttpSessionBindingListener
• SRV.15.1.13 HttpSessionEvent
• SRV.15.1.14 HttpSessionListener
• TODO: provide URL for latest servlet spec.
• TODO: AND J2EE spec

Optimisations

• DefaultServlet is stateless (session will never be fetched) so can be excluded from the equation. This means that requests for images etc can be excluded from concurrent request groups, reducing them substantially. A smart lb would know about this and could relax affinity as well (although a caching tier might be serving this content before you hit the lb - this tier also needs coordination with web-container teir so it can be selectively flushed upon webapp redeployment).
• lastAccessedTime - don't distribute, ignore until last minute ?

  If a node dies.all sessions upon it must be considered just touched, since a request thread for them may have caused the crash. Time of node death should be noted and associated with these sessions. upon rehydration this is the lastAcessed value that they should adopt.

• Batching of deltas

Further Issues

• HtpSessionActivationListener - TODO - WHAT ?
• ClassLoading

Further Notes

TODO - Look into Geronimo impl... SRV.10.6 Listener Exceptions TODO - can all my SRV refs be links ? TODO - we can use a SecurityManager to prevent background threads being created. We can prevent access from such a thread to a container managed object, but we can't prevent such a reference being hld by such a thread... NB