This page last changed on Mar 04, 2010 by robbie.

Broker Management via QMF

As part of the effort to bring more commonality between the Java and C++ brokers I have been attempting to implement QMF management in the Java Broker.  Currently the Java Broker can process the majority of the QMF Management commands (although it will not accept agents connecting to it).  Ultimatlely the aim must be to allow management and configuration of the brokers using the same set of tools.

In implementing QMF management I have noted a number of questions - some general and related to the nature of QMF, some more specific to the particular set of entities and methods in the management schema for the Qpid (C++) Broker, and some simply relating to the deifnition of individual properties or statistics.  I have also noted where I think additions to the existing schema would be useful.

Currently, even if fully implemengted, management through QMF would not be sufficient to duplicate the existing management functions available in the Java Broker (e.g. through the JMX console).

General Comments on the Schema

  • Why are there no methods to create entities? Although one can use AMQP commands to accomplish this task it means you have to switch between two distinct command sets to manage the broker; it also means you cannot simply piggyback the QMF management over other transports. (further, for AMQP 1-0 we are removing "management" commands from the core protocol)

GS: See QPID-2317; I'd like using this to be as simple as sending a correctly formatted/encoded message to a special address. I believe with QMFv2 that will indeed be the pattern. It is a useful addition as the new APIs in c++ and python aim to be be protocol independent and thus do not expose the 0-10 declare methods.

  • There are no tools for inspecting the contents of messages within the broker. Where a message is stuck on the queue, it is useful to be able to inspect its properties and/or content.

GS: You could do this by browsing the queue (though there is no support then for moving or modifying a message). A scheme that avoids duplicating too much of the standard messaging behaviour would be good.

RG: the problem with that is that it requires you to be on an AMQP 0-10 connection.  I would like it to be that we can conduct QMF over other protocols, namely AMQP0-9 or JMX.  I understand your concern about duplicating functionality... as you already mentioned, moving can't be done easily anyway, but I see this inspection somewhat differently to browsing... and (in particular) without some form of server side selector I don't think the C++ broker could do what I am talking about (pick an arbitrary message from the queue and inspect it).

  • Values with High/Low watermarks... the historic low for these will almost invariably be 0, so what is the point?
  • Absolute times are specified in nanoseconds... this is "unusual" and it's pretty much impossible to get an accurate absolute time in nanoseconds. Milliseconds for absolute time and nanoseconds for deltas would make more sense.
  • The model and methods are not designed to cope with a broker which can support multiple virtual hosts.
  • It would be nice to have the ability to configure dynamic event configuration... i.e. the ability to say "publish an event to this address every time this condition is met". You can build this on top of the statistic updates, but it requires you to have a fairly frequent management publish period and a process actively monitoring the output.

* GS: I'm very much in favour of aligning the two brokers. We would however need to come up with a mechanism for allowing the c++ broker to retain backwards compatibility with a schema as currently defined. E.g. support for multiple schemas?

  • The java broker exposes an interface for conducting live User Management, eg add/delete/view users, set password, set management access rights (read only, read/write, or admin which is read/write plus access to the more sensitive management mbeans such as UserManagement itself). It would be good to expose this type of management via QMF as well.
  • The Java broker exposes an interface for viewing and adjusting its logging levels while running. This too would be useful to expose via QMF.

General comments on QMF

  • have a generic way to be able to create new entities,
  • have a way to modify the mutable properties of existing entities.
  • when sending schema, where a particular value is of a restricted domain (e.g. has min and/or max values; or takes a value from a restricted enumerated set) then that restricted domain is communicated.
  • add ability to "reset" statistic counters... This would actually make low watermark counts useful.
  • add ability to query for objects based on criteria (e.g. get me all exchanges whose vhostRef is X)

Comments on Individual Schema Classes

System

Properties

General: what is this supposed to represent - the machine the broker is running on, or the process that is contains the broker?

Name
Type Description C++
Java
Notes
systemId uuid     Y Is this a qpid specific property?
osName sstr Operating System Name   Y  
nodeName sstr Node Name   Y What exactly is this supposed to be... hostname?
release sstr     Y What exactly is this supposed to be... release of the OS?
version sstr     Y What's the difference between this and release?
machine sstr     Y What is this?

Broker

Properties

Name
Type Description C++
Java
Notes
systemRef objId System ID   Y  
port uint16 TCP Port for AMQP Service   Y A broker may be listening on more than one port - I suggest that we want to have a new entity to represent listening ports/transports
workerThreads uint16 Thread pool size   Y  
maxConns uint16 Maximum allowed connections   N Java broker currently doesn't support a connection limit
connBacklog uint16 Connection backlog limit for listening socket   N What is this? It "defines the maximum length to which the queue of pending connections for sockfd may grow (from man page for the listen() system call"
stagingThreshold uint32 Broker stages messages over this size to disk   N Java Broker does not currently support stagingThe 'staging' functionality is poorly conceived on the c++ broker and needs reviewed; it is really just about handling very large messages by not requiring that the full contents ever be held in memory
mgmtPubInterval uint16 Interval for management broadcasts   Y  
version sstr Running software version   Y  
dataDir sstr Persistent configuration storage location   Y Java Broker currently displays the value of $QPID_HOME here

Statistics

Name
Type Description C++
Java
Notes
uptime deltaTime     Y Why have this as a statistic that is going to constantly change? start time as a property would do equally well
Sggested Additions
  • number of current / high watermark of connections

Methods

Signature Description C++
Java
Notes
( uint32 sequence, lstr body ) echo ( uint32 sequence, lstr body) Request a response to test the path to the management broker   Y  
void connect ( sstr host, uint32 port, bool durable, sstr authMechanism, sstr username, sstr password, sstr transport) Establish a connection to another broker   Y Doesn't allow a vhost to be specified, nor is transport defined (is ssl a transport?) yes, ssl would be a transport as would e.g. rdma. Perhaps a URL would be simpler?
void queueMoveMessages ( sstr srcQueue, sstr destQueue, uint32 qty) Move messages from one queue to another   N Queues are local to vhosts, not brokers... since you may have queues with the same name on two separate vhosts on the same broker - this method cannot be implemented on a broker which supports multiple vhosts. Should also add an arguments map to allow filters to be applied
Suggested Additions
  • ability to create virtual hosts
  • shutdown/restart broker?
  • reload (security) configuration.

Agent

The Java Broker does not currently support the Agent class

Properties

Name
Type Description C++
Java
Notes
connectionRef objId     N  
label sstr Label for agent   N  
registeredTo objId Broker agent is registered to   N  
systemId uuid Identifier of system where agent resides   N  
brokerBank uint32 Assigned object-id broker bank   N  
agentBank uint32 Assigned object-id agent bank   N  

Vhost

Properties

Name
Type Description C++
Java
Notes
brokerRef objId     Y  
name sstr     Y  
federationTag sstr     Y  

Statistics

Suggested Additions
  • number (and high watermark) of queues/exchanges
  • number (and high watermark) of connections

Methods

Suggested Additions
  • create queue
  • create exchange
  • create binding
  • delete queue
  • delete exchange
  • delete this vhost

Queue

Properties

Name
Type Description C++
Java
Notes
vhostRef objId     Y  
name sstr     Y  
durable bool     Y  
autoDelete bool     Y  
exclusive bool     Y  
arguments map Arguments supplied in queue.declare   Y  
altExchange objId     Y  

Statistics

Name
Type Description C++
Java
Notes
msgTotalEnqueues count64 Total messages enqueued   Y
 
msgTotalDequeues count64 Total messages dequeued   Y  
msgTxnEnqueues count64 Transactional messages enqueued      
msgTxnDequeues count64 Transactional messages dequeued      
msgPersistEnqueues count64 Persistent messages enqueued   Y  
msgPersistDequeues count64 Persistent messages dequeued   Y  
msgDepth count32 Current size of queue in messages   Y Is this with or without unacked messages (i.e. does it include all messages for which the dequeue has not yet been committed)?It includes unacked messages
Also would seem to make sense to provide high watermark for queue depth (bytes and msg)GS: +1 on high watermark for queue depth; would be very useful
byteDepth count32 Current size of queue in bytes   Y  
byteTotalEnqueues count64 Total messages enqueued   Y  
byteTotalDequeues count64 Total messages dequeued   Y  
byteTxnEnqueues count64 Transactional messages enqueued      
byteTxnDequeues count64 Transactional messages dequeued      
bytePersistEnqueues count64 Persistent messages enqueued   Y  
bytePersistDequeues count64 Persistent messages dequeued   Y  
consumerCount hilo32 Current consumers on queue   Y  
bindingCount hilo32 Current bindings   Y  
unackedMessages hilo32 Messages consumed but not yet acked     It would seem more useful on a per subscription basis (so we can see which subscriptions are holding on to messages without acking them).GS: I agree
messageLatency mmaTime Broker latency through this queue     This generates a number of statistics (Max/Min/Avg/Samples) each of which is said to be measured in nanoseconds. What does Samples mean in this context?this isn't actually implemented on c++ yet
Suggested Additions
  • age of oldest message on the queue (and high watermark),
  • high watermark for message size
  • high watermark for queue depth (bytes)
  • high watermark for message count
  • messages/bytes expired due to TTL

Methods

Signature Description C++
Java
Notes
void purge ( uint32 request) Discard all or some messages on a queue   Y Would be nice to add an arguments map so that filters could be supplied. Also making request an IO parameter so you can return how many messages were purged. Finally are the messages supposed to be purged from the head, the tail or randomly throughout the queue?
Suggested Additions:
  • mechanisms to move / copy messages to another queue (with filters to allow specifying arbitrary messages, not first X msg)
  • rename this queue
  • delete this queue
  • mechanism to inspect messages on the queue

Exchange

Properties

Name
Type Description C++
Java
Notes
vhostRef objId     Y  
name sstr     Y  
type sstr     Y  
durable bool     Y  
autoDelete bool     Y  
altExchange objId     Y  
arguments map Arguments supplied in exchange.declare      

Statistics

Name
Type Description C++
Java
Notes
producerCount hilo32 Current producers on exchange N N How is this supposed to be calculated? RGem: This is not actually implemented on the C++ broker either
bindingCount hilo32 Current bindings   Y  
msgReceives count64 Total messages received   Y  
msgDrops count64 Total messages dropped (no matching key)   Y  
msgRoutes count64 Total routed messages   Y  
byteReceives count64 Total bytes received   Y  
byteDrops count64 Total bytes dropped (no matching key)   Y  
byteRoutes count64 Total routed bytes   Y  

Methods

Suggested Additions
  • rename
  • set AlternateExchange
  • delete

Binding

Properties

Name
Type Description C++
Java
Notes
exchangeRef objId     Y  
queueRef objId     Y  
bindingKey sstr     Y  
arguments map     Y  
origin sstr     Y definition would be good... why is this not in "arguments" (since that is how it is set)?

Statistics

Name
Type Description C++
Java
Notes
msgMatched count64     Y  

Methods

Suggested Additions
  • delete

Subscription

Properties

Name
Type Description C++
Java
Notes
sessionRef objId     Y
 
queueRef objId     Y  
name sstr     Y  
browsing bool     Y
is this supposed to be a pre/non acquire indicator? Is this  true only when pre-acquiring and accept-mode is none?This is true of the acquire-mode was not-acquired (for 0-10). It is independent of the accept-mode which is exposed via the acknowledged property
acknowledged bool     Y
is this simply accept-mode = explicit?yes
exclusive bool     Y
 
creditMode sstr WINDOW or CREDIT   Y
 
arguments map     Y
 

Statistics

Name
Type Description C++
Java
Notes
delivered count64 Messages delivered   Y
 
Suggested Additions
  • available credit
  • unacknowledged msg/bytes
  • delivered bytes

Methods

Suggested Additions
  • delete

Connection

Properties

Name
Type Description C++
Java
Notes
vhostRef objId     Y
 
address sstr     Y
Is this the local or remote address?  Whichever, we should also have the other one. It is the remote address
incoming bool     Y
 
SystemConnection bool Infrastucture/ Inter-system connection (Cluster, Federation, ...)   Y
 
federationLink bool Is this a federation link   Y
 
authIdentity sstr authId of connection if authentication enabled   Y
 
remoteProcessName sstr Name of executable running as remote client     These are a subset of the clientProperties sent by the client - should we not just show the client Props?
remotePid uint32 Process ID of remote client      
remoteParentPid uint32 Parent Process ID of remote client      
Suggested Additions
  • vhost
  • server/client properties maps
  • protocol version

Statistics

Name
Type Description C++
Java
Notes
closing bool This client is closing by management request     Why is this a statistic???
framesFromClient count64        
framesToClient count64        
bytesFromClient count64        
bytesToClient count64        
Suggested Additions
  • attached sessions (with high watermark)

Methods

Signature Description C++
Java
Notes
void close ( )        

Link

Properties

Name
Type Description C++
Java
Notes
vhostRef objId     Y
 
host sstr     Y
 
port uint16     Y
 
transport sstr     Y
What are the valid transports? at present, tcp, rdma and ssl(which is over tcp)
durable bool     Y
 
Suggested Additions
  • remote vhost

Statistics

Name
Type Description C++
Java
Notes
state sstr Operational state of the link     Are the valid states defined anywhere? c++ broker currently includes the following states: WAITING, CONNECTING, OPERATIONAL, FAILED, CLOSED, PASSIVE (from Link.h)
lastError sstr Reason link is not operational      

Methods

Signature Description C++
Java
Notes
void close ( )        
void bridge ( bool durable, sstr src, sstr dest, sstr key, sstr tag, sstr excludes, bool srcIsQueue, bool srcIsLocal, bool dynamic, uint16 sync) Bridge messages over the link   Y
 

Bridge

Properties

Name
Type Description C++
Java
Notes
linkRef objId     Y
 
channelId uint16     Y
 
durable bool     Y
 
src sstr     Y
 
dest sstr     Y
 
key sstr     Y
 
srcIsQueue bool     Y
 
srcIsLocal bool     Y
 
tag sstr       what is this and what does it do? It is used for loop detection and is the value to be used for qpid.trace.id on the subscription queue
excludes sstr    
what is this and what does it do? Also used for loop detection and is the value to be used for qpid.trace.excludes on the subscription queue
dynamic bool     Y
 
sync uint16        

Methods

Signature Description C++
Java
Notes
void close ( )        

Session

Properties

Name
Type Description C++
Java
Notes
vhostRef objId     Y
 
name sstr     Y
 
channelId uint16     Y
 
connectionRef objId     Y
 
detachedLifespan uint32        
attached bool     Y
 
expireTime absTime        
maxClientRate uint32        
Suggested Additions
  • transactional mode

Statistics

Name
Type Description C++
Java
Notes
framesOutstanding count32        
TxnStarts count64 Total transactions started     Are these supposed to be tx or dtx transactions? or either
TxnCommits count64 Total transactions committed      
TxnRejects count64 Total transactions rejected      
TxnCount count32 Current pending transactions      
clientCredit count32 Client message credit      
Suggested Additions:
  • subscription count/watermark
  • unacked messages/bytes

Methods

Signature Description C++
Java
Notes
void solicitAck ( )        
void detach ( )        
void resetLifespan ( )        
void close ( )        
Document generated by Confluence on May 26, 2010 10:33