Message Service Update

The latest adaptation of my exploratory project is known as the Message Service. 

This is an update to the previous memo.  It includes the addition of communication with a non-framework based application.  The non-framework based application chosen is an ARINC-661 protocol Display Application.  The changes to the framework based application needed to interface to a non-framework based application will be the updated portion of this memo.  A minimal example of a Display Application was developed to verify that the communications between the two kinds of applications was successfully handled by the changes to the Message Service based application.

The Message Service (MS) is a framework for the delivery of individual messages from a producer component to the consumer components that have registered for the topic of the message.  The delivery will occur within the application of the producer as well as other (remote) applications based upon the Message Service.

The Message Service is a Microsoft Windows PC based application written almost exclusively in Ada.  However, it consigns Operating System (OS) interfaces to particular Ada packages so that it can easily be supported by Linux as well (and has been in the past).

The Message Service uses its own mechanism for the delivery of local messages and Microsoft Pipes for delivery between applications.  (Linux named pipes will support delivery between applications when running under Linux as the OS.  This was implemented in previous versions of the exploratory project so the modifications are known.)  In addition, previous versions also supported the use of TCP/IP for delivery between applications – either hosted by a single PC or multiple PCs.

The OS interfaces are mainly hidden within the Message Service framework and not accessible to the User Components.  Where the term User Components refers to OS threads whose purpose is to support the objective of the application.  This adaptation moved Remote IO to a user component.  To support the move of this feature specific OS interfaces were made visible via the Remote_Itf package.

This adaptation of the exploratory project consigns the actual message delivery between applications to an input/output (IO) user component.  Previously the delivery of topic messages between applications was implemented within the framework.  With this modification of the project architecture other such user components can be developed to interface to non-framework based applications – for instance, A661 display applications.  Or, at least, separate Ada packages can be implemented within the Remote IO component to interface to each particular category of remote application.

The User Components are Installed by an application specific Ada package that are invoked directly from the Ada main procedure of the Ada based application.  Otherwise, each application contains common code.  The OS folders that contain the application consist of the common code as well as folders that are specific to the application.  The OS interface code is written to recognize which operating system the application is running under.  If this wasn't so builds could be done by operating system by also having OS specific folders.

The current architecture has the following components/threads (figure Message Service Overall Structure) where the IO component has multiple threads.  The framework has the threads and packages of figure Message Service Framework Structure.

  

The above figure illustrates six User Components along with the Apps package and the IO Framework Friend Component which is actually a component with subcomponents to be illustrated below.  It is considered a friend of the Message Service Framework since it is installed by the Framework rather than the Apps package that is invoked by the Ada main procedure to install the user components of the particular application.  As a friend component it enjoys access to selected IO procedures/functions and uses an OS event to start the framework Support thread and is likewise started by an OS event rather than the framework topics selected by the user components. 

The Apps package is invoked when the OS launches the application.  It reads the Configuration file to obtain the allowed applications with their executable names and whether the application is a Message Service Framework based application or another non-MS application such as an A661 Display application and the communication methods that the application supports.  The Apps package uses this information to build a table including the information.  The Apps package contains an application unique procedure that invokes the Install procedures of the User Components of the application.

The messages between the IO component and the framework are communicated via queues where the address of the message is passed in the queue rather than the message data to avoid unnecessary copying of the message data.  Note: "To Queue" is an abbreviation of "To IO Framework Component" and similarly for "From Queue".  The Start Event causes the thread to recover from an OS Wait and to read the queue for attributes of received messages to be delivered to local user components or transmitted to a remote application.

The messages between the user components are delivered via a framework topic feature.  This mechanism is supported by each user component registering with the framework which topics it will publish and which it wants to have delivered to it.  The framework keeps track of this and when a user component publishes a topic it delivers the message data of the topic to the components that registered to consume it.

An application that isn't based upon the Message Service framework will, of course, have its own mechanism to deliver messages.  Such applications will communicate with MS applications via communication buses such as pipes and TCP/IP and the MS framework will deliver the received message as a topic to the user component that has registered to consume it.  Likewise, topic messages to be transmitted will be converted as necessary for the particular remote application by that application's IO component.

The diagram of the IO Framework Friend Component prior to the inclusion of non-MS framework protocols is above. 

With the inclusion a set of protocol packages has been added similar to the communication method packages (MS_Pipe, etc) with each protocol package treating specifics of the particular protocol to communicate between the MS Framework based application and the remote application for which the communications protocol applies.

The IO Framework Friend Component consists of

1) the Remote IO thread that receives OS wakeup events from the MS Framework to transmit messages to the remote MS Framework based applications,

2) the A661 IO thread that receives OS wakeup events to treat requests and to transmit messages to remote A661 Display applications,

3) a Receive thread for each remote application of the configuration, and

3) the Monitor thread to watch for disconnects between the local application and a remote application.

The Receive threads queue notifications of messages received from remote applications as they are received.  If the received message is from a framework based application, the message is forwarded to the MS Based package for treatment.  Likewise, if the message is from an A661 display application, the message is forwarded to the A661 package for treatment.

The MS Based child package of Remote IO will check whether the received message is a Heartbeat, Remote Register, or user component message and, if one the latter two, queue the message and send the OS wakeup event to the Remote thread of the MS Framework.  When awoken from its Wait, the Remote thread will publish the topic data portion of the received message to the user components that have registered for the topic.

The A661 child package of Remote IO will queue the received message to the A661 IO thread package for treatment.  This A661 IO package (just identified as A661 in the implementation and not to be confused with the A661 child package of Remote IO) has its own set of child packages to be a mini-framework for the treatment of A661 messages.  As such it is incomplete.  That is, in previous versions of the Exploratory Project, I had implemented a more complete (yet still not full-fledged) implementation that interfaced to a C# (Mono for Linux) Display application that could display widgets in a window on the terminal screen.  Since Microsoft has not renewed my access to C# I implemented the limited A661 display application to be used to debug the interface to a MS Framework application in Ada.  This limited version just decodes the trial message and sends the response to illustrate that the concept design works.

When user components publish topics that a remote application has registered to consume them, the OS Framework will deliver the topic message to the Remote component which will then queue it for delivery to the IO Framework Friend Component.  This will send the OS wakeup event of the Remote IO thread.  The top-level Transmit procedure of Remote IO then forwards the message to the MS Based or A661 child package as appropriate for the specialize protocol processing.  The applicable child package will then cause the message to be transmitted via the correct communications driver package. 

This allows user components to treat either MS Based message topics or A661 messages and publish their response.  The response message will then be queued to the "To Remote IO" message queue, Remote IO will be woken up and it will forward the response message to the correct child package to transmit to the specified remote application.

For messages received from MS Based applications, the messages are queued to the "From Remote IO" message queue for delivery directly to the Message Remote package which, in turn, will deliver them to the applicable user component.  See the following diagram. 

For messages received from an A661 display application, the messages are queued to the "To A661" queue for delivery to the A661 for specialized processing.  When and if more completely implemented like that of the previous Exploratory Project, it would look up the user component action procedure that is to treat the A661 widget event and then cause it to be delivered to user component.

Both the Receive threads and Transmit use particular IO driver packages.  For instance, there will be a Receive thread for each supported method of external communications for each remote application of the configuration.  Transmit, on the other hand, will select the supported method of communications and invoke its transmit routine.

The Remote IO thread can also be awakened by the Monitor thread to treat changes in the status of the connection between the local application and a remote application.  Likewise, the dequeued To Remote IO items can indicate errors that should be logged rather than a message to be transmitted.

The Remote_Itf package is a friend interface to particular OS functions and procedures otherwise hidden within the Framework.  It also contains the circular queues and the mechanism for sending the OS wakeup events to either the Remote thread of the Framework or the Remote_IO thread.  The OS event selected depends upon which queue had an added message element.

There are other support procedures, of course, besides those illustrated.

The Message Service Framework is illustrated above.

The User Components interface with the Message Service Framework via the Itf package which, in turn, invokes the Itf package procedures of the Message-Itf package that is a child package of Message. 

If the request is to register the User Component or one of the topics it will produce or consume, the Message-Itf procedure will invoke the Scheduler package to allocate the component or the Message-Registry package to update the topic tables.  Since these requests precede the starting of the threads, each such request will be treated without thread concurrency.  That is, these requests run under the Operating System launch thread where there are no other running threads.

If the request is to obtain a message buffer, publish an instance of a topic (the message buffer), or get any recently published topics that the user component has registered to consume the Message-Itf package buffers the request and sends an OS wakeup event to the Support thread within the Message-Support package.  The Support thread then invokes one of Message-Delivery procedures.  Since the Support thread is the highest priority thread, only one such topic request will be treated at a time.

After all User Components of the particular application have been installed, Scheduler will start the User Component threads as well as the framework threads.  Besides the Support and Remote threads, these include the Event Driven and the three different periodic threads that implement the delivery of topics to the various user components. 

Note: If a user component has installed itself to run as a Periodic thread it will run as one of the Scheduler Periodic threads.  Similarly for Aperiodic and Pseudo Periodic.  If the user component installed itself as an Event Driven thread, it will be run as one of the Scheduler Event Driven threads when its topic is published.  That is, the Scheduler creates a thread farm with an instance of each kind of thread as the user components install themselves.  Any particular User Component thread is a thread from the Scheduler thread farm.

Periodic threads are run at a rate specified by the install.  Aperiodic threads run and then Scheduler Aperiodic delays for the specified interval and then activates the user component entry point once again.  Pseudo Periodic threads wait for their OS start event every time they return from the user component.  Thus they are all similar but have a different mechanism to provide their periodicity. 

No matter the cause for how a user component is run, the availability of the topic instances (i.e., messages) disappears after the user component returns control to the Scheduler.  Following the return the Scheduler thread invokes a Message Release procedure that releases the message data buffer of the topic that was temporarily allocated to the user component.  So if a component needs to retain particular data from a received message it must copy it from the message.

A user component is provided a pointer to the message data buffer upon its return from its Itf request for a particular topic (or for any newly available topic).  It can then type cast the pointer to the format of the topic's data and examine it without the need to copy it from the data buffer.  When the component is going to publish a topic message it, of course, must copy data values into the data buffer via the supplied pointer.

Each consumer of a particular topic is provided its own data buffer each time the topic is published.  Therefore, a component can attempt to modify the message that it receives but such a modification will not affect the message delivered to another component and will also not be useful.  In this manner the Message Service attempts to avoid excessive copying of data which can effect performance.

This feature also exists for the messages created to be transmitted to other applications or those received from other applications.  Since the Remote and Receive threads do not know in advance which topic will be delivered to the Remote_IO thread for transmit or from a Receive thread for delivery to a local user consumer component, these message buffers are pre-allocated at the maximum size.  As a buffer is needed for transmit or for the next receive, the assigned buffer is marked as no longer available.  To avoid copying only the address of the buffer is passed in the From Remote IO or To Remote IO queues.  After the data has been transmitted by Remote IO or published by Remote for delivery to a local user component a Release command is queued to enable the buffer to once again be marked as available.