Saturday, April 2, 2011

Exploratory Project Component Specified Verification Function

Exploratory Project Component Specified Verification Function


After solving the latest thread interaction problems that occurred as I added more test components to interact (as I mentioned in the previous post), the integration/consolation of the verification of topic messages to be done before a component can read the data – see the Exploratory Project Component Specified Verification Function post – took no time at all and was completed in an afternoon.

The verification of the topic messages has been integrated to occur at two different places. 

One, for messages received from a remote application, the initial verification occurs as the message is received.  The message header contains information as to what component sent it including the identifier of the application, the message CRC, the topic identifier, etc.  This information is used to verify the message matches the CRC; that the message source is an expected publishing component for the topic; etc.  This verification was implemented long ago in the lifetime of the Exploratory Project.

Two, for both remotely received and local messages, validation is performed when a component instantiates a Reader of the topic.

First, common validation that is done for any topic is performed.  Further validation of the header can be done for remotely received messages (that is, with a source application that is different from that of the running application).  Such validation makes no sense for locally published messages since the header is rarely used for such messages.  But if a CRC is produced in the future, it should be validated here.  Likewise, validations for correct combinations of attributes can be performed.  For instance, that the reading component is the consumer of the topic.

Second, if these common validations indicate that the topic instance is valid, the Valid function of the topic is invoked to perform common validations of the particular topic.  These validations are encoded in the topic’s Ada package (or C++, C# class/structure).  The topic’s class declares the structure of the topic data (other than the header) to provide the format of the data in the data buffer containing an instance of the topic.  The topic’s Valid function can perform checks on this data to verify that the data is usable by any reader.

Third, if the topic specific validation indicates that the message is usable, the Topic Validation function checks if the component of the topic reader has registered its own verify / filter function.  If it has, it passes the address of the data to this function where the component can check that the data values are such that it should act upon the new instance of the topic.

If all the validations succeed, the pointer to the data buffer is returned by the Reader to the consumer component.  Otherwise, the pointer is set to null to prevent the component from accessing the data.  This pointer is such that it references the data in the structure declared by the topic class but only allows the data to be read.  (A different pointer is returned to a producer component Writer that allows data to be copied (that is, written) to the newly assigned data buffer.  This pointer becomes invalid as soon as the component publishes the topic.)  When the data buffer is released, the reader pointer is reset to null indicating that the data buffer is no longer available to be read.

Before the latest change, the topic Valid function was callable by the component after the Reader returned the pointer to the data.  And the component could perform any filtering that it wanted on the data as well.  It still is able to do the later rather than registering a verify/filter function.  Also, a publishing component can fill in the data and then invoke the topic’s Valid function to check whether the combination of values are valid as far as the particular topic class is concerned.

These changes were made so that all the Reader validations could be automatically done prior to the use of the data by a component for its purposes.  That is, to encourage complete validation of a received message prior to using the data rather than scattering it throughout the user’s code and allowing checking of data fields prior to knowing if the message itself was valid.

This comes with some negatives as well as far as component specific validation / filtering.  This is because, with the current implementation, the address must be typecast to a pointer of the correct format.  Thus the component specific function could modify the data.  (The topic Valid function could as well although it should be better controlled as a more general package/class.)

These negatives can be corrected by implementing a Verify instantiation of the topic that would return the pointer as the Reader does.  (A second Reader can’t be used since that would cause circular execution.  Although, most likely, this could be prevented by an object that would indicate that the second Reader has to be that of the component’s Verify function such that most of the validation has already been done.)  These negatives will be corrected in a future implementation.

An example using snippets of Ada code follows.

One, the registration of a topic consumer by a component to also provide a Verify function.
  procedure Install is

    Deliver_Id_List
    --| Treat all widgets with ids between 100 and 199
    : constant mC.Itf_Types.Delivery_Id_Array_Type
    := ( ( 100, 199 ), ( 0, 0 ), ( 0, 0 ), ( 0, 0 ), ( 0, 0 ) );

    Delivery_Id
    --| Deliver topic when identifier is between 100 and 199
    : constant mC.Itf_Types.Delivery_List_Type
    := ( Count => 1,
         List  => Deliver_Id_List );

  begin – Install

    ...

    --| Logic_Step:
    --|   Register to consume all delivery (widget) ids of the "Topic Widget
    --|   Event" One-of-N message between 100 and 199.

    mT.Topic_Widget_Event_Layer1.Data.Register
    ( Participant => Component_Key,
      Detection   => mC.Itf_Types.Event_Driven,
      Role        => mC.Itf_Types.Consumer,
      Delivery_Id => Delivery_Id,
      Callback    => Treat_Widget_Event'access,
      Verify      => Verify_Delivery_Event'access,
      Access_Key  => Topic_Widget_Event_Access_Key,
      Status      => Data_Status );

    ...

  end Install;
where  the Verify parameter provides the callback to the Verify_Delivery_Event function. 

This function has the same characteristics as the Verify_Filter_Entry_Point_Type access type
  type Verify_Filter_Entry_Point_Type
  --| Callback to execute component and topic specific verify/filter function
  -- ++
  --| Overview:
  --|   This function is a component specific verify/filter function to be
  --|   executed when the component instantiates the reader for a topic.
  --|   If the component includes a verify/filter function callback when it
  --|   registers to consume a topic, the component supplied function will
  --|   be executed following after the Valid function of the topic.  Both
  --|   must be successful for pointer to the instance of the topic data
  --|   to be returned to the component to enable the data to be examined.
  -- --
  is access function
  ( Data : in System.Address
  ) return Boolean;

Two, an example of a topic package specification (visible part) and body (implementation).

package mT.Topic_Widget_Event_Layer1 is
-- ++
--| Overview:
--|   This package declares the widget event topic where the producer
--|   component mimics one that receives ARINC-661 events from a display
--|   processor / application and then publishes these events as a 1-of-N
--|   topic to notify the particular consumer component that registered
--|   to be notified of the topic when the ARINC-661 event identifier
--|   matches the Delivery Identifier for which it registered.
--|
--|   This topic is delivered such that the consumer receives each update
--|   rather than the most recent as for Data Flow and Event Driven.
--|   Therefore, multiple buffers may be needed to retain past updates that
--|   the consumer, due to its relative priority or processing time, the
--|   nature of the producer component, or other reasons prevent it from
--|   treating each update before another is produced.  The registering
--|   consumer component is able to specify the number of additional buffers
--|   that it needs.  If an insufficient number are specified (or the
--|   consumer component is no longer able to read the topic) the producer
--|   will fail to obtain a new write buffer until a new buffer is again
--|   available.
-- --

  Id
  : constant mC.Message.Topic_Id_Type
  := mC.Message.Topic_Widget_Event_Layer1_Id;

  Name
  : constant String(1..25) := "Topic Widget Event Layer1";

  type Data_Type is
  record

    Widget_Id : Display_Interface.Widget_Id_Type;
    --| Identifier of widget that sent the event

    Event_Tag : Display_Interface.Widget_Event_Type;
    --| Identifier of the type of event

    Text      : String_Tools.Variable_String_Type;
    --| Any text associated with the event

  end record;
  for Data_Type'size use 67*32; -- bits
  for Data_Type'alignment use mC.Message.Message_Alignment; -- bytes

  type Topic_Type
  is record
    Header : mC.Message.Message_Header_Type;
    Data   : Data_Type;
  end record;
  for Topic_Type'alignment use mC.Message.Message_Alignment; -- bytes
  for Topic_Type'size use mC.Message.Message_Header_Size*8+(67*32); -- bits

  ---------------------------------------------------------------------------

  function Valid
  ( Data : in mC.Message.Data_Address_Type
    --| Address of particular data buffer
  ) return Boolean;
  --| Validate the Topic Data
  -- ++
  --| Overview:
  --|   This function returns TRUE if Message data can be accessed by its
  --|   pointer and passes the validation.
  --| Limitations:
  --|   This function must ONLY be used to validate data following the
  --|   instantiation of a topic Reader.  It must NOT be used after the
  --|   instantiation of a topic Writer since it will overwrite the Valid
  --|   flag of the instantiation parameter causing the lack of the
  --|   assignment of a write buffer pointer to appear to be valid.
  -- --

  type Topic_Type_Access is access Topic_Type;
  for Topic_Type_Access'storage_size use 0;
  --| Make sure a user doesn't allocate storage for these objects

  function Address_to_Data_Ptr
  ( Data : in System.Address
    --| Address of data buffer provided by Reader
  ) return Topic_Type_Access;
  --| Return a pointer to the topic data
  -- ++
  --| Overview:
  --|   This function returns a pointer to an instance of the data of the
  --|   topic as instantiated by the topic reader.
  -- --

  ---------------------------------------------------------------------------

  package Data
  --| Accessor to topic data
  is new mC.Message.Topic( Id         => Id,
                           Name       => Name,
                           Protocol   => mC.Itf_Types.Point_to_Point,
                           Permanence => mC.Itf_Types.EP,
                           Exchange   => mC.Itf_Types.Delivery_Identifier,
                           Kind       => mC.Itf_Types.Request,
                           Data_Type  => Topic_Type,
                           Validation => Valid'access );

end mT.Topic_Widget_Event_Layer1;

package body mT.Topic_Widget_Event_Layer1 is
-- ++
--| Overview:
--|   This package declares the widget event topic where the producer
--|   component mimics one that receives ARINC-661 events from a display
--|   processor / application and then publishes these events as a 1-of-N
--|   topic to notify the particular consumer component that registered to be
--|   notified of the topic when the ARINC-661 event identifier matches the
--|   Delivery Identifier for which they registered.
-- --

  ---------------------------------------------------------------------------

  function Address_to_Data_Ptr
  ( Data : in System.Address
    --| Address of data buffer provided by Reader
  ) return Topic_Type_Access is

    function Address_to_Access
    --| Type cast the address of the topic data to a pointer to
    --| the record type
    is new Unchecked_Conversion( Source => System.Address,
                                 Target => Topic_Type_Access );

  begin -- Address_to_Data_Ptr

    return Address_to_Access( Data );

  end Address_to_Data_Ptr;

  ---------------------------------------------------------------------------

  function Valid
  ( Data : in mC.Message.Data_Address_Type
    --| Access key
  ) return Boolean is
  -- ++
  --| Logic_Flow:
  --|   Validate the data and return True if successful.
  -- --

    Data_Ptr
    --| Pointer to data buffer
    : Topic_Type_Access
    := Address_to_Data_Ptr( System.Address(Data) );

  begin -- Valid

    if Data_Ptr = null then
      return False;
    else

      -- example
      if Data_Ptr.Data.Widget_Id > 0 and then
         Data_Ptr.Data.Widget_Id < 10000
      then
        return True;
      else
        return False;
      end if;

    end if;

  end Valid;

end mT.Topic_Widget_Event_Layer1;

Three, the instantiation of a Reader by a topic consumer.  Note, Treat_Widget_Event is the callback registered above in the Install to be executed when the event occurs.
  procedure Treat_Widget_Event
  ( Topic : in mC.Topic_Name_Type
  ) is

    package Widget_Event_Data
    is new mT.Topic_Widget_Event_Layer1.Data.Reader
           ( Access_Key => Topic_Widget_Event_Access_Key,
             Fresh_Data => True );

  begin -- Treat_Widget_Event

    --| Logic_Step:
    --|   Check if data is usable.

    if Widget_Event_Data.Valid then

      --| Logic_Stop:
      --|   Act upon the received event.

      Create_Response
      ( Widget_Id => Widget_Event_Data.Ptr.Data.Widget_Id,
        Event     => Widget_Event_Data.Ptr.Data.Event_Tag,
        Text      => Widget_Event_Data.Ptr.Data.Text );

--  else -- not Widget_Event_Data.Valid
    end if; -- Widget_Event_Data.Valid

  end Treat_Widget_Event;
Instead of the check of Widget_Event_Data.Valid, a check that Widget_Event_Data.Ptr is non-null could have been done.  Any use of the topic data fields accessible via the Widget_Event_Data.Ptr.Data pointer could have been done.

No validation of the data had to be done in the above procedure because it had all been accomplished by the framework and the component’s Verify_Delivery_Event function when package Widget_Event_Data was instantiated (such as can occur using the constructor of a C++/C# class).

When the above topic Reader was instantiated, the data buffer was located and then the common validation was performed.
function Topic_Validation
( Validation : in Validation_Access_Type;
  Data       : in mC.Message.Data_Address_Type;
  Access_Key : in Key_Type
) return Boolean is

  Header_Ptr
  --| Pointer to topic header
  : Message_Header_Ptr_Type
  := Address_to_Header_Ptr( System.Address(Data) );

  Valid
  --| True if topic data is valid and contains the data values that the
  --| component needs to treat the topic
  : Boolean;

  use type Apps.Component_Selector_Type;
  use type mC.Itf_Types.Verify_Filter_Entry_Point_Type;

begin -- Topic_Validation

  --| Logic_Step:
  --|   Do any general validation of the header.
  --| Notes:
  --|   o For remote messages, the CRC checked when received as well as a
  --|     valid, expected Source for the topic.
  --|   o For local messages, the header really doesn't mean anything.

  -- The following is an illustration of examining the header.
  if Integer(Header_Ptr.Source.User_Id) = Integer(Local_App.Id) or else
     ( Header_Ptr.Source.User_Id = 0 and then
       Header_Ptr.Source.Com_Id  = 0 and then
       Header_Ptr.Source.Sub_Id  = 0 )
  then
    Console.Write( "Local Topic instance" );
  else
    Console.Write( "Remote Topic instance",
                   Integer(Header_Ptr.Source.User_Id),
                   Integer(Header_Ptr.Source.Com_Id),
                   Integer(Header_Ptr.Source.Sub_Id) );
  end if;

  --| Logic_Step:
  --|   Do general validation for the particular topic.

  if Validation /= null then
    Valid := Validation( Data => Data );
  else
    Valid := True; -- no validation routine to modify the original validity
  end if;

  if Valid then

    --| Logic_Step:
    --|   Do any component specific validation or filtering of the consumer.

    if Key_Type(Access_Key).User_Record.Verify /= null then
      Valid := Key_Type(Access_Key).User_Record.Verify
               ( Data => System.Address(Data) );
    end if;

  else
    Console.Write_Error("Topic_Validation Valid of False");
  end if;

  return Valid;

end Topic_Validation;

In the above, the topic header is first checked.  If no problems are found, other general validation could be done.  Next, the supplied Validation callback for the topic is executed.  If that returns Valid of True, any Verify callback of the component of the Reader is executed.  If Valid is returned as True, the reader will make the data pointer available to the topic consumer.

Last, the Verify / Filter function of the topic consumer as invoked by the last step of Topic_Validation.
  function Verify_Delivery_Event -- of Verify_Filter_Entry_Point_Type access
  ( Data : in System.Address
  ) return Boolean is
  -- ++
  --| Logic_Flow:
  --|   Verify/filter the topic when instantiate the Reader.
  -- --

    Data_Ptr
    --| Pointer to the topic data as provided by the above callback from
    --| the topic Reader instantiation
    : mT.Topic_Widget_Event_Layer1.Topic_Type_Access
    := mT.Topic_Widget_Event_Layer1.Address_to_Data_Ptr( Data );

  begin -- Verify_Delivery_Event

    if Data_Ptr.Data.Widget_Id = abc or else
       ( Data_Ptr.Data.Widget_Id >= xyz and then
         Data_Ptr.Data.Widget_Id <= zyx )
    then
      return True;
    else
      return False;
    end if;

  end Verify_Delivery_Event;

Service-Oriented Architecture Attributes Progress Report 4 – Thread Interaction

Service-Oriented Architecture Attributes Progress Report 4 – Thread Interaction


When I started making changes to support Component Specified Verification I made some modifications as well by adding some additional test components to produce different message combinations.  This broke the framework. 

It started out, if I remember correctly, with the timing of the message topics, their delivery, and their release.  Way back when during the predecessor of this personal project, there was the design goal of having the framework run in the same thread (APEX process) as the component that was interfacing to it to save process switching time.  I heard after I left – the only thing that I’ve heard about the project – that this approach was abandoned as not working.  When I started my personal PC based project I retained the same thread approach thinking I would find out why it became necessary to discard it.

The problem, of course, is that causing a new component to start running via a notification event (or periodically if not an Upon Demand component) can cause a component with a higher priority to run suspending the currently running thread that may be currently executing framework code.  For instance, when a component publishes a notification topic to cause another component to be run, that upon demand component will start immediately if it has a higher priority suspending the original component.  Therefore, the rest of its execution will be suspended as well as that of the framework to which it would return.  That delays the framework’s execution of the data buffer release of the delivered topic that has been read by the consumer component and hence no longer needed by it.

The newly running component can publish its own instance of a request that is also being requested by the suspended component.  When the second component finishes, the N-to-One Request/Response consumer component can then begin to run as having a priority intermediate to the first two.  This component will then produce a response to the first component before it has completed its execution that produced the request.

This caused a good deal of reevaluation as to when Read Queues were needed to contain multiple instances of a topic to be treated by a component and when multiple data buffers were needed when ordinarily only a single buffer is needed since it would be released before a new request or response would need the buffer. 

This likely answered why the original approach was abandoned for the aerospace project.  However, having solved some of these problems earlier as my own PC project proceeded, I continued the approach to see what would develop.  For the PC project, the problems are, most likely, greater since the number of threads available using Windows is quite limited so threads often run via time slicing.  Thus, threads of the same priority alternate with one running and then being suspended and another running and the various same priority threads alternating their execution.

After some unproductive approaches I have now succeeded once again and each request is delivered to its consumer and each data buffer is released after the consumer has had a chance to read the topic.

Wednesday, March 9, 2011

Exploratory Project Component Specified Verification Function

Exploratory Project Component Specified Verification Function


Reflecting upon a past aerospace project last night concerning implementation concerns that I had had while doing DO178B level A reviews a new feature to add to my exploratory project came to mind.

The aerospace code that came to mind was code that was involved with the receive and use of messages from particular ARINC 653 ports.  My problem with the code had to do with the haphazard way the received data was checked.  A message would have verification of its validity performed in various places in the code along with checks as to whether the values of various fields were such as to indicate that the message should be used.  The validation done would be inconsistent from the treatment of one port versus another.  And fields would be checked for usefulness before the validity of the message had been completely determined.  Also, various checks would be done in different paths of the code and in different called routines making it difficult to determine if all the necessary checks had been done / would get done. 

At the time I thought how such code should be redone to have a particular set of validation functions.  One to perform the necessary checks of any port message and another for each receive port to be called following the first that would check the values of the various fields as to whether the message was usable.

It occurred to me that my proposed routines to check the received values would have been similar to the Content Filtered messages of my Exploratory Project; except that they would be run after the message was received rather than prior to sending it.

Then I thought of how supplying a special verification routine to the message delivery system of the exploratory project would cause the checks to always be performed first.  That is, a particular consumer could reverse the existing Content Filtered message exchange mechanism to specify its own filter as a callback at the time it registers for the topic.

The framework already allows each topic to have a verify function specified for it to determine if a received topic is valid prior to allowing the instantiation of a Reader for it.  By also allowing a consumer component to specify its own filter/verify function would ensure that the topic message had values in the ranges that were acceptable for use by the consumer prior to its attempting to use them.  In fact, for point-to-point topics, the message communication framework could even avoid delivering the topic instance if the contained fields that would cause it to be ignored.  (But then would be executing the function in the thread of the framework rather than in the thread of the user component.)

In any case, when the topic verify was requested by a component to obtain the pointer to the message data, the framework could first execute the common verification of the topic as it does now and then run the particular filter/verify function of the component.  If the topic data didn’t pass both validations, invalid would be returned and the data pointer would be null preventing the component from using the data.

Also the message would always be validated prior to be used with the validations being done and their order easily identified for certification.

To be consistent, the framework most likely should not execute the verify/filter function to determine whether to deliver it to the component when the instance of the topic is published.  That is, it would be difficult to do this for publish/subscribe topics.  And the function would be executed in the thread of the framework sometimes and in the thread of the consumer component in others.  And, of course, for a remotely published topic it would have to be received by the local app before the local framework could execute the filter/verify function. 

Such a verify/filter function would be a combination framework attribute of the topic and the consumer and would initially be supplied at the time the consumer registers for the topic.  An additional feature could be the ability to allow a component to change the registered filter/verify callback to be executed as conditions change as the application runs or to supply a null callback to discard filtering.

Service-Oriented Architecture Attributes Progress Report 3

Service-Oriented Architecture Attributes Progress Report 3


This progress report has been significantly delayed – mainly because I have been interested in something else – but also since checking out some of the restructured attributes and added topic delivery methods was more challenging then expected.  Particularly when the topic had to be delivered to a component of a remote application.

One change was to treat Content Filtered topics similarly to Request/Response topics.  This made both methods of message exchange examples of the previous N-to-One delivery category. 

I hadn’t thought of Content Filtered as really N-to-One, thinking that multiple consumers could answer requests.  But I found this a problem when I went to implement the second instance of the topic since I didn’t have a mechanism to send the request to multiple consumers.  So I cloned the Content Filtered topic as a second such topic so the remote consumer component could treat a different topic. 

If I am to have multiple consumers to send responses to the multiple requesting components for a particular filter condition, then I need to have a different way of treating this.  And would might be the consequences if more than one component detected that the filter conditions were satisfied so that the requesting component got multiple responses. 

This message exchange would be M-to-N.  That is, a variety of both One-of-N and N-to-One where instead of the delivery identifier of One-of-N selecting which of the N consumers was to receive the message, the request message would be sent to each consumer.  That consumer would then need to decide if it could respond to one or more of the filter conditions and save those in its filter table.  Then when the conditions were satisfied, send its response to each of the requestors that ask for a response under those conditions.  It would be sending a One-to-One Point-to-Point response multiple times.

This would be a whole different ballgame from what I have had.  So, for now at least, Content Filtered is an example of N-to-One in that there is one consumer of the request.  And, instead of sending an immediate, synchronous response to the particular requestor, it sends the delayed asynchronous response to all the components that requested a response when the particular filter conditions were satisfied.

This was implemented and checked out with both locally delivered messages within a particular application and remotely delivered to components of another application for message exchange of No Exchange (broadcast to be delivered to any subscriber), Delivery Identifier (One-of-N), Request/Response and Content Filtered (both examples of N requestors to One consumer/responder).