Sunday, January 29, 2012

Interfacing to an Access Database via Visual C# 2010 Express


Interfacing to an Access Database via Visual C# 2010 Express

 Background

Looking for something to do, I decided to look into something that I had previously done a few times for aviation projects.  That is, to interface an Access database with a Visual Basic program to
   1.  Maintain the database and
   2.   Use the database to support an aviation project.

The first such project was to read the debug output of a cross-compiler build to find the addresses assigned to procedures, variables, type declarations, etc for use in gathering flight data, code path iterations, etc and then, after the flight, to report it by comparing the output to the database values to report by variable name, etc.

The second was to maintain the relationship between ARINC-661 widget identifiers and the user application actions to be ran upon receiving the widget event or to produce output commands to modify the cockpit displays.  Then, from an aircraft project xml file, to produce the A661 xml Definition File (DF) for loading into the Display Processor and another, user application oriented output version of the project file (containing some non-A661 tags such as the enumerated values of the actions) to be loaded into the user application.   These Visual Basic applications to keep the user application in sync with the A661 display DF as the definition file changed – such as the inclusion of new or changed widget identifiers and their associated application actions.

Between the first project and the second, the versions of Access and Visual Basic had changed so the interface procedures and techniques had to also be somewhat changed and had to be reworked for accessing the database of the second set of projects.

Now, at home for my amusement, I thought I would try to interface with an Access database.  I have never updated the version of the Microsoft Access database application that I purchased with Office 2000 back a decade or more ago.  But I thought I would see if the Visual Basic 2010 Express and Visual C# 2010 Express that I have been using would happen to be able to read databases produced by Access 2000.  As it turned out, I used Visual C# 2010 Express.

Since I had previously had to change my Visual Basic interface calls to read and update the Access database with new releases of Visual Basic and Access, I was pleasantly surprised when it turned out that I was able to use Visual C# 2010 Express to interface with Access 2000 of Windows 97.

Visual C# 2010 Express project initial problems

At first I managed to create a Visual C# project using my new 64-bit PC that referenced one of my .mdb files to update new records to its table, but never to read the file.  That is, each new run would detect zero rows in the database but would produce the rows over again and cause the timestamp on the .mdb to be updated.  If the program was modified to create another new row, the .mdb file then contained that row when examined using my old Windows Office 2000 Access application.

Searching the internet I found that a 32-bit program had to be used to interface to Access and that certain dll's had to be present in the Windows System32 folder.  Also, that the Express versions didn't support producing a 32-bit program on my 64-bit Windows 7 computer.

I therefore attempted the program on my 32-bit XP notebook where its version of Visual C# 2010 Express ran and was able to read the existing .mdb file. 

On the notebook a sample program from Microsoft ran and could retrieve the data from the sample's .mdb file.  On the Windows 7 PC it wouldn't run and resulted in an exception when attempting to fill the data set after opening the connection to the database. 

However, in attempting to run my own project on the XP notebook, it continued to ignore the original row or the two rows that were updated by the program.  However, the behavior was somewhat different than on the 64-bit PC.

On the Windows 7 PC the program could output (update) a new row the first time that particular row was created by the program but would always find zero rows when restarted.  After running, the mdb file timestamp would be updated, but it wouldn't have any new rows (unless the program had been modified to create yet another row).

On the XP PC each time the program is run, the two new rows that it created were duplicated in the mdb file so that its number of rows increased each time.  But it didn't recognize these rows the next time it is run.

So the operation is somewhat different on the XP notebook and the Microsoft example sample did recognize the two existing rows of its mdb file.

I noticed that the properties associated with the C# icon on the notebook have a Compatibility tab with a compatibility mode.  On the XP notebook the Compatibility is Windows 97 which happens to be where the Access 2000 came from.  On the Windows 7 PC, the Visual C# install resulted in Windows XP (Service Pack 3) compatibility.  I tried changing it to Windows 97 compatibility but C# projects can't be launched with this setting.

It took awhile to discover how to update the database using the generated database class.  There are various update methods that only update the loaded dataset but not the actual database.  There is just one Insert method to add a new row to the database.  That is, one can update the dataset a row at a time including a new row but, using the C# generated classes, there is only one method to output a new row to the database and it doesn't update the dataset at the same time.  (Note:  Creating a C# project for a particular Access .mdb file generates supporting classes in their own project files to be used by the user's Program while the Microsoft examples use OleDb classes directly for this type of database.)

Also, as I created the project, the generated database classes are such that the original database is copied to both the /bin/Debug and /bin/Release subfolders.  Running the C# debugger or the executable that is in the Release folder causes the original database to be copied to the subfolder being used.  Using the debugger causes this to happen each time it starts running.  Using the Release executable, the copy is made at least every time a new Solution is Built.

Therefore, trying to determine if my code was working caused me some wonderment.  Sometimes the changes would be there and then they would disappear again.  After I figured this out, I could move the executable to the folder of the database selected when building the generated classes and keep the modifications from one execution to the next.

Visual C# 2010 Express project example

The example code provided below has the project Program class Main procedure invoke a ReadUpdateReadwithDataSetDesignerClasses method.  (This naming used since I also plan to produce a similar example using the OleDB classes directly rather then the classes generated by referencing the database when the C# project is created.)

The database name is db1.mdb.  It has one table named HouseholdInventory with seven fields/columns named as described in the example code.  The first column is the CategoryID column.  It is the database key and is AutoNumber.

The example
o Opens a connection to the database,
o Loads the database into the internal dataset, and
o Instantiates a Reader and then displays the current contents of the dataset's table.

Then the example
o Adds a new row to the dataset and Inserts it into the database, and
o Once again creates a Reader and displays the current contents of the dataset's table.

Following this, the example
o Queries for a particular row by the unique value of the CategoryID field/column,
o Modifies a couple of fields in a copy of the queried row and Updates the modified row into the dataset and the database, and
o Creates a Reader once more and displays the updated contents of the dataset.

Lastly, the example closes the connection and waits for the operator to examine the Console output.  Before using ENTER/RETURN the operator can also use Access to check that the database matches the displayed dataset values.

Notes

Since the example attempts to lookup a CategoryID of 3, the original database must have a least three rows which, with AutoNumber will have values of 1, 2, and 3.

As mentioned in the example code, the database, dataset table, and table adapter objects are instantiated from the generated classes as static within the Program class along with an unused table adapter manager object.  They could, of course, be instantiated locally at the beginning of the ReadUpdateReadwithDataSetDesignerClasses method.

To be able to provide all the steps required in this post to select a particular Access database to be associated with a project, I went through all the steps once again.  In doing so, I discovered why I had what I had thought was strange behavior.  See step 16 below.

Steps to Create a C# Access database project

1) Launch Microsoft Visual C# 2010 Express
2) Select File|New Project
3) In New Project form select Console Application
4) With the ConsoleApplication1 (for instance) window opened, select Project|Add Class.
5) Out of the various options, select Local Database.
6) This will display the Data Source Configuration Wizard form to Choose a Database Model.  With Dataset highlighted, do Next>.
7) This will display that the database selected is new.  Do < Previous.
8) This will display whether to display the connection string.  Do < Previous again.
9) This will display the Choose Your Data Collection form with a default of Database1.sdf.  Select New Connection.
10) This will display the Add Connection form.  Select Change.
11) This will display the Change Data Source form.  Select Microsoft Access Database File.
12) This will display .NET Framework Data Provider for OLE DB as the Data provider.  Select OK.
13) This will return to the Add Connection form.  Browse for the .mdb database and Open it.  (Advanced Properties will display that the Provider=Microsoft.Jet.OLEDB.4.0 along with the selected Data Source.)
14) Test Connection will validate that the connection can succeed.  After OKing that, select OK on the Add Connection form.
15) Upon return to the Data Source Configuration Wizard, select Next >.
16) Then there will be whether to copy the data file to your project.  Selecting Yes will result in what I thought was the strange behavior when running the Debug and the Release versions of the project since I hadn't paid enough attention.  Selecting No will always use the selected file.
17) Upon return to the  Data Source Configuration Wizard, Yes will be checked to save the connection string as a particular name.  Select NEXT >.
18) This will display a Choose Your Database Objects form of the Wizard.  Select the + to display the Tables in the database and then the + of the desired tables and then use the checkboxes to select the tables and fields/columns objects wanted.  The Views checkbox can also be used to include this object.  Then select Finish.

The above steps create a Program.cs code file for a Program class with an empty Main procedure along with subfolders of Properties and References. 

Example Code

  static void ReadUpdateReadwithDataSetDesignerClasses()
  {
      // This method reads and displays an existing Access database and
      // then updates the loaded dataset and inserts new rows into the
      // database and then redisplays the dataset.  To do this it uses
      // the Microsoft Visual C# 2010 Express generated DataSet classes
      // that were created at the time the new project was created as
      // well as the Properties folder, etc.
      //
      // The existing Access database that was used to create the
      // generated classes.  This database has one table named
      // HouseholdInventory.  This table has 7 fields/columns where
      // the first column is named CategoryID and has a data type of
      // AutoNumber.  Five of the other 6 columns are Text strings
      // while the DatePurchased column has a data type of Date/Time.
      //
      // New static objects of these generated classes of
      //   static db1DataSet1 db1 = new db1DataSet1();
      //   static db1DataSet1.HouseholdInventoryDataTable table =
      //     new db1DataSet1.HouseholdInventoryDataTable();
      //   static HouseholdInventoryTableAdapter adapter =
      //     new HouseholdInventoryTableAdapter();
      //   static TableAdapterManager manager = new TableAdapterManager();
      // are instantiated in the Program class of which this method is a
      // part and are referenced by this method.

      adapter.Connection.Open(); // Open a connection to the database

      table = adapter.GetData(); // Get/Load the database into the dataset table

      int rows = table.Count;    // Display original number of rows
      Console.WriteLine("original number of table rows {0}", rows.ToString());

      // Display the original data and capture last current CategoryID value.
      DataTableReader reader = table.CreateDataReader();
      while (reader.Read())
      {
          Console.WriteLine("Category ID {0}", reader["CategoryID"].ToString());
          Console.WriteLine("Notes {0}", reader["Notes"].ToString());
          Console.WriteLine("Model Number {0}", reader["ModelNumber"].ToString());
          Console.WriteLine("Date Purchased {0}",
                            reader["DatePurchased"].ToString());
          Console.WriteLine("Description {0}", reader["Description"].ToString());
          Console.WriteLine("Item Name {0}", reader["ItemName"].ToString());
          Console.WriteLine("Item Type {0}", reader["ItemType"].ToString());
      }
      Int32 CategoryIDValue = Convert.ToInt32(reader["CategoryID"].ToString());
      reader.Close();

      // Add a new row to the table and the data base.
      db1DataSet1.HouseholdInventoryRow row = table.NewHouseholdInventoryRow();
      row.CategoryID = CategoryIDValue + 1;
      row.Description = "new item";
      row.ModelNumber = "45123";
      row.Notes = "new note";
      DateTime dateTime = System.DateTime.Now;
      row.DatePurchased = dateTime;
      row.ItemName = "hammer";
      row.ItemType = "tool";
      UpdateInsert(row);

      // Display the updated data table.
      rows = table.Count;    // Display updated number of rows
      Console.WriteLine(" ");
      Console.WriteLine("number of table rows after add {0}", rows.ToString());
      reader = table.CreateDataReader();
      while (reader.Read())
      {
          Console.WriteLine("Category ID {0}", reader["CategoryID"].ToString());
          Console.WriteLine("Notes {0}", reader["Notes"].ToString());
          Console.WriteLine("Model Number {0}", reader["ModelNumber"].ToString());
          Console.WriteLine("Date Purchased {0}",
                            reader["DatePurchased"].ToString());
          Console.WriteLine("Description {0}", reader["Description"].ToString());
          Console.WriteLine("Item Name {0}", reader["ItemName"].ToString());
          Console.WriteLine("Item Type {0}", reader["ItemType"].ToString());
      }
      reader.Close();

      // Query for row with CategoryID of 3 and display values.
      row = table.FindByCategoryID(3);
      Console.WriteLine(" ");
      Console.WriteLine("Table row found");
      Console.WriteLine("Category ID {0}", row.CategoryID.ToString());
      Console.WriteLine("Notes {0}", row.Notes.ToString());
      Console.WriteLine("Model Number {0}", row.ModelNumber.ToString());
      Console.WriteLine("Date Purchased {0}", row.DatePurchased.ToString());
      Console.WriteLine("Description {0}", row.Description.ToString());
      Console.WriteLine("Item Name {0}", row.ItemName.ToString());
      Console.WriteLine("Item Type {0}", row.ItemType.ToString());

      // Modify the Notes and Date Purchased values and Update.
      db1DataSet1.HouseholdInventoryRow newRow = table.NewHouseholdInventoryRow();
      newRow = row;
      newRow.Notes = "modified note to show that updated";
      dateTime = System.DateTime.Now;
      newRow.DatePurchased = dateTime;
      Update(newRow,row);

      // Display data once again.
      Console.WriteLine(" ");
      Console.WriteLine("Table rows after update of row with CategoryID of 3");
      reader = table.CreateDataReader();
      while (reader.Read())
      {
          Console.WriteLine("Category ID {0}", reader["CategoryID"].ToString());
          Console.WriteLine("Notes {0}", reader["Notes"].ToString());
          Console.WriteLine("Model Number {0}", reader["ModelNumber"].ToString());
          Console.WriteLine("Date Purchased {0}",
                            reader["DatePurchased"].ToString());
          Console.WriteLine("Description {0}", reader["Description"].ToString());
          Console.WriteLine("Item Name {0}", reader["ItemName"].ToString());
          Console.WriteLine("Item Type {0}", reader["ItemType"].ToString());
      }
      reader.Close();

      // Close the connection.
      adapter.Connection.Close();

      // Allow viewing of console output.
      Console.WriteLine(" ");
      Console.Write("Use ENTER/RETURN when finished viewing output: ");
      Console.Read();

  } // end method ReadUpdateReadwithDataSetDesignerClasses

  private static void Update(db1DataSet1.HouseholdInventoryRow newRow,
                             db1DataSet1.HouseholdInventoryRow oldRow)
  {
      // This method Updates a current row to the dataset and to the
      // database.

      adapter.Update(newRow);
      adapter.Update(newRow.Notes, newRow.ModelNumber, newRow.DatePurchased,
                     newRow.Description, newRow.ItemName, newRow.ItemType,
                     oldRow.CategoryID, oldRow.ModelNumber,
                     oldRow.DatePurchased, oldRow.Description,
                     oldRow.ItemName, oldRow.ItemType);
  }

  private static void UpdateInsert(db1DataSet1.HouseholdInventoryRow row)
  {
      // This method Updates a new row to the dataset and Inserts the row
      // into the database.

      table.AddHouseholdInventoryRow(row);
      adapter.Insert(row.Notes, row.ModelNumber, row.DatePurchased,
                     row.Description, row.ItemName, row.ItemType);
  }




Thursday, January 12, 2012

Launch of User Apps by Display App


Launch of User Apps by Display App

I have been thinking about having the first launched application launch the others in the configuration.  The Visual C# interface to Windows functions seems to allow the launch of both local and remote applications – that is, applications on the same PC as the already running application/process and applications that are on another PC.  The older Win32Ada interface only seems to have a function to launch another process on the same PC as the invoking application/process.

I have just modified the Visual C# display application to start the user applications of the local PC.  It first checks if the application is already running.  Then, if not, it starts the application process.  This turned out to be very easy.

Nothing has been done as yet about checking whether the application is running on a remote computer or to start an application on a remote computer as can be specified in the configuration.  The change was only made to the display application since the user applications would need access to recent Windows interface functions.  Therefore, to use this feature the operator needs to start by launching the display application.

There is the possibility of race conditions.  That is, the currently running processes are obtained and then a check is made for each user application in the configuration to determine if it is in the list of running processes.  If not, it is launched.  Therefore, there is the possibility that the user application could be independently launched in the interval between obtaining the list of running processes and programmatically launching it due to not being in the list.  This is highly unlikely to happen since a single operator would be running the set of applications (that is, Windows processes).

This change required three modifications to the display application.  First, an additional configuration field for user applications was extracted – the application executable path and name.  (The PC of the application will also need to be extracted and stored when also do remote PC applications.) 

Second, this application path and name was stored in the internal table (C# struct) of the instance of the Program class as executable as well as adding additional entries for whether the application is running (opened) and the process name (processName) portion of the executable path.

Third, the Remote class Initialize method was changed to invoke a new runRemoteApplications method.  This new method was implemented as

private void runRemoteApplications()
{ // Launch the other applications of the configuration

    // Get the list of process identifiers.
    System.Diagnostics.Process[] aProccesses =
      System.Diagnostics.Process.GetProcesses();

    string processName = "";
    for (int r = 0; r < Program.userAppConfiguration.count; r++)
    {
        if (!Program.userAppConfiguration.opened[r])
         { // check if the application is running
            for (int i = 0; i < aProccesses.Length - 1; i++)
            {
                if (aProccesses[i].ProcessName ==
                    Program.userAppConfiguration.processName[r])
                {
                    processName = Program.userAppConfiguration.processName[r];
                    // application is running via manual start
                    Program.userAppConfiguration.opened[r] = true;
                    break; // exit inner loop
                }
            } // end for
        }

        if (!Program.userAppConfiguration.opened[r])
        { // Start the application
            try
            {
                Program.userAppConfiguration.userProcess[r] =
                  System.Diagnostics.Process.Start(
                    Program.userAppConfiguration.executable[r]);
                Program.userAppConfiguration.opened[r] = true;
            }
            catch { }
        }

    } // end remote user app of configuration search

} // end method runRemoteApplications

This worked on the first attempt, whether or not a user app had already been launched.  So I give myself a pat on the back.

Monday, January 9, 2012

Addition of Handshake to Display Protocol


Addition of Handshake to Display Protocol

Recently I changed the display widgets / windows controls to be disabled in the Definition File (DF) so disabled when the default forms are initially displayed.  Then, when the connection between the user app and the display app occurs, the user app enables the form and hence any controls that were enabled in the Definition File. 

The DF was changed so that widgets/controls that only need the user app that interfaces with the display layer remained initially enabled so that as soon as the parent form was enabled these widgets became active and usable.  However, widgets/controls that had user code that required communication with another user app to fully determine the command to be transmitted to the display were disabled in the DF.  Then, a command to enable them was sent when the user app to user app connection was established.

With these changes, the controls couldn't clicked, etc until the user application(s) could treat the event.

At times, after these changes were made, the form would never be enabled or the widgets that required connection another user app would never be enabled.  This seemed to occur, although perhaps not in every instance, when the form was hidden under the other form.

I have not discovered whether the display layer failed to receive the command or whether Windows failed to change the status of the control or whether Windows changed the status of the control but failed to update the displayed form and its widgets.  Therefore, I have been adding a handshake to the Display Protocol for communication between a user app and its display app layer.

In the display app, when the command has been received and acted upon to update the display, an acknowledge "event" is sent to the sender of the command with the sequence number of the received extended header as data to identify the received command.

In the user app, each command that is to be transmitted is added to a list of sent commands along with the current value of a timeout counter.  Each acknowledge that is received removes the command with the corresponding sequence number from the sent commands list.  A periodic timeout subcomponent was added to the framework remote component to increment the timeout counter in a thread safe manner.  (That is, the periodic timeout thread only updates the timer and then sends an event to wakeup the remote component main thread.  All other processing is down in the remote component main thread so that the two threads are not both using the same data except for the main thread making a copy of the current value of the timeout counter.)

When the remote main thread receives notification that the timeout counter has been updated, it checks the sent command list for old entries.  Any such entries have their sequence number updated to the next value along with the current timeout counter and the command message is resent to the display.  This should take care of any problems due to the display app missing the command.

In case Windows fails to change the enabled status of a form or widget under some circumstances, I will next modify the display app to check that the Windows interface reports the same status for a control after the display app attempted to change it as the command message specified.  Only if all the widgets specified in the command have their updated status will the acknowledge "event" message be sent to the user application.  Therefore, if Windows doesn't report back the new status the user app will resend the command.

After this second change, if the display ever doesn't match the commanded state then I think the problem would be that Windows updated the state but failed to update the displayed form.

Thursday, November 17, 2011

Restructuring of Framework Remote Component Update 1


Restructuring of Framework Remote Component Update 1


Lack of Topic Data Buffer Release

Immediately after publishing the last post – Restructuring of Framework Remote Component – I found that Remote Register Complete topic still wasn't treated correctly in all circumstances.

Looking into this I finally found that it didn't really have anything to do with this topic per se.  Instead, it was due to a topic that the framework, upon its being published, determines that it is to be transmitted.  When this is the case the framework copies the data to a record that is added to the Transmit Queue and invokes a procedure to release the data buffer.

The problem is that the Release procedures, except for one, only release the data buffer if it has been marked as being read; i.e., that a Reader has been instantiated for the topic instance.  When the framework just copies it into the Transmit Queue this doesn't happen.  Therefore, the data buffer wasn't being released.  Since the topic was specified as one to use multiple buffers the next time a writer for the topic was instantiated for the same producer another data buffer was assigned. 

Since the Remote component is both the publisher and the consumer of this topic, a new buffer could be assigned upon the publish of a received message from another application.  Then, when Remote, as the consumer component, read the topic it got the oldest of the multiple data buffers.  Depending upon the timing, this could be the instance that the application transmitted to another app rather than the one received.

In trying to determine why this was taking place I happened to see the one Release procedure that also checked whether the Reference Numbers matched when the Being Read boolean was not set.  I reasoned that this was because at some time in the past I had found the same problem for a different method of topic delivery and had corrected the problem in this manner.  That is, looking for a matching Reference Number when the topic didn't have Being Read set indicating that it had been transmitted rather that delivered to a local component that had instantiated a Reader.

I updated each of the various Release procedures to also make a similar check if the procedure was being called following transmit.  This fixed the problem.


Use of Remote Register Complete message

While locating the source of the seeming mis-delivery of the Remote Register Complete topic, the reason for needing this topic was well illustrated.  That is, the Aperiodic component of App2 sent a N-to-One request to App1 each time that it ran if there was a connection to App1.  However, until the Remote Register messages were received and treated sometime after the connection, the remote app – in this instance App1 – didn't know what to do with the Response.

Therefore, the response wasn't transmitted back to App2 and its data buffer didn't get released.  In this situation the topic wasn't one that specified multiple data buffers.  Therefore, the next time such a topic was received from App2, the response couldn't be created because the data buffer was still in use.

Also, a table exists to match responses to received requests to look up where to transmit the response.  With no response to match a request, received requests kept being added to the table until its limit was reached.

This was fixed – as point 4) in the previous post mentioned that it would be – by changing when the Writer could obtain a data buffer for a topic to be transmitted.  Instead of only requiring that there be a connection to the remote app that is to consume the topic, another consideration was added.  When the topic is a non-registration topic, Remote Register Complete also has to have been received from the remote app to which the topic message is to be transmitted.

This prevented topic data buffers being created, and hence written and published, until the remote application is ready to fully treat them.

Tuesday, November 8, 2011

Restructuring of Framework Remote Component


Restructuring of Framework Remote Component

 Now that the treatment of the message protocol for communications between the user applications and the display application had been incorporated into the mC framework it became time to restructure the Remote Component to treat the interface to the Display application using the same Ada packages and methods as those for other User applications. 

This took me longer than expected and is not yet finished.  Mostly however because some features were also added that are not strictly related to employing the user application Remote methods in common with the display interface to avoid the duplication that occurred when the cloned user display components were moved into the framework Remote component.  Four of these were

   1) to get application App3 that contains C and C++ user components up-to-date after having been ignored for quite some time while a number of framework changes were being made.
   2) to use Windows interfaces to check which of the other applications of the configuration of the current PC are running,
   3) to use Windows to do the same for a different PC in the configuration,  and
   4) to send a Remote Register Complete topic between running user applications to indicate to a connected application that the sending application has completed its Remote Registration with the receiving application.  Sometime in the future this can be used to avoid transmitting other topics until the sending application is able to determine that the connected application has completed its remote registration with the sending application and hence is ready to treat remote topics.

Another feature to be added will be to monitor the communications between the user application and the display application to check that the two applications remain connected or, if become reconnected, to continue as best as possible from where communications was lost.  Yet another (structure only change) will be to package the remote registration procedures into a Registration unit for easier recognition of their common rationale.

An unrelated additional feature will be to have the first running application of the configuration launch the others.

As part of this change, the configuration file was changed to have a user applications portion and a display applications portion so that one such file can be used both by the user application framework and by the display application.  The user application framework needed this change as part of incorporating the treatment of the display interface via the same methods as the user applications.

While making these changes an error was detected where the application identifier was treated (in various places) as an index into arrays of data concerning applications of the configuration and connected applications.  This previously had not become apparent since the debug cases had been for user applications 1, 2, and 3.  When I added the display application for common treatment, I separated the possible values for the identifiers into one range values for user applications and another range for display applications.  Thus the display application came to be assigned an identifier of 15 while the arrays were sized as 1 to 5 so that a direct index of the identifier could no longer be used.  Further, I could find this sort of thing better if I switched app 2, for instance, to app 5 to have gaps in the user app identifiers which I will need to do.
Above is a figure showing how the user Display components were previously moved into a common framework Display component. 

The following second figure illustrates how the Display interface now uses the same Methods as the user components and how, in the future, it will use the same communications Monitor package and package the remote registration into its own unit.
The new diagram is meant to show that the Remote Component with its thread (the top middle oval to indicate the thread – enclosed icon to indicate the Ada package) includes inputs from and outputs to the tables and reads the Transmit, Receive and Watch Queues upon receiving a wakeup event (that is, the notify of a dataless topic that is published as part of adding the entry to the queue).

The Transmit Queue is written via the publish of a topic from various components while the Receive Queue is written to contain messages received from other applications including the Display App via the receive ports of the MS_Pipe and WinSock communication methods that each have a thread that blocks to wait for a new message from its associated application. 

Received display messages are passed to the Display subpackage that converts them to instances of the Display Event Request topic message and then publishes them for distribution.  In this case the Remote component also queues to the Transmit Queue. 

A received Display Event Response topic (dequeued from the Receive Queue) as well as an unsolicited Display Command topic message are treated via Display as received by the framework (while running under the publishing component's thread).  The Display package supplies the Display protocol header in place of that of the Topic protocol and transmits to the display app via the selected Remote Method.

Hence the Display package is the Display Protocol to Topic Protocol converter and vice versa.  This design, as mentioned in the previous post, allows for the use of other protocols to be added; such as an A661 Protocol.  Only another protocol converter need be added, just as can be done for communication methods, by adding another supported method to be selected by the Method package.  When this becomes necessary a protocol selector package should be added to invoke callback procedures of the particular converter package as is done by Method to select those of MS_Pipe or WinSock.  Other communication drivers can, of course, be added as well where Method would then also contain callback procedure entries to the new method.

Monitor / Registration

The Monitor package will be discussed after monitoring of the user-display application traffic has been implemented.  The Registration package will just be grouping the remote register procedures within such a package. 

Windows Interface to Detect Running Applications

There are supposed to be newer Windows interfaces to return whether a particular process (that is, application) is running on the current PC or on a named one.  Others to start a process on the current PC. 

Since these interfaces don't seem to be available via the GNAT libraries (and certainly not via Win32Ada that contains interfaces to much older versions of Windows), I attempted to access them through a Visual C++ function and link it via gnatlink with the rest of exploratory project.  After various attempts I put that aside to return to later.

I can, however, detect if a specific application is currently running via the Windows interfaces provided with GNAT and Win32Ada.
  BOOL WINAPI EnumProcessModules
  ( __in   HANDLE hProcess,
    __out  HMODULE *lphModule,
    __in   DWORD cb,
    __out  LPDWORD lpcbNeeded
  );
can be used.  To do this, the appRunningC.c function was created and imported to Ada via
  pragma Import(C, App_Running, "appRunningC");

The Ada declaration used was
  function App_Running
  ( Application : in Interfaces.C.Strings.Chars_Ptr
  ) return Interfaces.C.char;
where the Chars_Ptr is used to point to a null terminated character array containing the name of the application along with its path as obtained from the Apps-Configuration.dat file.  The returned character is typecast to a byte value to indicate whether the application is running or not.

The code of appRunningC is
#include <mC-ItfC.h>
#include <ctype.h>
#include <psapi.h>
#include <stdio.h>
#include <string.h>
#include <tchar.h>
#include <windows.h>


// Microsoft Help comment:
//   To ensure correct resolution of symbols, add Psapi.lib to TARGETLIBS
//   and compile with -DPSAPI_VERSION=1
// My comment:
//   For GNAT, include C:\GNAT\2011\lib\gcc\i686-pc-mingw32\4.5.3\libpsapi.a
//   in the gnatlink command. 


void strConvert( char * inStr, char * outStr )
{ // Convert string to all lower case while switching any forward slash
  // in pathname to a backward slash
  int i = 0;
  while (inStr[i] != 0)
  {
     if (inStr[i] == '/')
     { outStr[i] = '\\';
     }
     else
     { outStr[i] = tolower(inStr[i]);
     }
     i++;
  } // end loop
  outStr[i] = 0; // attach trailing null


} // end method strConvert 


int MatchModulePathname( DWORD processID, char * Application )
   { // Check application name at beginning of module list for process for match.


    HMODULE hMods[1024];
    HANDLE hProcess;
    DWORD cbNeeded;
    unsigned int i;


    // Get a handle to the process (that is, application).
    hProcess = OpenProcess( PROCESS_QUERY_INFORMATION |
                            PROCESS_VM_READ,
                            FALSE, processID );
    if (NULL == hProcess)
        return 0; // false


    // Get a list of all the modules in this process.
    if ( EnumProcessModules(hProcess, hMods, sizeof(hMods), &cbNeeded) )
    {


        // Only the first name will contain the application name of the process.
        TCHAR szModName[MAX_PATH], ModName[MAX_PATH];


        // Get the full path to the module's file.
        if ( GetModuleFileNameEx( hProcess, hMods[0], szModName,
             sizeof(szModName) / sizeof(TCHAR)) )
        {
            strConvert( szModName, ModName );


            // Check if module name with path matches that of the Application.
            if (strcmp(Application,ModName) == 0)
            {
               // Release the handle to the process and return found.
               CloseHandle( hProcess );
               return 1; // true
            }
        }


    } // end if ( EnumProcessModules(hProcess, hMods, sizeof(hMods), &cbNeeded) )


    // Release the handle to the process.
    CloseHandle( hProcess );
    return 0; // false


} // end method MatchModulePathname


extern "C" char appRunningC(char * Application )
{ // Determine if Application is running.
    DWORD aProcesses[1024];
    DWORD cbNeeded;
    DWORD cProcesses;
       unsigned int i;


    // Get the list of process identifiers.
    if ( !EnumProcesses( aProcesses, sizeof(aProcesses), &cbNeeded ) )
        return 2; // Application cannot be found due to problem


    // Calculate how many process identifiers were returned.
    cProcesses = cbNeeded / sizeof(DWORD);


    // Examine module names for each process for match to input.
    char AppPath[MAX_PATH];
        strConvert(Application, AppPath);
         for ( i = 0; i < cProcesses; i++ )
         {
            if (MatchModulePathname( aProcesses[i], AppPath ) == 1)
            {
                return 1; // Application found
            }
        }


    return 0; // Application not found


} // end method appRunningC

This routine is currently being used to check whether each application of the configuration (other than the application that is doing the checking) is running.  This can be changed to pass a list of all of the other applications in the configuration to appRunningC and return an array of those that are running to avoid multiple calls to the OpenProcess and EnumProcessModules functions.

GNAT Compiler

While making the various changes it seemed that GNAT GPS no longer rebuilt changed separates.  This seemed to be the case after the addition of the C function for use by the Ada framework Remote-Method-MS_Pipe package to interface to Windows to check what applications of the configuration were running.  Therefore I had to start compiling the packages that I changed with statements such as
c:\gnat\2011\bin\g++ -c -g -IC:\Source\EP\UserApp\Try10 -IC:\Source\EP\Util
  -IC:\Win32Ada\src -aLC:\Source\EP\ObjectC
  C:\Source\EP\UserApp\Try10\mc-message-remote.adb -o mc-message-remote.o
in a batch file (where the indented lines are actually part of the first line) as well as the .c file.  The -g switch includes debug symbols in the .o object file to be included in the eventual .exe file by gnatlink.

Remote Register Complete topic

The implementation of the Remote Register Complete topic took longer than expected.  This topic is both produced and consumed by the Remote framework component.  However, the instance produced by the Remote component of one application is meant for that of another application. 

Therefore the topic used the recently added Requested Delivery method of message exchange so each application could register to consume a delivery identifier that equaled its application identifier.  The Remote component of the publishing application supplies the delivery identifier of the application to receive the topic as it publishes the topic.  Therefore there are as many possible publishers of the topic as there are applications and the same number of consumers.

This is as the message exchange delivery method was intended to treat.  The difference in this case was that the same component was both a publisher and a consumer so its Role was Either.  Since this Role hadn't been previously used, this caused a few problems that had to be tracked down and corrected.  However, another problem caused confusion in identifying what problems were caused by this new role.

That was due to one application of a pair seeming to transmit its instance of the topic to the other application and that application seeming to create its instance but not doing the transmit.  Finally (I must be getting old) I identified the reason.

The registering of a delivery identifier to be consumed involves, as has been done in the past, two variables where the second one is referenced in the Register call.  The first of these is an array of identifier pairs indicating five possible pairs of delivery identifiers that indicate the identifiers the component is to treat.  Each pair indicates a range of values where the second entry can be 0 to indicate that there is no range; only the particular identifier of the first entry.  The second variable contains the count of the number of pairs to examine and the doubly indexed array of values.  Normally, of course, the count will be one.

Registering to consume the Remote Register Complete topic, the Remote Install procedure declared
    Delivery_Id_List
    --| Treat all "Remote Register Complete Topic" messages with ids of the app
    : mC.Itf_Types.Delivery_Id_Array_Type
    := ( ( 1, 1 ), ( 0, 0 ), ( 0, 0 ), ( 0, 0 ), ( 0, 0 ) );
    Delivery_Id
    --| Deliver topic when identifier is that of local app
    : mC.Itf_Types.Delivery_List_Type
    := ( Count => 1,
         List  => Delivery_Id_List );
mimicking what had been done in the past for other Requested Delivery and Delivery Identifier delivery methods.  Except in those cases, the Delivery_Id_List object was declared as constant since the identifiers were known at compile-time. 

In this case, since the pair of delivery ids depends upon the application's own identifier – that isn't known until initialization-time, the value was initialized as shown above and then modified just prior to the Register call to contain the running application's id as shown below.


  Delivery_Id_List := ( ( Integer(Local_App.Id), Integer(Local_App.Id) ),
                        ( 0, 0 ), ( 0, 0 ), ( 0, 0 ), ( 0, 0 ) );


  mT.Remote_Register_Complete_Topic.Request.Data.Register
  ( Participant => Component_Main_Key,
    Detection   => mC.Itf_Types.Event_Driven,
    Role        => mC.Itf_Types.Either,
    Delivery_Id => Delivery_Id,
    Callback    => Monitor.Treat_Register_Complete'access,
    Access_Key  => Remote_Register_Complete_Topic_Access_Key,
    Status      => Data_Status );
where the Count in the Delivery_Id variable remains as 1.

Since this kind of Register (other than the use of Either for the Role instead of Consumer) had been working on the order of a year or more it didn't occur to me that this late resetting of the Delivery_Id_List could be causing any problems. 

However, I finally determined that the compiler was ignoring the resetting of the value to be used by Delivery_Id in the object code that it generated.  So both applications of the pair were specifying that they would consume delivery id 1.  I changed the source code to


  declare


    Delivery_Id_List
    --| Treat all "Remote Register Complete Topic" messages with ids of the app
    : mC.Itf_Types.Delivery_Id_Array_Type
    := ( ( Integer(Local_App.Id), Integer(Local_App.Id) ),
         ( 0, 0 ), ( 0, 0 ), ( 0, 0 ), ( 0, 0 ) );


    Delivery_Id
    --| Deliver topic when identifier is that of local app
    : mC.Itf_Types.Delivery_List_Type
    := ( Count => 1,
         List  => Delivery_Id_List );


  begin

    mT.Remote_Register_Complete_Topic.Request.Data.Register
    ( Participant => Component_Main_Key,
      Detection   => mC.Itf_Types.Event_Driven,
      Role        => mC.Itf_Types.Either,
      Delivery_Id => Delivery_Id,
      Callback    => Monitor.Treat_Register_Complete'access,
      Access_Key  => Remote_Register_Complete_Topic_Access_Key,
      Status      => Data_Status );


  end;

This fixed the problem with Remote of each application transmitting the instance of the topic that was published by it to the other application of the pair where it was then delivered to the
Treat_Register_Complete procedure of the Monitor subpackage.

This might be a case that could have been fixed by pragma Volatile if it had been recognized.  That is, an example of the same thing that can happen when an object is changed by one thread and read by another without the new value being used by the code that was generated by the compiler due to its not recognizing that it could be changed.  (Or, as happens when an address is passed to a procedure that then uses it to change a value used later by the calling routine.)

The remaining gotchas were then immediately found and fixed.

Wednesday, September 21, 2011

Display Protocol Moved to Framework and Added Message Exchange Method


Background

As part of the monitoring of the Exploratory Project communications in order to detect a dropped connection I decided it was time to move the treatment of the communications with the Display application from user components to the framework.  This was overdue since, with the addition of a second user application to interface to the display application, the user component had to be cloned and would have to be cloned again for every user application that needs to interface with the Display application.  And, as part of the framework, additional protocols can be supported (as specified in the configuration files) and additional communication interfaces such as the WinSock interface supported for communications between user applications.

When I did this I got a surprise due to a fading of my memory of what I had had to do when I had added the second user application to interface with the display.  With the making of the user component into a framework component I had to remove my cloned topics that the user components supported so as to have only one Display Widget Event topic to translate a display protocol event message into a framework topic.  This because can't continue to add cloned topics since now a part of what should be a stable framework.  Therefore, it was necessary to come up with a way that the common framework of each user application could determine when the Display Widget Event topic needed to be forwarded to a remote user app when the local app also had a consumer for the topic.

In addition, the move of the display protocol to the framework highlighted another problem with the previous framework topics.  That is, I originally created the Delivery Identifier message exchange topic category knowing that I wanted to be able to forward a display widget event to the particular component that registered to treat the particular event identifier.  That is, one topic produced by possibly multiple components and only delivered to the one of multiple potential consumers that registered to treat the particular delivery identifier.

When it came time to actually implement this the treating component had to send a display update command topic to be translated to the display protocol message to be transmitted to the display application.  At the time, this was handled by using a Request / Response message exchange topic category (with no response requested) with the particular user display component registering to consume the request.  Again, with the display component becoming part of the framework, it would only be able to register for a common single such topic since not able to treat extra cloned topics as additional user applications interface to the display.

This again was something I should have recognized at the time that I added the interface of the user application with the display and had to have the user component respond to the received widget event.

Requested Delivery Message Exchange

Therefore, while implementing a common Display Widget Event topic to be published by the framework display component upon the receipt of the display protocol message, I created a new message exchange topic category of Requested Delivery.  This category combines Delivery Identifier and Request / Response where there can be multiple producers of the topic (in this particular case the multiple instantiations of the framework display component – one per user application) and multiple consumers with only one particular consumer registering to treat a particular delivery identifier.  This is the same as Delivery Identifier.  In addition, there is a paired response topic as with Request / Response where the response is returned to the particular request publisher. 

Instead of One-of-N delivery (one consumer of N possible) as with Delivery Identifier and N-to-One delivery (N producers to one possible consumer with the response returned to the particular request producer) as with Request / Response, there is One-of-N-to-One delivery where the message is delivered to one consumer of N possible consumers and the response is delivered to the particular request producer.

Therefore, the framework of one user application can receive a Widget Event message from the Display application, translate it to the common Display Widget Event topic and publish it.  The framework then determines which component of what application registered to treat the widget id of the message and forwards it to that component.  The component treats the event and creates the new instance of the response topic and publishes it.  The framework then returns the topic instance to the component that published the request.  Since this will be the framework display component of a particular user application, it converts it to a display protocol message and transmits it to the Display application.

The implementation of this new Requested Delivery message exchange resulted in a number of challenges to be solved.  That is, when there were cloned topics, the consumer of a particular delivery id by a particular component could be determined at topic registration time and the publisher of the particular cloned topic was known.  With the change to Requested Delivery, the consumer can be known but not the particular producer since the delivery id to be produced isn't known until it is received from the Display app and hence published by the framework display component.  For instance, the framework Display component of each user application will register as the producer of the Display Widget Event request message.

The registration of the producer of a particular delivery identifier has to be delayed until the topic is actually published with the delivery id although it continues to register as a producer of the topic in general at registration time.  To do this, another group of items was added to the Register Request messages exchanged by the user apps along with tables to know what components have registered to consume particular delivery ids.  This group of items contains all the components that have registered to consume the Requested Delivery topic along with the delivery ids that each component has registered to consume.  Then, when a topic is published, the table created due to these lists can be used to determine what application registered to consume the particular identifier.

If the consuming component is in a remote application, the framework sends the topic to that application.  The framework of that application then looks it up in the table used prior to this change to determine the component to which to forward the topic.  If not in that table, it registers the topic and component as specified in the newly added table so it can be processed in the usual way for the response and any additional instances can be found as received.  That is, the producer of the delivery id is now known so the framework can determine the component to which to return the response.

With these changes, messages such as those received from the display can be forwarded to a user component that has raised its hand (registered) to treat a particular instance of the topic (as identified by a numeric value) and the resulting response message can be returned.  And there can be multiple components that produce the topic although, hopefully, not with the same numeric identifier. 

This is the case with the ARINC-661 protocol where different layers can be interfaced with different user applications – one layer to one user application.  Or, with the ARINC supplement 3 draft document (http://wn.com/ARINC_661), one application treating multiple layers as long as the widget identifiers of the layers are unique.  There must also be other examples where this new message exchange category can be applied.
 

Framework Component Structure

The structure of the components of the framework that are directly involved with communications is illustrated in the Framework Components diagram.

Note, framework components are similar to user components except for being an integral part of the component.  That is, they register as a component with the framework, register the topics that they will produce and consume, have a process/thread in which to execute, etc.  They are dissimilar in that they can reference various framework types and variables.

As shown in the diagram, there is a Remote package installs itself as a component while requesting that its non-install code run under a Remote process thread.  In this package are a group of tables that are used to determine topics to be transmitted and validate received messages as to whether they are expected.  (If not, they are rejected.) 

Validated received messages (each received in one of the Receive process threads of MS_Pipe or WinSock packages – one thread per possible remote user application) are added to the Receive Queue and with the framework then signaled to run the Remote thread.  This also updates the Connections package watch tables used to monitor whether reconnect attempts are needed.


Messages to be transmitted are added to the Transmit Queue and the framework is signaled to run the Remote thread.  (Each time the thread is run it checks both queues.)  Messages in the Transmit Queue are passed along to the Method package to transmit the message by each supported and connected communication method as implemented in transmit procedures of the MS_Pipe and WinSock packages.

The framework also wakes up the Remote thread periodically to send heartbeat messages in case components haven't queued Topic messages to be transmitted.

The Display interface is implemented within a Display package.  It installs Client and Response components where its install registers with the framework to provide corresponding Client and Response process threads.  Received display messages are translated to Requested Delivery request topic messages in the Parse package and then published for the framework to deliver to the consumer of the widget identifier.   The Client component acts as if it is the Response component when doing the publish so that the response will be delivered to the Response component.  The Response component then transmits the response, via the display protocol, to the Display app.

If the framework determines that the consumer for the published Requested Delivery request is in a remote application, it uses the Remote component by inserting the message in the Transmit Queue.

Maintaining the Connections status for the communications with the Display has yet to be implemented.  The initial move of the display protocol to the framework retained the previous user component structure for Client and Response while adding the Method package to allow for different methods to be supported.  While doing the display communications monitoring the use of Remote Connections and common method packages will be examined. 


Protocols

With the move of the Display protocol to the framework it became necessary to be able to distinguish between the protocol that the framework previously implemented and the new one that it can also handle.  As well as the ability to designate other protocols in the future.  Therefore, as mentioned in the most recent previous post, the prior framework protocol will now be referred to as the Topic Protocol while the current protocol between the user apps and the display app will be referred to as the Display Protocol.  This is because it isn't really ARINC-661 and naming it the Display Protocol will allow a future implementation of ARINC-661 to be referred to as the ARINC-661 Protocol.

So now the framework supports Framework Topic protocol messages and Display protocol messages in a more or less common structure.