Data Table - Shared between host and plugin

Matthew_Vesperman · September 25, 2019, 1:19pm

Hello,

I was wondering what would be the best method of sharing database tables between the host and plugins.

Is this something that could be done using RO SDK or DataAbstract?

Thanks

EvgenyK · September 25, 2019, 1:53pm

Hi,

what platform you are using (Delphi, .NET, etc)?

Matthew_Vesperman · September 25, 2019, 2:55pm

The app is using both.

Delphi Host
.NET Plugins

EvgenyK · September 25, 2019, 3:16pm

Hi,

you can save TDAMemDataTable (or IDADataset) to stream, and pass it as array of bytes to other side, and read it on other side to datatable.
in the same way, you can pass delta changes back.

Data Abstract is required.

Matthew_Vesperman · September 25, 2019, 3:31pm

Thanks.

Off-hand is there a sample of this anywhere?

Also, can this be done without creating a DA server/client setup?

EvgenyK · September 27, 2019, 9:17am

DA server/client setup isn’t required.

Delphi side:
imagine you already have some TDAMemDataTable with data:

// write datatable to stream
DABin2DataStreamer.Initialize(MemoryStream,aiWrite);
try
  DABin2DataStreamer.WriteDataset(DAMemDataTable,[woRows, woSchema]);
finally
  DABin2DataStreamer.Finalize;
end;

// move content of stream into `array of bytes` (i.e. TBytes)
MemoryStream.Position := 0;
SetLength(bytes, MemoryStream.Size);
MemoryStream.Read(bytes, MemoryStream.Size);

.NET side:

dataStreamer.InitializeStreamer(remoteData, StreamerInitialization.ReadFromBeginning);
dataStreamer.ReadDataTable(tableName, table, applySchema, true);
dataStreamer.FinalizeStreamer();

Matthew_Vesperman · September 27, 2019, 2:51pm

Thanks, that looks perfect for my first issue.

On another note is it possible to use Data Abstract to allow a Delphi Hydra Host to act as a client to a Hydra .NET plugin acting as a server, communicating using named pipes?

EvgenyK · September 27, 2019, 2:54pm

yes, it should be possible too.

Matthew_Vesperman · September 27, 2019, 2:59pm

I was hoping so, thanks.

Matthew_Vesperman · September 27, 2019, 3:03pm

Is there a step-by-step guide to manually creating the server portion inside .Net?

antonk · September 30, 2019, 10:57am

Hello

I will use a just-created empty C# console app as a starting point.
The following are detailed steps to turn this project into a Data Abstract server.

Add references to the following assemblies: RemObjects.SDK.dll, RemObjects.SDK.Server.dll, RemObjects.DataAbstract.dll, RemObjects.DataAbstract.Server.dll
Add a new item to the project: Data Abstract -> Schema File
Add a new item to the project: Data Abstract -> Connections File
Rename both newly added files as you need.
Set Build Action of both newly added files to Embedded Resource

Add the following class to the project (ServiceSchemaName value should match to the item name used at step (4)):

[RemObjects.SDK.Server.Service]
public class DataService : RemObjects.DataAbstract.Server.DataAbstractService
{
       public DataService()
       {
              this.AcquireConnection = true;
              this.ServiceDataStreamer = new Bin2DataStreamer();
              this.ServiceSchemaName = "Sample";
       }
}

Add a file to the project named licenses.licx (or open already existing one)
Double-check that build action of this file is set to Embedded Resource

Add the following lines to licenses.licx:

RemObjects.SDK.Server.IpHttpServerChannel, RemObjects.SDK.Server
RemObjects.SDK.Server.NamedPipeServerChannel, RemObjects.SDK.Server
RemObjects.DataAbstract.Server.ConnectionManager, RemObjects.DataAbstract.Server

Add the following code to the Main method (connectionManager.LoadFromResource parameter value should match to the item name used at step (4)):

static void Main(string[] args)
{
	// Load service definitions
	RemObjects.SDK.Server.Configuration.Load("SampleServer", "SampleServer");

	// Load Data Abstract configuration, including DB driver definitions
	RemObjects.DataAbstract.Server.Configuration.Load();

	// Load connection definitions from embedded resource
	var connectionManager = new ConnectionManager();
	connectionManager.LoadFromResource("Sample.daConnections");
	
	// Setup server channel
	var serverChannel = new IpHttpServerChannel();
	var message = new BinMessage();
	serverChannel.Dispatchers.Add("bin", message);

	// Start server
	serverChannel.Open();

	Console.WriteLine("Press ENTER to exit");
	Console.ReadLine();

	// Stop server
	serverChannel.Close();
}

Dbl-click the .daSchema file and define connection and schema tables as needed
Start the application

Matthew_Vesperman · October 2, 2019, 1:43pm

How would this work if I wanted to use memory tables instead of a database?

More specifically, the .NET server plugin needs to server data (from memory) to the host as a data table.

antonk · October 2, 2019, 2:08pm

Let’s go back to the roots.

What exactly do you need?

1.You need to pass a table data across .NET / Delphi boundaries. That data is not going to be changed by the receiver.
Best approach: Define a plugin interface method that returns a byte array. Call this method on the host to get data. Use streamer to encode data to a byte stream plugin-side and then use corresponding streamer host-side to convert data back to a table.
Another approach: Define a server method that would return a Binary object. Encode data table in that method and return it as a method result. Host-side call a “server” method and again use streamer to decode data. More overhead.

2.You need to pass a table data across .NET / Delphi boundaries. That data can be changed host-side.
Best approach:
Create in-memory SQLite database. Put there your data. Let the DataAbstract data service do the work for you.
Pros - host-side all DataAbstract methods can be used. In-memory database has no file footprint and is really fast.
Cons - requires manual connection management to not let the in-memory database vanish

Another approach: Override the DataServer’s GetData method. There analyze the incoming requests and return the pre-encoded table data if your memory table has been requested.

Matthew_Vesperman · October 2, 2019, 3:14pm

I am pretty much looking to do #2. Although the DataServer’s GetData option sounds interesting too.

I have data in the plugin (in memory) that I want to access (read-write) from the host.

antonk · October 3, 2019, 10:49am

So, you want to use a memory-based database to store data server-side.

Let’s amend the server app I described earlier.

Go to https://github.com/Phrynohyas/FluentSQLite and take this file: https://github.com/Phrynohyas/FluentSQLite/blob/master/FluentSQLite/FluentSQLite/FluentSQLite.cs
Add to the project.

Also add this class:

 using System;
 using RemObjects.DataAbstract.Server;
 using RemObjects.SDK.Pooling;

 namespace FluentSQLite
 {
 	static class DADatabase
 	{
 		private const string DEFAULT_CONNECTION_NAME = "InMemory SQLite Connection";
     	private const string DEFAULT_CONNECTION_TYPE = "SQLite.NET";

 		public static IDatabaseContext CreateDatabase(ConnectionManager connectionManager)
 		{
 			connectionManager.PoolingBehavior = PoolBehavior.Wait;
     		connectionManager.MaxPoolSize = 1;

 	    	connectionManager.ConnectionDefinitions.Clear();

 			var connectionDefinition = connectionManager.AddDefinition(DADatabase.DEFAULT_CONNECTION_NAME, String.Format("{0}?{1}", DADatabase.DEFAULT_CONNECTION_TYPE, Database.DefaultConnectionString), true);
 			connectionDefinition.ConnectionType = DADatabase.DEFAULT_CONNECTION_TYPE;

 			var connection = connectionManager.AcquireConnection(DADatabase.DEFAULT_CONNECTION_NAME, true);

     		var database = new DatabaseContext(((BaseConnection)connection).ActualConnection);

 	    	connectionManager.ReleaseConnection(connection);

 	    	return database;
 		}
 	}
 }

Amend the Main code:

    ...
 	var connectionManager = new ConnectionManager();
 	//connectionManager.LoadFromResource("Sample.daConnections");

 	var database = CreateDatabase(connectionManager);
    ...

Add this method to the Program class:

 private static IDatabaseContext CreateDatabase(ConnectionManager connectionManager)
 {
 	return DADatabase.CreateDatabase(connectionManager)
 		.AddTable("Customers")
 			.AddField("Id", FieldType.String, true, true)
 			.AddField("Name", FieldType.String, true)
 			.AddField("Phone", FieldType.String)
 			.AddField("Address", FieldType.String)
 			.AddField("Remarks", FieldType.String)
 			.AddField("Discount", FieldType.Float)
 		.CommitStructure()
 		.SelectTable("Customers")
 			.InsertRow("ID", "Test Name", null, null, null, 42.0)
 		.CommitData();
 }

In this method a new in-memory database is being created and its structure is initialized. The a sample data row is added to it.

In your app you will need to initialize your table(s) structure according to your Schema.

Server-side to insert data into the memory table you can either use LocalDataAdapter or just use the same approach as above (store table context acquired via database.SelectTable call and add data directly to the database using InsertRow calls + CommitData). Note that while this approach is faster than LocalDataAdapter-based one you have to ensure that no other processes are trying to read or write data while you perform insert operations.