Is this a Use Case for DataAbstract?

Data Abstract
1

I have a scientific application, that generates a huge amount of time series data, say, every 10 milliseconds. I saved the data to a database table - let’s call it RAW_DATA_TABLE

The RAW_DATA_TABLE has a data resolution of 10 milliseconds.

I need to do some statistical analysis to generate 5-minutes aggregate of the raw data, for the past 48 hours. That is, the 48 hours RAW_DATA will need to be grouped into 5-min bins. This process will generate new data, to be saved in a new table DATA_5MIN_TABLE

The bin-size can also be 15 minutes, so it could be purely a user input.

I would like to move the computation of the statistical analysis into some middle tier, leaving the backend database only have RAW_DATA.

The problem is, the computation itself is time consuming, if performed on the fly.

Will this fit what DataAbstract does well? Can DataAbstract server l do “prefetch” and do the computation in the background silently, so when a user queries, the processed data will be always ready for the immediately past 48 hours data, rather than doing the data aggregation on demand?

What would be the best practice?

Thanks

Wuping

2

Hi,

DAD server can perform this.

calculated data can be stored in TDAMemDataTable on server-side, so from client-side sight this table can be used as usual DB table.

for this, setup DAMemDataTable(s) as virtual datasource for DAService:

  • drop TDAMemDatatable to server DataModule
  • create required fields manually
  • set table.RemoteFetchEnabled to False, so it will work w/o remote datasource
  • register table in daservice.ExportedDataTables.
3

EvgenyK

As always, thank you.

I understand DataAbstract provides client library to deal with tables.

Is it possible that a DA server acts like a database server transparently to a client? For example, to a client, it just acts like a MySQL server? I am asking to see if I can hook up a DA server as SQLDataSource to DevExpress Dashboard (.NET).

Thanks again!

Wuping

4

Hi,

DAD server provides IDataAbstractService interface that can be consumed by any DataAbstract client, including DataAbstract for .NET.

DAN classes allows to work with standard DataTable object that can be used as datasource for DevExpress classes.

5

Thank you for the advice.

I am aware that DA for .NET uses ADO.NET, while DA fir Delphi uses third party drivers such as UniDAC.

Any experience or insights that which is generally faster in terms of the connection between the middle tier and the backend database drivers? Is ADO.NET generally slower than UniDAC which has direct connection drivers?

Thank you.

6

we didn’t perform such tests between different platforms.
if you think, that Devart drivers better than default ADO.NET driver, you can use ADO.NET driver from Devart instead of default one

7

I think I’ll stick to the Delphi version, as UniDAC supports more databases (for example, MangoDB) than their ADO.NET driver.