Is connectionmanager thread safe? / How to detect running transaction / async tasks?

I cannot imagine it not being the case but I’ll ask it anyway:
Is the connection manager thread safe?

I ask this because I get a ‘transaction still active’ exception.

What could be the causes of this error?
AFAIK: I only use a particular dataAdapter in one thread and I do not use async functions (being: ToDABindingList and ApplyUpdates)

for example:

using (LinqLocalDataAdapter dataAdapter = CloudServerUtil.prepareSyncMasterLocalLinqAdapter())
{
...
   var runinfoQuery = from c in dataAdapter.GetTable<OffCloudServerData.SyncMasterDAClasses.SyncRunInfo>() select c;
              var runinfolist = runinfoQuery.ToDABindingList();  // This does the SQL SELECT
...
  runinfo = new OffCloudServerData.SyncMasterDAClasses.SyncRunInfo();
...
  runinfolist.Add(runinfo);
...
  dataAdapter.ApplyChanges();
...
}
Exception: Cannot release connection because its transaction is still active  [Source:RemObjects.DataAbstract.Server]
   at RemObjects.DataAbstract.Server.ConnectionManager.ReleaseConnection(IAbstractConnection connection, Boolean keepInPool)
   at RemObjects.DataAbstract.Server.DataAbstractService.InternalReleaseConnection(IAbstractConnection connection, Boolean keepInPool)
   at RemObjects.DataAbstract.Server.DataAbstractService.BeginTransaction(IAbstractConnection& connection)
   at RemObjects.DataAbstract.Server.DataAbstractService.GetDataViaDASql(String tableName, TableRequestInfoV6 requestInfo, DataStreamer streamer, Boolean allowPrivateAccess)
   at RemObjects.DataAbstract.Server.DataAbstractService.InternalGetData(String[] tableNames, TableRequestInfo[] requestInfo, DataStreamer streamer, Boolean allowPrivateAccess)
   at RemObjects.DataAbstract.Server.DataAbstractService.LocalGetData(String[] tableNames, TableRequestInfo[] requestInfo, DataStreamer streamer)
   at RemObjects.DataAbstract.Linq.LinqLocalDataAdapter.FetchData(TableRequestInfo[] requests, String[] names, Action`3 fillMethod)
   at RemObjects.DataAbstract.Linq.RemoteTable`1.Execute(Expression expression, DataParameter[] parameters)
   at RemObjects.DataAbstract.Linq.RemoteTableQuery`1.GetEnumerable()
   at RemObjects.DataAbstract.Linq.BindingListExtensions.ToDABindingList[T](IEnumerable`1 data)
   at SyncMaster.Logic.SyncScheduler.processSchedule()

FYI: This is how I create my local adapter

public static LinqLocalDataAdapter prepareLinqLocalAdapter(string servicename)
{
  /*Code that works for RO < 10.0.0.149x var sesmgr = RemObjects.SDK.Server.SessionManager.GlobalSessionManager;
  if (sesmgr == null)
    return null;
  Guid guid = Guid.NewGuid();
  var ses = sesmgr.GetSession(guid);
  var dataAdapter = new LinqLocalDataAdapter(servicename, guid);
  sesmgr.ReleaseSession(ses);*/
  //var dataStreamer = new RemObjects.DataAbstract.Bin2DataStreamer();

  /*Another attempt 
  var sesmgr=new RemObjects.SDK.Server.MemorySessionManager();
  if (sesmgr == null)
    return null;
  Guid guid = Guid.NewGuid();
  var ses = sesmgr.GetSession(guid);
  //var ses = RemObjects.SDK.Server.SessionManager.CreateNewSession(guid);
  var dataAdapter = new LinqLocalDataAdapter(servicename, guid);
  sesmgr.ReleaseSession(ses);*/

  /*Code suggested by RO but which results in crash and RO internal bugreport https://talk.remobjects.com/t/remobjects-sdk-server-sessionmanager-globalsessionmanager-does-not-exist-anymore-in-ro-10-0-0-1521/25527/7
  var dataAdapter = new LinqLocalDataAdapter(servicename, Guid.NewGuid());*/

  var sesmgr = RemObjects.SDK.Server.Engine.SessionManager;
  if (sesmgr == null)
    return null;
  Guid guid = Guid.NewGuid();
  var ses = sesmgr.GetSession(guid);
  var dataAdapter = new LinqLocalDataAdapter(servicename, guid);
  sesmgr.ReleaseSession(ses);

  return dataAdapter;
}

There are multiple local adapters and multiple remote adapters active at the same time

It should be/is supposed to be, yes.

Curious. this happens with an app that does no other data access except this one insert (multiple times, i imagine)? How is this code triggered and how do you ensure it always runs in the same thread (not that that should matter, mind you — this should be save to happen from 50 threads at once, too).

What database and driver is this using?

It is a windows service application running permanently and using a MSSQL.NET data connection

The crash happens for a worker thread where a dataAdapter is created inside a loop.
The dataAdapter (or LINQ objects) do not get passed as arguments to other functions (other threads) so all uses should be from the same thread.

A lot of other data access happens however in this process.
Other threads also create dataAdapters in a similar way and perform DB operations.
FYI: At present we see ~ 1 crash/day in production.

Code showing the loop (I assume the try catch arround the dispose should also be safe: via the using statement a new instance should be created)

  do
  {
        try
        {
          // Create a RO DA adapter for a local DA schema
          using (LinqLocalDataAdapter dataAdapter = CloudServerUtil.prepareSyncMasterLocalLinqAdapter())
          {
          } // using
    } catch (Exception ex)
      ...      
} while ([forever])

Thanks for the additional detail. What i think is happening is this: thread-safe or not, connections do of course (by default) get cached and reused. I don’t think (for now) that there’s a bug in Connection Manager doling out the same connection twice (we;'d see much worse problems then), what instead is happening, in think, is that some other data processing leaves a connection in a bad (unfinished) state, it goes back to the pool, and then your code happens to get the bad connection.

So a lot of data access is happening, including this one. But the error only/always happens for this particular update, not randomly elsewhere? If so, that’s the more curious part…

That looks fine to me…

  1. Indeed the error seems to report that while releasing a “free” connection of the pool it notices that the connection is not really “free”.
    The question arises: What makes the connectionmanager mark a connection as “free” (available).

  2. No I have the error in another place too. Here it is while really freeing the connection that was being used.
    The problem is that I also do not see an async call which would validate the “active” state.
    These dataAdapter objects however are maintained for a longer time however.
    So everything indicates this is causing the issue.
    Any idea how costly it is to create/destroy a localdataadapter?
    One solution is to create an adapter for every data update (very short lifetime)
    Another solution would be to use critical sections (if it is a threading issue and not an async call issue)

Here’s the alternate callstack

Exception: Cannot release connection because its transaction is still active  [Source:RemObjects.DataAbstract.Server]
   at RemObjects.DataAbstract.Server.ConnectionManager.ReleaseConnection(IAbstractConnection connection, Boolean keepInPool)
   at RemObjects.DataAbstract.Server.DataAbstractService.InternalReleaseConnection(IAbstractConnection connection, Boolean keepInPool)
   at RemObjects.DataAbstract.Server.DataAbstractService.InternalDeactivate(Guid sessionId)
   at RemObjects.SDK.Server.Service.Deactivate(Guid clientId)
   at RemObjects.DataAbstract.Server.LocalServiceConnection.ReleaseServiceInstance()
   at RemObjects.DataAbstract.Linq.LinqLocalDataAdapter.Dispose(Boolean disposing)
   at System.ComponentModel.Component.Dispose()
   at SyncMaster.Logic.SyncJobs.<>c.<RemoveAllFinished>b__5_0(SyncMasterJob job)
   at System.Collections.Generic.List`1.RemoveAll(Predicate`1 match)
   at SyncMaster.Logic.SyncJobs.RemoveAllFinished()
   at SyncMaster.Logic.SyncScheduler.processSchedule()

TIA,
Frederic

Ok, that’s “good”. So the issue is that one of the many data accesses your app makes (not necessarily, and in fact most likely, not the one that fails) leaves a connection with an active transaction and lets it return to the pool like that. Later, some other task gets assigned that connection to re-use, and it fails.

The problem is, nhow to find tha actual culprit…

I don’t think thats the issue, but worth a try. Data Adapters should be trivially cheap to instantiate and discard as you see fit, so a first step could be to just stop reusing those and create fresh ones instead, to see if that has any effect (but i’d expect it won’t).

wait, the same DataAdapter instance should definitely not be used from two threads at the same time. If that is haopenng, that could be causing many unforseen issues, yes!

Could it be that the code below leaves a transaction open?
e.g. The query object does not get disposed immediately and FirstOrDefault does not really finish it off?
I assume not since RemoteTable is not disposable.

var runinfoQuery = from c in dataAdapter.GetTable<OffCloudServerData.SyncMasterDAClasses.SyncRunInfo>() where c.sri_primkey == sri_primkey_ select c;
      runinfo_ = runinfoQuery.FirstOrDefault();

Another thought: How should “transaction” be interpreted?
Can this only be a modify operation?
More specifically : Can it only come from ApplyChanges() (or variant)
Does one of the DAClasses setters already start a transaction?

e.g.

[RemObjects.DataAbstract.Linq.FieldName("sri_runinfo")]		[RemObjects.DataAbstract.Linq.DataType(RemObjects.DataAbstract.Schema.DataType.WideString)]
[RemObjects.DataAbstract.Linq.LogChanges]
public System.String sri_runinfo
{
	get
	{
		return this.f____sri_runinfo;
	}
	set
	{
		if (System.Collections.Generic.Comparer<System.String>.Default.Compare(this.f____sri_runinfo, value) != 0)
		{
			this.OnPropertyChanging("sri_runinfo");
			this.f____sri_runinfo = value;
			this.OnPropertyChanged("sri_runinfo");
		}
	}
}

I don’t believe this needs disposing, no.

This is a transaction inside the back-end database, yes. The only thing that would start (and should close) a database transaction would be updates to the database (e.g. one or more related ApplyUpdates calls, or anything else that actually hits gthe database.

Local chnages to DA LINQ objects would not affect transactions, as those are just making chages to local in-memory objects until the changes are applied.

Ok, reviewing the code more, it seems that

‘Cannot release connection because its transaction is still active’

is an explicit check we perform on DA level, when returning a transaction t the pool if the PoolTransactionBehaviour is set to .Fail (the default). That means my previous expectation is probably wrong, as it should not be possible for one bit of code to erroneously return a connection to the pool with open transactions and another bit of code to trhen get the bad connection — because the error would already happen the first time time.

This means one of two things:

(a) the code that triggers the above exception is actually the code that doesn’t properly close the transaction (unlikely, given how trivial it is)

or

(b) we actually do have a race condition (more likely).

Reviewing the code for ConnectionManager, it does (as expected) look to be thread safe. Now, there’s always the potential for bugs in any threaded code, but given that this is long-time established code running for 15+ years that hasn’t been changed much in forever and that we’re not seeing threading issues here en mass, my gut feeling is that this is not the culprit, even if i cannot rule it out 100% yet.

Since from the last few messages it does seem in doubt whether your use of the LinqDataAdapters is single-threaded or not, i would focus on making sure they are, as a first step. The easiest way to do that would be to just make sure Data Adapters are never recycled and always created fresh for a single* use.

Where “single” does not necessarily mean for a single GetTable/Apply call only — related data fetches and updates (eg master/detail) must use the same DA. But make sure they are created used, and then discarded within the same method/thread, and not pooled, recycled or stored globally anywhere.

If this solves the issue, good. If not, we can/have to dig deeper.

Thanks for the reply.
Does this mean I should fetch my database records over and over again?
(it appears that way)

The DB writes are some soft of state changes (e.g. ‘JOB QUEUED’, ‘JOB STARTED’, ‘JOB FINISHED’)
My current code has a long living linq table which I then modify but of course this object which was fetched with a killed dataAdapter does not modify when I do applyupdates on a new dataadapter.

What is typically done here?
Can I attach the table to an adapter?

[RemObjects.DataAbstract.Linq.TableName(“SyncRunInfo”)]
public partial class SyncRunInfo : System.ICloneable, System.ComponentModel.INotifyPropertyChanged, System.ComponentModel.INotifyPropertyChanging

Maybe explicit calls to BeginUpdate and EndUpdate?

No, that shouldn’t be necessary. you can fetch a DA LINQ sequence, ToList it, and keep that around after the Adapter is disposed (make sure you do ToList or an equivalent though, if you keep the sequence itself, each access will re-run the query remotely, as that’s how LINQ works). You can then later make changes and apply them using a new Data Adapter (as long as you apply related changes (eg a new parent record and its children) with the same adapter.

Does that make sense?

Your answer makes sense except the “No, that shouldn’t be necessary.”

How does the dataAdapter get connected to the list of dataabstract records in your situation?

FYI: The BeginUpdate/EndUpdate works but maybe you have a better solution?

e.g., form my own code (adapted)

var rda := DataAccess.UniqueLinqRDA;
var lUser := (from u in rda.GetTable<Users>)
var lSavedUsers := lUsers.ToList; // lSavedUsers is now local data, detached from the query

... later

var newRda := DataAccess.UniqueLinqRDA;
lUser.Name := "Paul";
newRda.UpdateRow(lUser);  // associates/registers the row and its updates with the new RDA
newRda.ApplyChanges; // apply it

ofc newRda isn’t “needed” if this is linear code, just use the same rda to fetch and apply. This assumes the “later” part happens after the first rda has gone out of scope, say in a different method.

Does that make sense?

OK, thanks, got it

1 Like