I upgraded a customer a few days ago. On that day and also twice on a subsequent day, I’ve seen an error that has never triggered previously in several years of our software running on the customers setup.
Prior to the upgrade, the RemObjects SDK for .NET version was 126.96.36.1993. After the upgrade, they are using 188.8.131.527. The customer is probably our heaviest user with hundreds of sessions coming and going during any one day. We never actively restart our .NET server process, except for system patching etc.
On the first occasion the error triggered, I saw our .NET server heaving consuming CPU. Memory usage looked to be heading towards 10GB. On that occasion I had been alerted before the process fell over and I actively killed it, so there was no event recorded i.e. I’m assuming this was the same error.
Yesterday, the error occurred twice within 6 hours, resulting in our service not running for a period of time. These I did not see until they had crashed and where the event log did record the event.
Application: oxy003.exe Framework Version: v4.0.30319 Description: The process was terminated due to an unhandled exception. Exception Info: System.OutOfMemoryException Stack: at System.Collections.Generic.Queue`1[[System.__Canon, mscorlib, Version=184.108.40.206, Culture=neutral, PublicKeyToken=b77a5c561934e089]].SetCapacity(Int32) at System.Collections.Generic.Queue`1[[System.__Canon, mscorlib, Version=220.127.116.11, Culture=neutral, PublicKeyToken=b77a5c561934e089]].Enqueue(System.__Canon) at RemObjects.SDK.Server.MemoryMessageQueue.QueueMessage(System.IO.Stream) at RemObjects.SDK.Server.MessageQueueManager.QueueMessage(System.IO.Stream, RemObjects.SDK.Server.IMessageQueue) at RemObjects.SDK.Server.MessageQueueManager.QueueMessage(System.IO.Stream, System.Guid) at RemObjects.SDK.Server.MessageQueueManager.QueueMessage(RemObjects.SDK.IMessage, System.Guid) at RemObjects.SDK.Server.EventSinkManager.RemObjects.SDK.Server.ICustomMessageQueueHandler.DispatchEvent(RemObjects.SDK.IMessage, System.Type, RemObjects.SDK.Server.IEventTargets) at OXY003.ServerEvents_EventSinkProxy.onLogout(System.Guid, System.Guid) at OXY003.Engine.sessionManager_OnSessionExpired(System.Object, RemObjects.SDK.Server.SessionEventArgs) at RemObjects.SDK.Server.MemorySessionManager.ExpireSessions() at RemObjects.SDK.Server.SessionManager.ExpireTimerHandler(System.Object) at System.Threading.ExecutionContext.RunInternal(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object, Boolean) at System.Threading.ExecutionContext.Run(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object, Boolean) at System.Threading.TimerQueueTimer.CallCallback() at System.Threading.TimerQueueTimer.Fire() at System.Threading.QueueUserWorkItemCallback.System.Threading.IThreadPoolWorkItem.ExecuteWorkItem() at System.Threading.ThreadPoolWorkQueue.Dispatch()
The above would seem to fit with the first incident I witnessed i.e. high memory consumption to the point of failure.
We do have code in the server side OnSessionExpired.
method Engine.sessionManager_OnSessionExpired(sender: System.Object; e: RemObjects.SDK.Server.SessionEventArgs); var lCompanyGuid : System.Guid; FormGuid : System.Guid; iLoop : Integer; rRow : System.Data.DataRow; Sessions : IList<Guid>; lSessions : Array of Guid; begin Sessions := self.sessionManager.GetAllSessions; Array.Resize(var lSessions, Sessions.Count); Sessions.CopyTo(lSessions,0); var Targets : RecipientListEventTargets := new RecipientListEventTargets(lSessions); var ev2: IServerEvents := eventSinkManager1.GetEventSink(e.Session.SessionID,typeOf(IServerEvents), Targets) as IServerEvents; lCompanyGuid := new Guid(e.Session['CompanyGuid'].ToString); FormGuid := new Guid(e.Session['FormGuid'].ToString); ev2.onLogout(lCompanyGuid,FormGuid); for iLoop := ServerActivity.MessageDataTable.Rows.Count-1 downto 0 do begin rRow := ServerActivity.MessageDataTable.Rows[iLoop]; case (rRow['CompanyGuid'].ToString = lCompanyGuid.ToString) and (rRow['FormGuid'].ToString = FormGuid.ToString) of true : begin rRow.Delete; break; end; end; end; end;
The above code has not changed in several years and I’ve never seen the application fail with the above exception on any customer installation.
I’ve tested locally today with low timeout settings (30 seconds compared to the production setting of 10800) on the session manager. The code still appears to perform as it has always done, triggering the onLogout event to pass back to clients, notifying them of an expired session (used in a monitoring screen to view the sessions still active).
One workaround would be to remove the above code, but I’d rather not resort to that without understanding why we now have an issue. However, this is a key customer and so I do need to implement some remedy fairly quickly.
Any thoughts on what might be happening here ? Changes in the RemObjects framwork that might now be exposing flaws in our code ?