Uncaught stackoverflow crashing server

In C# Dot Net 7, Rem Objects 10.0.0.1559

I am encountering a server crash caused by a stack overflow when sending a sufficiently large stream, RemObjects.SDK.Server.AsyncHttpServerWorker.ResponseBodyCallback seems to work via recursion, each recursion, a byte[4096] buffer is sent to the client. I seem to be running out of stack memory before my file is sent. From a quick visual inspection, I recurse through

ResponseBodyCallback->
BeginWrite->
IntBeginWrite->
TaskAsyncResult->
ResponseBodyCallback ^

5200 times before a stack overflow is triggered, causing the entire server to crash.
5200 calls of BeginWrite, 4096 bytes per buffer, meaning this crashed at around 5200x4096=20MB (21,299,200 bytes) (<- This is my interpretation, true if I understand the structure of AsyncHttpServerWorker .ResponseBodyCallback)

This could be solved with a different architecture of the ResponseBodyCallback function (A while loop, instead of a recursion)
But equally importantly, this should be handled such as to not cause a full server crash.

Here is a part of the stack trace (With some of the recursions trimmed):

2024-02-01T10:02:43.898442100Z Stack overflow.
2024-02-01T10:02:43.905874900Z    at System.Threading.Tasks.Task.FromResult[[System.Int32, System.Private.CoreLib, Version=7.0.0.0, Culture=neutral, PublicKeyToken=xxxxxxxxxxxxxxxx]](Int32)
2024-02-01T10:02:43.905874900Z    at RemObjects.SDK.Connection.IntBeginWrite(Byte[], Int32, Int32, System.AsyncCallback, System.Object)
2024-02-01T10:02:43.905874900Z    at RemObjects.SDK.Connection.BeginWrite(Byte[], Int32, Int32, System.AsyncCallback, System.Object)
2024-02-01T10:02:43.905874900Z    at RemObjects.SDK.Server.AsyncHttpServerWorker.ResponseBodyCallback(System.IAsyncResult)
2024-02-01T10:02:43.905874900Z    at System.Threading.Tasks.TaskToApm+TaskAsyncResult..ctor(System.Threading.Tasks.Task, System.Object, System.AsyncCallback)
2024-02-01T10:02:43.906537200Z    at RemObjects.SDK.Connection.IntBeginWrite(Byte[], Int32, Int32, System.AsyncCallback, System.Object)
2024-02-01T10:02:43.906698400Z    at RemObjects.SDK.Connection.BeginWrite(Byte[], Int32, Int32, System.AsyncCallback, System.Object)
2024-02-01T10:02:43.906869300Z    at RemObjects.SDK.Server.AsyncHttpServerWorker.ResponseBodyCallback(System.IAsyncResult)
2024-02-01T10:02:43.907165300Z    at System.Threading.Tasks.TaskToApm+TaskAsyncResult..ctor(System.Threading.Tasks.Task, System.Object, System.AsyncCallback)
2024-02-01T10:02:43.907165300Z    at RemObjects.SDK.Connection.IntBeginWrite(Byte[], Int32, Int32, System.AsyncCallback, System.Object)
2024-02-01T10:02:43.907165300Z    at RemObjects.SDK.Connection.BeginWrite(Byte[], Int32, Int32, System.AsyncCallback, System.Object)
2024-02-01T10:02:43.907910500Z    at RemObjects.SDK.Server.AsyncHttpServerWorker.ResponseBodyCallback(System.IAsyncResult)
2024-02-01T10:02:45.840557600Z    at System.Threading.Tasks.TaskToApm+TaskAsyncResult..ctor(System.Threading.Tasks.Task, System.Object, System.AsyncCallback)
2024-02-01T10:02:45.840557600Z    at RemObjects.SDK.Connection.IntBeginWrite(Byte[], Int32, Int32, System.AsyncCallback, System.Object)
2024-02-01T10:02:45.840557600Z    at RemObjects.SDK.Connection.BeginWrite(Byte[], Int32, Int32, System.AsyncCallback, System.Object)
2024-02-01T10:02:45.840557600Z    at RemObjects.SDK.Server.AsyncHttpServerWorker.ResponseBodyCallback(System.IAsyncResult)
2024-02-01T10:02:45.840557600Z    at System.Threading.Tasks.TaskToApm+TaskAsyncResult..ctor(System.Threading.Tasks.Task, System.Object, System.AsyncCallback)
2024-02-01T10:02:45.840557600Z    at RemObjects.SDK.Connection.IntBeginWrite(Byte[], Int32, Int32, System.AsyncCallback, System.Object)
2024-02-01T10:02:45.840557600Z    at RemObjects.SDK.Connection.BeginWrite(Byte[], Int32, Int32, System.AsyncCallback, System.Object)
2024-02-01T10:02:45.843658100Z    at RemObjects.SDK.Server.AsyncHttpServerWorker.ResponseBodyCallback(System.IAsyncResult)
2024-02-01T10:02:45.843658100Z    at System.Threading.Tasks.TaskToApm+TaskAsyncResult..ctor(System.Threading.Tasks.Task, System.Object, System.AsyncCallback)
2024-02-01T10:02:45.843658100Z    at RemObjects.SDK.Connection.IntBeginWrite(Byte[], Int32, Int32, System.AsyncCallback, System.Object)
2024-02-01T10:02:45.843658100Z    at RemObjects.SDK.Connection.BeginWrite(Byte[], Int32, Int32, System.AsyncCallback, System.Object)
2024-02-01T10:02:45.843658100Z    at RemObjects.SDK.Server.AsyncHttpServerWorker.ResponseBodyCallback(System.IAsyncResult)

<-Recursion start->

2024-02-01T10:02:45.843658100Z    at System.Threading.Tasks.TaskToApm+TaskAsyncResult..ctor(System.Threading.Tasks.Task, System.Object, System.AsyncCallback)
2024-02-01T10:02:45.843658100Z    at RemObjects.SDK.Connection.IntBeginWrite(Byte[], Int32, Int32, System.AsyncCallback, System.Object)
2024-02-01T10:02:45.843658100Z    at RemObjects.SDK.Connection.BeginWrite(Byte[], Int32, Int32, System.AsyncCallback, System.Object)
2024-02-01T10:02:45.843658100Z    at RemObjects.SDK.Server.AsyncHttpServerWorker.ResponseBodyCallback(System.IAsyncResult)
2024-02-01T10:02:45.843658100Z    at System.Threading.ExecutionContext.RunInternal(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object)
2024-02-01T10:02:45.843658100Z    at System.Threading.Tasks.AwaitTaskContinuation.RunCallback(System.Threading.ContextCallback, System.Object, System.Threading.Tasks.Task ByRef)
2024-02-01T10:02:45.843658100Z    at System.Threading.Tasks.Task.RunContinuations(System.Object)
2024-02-01T10:02:45.843658100Z    at System.Threading.Tasks.Task`1[[System.Int32, System.Private.CoreLib, Version=7.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e]].TrySetResult(Int32)
2024-02-01T10:02:45.843658100Z    at System.Threading.Tasks.ValueTask`1+ValueTaskSourceAsTask+<>c[[System.Int32, System.Private.CoreLib, Version=7.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e]].<.cctor>b__4_0(System.Object)
2024-02-01T10:02:45.843658100Z    at System.Net.Sockets.Socket+AwaitableSocketAsyncEventArgs.InvokeContinuation(System.Action`1<System.Object>, System.Object, Boolean, Boolean)
2024-02-01T10:02:45.843658100Z    at System.Net.Sockets.Socket+AwaitableSocketAsyncEventArgs.OnCompleted(System.Net.Sockets.SocketAsyncEventArgs)
2024-02-01T10:02:45.843658100Z    at System.Net.Sockets.SocketAsyncEventArgs+<>c.<.cctor>b__176_0(UInt32, UInt32, System.Threading.NativeOverlapped*)
2024-02-01T10:02:45.843658100Z    at System.Threading.ThreadPoolTypedWorkItemQueue`2[[System.Threading.PortableThreadPool+IOCompletionPoller+Event, System.Private.CoreLib, Version=7.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e],[System.Threading.PortableThreadPool+IOCompletionPoller+Callback, System.Private.CoreLib, Version=7.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e]].System.Threading.IThreadPoolWorkItem.Execute()
2024-02-01T10:02:45.843658100Z    at System.Threading.ThreadPoolWorkQueue.Dispatch()
2024-02-01T10:02:45.843658100Z    at System.Threading.PortableThreadPool+WorkerThread.WorkerThreadStart()

Agreed on both accounts.

To have a more complete test case; what server channel are you using, and what does the method you’re calling look like – I assume a simple method that takes a Binary and then passing it a large one will suffice to reproduce this?

Logged as bugs://D19426.

That is correct, I don’t have it on hand anymore. But sending a binary of a sufficient size using a Service Method will indeed trigger this stack overflow (I’m running my server as an IpSuperHttpServerChannel if its relevant).

(Its still happening in version 10.0.0.1587)

Thanx! I will try and reproduce the here, next week. I assume you have thje message size limited disabled/set higher than the binary your sending?

thanx,
marc

Yes, you are correct. Limit is 200mb I believe (With timeouts disabled, and max message size disabled on the client, and on the server.

        IpTcpServerChannel ipTcpServerChannel = new IpTcpServerChannel();
        
        ipTcpServerChannel.TcpServer.TimeoutEnabled = false;
        ipTcpServerChannel.TcpServer.MaxLineLengthEnabled = false; 

I did notice that we are using IpTcpServerChannel now, instead of the IpSuperHttpServerChannel that we used to use before. I’m not certain if that will actually changed anything. Just communicating it for clarity.

1 Like

Thanx.

the same issue persists with the Tcp server as well, I take it?

—marc

So in my original post, I believe it was an IpTcpServerChannel.

However:

(Whats below is confusion, it is unclear to me is the summary)

The issue today is between two servers, (One running on IpSuperHttpServerChannel, the other running a IpTcpServerChannel), and call each others service methods (among other things they do).

The server with the IpSuperHttpServerChannel crashes with the stack overflow. But it might be crashing as a client replying to a service method like this:

 // This is running behind an IpTcpServerChannel
  [ServiceMethod]
        public void CreateReport(out Binary[] files, out Binary[] logs)

To the best of my knowledge, this issue is concerning the IpTcpServerChannel. However sadly its not fully clear to me at this point because I’m having issues with reproducing it currently. If I get more information I’ll let you know.

Hm now I am confused :grin:.

even if both apps are servers, for the purpose of this, lets think of one of them as a client, as the crash, I presume, happens due to one app (the “client”) calling a method on the other (the “server”). It doesnt matter then that the “client” also acts as a server.

Which channel is the one receiving the call (and crashing due to the call) using? IpSuperHttpServerChannel?