Uncaught stackoverflow crashing server

Remoting SDK
, , ,
1

In C# Dot Net 7, Rem Objects 10.0.0.1559

I am encountering a server crash caused by a stack overflow when sending a sufficiently large stream, RemObjects.SDK.Server.AsyncHttpServerWorker.ResponseBodyCallback seems to work via recursion, each recursion, a byte[4096] buffer is sent to the client. I seem to be running out of stack memory before my file is sent. From a quick visual inspection, I recurse through

ResponseBodyCallback->
BeginWrite->
IntBeginWrite->
TaskAsyncResult->
ResponseBodyCallback ^

5200 times before a stack overflow is triggered, causing the entire server to crash.
5200 calls of BeginWrite, 4096 bytes per buffer, meaning this crashed at around 5200x4096=20MB (21,299,200 bytes) (<- This is my interpretation, true if I understand the structure of AsyncHttpServerWorker .ResponseBodyCallback)

This could be solved with a different architecture of the ResponseBodyCallback function (A while loop, instead of a recursion)
But equally importantly, this should be handled such as to not cause a full server crash.

Here is a part of the stack trace (With some of the recursions trimmed):

2024-02-01T10:02:43.898442100Z Stack overflow.
2024-02-01T10:02:43.905874900Z    at System.Threading.Tasks.Task.FromResult[[System.Int32, System.Private.CoreLib, Version=7.0.0.0, Culture=neutral, PublicKeyToken=xxxxxxxxxxxxxxxx]](Int32)
2024-02-01T10:02:43.905874900Z    at RemObjects.SDK.Connection.IntBeginWrite(Byte[], Int32, Int32, System.AsyncCallback, System.Object)
2024-02-01T10:02:43.905874900Z    at RemObjects.SDK.Connection.BeginWrite(Byte[], Int32, Int32, System.AsyncCallback, System.Object)
2024-02-01T10:02:43.905874900Z    at RemObjects.SDK.Server.AsyncHttpServerWorker.ResponseBodyCallback(System.IAsyncResult)
2024-02-01T10:02:43.905874900Z    at System.Threading.Tasks.TaskToApm+TaskAsyncResult..ctor(System.Threading.Tasks.Task, System.Object, System.AsyncCallback)
2024-02-01T10:02:43.906537200Z    at RemObjects.SDK.Connection.IntBeginWrite(Byte[], Int32, Int32, System.AsyncCallback, System.Object)
2024-02-01T10:02:43.906698400Z    at RemObjects.SDK.Connection.BeginWrite(Byte[], Int32, Int32, System.AsyncCallback, System.Object)
2024-02-01T10:02:43.906869300Z    at RemObjects.SDK.Server.AsyncHttpServerWorker.ResponseBodyCallback(System.IAsyncResult)
2024-02-01T10:02:43.907165300Z    at System.Threading.Tasks.TaskToApm+TaskAsyncResult..ctor(System.Threading.Tasks.Task, System.Object, System.AsyncCallback)
2024-02-01T10:02:43.907165300Z    at RemObjects.SDK.Connection.IntBeginWrite(Byte[], Int32, Int32, System.AsyncCallback, System.Object)
2024-02-01T10:02:43.907165300Z    at RemObjects.SDK.Connection.BeginWrite(Byte[], Int32, Int32, System.AsyncCallback, System.Object)
2024-02-01T10:02:43.907910500Z    at RemObjects.SDK.Server.AsyncHttpServerWorker.ResponseBodyCallback(System.IAsyncResult)
2024-02-01T10:02:45.840557600Z    at System.Threading.Tasks.TaskToApm+TaskAsyncResult..ctor(System.Threading.Tasks.Task, System.Object, System.AsyncCallback)
2024-02-01T10:02:45.840557600Z    at RemObjects.SDK.Connection.IntBeginWrite(Byte[], Int32, Int32, System.AsyncCallback, System.Object)
2024-02-01T10:02:45.840557600Z    at RemObjects.SDK.Connection.BeginWrite(Byte[], Int32, Int32, System.AsyncCallback, System.Object)
2024-02-01T10:02:45.840557600Z    at RemObjects.SDK.Server.AsyncHttpServerWorker.ResponseBodyCallback(System.IAsyncResult)
2024-02-01T10:02:45.840557600Z    at System.Threading.Tasks.TaskToApm+TaskAsyncResult..ctor(System.Threading.Tasks.Task, System.Object, System.AsyncCallback)
2024-02-01T10:02:45.840557600Z    at RemObjects.SDK.Connection.IntBeginWrite(Byte[], Int32, Int32, System.AsyncCallback, System.Object)
2024-02-01T10:02:45.840557600Z    at RemObjects.SDK.Connection.BeginWrite(Byte[], Int32, Int32, System.AsyncCallback, System.Object)
2024-02-01T10:02:45.843658100Z    at RemObjects.SDK.Server.AsyncHttpServerWorker.ResponseBodyCallback(System.IAsyncResult)
2024-02-01T10:02:45.843658100Z    at System.Threading.Tasks.TaskToApm+TaskAsyncResult..ctor(System.Threading.Tasks.Task, System.Object, System.AsyncCallback)
2024-02-01T10:02:45.843658100Z    at RemObjects.SDK.Connection.IntBeginWrite(Byte[], Int32, Int32, System.AsyncCallback, System.Object)
2024-02-01T10:02:45.843658100Z    at RemObjects.SDK.Connection.BeginWrite(Byte[], Int32, Int32, System.AsyncCallback, System.Object)
2024-02-01T10:02:45.843658100Z    at RemObjects.SDK.Server.AsyncHttpServerWorker.ResponseBodyCallback(System.IAsyncResult)

<-Recursion start->

2024-02-01T10:02:45.843658100Z    at System.Threading.Tasks.TaskToApm+TaskAsyncResult..ctor(System.Threading.Tasks.Task, System.Object, System.AsyncCallback)
2024-02-01T10:02:45.843658100Z    at RemObjects.SDK.Connection.IntBeginWrite(Byte[], Int32, Int32, System.AsyncCallback, System.Object)
2024-02-01T10:02:45.843658100Z    at RemObjects.SDK.Connection.BeginWrite(Byte[], Int32, Int32, System.AsyncCallback, System.Object)
2024-02-01T10:02:45.843658100Z    at RemObjects.SDK.Server.AsyncHttpServerWorker.ResponseBodyCallback(System.IAsyncResult)
2024-02-01T10:02:45.843658100Z    at System.Threading.ExecutionContext.RunInternal(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object)
2024-02-01T10:02:45.843658100Z    at System.Threading.Tasks.AwaitTaskContinuation.RunCallback(System.Threading.ContextCallback, System.Object, System.Threading.Tasks.Task ByRef)
2024-02-01T10:02:45.843658100Z    at System.Threading.Tasks.Task.RunContinuations(System.Object)
2024-02-01T10:02:45.843658100Z    at System.Threading.Tasks.Task`1[[System.Int32, System.Private.CoreLib, Version=7.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e]].TrySetResult(Int32)
2024-02-01T10:02:45.843658100Z    at System.Threading.Tasks.ValueTask`1+ValueTaskSourceAsTask+<>c[[System.Int32, System.Private.CoreLib, Version=7.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e]].<.cctor>b__4_0(System.Object)
2024-02-01T10:02:45.843658100Z    at System.Net.Sockets.Socket+AwaitableSocketAsyncEventArgs.InvokeContinuation(System.Action`1<System.Object>, System.Object, Boolean, Boolean)
2024-02-01T10:02:45.843658100Z    at System.Net.Sockets.Socket+AwaitableSocketAsyncEventArgs.OnCompleted(System.Net.Sockets.SocketAsyncEventArgs)
2024-02-01T10:02:45.843658100Z    at System.Net.Sockets.SocketAsyncEventArgs+<>c.<.cctor>b__176_0(UInt32, UInt32, System.Threading.NativeOverlapped*)
2024-02-01T10:02:45.843658100Z    at System.Threading.ThreadPoolTypedWorkItemQueue`2[[System.Threading.PortableThreadPool+IOCompletionPoller+Event, System.Private.CoreLib, Version=7.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e],[System.Threading.PortableThreadPool+IOCompletionPoller+Callback, System.Private.CoreLib, Version=7.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e]].System.Threading.IThreadPoolWorkItem.Execute()
2024-02-01T10:02:45.843658100Z    at System.Threading.ThreadPoolWorkQueue.Dispatch()
2024-02-01T10:02:45.843658100Z    at System.Threading.PortableThreadPool+WorkerThread.WorkerThreadStart()
3

Agreed on both accounts.

4

To have a more complete test case; what server channel are you using, and what does the method you’re calling look like – I assume a simple method that takes a Binary and then passing it a large one will suffice to reproduce this?

5

Logged as bugs://D19426.