Uncaught stackoverflow crashing server

infokevercom · February 2, 2024, 3:41pm

In C# Dot Net 7, Rem Objects 10.0.0.1559

I am encountering a server crash caused by a stack overflow when sending a sufficiently large stream, RemObjects.SDK.Server.AsyncHttpServerWorker.ResponseBodyCallback seems to work via recursion, each recursion, a byte[4096] buffer is sent to the client. I seem to be running out of stack memory before my file is sent. From a quick visual inspection, I recurse through

ResponseBodyCallback->
BeginWrite->
IntBeginWrite->
TaskAsyncResult->
ResponseBodyCallback ^

5200 times before a stack overflow is triggered, causing the entire server to crash.
5200 calls of BeginWrite, 4096 bytes per buffer, meaning this crashed at around 5200x4096=20MB (21,299,200 bytes) (<- This is my interpretation, true if I understand the structure of AsyncHttpServerWorker .ResponseBodyCallback)

This could be solved with a different architecture of the ResponseBodyCallback function (A while loop, instead of a recursion)
But equally importantly, this should be handled such as to not cause a full server crash.

Here is a part of the stack trace (With some of the recursions trimmed):

2024-02-01T10:02:43.898442100Z Stack overflow.
2024-02-01T10:02:43.905874900Z    at System.Threading.Tasks.Task.FromResult[[System.Int32, System.Private.CoreLib, Version=7.0.0.0, Culture=neutral, PublicKeyToken=xxxxxxxxxxxxxxxx]](Int32)
2024-02-01T10:02:43.905874900Z    at RemObjects.SDK.Connection.IntBeginWrite(Byte[], Int32, Int32, System.AsyncCallback, System.Object)
2024-02-01T10:02:43.905874900Z    at RemObjects.SDK.Connection.BeginWrite(Byte[], Int32, Int32, System.AsyncCallback, System.Object)
2024-02-01T10:02:43.905874900Z    at RemObjects.SDK.Server.AsyncHttpServerWorker.ResponseBodyCallback(System.IAsyncResult)
2024-02-01T10:02:43.905874900Z    at System.Threading.Tasks.TaskToApm+TaskAsyncResult..ctor(System.Threading.Tasks.Task, System.Object, System.AsyncCallback)
2024-02-01T10:02:43.906537200Z    at RemObjects.SDK.Connection.IntBeginWrite(Byte[], Int32, Int32, System.AsyncCallback, System.Object)
2024-02-01T10:02:43.906698400Z    at RemObjects.SDK.Connection.BeginWrite(Byte[], Int32, Int32, System.AsyncCallback, System.Object)
2024-02-01T10:02:43.906869300Z    at RemObjects.SDK.Server.AsyncHttpServerWorker.ResponseBodyCallback(System.IAsyncResult)
2024-02-01T10:02:43.907165300Z    at System.Threading.Tasks.TaskToApm+TaskAsyncResult..ctor(System.Threading.Tasks.Task, System.Object, System.AsyncCallback)
2024-02-01T10:02:43.907165300Z    at RemObjects.SDK.Connection.IntBeginWrite(Byte[], Int32, Int32, System.AsyncCallback, System.Object)
2024-02-01T10:02:43.907165300Z    at RemObjects.SDK.Connection.BeginWrite(Byte[], Int32, Int32, System.AsyncCallback, System.Object)
2024-02-01T10:02:43.907910500Z    at RemObjects.SDK.Server.AsyncHttpServerWorker.ResponseBodyCallback(System.IAsyncResult)
2024-02-01T10:02:45.840557600Z    at System.Threading.Tasks.TaskToApm+TaskAsyncResult..ctor(System.Threading.Tasks.Task, System.Object, System.AsyncCallback)
2024-02-01T10:02:45.840557600Z    at RemObjects.SDK.Connection.IntBeginWrite(Byte[], Int32, Int32, System.AsyncCallback, System.Object)
2024-02-01T10:02:45.840557600Z    at RemObjects.SDK.Connection.BeginWrite(Byte[], Int32, Int32, System.AsyncCallback, System.Object)
2024-02-01T10:02:45.840557600Z    at RemObjects.SDK.Server.AsyncHttpServerWorker.ResponseBodyCallback(System.IAsyncResult)
2024-02-01T10:02:45.840557600Z    at System.Threading.Tasks.TaskToApm+TaskAsyncResult..ctor(System.Threading.Tasks.Task, System.Object, System.AsyncCallback)
2024-02-01T10:02:45.840557600Z    at RemObjects.SDK.Connection.IntBeginWrite(Byte[], Int32, Int32, System.AsyncCallback, System.Object)
2024-02-01T10:02:45.840557600Z    at RemObjects.SDK.Connection.BeginWrite(Byte[], Int32, Int32, System.AsyncCallback, System.Object)
2024-02-01T10:02:45.843658100Z    at RemObjects.SDK.Server.AsyncHttpServerWorker.ResponseBodyCallback(System.IAsyncResult)
2024-02-01T10:02:45.843658100Z    at System.Threading.Tasks.TaskToApm+TaskAsyncResult..ctor(System.Threading.Tasks.Task, System.Object, System.AsyncCallback)
2024-02-01T10:02:45.843658100Z    at RemObjects.SDK.Connection.IntBeginWrite(Byte[], Int32, Int32, System.AsyncCallback, System.Object)
2024-02-01T10:02:45.843658100Z    at RemObjects.SDK.Connection.BeginWrite(Byte[], Int32, Int32, System.AsyncCallback, System.Object)
2024-02-01T10:02:45.843658100Z    at RemObjects.SDK.Server.AsyncHttpServerWorker.ResponseBodyCallback(System.IAsyncResult)

<-Recursion start->

2024-02-01T10:02:45.843658100Z    at System.Threading.Tasks.TaskToApm+TaskAsyncResult..ctor(System.Threading.Tasks.Task, System.Object, System.AsyncCallback)
2024-02-01T10:02:45.843658100Z    at RemObjects.SDK.Connection.IntBeginWrite(Byte[], Int32, Int32, System.AsyncCallback, System.Object)
2024-02-01T10:02:45.843658100Z    at RemObjects.SDK.Connection.BeginWrite(Byte[], Int32, Int32, System.AsyncCallback, System.Object)
2024-02-01T10:02:45.843658100Z    at RemObjects.SDK.Server.AsyncHttpServerWorker.ResponseBodyCallback(System.IAsyncResult)
2024-02-01T10:02:45.843658100Z    at System.Threading.ExecutionContext.RunInternal(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object)
2024-02-01T10:02:45.843658100Z    at System.Threading.Tasks.AwaitTaskContinuation.RunCallback(System.Threading.ContextCallback, System.Object, System.Threading.Tasks.Task ByRef)
2024-02-01T10:02:45.843658100Z    at System.Threading.Tasks.Task.RunContinuations(System.Object)
2024-02-01T10:02:45.843658100Z    at System.Threading.Tasks.Task`1[[System.Int32, System.Private.CoreLib, Version=7.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e]].TrySetResult(Int32)
2024-02-01T10:02:45.843658100Z    at System.Threading.Tasks.ValueTask`1+ValueTaskSourceAsTask+<>c[[System.Int32, System.Private.CoreLib, Version=7.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e]].<.cctor>b__4_0(System.Object)
2024-02-01T10:02:45.843658100Z    at System.Net.Sockets.Socket+AwaitableSocketAsyncEventArgs.InvokeContinuation(System.Action`1<System.Object>, System.Object, Boolean, Boolean)
2024-02-01T10:02:45.843658100Z    at System.Net.Sockets.Socket+AwaitableSocketAsyncEventArgs.OnCompleted(System.Net.Sockets.SocketAsyncEventArgs)
2024-02-01T10:02:45.843658100Z    at System.Net.Sockets.SocketAsyncEventArgs+<>c.<.cctor>b__176_0(UInt32, UInt32, System.Threading.NativeOverlapped*)
2024-02-01T10:02:45.843658100Z    at System.Threading.ThreadPoolTypedWorkItemQueue`2[[System.Threading.PortableThreadPool+IOCompletionPoller+Event, System.Private.CoreLib, Version=7.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e],[System.Threading.PortableThreadPool+IOCompletionPoller+Callback, System.Private.CoreLib, Version=7.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e]].System.Threading.IThreadPoolWorkItem.Execute()
2024-02-01T10:02:45.843658100Z    at System.Threading.ThreadPoolWorkQueue.Dispatch()
2024-02-01T10:02:45.843658100Z    at System.Threading.PortableThreadPool+WorkerThread.WorkerThreadStart()

mh · February 2, 2024, 6:12pm

Agreed on both accounts.

mh · February 2, 2024, 6:13pm

To have a more complete test case; what server channel are you using, and what does the method you’re calling look like – I assume a simple method that takes a Binary and then passing it a large one will suffice to reproduce this?

RemObjectsSoftware · February 2, 2024, 6:14pm

Logged as bugs://D19426.

infokevercom · June 28, 2024, 12:05pm

That is correct, I don’t have it on hand anymore. But sending a binary of a sufficient size using a Service Method will indeed trigger this stack overflow (I’m running my server as an IpSuperHttpServerChannel if its relevant).

(Its still happening in version 10.0.0.1587)

mh · June 28, 2024, 1:11pm

Thanx! I will try and reproduce the here, next week. I assume you have thje message size limited disabled/set higher than the binary your sending?

thanx,
marc

infokevercom · June 28, 2024, 1:33pm

Yes, you are correct. Limit is 200mb I believe (With timeouts disabled, and max message size disabled on the client, and on the server.

        IpTcpServerChannel ipTcpServerChannel = new IpTcpServerChannel();
        
        ipTcpServerChannel.TcpServer.TimeoutEnabled = false;
        ipTcpServerChannel.TcpServer.MaxLineLengthEnabled = false;

I did notice that we are using IpTcpServerChannel now, instead of the IpSuperHttpServerChannel that we used to use before. I’m not certain if that will actually changed anything. Just communicating it for clarity.

mh · June 28, 2024, 1:36pm

Thanx.

the same issue persists with the Tcp server as well, I take it?

—marc

infokevercom · June 28, 2024, 2:10pm

So in my original post, I believe it was an IpTcpServerChannel.

However:

(Whats below is confusion, it is unclear to me is the summary)

The issue today is between two servers, (One running on IpSuperHttpServerChannel, the other running a IpTcpServerChannel), and call each others service methods (among other things they do).

The server with the IpSuperHttpServerChannel crashes with the stack overflow. But it might be crashing as a client replying to a service method like this:

 // This is running behind an IpTcpServerChannel
  [ServiceMethod]
        public void CreateReport(out Binary[] files, out Binary[] logs)

To the best of my knowledge, this issue is concerning the IpTcpServerChannel. However sadly its not fully clear to me at this point because I’m having issues with reproducing it currently. If I get more information I’ll let you know.

mh · June 29, 2024, 1:34pm

Hm now I am confused .

even if both apps are servers, for the purpose of this, lets think of one of them as a client, as the crash, I presume, happens due to one app (the “client”) calling a method on the other (the “server”). It doesnt matter then that the “client” also acts as a server.

Which channel is the one receiving the call (and crashing due to the call) using? IpSuperHttpServerChannel?

infokevercom · July 24, 2024, 3:22pm

Sorry for the slow responses. We were in the middle of other issues that needed attention.

The crash happened on the server with an IpSuperHttpServerChannel. As it responds to the message it received. I think if you remove the message size limit, I would hypothesis that the crash will happen regardless of the server channel (Assuming it also has this recursive method to sending the data).

ResponseBodyCallback->
BeginWrite->
IntBeginWrite->
TaskAsyncResult->
ResponseBodyCallback ^

I believe this is the reason why it causes a server crash.

Starting with the .NET Framework 2.0, you can’t catch a StackOverflowException object with a try /catch block, and the corresponding process is terminated by default.

The Server in this case is not exiting (If it did, that’s not so bad, as in our environment at least it would simply restart). Instead it simply becomes unreachable.

mh · July 24, 2024, 3:34pm

Same here; I ha meant to start ,looking into this issue regardless, but was pulled into other stuff. I’ll try to reproduce this and have a look, this week.

Yes, that is expected.StackOverflows are always fatal, which makes Italy the mor important to avoid/fix them.

thanx,
marc

mh · July 24, 2024, 4:38pm

Hmm. I’m afraid I cannot reproduce any error with a really simple case. I’m transfering a random ~45MB file that happened to me in my Downloads folder as Binary, and return its size.

    [ServiceMethod]
    method DoSomething(aBinary: Binary): String;
    begin
      result := aBinary.Length.ToString();
    end;

      var lBytes := File.ReadBytes("/Users/mh/Downloads/...");
      var lBinary := new RemObjects.SDK.Types.Binary(lBytes);
      try
        writeLn(new ServerAccess().Service1.DoSomething(lBinary));
      except
        on E: Exception do
          Log($"E {E}");
      end;

Can you have a look at the attached test case and see if either (a) you can run it and see if it works or fails for you (adjust the file path in the client) or (b) anything about the setup of the channel strikes you as that I missed something that might be crucial?

thanx,
marc

ServerApp.zip (93.6 KB)

infokevercom · July 25, 2024, 12:31pm

I reproduced it here, I think the key part that was missing is:

IpSuperHttpServerChannel serverChannel = new IpSuperHttpServerChannel();
serverChannel.ServePlainHttpRequests = true;

Then connecting doing the call via http (or https).

RemObjectsSizeTest.zip (61.9 KB)

Here it is as code:

        public static int Main(string[] args)
        {
            Console.WriteLine("Starting service!");

            ApplicationServer server = new ApplicationServer("test", "test.test.test", "testServiceLibrary", "test test");

            Console.WriteLine("Server created! Opening port....");
            int portNumber = 9050;


            IpSuperHttpServerChannel serverChannel = new IpSuperHttpServerChannel();
            
            server.NetworkServer.ServerChannel = serverChannel;
            server.NetworkServer.Port = 9050;

            serverChannel.ServePlainHttpRequests = true;

            Console.WriteLine("Starting reporting micro service in port: " + portNumber);
            
            // Start another thread, that will call Service1.GetBigBinaryFile service method, after 3 seconds.
            var t = new System.Threading.Thread(() =>
            {
                System.Threading.Thread.Sleep(3000);
                Console.WriteLine("Calling GetBigBinaryFile service method...");
                
                ServerAccess serverAccess = new ServerAccess();
                var service1 = serverAccess.Service1;
                service1.GetBigBinaryFile(out var log);
                
                
                
                Console.WriteLine($"GetBigBinaryFile service method called! Size is {log.Length}");
            });
            
            t.Start();
            
            server.Run(args);
            return 0;
        }

Also in my ServerAccess I have:

        public ServerAccess()
        {
            this._serverUrl = "http://localhost:9050/bin";
            this._clientChannel = ClientChannel.ChannelMatchingTargetUri(this._serverUrl);
            BinMessage binMessage = new BinMessage();
            binMessage.EnforceMaxMessageSize = false;
            this._message = binMessage;
        }

This gives me the stack overflow issue before.

This is the GetBigBinaryFile btw:

        [ServiceMethod]
        public void GetBigBinaryFile(out Binary log)
        {
            // Make a 100MB binary file with random data
            byte[] data = new byte[100 * 1024 * 1024];
            new Random().NextBytes(data);
            log = new Binary(data);
        }

mh · July 25, 2024, 12:43pm

Ah, so the client uses plain HTTP?

Still works fine for me though, with new IpHttpClientChannel(TargetUrl := "http://localhost:8099/bin");

mh · July 25, 2024, 12:49pm

infokevercom:

This is the GetBigBinaryFile btw:

        [ServiceMethod]
        public void GetBigBinaryFile(out Binary log)
        {
            // Make a 100MB binary file with random data
            byte[] data = new byte[100 * 1024 * 1024];
            new Random().NextBytes(data);
            log = new Binary(data);
        }

Oh you’re sending the big data BACK. I’ll test.i was sending it as an in parameter FROM the client.

mh · July 25, 2024, 12:53pm

Still fine, here for me

infokevercom · July 25, 2024, 12:53pm

Yes you appear to be correct, it works when its IpHttpServerChannel.

The issue appears to be with IpSuperHttpServerChannel, we have different clients, and some (this is from vague memory, I can’t recall why) required ServePlainHttpRequests to be true.

I think it might have been javascript clients needing a non IpSuperHttpServerChannel connection? But for our other clients we did want the super connection.

Maybe I’m missing something about the function of ServePlainHttpRequests.
But its still nice to know that this isn’t an issue with IpHttpServerChannel does not have this issue.

mh · July 25, 2024, 12:55pm

FWIW, your testcase also works for me:

~> Process ReportingService started, took 1.263.
~> Attaching to 7307
Starting service!
Server created! Opening port....
Starting reporting micro service in port: 9050
test test

Press Ctrl+C to terminate
Calling GetBigBinaryFile service method...
GetBigBinaryFile service method called! Size is 104857600

infokevercom · July 25, 2024, 12:56pm

Oh, what version of rem objects are you using? Maybe this has already been addressed? (I’m currently on 10.0.0.1589)