Note: It might be that the issue is a bug outside RO/DA, we are still investigating, but I already wanted to query if there are known issues.
Also if a stable new RO/DA version is released (ideally both .NET & delphi versions) then I can test if the latest version solves this issue. Please keep me informed.
Hi Frederic,
It turns out to be the server side of the problem you reported earlier, which we already fixed, and the fix is in the build we sent you.)
the workaround is the same as the client-side one in the other thread
using System.Net.Sockets;
using RemObjects.SDK;
public class TimeoutAwareSslServerFactory : IConnectionFactory
{
private readonly SslConnectionFactory fSslOptions;
private readonly int fHandshakeTimeout;
public TimeoutAwareSslServerFactory(SslConnectionFactory sslOptions, int handshakeTimeout)
{
this.fSslOptions = sslOptions;
this.fHandshakeTimeout = handshakeTimeout;
}
public Connection CreateServerConnection(Socket socket)
{
this.fSslOptions.LoadCertificate();
var inner = new InnerConnection(socket);
inner.TimeoutEnabled = true;
inner.Timeout = this.fHandshakeTimeout;
return new SslConnection(this.fSslOptions, inner);
}
public Connection CreateClientConnection(Binding binding)
{
return this.fSslOptions.CreateClientConnection(binding);
}
// SslConnection reads whatever has already arrived. A plain Connection would
// block for a full buffer here and stall healthy handshakes.
private class InnerConnection : Connection
{
public InnerConnection(Socket socket) : base(socket) { }
public override int Read(byte[] buffer, int offset, int size)
{
return this.ReceiveWhatsAvailable(buffer, offset, size);
}
}
}
Set it on the channel after SslOptions is configured, before the server is opened:
var channel = (IpTcpServerChannel)server.NetworkServer.ServerChannel;
channel.TcpServer.ConnectionFactory =
new TimeoutAwareSslServerFactory(channel.SslOptions, channel.TcpServer.Timeout);
This only applies with TLS enabled on that channel. If SslOptions.Enabled is off there, let us know then the threads are following connections that stay open, and we should look at that instead.
Thanks, I will try the build you sent me (i was a bit hesitant up until now because we have an entire toolchain using RO and deadlines coming up so switching is delicate)
Please note that the memory/thread/handle issue is on a .NET FW 4.7.2 RO server inside the LAN of our customers and this RO server only listens to a plain TCP channel. There is no SSL involved.
that is exactly what I meant by “if SslOptions.Enabled is off there, let us know”…
With a plain TCP channel that fix does not apply, so that build will not change this, unfortunately.
Could you tell me what Server.Timeout and Server.TimeoutEnabled are set to on that channel? And if you can catch the process while the thread count is high, a Parallel Stacks view (or a dump) would show where those threads are actually standing.
Note: I posted a callstack (with dubious info) from sysinternals process explorer in a previous post.
On our development machines we cannot reproduce the issue.
This might be an issue totally unrelated to RO/DA, the server uses a library (from the belgian government) of questionable quality.
Maybe a simple question which already helps pinpointing the issue: We use pooled services & no or very few extra services need to be created. Does RO/DA itself create extra threads in that case?
It looks to be the case, see the log at the bottom. This surprises me a bit. Service instances are pooled for performance reasons but there is no thread pool (for the same reason)?
2026-09-02 12:26:39.2784|DEBUG|EHealthProxy.HelperClasses.ConnectorHelper|About to call PrepareService for EHealthProxy RO service function GFDGetStatus service worker object 29 OSThrID 17444 ManagedThrId 451
2026-09-02 12:27:26.6511|DEBUG|EHealthProxy.HelperClasses.ConnectorHelper|About to call PrepareService for EHealthProxy RO service function RecipeListReservations service worker object 29 OSThrID 28736 ManagedThrId 599
2026-09-02 12:27:33.0415|INFO|EHealthProxy.HelperClasses.ConnectorHelper|Recip-e found 0 reservations (page:0) after 1/01/0001 0:00:00
2026-09-02 12:27:33.0415|INFO|EHealthProxy.HelperClasses.ConnectorHelper|Recipe ListReservations for startdate 1/01/0001 0:00:00 returned code:100 message::? prescrstatus:
2026-09-02 12:27:58.2506|DEBUG|EHealthProxy.HelperClasses.ConnectorHelper|About to call PrepareService for EHealthProxy RO service function RecipeListReservations service worker object 29 OSThrID 692 ManagedThrId 546
2026-09-02 12:28:04.3164|INFO|EHealthProxy.HelperClasses.ConnectorHelper|Recip-e found 0 reservations (page:0) after 30/12/1899 0:00:00
2026-09-02 12:28:04.3164|INFO|EHealthProxy.HelperClasses.ConnectorHelper|Recipe ListReservations for startdate 30/12/1899 0:00:00 returned code:100 message::? prescrstatus:
2026-09-02 12:28:33.0745|DEBUG|EHealthProxy.HelperClasses.ConnectorHelper|About to call PrepareService for EHealthProxy RO service function RecipeListReservations service worker object 29 OSThrID 25168 ManagedThrId 737
2026-09-02 12:28:39.2609|INFO|EHealthProxy.HelperClasses.ConnectorHelper|Recip-e found 0 reservations (page:0) after 1/01/0001 0:00:00
2026-09-02 12:28:39.2609|INFO|EHealthProxy.HelperClasses.ConnectorHelper|Recipe ListReservations for startdate 1/01/0001 0:00:00 returned code:100 message::? prescrstatus:
2026-09-02 12:29:39.2830|DEBUG|EHealthProxy.HelperClasses.ConnectorHelper|About to call PrepareService for EHealthProxy RO service function RecipeListReservations service worker object 29 OSThrID 6692 ManagedThrId 679
2026-09-02 12:29:45.1892|INFO|EHealthProxy.HelperClasses.ConnectorHelper|Recip-e found 0 reservations (page:0) after 1/01/0001 0:00:00
2026-09-02 12:29:45.1892|INFO|EHealthProxy.HelperClasses.ConnectorHelper|Recipe ListReservations for startdate 1/01/0001 0:00:00 returned code:100 message::? prescrstatus:
2026-09-02 12:30:45.1955|DEBUG|EHealthProxy.HelperClasses.ConnectorHelper|About to call PrepareService for EHealthProxy RO service function RecipeListReservations service worker object 29 OSThrID 5648 ManagedThrId 682
2026-09-02 12:30:52.1122|INFO|EHealthProxy.HelperClasses.ConnectorHelper|Recip-e found 0 reservations (page:0) after 1/01/0001 0:00:00
2026-09-02 12:30:52.1122|INFO|EHealthProxy.HelperClasses.ConnectorHelper|Recipe ListReservations for startdate 1/01/0001 0:00:00 returned code:100 message::? prescrstatus:
2026-09-02 12:31:06.2818|DEBUG|EHealthProxy.HelperClasses.ConnectorHelper|About to call PrepareService for EHealthProxy RO service function RecipeListReservations service worker object 29 OSThrID 24004 ManagedThrId 435
2026-09-02 12:31:12.6262|INFO|EHealthProxy.HelperClasses.ConnectorHelper|Recip-e found 0 reservations (page:0) after 30/12/1899 0:00:00
2026-09-02 12:31:12.6262|INFO|EHealthProxy.HelperClasses.ConnectorHelper|Recipe ListReservations for startdate 30/12/1899 0:00:00 returned code:100 message::? prescrstatus:
2026-09-02 12:31:52.1133|DEBUG|EHealthProxy.HelperClasses.ConnectorHelper|About to call PrepareService for EHealthProxy RO service function RecipeListReservations service worker object 29 OSThrID 26532 ManagedThrId 595
2026-09-02 12:31:58.0180|INFO|EHealthProxy.HelperClasses.ConnectorHelper|Recip-e found 0 reservations (page:0) after 1/01/0001 0:00:00
2026-09-02 12:31:58.0180|INFO|EHealthProxy.HelperClasses.ConnectorHelper|Recipe ListReservations for startdate 1/01/0001 0:00:00 returned code:100 message::? prescrstatus:
2026-09-02 12:32:58.0231|DEBUG|EHealthProxy.HelperClasses.ConnectorHelper|About to call PrepareService for EHealthProxy RO service function RecipeListReservations service worker object 29 OSThrID 29384 ManagedThrId 548
2026-09-02 12:33:05.9758|INFO|EHealthProxy.HelperClasses.ConnectorHelper|Recip-e found 0 reservations (page:0) after 1/01/0001 0:00:00
2026-09-02 12:33:05.9758|INFO|EHealthProxy.HelperClasses.ConnectorHelper|Recipe ListReservations for startdate 1/01/0001 0:00:00 returned code:100 message::? prescrstatus:
In your log the pooled instance stays the same while the thread changes on every call, so every call arrives on a new connection. Your thread count is really your open connection count.
After the 300s timeout, inactive connections are closed, so the accumulated connections aren’t inactive - either something is still flowing through them, or the call hasn’t completed yet
>Your thread count is really your open connection count.
This is very useful information. Below a thread count of 142 on a customers machine and the TCP info of that same proces.
Our server is listening on port 8988 so there are no “clients” that have left connections open.
The other connections are towards the internet.
To rule out RO/DA code we would need to know if the threads that got started at some point (e.g. in the second screenshot on 18,19 & 20 august) were created by RO/DA.
I initially suspected RO/DA because I log the RO/DA thread IDs and I found them in my logs but now I am reconsidering because the timing is off. e.g TID 17228 was created on 17/8/26 12:21 by RO/DA but the thread that hangs was created later 19/8/26 11:28 (but also happens to have TID 17228)
So you needn’t investigate this any further. The issue is probably in another 3rd party library.
Sorry for the load & thanks for the help.
Process in production with slowly increasing thread count + open network connections
