Issues with response times for IpSuperHttpClientChannel on Microsoft Azure?

fvancrae · April 22, 2022, 1:19pm

Are there known issues with a RO super https server running on Azure?

I am using IpSuperHttpClientChannel and it works on most machines.
It also works on Azure with fast access times.
However on azure after a period of inactivity the client call takes 10 seconds extra.
After this one time speed penalty the speed is OK again.

For example a call takes 0.100 seconds, then about 5 minutes of no activity on the same channel we make the same call again and the call will take 10.100 seconds.

Maybe some firewall on azure throws away idle TCP connections which confues the RO super http channel?

When I trace the issue it is the Dispatch call which takes 10 seconds.

Client side we have:
this.clientChannel = RemObjects.SDK.ClientChannel.ChannelMatchingTargetUri(uri);
this.message = RemObjects.SDK.Message.MessageMatchingTargetUri(uri);
if (message is RemObjects.SDK.BinMessage)
{
((RemObjects.SDK.BinMessage)message).MaxMessageSize = Int32.MaxValue;
((RemObjects.SDK.BinMessage)message).MaxDecompressedMessageSize = Int32.MaxValue;
((RemObjects.SDK.BinMessage)message).UseCompression = true; // FVC: Note: By default true
}
if (this.clientChannel is RemObjects.SDK.IpSuperHttpClientChannel)
{
((RemObjects.SDK.IpSuperHttpClientChannel)this.clientChannel).ConnectTimeout = connecttimeout * 1000;
((RemObjects.SDK.IpSuperHttpClientChannel)this.clientChannel).RequestTimeout = 5601000;
}

Server side:
serverROSuperHTTP_ = new RemObjects.SDK.Server.ApplicationServer(serverName, serviceTypes);
// Enable traffic encryption
serverROSuperHTTP_.AutoCreateSelfSignedCertificate = false;
serverROSuperHTTP_.NetworkServer.UseTLS = true;
serverROSuperHTTP_.NetworkServer.Certificate = servercert;
serverROSuperHTTP_.NetworkServer.ServerChannel = new RemObjects.SDK.Server.IpSuperHttpServerChannel();
serverROSuperHTTP_.NetworkServer.Port = listenport;
serverROSuperHTTP_.NetworkServer.ServerChannel.BeforeSendStream+= networkstats_.onBeforeSendStream;
serverROSuperHTTP_.NetworkServer.ServerChannel.AfterReceiveStream += networkstats_.onAfterReceiveStream;

Is there a way to investigate this issue?

mh · April 22, 2022, 7:22pm

Just an idea (i don’t use Azure much, myself), but could it be that there’s a setting where Azure suspends your “machine” after inactivity, so it has to be woken up again for the next request?

fvancrae · April 22, 2022, 7:59pm

No, I don’t think it’s that. The machine is running lots of tasks.
The issue also seems client channel related. A second instance has a fast response at the exact same time.
Meanwhile I have switched to a regular RO HTTP channel and this fixes the problem.

mh · April 22, 2022, 8:02pm

Ah, curious. so this happens to the client, but only against an Azure server? I’m afraid I’ll have to leave this for my colleague Anton to look into next week.

Good to know. did you switch only the client, or did you have to switch the server channel too, to hide the issue? (SuperHttp servers can handle standard HTTP client requests, too)

fvancrae · April 23, 2022, 5:45am

Ach, I did not know this. I changed both sides.

FYI: I don’t think the issue is Azure specific. It surfaces very easy for me on Azure however.

antonk · April 24, 2022, 1:59pm

Actually it seems it IS specific to Azure
Azure WAF terminates idle connections after 4 minute timeout.

SuperHttp server channel sends a heartbeat messages to the connected clients.
However the issue is that these messages are used to ping servers to check client - server connectivity and to silently reconnect if needed.
SuperHttp connection consists of 2 Http connections - one for sending requests and another one to retrieve responses from the server.

So apparently client side receives heartbeat messages from the server and assumes that everything is OK. While at the same time its connection used to SEND messages to the server was silently closed by WAF. So the client channel is put into an inconsistent stare (please note that it manages to recover even from this half-closed state).

It seems that a possible solution is to call this method during server application startup: ServicePoint.SetTcpKeepAlive(Boolean, Int32, Int32) Method (System.Net) | Microsoft Learn

It enables TCP-level Keep-Alive packages to be sent automatically:

System.Net.ServicePoint.SetTcpKeepAlive(true, 30*1000, 30*1000);

Could you if possible test this solution on your environment? If it works it will be added as a part of AppServer initialization process.

Thanks in advance

fvancrae · April 26, 2022, 10:33am

FYI: I tried https client with a super https server but this fails with exception ‘Unexpected end of stream’

fvancrae · April 26, 2022, 10:40am

I believe other firewalls/routers also terminate idle connections so I consider it not to be azure specific

Regarding “System.Net.ServicePoint.SetTcpKeepAlive(true, 301000, 301000);”
SetTcpKeepAlive is an instance member function
How do I access the correct ServicePoint object?

mh · April 26, 2022, 12:17pm

Curious. What exact client/server channels. This not only should work, i know it does work for me, as I have servers live that respond to both.

i’ll have to leave this to @antonk.

fvancrae · April 26, 2022, 12:53pm

My server side code is

        serverROSuperHTTP_ = new RemObjects.SDK.Server.ApplicationServer("ServiceBroker", serviceTypes);
        // Enable traffic encryption
        serverROSuperHTTP_.AutoCreateSelfSignedCertificate = false;
        serverROSuperHTTP_.NetworkServer.UseTLS = true;
        serverROSuperHTTP_.NetworkServer.Certificate = new System.Security.Cryptography.X509Certificates.X509Certificate2(System.IO.Path.Combine(AppDomain.CurrentDomain.BaseDirectory, XXXXXXXXXXXXXXXXX);
        serverROSuperHTTP_.NetworkServer.ServerChannel = new RemObjects.SDK.Server.IpSuperHttpServerChannel();
        serverROSuperHTTP_.NetworkServer.Port = cfg.settings.tcpport;// Properties.Settings.Default.TCPPort;
        serverROSuperHTTP_.NetworkServer.ServerChannel.BeforeSendStream += networkstats_.onBeforeSendStream;
        serverROSuperHTTP_.NetworkServer.ServerChannel.AfterReceiveStream += networkstats_.onAfterReceiveStream;

My client side code is

private void SetupROConnection(int connecttimeout)
    {
      string uri = coninfo.getROUri();
      this.clientChannel = RemObjects.SDK.ClientChannel.ChannelMatchingTargetUri(uri);
      this.message = RemObjects.SDK.Message.MessageMatchingTargetUri(uri);
      if (message is RemObjects.SDK.BinMessage)
      {
        ((RemObjects.SDK.BinMessage)message).MaxMessageSize = Int32.MaxValue;
        ((RemObjects.SDK.BinMessage)message).MaxDecompressedMessageSize = Int32.MaxValue;
        ((RemObjects.SDK.BinMessage)message).UseCompression = true;  // FVC: Note: By default true
      }
      if (this.clientChannel is RemObjects.SDK.IpSuperHttpClientChannel)
      {
        ((RemObjects.SDK.IpSuperHttpClientChannel)this.clientChannel).ConnectTimeout = connecttimeout * 1000;
        ((RemObjects.SDK.IpSuperHttpClientChannel)this.clientChannel).RequestTimeout = 5*60*1000;
      }
      if (this.clientChannel is RemObjects.SDK.IpHttpClientChannel)
      {
        ((RemObjects.SDK.IpHttpClientChannel)this.clientChannel).Timeout = 5*60 * 1000; // FVC: To check if this is works
      }
      if (this.clientChannel is RemObjects.SDK.IpTcpClientChannel)
      {
        ((RemObjects.SDK.IpTcpClientChannel)this.clientChannel).Timeout = 5*60 * 1000; // FVC: To check if this is works        
      }
    }

The URL I feed to the function is “https://broker.officinall.be:7470/bin”
Maybe the https is the issue? Or the /bin?

mh · April 26, 2022, 1:25pm

Both should be ok, to access this server from a plain, non-super HTTP channel with TLS.

But I can reproduce that it doesn’t work for your server, I get “Message: Unexpected end of stream.”. will log.

RemObjectsSoftware · April 26, 2022, 1:27pm

Logged as bugs://D19259.