We are using Delphi 10.3.2, Remoting sdk VERSION 10.0.0.1555, Synapse SuperTcp , Olympia/InMemorySessoinManager, Windows
(the problem happens using olympia or not)
In my application from time to time we get a deadlock situation where new requests cant be processed, in the last RO version the lock in the sessionmanager was fixed, but now we notice more deadlocks, in this case when using callbacks
After analyze some callstacks we belive we identify the problem, in the uROBaseSuperTCPServer.pas file, function TSendEvent.Callback
fOwner.fClients.LockList;
try
{$IFDEF ROUseGenerics}
if not fOwner.fGuidToClientMap.TryGetValue(fClientGuid, l_worker) then Exit;
if fWorkerOverride <> nil then
l_worker := TROSCServerWorker(fWorkerOverride.GetTransportObject);
{$ELSE}
i := fOwner.fGuidToClientMap.IndexOf(GUIDToString(fClientGuid));
if i = -1 then Exit;
if fWorkerOverride <> nil then
l_worker := TROSCServerWorker(fWorkerOverride.GetTransportObject)
else
l_worker := TROSCServerWorker(fOwner.fGuidToClientMap.Objects[i]);
{$ENDIF}
//Isso não deveria estar fora deste critical?
l_ask := l_worker.SendPackage(l_stream, 0);
finally
fOwner.fClients.UnlockList;
end;
If l_worker.SendPackage takes too much time all my server will be block for that time, so lets say we have a really bad connection that takes 10s to fulfill the request, every client will wait for 10 seconds to process new requests, wich is not ideal
here is where all client will wait
procedure TROBaseSuperTCPServer.IntExecute(ABaseSuperConnection: TROBaseSuperTcpConnection);
...
l_worker := TROSCServerWorker.Create(Self, ABaseSuperConnection);
l_worker.MaxPackageSize := Self.MaxPackageSize;
//fClient esta bloqueado
fClients.Add(l_worker);
try
l_worker.DoExecute;
finally
fClients.Remove(l_worker);
....
end;
Now, in production we have from time to time a very very very long frozen application situation (at least a few minutes, we never wait to see if will finish or not) we couldn’t reproduce that, but we can reproduce freeze all requests on the server for a few seconds.
Steps to reproduce
- Compile both client and server projetcs
- Open 1 server and 2 client.exe (we use two computers, but i think will work on the same pc)
- Configure client.exe if ipadress/port
- Open clumsy, or any other program that you like to fake a bad connection
- On client1 click “Login and register for callback” them reserve, everything will works fine
- On server click “Send callback”, on the client1 you will see a callback counter increase, as expected
- Open perfmom on windows, go in network and get the port the client use to connect on server
- On clumsy put this filter tcp.DstPort == PORT_HERE or tcp.SrcPort == PORT_HERE
- On clumy choose 2000 on the lag field and click start
- On server click “Send Callback” befor the 2 seconds try to connecton on the second client, you will not be able until the 2 seconds mark
- To be sure the clumsy will not affect the second client, you can stop it and test i you like
Now, im not sure why this critical is beggin use to be honest, there is 3 locks total when a callback is send, this one fOwner.fClients.LockList; dosent seems right to me, but off course im not 100% sure
Demo.rar (2.2 MB)