Severe Code First startup scalability issue during RTTI → RODL generation (benchmark included)

Remoting SDK
1

Hello,

We have been investigating a severe startup-time issue in one of our RemObjects SDK for Delphi servers that uses Code First services.

Before posting this, we profiled the complete RTTI → RODL generation pipeline and validated multiple hypotheses. One of our initial assumptions turned out to be wrong, so the conclusions below are based entirely on measurements.

Our intention is not to suggest a specific implementation, but rather to verify whether the behavior we found is expected.

Environment

  • RemObjects SDK for Delphi 10.0.0.1613
  • Delphi 12 Athens
  • Reproducible on both Win32 and Win64

Benchmark

To isolate the problem, we created a benchmark consisting only of trivial Code First services.

Each service contains:

  • one [ROService] class
  • one simple method
  • no DTOs
  • no HTTP attributes
  • no Swagger attributes
  • no business logic

The benchmark measures only RTTI → RODL generation:

srv := TMyServer.Create(nil);

reader := srv.RTTIReader;

sw := TStopwatch.StartNew;

reader.ReadRODLResource(Stream, ‘’, True, False);

sw.Stop;

Results:

Services Time Generated RODL
250 37 ms 137 KB
500 239 ms 275 KB
1000 1619 ms 550 KB
2000 13550 ms 1101 KB
4000 109674 ms 2203 KB

The generated RODL grows linearly.

Generation time grows dramatically (approximately O(N²·⁹) in this benchmark).

First hypothesis (discarded)

Initially we suspected the repeated TList.Contains() calls inside FindAllROTypes().

We replaced those lookups with hash-based lookups.

The result was almost no measurable improvement, so that hypothesis turned out to be incorrect.

Profiling

Profiling shows that almost the entire execution time is spent inside the generation loop of TRORTTIRODLReader.DoProcess().

FindAllROTypes() itself represents only a very small fraction of the total runtime.

What appears to be the bottleneck

The dominant cost appears to come from repeated linear lookups performed during the Generate* phase.

For example:

  • GenerateService() → TRODLLibrary.FindService()
  • GenerateStruct() → TRODLLibrary.FindStruct()
  • GenerateArray() → TRODLLibrary.FindArray()
  • GenerateEnum() → TRODLLibrary.FindEnum()
  • GenerateException() → TRODLLibrary.FindException()

Each Find*() performs a linear scan over a collection that keeps growing during generation.

Proof of concept

As an experiment, we replaced those repeated linear lookups with hash-based lookups during generation.

Results:

Services Original Experimental lookup
500 239 ms 30 ms
1000 1619 ms 59 ms
2000 13550 ms 126 ms
4000 109674 ms 291 ms

The generated RODL remained byte-identical.

We are not suggesting this proof-of-concept as the final implementation.

Our intention is simply to report what appears to be an algorithmic scalability issue and verify whether this matches your understanding of the current implementation.

If useful, we can also provide:

  • the benchmark project;
  • the profiling instrumentation;
  • the experimental patch used during the investigation.

Thank you.

2

Hi,

Thx for your benchmark.
We will optimize TRODLLibrary class.

3

Logged as bugs://D19689.

4

bugs://D19689 was closed as fixed.

5

Hi,

I’ve used code like

code 
const
  total = 4000;
var
  i: Integer;
  svc: TRODLService;
  str: TRODLStruct;
  str_f: TRODLTypedEntity;
  arr: TRODLArray;
  op: TRODLOperation;
  op_p: TRODLOperationParam;
  sw: TStopwatch;
  enum: TRODLEnum;
begin
  fRodl.Clear;
  memo1.Lines.Add('total iterations =' + total.ToString);
  sw := TStopwatch.StartNew;
  for i := 0 to total - 1 do begin
    enum := TRODLEnum.Create;
    enum.Name := 'enum'+ i.ToString;
    enum.Add.Name:='value0';
    enum.Add.Name:='value1';

    str := TRODLStruct.Create;
    str.Name := 'struct'+ i.ToString;
    str_f := str.Add;
    str_f.Name := 'prop'+ i.ToString;
    str_f.DataType := 'Integer';
    fRodl.Add(str);

    arr := TRODLArray.Create;
    arr.Name := 'array' + i.ToString;
    arr.ElementType := str.Name;
    fRodl.Add(arr);

    svc := TRODLService.Create;
    svc.Name := 'svc' + i.ToString;
    if svc.Default = nil then svc.Add;
    op := svc.Default.Add;
    op.Name := 'operation'+ i.ToString;
    op_p := op.Add;
    op_p.Name := 'parameter'+ i.ToString;
    op_p.DataType := str.Name;
    op_p := op.AddResult;
    op_p.DataType := arr.Name;
    fRodl.Add(svc);
  end;
  sw.Stop;
  memo1.Lines.Add('generation = '+sw.ElapsedMilliseconds.ToString+'ms');
  memo1.Lines.Add('total elements in RODL library = '+ fRodl.Count.ToString);
  sw := TStopwatch.StartNew;
  for i := 0 to total - 1 do begin
    fRodl.FindService('svc' + i.ToString);
  end;
  sw.Stop;
  memo1.Lines.Add('findService ='+sw.ElapsedMilliseconds.ToString+'ms');

was:

total iterations =4000
generation = 21ms
total elements in RODL library = 12000
findService  =135463ms

now:

total iterations =4000
generation = 19ms
total elements in RODL library = 12000
findService =35ms
6

Thank you, Evgeny. Those results confirm the bottleneck very clearly.

Could you please let us know in which SDK build/version this fix will be available, and whether there is a beta or interim build we can test?

We would also like to rerun our production benchmark after updating, where RTTI → RODL generation currently takes approximately 43 seconds.

7

Hi,

drop email to support@ where specify your account name which has valid license and I’ll send updated units to you.

by other hand, you can drop PM here to me or support team with the same info.