WebAssembly and file upload to a module

Please view a project from the first post. In js I have a buffer :

array = new Uint8Array(arrayBuffer);
program.Parse(array);

then in program Parse :

    method Parse( buf : EcmaScriptObject ) ;
    begin
      var buffer := new EcmaScriptObject( WebAssemblyCalls.GetArray(buf.Handle, 0) );
      var lTotal := WebAssemblyCalls.GetStringLength(buffer);
      var res := new Byte[lTotal];
      WebAssemblyCalls.ResponseBinaryTextToArray(buffer, @res[0]);

      // parse buffer[] or memory stream
    end;

I’m not sure how to handle EcmaScriptObject to process that buf array. I want to copy buf to res array. Is it possible with a copy method or byte per byte?

This:

    method Parse( buf : EcmaScriptObject ) ;
    begin
      var mem := new Byte[Convert.ToInt32(buf.GetMember('length', 0, []))];
      buf.CopyToNativeArray(0, mem, 0, mem.Length);
      
      for i: Integer := 0 to length(mem)-1 do
        writeln(mem[i]);
    end;

Will work for the next build. CopyToNativeArray (and from) can be used to interact with TypedArrays. (Alternatively, grab my last change in IslandRTL and you can do the same locally)

1 Like

I moved forward with upload of geojson file but for bigger (2-3MB) files during parsing json (custom parser) I get error:
5e574853-a7f8-4a79-b3a8-4a5859f05fbe
Have you ever seen this error before? I get this crash when using str.ToUpper or str.ToLower methods on string variable in wasm. The VS debug watch shows that str variable is empty ‘’ but str.Length or length(str) returns 4. I thought about rtl function error, but a simple case works fine. It must be a memory issue of a bigger project. Small json files are parsed correctly, so we are executing the same code but for different files. We are passing a FileReader result to wasm, maybe a garbage collector messes here?
Any idea what could be wrong? I’m unable to reproduce it using a small test case.

This is pretty weird yeah. I don’t think. So what happens is that:

export function readCharsFromMemory(offs, len: number): string
    {
        var arr = new Int16Array(mem.buffer, offs, len);
        var s: string = "";
        for (var i: number = 0; i < len; i++)
        s+= String.fromCharCode(arr[i]);
        return s;
    }

-2092578804 is a rather curious number though. That’s in a call to the dynamic dispatch, but the name is complete garbage. Any chance I can see this? It’s like it completely runs out of memory.

I have another question. Here is the test project.Module5.zip (5.1 KB)
I’m trying to return an object from wasm to javascript and call its method. How to do it?

        var c = result.Class1();
        var obj = result.Class2();
        c.Test(obj,'js');
        var c3 = result.Class3();
        c3.Log();
        var o = c3.GetClass() ;
        console.log(o);
        o.Log('ok'); <-- ??? how to use it 

I get : Message: TypeError: o.Log is not a function

This sound like it might be the same issue as WebAssembly and factory method - #7 by mh, or at least related…

Logged as bugs://E25391.

Hi this one:

 var o = c3.GetClass ;

should be:

 var o = c3.GetClass();

With this modification and the fixes from today’s build should be working properly.
Thanks!

1 Like

bugs://E25391 was closed as fixed.

Carlo, I think I managed to reproduce the problem in a short test case (actually I’m the one who came across this one and told Artur). I consistently get the following stack trace:

Uncaught RangeError: invalid or out-of-range index
    readCharsFromMemory RemObjectsElements.js:228
    __island_to_upper RemObjectsElements.js:329
    res RemObjectsElements.js:267
    readCharsFromMemory RemObjectsElements.js:228
    __island_to_upper RemObjectsElements.js:329
    mi_s7_ToUppernb RemObjectsElements.js line 699 > WebAssembly.instantiate:1758754
    mi_t15_wasm__input_d_Program6_doLoadnt13__1sEcmaScriptObject RemObjectsElements.js line 699 > WebAssembly.instantiate:1847484
    mi_t15_wasm__input_d_Program14__l_OpenFile_g_b____0nt13__1sEcmaScriptObject RemObjectsElements.js line 699 > WebAssembly.instantiate:1850142
    mi_t16__1sWebAssemblyDelegate6_Invokent13__1sEcmaScriptObject RemObjectsElements.js line 699 > WebAssembly.instantiate:1557079
    __island_call_delegate RemObjectsElements.js line 699 > WebAssembly.instantiate:1765788
    res RemObjectsElements.js:267

Compiled with Elements 11.0.0.2675, tested on the latest Firefox and Chrome. There is a sample file in the GML directory that you need to select in the file input element (it probably can be any other few-MBs-or-larger file but this is what I tested on). The index.html is in Bin\Debug\WebAssembly and the project is updated accordingly.

wasm_input.zip (3.8 MB)

Logged as bugs://E25421.

Thanks @asiwecki !

Does this work for you?

      var len : Integer := buf.length;
      var arr := new Byte[len];
      EcmaScriptObject(buf).CopyToNativeArray(0, arr, 0, len);

The problem with the original approach is that due to the dynamic dispatch, you’re doing 6 million * ~10 allocations. I think it’s just running out of memory faster than it can GC. Above works for me. Canyou try and see if that works in the full testcase too?

It does fix the test case and improve the situation in our project but still there are few-MBs-long files which fail in the same way. The test case is not a realistic scenario, it just loads the file into memory. Normally you need to parse the file, create objects etc. and when this happens the memory consumption goes through the roof.

I think that the real problem is the memory management. I compared the same parser compiled for .NET and WASM on the same files and WASM takes at least 10 times more memory (at least, could be much more) e.g. 30 MB vs 400 MB, 50 MB vs 2.5 GB etc.

We are aware of the WASM limitations but this needs to be improved to be usable for us. Is there any way to force object disposal or, at least, force GC to collect (like GC.Collect() in .NET)?

You should be able to call GC.Collect(), yes.

note that WASM has a limit of 2 GB memory total (that includes the stack)

Hi
Could you be so kind and verify whether GC.Collect() really works for wasm? Can you also explain when the GC actually activates (is it monitoring available memory) and how to give it a chance to work?
Does it work “the same way” as in Island for Windows?
Calling GC.Collect() manually is not a good idea for the same reason as in .NET.
I understand that a wasm object requires more wrapped code around but why there is so much memory overhead compared to other platforms?
As Adam said, we e.g. parse gml or json files creating many objects in a complex manner but also releasing them in the meantime. Do you need more assistance or a complex project for this issue to help us in finding a solution?

Damn. Completely lost track of this thread. Sorry about that. The GC should run automatically in a (javascript) timer (ie after your code isn’t running anymore and control returned to javascript).

Note that all native Island stuff has the same memory usage as on other platforms, what has a bit of overhead is dynamic dispatch, but there are tricks to avoid that. One example is above, instead of asking for data by iterating a js byte array , get all data over in 1 go.

1 Like

Carlo, thank you for the clues. As I’ve mentioned before I made changes in my project and now I copy JS arrays instead of iterating through them. It improved the situation a little bit but still the amount of consumed memory is excessive. The same code run on .NET takes a tiny fraction of memory compared to WASM. I’ve tried to tinker with GC, hand control over back to JS but nothing seems to help. Would you please write me a piece of code that forces garbage collection and/or disposal of an object?

GC.Collect should force a GC ofc, but I’m now curious as to what you’re doing exactly that causes this. Can you show a hot path? So I can see whats going on.

Sorry for the long pause, unfortunately covid hit me few days ago and I wasn’t in a “working condition” up to now…

GC.Collect should force a GC

I don’t see a global GC object, only SimpleGC but it doesn’t have a Collect method. SimpleGCExt has it but what does the c argument mean? Is it the max generation to collect (as in .NET)? How to get the max generation value then?

but I’m now curious as to what you’re doing exactly that causes this. Can you show a hot path?

I’ve discarded all the code changes before the last post in hope that you’ll send me something that works :smile:.

1 Like