WebAssembly and file upload to a module

Hi
What is the best solution to upload (or stream) a file (selected from a disc) to a wasm module? I would like to pass different files (also big ones) as a binary buffer (in chunks) or stream and consume in the wasm method for further parsing. Here is the test project :
Module2.zip (4.6 KB)
I know that using FileReader and file slice I can read in chunks, but don’t know how to use a buffer in wasm module :

var buffer := new EcmaScriptObject( WebAssemblyCalls.GetArray(buf.Handle, 0) );
how to convert buffer into byte[] or stream variables? The doc doesn’t say much on how to use webassembly API from island rtl.

Regards
Artur

This is uploading a file from browser → wasm right? You don’t want it to go to a server but process it wasm side?

Yes. Just JS and WASM.

So it’s been a while since I did this javascript side, but you end up with a uint8Array iirc. What needs to be done is that you need to allocate some memory Island side (new byte[size] will do), then, then I think we need to build some js side code that can copy from 1 buffer to the other buffer (the wasm memory is a buffer too).

Ill have to find some time to demonstrate this, unless you think you know how to do it yourself.

1 Like

Please view a project from the first post. In js I have a buffer :

array = new Uint8Array(arrayBuffer);
program.Parse(array);

then in program Parse :

    method Parse( buf : EcmaScriptObject ) ;
    begin
      var buffer := new EcmaScriptObject( WebAssemblyCalls.GetArray(buf.Handle, 0) );
      var lTotal := WebAssemblyCalls.GetStringLength(buffer);
      var res := new Byte[lTotal];
      WebAssemblyCalls.ResponseBinaryTextToArray(buffer, @res[0]);

      // parse buffer[] or memory stream
    end;

I’m not sure how to handle EcmaScriptObject to process that buf array. I want to copy buf to res array. Is it possible with a copy method or byte per byte?

This:

    method Parse( buf : EcmaScriptObject ) ;
    begin
      var mem := new Byte[Convert.ToInt32(buf.GetMember('length', 0, []))];
      buf.CopyToNativeArray(0, mem, 0, mem.Length);
      
      for i: Integer := 0 to length(mem)-1 do
        writeln(mem[i]);
    end;

Will work for the next build. CopyToNativeArray (and from) can be used to interact with TypedArrays. (Alternatively, grab my last change in IslandRTL and you can do the same locally)

1 Like

I moved forward with upload of geojson file but for bigger (2-3MB) files during parsing json (custom parser) I get error:
5e574853-a7f8-4a79-b3a8-4a5859f05fbe
Have you ever seen this error before? I get this crash when using str.ToUpper or str.ToLower methods on string variable in wasm. The VS debug watch shows that str variable is empty ‘’ but str.Length or length(str) returns 4. I thought about rtl function error, but a simple case works fine. It must be a memory issue of a bigger project. Small json files are parsed correctly, so we are executing the same code but for different files. We are passing a FileReader result to wasm, maybe a garbage collector messes here?
Any idea what could be wrong? I’m unable to reproduce it using a small test case.

This is pretty weird yeah. I don’t think. So what happens is that:

export function readCharsFromMemory(offs, len: number): string
    {
        var arr = new Int16Array(mem.buffer, offs, len);
        var s: string = "";
        for (var i: number = 0; i < len; i++)
        s+= String.fromCharCode(arr[i]);
        return s;
    }

-2092578804 is a rather curious number though. That’s in a call to the dynamic dispatch, but the name is complete garbage. Any chance I can see this? It’s like it completely runs out of memory.

I have another question. Here is the test project.Module5.zip (5.1 KB)
I’m trying to return an object from wasm to javascript and call its method. How to do it?

        var c = result.Class1();
        var obj = result.Class2();
        c.Test(obj,'js');
        var c3 = result.Class3();
        c3.Log();
        var o = c3.GetClass() ;
        console.log(o);
        o.Log('ok'); <-- ??? how to use it 

I get : Message: TypeError: o.Log is not a function

This sound like it might be the same issue as WebAssembly and factory method - #7 by mh, or at least related…

Logged as bugs://E25391.

Hi this one:

 var o = c3.GetClass ;

should be:

 var o = c3.GetClass();

With this modification and the fixes from today’s build should be working properly.
Thanks!

1 Like

bugs://E25391 was closed as fixed.

Carlo, I think I managed to reproduce the problem in a short test case (actually I’m the one who came across this one and told Artur). I consistently get the following stack trace:

Uncaught RangeError: invalid or out-of-range index
    readCharsFromMemory RemObjectsElements.js:228
    __island_to_upper RemObjectsElements.js:329
    res RemObjectsElements.js:267
    readCharsFromMemory RemObjectsElements.js:228
    __island_to_upper RemObjectsElements.js:329
    mi_s7_ToUppernb RemObjectsElements.js line 699 > WebAssembly.instantiate:1758754
    mi_t15_wasm__input_d_Program6_doLoadnt13__1sEcmaScriptObject RemObjectsElements.js line 699 > WebAssembly.instantiate:1847484
    mi_t15_wasm__input_d_Program14__l_OpenFile_g_b____0nt13__1sEcmaScriptObject RemObjectsElements.js line 699 > WebAssembly.instantiate:1850142
    mi_t16__1sWebAssemblyDelegate6_Invokent13__1sEcmaScriptObject RemObjectsElements.js line 699 > WebAssembly.instantiate:1557079
    __island_call_delegate RemObjectsElements.js line 699 > WebAssembly.instantiate:1765788
    res RemObjectsElements.js:267

Compiled with Elements 11.0.0.2675, tested on the latest Firefox and Chrome. There is a sample file in the GML directory that you need to select in the file input element (it probably can be any other few-MBs-or-larger file but this is what I tested on). The index.html is in Bin\Debug\WebAssembly and the project is updated accordingly.

wasm_input.zip (3.8 MB)

Logged as bugs://E25421.

Thanks @asiwecki !

Does this work for you?

      var len : Integer := buf.length;
      var arr := new Byte[len];
      EcmaScriptObject(buf).CopyToNativeArray(0, arr, 0, len);

The problem with the original approach is that due to the dynamic dispatch, you’re doing 6 million * ~10 allocations. I think it’s just running out of memory faster than it can GC. Above works for me. Canyou try and see if that works in the full testcase too?

It does fix the test case and improve the situation in our project but still there are few-MBs-long files which fail in the same way. The test case is not a realistic scenario, it just loads the file into memory. Normally you need to parse the file, create objects etc. and when this happens the memory consumption goes through the roof.

I think that the real problem is the memory management. I compared the same parser compiled for .NET and WASM on the same files and WASM takes at least 10 times more memory (at least, could be much more) e.g. 30 MB vs 400 MB, 50 MB vs 2.5 GB etc.

We are aware of the WASM limitations but this needs to be improved to be usable for us. Is there any way to force object disposal or, at least, force GC to collect (like GC.Collect() in .NET)?

You should be able to call GC.Collect(), yes.

note that WASM has a limit of 2 GB memory total (that includes the stack)

Hi
Could you be so kind and verify whether GC.Collect() really works for wasm? Can you also explain when the GC actually activates (is it monitoring available memory) and how to give it a chance to work?
Does it work “the same way” as in Island for Windows?
Calling GC.Collect() manually is not a good idea for the same reason as in .NET.
I understand that a wasm object requires more wrapped code around but why there is so much memory overhead compared to other platforms?
As Adam said, we e.g. parse gml or json files creating many objects in a complex manner but also releasing them in the meantime. Do you need more assistance or a complex project for this issue to help us in finding a solution?