Island C source file import question, handling C inline functions

Thank you very much. The importing works for Windows/Island now. But it seems still having the following problems:

  1. still complains about endianness for Linux importing…
  2. seems it only imports the struct types in cstring_internal.h, none of the inline functions are imported.

That’s good to hear. I’ll reopen the issue but need help from marc to import this properly on windows (Some error about tgz).

bugs://85752 got reopened.

I fixed the endianness defines. One reason you don’t get the inline methods like:

static inline uint32_t TF_swap32(uint32_t host_int) {

is that those aren’t included yet. If I add:


#import "include/tensorflow/core/platform/ctstring.h"
#import "include/tensorflow/core/platform/ctstring_internal.h"

to import.h it fixes them. I do get some warnings on some of them though:

Can you tell me which ones you actually need? So I can prioritize which ones should work first.

_byteswap_ulong
memset
free
switch
memcpy
memset

I think all of them are needed… :smile:

So you need TF_String* to use this properly?

Yes. Indeed need those TF_String_XXXX functions from encoding/deconding TensorFlow data. I believe those inline functions would enhance runtime performance for large-scale data - that is the reason they are introduced.

Many thanks.

I need a tgz library for Echoes to support downloading and unzipping .tar.gz files in Windows (on Mac I shell out to the command line tool).

if you know.have one, I can add it easily;. else, workaround, download/extract yourself and fix the reference.

Marc

Is there any literature (documentation, blog) providing more details on Island fx and the importer, especially the functionalities to interop with C?

Apart from DLL, this importer also works with static library, and C obj file, right?

I think this is a great feature (like the capability of importing C inline functions). I would like to add the information to a research paper that I am working on.

The importer just turns the “.h” into binary data in there “.fx”. it’s really unrelated to where the implementation comes from (.dll, .lib or object) — aside from the LinkLibrary setting, which gets encoded in the .fx to tell the build chain what binaries to pass to the linker.

Right now, the only official documentation we have is what’s at FXGen.

yours,
marc

Understand. The capability of processing C inline functions is a nice story to tell.

I think we support pretty much everything except switch at the moment. The missing symbols are just that, things it doen’t find properly.

2 Likes

Oke so it processes it now, but fails with these:

which are just missing symbols, I have to think where to put those best.

Is this finalized? I downloaded the latest, and having the following method calling inline functions TF_TString_Init and TF_TString_Copy, but got Internal Error. Let me know if you need a new test case.

  method ScalarStringTensor(const aStr: ^AnsiChar; aStatus: ^TF_Status): ^TF_Tensor;
  begin
    const TF_TSTRING_SIZE: Integer = 24;
    var strlen := lstrlenA(aStr);
    result := TF_AllocateTensor(TF_DataType.TF_STRING, nil, 0, TF_TSTRING_SIZE);
    var data := ^TF_TString(TF_TensorData(result));
    TF_TString_Init(data);
    TF_TString_Copy(data, aStr, strlen);
  end; 

Severity Code Description Project File Line Suppression State
Error (E0) Internal error: System.InvalidOperationException: Not valid type for GEP
at LLVM.BitCode.LLVMModule.InstGEP(LLVMBaseValue aVal, LLVMBaseValue[] aArgs)
at RemObjects.Elements.Island.IslandOutput.GenerateArrayAccessExpression(Expression aExpr, Boolean aAddress, Boolean aSkipBounds)
at RemObjects.Elements.Island.IslandOutput.GenerateExpression(Expression aExpr, Boolean aAddress)
at RemObjects.Elements.Island.IslandOutput.GenerateStatement(Statement aSt)
at RemObjects.Elements.Island.IslandOutput.GenerateStatement(Statement aSt)
at RemObjects.Elements.Island.IslandOutput.GenerateStatement(ScopeStatement asT)
at RemObjects.Elements.Island.IslandOutput.GenerateStatement(Statement aSt)
at RemObjects.Elements.Island.IslandOutput.GenerateStatement(Statement aSt)
at RemObjects.Elements.Island.IslandOutput.GenerateStatement(ScopeStatement asT)
at RemObjects.Elements.Island.IslandOutput.GenerateStatement(Statement aSt)
at RemObjects.Elements.Island.IslandOutput.GenerateStatement(Statement aSt)
at RemObjects.Elements.Island.IslandOutput.GenerateStatement(ScopeStatement asT)
at RemObjects.Elements.Island.IslandOutput.GenerateStatement(Statement aSt)
at RemObjects.Elements.Island.IslandOutput.GenerateMethod(IMethodInfo ameth)
at RemObjects.Oxygene.Code.CombinedParsedType.ForAllMethods(Func2 action) at RemObjects.Elements.Island.IslandOutput.GenerateParsedType(IMutableParsedType aType) at RemObjects.Elements.Island.IslandOutput.<GenerateExecutable>b__19(IParsedType a) at System.Linq.Enumerable.WhereSelectEnumerableIterator2.MoveNext()
at RemObjects.Elements.Island.IslandOutput.GenerateExecutable()
at RemObjects.Oxygene.Code.Compiler.Compiler.GenerateExecutable()
at RemObjects.Oxygene.Code.Compiler.Compiler.Compile() ApiHelpers C:\Program Files (x86)\Microsoft Visual Studio\2019\Professional\MSBuild\RemObjects Software\Elements\RemObjects.Elements.Island.Windows.targets 70

Not yet. Some missing symbols and I just haven’t gotten around to testing it. the tricky part is always making all moving parts work well together

No, I believe that’s still ToDo.

Yes, please; this would be a different/unrelated error to the missing symbols.

bugs://85752 got closed with status fixed.

I downloaded the latest 2618 - I still have the following warning for the IMPORT project, on Windows?
inline function “TF_swap32” has unsupported element: include/tensorflow/core/platform/ctstring_internal.h (33) unknown identifier _byteswap_ulong [C:\DEVL

bugs://85752 got reopened.

so when you import now, you have 1 or 2 .h files extra with custom content right?

What needs to happen now is that we need to provide these ourselves, as our Islandrtl does not have everything the c library has (I only added stuff on demand). What I think the sanest approach here is, is in stdio.h or stdlib.h (the empty files) you add the signatures for these functions, like:

extern unsigned long _byteswap_ulong ( unsigned long val );

Then it imports and you can provide this function in your own code with:

[SymbolName('_byteswap_ulong')]
class method _byteswap_ulong(l: Cardinal): Cardinal; 
begin 
  exit (l shr 24) or (l shl 24) or ((l shr 8) and $ff00)or ((l shl 8) and $ff0000); 
// note i wrote this from memory; it might be wrong!
end;