Island C source file import question, handling C inline functions

In the latest Google TensorFlow version 2.4, the C API has a new header file which is composed of a few in-line functions for string processing.

Previously similar functionalities are provided through DLL exports (or TensorFlow.dll) but now they are provided as inline source functions.

The questions - can Island cross compile this header file? How can I Import this file into Island project?

@ck @mh

Have you tried, and got errors?

IIRC we support inline functions, if the C parser (which is mostly complete, even though targeted at .h files and not implementations) can handle them, and they would be represented in the .fx as syntax tree (much as they wudkl if you yourself define an inline function and build the project to a library+fx…

This is great to know that Island has C Parser. On TensorFlow official website, this new C API change (of directly using C inline functions) causes quite big headache for .NET and Rust bindings. I guess that is the reason why we should use Island!

That being said, I tried to import the latest TensorFlow 2.4, but wasn’t very successful. Could you help me take a look? Import-tf-240.zip (39.9 KB)

Two things to NOTE:

  1. They reference standard C header files like stdlib.h etc. I am not 100% sure how to handle that.

  2. The inline functions have MACROS like below, for different platforms (Windows, MacOS, Linux), again, I am not sure how to handle that:

    #if (defined(__BYTE_ORDER__) && defined(__ORDER_LITTLE_ENDIAN__) && \
         __BYTE_ORDER__ == __ORDER_LITTLE_ENDIAN__) ||                  \
        defined(_WIN32)
    #define TF_TSTRING_LITTLE_ENDIAN 1
    #elif defined(__BYTE_ORDER__) && defined(__ORDER_BIG_ENDIAN__) && \
        __BYTE_ORDER__ == __ORDER_BIG_ENDIAN__
    #define TF_TSTRING_LITTLE_ENDIAN 0
    #else
    #error "Unable to detect endianness."
    #endif
    
    #if defined(__clang__) || \
        (defined(__GNUC__) && \
         ((__GNUC__ == 4 && __GNUC_MINOR__ >= 8) || __GNUC__ >= 5))
    static inline uint32_t TF_swap32(uint32_t host_int) {
      return __builtin_bswap32(host_int);
    }
    
    #elif defined(_MSC_VER)
    static inline uint32_t TF_swap32(uint32_t host_int) {
      return _byteswap_ulong(host_int);
    }
    
    #elif defined(__APPLE__)
    static inline uint32_t TF_swap32(uint32_t host_int) {
      return OSSwapInt32(host_int);
    }

:wink:

This should be covered in rtf.fx, assuming you have that referenced in your import project?

we do not support “function” macros, but these just defines just add constants, so that should work fine… I’ll have a look and log as needed.

FWIW, I get

               Extracting '/Users/mh/Library/Application Support/RemObjects Software/EBuild/Remotes/27964E082C18C507E05AE4DD3633437580963978/libtensorflow-cpu-darwin-x86_64-2.4.0.tar.gz'
W:             File '/Users/mh/Library/Application Support/RemObjects Software/EBuild/Remotes/27964E082C18C507E05AE4DD3633437580963978/z/lib/libtensorflow_framework.2.3.0.dylib' was not found in downloaded archive.
E:             Import file '/Users/mh/Library/Application Support/RemObjects Software/EBuild/Remotes/27964E082C18C507E05AE4DD3633437580963978/z/lib/libtensorflow_framework.2.3.0.dylib' does not exist.

the other two platforms are ok, so i’ll just disable macOS for now…

with that out of the way I get

E: “Unable to detect endianness.” [/Users/mh/Downloads/Import 3/include/tensorflow/core/platform/ctstring_internal.h (29)]

for Linux and

E: Cannot find include file: stdlib.h [/Users/mh/Downloads/Import 3/include/tensorflow/core/platform/ctstring.h (20)]

for Windows.

both seem odd, so I’ll go that for the compiler team.

Thanks, logged as bugs://85752

workaround for the first might be to define BYTE_ORDER and ORDER_BIG_ENDIAN manually yourself…

#if defined(_MSC_VER) && !defined(__clang__)
    #define __ORDER_LITTLE_ENDIAN__    0x4d2
    #define __ORDER_BIG_ENDIAN__       0x10e1
    #define __BYTE_ORDER__             __ORDER_LITTLE_ENDIAN__
#endif

So I added the above macros to import.h, and it seems working, and the importer no longer complains endianness. But, now the importer complains that "Cannot find include file: stdlib.h [C:\DEVLIBS\RO\Element\TensorFlow-Island\Island\Projects\Import\include\tensorflow\core\platform\ctstring.h (20)]

stdlib.h is standard C header files, located at C:\Program Files (x86)\Windows Kits\10\Include\10.0.19041.0\ucrt

  • Is there a way to add this include to search path of the importer?

I’ll have to leave that one to @ck to look at.

You can but is there actually something in there it needs? Sometimes they just get over excited in adding includes. If you add an empty stdlib.h next to the files, does it work then?

That is certainly a good idea. I commented out all those includes such as “stdlib.h”.

Now the importer doesn’t complain missing stdlib.h or any other headers. But, gives the following error:

:                include/tensorflow/core/platform/ctstring_internal.h (190340) One of semicolon, comma, (eof) expected Identifier (i64)
  // We assume SIZE_MAX % 16 == 0.
  size_t curr_cap_x2 = curr_cap >= 0xffffffffffffffffui64    / 2 ? 0xffffffffffffffffui64    - 1 : curr_cap * 2;

 [C:\DEVLIBS\RO\Element\TensorFlow-Island\Island\Projects\Import\include\tensorflow\core\platform\ctstring_internal.h (252)]
                  Parsing (root) file: include/tensorflow/c/tf_attrtype.h
                  Parsing (root) file: include/tensorflow/c/tf_datatype.h
                  Parsing (root) file: include/tensorflow/c/tf_status.h
                  Parsing (root) file: include/tensorflow/c/tf_tensor.h
                  Parsing (root) file: include/tensorflow/c/tf_tstring.h
E:                include/tensorflow/core/platform/ctstring_internal.h (248) One of semicolon, comma, (eof) expected Identifier (i64)
  // We assume SIZE_MAX % 16 == 0.
  size_t curr_cap_x2 = curr_cap >= 0xffffffffffffffffui64    / 2 ? 0xffffffffffffffffui64    - 1 : curr_cap * 2;

 [C:\DEVLIBS\RO\Element\TensorFlow-Island\Island\Projects\Import\include\tensorflow\core\platform\ctstring_internal.h (252)]
E:                include/tensorflow/core/platform/ctstring_internal.h (190343) One of semicolon, comma, (eof) expected Identifier (i64)
  // We assume SIZE_MAX % 16 == 0.
  size_t curr_cap_x2 = curr_cap >= 0xffffffffffffffffui64    / 2 ? 0xffffffffffffffffui64    - 1 : curr_cap * 2;

 [C:\DEVLIBS\RO\Element\TensorFlow-Island\Island\Projects\Import\include\tensorflow\core\platform\ctstring_internal.h (252)]
                  Importing failed due to these files:
                    include/tensorflow/c/c_api.h - include/tensorflow/core/platform/ctstring_internal.h (190340) One of semicolon, comma, (eof) expected Identifier (i64)
  // We assume SIZE_MAX % 16 == 0.
  size_t curr_cap_x2 = curr_cap >= 0xffffffffffffffffui64    / 2 ? 0xffffffffffffffffui64    - 1 : curr_cap * 2;


                    include/tensorflow/c/tf_tstring.h - include/tensorflow/core/platform/ctstring_internal.h (248) One of semicolon, comma, (eof) expected Identifier (i64)
  // We assume SIZE_MAX % 16 == 0.
  size_t curr_cap_x2 = curr_cap >= 0xffffffffffffffffui64    / 2 ? 0xffffffffffffffffui64    - 1 : curr_cap * 2;


                    __ImportSpec__TensorFlow.Island.Api.h - include/tensorflow/core/platform/ctstring_internal.h (190343) One of semicolon, comma, (eof) expected Identifier (i64)
  // We assume SIZE_MAX % 16 == 0.
  size_t curr_cap_x2 = curr_cap >= 0xffffffffffffffffui64    / 2 ? 0xffffffffffffffffui64    - 1 : curr_cap * 2;

This sounds like something I should look at yes.

2 Likes

This seems to suggest that Island C Parser has some problem parsing the value of SIZE_MAX 0xffffffffffffffffui64 as 64bit unsigned integer literal. SIZE_MAX is defined in both limits.h and stdint.h

The ui64 suffix is not C-standard integer literal suffix (it is Microsoft specific). See this discussion - c++ - Microsoft Integer Literal Extensions -- Where documented? - Stack Overflow

and this C Integer Constants | Microsoft Docs

So I guess the problem is caused by the non-standard, Microsoft-specific integer literal?

We’ll have to add support for that, yes. Workaround, can you just remove it? It is it in too many places?

It is not in many places. Just referenced in one single file.

But it is “compiled” in Island rtl file (as part of the C run time). I guess I can override it somewhere as a workaround. But I’d like to wait for the more graceful importer solution with an enhanced C parser.

Thanks, logged as bugs://85758

bugs://85758 got closed with status fixed.

bugs://85752 got closed with status fixed.

the main issue seems to be those ui64/i64 constants. It imports now.

2 Likes