Island C source file import question, handling C inline functions

wuping · March 21, 2021, 10:25am

In the latest Google TensorFlow version 2.4, the C API has a new header file which is composed of a few in-line functions for string processing.

Previously similar functionalities are provided through DLL exports (or TensorFlow.dll) but now they are provided as inline source functions.

github.com

tensorflow/tensorflow/blob/master/tensorflow/core/platform/ctstring_internal.h

/* Copyright 2019 The TensorFlow Authors. All Rights Reserved.

Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at

    http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
==============================================================================*/

#ifndef TENSORFLOW_CORE_PLATFORM_CTSTRING_INTERNAL_H_
#define TENSORFLOW_CORE_PLATFORM_CTSTRING_INTERNAL_H_

#include <limits.h>
#include <stdint.h>

This file has been truncated. show original

The questions - can Island cross compile this header file? How can I Import this file into Island project?

@ck @mh

mh · March 21, 2021, 4:58pm

Have you tried, and got errors?

IIRC we support inline functions, if the C parser (which is mostly complete, even though targeted at .h files and not implementations) can handle them, and they would be represented in the .fx as syntax tree (much as they wudkl if you yourself define an inline function and build the project to a library+fx…

wuping · March 21, 2021, 6:01pm

This is great to know that Island has C Parser. On TensorFlow official website, this new C API change (of directly using C inline functions) causes quite big headache for .NET and Rust bindings. I guess that is the reason why we should use Island!

That being said, I tried to import the latest TensorFlow 2.4, but wasn’t very successful. Could you help me take a look? Import-tf-240.zip (39.9 KB)

Two things to NOTE:

They reference standard C header files like stdlib.h etc. I am not 100% sure how to handle that.

The inline functions have MACROS like below, for different platforms (Windows, MacOS, Linux), again, I am not sure how to handle that:

#if (defined(__BYTE_ORDER__) && defined(__ORDER_LITTLE_ENDIAN__) && \
     __BYTE_ORDER__ == __ORDER_LITTLE_ENDIAN__) ||                  \
    defined(_WIN32)
#define TF_TSTRING_LITTLE_ENDIAN 1
#elif defined(__BYTE_ORDER__) && defined(__ORDER_BIG_ENDIAN__) && \
    __BYTE_ORDER__ == __ORDER_BIG_ENDIAN__
#define TF_TSTRING_LITTLE_ENDIAN 0
#else
#error "Unable to detect endianness."
#endif

#if defined(__clang__) || \
    (defined(__GNUC__) && \
     ((__GNUC__ == 4 && __GNUC_MINOR__ >= 8) || __GNUC__ >= 5))
static inline uint32_t TF_swap32(uint32_t host_int) {
  return __builtin_bswap32(host_int);
}

#elif defined(_MSC_VER)
static inline uint32_t TF_swap32(uint32_t host_int) {
  return _byteswap_ulong(host_int);
}

#elif defined(__APPLE__)
static inline uint32_t TF_swap32(uint32_t host_int) {
  return OSSwapInt32(host_int);
}

mh · March 22, 2021, 12:47pm

This should be covered in rtf.fx, assuming you have that referenced in your import project?

we do not support “function” macros, but these just defines just add constants, so that should work fine… I’ll have a look and log as needed.

mh · March 22, 2021, 12:49pm

FWIW, I get

               Extracting '/Users/mh/Library/Application Support/RemObjects Software/EBuild/Remotes/27964E082C18C507E05AE4DD3633437580963978/libtensorflow-cpu-darwin-x86_64-2.4.0.tar.gz'
W:             File '/Users/mh/Library/Application Support/RemObjects Software/EBuild/Remotes/27964E082C18C507E05AE4DD3633437580963978/z/lib/libtensorflow_framework.2.3.0.dylib' was not found in downloaded archive.
E:             Import file '/Users/mh/Library/Application Support/RemObjects Software/EBuild/Remotes/27964E082C18C507E05AE4DD3633437580963978/z/lib/libtensorflow_framework.2.3.0.dylib' does not exist.

the other two platforms are ok, so i’ll just disable macOS for now…

mh · March 22, 2021, 12:50pm

with that out of the way I get

E: “Unable to detect endianness.” [/Users/mh/Downloads/Import 3/include/tensorflow/core/platform/ctstring_internal.h (29)]

for Linux and

E: Cannot find include file: stdlib.h [/Users/mh/Downloads/Import 3/include/tensorflow/core/platform/ctstring.h (20)]

for Windows.

both seem odd, so I’ll go that for the compiler team.

RemObjectsSoftware · March 22, 2021, 12:51pm

Thanks, logged as bugs://85752

mh · March 22, 2021, 12:58pm

workaround for the first might be to define BYTE_ORDER and ORDER_BIG_ENDIAN manually yourself…

wuping · March 22, 2021, 5:11pm

#if defined(_MSC_VER) && !defined(__clang__)
    #define __ORDER_LITTLE_ENDIAN__    0x4d2
    #define __ORDER_BIG_ENDIAN__       0x10e1
    #define __BYTE_ORDER__             __ORDER_LITTLE_ENDIAN__
#endif

So I added the above macros to import.h, and it seems working, and the importer no longer complains endianness. But, now the importer complains that "Cannot find include file: stdlib.h [C:\DEVLIBS\RO\Element\TensorFlow-Island\Island\Projects\Import\include\tensorflow\core\platform\ctstring.h (20)]

stdlib.h is standard C header files, located at C:\Program Files (x86)\Windows Kits\10\Include\10.0.19041.0\ucrt

Is there a way to add this include to search path of the importer?

mh · March 22, 2021, 6:04pm

I’ll have to leave that one to @ck to look at.

ck · March 22, 2021, 8:12pm

You can but is there actually something in there it needs? Sometimes they just get over excited in adding includes. If you add an empty stdlib.h next to the files, does it work then?

wuping · March 22, 2021, 9:16pm

That is certainly a good idea. I commented out all those includes such as “stdlib.h”.

Now the importer doesn’t complain missing stdlib.h or any other headers. But, gives the following error:

:                include/tensorflow/core/platform/ctstring_internal.h (190340) One of semicolon, comma, (eof) expected Identifier (i64)
  // We assume SIZE_MAX % 16 == 0.
  size_t curr_cap_x2 = curr_cap >= 0xffffffffffffffffui64    / 2 ? 0xffffffffffffffffui64    - 1 : curr_cap * 2;

 [C:\DEVLIBS\RO\Element\TensorFlow-Island\Island\Projects\Import\include\tensorflow\core\platform\ctstring_internal.h (252)]
                  Parsing (root) file: include/tensorflow/c/tf_attrtype.h
                  Parsing (root) file: include/tensorflow/c/tf_datatype.h
                  Parsing (root) file: include/tensorflow/c/tf_status.h
                  Parsing (root) file: include/tensorflow/c/tf_tensor.h
                  Parsing (root) file: include/tensorflow/c/tf_tstring.h
E:                include/tensorflow/core/platform/ctstring_internal.h (248) One of semicolon, comma, (eof) expected Identifier (i64)
  // We assume SIZE_MAX % 16 == 0.
  size_t curr_cap_x2 = curr_cap >= 0xffffffffffffffffui64    / 2 ? 0xffffffffffffffffui64    - 1 : curr_cap * 2;

 [C:\DEVLIBS\RO\Element\TensorFlow-Island\Island\Projects\Import\include\tensorflow\core\platform\ctstring_internal.h (252)]
E:                include/tensorflow/core/platform/ctstring_internal.h (190343) One of semicolon, comma, (eof) expected Identifier (i64)
  // We assume SIZE_MAX % 16 == 0.
  size_t curr_cap_x2 = curr_cap >= 0xffffffffffffffffui64    / 2 ? 0xffffffffffffffffui64    - 1 : curr_cap * 2;

 [C:\DEVLIBS\RO\Element\TensorFlow-Island\Island\Projects\Import\include\tensorflow\core\platform\ctstring_internal.h (252)]
                  Importing failed due to these files:
                    include/tensorflow/c/c_api.h - include/tensorflow/core/platform/ctstring_internal.h (190340) One of semicolon, comma, (eof) expected Identifier (i64)
  // We assume SIZE_MAX % 16 == 0.
  size_t curr_cap_x2 = curr_cap >= 0xffffffffffffffffui64    / 2 ? 0xffffffffffffffffui64    - 1 : curr_cap * 2;


                    include/tensorflow/c/tf_tstring.h - include/tensorflow/core/platform/ctstring_internal.h (248) One of semicolon, comma, (eof) expected Identifier (i64)
  // We assume SIZE_MAX % 16 == 0.
  size_t curr_cap_x2 = curr_cap >= 0xffffffffffffffffui64    / 2 ? 0xffffffffffffffffui64    - 1 : curr_cap * 2;


                    __ImportSpec__TensorFlow.Island.Api.h - include/tensorflow/core/platform/ctstring_internal.h (190343) One of semicolon, comma, (eof) expected Identifier (i64)
  // We assume SIZE_MAX % 16 == 0.
  size_t curr_cap_x2 = curr_cap >= 0xffffffffffffffffui64    / 2 ? 0xffffffffffffffffui64    - 1 : curr_cap * 2;

ck · March 22, 2021, 9:16pm

This sounds like something I should look at yes.

wuping · March 23, 2021, 3:24am

This seems to suggest that Island C Parser has some problem parsing the value of SIZE_MAX 0xffffffffffffffffui64 as 64bit unsigned integer literal. SIZE_MAX is defined in both limits.h and stdint.h

The ui64 suffix is not C-standard integer literal suffix (it is Microsoft specific). See this discussion - c++ - Microsoft Integer Literal Extensions -- Where documented? - Stack Overflow

and this C Integer Constants | Microsoft Learn

So I guess the problem is caused by the non-standard, Microsoft-specific integer literal?

mh · March 23, 2021, 12:34pm

We’ll have to add support for that, yes. Workaround, can you just remove it? It is it in too many places?

wuping · March 23, 2021, 1:52pm

It is not in many places. Just referenced in one single file.

But it is “compiled” in Island rtl file (as part of the C run time). I guess I can override it somewhere as a workaround. But I’d like to wait for the more graceful importer solution with an enhanced C parser.

RemObjectsSoftware · March 23, 2021, 2:02pm

Thanks, logged as bugs://85758

RemObjectsSoftware · March 23, 2021, 8:39pm

bugs://85758 got closed with status fixed.

RemObjectsSoftware · March 23, 2021, 8:55pm

bugs://85752 got closed with status fixed.

ck · March 23, 2021, 8:56pm

the main issue seems to be those ui64/i64 constants. It imports now.