Linq does not recognize utf-8 characters

Hi

Help please, I am connecting a Delphi server with a net core 2.1 application and when making a query through Linq the result does not recognize utf-8 characters, but if I call a function through the sdk it does recognize the utf-8 characters.

Please provide more details

This is an example where not recognize utf-8 characters .

Are you sure it’s not just the output that cannot display the characters properly? AFAIL the Windows console and Visual Studio’s debug console do not support Unicode.

can you look at the value of the string in the debugger?

This is an example debugger

Regarding Visual Studio output - seems this is the issue you have: https://developercommunity.visualstudio.com/content/problem/176587/unicode-characters-in-output-window.html

Now to the more pressing concern: could you say exact type of the Description field in the daSchema file and also show how the Description property is defined in the TableDefinitions file (including attributes applied to this definition)?

Thanks in advance

Level RO. 9.4.109.1377

image

TableDefinitions file (including attributes applied to this definition)

[RemObjects.DataAbstract.Linq.TableName("SA_Student_Status")]
public partial class SA_Student_Status : System.ComponentModel.INotifyPropertyChanged
{
	public event System.ComponentModel.PropertyChangedEventHandler PropertyChanged;

	protected void OnPropertyChanged(string parameterName)
	{
		if (this.PropertyChanged != null)
			this.PropertyChanged.Invoke(this, new System.ComponentModel.PropertyChangedEventArgs(parameterName));
	}

	private System.Guid f____Student_Status_GUID;

	[RemObjects.DataAbstract.Linq.FieldName("Student_Status_GUID")]
	[RemObjects.DataAbstract.Linq.DataType(RemObjects.DataAbstract.Schema.DataType.Guid)]
	[RemObjects.DataAbstract.Linq.PrimaryKey]
	[RemObjects.DataAbstract.Linq.LogChanges]
	public System.Guid Student_Status_GUID
	{
		get
		{
			return this.f____Student_Status_GUID;
		}
		set
		{
			if (System.Collections.Generic.Comparer<System.Guid>.Default.Compare(this.f____Student_Status_GUID, value) != 0)
			{
				this.f____Student_Status_GUID = value;
				this.OnPropertyChanged("Student_Status_GUID");
			}
		}
	}

	private System.String f____Description;

	[RemObjects.DataAbstract.Linq.FieldName("Description")]
	[RemObjects.DataAbstract.Linq.DataType(RemObjects.DataAbstract.Schema.DataType.String)]
	[RemObjects.DataAbstract.Linq.LogChanges]
	public System.String Description
	{
		get
		{
			return this.f____Description;
		}
		set
		{
			if (System.Collections.Generic.Comparer<System.String>.Default.Compare(this.f____Description, value) != 0)
			{
				this.f____Description = value;
				this.OnPropertyChanged("Description");
			}
		}
	}

	private System.Boolean f____Active;

	[RemObjects.DataAbstract.Linq.FieldName("Active")]
	[RemObjects.DataAbstract.Linq.DataType(RemObjects.DataAbstract.Schema.DataType.Boolean)]
	[RemObjects.DataAbstract.Linq.LogChanges]
	public System.Boolean Active
	{
		get
		{
			return this.f____Active;
		}
		set
		{
			if (System.Collections.Generic.Comparer<System.Boolean>.Default.Compare(this.f____Active, value) != 0)
			{
				this.f____Active = value;
				this.OnPropertyChanged("Active");
			}
		}
	}
}

Thanks.

Let me first explain what causes the issue:

From what I see the database stores strings using 1-byte char encoding.

Then the Delphi server reads data and sends them to the client (in our case to a .NET app) using 1-byte char encoding.

Unfortunately the .NET code doesn’t know which exactly code page should it use to convert this 1-byte chars to the UTF-16 encoding it uses. So the default code page is used. This works well for usual English alphabet, but causes issues for extended chars.

There are 2 possible solutions:

  1. To change the way data is stored. No always feasible as it might break existing clients.

  2. To help the .NET client to understand which exactly code page it should be using to convert 1-byte encoding.

Assuming you are using the default DataModule class generated fro the .NET client code the code you need will look as this:

    public DataModule()
    {
        this.InitializeComponent();
        this.message.ClientID = Guid.NewGuid();
        this.dataStreamer.DefaultStringEncoding = System.Text.Encoding.GetEncoding("en-US");
    }

Note the last line here. You’ll have to provide the code page name used by your server app instead of “en-US”. After that the issue should be resolved.

THANK YOU

In the end I was fine with the following

dataStreamer.DefaultStringEncoding = System.Text.Encoding.GetEncoding(“ISO-8859-1”);

To support all other missing encoding of .net core, see also here: