Delphi WideString and Delphi 2009+

Question

I am writing a class that will save wide strings to a binary file. I'm using Delphi 2005 for this but the app will later be ported to Delphi 2010. I'm feeling very unsure here, can someone confirm that:

1. A Delphi 2005 WideString is exactly the same type as a Delphi 2010 String

2. A Delphi 2005 WideString char as well as a Delphi 2010 String char is guaranteed to always be 2 bytes in size.

With all the Unicode formats out there I don't want to be hit with one of the chars in my string suddenly being 3 bytes wide or something like that.

Edit: Found this: "I indeed said UnicodeString, not WideString. WideString still exists, and is unchanged. WideString is allocated by the Windows memory manager, and should be used for interacting with COM objects. WideString maps directly to the BSTR type in COM." at http://www.micro-isv.asia/2008/08/get-ready-for-delphi-2009-and-unicode/

Now I'm even more confused. So a Delphi 2010 WideString is not the same as a Delphi 2005 WideString? Should I use UnicodeString instead?

Edit 2: There's no UnicodeString type in Delphi 2005. FML.

Answer 1

For your first question: WideString is not exactly the same type as D2010's string. WideString is the same COM BSTR type that it has always been. It's managed by Windows, with no reference counting, so it makes a copy of the whole BSTR every time you pass it somewhere.

UnicodeString, which is the default string type in D2009 and on, is basically a UTF-16 version of the AnsiString we all know and love. It's got a reference count and is managed by the Delphi compiler.

For the second, the default char type is now WideChar, which are the same chars that have always been used in WideString. It's a UTF-16 encoding, 2 bytes per char. If you save WideString data to a file, you can load it into a UnicodeString without trouble. The difference between the two types has to do with memory management, not the data format.

Answer 2

As others mentioned, string (actually UnicodeString) data type in Delphi 2009 and above is not equivalent to WideString data type in previous versions, but the data content format is the same. Both of them save the string in UTF-16. So if you save a text using WideString in earlier versions of Delphi, you should be able to read it correctly using string data type in the recent versions of Delphi (2009 and above).

You should take note that performance of UnicodeString is way superior than WideString. So if you are going to use the same source code in both Delphi 2005 and Delphi 2010, I suggest you use a string type alias with conditional compiling in your code, so that you can have the best of both worlds:

type
  {$IFDEF Unicode}
  MyStringType = UnicodeString;
  {$ELSE}
  MyStringType = WideString;
  {$ENDIF}

Now you can use MyStringType as your string type in your source code. If the compiler is Unicode (Delphi 2009 and above), then your string type would be an alias of UnicodeString type which is introduced in Delphi 2009 to hold Unicode strings. If the compiler is not unicode (e.g. Delphi 2005) then your string type would be an alias for the old WideString data type. And since they both are UTF-16, data saved by any of the versions should be read by the other one correctly.

Answer 3

1. A Delphi 2005 WideString is exactly the same type as a Delphi 2010 String

That is not true - ex Delphi 2010 string has hidden internal codepage field - but probably it does not matter for you.

1. A Delphi 2005 WideString char as well as a Delphi 2010 String char is guaranteed to always be 2 bytes in size.

That is true. In Delphi 2010 SizeOf(Char) = 2 (Char = WideChar).

---

There cannot be different codepage for unicode strings - codepage field was introduced to create a common binary format for both Ansi strings (that need codepage field) and Unicode string (that don't need it).

If you save WideString data to stream in Delphi 2005 and load the same data to string in Delphi 2010 all should work OK.

WideString = BSTR and that is not changed between Delphi 2005 and 2010

UnicodeString = WideString in Delphi 2005 (if UnicodeString type exists in Delphi 2005 - I don't know) UnicodeString = string in Delphi 2009 and above.

---

@Marco - Ansi and Unicode strings in Delphi 2009+ have common binary format (12-byte header).

UnicodeString codepage CP_UTF16 = 1200;

Answer 4

The rule is simple:

• If you want to work with unicode strings inside your module only - use UnicodeString type (*).
• If you want to communicate with COM or with other cross-module purposes - use WideString type.

You see, WideString is a special type, since it's not native Delphi type. It is an alias/wrapper for BSTR - a system string type, intendent for using with COM or cross-module communications. Being a unicode - is just a side-effect.

On the other hand, AnsiString and UnicodeString - are native Delphi types, which have no analog in other languages. String is just an alias for either AnsiString or UnicodeString.

So, if you need to pass string to some other code - use WideString, otherwise - use either AnsiString or UnicodeString. Simple.

P.S.

(*) For old Delphi - just place

{$IFNDEF Unicode}

type
  UnicodeString = WideString;

{$ENDIF}

somewhere in your code. This fix will allow you to write the same code for any Delphi version.

Answer 5

While a D2010 char is always and exactly 2 bytes, the same character folding and combining issues are present in UTF-16 characters as in UTF-8 characters. You don't see this with narrow strings because they're codepage based, but with unicode strings it's possible (and in some situations common) to have affective but non-visible characters. Examples include the byte order mark (BOM) at the start of a unicode file or stream, left to right/right to left indicator characters, and a huge range of combining accents. This mostly affects questions of "how many pixels wide will this string be on the screen" and "how many letters are in this string" (as distinct from "how many chars are in this string"), but also means that you can't randomly chop characters out of a string and assume they're printable. Operations like "remove the last letter from this word" become non-trivial and depend on the language in use.

The question about "one of the chars in my string suddenly being 3 bytes long" reflects a little confustion about how UTF works. It's possible (and valid) to take three bytes in a UTF-8 string to represent one printable character, but each byte will be a valid UTF-8 character. Say, a letter plus two combining accents. You will not get a character in UTF-16 or UTF-32 being 3 bytes long, but it might be 6 bytes (or 12 bytes) long, if it's represented using three code points in UTF-16 or UTF-32. Which brings us to normalisation (or not).

But provided you are only dealing with the strings as whole things, it's all very simple - you just take the string, write it to a file, then read it back in. You don't have to worry about the fine print of string display and manipulation, that's all handled by the operating system and libraries. Strings.LoadFromFile(name) and Listbox.Items.Add(string) work exactly the same in D2010 as in D2007, the unicode stuff is all transparent to you as a programmer.

Answer 6

I am writing a class that will save wide strings to a binary file.

When you write the class in D2005 you will be using Widestring When you migrate to D2010 Widestring will still be valid and work properly. Widestring in D2005 is the same as WideString in D2010.

The fact that String=WideString in D2010 need not be considered since the compiler deals with those issues easily.

Your input routine to save with (AString: String) need only one line entering the proc

procedure SaveAStringToBIN_File(AString:String);
var wkstr : Widestring;
begin
{$IFDEF Unicode}  wkstr := AString;      
{$ELSE}           wkstr := UTF8Decode(AString);   {$ENDIF}
...
   the rest is the same saving a widestring to a file stream
  write the length (word) of string then data 

end;