Delphi - Problem With Set String and PAnsiChar and Other Strings not Displaying

Question

I was getting advice from Rob Kennedy and one of his suggestions that greatly increased the speed of an app I was working on was to use SetString and then load it into the VCL component that displayed it.

I'm using Delphi 2009 so now that PChar is Unicode,

SetString(OutputString, PChar(Output), OutputLength.Value);
edtString.Text := edtString.Text + OutputString;

Works and I changed it to PChar myself but since the data being moved isn't always Unicode in fact its usually ShortString Data.... so onto what he actually gave me to use:

SetString(OutputString, PAnsiChar(Output), OutputLength.Value);
edtString.Text := edtString.Text + OutputString;

Nothing shows up but I check in the debugger and the text that normally appeared the way I did it building 1 char at a time in the past was in the variable.

Oddly enough this is not the first time I ran into this tonight. Because I was trying to come up with another way, I took part of his advice and instead of building into a VCL's TCaption I built it into a string variable and then copied it, but when I send it over nothing's displayed. Once again in the debugger the variable that the data is built in... has the data.

for I := 0 to OutputLength.Value - 1 do
begin
  OutputString := OutputString + Char(OutputData^[I]);
end;
edtString.Text := OutputString;

The above does not work but the old slow way of doing it worked just fine....

for I := 0 to OutputLength.Value - 1 do
begin
  edtString.Text := edtString.Text + Char(OutputData^[I]);
end;

I tried making the variable a ShortString, String and TCaption and nothing is displayed. What I also find interesting is while I build my hex data from the same array into a richedit it's very fast while doing it inside an edit for the text data is very very slow. Which is why I haven't bothered trying to change the code for the richedit as it works superfast as it is.

Edit to add - I think I sorta found the problem but I have no solution. If I edit the value in the debugger to remove anything that can't be displayed (which by the old method used to just not display... not fail) then what I have left is displayed. So if it's just a matter of getting rid of bytes that were turned to characters that are garbage how can I fix that?

I basically have incoming raw data from a SCSI device that's being displayed hex-editor style. My original slow style of adding one char at a time successfully displayed strings and Unicode strings that did not have Unicode-specific characters in them. The faster methods even if working won't display ShortStrings one way and the other way wont display UnicodeStrings that aren't using non 0-255 characters. I really like and could use the speed boost but if it means sacrificing the ability to read the string... then what's the point in the app?

EDIT3 - Alright now that I've figured out that 0-31 is control char and 32 and up is valid I think I'm gonna make an attempt to filter char and replace those not valid with a . which is something I was planning on doing later to emulate hex editor style.

If there are any other suggestions I'd be glad to hear about them but otherwise I think I can craft a solution that's faster than the original and does what I need it to at the same time.

Answer 1

I used PAnsiChar in my example for a reason. It looked like OutputLength was being measured in bytes, not characters, so I made sure to use a type whose length is always measured in bytes. You'll also notice that I showed the declaration of OutputString as an AnsiString.

Since the edit control stored Unicode, though, there will be a conversion between AnsiString and UnicodeString. That will take the system's current code page into account, but that's probably not what you want. You might want to declare the variable as a RawByteString instead. That won't have any code page associated with it, so there won't be any unexpected conversions.

Don't use strings for storing binary data. If you're building what amounts to a hex editor, then you're working with binary data. It's important to remember that. Even if your binary data happens to consist mostly of bytes that can be interpreted as text, you can't treat the data as text or you'll run into exactly the problems you're seeing — characters that don't appear as expected. If you get a bunch of bytes from your SCSI device, then store them in an array of bytes, not characters.

In hex editors, you'll notice that they always show the hexadecimal values of the bytes. They might show those bytes interpreted as characters, but that's secondary, and they generally only show the bytes that can represent ASCII characters; they don't try to get too fancy with the basic display. The good hex editors will offer to display the data interpreted as wide characters, too. This aids in debugging because the user can look at the same data in multiple ways. But they're just views of the data. They're not actually changing the binary contents of the data.

Answer 2

Some comments:

1. Your question is very unclear. What exactly do you want to do?
2. Your question reads terrible, please check your text with a spelling checker.
3. The question you are referring to is this one: Delphi accessing data from dynamic array that is populated from an untyped pointer
4. Please give a complete code sample of your function like you did in your previous question, I want to know if you implemented Rob Kennedy's suggestion or the code you gave yourself in a following answer (let's hope not :) )
5. As far a I understand your question: You're sending a query to your SCSI device and you get an array of bytes back which you store in the variable OutputData. After that you want to show your data to the user. So your real question is: How to show an array of bytes to the user?
6. Login as the same user and don't create an account for every new question. That way we can track your question history an can find out what you mean by 'getting advice'.

Some assumptions and suggestions if I'm right about the true meaning of your question:

1. Showing your data as a hex string won't give any problems
2. Showing your data in a normal Memo field gives you problems, although a Delphi string can contain any character, including 0 bytes, displaying them will give you problems. A TMemo for example will show your data till the first 0 byte. What you have to do (and you gave the answer yourself), is replacing non viewable characters with a dummy. After that you can show your data in a TMemo. Actually all hex viewers do the same, characters that cannot be printed will be shown as a dot.

Answer 3

When you filter out non viewable characters...You'll probably need to decide what to do with a couple of them like #9(Tab), #10(LF), #11(Verticle Tab), #12(FF-or New Page),#13(CR)