Reading/writing dynamic arrays of objects to a file - Delphi

Question

Im trying to write some code which will read/write a dynamic array of objects to a file. The objects represent the structure of the java source code. I need to be able to scan the whole source code and gather information on Fields, Methods and Classes. I have an algorithm which does this and the result is kept in a structure of TFieldStruc, TMethodStruc and TClassStruc, all descendants of the TCoreStruc (a descendant of TObject). The java source code takes a couple of minutes to be scanned and have the virtual structure generated. Because of this my application scans all the source code once and saves it into a much more accessible format which is loaded when ever the IDE launches.

Is there a way (other than exporting the objects 'to string' and then re-creating them again when they are loaded) to stream the entire three arrays of TFieldStruc, TMethodStruc and TClassStruc, to a file so they can be read later?

I have tried reading and writing to a 'File of TFieldStruc..' and the TFileStream to save the objects to a file and read them back, but in both cases I get 'inaccessible value' in the debugger followed by an 'Access Violation' error as soon as the object is accessed again. If anyone has ideas on how to solve this problem it would be appreciated. Below is the code to TCodeStruc if any of its fields/methods may be causing issues:

type
  TCoreStruc = class(TObject)
    public
      LowerPointer : integer;
      HigherPointer : integer;
      Line : integer;
      Word : integer;
      Char : integer;
      CoreType : ansistring;
      IsPublic : boolean;
      IsPrivate : boolean;
      IsStatic : boolean;
      IsFinal : boolean;
      Name : ansistring;
      NestedStruc : TCoreStruc;
      constructor Create(Name, CoreType : ansistring; NestedStruc : TCoreStruc; IsPublic, IsPrivate, IsStatic, IsFinal : boolean);
      procedure UpdateValues(NestedStruc : TCoreStruc; IsPublic, IsPrivate, IsStatic, IsFinal : boolean);
      procedure SetPosition(Line, Word, Char : integer);
  end;

Answer 1

Here is an example using your structure.

A few notes about this:

• There are lots of different ways to go about this. Take David Heffernan's advice and do some searches for serialization. I use the approach include below in one of my applications but other include using RTTI/Persistent objects to iterate the published properties of an object. There are libraries that will iterate object for you and do all the work.

• You need to actually write out to the string the sizes of dynamic objects. That includes things like arrays and strings.

• In my example each object knows how to read and write itself to the stream.

• I use a struct for the fixed length parts of the object. That saves me from needing to write each data element individually.

• Strings need to be written on their own to include their size (you could used a fixed length buffer like Delphi short strings but it is not that much work to write out a regular string. You need to decide what type of format you want string data written in. I picked UTF8 for my application.

• For your other arrays you can write their data (including length) after the first array is written out. Sometime there will be a header section that includes the lengths for all the dynamic sections at the top, others will write the length write before the structure starts. The key part is to always write things in the same order and included somewhere it can be re-read how many there are.

• There is no error checking or verification in the file structure below. If anything is different between the read and write it will blow up - probably with a stream read error.

• Any change to the structures will cause old files to not be read properly. There are a number of ways to version a file to ensure you can still read old formats. Not included here.

• In your application you would pass a TFileStream to the read and write function. I like to write the actual read/write functions with just a TStream. Then the object does not care where the data is coming from. It could be file, or it could already be in memory.

• If you drop the following unit into a console application you should be able to add a call to Main and step through the example.

    
    unit CoreStruct;

    interface

    uses Classes, Types;

    type

      TCoreStructData = packed record
        LowerPointer : integer;
        HigherPointer : integer;
        Line : integer;
        Word : integer;
        Char : integer;
        IsPublic : boolean;
        IsPrivate : boolean;
        IsStatic : boolean;
        IsFinal : boolean;
        HasNested: boolean;
      end;

      TCoreStruc = class(TObject)
        private
          FCoreData: TCoreStructData;
          FNestedStruc : TCoreStruc;

          procedure SetNestedStruc(AValue: TCoreStruc);
        public
          CoreType : String;
          Name : String;


          constructor Create(); overload;
          procedure WriteToStream(Stream: TStream);
          procedure ReadFromStream(Stream: TStream);

          //constructor Create(Name, CoreType : ansistring; NestedStruc : TCoreStruc; IsPublic, IsPrivate, IsStatic, IsFinal : boolean); overload;
          //procedure UpdateValues(NestedStruc : TCoreStruc; IsPublic, IsPrivate, IsStatic, IsFinal : boolean);
          //procedure SetPosition(Line, Word, Char : integer);

          property LowerPointer: integer read FCoreData.LowerPointer write FCoreData.LowerPointer;
          property HigherPointer: integer read FCoreData.HigherPointer write FCoreData.HigherPointer;
          property Line: integer read FCoreData.Line write FCoreData.Line;
          property Word: integer read FCoreData.Word write FCoreData.Word;
          property Char: integer read FCoreData.Char write FCoreData.Char;

          property NestedStruc: TCoreStruc read FNestedStruc write SetNestedStruc;
      end;

      procedure Main();

    implementation


    function ReadUTF8StringFromStream(const Stream: TStream): String;
    var
      n: Integer;
      Buffer8: Utf8String;
    begin
      Result := '';

      Stream.ReadBuffer(n, SizeOf(n));

      if n = 0 then
        Exit;

      SetLength(Buffer8, n);
      Stream.ReadBuffer(Pointer(Buffer8)^, n);

      Result := String(Buffer8);

    end;


    procedure WriteUtf8StringToStream(const Data: String; Stream: TStream);
    var
      Buffer8: Utf8String;
      n: Integer;
    begin
      // When writing strings we need to make sure the length of the
      // string is written out to the stream.  That goes first so the
      // reader knows how long the buffer is.
      //
      // You could you different formats to write to the file depending on
      // needs.  I like using UTF8 when writing out to file, but it does
      // require an extra buffer copy when turning it back into a native
      // Delphi unicode string.
      Buffer8 := Utf8String(Data);
      n := Length(Buffer8);

      Stream.WriteBuffer(n, SizeOf(n));

      Stream.WriteBuffer(Pointer(Buffer8)^, n);

    end;


    procedure Main();
    var
      Structs: array of TCoreStruc;

      ArraySize: integer;
      DataStream: TMemoryStream;

      ArraySize_A: integer;
      Structs_A: array of TCoreStruc;
      i: integer;
    begin
      // Create and write some data
      SetLength(Structs, 3);
      Structs[0] := TCoreStruc.Create();
      Structs[0].HigherPointer := 1;
      Structs[0].Name := 'Testing';

      Structs[0].NestedStruc := TCoreStruc.Create();
      Structs[0].NestedStruc.HigherPointer := 100;

      Structs[1] := TCoreStruc.Create();
      Structs[1].HigherPointer := 2;

      Structs[2] := TCoreStruc.Create();
      Structs[2].HigherPointer := 3;


      DataStream := TMemoryStream.Create();

      // We need to start with the count we are writing out so
      // the reader knows how many times to loop.
      ArraySize := Length(Structs);
      DataStream.WriteBuffer(ArraySize, SizeOf(integer));

      for i := 0 to ArraySize - 1 do
      begin
        Structs[i].WriteToStream(DataStream);
      end;


      // Read the data into a new set of objects
      DataStream.Position := 0;

      DataStream.ReadBuffer(ArraySize_A, SizeOf(integer));
      SetLength(Structs_A, ArraySize_A);

      for i := 0 to ArraySize_A - 1 do
      begin
        Structs_A[i] := TCoreStruc.Create();
        Structs_A[i].ReadFromStream(DataStream);
      end;

    end;

    { TCoreStruc }

    constructor TCoreStruc.Create;
    begin
      Self.LowerPointer := 0;
      Self.HigherPointer := 0;
      Self.Line := 0;
      Self.Word := 0;
      Self.Char := 0;

      Self.NestedStruc := nil;

    end;

    procedure TCoreStruc.WriteToStream(Stream: TStream);
    begin
      Stream.WriteBuffer(FCoreData, SizeOf(TCoreStructData));
      WriteUtf8StringToStream(Name, Stream);
      WriteUtf8StringToStream(CoreType, Stream);

      if FCoreData.HasNested = true then
      begin
        FNestedStruc.WriteToStream(Stream)
      end;

    end;

    procedure TCoreStruc.ReadFromStream(Stream: TStream);
    begin

      Stream.ReadBuffer(FCoreData, SizeOf(TCoreStructData));
      Name := ReadUtf8StringFromStream(Stream);
      Name := ReadUtf8StringFromStream(Stream);

      if FCoreData.HasNested = true then
      begin
        FNestedStruc := TCoreStruc.Create();
        FNestedStruc.ReadFromStream(Stream);
      end;

    end;

    procedure TCoreStruc.SetNestedStruc(AValue: TCoreStruc);
    begin
      FNestedStruc := AValue;

      if AValue = nil then
        FCoreData.HasNested := false
      else
        FCoreData.HasNested := true;

    end;



    end.