VNC DES authentication algorithm

0
votes

I'm doing a client for VNC protocol (RFB). Everything is working from the part where I connect to server, to the part where I receive 'challenge' from server. The challenge is a 16-byte random, that should be encrypted with the password provided by user as key in DES algorithm. I tried to use 2 units for DES encryption with this project, one was a total failure, and another was this one... I should get a 16-byte encrypted string to send, and when I'm debugging I see my AnsiString have 16 length (I believe for AnsiStrings we have LENGTH = BYTES)... Anyway, my code is simple and I'll paste the most important parts involving this problem here.

Challenge: array[0..15] of AnsiChar;
PasswdDES: AnsiString;
Result: array[0..3] of AnsiChar;
  begin
      MySocket:= TClientSocket.Create(Nil);
      ....
      MySocket.Socket.ReceiveBuf(Challenge, SizeOf(Challenge));
      PasswdDES:= EncryStr(AnsiString(Challenge), AnsiString('123'));
      MySocket.Socket.SendText(PasswdDES);
      MySocket.Socket.ReceiveBuf(Result, SizeOf(Result));

I always receive on 'Result' the value #1 what means Authentication Failure... EncryStr is one function from the unit. I readed this link and tried to implement the solution in Delphi, but still not working.

Notes: I already tried using Challenge as array of bytes, and did something like:

PasswdDES:= EncryStr(LPCSTR(Challenge[0]), '123');

But no success.. As i'm using Delphi 2010, I'm using AnsiString... Thank's for your attention!

The java solution I tried to implement on delphi was this:

Function BitFlip(B: Byte): Byte; //stackoverflow
const
  N: array[0..15] of Byte = (0, 8, 4, 12, 2, 10, 6, 14, 1, 9, 5, 13, 3, 11, 7, 15);
begin
  Result := N[B div 16] or N[B mod 16] shl 4;
End;

Function Invert(input: PAnsiChar): AnsiString;
var
  I : int32;
Begin
  Result := '';
  for I := 0 to length(input)-1 do
    Result := Result + AnsiChar(BitFlip(Byte(Input[I])));
End;
2
algorithmdelphidelphi-2010vnc
Since you won't provide your translation of the VNC code you link to, nobody else than you can debug it. Here's how to do that. Run the original code for a series of inputs. Then run the translation for the same series of inputs. Then debug the differences. – David Heffernan
Sorry man, but I didn't got you. I'm using that unit, the same unit I linked here, no differences... – user1526124
I mean this: vidarholen.net/contents/junk/vnc.html – David Heffernan
I added it to my post... Anyway it's hard to understand, I almost can't understand the DES unit and this code (I got it on StackOverflow)... I don't know why VNC guys couldn't do something easier to implement, they changed DES algorithm and don't explain in any place how to get it working... – user1526124
Have you read this text http://www.vidarholen.net/contents/junk/vnc.html? You cannot use '123' as a password. You have to pass it as 8 bytes long block starting with password text having mirrored bits within each ANSI character (byte) and padded by 0 to 8 bytes (DES is a block cipher) – pf1957

2 Answers

1
votes

This is Free Pascal conversion of des.c file from UltraVNC project.

unit pasd3des;

{$mode objfpc}{$H+}

interface

const
  EN0 = 0; (* MODE == encrypt *)
  DE1 = 1; (* MODE == decrypt *)

procedure des(inblock, outblock: PChar);
procedure deskey(key: PChar; edf: ShortInt);

implementation

const
  Df_Key: array[0..23] of byte = (
    $01, $23, $45, $67, $89, $AB, $CD, $EF,
    $FE, $DC, $BA, $98, $76, $54, $32, $10,
    $89, $AB, $CD, $EF, $01, $23, $45, $67);

  bytebit: array[0..7] of word = (1, 2, 4, 8, 16, 32, 64, 128);

  bigbyte: array[0..23] of Longword = (
    $800000, $400000, $200000, $100000,
    $80000,  $40000,  $20000,  $10000,
    $8000,   $4000,   $2000,   $1000,
    $800,    $400,    $200,    $100,
    $80,     $40,     $20,     $10,
    $8,      $4,      $2,      $1);

  pc1: array[0..55] of byte = (
    56, 48, 40, 32, 24, 16,  8,  0, 57, 49, 41, 33, 25, 17,
     9,  1, 58, 50, 42, 34, 26, 18, 10,  2, 59, 51, 43, 35,
    62, 54, 46, 38, 30, 22, 14,  6, 61, 53, 45, 37, 29, 21,
    13,  5, 60, 52, 44, 36, 28, 20, 12,  4, 27, 19, 11,  3);

  totrot: array[0..15] of byte = (1, 2, 4, 6, 8, 10, 12, 14, 15, 17, 19, 21, 23, 25, 27, 28);

  pc2: array[0..47] of byte = (13, 16, 10, 23, 0, 4, 2, 27, 14, 5, 20, 9,
    22, 18, 11, 3, 25, 7, 15, 6, 26, 19, 12, 1,
    40, 51, 30, 36, 46, 54, 29, 39, 50, 44, 32, 47,
    43, 48, 38, 55, 33, 52, 45, 41, 49, 35, 28, 31);

  SP1: array[0..63] of Longword = (
    $01010400, $00000000, $00010000, $01010404,
    $01010004, $00010404, $00000004, $00010000,
    $00000400, $01010400, $01010404, $00000400,
    $01000404, $01010004, $01000000, $00000004,
    $00000404, $01000400, $01000400, $00010400,
    $00010400, $01010000, $01010000, $01000404,
    $00010004, $01000004, $01000004, $00010004,
    $00000000, $00000404, $00010404, $01000000,
    $00010000, $01010404, $00000004, $01010000,
    $01010400, $01000000, $01000000, $00000400,
    $01010004, $00010000, $00010400, $01000004,
    $00000400, $00000004, $01000404, $00010404,
    $01010404, $00010004, $01010000, $01000404,
    $01000004, $00000404, $00010404, $01010400,
    $00000404, $01000400, $01000400, $00000000,
    $00010004, $00010400, $00000000, $01010004);

  SP2: array[0..63] of Longword = (
    $80108020, $80008000, $00008000, $00108020,
    $00100000, $00000020, $80100020, $80008020,
    $80000020, $80108020, $80108000, $80000000,
    $80008000, $00100000, $00000020, $80100020,
    $00108000, $00100020, $80008020, $00000000,
    $80000000, $00008000, $00108020, $80100000,
    $00100020, $80000020, $00000000, $00108000,
    $00008020, $80108000, $80100000, $00008020,
    $00000000, $00108020, $80100020, $00100000,
    $80008020, $80100000, $80108000, $00008000,
    $80100000, $80008000, $00000020, $80108020,
    $00108020, $00000020, $00008000, $80000000,
    $00008020, $80108000, $00100000, $80000020,
    $00100020, $80008020, $80000020, $00100020,
    $00108000, $00000000, $80008000, $00008020,
    $80000000, $80100020, $80108020, $00108000);

  SP3: array[0..63] of Longword = (
    $00000208, $08020200, $00000000, $08020008,
    $08000200, $00000000, $00020208, $08000200,
    $00020008, $08000008, $08000008, $00020000,
    $08020208, $00020008, $08020000, $00000208,
    $08000000, $00000008, $08020200, $00000200,
    $00020200, $08020000, $08020008, $00020208,
    $08000208, $00020200, $00020000, $08000208,
    $00000008, $08020208, $00000200, $08000000,
    $08020200, $08000000, $00020008, $00000208,
    $00020000, $08020200, $08000200, $00000000,
    $00000200, $00020008, $08020208, $08000200,
    $08000008, $00000200, $00000000, $08020008,
    $08000208, $00020000, $08000000, $08020208,
    $00000008, $00020208, $00020200, $08000008,
    $08020000, $08000208, $00000208, $08020000,
    $00020208, $00000008, $08020008, $00020200);

  SP4: array[0..63] of Longword = (
    $00802001, $00002081, $00002081, $00000080,
    $00802080, $00800081, $00800001, $00002001,
    $00000000, $00802000, $00802000, $00802081,
    $00000081, $00000000, $00800080, $00800001,
    $00000001, $00002000, $00800000, $00802001,
    $00000080, $00800000, $00002001, $00002080,
    $00800081, $00000001, $00002080, $00800080,
    $00002000, $00802080, $00802081, $00000081,
    $00800080, $00800001, $00802000, $00802081,
    $00000081, $00000000, $00000000, $00802000,
    $00002080, $00800080, $00800081, $00000001,
    $00802001, $00002081, $00002081, $00000080,
    $00802081, $00000081, $00000001, $00002000,
    $00800001, $00002001, $00802080, $00800081,
    $00002001, $00002080, $00800000, $00802001,
    $00000080, $00800000, $00002000, $00802080);

  SP5: array[0..63] of Longword = (
    $00000100, $02080100, $02080000, $42000100,
    $00080000, $00000100, $40000000, $02080000,
    $40080100, $00080000, $02000100, $40080100,
    $42000100, $42080000, $00080100, $40000000,
    $02000000, $40080000, $40080000, $00000000,
    $40000100, $42080100, $42080100, $02000100,
    $42080000, $40000100, $00000000, $42000000,
    $02080100, $02000000, $42000000, $00080100,
    $00080000, $42000100, $00000100, $02000000,
    $40000000, $02080000, $42000100, $40080100,
    $02000100, $40000000, $42080000, $02080100,
    $40080100, $00000100, $02000000, $42080000,
    $42080100, $00080100, $42000000, $42080100,
    $02080000, $00000000, $40080000, $42000000,
    $00080100, $02000100, $40000100, $00080000,
    $00000000, $40080000, $02080100, $40000100);

  SP6: array[0..63] of Longword = (
    $20000010, $20400000, $00004000, $20404010,
    $20400000, $00000010, $20404010, $00400000,
    $20004000, $00404010, $00400000, $20000010,
    $00400010, $20004000, $20000000, $00004010,
    $00000000, $00400010, $20004010, $00004000,
    $00404000, $20004010, $00000010, $20400010,
    $20400010, $00000000, $00404010, $20404000,
    $00004010, $00404000, $20404000, $20000000,
    $20004000, $00000010, $20400010, $00404000,
    $20404010, $00400000, $00004010, $20000010,
    $00400000, $20004000, $20000000, $00004010,
    $20000010, $20404010, $00404000, $20400000,
    $00404010, $20404000, $00000000, $20400010,
    $00000010, $00004000, $20400000, $00404010,
    $00004000, $00400010, $20004010, $00000000,
    $20404000, $20000000, $00400010, $20004010);

  SP7: array[0..63] of Longword = (
    $00200000, $04200002, $04000802, $00000000,
    $00000800, $04000802, $00200802, $04200800,
    $04200802, $00200000, $00000000, $04000002,
    $00000002, $04000000, $04200002, $00000802,
    $04000800, $00200802, $00200002, $04000800,
    $04000002, $04200000, $04200800, $00200002,
    $04200000, $00000800, $00000802, $04200802,
    $00200800, $00000002, $04000000, $00200800,
    $04000000, $00200800, $00200000, $04000802,
    $04000802, $04200002, $04200002, $00000002,
    $00200002, $04000000, $04000800, $00200000,
    $04200800, $00000802, $00200802, $04200800,
    $00000802, $04000002, $04200802, $04200000,
    $00200800, $00000000, $00000002, $04200802,
    $00000000, $00200802, $04200000, $00000800,
    $04000002, $04000800, $00000800, $00200002);

  SP8: array[0..63] of Longword = (
    $10001040, $00001000, $00040000, $10041040,
    $10000000, $10001040, $00000040, $10000000,
    $00040040, $10040000, $10041040, $00041000,
    $10041000, $00041040, $00001000, $00000040,
    $10040000, $10000040, $10001000, $00001040,
    $00041000, $00040040, $10040040, $10041000,
    $00001040, $00000000, $00000000, $10040040,
    $10000040, $10001000, $00041040, $00040000,
    $00041040, $00040000, $10041000, $00001000,
    $00000040, $10040040, $00001000, $00041040,
    $10001000, $00000040, $10000040, $10040000,
    $10040040, $10000000, $00040000, $10001040,
    $00000000, $10041040, $00040040, $10000040,
    $10040000, $10001000, $10001040, $00000000,
    $10041040, $00041000, $00041000, $00001040,
    $00001040, $00040040, $10000000, $10041000);

var
  KnL: array[0..31] of Longword;

procedure usekey(from: PLongWord);
var
  ato, endp: PLongWord;
begin
  ato := @KnL[0];
  endp := @KnL[31];
  while ( ato <= endp ) do
  begin
    ato[0] := from[0];
    ato := @ato[1];
    from := @from[1];
  end;
end;

procedure cookey(raw1: PLongWord);
var
  cook, raw0: PLongWord;
  dough: array[0..31] of LongWord;
  i: Integer;
begin
  cook := @dough[0];
  for i := 0 to 15 do
  begin
    raw0 := raw1;
    raw1 := @raw1[1];
    cook[0] := (raw0[0] and $00fc0000) shl 6;
    cook[0] := cook[0] or (raw0[0] and $00000fc0) shl 10;
    cook[0] := cook[0] or (raw1[0] and $00fc0000) shr 10;
    cook[0] := cook[0] or (raw1[0] and $00000fc0) shr 6;
    cook := @cook[1];
    cook[0] := (raw0[0] and $0003f000) shl 12;
    cook[0] := cook[0] or (raw0[0] and $0000003f) shl 16;
    cook[0] := cook[0] or (raw1[0] and $0003f000) shr 4;
    cook[0] := cook[0] or (raw1[0] and $0000003f);
    cook := @cook[1];
    raw1 := @raw1[1];
  end;
  usekey(@dough);
end;

procedure deskey(key: PChar; edf: ShortInt);
var
  i, j, l, m, n: Integer;
  pc1m, pcr: array[0..55] of Byte;
  kn: array[0..31] of LongWord;
begin
  for j := 0 to 55 do
  begin
    l := pc1[j];
    m := l and 07;
    if (Byte(key[l shr 3]) and bytebit[m]) > 0 then
      pc1m[j] := 1
    else
      pc1m[j] := 0;
  end;
  for i := 0 to 15 do
  begin
    if (edf = DE1) then
      m := (15 - i) shl 1
    else
      m := i shl 1;
    n := m + 1;
    kn[m] := 0;
    kn[n] := 0;
    for j := 0 to 27 do
    begin
      l := j + totrot[i];
      if (l < 28) then
        pcr[j] := pc1m[l]
      else
        pcr[j] := pc1m[l - 28];
    end;
    for j := 28 to 55 do
    begin
      l := j + totrot[i];
      if (l < 56) then
        pcr[j] := pc1m[l]
      else
        pcr[j] := pc1m[l - 28];
    end;
    for j := 0 to 23 do
    begin
      if (pcr[pc2[j]] <> 0) then
        kn[m] := kn[m] or bigbyte[j];
      if (pcr[pc2[j+24]] <> 0) then
        kn[n] := kn[n] or bigbyte[j];
    end;
  end;
  cookey(@kn);
end;

procedure cpkey(into: PLongWord);
var
  from, endp: PLongWord;
begin
  from := KnL;
  endp := @KnL[31];
  while (from <= endp) do
  begin
    into[0] := from[0];
    into := @into[1];
    from := @from[1];
  end;
end;

procedure scrunch(outof: PByte; into: PLongWord);
begin
  into[0] := (outof[0] and $ff) shl 24;
  outof := @outof[1];
  into[0] := into[0] or (outof[0] and $ff) shl 16;
  outof := @outof[1];
  into[0] := into[0] or (outof[0] and $ff) shl 8;
  outof := @outof[1];
  into[0] := into[0] or (outof[0] and $ff);
  outof := @outof[1];
  into := @into[1];
  into[0] := (outof[0] and $ff) shl 24;
  outof := @outof[1];
  into[0] := into[0] or (outof[0] and $ff) shl 16;
  outof := @outof[1];
  into[0] := into[0] or (outof[0] and $ff) shl 8;
  outof := @outof[1];
  into[0] := into[0] or (outof[0] and $ff);
end;

procedure unscrun(outof: PLongWord; into: PByte);
begin
  into[0] := Byte((outof[0] shr 24) and $ff);
  into := @into[1];
  into[0] := Byte((outof[0] shr 16) and $ff);
  into := @into[1];
  into[0] := Byte((outof[0] shr 8) and $ff);
  into := @into[1];
  into[0] := Byte(outof[0] and $ff);
  outof := @outof[1];
  into := @into[1];
  into[0] := Byte((outof[0] shr 24) and $ff);
  into := @into[1];
  into[0] := Byte((outof[0] shr 16) and $ff);
  into := @into[1];
  into[0] := Byte((outof[0] shr 8) and $ff);
  into := @into[1];
  into[0] := Byte(outof[0] and $ff);
end;

procedure desfunc(block, keys: PLongWord);
var
  fval, work, right, leftt: LongWord;
  roundd: Integer;
begin
  leftt := block[0];
  right := block[1];
  work := ((leftt shr 4) xor right) and $0f0f0f0f;
  right := right xor work;
  leftt := leftt xor (work shl 4);
  work := ((leftt shr 16) xor right) and $0000ffff;
  right := right xor work;
  leftt := leftt xor (work shl 16);
  work := ((right shr 2) xor leftt) and $33333333;
  leftt := leftt xor work;
  right := right xor (work shl 2);
  work := ((right shr 8) xor leftt) and $00ff00ff;
  leftt := leftt xor work;
  right := right xor (work shl 8);
  right := ((right shl 1) or ((right shr 31) and 1)) and $ffffffff;
  work := (leftt xor right) and $aaaaaaaa;
  leftt := leftt xor work;
  right := right xor work;
  leftt := ((leftt shl 1) or ((leftt shr 31) and 1)) and $ffffffff;
  for roundd := 0 to 7 do
  begin
    work  := (right shl 28) or (right shr 4);
    work := work xor keys[0];
    keys := @keys[1];
    fval := SP7[work and $3f];
    fval := fval or SP5[(work shr  8) and $3f];
    fval := fval or SP3[(work shr 16) and $3f];
    fval := fval or SP1[(work shr 24) and $3f];
    work := right xor keys[0];
    keys := @keys[1];
    fval := fval or SP8[work and $3f];
    fval := fval or SP6[(work shr  8) and $3f];
    fval := fval or SP4[(work shr 16) and $3f];
    fval := fval or SP2[(work shr 24) and $3f];
    leftt := leftt xor fval;
    work := (leftt shl 28) or (leftt shr 4);
    work := work xor keys[0];
    keys := @keys[1];
    fval := SP7[work and $3f];
    fval := fval or SP5[(work shr  8) and $3f];
    fval := fval or SP3[(work shr 16) and $3f];
    fval := fval or SP1[(work shr 24) and $3f];
    work := leftt xor keys[0];
    keys := @keys[1];
    fval := fval or SP8[work and $3f];
    fval := fval or SP6[(work shr  8) and $3f];
    fval := fval or SP4[(work shr 16) and $3f];
    fval := fval or SP2[(work shr 24) and $3f];
    right := right xor fval;
  end;
  right := (right shl 31) or (right shr 1);
  work := (leftt xor right) and $aaaaaaaa;
  leftt := leftt xor work;
  right := right xor work;
  leftt := (leftt shl 31) or (leftt shr 1);
  work := ((leftt shr 8) xor right) and $00ff00ff;
  right := right xor work;
  leftt := leftt xor (work shl 8);
  work := ((leftt shr 2) xor right) and $33333333;
  right := right xor work;
  leftt := leftt xor (work shl 2);
  work := ((right shr 16) xor leftt) and $0000ffff;
  leftt := leftt xor work;
  right := right xor (work shl 16);
  work := ((right shr 4) xor leftt) and $0f0f0f0f;
  leftt := leftt xor work;
  right := right xor (work shl 4);
  block[0] := right;
  block[1] := leftt;
end;

procedure des(inblock, outblock: PChar);
var
  work: array[0..1] of LongWord;
begin
  scrunch(PByte(inblock), @work);
  desfunc(@work, @KnL);
  unscrun(@work, PByte(outblock));
end;

end.

and function to crypt password

function CryptChal(AChal, APasswd: String): String;
const
  CHALLENGESIZE = 16;
var
  key, bytes, response: pChar;
  i: integer;
begin
  key := StrAlloc((CHALLENGESIZE div 2) + 1);
  response := StrAlloc(CHALLENGESIZE + 1);
  try
    for i := 0 to 7 do
    begin
      if (i < length(Apasswd)) then
        key[i] := Apasswd[i + 1]
      else
        key[i] := #0;
    end;
    if length(Apasswd)>8 then
      Key[8] := #0;

    deskey(key, 0);

    bytes := pChar(AChal);
    i := 0;
    while (i < CHALLENGESIZE) do
    begin
      des(bytes + i, response + i);
      inc(i, 8);
    end;
    Result := response;
    SetLength(Result, CHALLENGESIZE);
  finally
    StrDispose(response);
    StrDispose(key);
  end;
end;
1
votes

VNC use The Remote Framebuffer Protocol

Some important things were not described in documentation.

But we need to begin from doc: https://www.rfc-editor.org/rfc/rfc6143.html

There are some important items:

7.1.2. Security Handshake

Once the protocol version has been decided, the server and client must agree on the type of security to be used on the connection.

The security types defined in this document are:

                  +--------+--------------------+
                  | Number | Name               |
                  +--------+--------------------+
                  | 0      | Invalid            |
                  | 1      | None               |
                  | 2      | VNC Authentication |
                  +--------+--------------------+

Other security types exist but are not publicly documented.

7.2.2. VNC Authentication

VNC authentication is to be used. The server sends a random 16-byte challenge:

           +--------------+--------------+-------------+
           | No. of bytes | Type [Value] | Description |
           +--------------+--------------+-------------+
           | 16           | U8           | challenge   |
           +--------------+--------------+-------------+

The client encrypts the challenge with DES, using a password supplied by the user as the key. To form the key, the password is truncated to eight characters, or padded with null bytes on the right. The client then sends the resulting 16-byte response:

           +--------------+--------------+-------------+
           | No. of bytes | Type [Value] | Description |
           +--------------+--------------+-------------+
           | 16           | U8           | response    |
           +--------------+--------------+-------------+

IMPORTANT UNDOCUMENTED INFORMATION:

  1. DES has some block cipher modes, you should use ECB. DES working with 8-bytes sequences but we have 16-bytes challenge. There are some methods of work with block sequence, check it.

  2. When you make 8-bytes password from source password, you should reverse order of bits in bytes. For example, "00100111" byte should be reversed into "11100100". You can't pass VNC authentication without this transposition.

  3. Look "Remote Framebuffer Security Types" in https://www.iana.org/assignments/rfb/rfb.xhtml and check do you use: 2 - "VNC Authentication"?

When you write a program, take a real data from tcp-conversation and try to reproduce the same 16-bytes responce for overheard challenge.