What does the SSL error 140612029986456:error:140780E5:SSL mean?

Question

I am trying to debug the SSL connection of a client I do not have control over.

A possibility I discovered is to use openssl as a server to check whether the SSL connection is correctly initiated, and if not - why.

The basic connection ends with

root@elk:~# openssl s_server -key sonoff.key -cert sonoff.crt -accept 5555 -www
Using default temp DH parameters
ACCEPT
bad gethostbyaddr
140612029986456:error:140780E5:SSL routines:ssl23_read:ssl handshake failure:s23_lib.c:137:

What does this error actually mean?

The searches I did point to various bugs, but they all gravitate around "incorrect certificates". The certificates I use are correct, although self-signed. What in the message (or the -debug version below) could suggest this? Specifically: could this be an indication of the device checking the certificate and aborting on a self-signed?

root@elk:~# openssl s_server -key sonoff.key -cert sonoff.crt -accept 5555 -www -debug
Using default temp DH parameters
ACCEPT

bad gethostbyaddr
read from 0xb2ca20 [0xb32110] (11 bytes => 11 (0xB))
0000 - 16 03 01 00 4f 01 00 00-4b 03 03                  ....O...K..
read from 0xb2ca20 [0xb3211e] (73 bytes => 73 (0x49))
0000 - 1b da e3 8f 9a d2 af 66-81 b5 62 95 7e 44 fc 3b   .......f..b.~D.;
0010 - 75 48 90 da 51 03 f6 ad-33 81 ec 15 16 42 2b f2   uH..Q...3....B+.
0020 - 00 00 12 00 3d 00 35 00-3c 00 2f 00 b7 00 95 00   ....=.5.<./.....
0030 - b6 00 94 00 ff 01 00 00-10 00 0d 00 0c 00 0a 06   ................
0040 - 01 05 01 04 01 03 01 02-01                        .........
write to 0xb2ca20 [0xb3bc00] (86 bytes => 86 (0x56))
0000 - 16 03 03 00 51 02 00 00-4d 03 03 2b ca 7c 43 47   ....Q...M..+.|CG
0010 - 06 ae 7e 1e b3 9c 71 7a-ca 5d 6d 98 54 4e 1c 5a   ..~...qz.]m.TN.Z
0020 - b7 f9 38 2c 0d 3e de e0-62 a6 8a 20 2a 42 cf 9e   ..8,.>..b.. *B..
0030 - ba 1e 8c a3 03 24 e1 34-9e 15 2e d8 90 77 73 74   .....$.4.....wst
0040 - 78 33 e6 45 83 34 ca 11-ae 97 4e 18 00 3d 00 00   x3.E.4....N..=..
0050 - 05 ff 01 00 01                                    .....
0056 - <SPACES/NULS>
write to 0xb2ca20 [0xb3bc00] (1298 bytes => 1298 (0x512))
0000 - 16 03 03 05 0d 0b 00 05-09 00 05 06 00 05 03 30   ...............0
0010 - 82 04 ff 30 82 02 e7 a0-03 02 01 02 02 09 00 df   ...0............
0020 - 41 71 c6 48 ff eb 19 30-0d 06 09 2a 86 48 86 f7   Aq.H...0...*.H..
0030 - 0d 01 01 0b 05 00 30 16-31 14 30 12 06 03 55 04   ......0.1.0...U.
0040 - 03 0c 0b 65 78 61 6d 70-6c 65 2e 63 6f 6d 30 1e   ...example.com0.
0050 - 17 0d 31 38 30 32 30 34-31 38 31 33 35 36 5a 17   ..180204181356Z.
0060 - 0d 32 38 30 32 30 32 31-38 31 33 35 36 5a 30 16   .280202181356Z0.
0070 - 31 14 30 12 06 03 55 04-03 0c 0b 65 78 61 6d 70   1.0...U....examp
0080 - 6c 65 2e 63 6f 6d 30 82-02 22 30 0d 06 09 2a 86   le.com0.."0...*.
0090 - 48 86 f7 0d 01 01 01 05-00 03 82 02 0f 00 30 82   H.............0.
00a0 - 02 0a 02 82 02 01 00 e0-22 b0 41 79 e5 60 92 2c   ........".Ay.`.,
00b0 - c7 1c 97 84 f8 16 46 39-27 67 26 ad d7 c8 dc a4   ......F9'g&.....
00c0 - 18 2e 5a 1a aa f7 92 b0-71 8f f8 a5 e0 d9 52 d3   ..Z.....q.....R.
00d0 - 6a c9 e1 ae 7c ce f1 1c-92 99 dd 77 b7 f7 db b3   j...|......w....
00e0 - 2b f4 96 23 d4 d8 08 e8-e2 f2 15 7b 41 16 30 3d   +..#.......{A.0=
00f0 - 21 cd 5f b0 3d 5a 0c ec-60 9b d8 78 99 32 c4 9f   !._.=Z..`..x.2..
0100 - 3c 1b 50 d9 d4 cf f0 4b-58 ca c6 6f 56 f5 57 04   <.P....KX..oV.W.
0110 - c9 d5 60 56 d2 25 66 ee-fe f0 da 79 d6 f2 a5 67   ..`V.%f....y...g
0120 - 08 7a 19 e6 82 90 5c e0-a1 63 cf 4b a8 43 c6 d3   .z....\..c.K.C..
0130 - 08 a0 22 4a df be 42 22-8b c9 6f 81 d1 b0 d8 e0   .."J..B"..o.....
0140 - 24 79 b3 3a ad 10 af bd-7a 43 56 a6 87 ad d2 d4   $y.:....zCV.....
0150 - 96 cf 12 f9 9a 42 7a c7-63 07 f7 cc a9 a5 50 1e   .....Bz.c.....P.
0160 - bf 4d 2d ae e3 74 b7 a7-66 ca ab 44 66 db f5 e2   .M-..t..f..Df...
0170 - 28 dd a8 6f 76 0a 5a d4-8d 42 c6 a0 16 c2 bd d9   (..ov.Z..B......
0180 - 25 38 74 dc bc d3 95 03-b9 13 39 c0 4a cd d9 c9   %8t.......9.J...
0190 - cb 34 bf 9e 48 1e e4 56-8f 47 9c 33 ed eb d6 db   .4..H..V.G.3....
01a0 - 5e b7 46 85 d4 49 48 f3-24 ec 01 c3 98 74 50 f9   ^.F..IH.$....tP.
01b0 - 05 71 01 b5 96 f4 b5 bc-a3 57 78 3f c7 4f 10 1e   .q.......Wx?.O..
01c0 - da 96 59 c0 6d c1 13 64-52 7b f8 d7 88 59 29 ba   ..Y.m..dR{...Y).
01d0 - b7 5a ec a2 a7 1e 86 bb-ea 8b f8 03 e3 73 b3 6e   .Z...........s.n
01e0 - 70 27 29 b6 92 31 4f 52-73 f0 62 91 3e 90 53 40   p')..1ORs.b.>.S@
01f0 - d0 00 e2 1d 80 5e 82 a9-5e fc 56 2f 76 d9 40 39   .....^..^.V/v.@9
0200 - 85 e3 4e 55 d2 a7 47 06-27 92 d9 ec dd 7e 7d c2   ..NU..G.'....~}.
0210 - 07 6d 73 9b 58 75 da ba-cd 65 e3 d9 3b da 78 4e   .ms.Xu...e..;.xN
0220 - 96 81 c1 d1 2f f2 c0 65-e0 a2 12 33 1c eb 0c 2a   ..../..e...3...*
0230 - 10 c6 ea 87 4c 7a a5 d7-c3 4c a8 45 06 8d ca a5   ....Lz...L.E....
0240 - 90 7d e0 57 be e5 17 f0-13 41 28 6e f6 f3 cb b3   .}.W.....A(n....
0250 - d4 9b fa b1 f9 30 a1 b4-2e a4 de 73 39 35 82 32   .....0.....s95.2
0260 - de 55 f9 70 ee ca 8a d2-62 64 fc fb d7 56 17 c7   .U.p....bd...V..
0270 - 9a 3c 4f 3d b1 4a 1d 37-fb ad 40 ef c4 14 cf 08   .<O=.J.7..@.....
0280 - 57 8e 68 5f 2b d4 3b cc-bb 23 9f 83 64 e3 22 39   W.h_+.;..#..d."9
0290 - 53 30 5c 4d 0f d8 4d 2a-5a 23 e6 ee 06 b6 38 5d   S0\M..M*Z#....8]
02a0 - 83 ef 91 f8 25 a6 e3 02-03 01 00 01 a3 50 30 4e   ....%........P0N
02b0 - 30 1d 06 03 55 1d 0e 04-16 04 14 bf 22 b8 2a ec   0...U.......".*.
02c0 - f6 61 a9 05 50 d8 6a f5-35 bd 32 3e 7c 9b d7 30   .a..P.j.5.2>|..0
02d0 - 1f 06 03 55 1d 23 04 18-30 16 80 14 bf 22 b8 2a   ...U.#..0....".*
02e0 - ec f6 61 a9 05 50 d8 6a-f5 35 bd 32 3e 7c 9b d7   ..a..P.j.5.2>|..
02f0 - 30 0c 06 03 55 1d 13 04-05 30 03 01 01 ff 30 0d   0...U....0....0.
0300 - 06 09 2a 86 48 86 f7 0d-01 01 0b 05 00 03 82 02   ..*.H...........
0310 - 01 00 35 e5 1e a9 74 4f-78 59 c0 fd da a4 71 7a   ..5...tOxY....qz
0320 - 4e 9d b7 13 42 5b d5 f3-95 1b 48 a6 91 c8 84 be   N...B[....H.....
0330 - fc 67 ed 92 4e 2e f2 e5-6e 14 38 a5 af 51 7f 54   .g..N...n.8..Q.T
0340 - 11 5f ed 15 bd a6 d9 ef-9c ef ee db ed fb 39 45   ._............9E
0350 - e2 5e a9 53 66 73 40 98-27 38 4d a3 1d 5f eb d3   .^.Sfs@.'8M.._..
0360 - f8 33 01 3d ce 08 61 60-e1 ee 91 28 98 0a a2 17   .3.=..a`...(....
0370 - 28 17 39 c1 af 1e d2 f7-42 c6 5f ff a2 fa 77 2a   (.9.....B._...w*
0380 - f0 4b ed 4f ea d4 55 b4-4d 02 63 4f 9f 12 ab 86   .K.O..U.M.cO....
0390 - 26 a6 24 26 7e 26 e2 55-89 6b 7d 52 3c 48 ad 4c   &.$&~&.U.k}R<H.L
03a0 - 0c 76 64 7d a7 8f ff 9e-23 28 4c d5 79 85 ea 0c   .vd}....#(L.y...
03b0 - ee 54 31 24 e2 66 53 4c-ae a6 a6 43 c5 e1 16 c7   .T1$.fSL...C....
03c0 - 48 b3 af 49 c9 f6 30 e6-4a da 2f 9b 1a 9d 5f 15   H..I..0.J./..._.
03d0 - ff ac 9e c7 d3 e4 85 55-1d d6 42 af 58 1e 57 d1   .......U..B.X.W.
03e0 - 84 4f 87 a7 43 10 e8 13-42 fd 49 fa c1 a2 ba 6a   .O..C...B.I....j
03f0 - 30 a4 1e fb db be 99 16-a1 dc 1e 45 45 54 c3 89   0..........EET..
0400 - b8 b6 b8 74 90 2f 8b 8c-68 49 07 a2 7c 0d a6 d6   ...t./..hI..|...
0410 - 3c 80 e2 94 66 b0 83 1f-e3 29 02 4d 76 44 1e 65   <...f....).MvD.e
0420 - 57 7c 9e f2 01 0a 77 cd-c5 85 43 53 9e e7 56 21   W|....w...CS..V!
0430 - 4b 55 05 78 0e 6b 7a 14-44 69 d8 cb 06 de 65 9a   KU.x.kz.Di....e.
0440 - 0a 8b 15 11 4e 21 c6 1d-be d4 6c 45 96 88 f2 04   ....N!....lE....
0450 - d5 ff 03 fd d3 24 7d 25-22 8b 22 37 99 4b 22 fa   .....$}%"."7.K".
0460 - 8f 99 ec 71 88 58 8e 03-9f 76 19 a2 05 a6 3b bc   ...q.X...v....;.
0470 - 87 d8 90 db c1 cc 98 f9-1f c5 00 96 e4 90 3d ad   ..............=.
0480 - d0 fa a1 d5 b6 71 74 7d-0c a3 f2 79 5a e6 31 fb   .....qt}...yZ.1.
0490 - de 65 41 46 39 78 bd d6-c1 b0 1a 44 16 a1 ed 27   .eAF9x.....D...'
04a0 - c6 e7 d6 ab f0 2f 0f 78-a6 78 01 2c ed 18 bb 47   ...../.x.x.,...G
04b0 - 34 54 ee 73 1b 99 83 ca-bf d7 07 65 40 03 f2 75   [email protected]
04c0 - 98 d4 f3 22 6c 2f 77 1e-ec e7 1e bf 31 28 90 c4   ..."l/w.....1(..
04d0 - 0c 7a 37 cc 85 37 59 42-94 21 90 05 9e e0 60 51   .z7..7YB.!....`Q
04e0 - 0a 83 b8 29 45 37 06 5a-5e a3 b3 e0 08 cf 48 9b   ...)E7.Z^.....H.
04f0 - 51 17 7b 4c fb 6d 06 e9-4f dd 45 42 42 83 c2 f6   Q.{L.m..O.EBB...
0500 - b3 70 29 86 08 c4 28 f7-b4 d6 1f a1 63 94 f7 75   .p)...(.....c..u
0510 - 8f ba                                             ..
write to 0xb2ca20 [0xb3bc00] (9 bytes => 9 (0x9))
0000 - 16 03 03 00 04 0e                                 ......
0009 - <SPACES/NULS>
read from 0xb2ca20 [0xb32113] (5 bytes => 0 (0x0))
140570414024344:error:140780E5:SSL routines:ssl23_read:ssl handshake failure:s23_lib.c:137:

EDIT: a K12 dump of the session:

+---------+---------------+----------+
18:19:07,529,418   ETHER
|0   |8a|4b|47|20|ce|e4|1e|33|c8|aa|1b|40|08|00|45|00|00|2c|05|fb|00|00|7f|06|16|28|0a|64|0a|71|0a|c8|00|0d|58|a7|15|b3|00|36|bf|c5|00|00|00|00|60|02|16|d0|33|57|00|00|02|04|05|b4|

+---------+---------------+----------+
18:19:07,529,450   ETHER
|0   |1e|33|c8|aa|1b|40|8a|4b|47|20|ce|e4|08|00|45|00|00|2c|00|00|40|00|40|06|1b|23|0a|c8|00|0d|0a|64|0a|71|15|b3|58|a7|62|61|d4|6d|00|36|bf|c6|60|12|72|10|1f|c8|00|00|02|04|05|b4|

+---------+---------------+----------+
18:19:07,668,343   ETHER
|0   |8a|4b|47|20|ce|e4|1e|33|c8|aa|1b|40|08|00|45|00|00|7c|05|fc|00|00|7f|06|15|d7|0a|64|0a|71|0a|c8|00|0d|58|a7|15|b3|00|36|bf|c6|62|61|d4|6e|50|18|16|d0|67|d0|00|00|16|03|01|00|4f|01|00|00|4b|03|03|6e|30|37|19|a3|c0|a4|d5|d1|3c|8b|ed|76|72|e2|9f|d5|5e|57|3e|9e|98|b7|33|a9|12|54|de|40|32|df|34|00|00|12|00|3d|00|35|00|3c|00|2f|00|b7|00|95|00|b6|00|94|00|ff|01|00|00|10|00|0d|00|0c|00|0a|06|01|05|01|04|01|03|01|02|01|

+---------+---------------+----------+
18:19:07,668,359   ETHER
|0   |1e|33|c8|aa|1b|40|8a|4b|47|20|ce|e4|08|00|45|00|00|28|39|19|40|00|40|06|e2|0d|0a|c8|00|0d|0a|64|0a|71|15|b3|58|a7|62|61|d4|6e|00|36|c0|1a|50|10|72|10|1f|c4|00|00|

+---------+---------------+----------+
18:19:07,668,617   ETHER
|0   |1e|33|c8|aa|1b|40|8a|4b|47|20|ce|e4|08|00|45|00|05|99|39|1a|40|00|40|06|dc|9b|0a|c8|00|0d|0a|64|0a|71|15|b3|58|a7|62|61|d4|6e|00|36|c0|1a|50|18|72|10|25|35|00|00|16|03|01|00|51|02|00|00|4d|03|01|e9|4e|66|d1|56|43|c2|f8|df|95|62|d2|a1|8f|99|85|39|d3|22|4e|92|64|85|75|47|8f|29|f5|15|11|8e|10|20|b7|07|bc|84|e7|15|1e|6b|53|ed|6c|d5|1c|f6|15|10|ab|03|94|db|15|19|47|b7|43|04|30|59|fb|f2|52|e5|00|35|00|00|05|ff|01|00|01|00|16|03|01|05|0d|0b|00|05|09|00|05|06|00|05|03|30|82|04|ff|30|82|02|e7|a0|03|02|01|02|02|09|00|df|41|71|c6|48|ff|eb|19|30|0d|06|09|2a|86|48|86|f7|0d|01|01|0b|05|00|30|16|31|14|30|12|06|03|55|04|03|0c|0b|65|78|61|6d|70|6c|65|2e|63|6f|6d|30|1e|17|0d|31|38|30|32|30|34|31|38|31|33|35|36|5a|17|0d|32|38|30|32|30|32|31|38|31|33|35|36|5a|30|16|31|14|30|12|06|03|55|04|03|0c|0b|65|78|61|6d|70|6c|65|2e|63|6f|6d|30|82|02|22|30|0d|06|09|2a|86|48|86|f7|0d|01|01|01|05|00|03|82|02|0f|00|30|82|02|0a|02|82|02|01|00|e0|22|b0|41|79|e5|60|92|2c|c7|1c|97|84|f8|16|46|39|27|67|26|ad|d7|c8|dc|a4|18|2e|5a|1a|aa|f7|92|b0|71|8f|f8|a5|e0|d9|52|d3|6a|c9|e1|ae|7c|ce|f1|1c|92|99|dd|77|b7|f7|db|b3|2b|f4|96|23|d4|d8|08|e8|e2|f2|15|7b|41|16|30|3d|21|cd|5f|b0|3d|5a|0c|ec|60|9b|d8|78|99|32|c4|9f|3c|1b|50|d9|d4|cf|f0|4b|58|ca|c6|6f|56|f5|57|04|c9|d5|60|56|d2|25|66|ee|fe|f0|da|79|d6|f2|a5|67|08|7a|19|e6|82|90|5c|e0|a1|63|cf|4b|a8|43|c6|d3|08|a0|22|4a|df|be|42|22|8b|c9|6f|81|d1|b0|d8|e0|24|79|b3|3a|ad|10|af|bd|7a|43|56|a6|87|ad|d2|d4|96|cf|12|f9|9a|42|7a|c7|63|07|f7|cc|a9|a5|50|1e|bf|4d|2d|ae|e3|74|b7|a7|66|ca|ab|44|66|db|f5|e2|28|dd|a8|6f|76|0a|5a|d4|8d|42|c6|a0|16|c2|bd|d9|25|38|74|dc|bc|d3|95|03|b9|13|39|c0|4a|cd|d9|c9|cb|34|bf|9e|48|1e|e4|56|8f|47|9c|33|ed|eb|d6|db|5e|b7|46|85|d4|49|48|f3|24|ec|01|c3|98|74|50|f9|05|71|01|b5|96|f4|b5|bc|a3|57|78|3f|c7|4f|10|1e|da|96|59|c0|6d|c1|13|64|52|7b|f8|d7|88|59|29|ba|b7|5a|ec|a2|a7|1e|86|bb|ea|8b|f8|03|e3|73|b3|6e|70|27|29|b6|92|31|4f|52|73|f0|62|91|3e|90|53|40|d0|00|e2|1d|80|5e|82|a9|5e|fc|56|2f|76|d9|40|39|85|e3|4e|55|d2|a7|47|06|27|92|d9|ec|dd|7e|7d|c2|07|6d|73|9b|58|75|da|ba|cd|65|e3|d9|3b|da|78|4e|96|81|c1|d1|2f|f2|c0|65|e0|a2|12|33|1c|eb|0c|2a|10|c6|ea|87|4c|7a|a5|d7|c3|4c|a8|45|06|8d|ca|a5|90|7d|e0|57|be|e5|17|f0|13|41|28|6e|f6|f3|cb|b3|d4|9b|fa|b1|f9|30|a1|b4|2e|a4|de|73|39|35|82|32|de|55|f9|70|ee|ca|8a|d2|62|64|fc|fb|d7|56|17|c7|9a|3c|4f|3d|b1|4a|1d|37|fb|ad|40|ef|c4|14|cf|08|57|8e|68|5f|2b|d4|3b|cc|bb|23|9f|83|64|e3|22|39|53|30|5c|4d|0f|d8|4d|2a|5a|23|e6|ee|06|b6|38|5d|83|ef|91|f8|25|a6|e3|02|03|01|00|01|a3|50|30|4e|30|1d|06|03|55|1d|0e|04|16|04|14|bf|22|b8|2a|ec|f6|61|a9|05|50|d8|6a|f5|35|bd|32|3e|7c|9b|d7|30|1f|06|03|55|1d|23|04|18|30|16|80|14|bf|22|b8|2a|ec|f6|61|a9|05|50|d8|6a|f5|35|bd|32|3e|7c|9b|d7|30|0c|06|03|55|1d|13|04|05|30|03|01|01|ff|30|0d|06|09|2a|86|48|86|f7|0d|01|01|0b|05|00|03|82|02|01|00|35|e5|1e|a9|74|4f|78|59|c0|fd|da|a4|71|7a|4e|9d|b7|13|42|5b|d5|f3|95|1b|48|a6|91|c8|84|be|fc|67|ed|92|4e|2e|f2|e5|6e|14|38|a5|af|51|7f|54|11|5f|ed|15|bd|a6|d9|ef|9c|ef|ee|db|ed|fb|39|45|e2|5e|a9|53|66|73|40|98|27|38|4d|a3|1d|5f|eb|d3|f8|33|01|3d|ce|08|61|60|e1|ee|91|28|98|0a|a2|17|28|17|39|c1|af|1e|d2|f7|42|c6|5f|ff|a2|fa|77|2a|f0|4b|ed|4f|ea|d4|55|b4|4d|02|63|4f|9f|12|ab|86|26|a6|24|26|7e|26|e2|55|89|6b|7d|52|3c|48|ad|4c|0c|76|64|7d|a7|8f|ff|9e|23|28|4c|d5|79|85|ea|0c|ee|54|31|24|e2|66|53|4c|ae|a6|a6|43|c5|e1|16|c7|48|b3|af|49|c9|f6|30|e6|4a|da|2f|9b|1a|9d|5f|15|ff|ac|9e|c7|d3|e4|85|55|1d|d6|42|af|58|1e|57|d1|84|4f|87|a7|43|10|e8|13|42|fd|49|fa|c1|a2|ba|6a|30|a4|1e|fb|db|be|99|16|a1|dc|1e|45|45|54|c3|89|b8|b6|b8|74|90|2f|8b|8c|68|49|07|a2|7c|0d|a6|d6|3c|80|e2|94|66|b0|83|1f|e3|29|02|4d|76|44|1e|65|57|7c|9e|f2|01|0a|77|cd|c5|85|43|53|9e|e7|56|21|4b|55|05|78|0e|6b|7a|14|44|69|d8|cb|06|de|65|9a|0a|8b|15|11|4e|21|c6|1d|be|d4|6c|45|96|88|f2|04|d5|ff|03|fd|d3|24|7d|25|22|8b|22|37|99|4b|22|fa|8f|99|ec|71|88|58|8e|03|9f|76|19|a2|05|a6|3b|bc|87|d8|90|db|c1|cc|98|f9|1f|c5|00|96|e4|90|3d|ad|d0|fa|a1|d5|b6|71|74|7d|0c|a3|f2|79|5a|e6|31|fb|de|65|41|46|39|78|bd|d6|c1|b0|1a|44|16|a1|ed|27|c6|e7|d6|ab|f0|2f|0f|78|a6|78|01|2c|ed|18|bb|47|34|54|ee|73|1b|99|83|ca|bf|d7|07|65|40|03|f2|75|98|d4|f3|22|6c|2f|77|1e|ec|e7|1e|bf|31|28|90|c4|0c|7a|37|cc|85|37|59|42|94|21|90|05|9e|e0|60|51|0a|83|b8|29|45|37|06|5a|5e|a3|b3|e0|08|cf|48|9b|51|17|7b|4c|fb|6d|06|e9|4f|dd|45|42|42|83|c2|f6|b3|70|29|86|08|c4|28|f7|b4|d6|1f|a1|63|94|f7|75|8f|ba|16|03|01|00|04|0e|00|00|00|

+---------+---------------+----------+
18:19:08,150,907   ETHER
|0   |1e|33|c8|aa|1b|40|8a|4b|47|20|ce|e4|08|00|45|00|05|99|39|1b|40|00|40|06|dc|9a|0a|c8|00|0d|0a|64|0a|71|15|b3|58|a7|62|61|d4|6e|00|36|c0|1a|50|18|72|10|25|35|00|00|16|03|01|00|51|02|00|00|4d|03|01|e9|4e|66|d1|56|43|c2|f8|df|95|62|d2|a1|8f|99|85|39|d3|22|4e|92|64|85|75|47|8f|29|f5|15|11|8e|10|20|b7|07|bc|84|e7|15|1e|6b|53|ed|6c|d5|1c|f6|15|10|ab|03|94|db|15|19|47|b7|43|04|30|59|fb|f2|52|e5|00|35|00|00|05|ff|01|00|01|00|16|03|01|05|0d|0b|00|05|09|00|05|06|00|05|03|30|82|04|ff|30|82|02|e7|a0|03|02|01|02|02|09|00|df|41|71|c6|48|ff|eb|19|30|0d|06|09|2a|86|48|86|f7|0d|01|01|0b|05|00|30|16|31|14|30|12|06|03|55|04|03|0c|0b|65|78|61|6d|70|6c|65|2e|63|6f|6d|30|1e|17|0d|31|38|30|32|30|34|31|38|31|33|35|36|5a|17|0d|32|38|30|32|30|32|31|38|31|33|35|36|5a|30|16|31|14|30|12|06|03|55|04|03|0c|0b|65|78|61|6d|70|6c|65|2e|63|6f|6d|30|82|02|22|30|0d|06|09|2a|86|48|86|f7|0d|01|01|01|05|00|03|82|02|0f|00|30|82|02|0a|02|82|02|01|00|e0|22|b0|41|79|e5|60|92|2c|c7|1c|97|84|f8|16|46|39|27|67|26|ad|d7|c8|dc|a4|18|2e|5a|1a|aa|f7|92|b0|71|8f|f8|a5|e0|d9|52|d3|6a|c9|e1|ae|7c|ce|f1|1c|92|99|dd|77|b7|f7|db|b3|2b|f4|96|23|d4|d8|08|e8|e2|f2|15|7b|41|16|30|3d|21|cd|5f|b0|3d|5a|0c|ec|60|9b|d8|78|99|32|c4|9f|3c|1b|50|d9|d4|cf|f0|4b|58|ca|c6|6f|56|f5|57|04|c9|d5|60|56|d2|25|66|ee|fe|f0|da|79|d6|f2|a5|67|08|7a|19|e6|82|90|5c|e0|a1|63|cf|4b|a8|43|c6|d3|08|a0|22|4a|df|be|42|22|8b|c9|6f|81|d1|b0|d8|e0|24|79|b3|3a|ad|10|af|bd|7a|43|56|a6|87|ad|d2|d4|96|cf|12|f9|9a|42|7a|c7|63|07|f7|cc|a9|a5|50|1e|bf|4d|2d|ae|e3|74|b7|a7|66|ca|ab|44|66|db|f5|e2|28|dd|a8|6f|76|0a|5a|d4|8d|42|c6|a0|16|c2|bd|d9|25|38|74|dc|bc|d3|95|03|b9|13|39|c0|4a|cd|d9|c9|cb|34|bf|9e|48|1e|e4|56|8f|47|9c|33|ed|eb|d6|db|5e|b7|46|85|d4|49|48|f3|24|ec|01|c3|98|74|50|f9|05|71|01|b5|96|f4|b5|bc|a3|57|78|3f|c7|4f|10|1e|da|96|59|c0|6d|c1|13|64|52|7b|f8|d7|88|59|29|ba|b7|5a|ec|a2|a7|1e|86|bb|ea|8b|f8|03|e3|73|b3|6e|70|27|29|b6|92|31|4f|52|73|f0|62|91|3e|90|53|40|d0|00|e2|1d|80|5e|82|a9|5e|fc|56|2f|76|d9|40|39|85|e3|4e|55|d2|a7|47|06|27|92|d9|ec|dd|7e|7d|c2|07|6d|73|9b|58|75|da|ba|cd|65|e3|d9|3b|da|78|4e|96|81|c1|d1|2f|f2|c0|65|e0|a2|12|33|1c|eb|0c|2a|10|c6|ea|87|4c|7a|a5|d7|c3|4c|a8|45|06|8d|ca|a5|90|7d|e0|57|be|e5|17|f0|13|41|28|6e|f6|f3|cb|b3|d4|9b|fa|b1|f9|30|a1|b4|2e|a4|de|73|39|35|82|32|de|55|f9|70|ee|ca|8a|d2|62|64|fc|fb|d7|56|17|c7|9a|3c|4f|3d|b1|4a|1d|37|fb|ad|40|ef|c4|14|cf|08|57|8e|68|5f|2b|d4|3b|cc|bb|23|9f|83|64|e3|22|39|53|30|5c|4d|0f|d8|4d|2a|5a|23|e6|ee|06|b6|38|5d|83|ef|91|f8|25|a6|e3|02|03|01|00|01|a3|50|30|4e|30|1d|06|03|55|1d|0e|04|16|04|14|bf|22|b8|2a|ec|f6|61|a9|05|50|d8|6a|f5|35|bd|32|3e|7c|9b|d7|30|1f|06|03|55|1d|23|04|18|30|16|80|14|bf|22|b8|2a|ec|f6|61|a9|05|50|d8|6a|f5|35|bd|32|3e|7c|9b|d7|30|0c|06|03|55|1d|13|04|05|30|03|01|01|ff|30|0d|06|09|2a|86|48|86|f7|0d|01|01|0b|05|00|03|82|02|01|00|35|e5|1e|a9|74|4f|78|59|c0|fd|da|a4|71|7a|4e|9d|b7|13|42|5b|d5|f3|95|1b|48|a6|91|c8|84|be|fc|67|ed|92|4e|2e|f2|e5|6e|14|38|a5|af|51|7f|54|11|5f|ed|15|bd|a6|d9|ef|9c|ef|ee|db|ed|fb|39|45|e2|5e|a9|53|66|73|40|98|27|38|4d|a3|1d|5f|eb|d3|f8|33|01|3d|ce|08|61|60|e1|ee|91|28|98|0a|a2|17|28|17|39|c1|af|1e|d2|f7|42|c6|5f|ff|a2|fa|77|2a|f0|4b|ed|4f|ea|d4|55|b4|4d|02|63|4f|9f|12|ab|86|26|a6|24|26|7e|26|e2|55|89|6b|7d|52|3c|48|ad|4c|0c|76|64|7d|a7|8f|ff|9e|23|28|4c|d5|79|85|ea|0c|ee|54|31|24|e2|66|53|4c|ae|a6|a6|43|c5|e1|16|c7|48|b3|af|49|c9|f6|30|e6|4a|da|2f|9b|1a|9d|5f|15|ff|ac|9e|c7|d3|e4|85|55|1d|d6|42|af|58|1e|57|d1|84|4f|87|a7|43|10|e8|13|42|fd|49|fa|c1|a2|ba|6a|30|a4|1e|fb|db|be|99|16|a1|dc|1e|45|45|54|c3|89|b8|b6|b8|74|90|2f|8b|8c|68|49|07|a2|7c|0d|a6|d6|3c|80|e2|94|66|b0|83|1f|e3|29|02|4d|76|44|1e|65|57|7c|9e|f2|01|0a|77|cd|c5|85|43|53|9e|e7|56|21|4b|55|05|78|0e|6b|7a|14|44|69|d8|cb|06|de|65|9a|0a|8b|15|11|4e|21|c6|1d|be|d4|6c|45|96|88|f2|04|d5|ff|03|fd|d3|24|7d|25|22|8b|22|37|99|4b|22|fa|8f|99|ec|71|88|58|8e|03|9f|76|19|a2|05|a6|3b|bc|87|d8|90|db|c1|cc|98|f9|1f|c5|00|96|e4|90|3d|ad|d0|fa|a1|d5|b6|71|74|7d|0c|a3|f2|79|5a|e6|31|fb|de|65|41|46|39|78|bd|d6|c1|b0|1a|44|16|a1|ed|27|c6|e7|d6|ab|f0|2f|0f|78|a6|78|01|2c|ed|18|bb|47|34|54|ee|73|1b|99|83|ca|bf|d7|07|65|40|03|f2|75|98|d4|f3|22|6c|2f|77|1e|ec|e7|1e|bf|31|28|90|c4|0c|7a|37|cc|85|37|59|42|94|21|90|05|9e|e0|60|51|0a|83|b8|29|45|37|06|5a|5e|a3|b3|e0|08|cf|48|9b|51|17|7b|4c|fb|6d|06|e9|4f|dd|45|42|42|83|c2|f6|b3|70|29|86|08|c4|28|f7|b4|d6|1f|a1|63|94|f7|75|8f|ba|16|03|01|00|04|0e|00|00|00|

+---------+---------------+----------+
18:19:08,556,603   ETHER
|0   |8a|4b|47|20|ce|e4|1e|33|c8|aa|1b|40|08|00|45|00|00|7c|05|fd|00|00|7f|06|15|d6|0a|64|0a|71|0a|c8|00|0d|58|a7|15|b3|00|36|bf|c6|62|61|d4|6e|50|18|16|d0|67|d0|00|00|16|03|01|00|4f|01|00|00|4b|03|03|6e|30|37|19|a3|c0|a4|d5|d1|3c|8b|ed|76|72|e2|9f|d5|5e|57|3e|9e|98|b7|33|a9|12|54|de|40|32|df|34|00|00|12|00|3d|00|35|00|3c|00|2f|00|b7|00|95|00|b6|00|94|00|ff|01|00|00|10|00|0d|00|0c|00|0a|06|01|05|01|04|01|03|01|02|01|

+---------+---------------+----------+
18:19:08,556,617   ETHER
|0   |1e|33|c8|aa|1b|40|8a|4b|47|20|ce|e4|08|00|45|00|00|28|39|1c|40|00|40|06|e2|0a|0a|c8|00|0d|0a|64|0a|71|15|b3|58|a7|62|61|d9|df|00|36|c0|1a|50|10|72|10|1f|c4|00|00|

+---------+---------------+----------+
18:19:08,683,863   ETHER
|0   |8a|4b|47|20|ce|e4|1e|33|c8|aa|1b|40|08|00|45|00|00|28|05|fe|00|00|7f|06|16|29|0a|64|0a|71|0a|c8|00|0d|58|a7|15|b3|00|36|c0|1a|62|61|d9|df|50|11|11|5f|13|df|00|00|

+---------+---------------+----------+
18:19:08,683,932   ETHER
|0   |1e|33|c8|aa|1b|40|8a|4b|47|20|ce|e4|08|00|45|00|00|28|39|1d|40|00|40|06|e2|09|0a|c8|00|0d|0a|64|0a|71|15|b3|58|a7|62|61|d9|df|00|36|c0|1b|50|11|72|10|1f|c4|00|00|

+---------+---------------+----------+
18:19:08,762,143   ETHER
|0   |8a|4b|47|20|ce|e4|1e|33|c8|aa|1b|40|08|00|45|00|00|28|05|ff|00|00|7f|06|16|28|0a|64|0a|71|0a|c8|00|0d|58|a7|15|b3|00|36|c0|1b|62|61|d9|e0|50|10|11|5e|13|df|00|00|

+---------+---------------+----------+
18:19:16,784,169   ETHER
|0   |8a|4b|47|20|ce|e4|1e|33|c8|aa|1b|40|08|00|45|00|00|2c|06|00|00|00|7f|06|16|23|0a|64|0a|71|0a|c8|00|0d|18|30|15|b3|00|36|e7|9f|00|00|00|00|60|02|16|d0|4b|f4|00|00|02|04|05|b4|

+---------+---------------+----------+
18:19:16,784,184   ETHER
|0   |1e|33|c8|aa|1b|40|8a|4b|47|20|ce|e4|08|00|45|00|00|2c|00|00|40|00|40|06|1b|23|0a|c8|00|0d|0a|64|0a|71|15|b3|18|30|98|bc|9a|61|00|36|e7|a0|60|12|72|10|1f|c8|00|00|02|04|05|b4|

+---------+---------------+----------+
18:19:16,886,424   ETHER
|0   |8a|4b|47|20|ce|e4|1e|33|c8|aa|1b|40|08|00|45|00|00|7c|06|01|00|00|7f|06|15|d2|0a|64|0a|71|0a|c8|00|0d|18|30|15|b3|00|36|e7|a0|98|bc|9a|62|50|18|16|d0|70|29|00|00|16|03|01|00|4f|01|00|00|4b|03|03|e5|7a|5a|74|be|6f|f0|54|cf|60|fd|45|d1|15|6e|25|28|82|23|11|ce|26|76|8f|3a|9b|db|bf|31|93|66|24|00|00|12|00|3d|00|35|00|3c|00|2f|00|b7|00|95|00|b6|00|94|00|ff|01|00|00|10|00|0d|00|0c|00|0a|06|01|05|01|04|01|03|01|02|01|

+---------+---------------+----------+
18:19:16,886,440   ETHER
|0   |1e|33|c8|aa|1b|40|8a|4b|47|20|ce|e4|08|00|45|00|00|28|6c|15|40|00|40|06|af|11|0a|c8|00|0d|0a|64|0a|71|15|b3|18|30|98|bc|9a|62|00|36|e7|f4|50|10|72|10|1f|c4|00|00|

+---------+---------------+----------+
18:19:16,886,658   ETHER
|0   |1e|33|c8|aa|1b|40|8a|4b|47|20|ce|e4|08|00|45|00|05|99|6c|16|40|00|40|06|a9|9f|0a|c8|00|0d|0a|64|0a|71|15|b3|18|30|98|bc|9a|62|00|36|e7|f4|50|18|72|10|25|35|00|00|16|03|01|00|51|02|00|00|4d|03|01|24|c5|4f|88|a2|2e|d6|95|40|f4|42|c1|97|78|b8|4a|33|43|32|94|d8|81|88|7f|09|44|df|3a|6c|79|d2|3a|20|1d|a7|32|d9|88|67|21|c6|69|f6|70|a7|20|28|0c|ac|a4|da|d0|95|3f|02|4f|51|68|78|7e|bf|1c|53|46|69|00|35|00|00|05|ff|01|00|01|00|16|03|01|05|0d|0b|00|05|09|00|05|06|00|05|03|30|82|04|ff|30|82|02|e7|a0|03|02|01|02|02|09|00|df|41|71|c6|48|ff|eb|19|30|0d|06|09|2a|86|48|86|f7|0d|01|01|0b|05|00|30|16|31|14|30|12|06|03|55|04|03|0c|0b|65|78|61|6d|70|6c|65|2e|63|6f|6d|30|1e|17|0d|31|38|30|32|30|34|31|38|31|33|35|36|5a|17|0d|32|38|30|32|30|32|31|38|31|33|35|36|5a|30|16|31|14|30|12|06|03|55|04|03|0c|0b|65|78|61|6d|70|6c|65|2e|63|6f|6d|30|82|02|22|30|0d|06|09|2a|86|48|86|f7|0d|01|01|01|05|00|03|82|02|0f|00|30|82|02|0a|02|82|02|01|00|e0|22|b0|41|79|e5|60|92|2c|c7|1c|97|84|f8|16|46|39|27|67|26|ad|d7|c8|dc|a4|18|2e|5a|1a|aa|f7|92|b0|71|8f|f8|a5|e0|d9|52|d3|6a|c9|e1|ae|7c|ce|f1|1c|92|99|dd|77|b7|f7|db|b3|2b|f4|96|23|d4|d8|08|e8|e2|f2|15|7b|41|16|30|3d|21|cd|5f|b0|3d|5a|0c|ec|60|9b|d8|78|99|32|c4|9f|3c|1b|50|d9|d4|cf|f0|4b|58|ca|c6|6f|56|f5|57|04|c9|d5|60|56|d2|25|66|ee|fe|f0|da|79|d6|f2|a5|67|08|7a|19|e6|82|90|5c|e0|a1|63|cf|4b|a8|43|c6|d3|08|a0|22|4a|df|be|42|22|8b|c9|6f|81|d1|b0|d8|e0|24|79|b3|3a|ad|10|af|bd|7a|43|56|a6|87|ad|d2|d4|96|cf|12|f9|9a|42|7a|c7|63|07|f7|cc|a9|a5|50|1e|bf|4d|2d|ae|e3|74|b7|a7|66|ca|ab|44|66|db|f5|e2|28|dd|a8|6f|76|0a|5a|d4|8d|42|c6|a0|16|c2|bd|d9|25|38|74|dc|bc|d3|95|03|b9|13|39|c0|4a|cd|d9|c9|cb|34|bf|9e|48|1e|e4|56|8f|47|9c|33|ed|eb|d6|db|5e|b7|46|85|d4|49|48|f3|24|ec|01|c3|98|74|50|f9|05|71|01|b5|96|f4|b5|bc|a3|57|78|3f|c7|4f|10|1e|da|96|59|c0|6d|c1|13|64|52|7b|f8|d7|88|59|29|ba|b7|5a|ec|a2|a7|1e|86|bb|ea|8b|f8|03|e3|73|b3|6e|70|27|29|b6|92|31|4f|52|73|f0|62|91|3e|90|53|40|d0|00|e2|1d|80|5e|82|a9|5e|fc|56|2f|76|d9|40|39|85|e3|4e|55|d2|a7|47|06|27|92|d9|ec|dd|7e|7d|c2|07|6d|73|9b|58|75|da|ba|cd|65|e3|d9|3b|da|78|4e|96|81|c1|d1|2f|f2|c0|65|e0|a2|12|33|1c|eb|0c|2a|10|c6|ea|87|4c|7a|a5|d7|c3|4c|a8|45|06|8d|ca|a5|90|7d|e0|57|be|e5|17|f0|13|41|28|6e|f6|f3|cb|b3|d4|9b|fa|b1|f9|30|a1|b4|2e|a4|de|73|39|35|82|32|de|55|f9|70|ee|ca|8a|d2|62|64|fc|fb|d7|56|17|c7|9a|3c|4f|3d|b1|4a|1d|37|fb|ad|40|ef|c4|14|cf|08|57|8e|68|5f|2b|d4|3b|cc|bb|23|9f|83|64|e3|22|39|53|30|5c|4d|0f|d8|4d|2a|5a|23|e6|ee|06|b6|38|5d|83|ef|91|f8|25|a6|e3|02|03|01|00|01|a3|50|30|4e|30|1d|06|03|55|1d|0e|04|16|04|14|bf|22|b8|2a|ec|f6|61|a9|05|50|d8|6a|f5|35|bd|32|3e|7c|9b|d7|30|1f|06|03|55|1d|23|04|18|30|16|80|14|bf|22|b8|2a|ec|f6|61|a9|05|50|d8|6a|f5|35|bd|32|3e|7c|9b|d7|30|0c|06|03|55|1d|13|04|05|30|03|01|01|ff|30|0d|06|09|2a|86|48|86|f7|0d|01|01|0b|05|00|03|82|02|01|00|35|e5|1e|a9|74|4f|78|59|c0|fd|da|a4|71|7a|4e|9d|b7|13|42|5b|d5|f3|95|1b|48|a6|91|c8|84|be|fc|67|ed|92|4e|2e|f2|e5|6e|14|38|a5|af|51|7f|54|11|5f|ed|15|bd|a6|d9|ef|9c|ef|ee|db|ed|fb|39|45|e2|5e|a9|53|66|73|40|98|27|38|4d|a3|1d|5f|eb|d3|f8|33|01|3d|ce|08|61|60|e1|ee|91|28|98|0a|a2|17|28|17|39|c1|af|1e|d2|f7|42|c6|5f|ff|a2|fa|77|2a|f0|4b|ed|4f|ea|d4|55|b4|4d|02|63|4f|9f|12|ab|86|26|a6|24|26|7e|26|e2|55|89|6b|7d|52|3c|48|ad|4c|0c|76|64|7d|a7|8f|ff|9e|23|28|4c|d5|79|85|ea|0c|ee|54|31|24|e2|66|53|4c|ae|a6|a6|43|c5|e1|16|c7|48|b3|af|49|c9|f6|30|e6|4a|da|2f|9b|1a|9d|5f|15|ff|ac|9e|c7|d3|e4|85|55|1d|d6|42|af|58|1e|57|d1|84|4f|87|a7|43|10|e8|13|42|fd|49|fa|c1|a2|ba|6a|30|a4|1e|fb|db|be|99|16|a1|dc|1e|45|45|54|c3|89|b8|b6|b8|74|90|2f|8b|8c|68|49|07|a2|7c|0d|a6|d6|3c|80|e2|94|66|b0|83|1f|e3|29|02|4d|76|44|1e|65|57|7c|9e|f2|01|0a|77|cd|c5|85|43|53|9e|e7|56|21|4b|55|05|78|0e|6b|7a|14|44|69|d8|cb|06|de|65|9a|0a|8b|15|11|4e|21|c6|1d|be|d4|6c|45|96|88|f2|04|d5|ff|03|fd|d3|24|7d|25|22|8b|22|37|99|4b|22|fa|8f|99|ec|71|88|58|8e|03|9f|76|19|a2|05|a6|3b|bc|87|d8|90|db|c1|cc|98|f9|1f|c5|00|96|e4|90|3d|ad|d0|fa|a1|d5|b6|71|74|7d|0c|a3|f2|79|5a|e6|31|fb|de|65|41|46|39|78|bd|d6|c1|b0|1a|44|16|a1|ed|27|c6|e7|d6|ab|f0|2f|0f|78|a6|78|01|2c|ed|18|bb|47|34|54|ee|73|1b|99|83|ca|bf|d7|07|65|40|03|f2|75|98|d4|f3|22|6c|2f|77|1e|ec|e7|1e|bf|31|28|90|c4|0c|7a|37|cc|85|37|59|42|94|21|90|05|9e|e0|60|51|0a|83|b8|29|45|37|06|5a|5e|a3|b3|e0|08|cf|48|9b|51|17|7b|4c|fb|6d|06|e9|4f|dd|45|42|42|83|c2|f6|b3|70|29|86|08|c4|28|f7|b4|d6|1f|a1|63|94|f7|75|8f|ba|16|03|01|00|04|0e|00|00|00|

+---------+---------------+----------+
18:19:16,933,784   ETHER
|0   |8a|4b|47|20|ce|e4|1e|33|c8|aa|1b|40|08|00|45|00|00|28|06|02|00|00|7f|06|16|25|0a|64|0a|71|0a|c8|00|0d|18|30|15|b3|00|36|e7|f4|98|bc|9f|d3|50|11|11|5f|30|2d|00|00|

+---------+---------------+----------+
18:19:16,933,874   ETHER
|0   |1e|33|c8|aa|1b|40|8a|4b|47|20|ce|e4|08|00|45|00|00|28|6c|17|40|00|40|06|af|0f|0a|c8|00|0d|0a|64|0a|71|15|b3|18|30|98|bc|9f|d3|00|36|e7|f5|50|11|72|10|1f|c4|00|00|

+---------+---------------+----------+
18:19:16,973,340   ETHER
|0   |8a|4b|47|20|ce|e4|1e|33|c8|aa|1b|40|08|00|45|00|00|28|06|03|00|00|7f|06|16|24|0a|64|0a|71|0a|c8|00|0d|18|30|15|b3|00|36|e7|f5|98|bc|9f|d4|50|10|11|5e|30|2d|00|00|

Answer 1

What does this error actually mean?

At the risk of stating the obvious it literally means "the handshake failed" and doesn't give you any more information than that. It could be caused by any number of possible issues. From your debug output we can see that:

1. The client issues a ClientHello to the server
2. The server responds with a ServerHello, a Certificate and a ServerHelloDone.
3. The server then attempts to read a response from the client but this read fails (most probably because the client has aborted - but without sending any kind of alert to indicate the cause of the problem).

Clearly something in the response that your server has sent back to the client has caused the client to just give up. The most likely culprit is the certificate, but its also possible that there is something about the ServerHello it doesn't like (e.g. the server didn't understand some extension that the client sent in its ClientHello, which it refuses to work without...although this is quite unlikely)

Answer 2

When looking in details to your debug log, it looks like a normal client/server handshake, in TLS 1.2. The last 9 bytes written mean the SERVER HELLO DONE is accomplished, and then the client (the last "read from") is sending 5 bytes that unfortunately we don't have. 5 bytes is a normal thing at this moment, these bytes are supposed to continue with the TLS exchange (Client key exchange,..) and they contain the "hello" byte, the protocol version (2 bytes), and the data length (2 bytes). All this work would be easier for me if you provided a network capture. But OK, we have some information.

There is one reason to make openssl bug at this moment, when we receive 5 bytes instead of 11. This code will later generate the same handshake failure that you get :

 * Request this many bytes in initial read.
 * We can detect SSL 3.0/TLS 1.0 Client Hellos
 * ('type == 3') correctly only when the following
 * is in a single record, which is not guaranteed by
 * the protocol specification:
 * Byte  Content
 *  0     type            \
 *  1/2   version          > record header
 *  3/4   length          /
 *  5     msg_type        \
 *  6-8   length           > Client Hello message
 *  9/10  client_version  /
 */
char buf_space[11];

But without any capture, it's almost impossible to say whether the client is sending a wrong Client key exchange, or if it's your openssl version which is faulty (more likely, it's the client). Can you check what openssl version you are running ?

Answer 3

This appears to be a network or stack problem.

Assuming we are the server at 10.200.0.13 and the client is 10.100.10.113 your network dump decodes as follows:

18:19:07,529,418 receive connect request (SYN) from port 22695 to port 5555 with starting seq 0x0036bfc5 
18:19:07,529,450 send accept (SYN ACK) with starting seq 0x6261d46d
18:19:07,668,343 receive segment containing ClientHello message with seq 0x0036bfc6 len 0x54 
18:19:07,668,359 send ack (through 0x0036c01a) 
18:19:07,668,617 send segment containing ServerHello, Certificate, ServerHelloDone with seq 0x6261d46e len 0x56+0x512+9=0x571 

18:19:08,150,907 after no response ~0.5s, resend segment 

18:19:08,556,603 receive resend of client 0x0036bfc6, which does NOT ack our data;
this implies client didn't get our data-ack and didn't get or accept (either copy of) our 0x6261d46e 
18:19:08,556,617 send probe (ACK with no additional data) 

18:19:08,683,863 receive probe from client, again neg-acking data but pos-acking SYN
18:19:08,683,932 after 2x neg-ack, send FIN (seq 0x6261d9df) 
18:19:08,762,143 receive ACK of our FIN (and data?) (ack=0x6261d9e0)

18:19:16,784,169 receive new connect (SYN) from port 6192 
...

This is screwy. If the client is any normal computer with an OS, usually those ack data at TCP level in the protocol stack even if their user program doesn't read (or accept) the data, at least up to some reasonable buffersize. The client is clearly getting at least some of our frames, since it correctly acks SYN and FIN, but it apparently refuses to ack our data.

If the client is something like a thermostat or an Arduino, where there is no actual OS but often instead a heap of code someone mashed together, often by copying from websites (including Stack?) without understanding it, so it doesn't work right. Or possibly the actual client is correct, but there is some broken middlebox like a firewall, IDS, 'smart' router, etc. in between.

Anyway if you can't get the TLS protocol messages, which are TCP data, delivered to the client at the TCP level, nothing else can work, so without any more useful info about or control over the client you're out of luck. Sorry.

Answer 4

bad gethostbyaddr

A reverse DNS is not working well on your server. With your server IP, run these commands and check if they return anything (I suppose you're on Linux) :

dig -x XXX.XXX.XXX.XXX
host XXX.XXX.XXX.XXX

Check that your hosts file solves correctly your hostname. If you have to configure a DNS server (/etc/resolv.conf), do it.