JDBC Could not establish SSL connection (Domino Java Agent FP 9) to SQL Server

Question

I have a Java Agent running on a Domino Server. It was running fine on version Domino 9.0.1 FP7. The MS Windows server is 2003. Waiting on SQL server version.

Now we upgraded the Domino Server to 9.0.1 FP9. The JVM went from 1.6 to 1.8.

I am assuming this Exception error is caused by the JVM or Java Policy because I upgraded the JDBC driver from 4.2 to 6.2.2 trying to fix this with no success (same error).

Connection String Original (JDBC Driver ...jvm/lib/ext/mssql-jdbc-6.2.2.jre8.jar)

String driver = "com.microsoft.sqlserver.jdbc.SQLServerDriver";
            String url = "jdbc:sqlserver://SERVER";

Connection String Same Result (JDBC Driver ...jvm/lib/ext/mssql-jdbc-6.2.2.jre8.jar) - added encrypt=false.

String driver = "com.microsoft.sqlserver.jdbc.SQLServerDriver";
            String url = "jdbc:sqlserver://SERVER:1433;encrypt=false;";

The Exception

com.microsoft.sqlserver.jdbc.SQLServerException: The driver could not establish a secure connection to SQL Server by using Secure Sockets Layer (SSL) encryption. Error: "SQL Server did not return a response. The connection has been closed. ClientConnectionId:e8a3a2c4-d9a9-4f82-a63f-967cae0c29f0".
    at com.microsoft.sqlserver.jdbc.SQLServerConnection.terminate(SQLServerConnection.java:2435)
    at com.microsoft.sqlserver.jdbc.TDSChannel.enableSSL(IOBuffer.java:1816)
    at com.microsoft.sqlserver.jdbc.SQLServerConnection.connectHelper(SQLServerConnection.java:2022)
    at com.microsoft.sqlserver.jdbc.SQLServerConnection.login(SQLServerConnection.java:1687)
    at com.microsoft.sqlserver.jdbc.SQLServerConnection.connectInternal(SQLServerConnection.java:1528)
    at com.microsoft.sqlserver.jdbc.SQLServerConnection.connect(SQLServerConnection.java:866)
    at com.microsoft.sqlserver.jdbc.SQLServerDriver.connect(SQLServerDriver.java:569)
    at java.sql.DriverManager.getConnection(DriverManager.java:675)
    at java.sql.DriverManager.getConnection(DriverManager.java:258)
    at OrderTrackOrders.NotesMain(Unknown Source)
    at lotus.domino.AgentBase.runNotes(Unknown Source)
    at lotus.domino.NotesThread.run(Unknown Source)
Caused by: java.io.IOException: SQL Server did not return a response. The connection has been closed. ClientConnectionId:e8a3a2c4-d9a9-4f82-a63f-967cae0c29f0
    at com.microsoft.sqlserver.jdbc.TDSChannel$SSLHandshakeInputStream.ensureSSLPayload(IOBuffer.java:774)
    at com.microsoft.sqlserver.jdbc.TDSChannel$SSLHandshakeInputStream.readInternal(IOBuffer.java:830)
    at com.microsoft.sqlserver.jdbc.TDSChannel$SSLHandshakeInputStream.read(IOBuffer.java:821)
    at com.microsoft.sqlserver.jdbc.TDSChannel$ProxyInputStream.readInternal(IOBuffer.java:1003)
    at com.microsoft.sqlserver.jdbc.TDSChannel$ProxyInputStream.read(IOBuffer.java:991)
    at com.ibm.jsse2.b.a(b.java:272)
    at com.ibm.jsse2.b.a(b.java:148)
    at com.ibm.jsse2.at.a(at.java:19)
    at com.ibm.jsse2.at.i(at.java:627)
    at com.ibm.jsse2.at.a(at.java:689)
    at com.ibm.jsse2.at.startHandshake(at.java:432)
    at com.microsoft.sqlserver.jdbc.TDSChannel.enableSSL(IOBuffer.java:1753)
    ... 10 more

What version of SQL Server? Has the SQL Server been configured to force protocol encryption? Are there any more details about the login failure in the SQL Log? — David Browne - Microsoft
Ms windows 2003 server. I will report back once I find out the SQL server version. If O/S does not support TLS 1.2, what are my options? — xpagesbeast
You really shouldn't change things on an out-of-support operating system. If Force Encryption is set on the SQL Server, perhaps you can turn that off and connect without encryption. — David Browne - Microsoft
That is an idea, do I have to be concerned with other processes that maybe connecting with SSL? For example web servers, will other things break expecting SSL? — xpagesbeast
@david, can you look at this one stackoverflow.com/questions/48007778/… — xpagesbeast

Tom Hillebrand Tom Hillebrand · Accepted Answer · 2017-12-22T17:46:11

Prior to Domino FeaturePack 9 this agent ran fine on FP6 and FP8.

This is a security issue related to disabled TLS security cyphers in the FP9 java.security file when connecting to a MS SQL server running on Windows 2003 server.

The solution is to make this change in the java.security file 'Domino Program Dir/jvm/lib/security/java.security' Change these two lines to remove 3DES_EDE_CBC, DESede From:

jdk.tls.disabledAlgorithms=SSLv3, RC4, MD5withRSA, DH keySize < 768, 3DES_EDE_CBC, DESede, \ EC keySize < 224

To:

jdk.tls.disabledAlgorithms=SSLv3, RC4, MD5withRSA, DH keySize < 768, EC keySize < 224

Feature Pack 9 makes the following changes to the java.security file in the Domino Program directory on the server: Domino Program Dir/jvm/lib/security/java.security

Change 1 Prior to FP9:

jdk.certpath.disabledAlgorithms=MD2, MD5, RSA keySize < 1024

Updated in FP9:

jdk.certpath.disabledAlgorithms=MD2, MD5, RSA keySize < 1024, \ 
    DSA keySize < 1024, EC keySize < 224

Change 2 Prior to FP9: jdk.jar.disabledAlgorithms=MD2, RSA keySize < 1024 Updated in FP9: jdk.jar.disabledAlgorithms=MD2, MD5, RSA keySize < 1024

Change 3 Prior to FP9: jdk.tls.disabledAlgorithms=SSLv3, RC4, MD5withRSA, DH keySize < 768 Updated in FP9:

jdk.tls.disabledAlgorithms=SSLv3, RC4, MD5withRSA, DH keySize < 768, 3DES_EDE_CBC, DESede, \ 
    EC keySize < 224

Change 4 (the following entries are NEW in FP9)

jdk.xml.dsig.secureValidationPolicy=\ 
disallowAlg http://www.w3.org/TR/1999/REC-xslt-19991116,\ 
disallowAlg http://www.w3.org/2001/04/xmldsig-more#rsa-md5,\ 
disallowAlg http://www.w3.org/2001/04/xmldsig-more#hmac-md5,\ 
disallowAlg http://www.w3.org/2001/04/xmldsig-more#md5,\ 
maxTransforms 5,\ 
maxReferences 30,\ 
disallowReferenceUriSchemes file http https,\ 
minKeySize RSA 1024,\ 
minKeySize DSA 1024,\ 
noDuplicateIds,\ 
noRetrievalMethodLoops 

sun.rmi.registry.registryFilter=javax.rmi.CORBA.Stub

JDBC Could not establish SSL connection (Domino Java Agent FP 9) to SQL Server

2 Answers