How to make https://localhost:port use self-signed certificate in embedded Jetty?

Question

Goal: I want this url ( https://localhost:8083 ) to use my self-signed certiciate on my localmachine.

First, I referenced this url (https://gist.github.com/oslego/f13e136ffeaa6174289a) and what I did was:

$ openssl genrsa -des3 -out server.orig.key 2048

$ openssl rsa -in server.orig.key -out server.key

$ openssl req -new -key server.key -out server.csr

Country Name (2 letter code) [AU]:

...

Common Name: localhost.ssl

...

$ openssl x509 -req -days 365 -in server.csr -signkey server.key -out server.crt

$ echo "127.0.0.1 localhost.ssl" | sudo tee -a /private/etc/hosts

I then convert server.crt to server.pem by executing

$ openssl x509 -in server.crt -out keystore.pem -outform PEM

$ keytool -import -trustcacerts -alias testCert -file keystore.pem -keypass testpassword -keystore keystore.jks -storepass testpassword

$ keytool -export -alias mykey -keystore keystore.jks -rfc -file truststore

$ vim keystore.password // manually created keystore.password via vim

But when I access to https://localhost:8083, it doesn't work with SSL.

Then,
I also created another certificate with

Common Name: localhost

$ echo "127.0.0.1 localhost" | sudo tee -a /private/etc/hosts

But this is not working as well. How can I make my https://localhost:8083 uses my self-signed certificate?

FYI, I use embedded Jetty and Java reads all the information correctly via config files which defines the locations of keystore.jks, truststore, and keystore.password files.

Answer 1

(1) Directing you to do genrsa with encryption and then rsa to remove encryption is a good indication of someone one doesn't know what they're talking about and hasn't read the man page for at least 10 years. Directing you to do req -new and then x509 -req -signkey is an excellent indication of someone who doesn't know what they're talking about and hasn't read the man page for at least 5 years. I advise completely ignoring any website or author who says those things. Also the cert output by either x509 -req -signkey or req -new -x509 is already PEM and doesn't need any conversion.

(2) You indicate putting name translations in /private/etc/hosts and give no clue what browser(s) and/or other client(s) or middleware(s) you are using, but I've never heard of any client(s) that use such a file for name resolution unless you are using a jail or container of some kind you left out of your question. (Practically all Unix systems do or can use /etc/hosts but that's not the same.)

(3) You also give no clue what 'doesn't work' means. But if it means handshake failure because of cipher mismatch (not always identified as such) it is undoubtedly because you put in the Java keystore and thus gave Jetty only your certificate when SSL/TLS server needs the privatekey. That makes this a duplicate of multiple questions linked at https://stackoverflow.com/a/37423399/2868801 which all explain, with slightly different details, that you need to use openssl pkcs12 -export to convert the certificate PLUS privatekey to a PKCS12 file, and then either use or convert the PKCS12 file in Java.

(4) jww is partly correct; current standards require certs to contain the SubjectAltName (SAN) extension, and you should do that -- which is much easier with req -new -x509 rather than the method you have, see (1). But so far only Chrome enforces this requirement, so an officially-obsolete cert with only Subject.CN will actually work in other clients -- once you set them to trust this cert, which is always an issue for selfsigned.