Technology: Java using SSL Certificate, or TLS1.2 - 1.3 encryption
Product: web server (https), SMTP email, force encryption MS SQL Server
There are several steps involve in setup SSL for various servers besides following standard
1. Create new SSL certificate for the virtual hostname used by the applications
2. Import the SSL certificate to the application servers
2.1. For Java, you need to import into Java keystore. For OpenJDK or Oracle Java (JRE), it is located in [java_jre_home]\lib\security, or [java_jdk_home\jre\lib\security.
2.2. For Linux or Windows binary programs, you need to import into OS. For RHEL, the SSL certificate file will be /etc/ssl/certs
3. Configures the application server to enable SSL, and indirectly specified the SSL certificate to use
4. Configures the application client to enable SSL, and optionally
4.1. Java - optionally specifies SSL keystore if not using default
Now, let's pick an application that leverages multiple technologies, and gives real picture what needs to be configured to enable SSL
Cognos Analytics report server
This reporting server has following functionality (typical, but could be more) that will has SSL encryption:
1. Web server - it is severing the report in html, so it is a web server
2. Web client - it is able to forward its traffic to a customer facing web server (IIS, Apache httpd, IBM httpd), so for this communication, it is a web client
3. SMTP email client - its interactive report and scheduled report can be sent out as email attachment (pdf, html)
4. DB client - it needs a "Content Store" DB to stores user credential and report template. It can has additional audit DB to store additional application data for audit functionality. Any DB that it is querying to display the reports are all act as a DB client
5. Authentication client - OOTB LDAP (Windows only) server, custom LDAP, Okta, IBM OpenConnect, etc authentication that it supports, it is acting as client
Therefore, for above 5 typical server & client functionality, SSL can be enabled and encrypts the traffic between remote servers and clients.
For all the above functionality, you need to understand that the product is a pure Java J2EE application. Therefore, all the SSL configuration will follow Java (standard) and IBM documentation (application specific).
Several and multiple SSL certificates involve in this encryption setup:
1. Java keystore located in [cognos_home]\configuration\certs\CAMKeystore (default password NoPassWordSet)
1.1. CAMKeystore keeps the SSL cert for web/https access for Cognos Analytics server. If there are multiple virtual hostname (such as internal access, intranet access, internet access), then there are multiple SSL certs (unless the SSL cert contains multiple virtual hostname)
1.2. CAMKeystore keeps the SSL cert for Cognos Analytics server's databases access, e.g. Content DB, audit DB, data store DB. Example is MS SQL Server force encryption. Each remote DB server uses different SSL cert, and CA root
1.3. CAMKeystore keeps the SSL cert for SMTP server, which used for email of report, or email notification functionality. Example is smtp protocol with STARTTLS that enforce SSL, or smtps protocol. Typically only 1 SMTP server, so only 1 SSL cert
1.4. CAMKeystore keeps the SSL cert for client side https with remote customer users facing web server
1.5. CAMKeystore keeps the SSL cert for remote authentication server, e.g. LDAP , ActiveDirectory, and SSO. Each authentication server has its own SSL cert
Procedure:
1. Determine the root certificate for all 5 components above. Each server will have its own SSL cert, and might not use the same CA root certificate
2. Verify in [cognos_home]\configuration\certs\CAMKeystore that the CA root certificate is already there, and will be recognize
2.1. Follows Cognos Analytics doc: DLS_SSL_CertImportTool.bat (Windows), or sh (Linux) - https://www.ibm.com/docs/en/cognos-analytics/11.2.0?topic=server-enabling-secure-tls-connection-your-email
3. Even if it is in CAMKeystore, verify that it is not expired
4. If not there, or expired, then export the CA root cert from remote server
5. Import into CAMKeystore, which is the IBM Java keystore
6. For DB used in data store, import the SSL cert into [cognos_home]\ibm-jre\jre\lib\security\cacerts
6. Follows various IBM Cognos Analytics document to enable SSL encryption
6.1. For Cognos Analytics 12.0.0 SMTP - https://www.ibm.com/docs/en/cognos-analytics/12.0.0?topic=server-enabling-secure-tls-connection-your-email
6.2. For Cognos Analytics 11.2.0 Content DB, audit DB, logging DB as database client - https://www.ibm.com/docs/en/cognos-analytics/11.2.0?topic=communications-enabling-ssl-db2-informix-databases
6.3. For data store as DB client - https://www.ibm.com/docs/en/cognos-analytics/11.2.0?topic=communications-enabling-ssl-db2-informix-databases
Tools
There are various DOS utilities that can troubleshoot SSL communication, but none for Java. Some sample Java programs are available, but you have to compile, as well as familiar with SSL logging parameters that you have to manually specify.
Tools for
1. Windows: openssl
2. UNIX, Linux - openssl, curl
3. Java SMTP - Oracle javamail-samples.zip that based on Oracle javax.mail.jar library that you need to download. It contains sample SMTP with STARTTLS sample code that you need to modify the Java program to specify SMTP port (default port 25)
4. Java MS SQL Server client - Microsoft provided sample https://learn.microsoft.com/en-us/sql/connect/jdbc/connection-url-sample?view=sql-server-ver16
Tool - opemnssl for SMTP
For SMTP that uses STARTTLS to enable encryption, uses following command to troubleshoot SSL issue
openssl s_client -starttls smtp -crlf -connect [remote email server hostname]:[smtp port]
This relies on OS SSL certificate store, so not applicable for Java SMTP troubleshooting
Tool - opemnssl for SMTPS
Same as above, but uses SMTPS SSL encryption, not SMTP + STARTTLS
openssl s_client smtps -crlf -connect [remote email server hostname]:[smtp port]
Tool - curl for SMTP
curl is preinstalled in Linux. It can login to SMTP email server, and send out email. Therefore, it is useful to ensure the remote SMTP server is properly configured to allow your application server to send out email. This uses SMTP + STARTTLS protocol
curl -v smtp://email-smtp.us-east-1.amazonaws.com:587 --user "login-user-name:password-to-be-filled" --mail-from "[username that allow to be used to send out" --mail-rcpt "email addr to receive the email" -T emailcontent.txt --ssl
Tool - curl for SMTPS
Same as above, but uses SMTPS instead of SMTP + STARTTLS
curl -v smtps://email-smtp.us-east-1.amazonaws.com:587 --user "login-user-name:password-to-be-filled" --mail-from "[username that allow to be used to send out" --mail-rcpt "email addr to receive the email" -T emailcontent.txt
Tool - Test-NetConnection
PowerShell Test-NetConnection can be used to verify the application server machine can reach remote SMTP/DB/LDAP/Okta/SSO/web server. This will give you a quick indication that the firewall, router, DNS, AWS Security Group, etc are configured to allow them to talk before you perform application side of testing, such as openssl, or curl above
test-netconnection [remote server hostname] -port [remote server port]
Example for MS SQL Server: test-netconnection [remote MS SQL Server virtual hostname] -port 1433
Example for web server: test-netconnection [remote web server virtual hostname] -port 443
Example for Oracle DB: test-netconnection [remote Oracle virtual hostname] -port 1521
Tool - Java
Enable following SSL debug parameter when running sample MS SQL Server, Oracle sample JavaMail program, etc
Example for using Oracle JavaMail sample called smtpsend.java: java -cp .;..\javax.mail.jar;..\javax.activation-api-1.2.0.jar -Djavax.net.debug=ssl:handshake smtpsend
