Using SSL/TLS

Telnet Secure Socket Layer (SSL) and Transport Layer Security (TLS) security protocols are available for 3270 and 5250 session types, and Telnet Extended SSL/TLS support is available for 3270 session types. These Telnet options apply to the connection between a host and the Host Integrator session server or Design Tool. They do not apply to the connection between the client and the Host Integrator session server.

How to enable TLS/SSL encryption

To configure SSL/TLS encryption in your model:

1. The Design Tool must be offline and disconnected.

2. To modify an existing model, click Connection > Session Setup. To configure a new model, click File > New to display the New Model dialog.

3. Select the Transport "Use SSL/TLS" checkbox (for 3270 or 5250).

4. After connecting to the host using the Design Tool, you can determine the negotiated cipher, see Settings > View Settings > Host Communication > Telnet > Secure Host SSL Negotiated Cipher. The TLS protocol version and negotiated cipher are also logged in model debug messages (.vmr files).

Enabling FIPS 140-2 Validated Encryption

The Federal Information Processing Standards (FIPS) is used by US government agencies. When using TLS/SSL, you can enable FIPS 140-2 validated encryption. To enable this feature, set an operating system environment variable, VHI_FIPS=1, before you start the session server or Design Tool.

To confirm FIPS 140-2 encryption is enabled:

1. Open the Administrative Console.

2. You can verify that FIPS is enabled on the Session Server > Properties > General > Security panel and in the session server log. FIPS mode is not supported on the IBM AIX platform.

Certificate Checking

This section describes how the Design tool and Session Server verify a secure connection to a 3270 or 5250 host. It does not apply to other TLS connections made by, or to, VHI components.

As part of the standard TLS handshake, the host will send a chain of certificates to the Design tool or Session Server.

The following checks are performed: 1. The certificate chain from the host must be complete, and signed by one of the trust anchors in the machine's trust store. 2. The name of the host must be equal to one of the Subject Alternative Names (SAN) listed in the server certificate. Note that in VHI, the host name can be specified in the model or in the deployment descriptors.

VHI does not perform revocation checks.

VHI does not download CA certificates using AIA extensions.

If the host presents a certificate chain from a well-known CA, then your machine's trust store should already be correct, and no changes are needed. If the host presents a self-signed certificate, or if the administrator has used a proprietary CA, then you can modify your trust store. Do not grab certificates from the connection, always contact the administrator of the host and ask for the correct certificate(s) to import.

Importing certificates

Windows: you can import a certificate using CertMgr /add <filename> /s /r localMachine root.

Alternatively, you can use certlm.msc. Place the certificate into the "Trusted Root Certification Authorities".

By importing the certificate into the "localMachine" store it becomes available to all users, which is necessary for the Session Server.

Linux: you can import a certificate (in PEM format) using sudo trust anchor <filename>.

Validation Errors

Error 3055 indicates a validation error and contains a detail string. Here are a few validation errors that can occur.

• unable to get issuer certificate
  The issuer certificate of a locally looked up certificate could not be found. This normally means the list of trusted certificates is not complete.
  Solution: ask the administrator to configure the host to send all intermediary certificates in the TLS handshake.
  Solution: contact the administrator of the host and ask for the correct certificate(s) to import.
• certificate has expired
  The certificate has expired: the notAfter date is before the current time.
  Solution: ask the administrator of the host to install a new certificate.
• self-signed certificate
  The host certificate is self-signed and the same certificate cannot be found in the list of trusted certificates.
  Solution: import the self-signed certificate into the trust store.
• self-signed certificate in certificate chain
  The certificate chain could be built up, but no suitable trust anchor (which typically is a self-signed root certificate) could be found in the trust store.
  Solution: contact the administrator of the host and ask for the correct certificate(s) to import.
• host name mismatch
  The name of the host in the model or in the model's deployment descriptor does not match.
  Solution: change the model or the deployment descriptor to use one of the SANs in the host certificate.
• ip address mismatch
  You are connecting to the host using an IP address and it is not one of the SANs.
  Solution: connect using a name, not an IP address. Certificates with a SAN IP address are rare.

Other errors can also occur. Refer to the OpenSSL documentation for a full list.

Altering the configuration of TLS connections

The configuration of host TLS connections is performed using an OpenSSL Configuration file, %VHI_ROOT%/bin/openssl-vhi.cnf. In this file, you will find a section [hostssl] where you can make changes to the configuration.

Note that the configuration file is read once, at startup. To see the effect of your edits, restart the Design Tool or Session Server.

Disabling TLS 1.3

If your host fails to negotiate TLS 1.3 connections, you can disable the use of this protocol version. In the configuration file, change MaxVersion = TLSv1.3 to MaxVersion = TLSv1.2.

Enabling SSL 3.0, TLS 1.0, or TLS 1.1

TLS 1.1 and earlier protocol versions are disabled by default. If your host does not yet support TLS 1.2 or TLS 1.3, you may see errors related to TLS version not supported in Design Tool, the session server log, or model debug messages (.vmr file). To enable these protocols:

1. change MinVersion = TLSv1.2 to MinVersion = TLSv1.1, MinVersion = TLSv1, or MinVersion = SSLv3
2. uncomment the CipherString line by removing the initial # character
3. In this line, change :@SECLEVEL=1 to :@SECLEVEL=0.

Modifying the ciphers offered to the host

If you wish, you can control the ciphers offered in the handshake. The CipherString setting controls the ciphers used for TLS 1.2 and earlier; the Ciphersuites setting controls the ciphers used for TLS 1.3. Refer to the OpenSSL 3.0 documentation to see possible values for these settings.

Client Authentication

If the host requires client authentication from Host Integrator, your private key and client certificate chain must be stored in a PEM file.

If your file is named %VHI_ROOT%/securehost/certificate.pem add the following line to the hostssl section:

Certificate = ${ENV::VHI_ROOT}securehost/certificate.pem

The file must be in PEM format with the unencrypted private key and the certificate chain in chain order.

If your certificate and private key are in PFX format, you can convert it to OpenSSL PEM format using the OpenSSL command line utility in the %VHIROOT%/bin folder.

Changing the Linux trust store

The actual location of the trust store on Linux is not standard. On startup, the code will test for the presence of an ordered list of candidates. If a file is present, it will be used as the trust store.

1. /etc/ssl/certs/ca-certificates.crt
2. /etc/pki/tls/certs/ca-bundle.crt
3. /etc/ssl/ca-bundle.pem
4. /etc/pki/tls/cacert.pem
5. /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem
6. /etc/ssl/cert.pem

You can change the file used as the trust store by changing the VerifyCAfile setting.

Disabling Certificate Validation

For troubleshooting purposes, you can temporarily disable verification. In the hostssl section, remove the line VerifyMode = Peer to disable verification.

Permanently disabling verification poses a security risk and is not recommended.