Certificates and private keys

X.509 certificates and their corresponding private keys are central to TLS. If you are involved in Enterprise Server administration, it is important that you understand at least the basics of certificates and private keys.

There are official standards for certificates and keys. The major ones are described here. Many of these (the RFCs and the CA/BF Baseline Requirements) are freely available online. ITU standards such as X.509 typically must be purchased from a standards organization, but summaries and discussions can generally be found online to answer most questions.

X.509
This is the ITU standard for the digital certificates used with TLS. The current version is 3, and all modern systems should use X.509v3 certificates, which provide a number of important features for security and interoperability. X.509 draws on a number of other standards, such as X.400 (which defines the distinguished name syntax used for subject and issuer names in certificates); X.500 (online directories, the precursor of LDAP); and ASN.1 (data representation and encoding). Regarding the last, you might see references to certificates in DER encoding; this stands for Distinguished Encoding Representation, a serialization format for ASN.1. Sometimes you might see a file described as being in "DER format". This typically means the file is the raw binary DER, as opposed to the Base64-encoded DER found in PEM-formatted certificate files.
PEM
The Privacy-Enhanced Mail standards were an early specification for encrypted email. While PEM as an email system was eclipsed by PGP and S/MIME, the PEM file and data formats (including Base64 encoding) are often used with certificates and private keys. PEM is defined in various RFCs, and the PEM file format in particular was most recently updated by RFC 7468.
PKCS
The Public Key Cryptography Standards, PKCS#1 through PKCS#15, were published by RSA Security. For users and administrators working with TLS, the PKCS standards most commonly encountered are PKCS#7, sometimes used as a certificate file format (typically with a .p7 or .p7b extension); PKCS#8, a format for private-key files (.p8 or .p8b); PKCS#10, used for Certificate Signing Requests to a CA, among other things; and PKCS#12, a file format for holding multiple pieces of information. PKCS#12 is the standardized version of Microsoft's PFX format, and most .pfx files are actually PKCS#12. (The .p12 extension is also used.) PKCS#12 files can hold multiple certificates and keys, and can be partly or entirely encrypted, so they are often used to carry both a certificate (or certificate chain) and its associated private key.
PKIX
The Public Key Infrastructure (X.509) specification, RFC 5280 (as updated by its errata and some succeeding RFCs), is the Internet standard for using X.509 certificates with TCP/IP protocols, including TLS. PKIX imposes a number of requirements on the contents and use of certificates. Some of these are noted below.
CA/BF Baseline Requirements
The CA / Browser Forum is an industry group of representatives from commercial CAs and browser makers who meet to discuss and specify additional requirements for the use of TLS on the public Internet. The Baseline Requirements (at version 1.7.3 as of this writing) documents their standard. In principle, the Baseline Requirements do not apply to organizational CAs and TLS used within an organization, or between private parties; but since browsers typically enforce the requirements, even in private use some of them might need to be observed at least for HTTPS. Below we note some of the consequences of the Baseline Requirements for certificates.

Certificate Authorities

Do not use the DemoCA certificate authority shipped with Enterprise Server in production. DemoCA is not supported for production use. Use an organizational or commercial CA.

For organizational CAs, Microsoft Windows Server includes a CA role which can be used to create suitable certificates and to distribute the required root and intermediate certificates to Windows client systems using Group Policy. This is a straightforward choice for organizations using Windows extensively, though configuring it to create suitable certificates takes some research. For Linux and UNIX, there are various open-source CA packages available.

Certificate validation types

When certificates are issued by a CA, the identity of the party requesting the certificate can be validated in one of three ways:

Organization Validation (OV)
An OV certificate is issued either within an organization, so (in theory) the request is authenticated using organizational channels; or it is issued by a commercial CA based on an official statement from the requesting organization that offers some proof of identity and authorization.
Domain Validation (DV)
A DV certificate is generated, usually through an automated process, for a specific fully-qualified domain name, and the request is validated by checking for certain DNS entries in the domain. The theory of DV certificates is that if someone controls DNS for a domain, they effectively own that domain and so it is appropriate to grant them DV certificates for systems in it. This is the approach used by Let's Encrypt.
Extended Validation (EV)
An EV certificate implies additional steps were taken to validate the certificate request. Typically these include additional paperwork and confirmation by staff at the issuing CA. EV certificates are also substantially more expensive, and are controversial in the industry, with some commentators suggesting they add little additional security. Browsers used to indicate to users when a site offered an EV certificate, but Chrome at least has stopped doing so.

The type of certificate validation does not matter to any Enterprise Server component. There is probably no significant benefit to using EV certificates with Enterprise Server.

Certificate properties

If an organization is generating its own certificates, it is important to ensure they follow best practices for their contents and options. Even with an external CA, the Certificate Signing Request (CSR) or other material provided to the CA should be sure to specify the appropriate parameters. Micro Focus recommends:

Names
Server certificates, and other certificates which identify computers by their network names — such as hostnames, fully-qualified domain names (FQDNs), and IP addresses — should list those names in Subject Alternative Name (SAN) extensions. Certificates which identify other entities such as people and applications should use a Subject Distinguished Name (Subject DN) with appropriate components.

At one time certificates for computers put the hostname or FQDN in the Common Name (CN) component of the Subject DN, but PKIX and CA/BF Baseline Requirements now require the use of SANs, and conforming implementations are not permitted to use the CN when validating a certificate they receive from a peer. The use of CN in a Subject or Issuer name is deprecated by CA/BF.

When generating a certificate or CSR for a server, specify a SAN for each name you might use to contact the server. Often that means a SAN for both the FQDN and bare-hostname forms of the server's canonical name, and if the server has aliases that clients might use, SANs for those (in both FQDN and bare-hostname forms) as well. If the system has a fixed IP address and clients can contact it using that address directly rather than resolving a hostname, you might want to have a SAN for the address as well.

For example, assume an enterprise server instance running on a system named corpserv, in the domain corpdom.com. For convenience, the organization has given this system the alias esserv. It has been permanently assigned the IPv4 address 10.2.5.18 and the IPv6 address fd0f:14d7:e62c:1234::0005:0012. For such a server we might want to request a certificate with the following Subject Alternative Names:
  • DNS:corpserv
  • DNS:corpserv.corpdom.com
  • DNS:esserv
  • DNS:esserv.corpdom.com
  • IP:10.2.5.18
  • IP:fd0f:14d7:e62c:1234::0005:0012
The idea is to create a certificate which adequately and uniquely identifies this computer, but accommodates all the names that a client might reasonably use in trying to connect to it using TLS.
Note: Some clients, such as the Chrome browser, now require a SAN when validating a server certificate.
Wildcards
Micro Focus recommends against the use of wildcard certificates for use with Enterprise Server, because of the risks associated with losing control of the private key for such a certificate.
Algorithms and key sizes
Each certificate has a signing algorithm and a public key. The signing algorithm must be a modern, strong algorithm; for example, signing algorithms using the MD5 hash are no longer allowed by many implementations. Keys should be adequately strong. The CA/BF requirements currently specify a minimum of 2048 bits for an RSA key, for example.
Validity dates
Every certificate contains two validity date-and-time fields: a not-valid-before and a not-valid-after. Currently the CA/BF requirements limit the total validity period for entity (server, client, user, and so forth) certificates to 398 days. Some browsers and other peers might enforce this. Currently, Chrome ignores this requirement for certificates which are issued by organizational CAs, but best practice is probably to issue certificates good for slightly more than one year, and schedule renewing them annually.
Serial number
Every certificate issued by a CA should have a unique serial number. CA/BF requires the serial number be positive and have at least 64 bits, which is why serial numbers for certificates issued by commercial CAs are very long (for example, 09:4a:51:9b:32:a5:b4:00:38:14:c5:ef:29:bf:8d:48 for the current certificate for www.microfocus.com). Sufficiently recent CA software should be able to generate serial numbers of this form. Some browsers and other peers may insist on serial numbers that meet this requirement.
Basic constraints
All entity certificates (server certificates, and other certificates which are not CA root or intermediate certificates) should have the Basic Constraints extension with the CA flag set to false. This is optional under the standards but good practice.
Key usage and extended key usage
Best practice is to issue each certificate for as narrow a use case as is feasible, and to set these appropriately for that use case. For example, a server certificate should have its Key Usage set to Digital Signature and Key Encipherment, and its Extended Key Usage to TLS Web Server Authentication.

Under the CA/BF rules, key usage is optional and extended key usage is required.

CRL distribution points and authority information access
These extensions deal with certificate revocation information. Certificate revocation is a fraught practice with numerous problems, and neither of the available mechanisms (Certificate Revocation Lists or CRLs, and the Online Certificate Status Protocol or OCSP) work very well, so many organizational CAs do not implement or make use of revocation, and many applications skip revocation checks. Currently no Enterprise Server components make revocation checks.

However, revocation still has a role in the industry, and the CA/BF have certain requirements regarding it.

CA/BF baseline requirements make the CRL Distribution Points extension optional, so it can be omitted.

CA/BF baseline requirements make the Authority Information Access extension mandatory, and require that it include an OCSP responder URL. They also note that the AIA extension should include a URL for retrieving the root certificate of the certificate chain. Currently it seems all browsers will ignore violations of this requirement, at least for organizational CAs, but that could change.

Authority key identifier (AKID) and subject key identifier (SKID)
The AKID and SKID extensions uniquely identify the public key of the certificate (SKID) and the certificate that signed it (AKID). They make it easier for an application which is validating a certificate to construct the certificate chain back to a trust anchor (typically a root certificate). CA/BF requires the AKID extension; best practice is to provide both.
Certificate policies extension
CA/BF requires the Certificate Policies extension. Micro Focus is unaware of any browsers or other clients which currently enforce this requirement.

Key hygiene

Key hygiene refers to good practices with private keys. Private keys are highly sensitive data, even when they are only used for testing purposes.

Consider the following scenario:

  • An organization creates a key pair for a test system, and a certificate for the system, so they can test with TLS.
  • For convenience, they use their organizational CA to issue the certificate. This way browsers within the organization (which have the appropriate trust anchors) will trust this certificate.
  • An attacker penetrates the internal network, obtains the certificate (from the server), and obtains the private key (because it was not adequately protected).
  • The attacker sets up a hostile server, and uses DNS cache poisoning or another technique to redirect traffic to the hostile server.
  • Now the attacker can use social engineering or other techniques to trick employees into connecting to the hostile server and submitting sensitive data such as credentials. Because the certificate is signed by the organizational CA, the browsers will not show any warnings.

Currently, Enterprise Server only supports storing private keys in key files, so protecting the private key is a matter of protecting the contents of those files and the files themselves.

Micro Focus recommends using an encrypted key file. Most key file formats, including PEM, PKCS#8, and PKCS#12 support encrypting the key data. Encrypting the file means the attacker must guess (including by brute force) the passphrase needed to unlock the key, or get the passphrase from another source. That increases the attacker's work factor.

Set permissions on the key file restrictively. Ideally, as discussed in the topic on filesystem permissions, Enterprise Server should run under a dedicated user account, and only that account should have read access to the private-key file. This is particularly important if the key is not encrypted, but a good practice in any case.

Restrict access to the private key to only those employees who have a business requirement for it. Do not share it with Micro Focus. Micro Focus Support, Professional Services, or Pre-Sales might ask for copies of your certificates; providing those is safe. Do not include the private keys. Micro Focus might occasionally ask for information about the private key, such as the modulus of an RSA key, in order to confirm that a private key matches the public key in a certificate; providing that is safe. Micro Focus will not ask for the private key itself.

Private keys corresponding to CA root and intermediate certificates are particularly sensitive. Industry best practice is to never store the private key for a CA root on non-removable media; keep it only on removable media or in a Hardware Security Module (HSM) and attach it only in order to generate new root or intermediate certificates.

Managing keys and supplying passphrases

If a private key is compromised, all certificates which use that key pair must immediately be revoked. A new key pair should be generated and the certificates re-issued and replaced.

A given key pair can be used for more than one certificate. Each customer will have to determine whether they prefer to manage a larger set of keys, or risk having to revoke and re-issue more certificates if a key is compromised.

In the previous section, Micro Focus recommended encrypting keys in key files. The disadvantage of encrypting the key is that the passphrase to unlock the key must be supplied when the key is first used by a process. For Enterprise Server that typically means supplying it at region startup. MFDS, ESCWA, and other cases are discussed below.

Note: Currently Enterprise Server also requests a certificate passphrase. In practice certificates are almost never encrypted, and so this passphrase is usually empty, but it needs to be supplied. Certificate passphrases can be provided in the same manner as key passphrases. See your product Help for more information.

There are various ways of providing the key passphrase:

  • Enter it manually after the enterprise server region has started. When a TLS-enabled listener is asked to start but the keyfile passphrase is not supplied in any other manner, the listener is put into a "start pending" state. Then a web form, linked to by ESCWA and MFDS, can be used to enter the passphrases and complete startup. This avoids having the passphrase stored for an attacker to find, but means that server startup requires this manual step.
  • Put it in the mf-server.dat file. In this case it is important to set restrictive file permissions on mf-server.dat. Beginning with release 8.0, the mf-server.dat file can also be used to specify passphrases contained in the Enterprise Server Vault facility.
    Note: You can use the Micro Focus Vault Facility to store a secret for the certificate and keyfile pass phrases. This can be specified in the mf-server.dat file and takes the following form: 
    mfsecret:configuration-name:secret-path

    or: 

    mfsecret::secret-path

    or: 

    mfsecret:secret-path
  • Use the ESCERTPAS user exit to supply the passphrase. ESCERTPAS is documented for supplying passphrases for the CICS Web Interface feature, but it can also be used for regular Enterprise Server listeners. An advantage of ESCERTPAS is that the exit can, in principle, do anything to get the passphrase. The sample exit program just hard-codes passphrases, but an implementation could retrieve them using some more-secure mechanism.

In future releases, Enterprise Server will likely support other passphrase and key storage options, such as the increased use of the Enterprise Server Vault facility, the Windows certificate and key store, and HSMs using the OpenSSL PKCS#11 interface.

For MFDS and ESCWA, the key passphrase is specified in the configuration or kept in the Vault Facility.

For clients using MFCC, key passphrases can be supplied programmatically, but few clients support this option. Instead, keys can be specified in mf-client.dat files, either directly or using the Vault.

Clients and servers that use CCI directly, such as Fileshare and the Mainframe Access client, can supply passphrases as part of the CCI machinename string.

See your product Help for more information.

Parent topic: Using and hardening TLS