Getting the cert and chain in one file in certmonger

Some servers want the server cert and CA chain all in one file. There isn’t an option in certmonger to do this but it can be completed using the post-save command. This is a command specified in the request that executes after a certificate has been issued and saved to disk.

The option does not accept bash syntax. It executes a single command. Generally speaking for complex operations your best bet is to put it into a separate bash script that is executed, which we’ll do here.

I created /usr/local/bin/catcerts.sh with the contents:

#!/bin/bash
#
# concatenate a server cert and the chain into a single file

cert=$1
chain=$2
target=$3

cat $cert $chain > $target

IMPORTANT: Add your own error checking.

Use certmonger to request a cert with this as the command:

ipa-getcert request -f /etc/pki/tls/certs/test.pem \
-k /etc/pki/tls/private/test.key \
-C "/usr/local/bin/catcerts.sh /etc/pki/tls/certs/test.pem /etc/ipa/ca.crt /etc/pki/tls/certs/whole.pem"

This is an example on an IPA-enrolled machine where the chain already exists in /etc/ipa/ca.crt. If you need the chain as well you can add -F /etc/pki/tls/certs/chain.pem and use that in the concatenation.

Is my IPA install ok?

This is a common and complex question.

IPA is not a single daemon. It is a collection of services configured to work together. It provides a great deal of customization, and therefore a lot of room for mis-configuration post-installation. Some changes can be made that won’t be generate immediately visible problems which can make diagnosis difficult and it is particularly difficult when a new administrator takes over a running system and lacks the background of the installation.

IPA puts a great deal of effort prior to installation to verify that a system is properly configured and ready to accept an IPA master installation. What it is missing is a way post-installation to ensure that the system, or a set of masters, is still running as expected.

One method to check this is the ipa-consistency tool written by Peter Pakos. It counts a number of different IPA entry types per master and alerts if the counts differ. It is a pretty useful tool to look for replication conflicts and to generally see if replication is working as expected.

The IPA team is working on an additional tool, freeipa-healthcheck, to check on the types of configuration problems we’ve seen come up again and again.

You might ask why IPA itself can’t do these checks automatically  to prevent mis-configuration. It’s a fair question, and it goes back to the architecture where IPA is not a single daemon. The primary system authentication methods in 2007 when IPA started, and largely true today, were LDAP, Kerberos, NIS and flat files. The architecture of IPA was to leverage existing services together and wrap it in a usable user-interface, hence 389-ds as the LDAP store, MIT Kerberos as the authentication provider and a plugin service to emulate the important parts of NIS. Bolt on a CA and DNS and there are quite a number of independent moving parts.

Will these checks ever be integrated directly into IPA? Probably, but it is safer to run them outside for now while we ensure both coverage and that we haven’t been over-eager in some areas and provide false positives. We would really like to run this automatically prior to upgrades to warn a user that they have problems and that upgrading is likely to be problematic.

So how can I ensure I haven’t done something to break my installation? Let’s start with the kinds of things that commonly go wrong, and why:

  1. Certificates don’t renew. IPA uses certmonger to automatically renew certificates. This works fine as long as one IPA master is defined as the renewal master but many times we’ve done this master removed. This leaves no master to do the renewal and things go badly from there.
  2. The certmonger certificate tracking was removed or modified.
  3. Filesystems running out of space. Databases don’t like this.
  4. File permissions. IPA writes to a ton of files across the filesystem. Sometimes, in this interest of security, these are tweaked in ways that prevent IPA services from having access to files it needs.
  5. Replication. Network hickups, misconfiguration, unfortunate timing. A lot can go into creating replication issues. Peter’s tool is quite good at rooting these out.
  6. A master has no DNA range so therefore can’t create new entries as a result of removing the one master that had the ranges.

freeipa-healthcheck is intended to look at a single running master to decide if it is configured as expected, and report when it finds a possible inconsistency.  The goals of the project were:

  1. Provide some basic level of assurance that the system is configured properly.
  2. Always provide an answer for every question so you know that everything has checked (the downside is a firehose of output).
  3. Machine readable output.
  4. In case of anything questionable, warn. This may generate false positives but better to ignore a non-issue than to miss a real one.
  5. Look only on one master at a time. We don’t want to require connectivity between all masters. This means that some future tool is going to need to take some of the data and do further analysis. DNA is an example of this. It is perfectly fine for a master to not have any DNA ranges configured, but the whole installation not having one is not good. freeipa-healthcheck today only reports what the current range is on a master.

How can you try it? It is shipping in Fedora 29+ and will ship in future versions of RHEL 8. It is a pretty simple command-line tool:

# ipa-healthcheck

This will execute all known checks and write JSON to stdout with an entry for each specific check and the outcome. Each check assigns a severity to outcome: SUCCESS, WARNING, ERROR or CRITICAL. For results that are not SUCCESS then additional details are provided which are hopefully enough to point a user in the right direction to address it.

For a contrived error, let’s say I messed up the certmonger tracking of the KDC cert. It would produce the following output:

[
  {
    "source": "ipahealthcheck.ipa.certs", 
    "kw": {
      "msg": "Unable to open cert file '/var/kerberos/krb5kdc/kdc2.crt': [Errno 2] No such file or directory: '/var/kerberos/krb5kdc/kdc2.crt'", 
      "certfile": "/var/kerberos/krb5kdc/kdc2.crt", 
      "key": "20191018192938", 
      "error": "[Errno 2] No such file or directory: '/var/kerberos/krb5kdc/kdc2.crt'"
    }, 
    "uuid": "9894e56a-83cc-42e6-9a77-b9066924fe73", 
    "duration": "0.296820", 
    "when": "20191021201731Z", 
    "check": "IPACertfileExpirationCheck", 
    "result": "ERROR"
  }, 
  {
    "source": "ipahealthcheck.ipa.certs", 
    "kw": {
      "msg": "Missing tracking for ca-name=None, cert-file=/var/kerberos/krb5kdc/kdc.crt, cert-postsave-command=/usr/libexec/ipa/certmonger/renew_kdc_cert, key-file=/var/kerberos/krb5kdc/kdc.key", 
      "key": "ca-name=None, cert-file=/var/kerberos/krb5kdc/kdc.crt, cert-postsave-command=/usr/libexec/ipa/certmonger/renew_kdc_cert, key-file=/var/kerberos/krb5kdc/kdc.key"
    }, 
    "uuid": "2fd7e3f4-d966-4a89-ae5f-c0071cacf33e", 
    "duration": "0.470736", 
    "when": "20191021201731Z", 
    "check": "IPACertTracking", 
    "result": "ERROR"
  }, 
  {
    "source": "ipahealthcheck.ipa.certs", 
    "kw": {
      "msg": "Unknown certmonger id 20191018192938", 
      "key": "20191018192938"
    }, 
    "uuid": "f8ab75c5-1055-4565-9973-f4e9b0a662d8", 
    "duration": "0.470772", 
    "when": "20191021201731Z", 
    "check": "IPACertTracking", 
    "result": "WARNING"
  }

]

Three separate errors for the same single issue? Yes, there is some overlap. This breaks down to:

  1. The first error is warning that the certificate file pointed to in the tracking doesn’t exist so it can’t be verified as not-expired.
  2. The second that the expected tracking for the KDC is not setup properly which could mean that renewal won’t work properly.
  3. Which means it is effectively an unknown tracking which may or may not be fine, but the tool defaults to warning when it doesn’t know for sure. This is often test certs or aborted efforts to correct tracking.

Remember that I said that every check reports in? This means that by default the output is rather voluminous. To narrow things down you can look for only output that isn’t at the SUCCESS level:

# ipa-healthcheck --failures-only

Or you can log to a file and re-parse that:

# ipa-healthcheck --output-file /var/log/ipa/healthcheck/healthcheck.log# ipa-healthcheck --input-file /var/log/ipa/healthcheck/healthcheck.log --failures-only

The hope is that this output can be incorporated into system tracking systems like Zabbix so stability over time can be easily tracked. It is probably not super useful to run healthcheck more than once a day but there is no harm in it. It only writes to stdout or the provided log file. It makes no other changes.

You can get a basic list of checks performed with:

# ipa-healthcheck –list-sources

Hopefully the names are descriptive enough to tell what it is looking for. The upsteram git repo README.md contains a fuller description of each one.

How do I revert back to using IPA-issued Web & LDAP certs?

For IPA v4.6.x.

So you have an IPA installation with a CA and you decided you don’t want your users to have to install the IPA CA certificate(s) so instead you want to use certificates for the Web and LDAP using some known 3rd party issuer. Sure, that works fine. You’d do something like:

Install the 3rd party CA chain and update your IPA master:

# ipa-cacert-manage install /path/to/root.pem -t CT,,
# ipa-cacert-manage install intermediate.cert.pem -t CT,,
# ipa-certupdate

Install the 3rd-party provided server certificate. In this case I have it as two separate files, the public cert and the private key.

# ipa-server-certinstall --dirman-password password -w -d --pin '' server.cert.pem server.cert.key root.pem \
intermediate.cert.pem

Great. IPA is working fine and my users don’t need to import the IPA CA.

Two years later…

My 3rd party certs are expiring soon and I don’t want to pay for new ones and I want to switch back to using IPA-issued certificates. We can use certmonger for that. This assumes that your CA is still up and functioning properly.

I’d start by backing up the two NSS databases. It is safest to do this offline (ipactl stop). You need to copy *.db from /etc/dirsrv/slapd-EXAMPLE-TEST and /etc/httpd/alias someplace safe, then restart the world (ipactl start).

First the web server:

# ipa-getcert request -d /etc/httpd/alias -n Server-Cert -K HTTP/`hostname` -D `hostname` -C /usr/libexec/ipa/certmonger/restart_httpd -p /etc/httpd/alias/pwdfile.txt

Edit /etc/httpd/conf.d/nss.conf and replace the value of NSSNickname with Server-Cert.

Wait a bit to be sure the cert is issued. You can run this to see the status:

# ipa-getcert list -d /etc/httpd/alias -n Server-Cert

Now the LDAP server:

# ipa-getcert request -d /etc/dirsrv/slapd-EXAMPLE-TEST -n Server-Cert -D `hostname` -K ldap/`hostname` -C "/usr/libexec/ipa/certmonger/restart_dirsrv EXAMPLE-TEST" -p /etc/dirsrv/slapd-EXAMPLE-TEST/pwdfile.txt

Similarly wait for it to be issued. To track the status:

# ipa-getcert list -d /etc/dirsrv/slapd-EXAMPLE-TEST -n Server-Cert

Once it is issued run:

# ipactl stop

Now edit /etc/dirsrv/slapd-EXAMPLE-TEST/dse.ldif. We could do this while the server is online but we need to restart anyway and your favorite editor is easier than ldapmodify. Replace the value of nsSSLPersonalitySSL with Server-Cert

Now restart the world:

# ipactl start

Connect to each port if you want to confirm that the certificate and chain are correct, e.g.

# openssl s_client -host `hostname` -port 443
CONNECTED(00000003)
depth=1 O = EXAMPLE.TEST, CN = Certificate Authority
verify return:1
depth=0 O = EXAMPLE.TEST, CN = ipa.example.test
verify return:1
---
Certificate chain
0 s:/O=EXAMPLE.TEST/CN=ipa.example.test
i:/O=EXAMPLE.TEST/CN=Certificate Authority
1 s:/O=EXAMPLE.TEST/CN=Certificate Authority
i:/O=EXAMPLE.TEST/CN=Certificate Authority
---
...

Setting up a Mac (OSX) as an IPA client

I periodically see people trying to setup a Mac running OSX as an IPA client. I don’t have one myself so can’t really provide assistance.

There is this guide which seems to be pretty thorough, https://linuxguideandhints.com/centos/freeipa.html#mac-clients

This upstream ticket also has some information on setting up a client, though it isn’t always directly related to simply configuring a client, https://pagure.io/freeipa/issue/4813

So I record this here so I know where to look later 🙂

How do I get a certificate for my web site with IPA?

That’s a bit of a loaded question that begs additional questions:

  1. Is the web server enrolled as an IPA client?
  2. What format does the private key and certificate need to be in? (OpenSSL-style PEM, NSS, other?)

If the answer to question 1 is YES then you can do this on that client machine (to be clear, this first step can be done anywhere or in the UI):

$ kinit admin$ ipa service-add HTTP/webserver.example.com

You can use certmonger to request and manage the certificate which includes renewals (bonus!).

If you are using NSS and let’s say mod_nss you’d do something like after creating the database and/or putting the password into /etc/httpd/alias/pwdfile.txt:

# ipa-getcert request -K HTTP/webserver.example.com -d /etc/httpd/alias -n MyWebCert -p /etc/httpd/alias/pwdfile.txt -D webserver.example.com

Let’s break down what these options mean:

  • -K is the Kerberos principal to store the certificate with. You do NOT need to get a keytab for this service
  • -d the NSS database directory. You can use whatever you want but be sure the service has read access and SELinux permission access to it.
  • -n the NSS nickname. This is just a shortcut name to your cert, use what you want.
  • -p the path to the pin/password for the NSS database
  • -D creates a DNS SAN for the hostname webserver.example.com. This is current best practice.

You’ll also need to add the IPA cert chain to the NSS database using certutil.

If you are using OpenSSL and say mod_ssl you’d do something like:

# ipa-getcert request -K HTTP/webserver.example.com -k /etc/pki/tls/private/httpd.key -f /etc/pki/tls/certs/httpd.pem -D webserver.example.com

Similar options as above but instad -f -d and -n:

  • -k path to the key file
  • -f path to the certificate file

To check on the status of your new request you can run:

# ipa-getcert list -n <numeric id that was spit out before>

It should be in status MONITORING.

If the answer to #1 is NO then you have two options: use certmonger on a different machine to generate the key and certificate and transfer them to the target or generate a CSR manually.

For the first case, using certmonger on a different machine, the steps are similar to the YES case.

Create a host and service for the web server:

$ kinit admin$ ipa host-add webserver.example.com$ ipa service-add HTTP/webserver.example.com

Now we need to grant the rights to the current machine to get certificates for the HTTP service of webserver.example.com

$ ipa service-add-host --hosts <your current machine FQDN> HTTP/webserver.example.com

Now run the appropriate ipa-getcert command above to match the format you need and check the status in a similar way.

Once it’s done you need to transfer the cert and key to the webserver.

Finally, if you want to get certificates on an un-enrolled system the basic steps are:

  1. Create a host entry and service as above
  2. Generate a CSR, see https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/7/html/linux_domain_identity_authentication_and_policy_guide/certificates#requesting-cert-certutil (or the next section)
  3. Submit that CSR per the above docs

If your webserver is not registered in DNS then you can use the –force option to host-add and service-add to force their creation.

This should pretty generically apply to all versions of IPA v4+, and probably to v3 as well.

 

Batch adding users

Doing bulk IPA operations from the command-line can be inefficient because each command requires a round trip. So a loop like this can be rather slow:

for line in $(cat /etc/passwd); do
        IFS=' '
        username=$(echo $line|cut -f1 -d:)
        password=$(echo $line|cut -f2 -d:)
        uid=$(echo $line|cut -f3 -d:)  
        gid=$(echo $line|cut -f4 -d:)
        ...
        ipa user-add $username --first=NIS --last=USER --password --gidnumber=$gid --uid=$uid --gecos=$gecos --homedir=$homedir --shell=$shell --setattr userpassword={crypt}$password
done

There is a round trip for every user.

The obvious way to improve this is to reduce the number of round trips by using the IPA batch command. Here is the skeleton of a program to read /etc/passwd. It lacks a whole ton of error checking and may be filled with errors but it should illustrate how the batch command works.

This will batch the creation of 50 users at a time.

from ipalib import api
from ipalib import errors
import sys


def add_batch_operation(command, *args, **kw):
    batch_args.append({
        "method": command,
        "params": [args, kw],
    })


def flush_batch_operation():
    if not batch_args:
        return None

    kw = {}

    try:
        return api.Command['batch'](*batch_args, **kw)
    except errors.CCacheError as e:
        print(e)
        sys.exit(1)


api.bootstrap(context='batch')
api.finalize()
api.Backend.rpcclient.connect()

lineno = 0
batch_args = 0
count = 0
batch_args = list()
with open("/etc/passwd", "r") as passwd:
    for line in passwd:
        lineno += 1
        try:
            (login, password, uid, gid, gecos, homedir, shell) = \
                line.strip().split(':')
        except ValueError as ve:
            print("Malformed line %d: %s" % (lineno, ve))

        if gecos:
            try:
                first, last = gecos.split(' ', 1)
            except ValueError:
                print("Unable to parse gecos line %d" % lineno)
        else:
            print("Missing gecos line %d" % lineno)

        params = [login]
        kw = {
            'givenname': first,
            'sn': last,
            'cn': gecos,
            'userpassword': '{crypt}' + password,
            'gecos': gecos,
            'homedirectory': homedir,
            'loginshell': shell,
        }

        add_batch_operation('user_add', *params, **kw)
        count += 1

        if count % 50 == 0:
            print("%d entries" % count)
            results = flush_batch_operation()
            for result in results.get('results'):
                if result.get('error') != None:
                    print(result.get('error'))
            batch_args = list()

flush_batch_operation()

certmonger CA subsystem renewal

The CA subsystem certificate (OCSP, Audit, etc) are renewed directly against dogtag rather than being processed through IPA like the Apache and 389-ds server certificates are.

certmonger does the renewa by issuing a request like this:

GET /ca/ee/ca/profileSubmit?profileId=caServerCert&serial_num=5&renewal=true&xml=true&requestor_name=IPA

The serial number value comes from the current certificate being tracked by certmonger. Dogtag will generate its own CSR based on the template values currently in LDAP, cn=5,ou=ca,ou=requests,o=ipaca

Frustrated rantings of a developer