All posts by rcritten

certmonger D-Bus introspection

I’m looking to do some certificate work related to certmonger and was thinking D-Bus would be a good way to get the data (freeIPA does something similar). The Using D-Bus Introspection blog post was key for me to figure out what certmonger could provide (without digging too much into the code).

I ended up running:

dbus-send --system --dest=org.fedorahosted.certmonger \
--type=method_call --print-reply \
/org/fedorahosted/certmonger \
org.freedesktop.DBus.Introspectable.Introspect

This provided me the list of interfaces I needed. First I started with getting the current requests:

dbus-send --system --dest=org.fedorahosted.certmonger \
--type=method_call --print-reply \
/org/fedorahosted/certmonger \
org.fedorahosted.certmonger.get_requests

Then you can pick or iterate through the requests to get the information you want. Here is how to get the serial number:

dbus-send --system --dest=org.fedorahosted.certmonger \
--type=method_call --print-reply \
/org/fedorahosted/certmonger/requests/Request1 \
org.freedesktop.DBus.Properties.Get \
string:org.fedorahosted.certmonger.request string:serial

You can find a list of possible values in src/tdbus.h

Advertisements

Peer certificate cannot be authenticated with given CA certificates

I’ve seen this twice now so I’m documenting the fix so I don’t have to go hunting again.

Basically it starts as a typical “oh crap my certs expired” question on #freeipa or freeipa-users. Sadly the usual things don’t seem to help (go back in time).

The last time this happened there was the added twist that the renewal master was gone so we had to first reconfigure a replica to do the renewal (you do have more than one CA right? RIGHT?)

Anyway, he would persistently get the Peer certificate cannot be authenticated message. We tried:

  • Confirming that ipaCert was the correct value in the IPA RA entry in LDAP
  • The CA is up and running: curl --cacert /etc/ipa/ca.crt -v https://`hostname`:8443/ca/ee/ca/getCertChain
  • Ensuring that certutil -L -d /etc/pki/pki-tomcat/alias-n 'caSigningCert cert-pki-ca' -a and cat /etc/ipa/ca.crt are the same cert

Then I remembered a fellow Red Hatter had reported a similar issue and discovered that the fix was to reset the NSS trust flags in the Apache NSS database (which certmonger uses).

# SSL_DIR=/etc/httpd/alias/ curl -v -o /dev/null --cacert /etc/ipa/ca.crt https://`hostname`:8443/ca/agent/ca/profileReview

You should get client certificate not found. If you don’t then try this:

# certutil -M -d /etc/httpd/alias -n 'EXAMPLE.COM IPA CA' -t ,,
# certutil -M -d /etc/httpd/alias -n 'EXAMPLE.COM IPA CA' -t CT,C,C

See https://lists.fedorahosted.org/archives/list/freeipa-users@lists.fedorahosted.org/thread/XSMWWPJU2VRUIGE6SRAHYAJF7BYBCNOE/

Setting up AD for winsync testing

It had literally been years since I had to setup an AD test environment to do basic winsync testing. I found some scraggly notes and decided to transcribe them here for posterity. They were written for AD 2003 and things for 2008 are a bit different but I still found it fairly easy to figure out (in 2008 there is less need to go to the Start menu).

I don’t in fact remember what a lot of these notes do so don’t kill the messenger.

Start with an AD 2008 instance by following http://www.freeipa.org/page/Setting_up_Active_Directory_domain_for_testing_purposes

Once that is booted:

  1. Change the hostname
  2. My Computer -> right click -> Properties -> Computer Name -> Change = win2003
  3. REBOOT
  4. Manage your Server
    1. Add or remove a role -> Next [Preliminary Steps]
    2. Custom -> Domain Controller
    3. Domain controller for a new domain
    4. Domain in a new forest
    5. Fill DNS name for new domain: example.com
  5. If conflict select Install and Configure DNS on this server
  6. REBOOT
  7. Start -> Control Panel -> Add or Remove Programs
    1. Add/Remove Windows Components
    2. Certificate Services, yes to the question
    3. Next
    4. Enterprise root CA
    5. AD CA for the common name
    6. Accept other defaults
    7. Ok about IIS
  8. REBOOT (or wait a little while for certs to issue)
  9. Start -> Admin Tools -> Certificate Authority
    1. Certificate Authority -> AD CA -> Issued Certificates
    2. Select the cert, double click
    3. Certificate Path
    4. Select AD CA, view certificate
    5. Details
    6. Copy to file
    7. Base 64-encoded x509 (.cer)
  10. Install WinSCP
  11. Copy cert to IPA

Now on the IPA master the agreement can be created:

# ipa-replica-manage connect win2003.example.com –winsync –cacert=/home/rcrit/adca.cer -v –no-lookup –binddn ‘cn=administrator,cn=users,dc=example,dc=com’ –bindpw <AD pw> –passsync <something>

As I recall I tended to put the AD hostname into /etc/hosts (hence the –no-lookup).

IPA configuration files and context

There are times when you may want more information out of the IPA server logs. I’ve seen people suggest adding debug = True to /etc/ipa/default.conf. This is fine (and it works) but it enables debugging in both the client and the server which can be annoying for command-line users.

What I do instead is create /etc/ipa/server.conf containing:

[global]
debug = True

The context that is set during initialization drives what configuration files are loaded so only the server will load this so the client remains quiet by default.

When the context is set during api.initialize it sets api.env.context. The original idea is this could drive different code paths depending on the context but in reality it hasn’t been used all that often. Being able to load context-specific configuration files is pretty neat though.

oslo messaging and notifications

novajoin needs to monitor nova notifications to know when an instance is deleted so the host can be removed from IPA as well. I originally coded it to use the notifications topic but ceilometer also uses this topic so novajoin-notify was getting only a subset of the deletes.

The fix is very easy, add a new topic to the topics option in nova.conf, e.g.

topics=notifications,novajoin

This isn’t at all obvious from any documentation.

novajoin and the 2016 OpenStack Barcelona Summit

The nova v2 vendordata (dynamic vendordata plugins) came up at the 2016 OpenStack Summit in Barcelona. It was expected at the outset to be a fairly short meeting since the two proposals were seemingly straightfoward: add authentication and caching. It ended up using all 40 minutes but in the end wasn’t all that contentious.

It was decided that authentication would be sent in all cases to the configured vendordata plugins. There was some brief discussion about whether the user token would be included but as I took it, nova would use its own (or some pre-configured) credential. This will require some paste configuration changes in novajoin but authentication should otherwise be supported.

The metadata server will also cache responses. Exactly how long, what, etc is TBD. If I remember I’ll update this post with the gerrit review link once it comes out.

The idea is to commit to master than back port to Newton stable.

The contentious part was related to that user token I mentioned. Adam Young from Keystone wanted that token to be sent along, even if expired, so one could know the roles of the user that kicked things off. The problem of course is that the user is just a snapshot in time. Roles change. Users are deleted. Apparently some users completely hammer on the metadata service today, some as frequently as every few minutes. At some point things could break if that user went away.

I was ambivalent about it. Adam’s point was that it could be used for access control which is a good idea. I think that if the roles were cached instead of the user, that might make more sense. But even then people would complain that they revoked or added access and the user can/can’t do things. It’s a no-win I think. I kept my mouth shut in any case.

In the end this is good news for novajoin. I was quite uncomfortable having unauthenticated requests at all (e.g. metadata requests from an instance) so that’ll go away soon.

The caching will solve the problems I had bending over backwards with the IPA OTP. There could still be problems if the time to enroll the instance > nova metadata cache so I’ll probably leave in my “last update wins” code, but this does make things a bit more predictable and will certainly be faster.

mod_nss and NSS Context part 2

I’ve made quite a lot more progress on migrating mod_nss to using NSS contexts, but I’m not quite done yet.

The server starts and handles some requests, so that’s something. It just freaks out a bit if there are multiple nicknames for the “same” virtual host (bug in SNI code) and it is generally unfriendly when it comes to nicknames.

When multiple NSS databases are initialized at the same time the nicknames become quite odd. The “first” database initialized, something probably not predictable in mod_nss, is a traditional database where references to the nickname without a token are handled.

Other databases use the slot in the nickname, e.g. NSS Certificate DB or in the case of mod_nss, internal. I have a huge hack in place now to try both iterations and it works when there are just two NSS databases and uniquely named nicknames. I think that I’ll need to detect duplicate nicks and blow up early otherwise things just don’t work as they should.

I spent an afternoon hacking on gencert to make it more generic so I can more easily generate multiple NSS certificate databases for testing. I started testing with 3 and like I said, it sorta works.

I’ve run into two difficult problems at the moment:

  1. The SNI code is based only on the vhost id so if there are multiple nicknames for a vhost (on different ports for example) then the wrong nickname could be returned
  2. Even if I have the right nickname looking it up using PK11_FindCertFromNickname() is failing to find it. I’ve tried using Server-Cert and internal:Server-Cert and nothing is found. I’m a bit baffled by this and unfortunately will have to dive into gdb. I say unfortunately because IIRC the PKCS#11 handling is a nightmare of callbacks.

Still, the progress has promise.

I need to double check on this but I’m pretty sure that NSS keeps a single trust table internally so my idea of being able to separate CA trust by vhost seems to be a pipe-dream for now.