I’ve seen this twice now so I’m documenting the fix so I don’t have to go hunting again.
Basically it starts as a typical “oh crap my certs expired” question on #freeipa or freeipa-users. Sadly the usual things don’t seem to help (go back in time).
The last time this happened there was the added twist that the renewal master was gone so we had to first reconfigure a replica to do the renewal (you do have more than one CA right? RIGHT?)
Anyway, he would persistently get the Peer certificate cannot be authenticated message. We tried:
- Confirming that ipaCert was the correct value in the IPA RA entry in LDAP
- The CA is up and running:
curl --cacert /etc/ipa/ca.crt -v https://`hostname`:8443/ca/ee/ca/getCertChain
- Ensuring that
certutil -L -d /etc/pki/pki-tomcat/alias-n 'caSigningCert cert-pki-ca' -a and cat /etc/ipa/ca.crt are the same cert
Then I remembered a fellow Red Hatter had reported a similar issue and discovered that the fix was to reset the NSS trust flags in the Apache NSS database (which certmonger uses).
# SSL_DIR=/etc/httpd/alias/ curl -v -o /dev/null --cacert /etc/ipa/ca.crt https://`hostname`:8443/ca/agent/ca/profileReview
You should get client certificate not found. If you don’t then try this:
# certutil -M -d /etc/httpd/alias -n 'EXAMPLE.COM IPA CA' -t ,,
# certutil -M -d /etc/httpd/alias -n 'EXAMPLE.COM IPA CA' -t CT,C,C
It had literally been years since I had to setup an AD test environment to do basic winsync testing. I found some scraggly notes and decided to transcribe them here for posterity. They were written for AD 2003 and things for 2008 are a bit different but I still found it fairly easy to figure out (in 2008 there is less need to go to the Start menu).
I don’t in fact remember what a lot of these notes do so don’t kill the messenger.
Start with an AD 2008 instance by following http://www.freeipa.org/page/Setting_up_Active_Directory_domain_for_testing_purposes
Once that is booted:
- Change the hostname
- My Computer -> right click -> Properties -> Computer Name -> Change = win2003
- Manage your Server
- Add or remove a role -> Next [Preliminary Steps]
- Custom -> Domain Controller
- Domain controller for a new domain
- Domain in a new forest
- Fill DNS name for new domain: example.com
- If conflict select Install and Configure DNS on this server
- Start -> Control Panel -> Add or Remove Programs
- Add/Remove Windows Components
- Certificate Services, yes to the question
- Enterprise root CA
- AD CA for the common name
- Accept other defaults
- Ok about IIS
- REBOOT (or wait a little while for certs to issue)
- Start -> Admin Tools -> Certificate Authority
- Certificate Authority -> AD CA -> Issued Certificates
- Select the cert, double click
- Certificate Path
- Select AD CA, view certificate
- Copy to file
- Base 64-encoded x509 (.cer)
- Install WinSCP
- Copy cert to IPA
Now on the IPA master the agreement can be created:
# ipa-replica-manage connect win2003.example.com –winsync –cacert=/home/rcrit/adca.cer -v –no-lookup –binddn ‘cn=administrator,cn=users,dc=example,dc=com’ –bindpw <AD pw> –passsync <something>
As I recall I tended to put the AD hostname into /etc/hosts (hence the –no-lookup).
There are times when you may want more information out of the IPA server logs. I’ve seen people suggest adding
debug = True to
/etc/ipa/default.conf. This is fine (and it works) but it enables debugging in both the client and the server which can be annoying for command-line users.
What I do instead is create
debug = True
The context that is set during initialization drives what configuration files are loaded so only the server will load this so the client remains quiet by default.
When the context is set during api.initialize it sets api.env.context. The original idea is this could drive different code paths depending on the context but in reality it hasn’t been used all that often. Being able to load context-specific configuration files is pretty neat though.
novajoin needs to monitor nova notifications to know when an instance is deleted so the host can be removed from IPA as well. I originally coded it to use the notifications topic but ceilometer also uses this topic so novajoin-notify was getting only a subset of the deletes.
The fix is very easy, add a new topic to the topics option in nova.conf, e.g.
This isn’t at all obvious from any documentation.
The nova v2 vendordata (dynamic vendordata plugins) came up at the 2016 OpenStack Summit in Barcelona. It was expected at the outset to be a fairly short meeting since the two proposals were seemingly straightfoward: add authentication and caching. It ended up using all 40 minutes but in the end wasn’t all that contentious.
It was decided that authentication would be sent in all cases to the configured vendordata plugins. There was some brief discussion about whether the user token would be included but as I took it, nova would use its own (or some pre-configured) credential. This will require some paste configuration changes in novajoin but authentication should otherwise be supported.
The metadata server will also cache responses. Exactly how long, what, etc is TBD. If I remember I’ll update this post with the gerrit review link once it comes out.
The idea is to commit to master than back port to Newton stable.
The contentious part was related to that user token I mentioned. Adam Young from Keystone wanted that token to be sent along, even if expired, so one could know the roles of the user that kicked things off. The problem of course is that the user is just a snapshot in time. Roles change. Users are deleted. Apparently some users completely hammer on the metadata service today, some as frequently as every few minutes. At some point things could break if that user went away.
I was ambivalent about it. Adam’s point was that it could be used for access control which is a good idea. I think that if the roles were cached instead of the user, that might make more sense. But even then people would complain that they revoked or added access and the user can/can’t do things. It’s a no-win I think. I kept my mouth shut in any case.
In the end this is good news for novajoin. I was quite uncomfortable having unauthenticated requests at all (e.g. metadata requests from an instance) so that’ll go away soon.
The caching will solve the problems I had bending over backwards with the IPA OTP. There could still be problems if the time to enroll the instance > nova metadata cache so I’ll probably leave in my “last update wins” code, but this does make things a bit more predictable and will certainly be faster.
I’ve made quite a lot more progress on migrating mod_nss to using NSS contexts, but I’m not quite done yet.
The server starts and handles some requests, so that’s something. It just freaks out a bit if there are multiple nicknames for the “same” virtual host (bug in SNI code) and it is generally unfriendly when it comes to nicknames.
When multiple NSS databases are initialized at the same time the nicknames become quite odd. The “first” database initialized, something probably not predictable in mod_nss, is a traditional database where references to the nickname without a token are handled.
Other databases use the slot in the nickname, e.g. NSS Certificate DB or in the case of mod_nss, internal. I have a huge hack in place now to try both iterations and it works when there are just two NSS databases and uniquely named nicknames. I think that I’ll need to detect duplicate nicks and blow up early otherwise things just don’t work as they should.
I spent an afternoon hacking on gencert to make it more generic so I can more easily generate multiple NSS certificate databases for testing. I started testing with 3 and like I said, it sorta works.
I’ve run into two difficult problems at the moment:
- The SNI code is based only on the vhost id so if there are multiple nicknames for a vhost (on different ports for example) then the wrong nickname could be returned
- Even if I have the right nickname looking it up using PK11_FindCertFromNickname() is failing to find it. I’ve tried using Server-Cert and internal:Server-Cert and nothing is found. I’m a bit baffled by this and unfortunately will have to dive into gdb. I say unfortunately because IIRC the PKCS#11 handling is a nightmare of callbacks.
Still, the progress has promise.
I need to double check on this but I’m pretty sure that NSS keeps a single trust table internally so my idea of being able to separate CA trust by vhost seems to be a pipe-dream for now.
There has been a long standing need to convert mod_nss from using a single NSS database to support NSS Contexts which will allow for much greater flexibility.
I started hacking on it between other work a couple of days ago and progress has been slow to say the least. It’s amazing the number of assumptions baked into the code that there is a single database. So far I’ve had to:
- Move the database config option from being treated only as a module config option to a proper Apache server context.
- Move a lot of initialization code around that verifies the NSS database filesystem read permissions and make it iterable
- Convert nss_pcache from validating the token password to just storing it. I now do a NSS_NoDB_Init() because there could be multiple sources and I don’t want to deal with that headache. This shifts the burden of identifying a bad password a bit later but it should be handled ok
I’ve run into another roadblock similar to the database config option being treated as global: the password dialog is also treated as a global. It really needs to be handled per-server context. And this is where I am.
Fortunately it has mostly been a matter of reorganizing some structures and fixing all uses of them, so it’s been more grunt work than anything else.
On the bright side I’ve got it passing the in-tree tests by specifying a single NSS database. Under the hood it’s using NSS contexts and the new nss_pcache backend.
Something I’ll need to look into is how to release this new code. I’m not 100% sure it isn’t going to blow up on people, depending on how they have things configured. This may also represent mod_nss 2.0, we’ll see. It won’t be much of a difference functionally, with the exception that different VS would be able to have different CA trust, but internally it’ll be quite different.
So stay tuned.