All posts by rcritten

novajoin and the 2016 OpenStack Barcelona Summit

The nova v2 vendordata (dynamic vendordata plugins) came up at the 2016 OpenStack Summit in Barcelona. It was expected at the outset to be a fairly short meeting since the two proposals were seemingly straightfoward: add authentication and caching. It ended up using all 40 minutes but in the end wasn’t all that contentious.

It was decided that authentication would be sent in all cases to the configured vendordata plugins. There was some brief discussion about whether the user token would be included but as I took it, nova would use its own (or some pre-configured) credential. This will require some paste configuration changes in novajoin but authentication should otherwise be supported.

The metadata server will also cache responses. Exactly how long, what, etc is TBD. If I remember I’ll update this post with the gerrit review link once it comes out.

The idea is to commit to master than back port to Newton stable.

The contentious part was related to that user token I mentioned. Adam Young from Keystone wanted that token to be sent along, even if expired, so one could know the roles of the user that kicked things off. The problem of course is that the user is just a snapshot in time. Roles change. Users are deleted. Apparently some users completely hammer on the metadata service today, some as frequently as every few minutes. At some point things could break if that user went away.

I was ambivalent about it. Adam’s point was that it could be used for access control which is a good idea. I think that if the roles were cached instead of the user, that might make more sense. But even then people would complain that they revoked or added access and the user can/can’t do things. It’s a no-win I think. I kept my mouth shut in any case.

In the end this is good news for novajoin. I was quite uncomfortable having unauthenticated requests at all (e.g. metadata requests from an instance) so that’ll go away soon.

The caching will solve the problems I had bending over backwards with the IPA OTP. There could still be problems if the time to enroll the instance > nova metadata cache so I’ll probably leave in my “last update wins” code, but this does make things a bit more predictable and will certainly be faster.


mod_nss and NSS Context part 2

I’ve made quite a lot more progress on migrating mod_nss to using NSS contexts, but I’m not quite done yet.

The server starts and handles some requests, so that’s something. It just freaks out a bit if there are multiple nicknames for the “same” virtual host (bug in SNI code) and it is generally unfriendly when it comes to nicknames.

When multiple NSS databases are initialized at the same time the nicknames become quite odd. The “first” database initialized, something probably not predictable in mod_nss, is a traditional database where references to the nickname without a token are handled.

Other databases use the slot in the nickname, e.g. NSS Certificate DB or in the case of mod_nss, internal. I have a huge hack in place now to try both iterations and it works when there are just two NSS databases and uniquely named nicknames. I think that I’ll need to detect duplicate nicks and blow up early otherwise things just don’t work as they should.

I spent an afternoon hacking on gencert to make it more generic so I can more easily generate multiple NSS certificate databases for testing. I started testing with 3 and like I said, it sorta works.

I’ve run into two difficult problems at the moment:

  1. The SNI code is based only on the vhost id so if there are multiple nicknames for a vhost (on different ports for example) then the wrong nickname could be returned
  2. Even if I have the right nickname looking it up using PK11_FindCertFromNickname() is failing to find it. I’ve tried using Server-Cert and internal:Server-Cert and nothing is found. I’m a bit baffled by this and unfortunately will have to dive into gdb. I say unfortunately because IIRC the PKCS#11 handling is a nightmare of callbacks.

Still, the progress has promise.

I need to double check on this but I’m pretty sure that NSS keeps a single trust table internally so my idea of being able to separate CA trust by vhost seems to be a pipe-dream for now.

mod_nss and NSS Contexts

There has been a long standing need to convert mod_nss from using a single NSS database to support NSS Contexts which will allow for much greater flexibility.

I started hacking on it between other work a couple of days ago and progress has been slow to say the least. It’s amazing the number of assumptions baked into the code that there is a single database. So far I’ve had to:

  • Move the database config option from being treated only as a module config option to a proper Apache server context.
  • Move a lot of initialization code around that verifies the NSS database filesystem read permissions and make it iterable
  • Convert nss_pcache from validating the token password to just storing it. I now do a NSS_NoDB_Init() because there could be multiple sources and I don’t want to deal with that headache. This shifts the burden of identifying a bad password a bit later but it should be handled ok

I’ve run into another roadblock similar to the database config option being treated as global: the password dialog is also treated as a global. It really needs to be handled per-server context. And this is where I am.

Fortunately it has mostly been a matter of reorganizing some structures and fixing all uses of them, so it’s been more grunt work than anything else.

On the bright side I’ve got it passing the in-tree tests by specifying a single NSS database. Under the hood it’s using NSS contexts and the new nss_pcache backend.

Something I’ll need to look into is how to release this new code. I’m not 100% sure it isn’t going to blow up on people, depending on how they have things configured. This may also represent mod_nss 2.0, we’ll see. It won’t be much of a difference functionally, with the exception that different VS would be able to have different CA trust, but internally it’ll be quite different.

So stay tuned.

CentOS 7, cloud-init and DataSourceConfigDriveNet

Something I ran into when developing the novajoin service was that my cloud-init script was not executed if either force_config_drive was True in nova.conf or if config_drive was enabled for a particular instance. What I’d see is that no metadata would come across and cloud-init would do very little work at all beyond adding keypairs and configuring networking.

The image I was working on was CentOS-7-x86_64-GenericCloud.qcow2 (1511). If I used a similar RHEL 7 image things worked fine.

I determined the issue to be with the version of cloud-init. You need cloud-init-0.7.6 for config-drive to work. I got a copy of the cloud-init rpm that Red Hat uses and used virt-customize to update my CentOS image and things worked after that.

$ virt-customize -a CentOS-7-x86_64-GenericCloud.qcow2 --install

With this image the novajoin service can push a cloud-init script that will enroll the instance into IPA.

nova metadata REST API

The nova service includes a metadata server where information about an instance is made available to that instance (for use during cloud-init, for example). This includes common things like the hostname, root password, ssh keypairs, etc.

A relatively new feature in Newton adds dynamic providers. When a request is made for metadata nova will contact the configured providers using a REST API and include the returned values in the metadata.

To enable dynamic metadata, add “DynamicJSON” to the vendordata_providers configuration option. This can also include “StaticJSON”

The vendordata_dynamic_targets configuration option specifies the URLs to be retrieved when metadata is requested.tance.

The format for an entry in vendordata_dynamic_targets is: @

Name is a string to distinguish this dynamic metadata from other dynamic providers. This will used as the key to the metadata returned to the instance.

Where name is a short string not including the ‘@’ character, and where the
URL can include a port number if so required. An example would be::

For example: test@

This dynamic metadata is available in a new file, openstack/2016-10-06/vendor_data2.json

It can be retrieved as an URL from within an instance using:

$ curl

The output will look something like:

    "test": {
        "key1": "somedata",
        "key2": "something else",

The following is passed to the dynamic REST server when nova receives a metadata request:

[table id=1 /]

Openstack services obtaining tokens

I had a difficult time finding information on how a service might obtain a token to talk to other services. There is certainly a lot of code to pull from but the calls are quite distributed. I got most of this from the ceilometer project.

The basic idea is that you register the keystoneauth1 options into your configuration, then instantiate it, then you can make calls to services.

I’m using a global session in this code. You can just as easily drop that and generate a new session every time you need to talk to something.

This code will obtain a token using the information from the [service_credentials] section in /etc/test/test.conf and make a request to nova and print the results.

from keystoneauth1 import loading as ks_loading
from oslo_config import cfg
from novaclient import client as nova_client


CFG_GROUP = "service_credentials"

_AUTH = None

def get_session():
    global _SESSION
    global _AUTH

    if not _AUTH:
        auth = ks_loading.load_auth_from_conf_options(cfg.CONF, 'service_credentials')

    if not _SESSION:
        _SESSION = ks_loading.load_session_from_conf_options(
            CONF, CFG_GROUP, auth=auth)

    return _SESSION

def novaclient():
    session = get_session()
    return nova_client.Client('2.1', session=session)

def register_keystoneauth_opts(conf):
    ks_loading.register_auth_conf_options(conf, CFG_GROUP)
        conf, CFG_GROUP,
        deprecated_opts={'cacert': [
            cfg.DeprecatedOpt('os-cacert', group=CFG_GROUP),
            cfg.DeprecatedOpt('os-cacert', group="DEFAULT")]

CONF([], project='test')

x = novaclient()
print x.flavors.list()

Create a configuration file

# mkdir /etc/test

Add this to /etc/test/test.conf (note that I’m re-using the nova credentials for this demo):


This was done using a Newton pre-release installation I’m developing on.

novajoin microservice integration

novajoin is a project for Openstack and IPA integration. It is a service that will allow instances created in nova to be added to IPA and a host OTP generated automatically. This OTP will then be passed into the instance to be used for enrollment during the cloud-init stage.

The end result is that a new instance will seamlessly be enrolled as an IPA client upon first boot.

Additionally, a class can be associated with an instance using Glance metadata so that IPA automember rules will automatically assign this new host to the appropriate hostgroups. Once that is done you can setup HBAC and sudo rules to grant the appropriate permissons/capabilities for all hosts in that group.

In short it can simplify administration significantly.

In the current iteration, novajoin consists of two pieces: a REST microservice and an AMQP notification listener.

The REST microservice is used to return dynamically generated metadata that will be added to the information that describes a given nova instance. This metadata is available at first boot and this is how novajoin injects the OTP into the instance for use with ipa-client-install. The framework for this change is being implemented in nova in this review: .

The REST server just handles theĀ  metadata, cloud-init does the rest. A cloud-init script is provided which glues the two together. It installs the needed packages, retrieves the metadata, then calls ipa-client-install with the requisite options.

The other server is an AMQP listener that will identify when an IPA-enrolled instance is deleted and remove host from IPA . It may eventually handle floating IP changes as well, automatically updating IPA DNS entries. The issue here is knowing what hostname to assign to the floating IP.

Glance images can have metadata as well which describes the image, such as OS distribution and version. If these have been set then novajoin will include this in the IPA entry it creates.

The basic flow looks something like this:

  1. Boot instance in nova. Add IPA metadata, specifying ipa_enroll True and optionally ipa_hostclass
  2. Instance boots. During cloud-init it will retrieve metadata
  3. During metadata retrieval ipa host-add is executed, adding the host to IPA and generating an OTP and any image metadata available.
  4. OTP and FQDN is returned in the metadata
  5. Our cloud-init script is called to install the IPA client packages and retrieve the OTP and FQDN
  6. Call ipa-client-install –hostname FQDN –password OTP

This leaves us with an IPA-enrolled client which can have permissions granted via HBAC and sudo rules (like who is allowed to log into this instance, what sudo commands are allowed, etc).