And, further optimizations - still much faster. Also, forgot to mention in the earlier, DNS notify is also a key component, at it generally causes quite timely propagation of DNS changes from master to authoritative slaves.
So, ... half a dozen non-trivial certs, from, the earlier ... month+ ago, a semi-manual process that would take usually about 30 minutes or so, with much of that being human time (at lot of manual editing of zone files, and semi-manual checking of DNS propagation), down to the earlier ... automated ... less than 4 minutes (3m21.050s) ... and now, further optimized ... all the way down to under 22 seconds! (0m21.974s):
$ time (for c in '*.balug.org,balug.org,*.archive.balug.org,*.beta.balug.org,*.ipv4.balug.org,*.ipv6.balug.org,*.new.balug.org,*.secure.balug.org,*.staging.balug.org,*.test.balug.org,*.php.test.balug.org,*.wiki.balug.org' '*.sf-lug.org,sf-lug.org,*.ipv4.sf-lug.org,*.ipv6.sf-lug.org,*.sflug.org,sflug.org,*.sflug.com,sflug.com,*.sflug.net,sflug.net,sf-lug.net,www.sf-lug.net,sf-lug.com,www.sf-lug.com' 'mpaoli.net,*.mpaoli.net,*.blackie.mpaoli.net,*.old-debian.mpaoli.net,digitalwitness.org,*.digitalwitness.org ' '*.balug.org,balug.org,*.lists.balug.org,berkeleylug.com,*.berkeleylug.com' 'berkeleylug.com,*.berkeleylug.com,berkeleylug.org,*.berkeleylug.org' '*.pi.berkeleylug.com,pi.berkeleylug.com'; do myCERTBOT_EMAIL= myCERTBOT_OPTS='--preferred-challenges dns --test-cert --manual-auth-hook mymanual-auth-hook --manual-cleanup-hook mymanual-cleanup-hook' Getcerts "$c" >>/dev/null 2>&1; done)
real 0m21.974s user 0m7.241s sys 0m0.769s $ (for f in *_cert.pem; do openssl x509 -noout -text -in "$f" 2>&1; done) | sed -ne 's/^ *(Not After :.*)$/\1/p;/DNS:/{s/^ *//;s/, */,/g;s/DNS://g;p}' Not After : Jul 12 11:30:16 2020 GMT *.archive.balug.org,*.balug.org,*.beta.balug.org,*.ipv4.balug.org,*.ipv6.balug.org,*.new.balug.org,*.php.test.balug.org,*.secure.balug.org,*.staging.balug.org,*.test.balug.org,*.wiki.balug.org,balug.org Not After : Jul 12 11:30:20 2020 GMT *.ipv4.sf-lug.org,*.ipv6.sf-lug.org,*.sf-lug.org,*.sflug.com,*.sflug.net,*.sflug.org,sf-lug.com,sf-lug.net,sf-lug.org,sflug.com,sflug.net,sflug.org,www.sf-lug.com,www.sf-lug.net Not After : Jul 12 11:30:24 2020 GMT *.blackie.mpaoli.net,*.digitalwitness.org,*.mpaoli.net,*.old-debian.mpaoli.net,digitalwitness.org,mpaoli.net Not After : Jul 12 11:30:28 2020 GMT *.balug.org,*.berkeleylug.com,*.lists.balug.org,balug.org,berkeleylug.com Not After : Jul 12 11:30:31 2020 GMT *.berkeleylug.com,*.berkeleylug.org,berkeleylug.com,berkeleylug.org Not After : Jul 12 11:30:34 2020 GMT *.pi.berkeleylug.com,pi.berkeleylug.com $
From: "Michael Paoli" Michael.Paoli@cal.berkeley.edu Subject: Re: Dynamic DNS updates and letsencrypt.org ... yummy! :-) Date: Sat, 28 Mar 2020 20:56:25 -0700
Date: Sat, 28 Mar 2020 13:32:44 +0100 Subject: Re: [BALUG-Talk] Dynamic DNS updates and letsencrypt.org ... yummy! :-) To: Michael Paoli Michael.Paoli@cal.berkeley.edu
What exactly do you mean when you say dynamic DNS? Any specific software you used?
[redirecting this to back on-list]
The dynamic DNS portion - and least as I've used it, uses protocol from RFC2136 https://tools.ietf.org/html/rfc2136 (see also: https://en.wikipedia.org/wiki/Dynamic_DNS) to provide a means of updating DNS, without the traditional manual editing of zone file. Essentially an API for making DNS changes, so such can be done (or at least submitted) quite quickly, and lends itself well to automation (and scalability, etc.).
The specific software used? At the "lowest levels" I'm using nsupdate(1). Generally with suitable key, nsupdate effectively authenticates to DNS master nameserver and corresponding key (and authorized access) on the DNS master nameserver, and then can proceed to submit permitted changes.
That's mostly it on the DNS side.
As for the letsencrypt.org certs, I'm using the certbot(1) client. The certbot client has capabilities to be configured to use a key with dynamic DNS. It can also be given options and option arguments to call programs for the "manual" parts of the verification process. These programs, however, needn't be manual - and can in fact also be leveraged to fully automate the process. So that's what I did. Essentially put in place (added) various infrastructure and configuration bits, to be able to fully automate the requesting and obtaining of certs, including the required validation portions done via DNS - and the requisite adding (and later removing) of the DNS records used for validation.
So, the requisite components, as I've implemented, are about like this:
o DNS master(s) o dynamic DNS - including key to allow the requisite DNS updates o some limited sudo access, so relevant ID can indirectly make use of said key, but never has access to itself read the key. The reasons it's done this way, is not only control of access to the key, but with the helper program that is allowed for that ID via sudo, there are additional further checks/restrictions, so the ID can only add and drop appropriate DNS records of appropriate name and format - tighter restrictions beyond just those configured (and that can be configured) on the key itself. o a pair of helper programs for certbot and calling those from certbot with appropriate options and option arguments. These not only leverage the other program (and via sudo) to add/drop the relevant DNS data, but also, in the case of adding, checks that the data that has been requested to be added has propagated to the responding authoritative nameservers before proceeding further - thus very much helping ensure that the verification step will pass (rather than fail due to letsencrypt.org checking an authoritative nameserver where the requisite validation data hasn't yet propagated to the nameserver checked). o there's also a wrapper program that handles creating private key(s) and CSR(s), and then invoking certbot to request the cert(s) for the CSR(s).
From: "Michael Paoli" Michael.Paoli@cal.berkeley.edu Subject: Dynamic DNS updates and letsencrypt.org ... yummy! :-) Date: Fri, 27 Mar 2020 22:43:22 -0700
Dynamic DNS updates and letsencrypt.org ... yummy! :-)
So, not too long back, I had a wee little mini-project that gave me motivation to start using, or at to at least seriously consider starting to use, dynamic DNS updates.
I'll skip the full details on that, but key bit was, to be able to give specific individual(s), ability to relatively easily and quickly edit DNS data ... but only very specific DNS data. Notably, generally, that of a specific subdomain (Pi.BerkeleyLUG.com) - and any subdomains thereunder. And without giving them ability to alter any other DNS data in or under the zone (BerkeleyLUG.com).
Without dynamic DNS updates, there would generally be two possible ways to approach that. One would be to create delegated subdomain. That then fully hands that off, but then burdens them with additional need to have DNS servers, and needing static Internet Protocol addresses (IPs) for those etc., and part of the earlier motivation was that they didn't have static IPs. So that also would burden them with DNS nameservers and need for static IPs - so that would at best be far from idea.
Other approach, would be to leverage existing DNS infrastructure, and allow them only to change data in and under the subdomain (Pi.BerkeleyLUG.com) of existing zone (BerkeleyLUG.com). That would be rather to quite difficult, as allowing them to only make DNS changes in the zone file appropriate for the subdomain (and any thereunder), would be a quite non-trivial configuration task. So, such an approach also wasn't very feasible. Not to mention there are generally hazards with allowing folks to edit DNS master zone file(s) - notably errors there can cause significant problems with DNS (up to and including breakage or other issues with up to the entire zone).
Well, enter dynamic DNS. With that, I could set up key that could be used only to alter DNS data for the subdomain (and any subdomains thereunder), but that key couldn't change any other DNS data at all. So that was the general approach I pursued. To test, and before enabling dynamic DNS on production (BerkeleyLUG.com) domain/zone, I created a temporary test delegated subdomain, and tested dynamic DNS on that zone (and restricting to a subdomain thereof, and any subdomains thereunder). I then created one more such temporary test delegated subdomain for some further testing.
After I was fully satisfied with all that, I moved on to dropping those two temporary test delegated subdomains, and enabling dynamic DNS updates for the production domain/zone, and setting up key that could only change the specifically configured subdomain (Pi.BerkeleyLUG.com) within (and any subdomains thereunder).
Anyway, after having that infrastructure in place, I thought of another potentially very useful use case. letsencrypt.org Certificate Authority (CA) TLS(/"SSL") certificates (certs). To validate those, http can be used - but not for wildcard certs, or DNS and including wildcard certs. Since many of the certs I use and prefer (if not "require" - does make things quite a bit simpler) use wildcards, well, DNS validation is the way to go on that. But that's been a quite semi-manual process. Notably, I'd been doing it where requesting client tells me what DNS data to add, I manually add it (add it to zone file and update serial number, reload, wait for that to propagate to the responding authoritative nameservers) ... and do that for each such verification data that needs be added, manually doing each, getting the cert, then removing those temporarily added no longer needed bits of DNS data. Well, certbot(8) (client for letsencrypt.org) has hooks/capabilities to use dynamic DNS updates and/or call programs in place of (or as part of) the "manual" verification (and cleanup) steps. So, by either by giving it access to suitable key (and/)or by having it call programs to handle creating the validation data, those steps can be automated. For DNS, such program(s) would be being given a [sub]domain, and a specific text string, and would then create the relevant TXT record under that [sub]domain. So, I set up a key with the appropriate access to use dynamic DNS (and had earlier set dynamic DNS up across all the zones where I'm master), and set up some helper programs, to allow certbot(8) to leverage the specific key, to make the DNS additions - and also remove them again after validations have completed.
So ... generating keys, CSRs, requesting certs, temporarily inserting DNS validation data, getting certs, and removing that temporarily inserted data. That used to be a semi-manual process. The manual part mostly being adding DNS data, checking its propagation, proceeding through that validation step ... for numerous domains across multiple certs. Typically that would take me, oh, maybe roughly 30 minutes or more, maybe once every about 85 days or so (the certs have a lifetime of only 90 days).
But with dynamic DNS, and certbot(8) then using the hooks to programs I wrote ... full automated. What used to typically take about 30 minutes or more about every 85 days or so ... much faster and fully automated. Now down to ... under 4 minutes (3m21.050s)!
Anyway, I show that below - the/my command used (I did this one against letsencrypt.org test environment, as I don't need to get new certs at present, and test doesn't have the rate limiting of the production environment, and also saves production from having that additional traffic where it's not needed - otherwise production is essentially identical).
$ time myCERTBOT_EMAIL= \
myCERTBOT_OPTS='--preferred-challenges dns --test-cert --server https://acme-v02.api.letsencrypt.org/directory --manual-auth-hook mymanual-auth-hook --manual-cleanup-hook mymanual-cleanup-hook' \ Getcerts \ '*.balug.org,balug.org,*.archive.balug.org,*.beta.balug.org,*.ipv4.balug.org,*.ipv6.balug.org,*.new.balug.org,*.secure.balug.org,*.staging.balug.org,*.test.balug.org,*.php.test.balug.org,*.wiki.balug.org' \ '*.sf-lug.org,sf-lug.org,*.ipv4.sf-lug.org,*.ipv6.sf-lug.org,*.sflug.org,sflug.org,*.sflug.com,sflug.com,*.sflug.net,sflug.net,sf-lug.net,www.sf-lug.net,sf-lug.com,www.sf-lug.com' \ 'mpaoli.net,*.mpaoli.net,*.blackie.mpaoli.net,*.old-debian.mpaoli.net,digitalwitness.org,*.digitalwitness.org' \ '*.balug.org,balug.org,*.lists.balug.org,berkeleylug.com,*.berkeleylug.com' \ 'berkeleylug.com,*.berkeleylug.com,berkeleylug.org,*.berkeleylug.org' \ '*.pi.berkeleylug.com,pi.berkeleylug.com'; echo "$?"
... real 3m21.050s user 0m23.092s sys 0m4.706s
$ (for f in *_cert.pem; do openssl x509 -noout -text -in "$f" 2>&1
done) | sed -ne 's/^ *(Not After :.*)$/\1/p /DNS:/{s/^ *//;s/, */,/g;s/DNS://g;p}'
Not After : Jun 25 16:57:17 2020 GMT *.archive.balug.org,*.balug.org,*.beta.balug.org,*.ipv4.balug.org,*.ipv6.balug.org,*.new.balug.org,*.php.test.balug.org,*.secure.balug.org,*.staging.balug.org,*.test.balug.org,*.wiki.balug.org,balug.org Not After : Jun 25 16:57:24 2020 GMT *.ipv4.sf-lug.org,*.ipv6.sf-lug.org,*.sf-lug.org,*.sflug.com,*.sflug.net,*.sflug.org,sf-lug.com,sf-lug.net,sf-lug.org,sflug.com,sflug.net,sflug.org,www.sf-lug.com,www.sf-lug.net Not After : Jun 25 16:57:28 2020 GMT *.blackie.mpaoli.net,*.digitalwitness.org,*.mpaoli.net,*.old-debian.mpaoli.net,digitalwitness.org,mpaoli.net Not After : Jun 25 16:57:33 2020 GMT *.balug.org,*.berkeleylug.com,*.lists.balug.org,balug.org,berkeleylug.com Not After : Jun 25 16:57:38 2020 GMT *.berkeleylug.com,*.berkeleylug.org,berkeleylug.com,berkeleylug.org Not After : Jun 25 16:57:43 2020 GMT *.pi.berkeleylug.com,pi.berkeleylug.com $
And at the end there, for the certs obtained I show when they expire and the domains/names in their SAN data. Note also in the above, the leading "> " is not literally entered, but PS2 (essentially the shell prompting us that it needs more input to complete the command).
So, 6 certs, each with Subject Alternative Name (SAN) having multiple names/domains, and each with at least one, if not more, having wildcard ... all done automatically in 3m21.050s elapsed time.
Oh, and another advantage with dynamic DNS updates. Serial numbers handled automatically - one less place for us mere mortal humans to screw up DNS.
Also, on the helper programs ... I only wanted the invoking ID to be able to make very certain specific changes to DNS records - the validation records are of a very specific type and format, etc. I could restrict the key a fair bit on that, but not fully to only and exactly just the needed. So, I handle that via the helper programs and a little bit of sudo. The ID doing the request never actually has access to the key itself. Instead, via sudo, the key is used on its behalf - and in program that further checks and restricts, such that only and exactly needed validation data is added (and later removed) - nothing else is allowed. Very feasible that way. Also, the helper programs that add the DNS data - also checks that it's propagated to the responding authoritative nameservers, before continuing (so it should then pass letsencrypt.org's validation tests).
But back to, e.g. Pi.BerkeleyLUG.com. Were that to instead be done the "old school" way of editing master zone file, would be quite challenging to set something up that could only update DNS data for that subdomain and subdomains thereunder ... but very easy to limit to only and exactly that on dynamic DNS.
So, dynamic DNS updates ... already able to do multiple useful things for me (and other(s)). :-)