[BALUG-Talk] Dynamic DNS updates and DNS notify and letsencrypt.org ... yummy! :-)

[Michael Paoli]

And, further optimizations - still much faster.
Also, forgot to mention in the earlier, DNS notify is also a
key component, at it generally causes quite timely propagation of
DNS changes from master to authoritative slaves.

So, ... half a dozen non-trivial certs, from, the earlier ...
month+ ago, a semi-manual process that would take usually about
30 minutes or so, with much of that being human time (at lot of
manual editing of zone files, and semi-manual checking of DNS
propagation), down to the earlier ... automated ... less than
4 minutes (3m21.050s) ... and now, further optimized ... all the way
down to under 22 seconds! (0m21.974s):

$ time (for c in  
'*.balug.org,balug.org,*.archive.balug.org,*.beta.balug.org,*.ipv4.balug.org,*.ipv6.balug.org,*.new.balug.org,*.secure.balug.org,*.staging.balug.org,*.test.balug.org,*.php.test.balug.org,*.wiki.balug.org' '*.sf-lug.org,sf-lug.org,*.ipv4.sf-lug.org,*.ipv6.sf-lug.org,*.sflug.org,sflug.org,*.sflug.com,sflug.com,*.sflug.net,sflug.net,sf-lug.net,www.sf-lug.net,sf-lug.com,www.sf-lug.com' 'mpaoli.net,*.mpaoli.net,*.blackie.mpaoli.net,*.old-debian.mpaoli.net,digitalwitness.org,*.digitalwitness.org ' '*.balug.org,balug.org,*.lists.balug.org,berkeleylug.com,*.berkeleylug.com' 'berkeleylug.com,*.berkeleylug.com,berkeleylug.org,*.berkeleylug.org' '*.pi.berkeleylug.com,pi.berkeleylug.com'; do myCERTBOT_EMAIL= myCERTBOT_OPTS='--preferred-challenges dns --test-cert --manual-auth-hook mymanual-auth-hook --manual-cleanup-hook mymanual-cleanup-hook' Getcerts "$c" >>/dev/null 2>&1;  
done)

real    0m21.974s
user    0m7.241s
sys     0m0.769s
$ (for f in *_cert.pem; do openssl x509 -noout -text -in "$f" 2>&1;  
done) | sed -ne 's/^ *\(Not After :.*\)$/\1/p;/DNS:/{s/^ *//;s/,  
*/,/g;s/DNS://g;p}'
Not After : Jul 12 11:30:16 2020 GMT
*.archive.balug.org,*.balug.org,*.beta.balug.org,*.ipv4.balug.org,*.ipv6.balug.org,*.new.balug.org,*.php.test.balug.org,*.secure.balug.org,*.staging.balug.org,*.test.balug.org,*.wiki.balug.org,balug.org
Not After : Jul 12 11:30:20 2020 GMT
*.ipv4.sf-lug.org,*.ipv6.sf-lug.org,*.sf-lug.org,*.sflug.com,*.sflug.net,*.sflug.org,sf-lug.com,sf-lug.net,sf-lug.org,sflug.com,sflug.net,sflug.org,www.sf-lug.com,www.sf-lug.net
Not After : Jul 12 11:30:24 2020 GMT
*.blackie.mpaoli.net,*.digitalwitness.org,*.mpaoli.net,*.old-debian.mpaoli.net,digitalwitness.org,mpaoli.net
Not After : Jul 12 11:30:28 2020 GMT
*.balug.org,*.berkeleylug.com,*.lists.balug.org,balug.org,berkeleylug.com
Not After : Jul 12 11:30:31 2020 GMT
*.berkeleylug.com,*.berkeleylug.org,berkeleylug.com,berkeleylug.org
Not After : Jul 12 11:30:34 2020 GMT
*.pi.berkeleylug.com,pi.berkeleylug.com
$

> From: "Michael Paoli" <Michael.Paoli@cal.berkeley.edu>
> Subject: Re: Dynamic DNS updates and letsencrypt.org ... yummy!  :-)
> Date: Sat, 28 Mar 2020 20:56:25 -0700

>> Date: Sat, 28 Mar 2020 13:32:44 +0100
>> Subject: Re: [BALUG-Talk] Dynamic DNS updates and letsencrypt.org  
>> ... yummy! :-)
>> To: Michael Paoli <Michael.Paoli@cal.berkeley.edu>
>
>> What exactly do you mean when you say dynamic DNS?
>> Any specific software you used?
>
> [redirecting this to back on-list]
>
> The dynamic DNS portion - and least as I've used it, uses protocol from
> RFC2136 https://tools.ietf.org/html/rfc2136
> (see also: https://en.wikipedia.org/wiki/Dynamic_DNS)
> to provide a means of updating DNS, without the traditional manual
> editing of zone file.  Essentially an API for making DNS changes,
> so such can be done (or at least submitted) quite quickly, and lends
> itself well to automation (and scalability, etc.).
>
> The specific software used?  At the "lowest levels" I'm using nsupdate(1).
> Generally with suitable key, nsupdate effectively authenticates to DNS
> master nameserver and corresponding key (and authorized access) on the DNS
> master nameserver, and then can proceed to submit permitted changes.
>
> That's mostly it on the DNS side.
>
> As for the letsencrypt.org certs, I'm using the certbot(1) client.
> The certbot client has capabilities to be configured to use a key with
> dynamic DNS.  It can also be given options and option arguments to call
> programs for the "manual" parts of the verification process.  These
> programs, however, needn't be manual - and can in fact also be leveraged
> to fully automate the process.  So that's what I did.  Essentially put
> in place (added) various infrastructure and configuration bits, to be
> able to fully automate the requesting and obtaining of certs, including
> the required validation portions done via DNS - and the requisite adding
> (and later removing) of the DNS records used for validation.
>
> So, the requisite components, as I've implemented, are about like this:
>
> o DNS master(s)
> o dynamic DNS - including key to allow the requisite DNS updates
> o some limited sudo access, so relevant ID can indirectly make use of
>   said key, but never has access to itself read the key.  The reasons
>   it's done this way, is not only control of access to the key, but with
>   the helper program that is allowed for that ID via sudo, there are
>   additional further checks/restrictions, so the ID can only add and
>   drop appropriate DNS records of appropriate name and format - tighter
>   restrictions beyond just those configured (and that can be configured)
>   on the key itself.
> o a pair of helper programs for certbot and calling those from certbot
>   with appropriate options and option arguments.  These not only
>   leverage the other program (and via sudo) to add/drop the relevant
>   DNS data, but also, in the case of adding, checks that the data that
>   has been requested to be added has propagated to the responding
>   authoritative nameservers before proceeding further - thus very much
>   helping ensure that the verification step will pass (rather than fail
>   due to letsencrypt.org checking an authoritative nameserver where the
>   requisite validation data hasn't yet propagated to the nameserver
>   checked).
> o there's also a wrapper program that handles creating private key(s)
>   and CSR(s), and then invoking certbot to request the cert(s) for the
>   CSR(s).
>
>
>> From: "Michael Paoli" <Michael.Paoli@cal.berkeley.edu>
>> Subject: Dynamic DNS updates and letsencrypt.org ... yummy!  :-)
>> Date: Fri, 27 Mar 2020 22:43:22 -0700
>
>> Dynamic DNS updates and letsencrypt.org ... yummy!  :-)
>>
>> So, not too long back, I had a wee little mini-project that gave me
>> motivation to start using, or at to at least seriously consider
>> starting to use, dynamic DNS updates.
>>
>> I'll skip the full details on that, but key bit was, to be able to give
>> specific individual(s), ability to relatively easily and quickly edit
>> DNS data ... but only very specific DNS data.  Notably, generally, that
>> of a specific subdomain (Pi.BerkeleyLUG.com) - and any subdomains
>> thereunder.  And without giving them ability to alter any other DNS
>> data in or under the zone (BerkeleyLUG.com).
>>
>> Without dynamic DNS updates, there would generally be two possible ways
>> to approach that.  One would be to create delegated subdomain. That
>> then fully hands that off, but then burdens them with additional need
>> to have DNS servers, and needing static Internet Protocol addresses
>> (IPs) for those etc., and part of the earlier motivation was that they
>> didn't have static IPs.  So that also would burden them with DNS
>> nameservers and need for static IPs - so that would at best be far from
>> idea.
>>
>> Other approach, would be to leverage existing DNS infrastructure, and
>> allow them only to change data in and under the subdomain
>> (Pi.BerkeleyLUG.com) of existing zone (BerkeleyLUG.com).  That would be
>> rather to quite difficult, as allowing them to only make DNS changes in
>> the zone file appropriate for the subdomain (and any thereunder), would
>> be a quite non-trivial configuration task.  So, such an approach also
>> wasn't very feasible.  Not to mention there are generally hazards with
>> allowing folks to edit DNS master zone file(s) - notably errors there
>> can cause significant problems with DNS (up to and including breakage
>> or other issues with up to the entire zone).
>>
>> Well, enter dynamic DNS.  With that, I could set up key that could be
>> used only to alter DNS data for the subdomain (and any subdomains
>> thereunder), but that key couldn't change any other DNS data at all.
>> So that was the general approach I pursued.  To test, and before
>> enabling dynamic DNS on production (BerkeleyLUG.com) domain/zone, I
>> created a temporary test delegated subdomain, and tested dynamic DNS on
>> that zone (and restricting to a subdomain thereof, and any subdomains
>> thereunder).  I then created one more such temporary test delegated
>> subdomain for some further testing.
>>
>> After I was fully satisfied with all that, I moved on to dropping those
>> two temporary test delegated subdomains, and enabling dynamic DNS
>> updates for the production domain/zone, and setting up key that could
>> only change the specifically configured subdomain (Pi.BerkeleyLUG.com)
>> within (and any subdomains thereunder).
>>
>> Anyway, after having that infrastructure in place, I thought of another
>> potentially very useful use case.  letsencrypt.org Certificate
>> Authority (CA) TLS(/"SSL") certificates (certs).  To validate those,
>> http can be used - but not for wildcard certs, or DNS and including
>> wildcard certs.  Since many of the certs I use and prefer (if not
>> "require" - does make things quite a bit simpler) use wildcards, well,
>> DNS validation is the way to go on that.  But that's been a quite
>> semi-manual process.  Notably, I'd been doing it where requesting
>> client tells me what DNS data to add, I manually add it (add it to zone
>> file and update serial number, reload, wait for that to propagate to
>> the responding authoritative nameservers) ... and do that for each such
>> verification data that needs be added, manually doing each, getting the
>> cert, then removing those temporarily added no longer needed bits of
>> DNS data.  Well, certbot(8) (client for letsencrypt.org) has
>> hooks/capabilities to use dynamic DNS updates and/or call programs in
>> place of (or as part of) the "manual" verification (and cleanup) steps.
>> So, by either by giving it access to suitable key (and/)or by having
>> it call programs to handle creating the validation data, those steps
>> can be automated.  For DNS, such program(s) would be being given a
>> [sub]domain, and a specific text string, and would then create the
>> relevant TXT record under that [sub]domain.  So, I set up a key with
>> the appropriate access to use dynamic DNS (and had earlier set dynamic
>> DNS up across all the zones where I'm master), and set up some helper
>> programs, to allow certbot(8) to leverage the specific key, to make the
>> DNS additions - and also remove them again after validations have
>> completed.
>>
>> So ... generating keys, CSRs, requesting certs, temporarily inserting
>> DNS validation data, getting certs, and removing that temporarily
>> inserted data.  That used to be a semi-manual process.  The manual part
>> mostly being adding DNS data, checking its propagation, proceeding
>> through that validation step ... for numerous domains across multiple
>> certs.  Typically that would take me, oh, maybe roughly 30 minutes or
>> more, maybe once every about 85 days or so (the certs have a lifetime of
>> only 90 days).
>>
>> But with dynamic DNS, and certbot(8) then using the hooks to programs I
>> wrote ... full automated.  What used to typically take about 30 minutes
>> or more about every 85 days or so ... much faster and fully automated.
>> Now down to ... under 4 minutes (3m21.050s)!
>>
>> Anyway, I show that below - the/my command used (I did this one against
>> letsencrypt.org test environment, as I don't need to get new certs at
>> present, and test doesn't have the rate limiting of the production
>> environment, and also saves production from having that additional
>> traffic where it's not needed - otherwise production is essentially
>> identical).
>>
>> $ time myCERTBOT_EMAIL= \
>>> myCERTBOT_OPTS='--preferred-challenges dns --test-cert --server  
>>> https://acme-v02.api.letsencrypt.org/directory --manual-auth-hook  
>>> mymanual-auth-hook --manual-cleanup-hook mymanual-cleanup-hook' \
>>> Getcerts \
>>> '*.balug.org,balug.org,*.archive.balug.org,*.beta.balug.org,*.ipv4.balug.org,*.ipv6.balug.org,*.new.balug.org,*.secure.balug.org,*.staging.balug.org,*.test.balug.org,*.php.test.balug.org,*.wiki.balug.org'  
>>> \
>>> '*.sf-lug.org,sf-lug.org,*.ipv4.sf-lug.org,*.ipv6.sf-lug.org,*.sflug.org,sflug.org,*.sflug.com,sflug.com,*.sflug.net,sflug.net,sf-lug.net,www.sf-lug.net,sf-lug.com,www.sf-lug.com'  
>>> \
>>> 'mpaoli.net,*.mpaoli.net,*.blackie.mpaoli.net,*.old-debian.mpaoli.net,digitalwitness.org,*.digitalwitness.org'  
>>> \
>>> '*.balug.org,balug.org,*.lists.balug.org,berkeleylug.com,*.berkeleylug.com'  
>>> \
>>> 'berkeleylug.com,*.berkeleylug.com,berkeleylug.org,*.berkeleylug.org' \
>>> '*.pi.berkeleylug.com,pi.berkeleylug.com'; echo "$?"
>> ...
>> real    3m21.050s
>> user    0m23.092s
>> sys     0m4.706s
>>
>> $ (for f in *_cert.pem; do openssl x509 -noout -text -in "$f" 2>&1
>>> done) |
>>> sed -ne 's/^ *\(Not After :.*\)$/\1/p
>>> /DNS:/{s/^ *//;s/, */,/g;s/DNS://g;p}'
>> Not After : Jun 25 16:57:17 2020 GMT
>> *.archive.balug.org,*.balug.org,*.beta.balug.org,*.ipv4.balug.org,*.ipv6.balug.org,*.new.balug.org,*.php.test.balug.org,*.secure.balug.org,*.staging.balug.org,*.test.balug.org,*.wiki.balug.org,balug.org
>> Not After : Jun 25 16:57:24 2020 GMT
>> *.ipv4.sf-lug.org,*.ipv6.sf-lug.org,*.sf-lug.org,*.sflug.com,*.sflug.net,*.sflug.org,sf-lug.com,sf-lug.net,sf-lug.org,sflug.com,sflug.net,sflug.org,www.sf-lug.com,www.sf-lug.net
>> Not After : Jun 25 16:57:28 2020 GMT
>> *.blackie.mpaoli.net,*.digitalwitness.org,*.mpaoli.net,*.old-debian.mpaoli.net,digitalwitness.org,mpaoli.net
>> Not After : Jun 25 16:57:33 2020 GMT
>> *.balug.org,*.berkeleylug.com,*.lists.balug.org,balug.org,berkeleylug.com
>> Not After : Jun 25 16:57:38 2020 GMT
>> *.berkeleylug.com,*.berkeleylug.org,berkeleylug.com,berkeleylug.org
>> Not After : Jun 25 16:57:43 2020 GMT
>> *.pi.berkeleylug.com,pi.berkeleylug.com
>> $
>>
>> And at the end there, for the certs obtained I show when they expire and
>> the domains/names in their SAN data.
>> Note also in the above, the leading "> " is not literally entered, but
>> PS2 (essentially the shell prompting us that it needs more input to
>> complete the command).
>>
>> So, 6 certs, each with Subject Alternative Name (SAN) having multiple
>> names/domains, and each with at least one, if not more, having wildcard
>> ... all done automatically in 3m21.050s elapsed time.
>>
>> Oh, and another advantage with dynamic DNS updates.  Serial numbers
>> handled automatically - one less place for us mere mortal humans to
>> screw up DNS.
>>
>> Also, on the helper programs ... I only wanted the invoking ID to be
>> able to make very certain specific changes to DNS records - the
>> validation records are of a very specific type and format, etc.  I
>> could restrict the key a fair bit on that, but not fully to only and
>> exactly just the needed.  So, I handle that via the helper programs and
>> a little bit of sudo.  The ID doing the request never actually has
>> access to the key itself.  Instead, via sudo, the key is used on its
>> behalf - and in program that further checks and restricts, such that
>> only and exactly needed validation data is added (and later removed) -
>> nothing else is allowed.  Very feasible that way.  Also, the helper
>> programs that add the DNS data - also checks that it's propagated to the
>> responding authoritative nameservers, before continuing (so it should
>> then pass letsencrypt.org's validation tests).
>>
>> But back to, e.g. Pi.BerkeleyLUG.com.  Were that to instead be done the
>> "old school" way of editing master zone file, would be quite
>> challenging to set something up that could only update DNS data for
>> that subdomain and subdomains thereunder ... but very easy to limit to
>> only and exactly that on dynamic DNS.
>>
>> So, dynamic DNS updates ... already able to do multiple useful things
>> for me (and other(s)).  :-)