[BALUG-Talk] Dynamic DNS updates and letsencrypt.org ... yummy! :-)

[Michael Paoli]

> Date: Sat, 28 Mar 2020 13:32:44 +0100
> Subject: Re: [BALUG-Talk] Dynamic DNS updates and letsencrypt.org  
> ... yummy! :-)
> To: Michael Paoli <Michael.Paoli@cal.berkeley.edu>

> What exactly do you mean when you say dynamic DNS?
> Any specific software you used?

[redirecting this to back on-list]

The dynamic DNS portion - and least as I've used it, uses protocol from
RFC2136 https://tools.ietf.org/html/rfc2136
(see also: https://en.wikipedia.org/wiki/Dynamic_DNS)
to provide a means of updating DNS, without the traditional manual
editing of zone file.  Essentially an API for making DNS changes,
so such can be done (or at least submitted) quite quickly, and lends
itself well to automation (and scalability, etc.).

The specific software used?  At the "lowest levels" I'm using nsupdate(1).
Generally with suitable key, nsupdate effectively authenticates to DNS
master nameserver and corresponding key (and authorized access) on the DNS
master nameserver, and then can proceed to submit permitted changes.

That's mostly it on the DNS side.

As for the letsencrypt.org certs, I'm using the certbot(1) client.
The certbot client has capabilities to be configured to use a key with
dynamic DNS.  It can also be given options and option arguments to call
programs for the "manual" parts of the verification process.  These
programs, however, needn't be manual - and can in fact also be leveraged
to fully automate the process.  So that's what I did.  Essentially put
in place (added) various infrastructure and configuration bits, to be
able to fully automate the requesting and obtaining of certs, including
the required validation portions done via DNS - and the requisite adding
(and later removing) of the DNS records used for validation.

So, the requisite components, as I've implemented, are about like this:

o DNS master(s)
o dynamic DNS - including key to allow the requisite DNS updates
o some limited sudo access, so relevant ID can indirectly make use of
   said key, but never has access to itself read the key.  The reasons
   it's done this way, is not only control of access to the key, but with
   the helper program that is allowed for that ID via sudo, there are
   additional further checks/restrictions, so the ID can only add and
   drop appropriate DNS records of appropriate name and format - tighter
   restrictions beyond just those configured (and that can be configured)
   on the key itself.
o a pair of helper programs for certbot and calling those from certbot
   with appropriate options and option arguments.  These not only
   leverage the other program (and via sudo) to add/drop the relevant
   DNS data, but also, in the case of adding, checks that the data that
   has been requested to be added has propagated to the responding
   authoritative nameservers before proceeding further - thus very much
   helping ensure that the verification step will pass (rather than fail
   due to letsencrypt.org checking an authoritative nameserver where the
   requisite validation data hasn't yet propagated to the nameserver
   checked).
o there's also a wrapper program that handles creating private key(s)
   and CSR(s), and then invoking certbot to request the cert(s) for the
   CSR(s).


> From: "Michael Paoli" <Michael.Paoli@cal.berkeley.edu>
> Subject: Dynamic DNS updates and letsencrypt.org ... yummy!  :-)
> Date: Fri, 27 Mar 2020 22:43:22 -0700

> Dynamic DNS updates and letsencrypt.org ... yummy!  :-)
>
> So, not too long back, I had a wee little mini-project that gave me
> motivation to start using, or at to at least seriously consider
> starting to use, dynamic DNS updates.
>
> I'll skip the full details on that, but key bit was, to be able to give
> specific individual(s), ability to relatively easily and quickly edit
> DNS data ... but only very specific DNS data.  Notably, generally, that
> of a specific subdomain (Pi.BerkeleyLUG.com) - and any subdomains
> thereunder.  And without giving them ability to alter any other DNS
> data in or under the zone (BerkeleyLUG.com).
>
> Without dynamic DNS updates, there would generally be two possible ways
> to approach that.  One would be to create delegated subdomain. That
> then fully hands that off, but then burdens them with additional need
> to have DNS servers, and needing static Internet Protocol addresses
> (IPs) for those etc., and part of the earlier motivation was that they
> didn't have static IPs.  So that also would burden them with DNS
> nameservers and need for static IPs - so that would at best be far from
> idea.
>
> Other approach, would be to leverage existing DNS infrastructure, and
> allow them only to change data in and under the subdomain
> (Pi.BerkeleyLUG.com) of existing zone (BerkeleyLUG.com).  That would be
> rather to quite difficult, as allowing them to only make DNS changes in
> the zone file appropriate for the subdomain (and any thereunder), would
> be a quite non-trivial configuration task.  So, such an approach also
> wasn't very feasible.  Not to mention there are generally hazards with
> allowing folks to edit DNS master zone file(s) - notably errors there
> can cause significant problems with DNS (up to and including breakage
> or other issues with up to the entire zone).
>
> Well, enter dynamic DNS.  With that, I could set up key that could be
> used only to alter DNS data for the subdomain (and any subdomains
> thereunder), but that key couldn't change any other DNS data at all.
> So that was the general approach I pursued.  To test, and before
> enabling dynamic DNS on production (BerkeleyLUG.com) domain/zone, I
> created a temporary test delegated subdomain, and tested dynamic DNS on
> that zone (and restricting to a subdomain thereof, and any subdomains
> thereunder).  I then created one more such temporary test delegated
> subdomain for some further testing.
>
> After I was fully satisfied with all that, I moved on to dropping those
> two temporary test delegated subdomains, and enabling dynamic DNS
> updates for the production domain/zone, and setting up key that could
> only change the specifically configured subdomain (Pi.BerkeleyLUG.com)
> within (and any subdomains thereunder).
>
> Anyway, after having that infrastructure in place, I thought of another
> potentially very useful use case.  letsencrypt.org Certificate
> Authority (CA) TLS(/"SSL") certificates (certs).  To validate those,
> http can be used - but not for wildcard certs, or DNS and including
> wildcard certs.  Since many of the certs I use and prefer (if not
> "require" - does make things quite a bit simpler) use wildcards, well,
> DNS validation is the way to go on that.  But that's been a quite
> semi-manual process.  Notably, I'd been doing it where requesting
> client tells me what DNS data to add, I manually add it (add it to zone
> file and update serial number, reload, wait for that to propagate to
> the responding authoritative nameservers) ... and do that for each such
> verification data that needs be added, manually doing each, getting the
> cert, then removing those temporarily added no longer needed bits of
> DNS data.  Well, certbot(8) (client for letsencrypt.org) has
> hooks/capabilities to use dynamic DNS updates and/or call programs in
> place of (or as part of) the "manual" verification (and cleanup) steps.
> So, by either by giving it access to suitable key (and/)or by having
> it call programs to handle creating the validation data, those steps
> can be automated.  For DNS, such program(s) would be being given a
> [sub]domain, and a specific text string, and would then create the
> relevant TXT record under that [sub]domain.  So, I set up a key with
> the appropriate access to use dynamic DNS (and had earlier set dynamic
> DNS up across all the zones where I'm master), and set up some helper
> programs, to allow certbot(8) to leverage the specific key, to make the
> DNS additions - and also remove them again after validations have
> completed.
>
> So ... generating keys, CSRs, requesting certs, temporarily inserting
> DNS validation data, getting certs, and removing that temporarily
> inserted data.  That used to be a semi-manual process.  The manual part
> mostly being adding DNS data, checking its propagation, proceeding
> through that validation step ... for numerous domains across multiple
> certs.  Typically that would take me, oh, maybe roughly 30 minutes or
> more, maybe once every about 85 days or so (the certs have a lifetime of
> only 90 days).
>
> But with dynamic DNS, and certbot(8) then using the hooks to programs I
> wrote ... full automated.  What used to typically take about 30 minutes
> or more about every 85 days or so ... much faster and fully automated.
> Now down to ... under 4 minutes (3m21.050s)!
>
> Anyway, I show that below - the/my command used (I did this one against
> letsencrypt.org test environment, as I don't need to get new certs at
> present, and test doesn't have the rate limiting of the production
> environment, and also saves production from having that additional
> traffic where it's not needed - otherwise production is essentially
> identical).
>
> $ time myCERTBOT_EMAIL= \
>> myCERTBOT_OPTS='--preferred-challenges dns --test-cert --server  
>> https://acme-v02.api.letsencrypt.org/directory --manual-auth-hook  
>> mymanual-auth-hook --manual-cleanup-hook mymanual-cleanup-hook' \
>> Getcerts \
>> '*.balug.org,balug.org,*.archive.balug.org,*.beta.balug.org,*.ipv4.balug.org,*.ipv6.balug.org,*.new.balug.org,*.secure.balug.org,*.staging.balug.org,*.test.balug.org,*.php.test.balug.org,*.wiki.balug.org'  
>> \
>> '*.sf-lug.org,sf-lug.org,*.ipv4.sf-lug.org,*.ipv6.sf-lug.org,*.sflug.org,sflug.org,*.sflug.com,sflug.com,*.sflug.net,sflug.net,sf-lug.net,www.sf-lug.net,sf-lug.com,www.sf-lug.com'  
>> \
>> 'mpaoli.net,*.mpaoli.net,*.blackie.mpaoli.net,*.old-debian.mpaoli.net,digitalwitness.org,*.digitalwitness.org'  
>> \
>> '*.balug.org,balug.org,*.lists.balug.org,berkeleylug.com,*.berkeleylug.com'  
>> \
>> 'berkeleylug.com,*.berkeleylug.com,berkeleylug.org,*.berkeleylug.org' \
>> '*.pi.berkeleylug.com,pi.berkeleylug.com'; echo "$?"
> ...
> real    3m21.050s
> user    0m23.092s
> sys     0m4.706s
>
> $ (for f in *_cert.pem; do openssl x509 -noout -text -in "$f" 2>&1
>> done) |
>> sed -ne 's/^ *\(Not After :.*\)$/\1/p
>> /DNS:/{s/^ *//;s/, */,/g;s/DNS://g;p}'
> Not After : Jun 25 16:57:17 2020 GMT
> *.archive.balug.org,*.balug.org,*.beta.balug.org,*.ipv4.balug.org,*.ipv6.balug.org,*.new.balug.org,*.php.test.balug.org,*.secure.balug.org,*.staging.balug.org,*.test.balug.org,*.wiki.balug.org,balug.org
> Not After : Jun 25 16:57:24 2020 GMT
> *.ipv4.sf-lug.org,*.ipv6.sf-lug.org,*.sf-lug.org,*.sflug.com,*.sflug.net,*.sflug.org,sf-lug.com,sf-lug.net,sf-lug.org,sflug.com,sflug.net,sflug.org,www.sf-lug.com,www.sf-lug.net
> Not After : Jun 25 16:57:28 2020 GMT
> *.blackie.mpaoli.net,*.digitalwitness.org,*.mpaoli.net,*.old-debian.mpaoli.net,digitalwitness.org,mpaoli.net
> Not After : Jun 25 16:57:33 2020 GMT
> *.balug.org,*.berkeleylug.com,*.lists.balug.org,balug.org,berkeleylug.com
> Not After : Jun 25 16:57:38 2020 GMT
> *.berkeleylug.com,*.berkeleylug.org,berkeleylug.com,berkeleylug.org
> Not After : Jun 25 16:57:43 2020 GMT
> *.pi.berkeleylug.com,pi.berkeleylug.com
> $
>
> And at the end there, for the certs obtained I show when they expire and
> the domains/names in their SAN data.
> Note also in the above, the leading "> " is not literally entered, but
> PS2 (essentially the shell prompting us that it needs more input to
> complete the command).
>
> So, 6 certs, each with Subject Alternative Name (SAN) having multiple
> names/domains, and each with at least one, if not more, having wildcard
> ... all done automatically in 3m21.050s elapsed time.
>
> Oh, and another advantage with dynamic DNS updates.  Serial numbers
> handled automatically - one less place for us mere mortal humans to
> screw up DNS.
>
> Also, on the helper programs ... I only wanted the invoking ID to be
> able to make very certain specific changes to DNS records - the
> validation records are of a very specific type and format, etc.  I
> could restrict the key a fair bit on that, but not fully to only and
> exactly just the needed.  So, I handle that via the helper programs and
> a little bit of sudo.  The ID doing the request never actually has
> access to the key itself.  Instead, via sudo, the key is used on its
> behalf - and in program that further checks and restricts, such that
> only and exactly needed validation data is added (and later removed) -
> nothing else is allowed.  Very feasible that way.  Also, the helper
> programs that add the DNS data - also checks that it's propagated to the
> responding authoritative nameservers, before continuing (so it should
> then pass letsencrypt.org's validation tests).
>
> But back to, e.g. Pi.BerkeleyLUG.com.  Were that to instead be done the
> "old school" way of editing master zone file, would be quite
> challenging to set something up that could only update DNS data for
> that subdomain and subdomains thereunder ... but very easy to limit to
> only and exactly that on dynamic DNS.
>
> So, dynamic DNS updates ... already able to do multiple useful things
> for me (and other(s)).  :-)