Okay, so, ... I think I got it effectively figured out. At least how to operate it, notably BIND 9 with dnssec-policy and including safely transitioning from from auto-dnssec maintain to dnssec-policy and doing ZSK and KSK rollovers.
So, first, I don't think there are any issues with DNSSEC itself, that's fine and solid as always. And sure, wee bit of care is required - as with DNS in general - as it is possible to shoot oneself in the foot. But it's not rocket science, and pretty straight forward.
I also think ISC BIND's dnssec-policy is a very significant improvement over their earlier auto-dnssec and such. But, despite their extensive documentation, alas, I think this time around that's where they fell significantly short. Some things just don't behave, at least under certain circumstances, at all like the documentation implies - and in manners that can significantly break things (most notably DNSSEC, and very much violating the principle of least surprise, etc.). So, there's definitely room for improvement there - most notably on the documentation. "Other than that", seems all quite solid as far as I can tell. And as far as I'm able to tell thus far, with proper procedures, all works basically fine. Alas, exactly what those proper procedures are - and aren't - not so clear from their documentation (and in some cases what their documentation implies is in fact not correct). So, maybe case of the code having gotten rather far ahead of and kind'a out'a step with the documentation, and ... documentation needs to properly catch up? Seems like it to me.
Anyway, I'll likely ... when I get around to it, and likely after yet more testing to fully confirm, probably do some wiki writeup bits on it to better document that (for myself, and others that may be looking for similar). At first I was thinking on www.wiki.balug.org, but probably more fitting is further enhancing Debian wiki: DNSSEC Howto for BIND 9.9+ https://wiki.debian.org/DNSSEC%20Howto%20for%20BIND%209.9+ (I'd earlier written/enhanced quite a bit of its content ... looks like time for yet some more of that). I should also give the ISC folks a bit of a nudge on the matter too ... most notably regarding their documentation. Also possible things were intended to go as the documentation implies, but at some point they significantly diverged.
And yes, alas, did have some more glitches with DNSSEC on sflug.org (non-canonical, notice no - in it). And did much further extensive testing on dnssec-test.balug.org. See also, e.g.: https://dnsviz.net/d/dnssec-test.balug.org/ZvaWoQ/dnssec/ etc.
And, short synopsis of key (no pun intended) bits? Goes about like this: auto-dnssec --> dnssec-policy: Make target policy and files, etc. fully consistent with existing keys, less instant breakage at conversion. Notably not only policy on key types and algorithms, aging/rotation, etc. but also, use dnssec-settime -f -s before switch to freshen creation time and provide any needed missing metadata. After that be sure to also readjust file ownerships/permissions, note also under dnssec-policy BIND user/group will need to not only be able to read files, but write directory/files. Rollover of ZSKs seems to work as documented, including automagically and/or per triggering with rndc dnssec -rollover. I've not tested CSK, but I'm presuming behaves similar to KSK. For KSK it's not nearly as smooth as documented, and though in some cases that seems to work as documented, that not uncommonly fails miserably. The proper way (/workaround) to do it smoothly appears to be like this: use dnssec-keygen to generate keypair consistent with policy use dnssec-settime -f -s to create all the relevant metadata appropriately adjust ownerships/permissions. use dnssec-dsfromkey (and/or other means as relevant) to get the needed data to add the new DS record for this new key - note on all these steps to wait the relevant TTLs to prevent issues (standard practice, particularly noteworthy where these changes are being done/triggered manually). use rndc sign zone use rndc dnssec -checkds -key newid published zone use rndc dnssec -checkds -key oldid withdrawn zone And probably highly similar applies to CSK What NOT to do, at least thus far (maybe this improves in future versions of BIND): for KSK and presumably also CSK keys: rndc dnssec -rollover -key oldid zone (though that appears perfectly fine for ZSK) as that will often instantly eject that key and instantly break DNSSEC, despite what current ISC BIND documentation suggest/implies. For KSK (and presumably likewise CSK) only use: rndc dnssec -rollover -key oldid zone after all the other relevant steps (and TTLs) have passed. Also appears using: rndc dnssec -rollover -key oldid zone straight off on KSK (and presumed CSK) may not always break things. It's possible this depends upon metadata on existing key(s) and how they got to that state, notably some of the metadata traced in the files alongside the primary zone files.