Second part (server authenticates the client):

The second commit, adds a dedicated port on the server in order to enforce TLS Authentication for connecting clients (secondary servers) and address #336 from the server's side.

nsd does not support multiple tls-port(s) so I had to put checks in multiple places to see if we are on tls-port or tls-auth-port.
Not the ideal way of doing this but it works.

Another approach would be to put a tls-auth: yes/no knob where we could enforce tls authentication on the normal tls-port.
This would be a much more simple solution. Happy to make it if this one is not accepted.

Now let me explain a little about my initial approach which was adding tls-auth on provide-xfr acl.
TLS handshake and the creation on nsd.tls_ctx and data->tls are created an a stage where we don't have access yet to the provide-xfr ACL in order to check if we require tls-auth.

I've tried the approach of doing SSL_CTX_set_verify with SSL_VERIFY_PEER | SSL_VERIFY_POST_HANDSHAKE
on the server in order to make the initial TLS handshake and request for a client authenticate at a later stage.

Client sends SSL_set_post_handshake_auth() in the first ClientHello to say that it supports Post-Handshake Authentication (PHA)

Server carries data->tls pointer down to query_process() where it calls a new function process_tls_auth() in order to
call SSL_verify_client_post_handshake() and issue another TLS handshake saying that it now requires client authentication.

query_process() was the first place where the ACL requiring tls-auth was available to me.
Note that PHA is only supposed to work for TLS1.3. TLS1.2 does renegotiation which is not available in TLS1.3

static nsd_rc_type
process_tls_auth(SSL *tls, struct query* q, nsd_type *nsd)
{
   struct zone_options* zone_opt;
   zone_opt = zone_options_find(nsd->options, q->qname);
   if (tls && zone_opt->pattern->provide_xfr->tls_auth_name) {
      // code taken from s_server.c from openssl
      SSL_set_verify(tls, SSL_VERIFY_PEER, NULL);
      if (!SSL_verify_client_post_handshake(tls)) {
         log_msg(LOG_ERR, "error in SSL_verify_client_post_handshake");
         return NSD_RC_REFUSE;
      } else {
         if(!SSL_do_handshake(tls)) {
            log_msg(LOG_ERR, "FAILED: SSL_do_handshake");
            return NSD_RC_REFUSE; // maybe not this, but fail in any case
         } else {
            log_msg(LOG_INFO, "TLS-AUTH SUCCESS!");
         }
      }
   }
   return NSD_RC_OK;
}

Up to this point I knew if we are on an ACL that requires TLS-AUTH in order to provide XFR to that specific client.

Don't know if the problem was calling SSL_do_handshake direct and not via tls_handshake() but it didn't work.
event code and handlers were complicated for me to play more with this.

This is where I stopped and though about tls-auth-port as a dedicated SSL_CTX that requires
SSL_VERIFY_PEER | SSL_VERIFY_FAIL_IF_NO_PEER_CERT

As I said, since this is the enforcing option, as well as passing a non-empty CAfile, this could also be done via a simpler YES/NO knob to enforce it on normal tls-port.

The NSD_RC_NOTAUTH return is not right, it means not authoritative for the zone, and to communicate that access is not granted, I think NSD_RC_REFUSE is a better choice of DNS RCODE value.

If the client has to set SSL_set_post_handshake_auth before connection starts, then perhaps it is nicer to support it somehow where the client action is not needed. With regular client authentication, this realistically more for compatibility, because I do not think any software sets that before the connection.

It is nicer if this option can exist together with regular tls service, so an on/off button is not really where I would take it either. But the check of client certificate at the initial handshake stage, on the tls_auth port, seems like a fine idea to me. If that works for the clients.

If the client has to set SSL_set_post_handshake_auth before connection starts, then perhaps it is nicer to support it somehow where the client action is not needed. With regular client authentication, this realistically more for compatibility, because I do not think any software sets that before the connection.

I agree that most clients do not set it. There are multiple talks online about this and various browsers that do not want to set it. Don't know about resolvers.

It is nicer if this option can exist together with regular tls service, so an on/off button is not really where I would take it either. But the check of client certificate at the initial handshake stage, on the tls_auth port, seems like a fine idea to me. If that works for the clients.

I'm not 100% decided that an on/off knob is that bad.
nsd is an authoritative server. Why would you need TLS in the first place except for zone transfers?
I can't think of a scenario where you want some clients to authenticate (on the TLS port) and some others to not.

Maybe if you want different policies per zone (stubs for instance). But then you need to go down to the ACL and tls-auth.

In any case the tls-auth-port seems the most friendly one in terms of compatibility and it does not sacrifice/alter the normal TLS behaviour.

As a side note, if this is finally accepted, I would also add something like the following in the manual page, which is important in my opinion:

Client's certificate name in not validated by the server. Server accepts any certificate presented by the client that is validated with tls-cert-bundle. If server uses the default verify location of the system, then any certificate signed by a valid CA is accepted. For this reason it is highly recommended to either setup a local PKI or create self signed certificates specifically for this purpose (XoT) for your secondary servers and only add those to a custom tls-cert-bundle file on the primary.
In advance primary should also filter (firewall) connections from secondaries to other listening ports in order to prohibit unauthenticated zone transfers and specifically only allow them via the tls-auth-port.

@wcawijngaards do you know if this will be accepted?

If yes, I would like to extend my work on top of this and address the 2 issues I've mentioned above.

In advance primary should also filter (firewall) connections from secondaries to other listening ports in order to prohibit unauthenticated zone transfers and specifically only allow them via the tls-auth-port.

This can be solved on the nsd side, by implementing something like a "tls-auth-xfr-only" yes/no knob
If yes, only xfrs that are in data->tls_auth (authenticated) will be accepted

Client's certificate name in not validated by the server. Server accepts any certificate presented by the client that is validated with tls-cert-bundle. If server uses the default verify location of the system, then any certificate signed by a valid CA is accepted. For this reason it is highly recommended to either setup a local PKI or create self signed certificates specifically for this purpose (XoT) for your secondary servers and only add those to a custom tls-cert-bundle file on the primary.

This can be solved by adding "tls-auth" in provide-xfr like it is in request-xfr, and use auth-domain-name from specified tls-auth, in order to verify client name is matching the presented certificate. The same way client verifies server's certificate.

Interested on these? I believe they are not hard to implement and will add to the security.

Now, for both of these to work
data->tls and
data->tls_auth must go from handle_tcp_reading()/handle_tls_reading() down to query_process()

I can pass both of them as args or the whole data struct pointer. What do you think is best?

I would like to have this accepted. For passing the parameter, it is now part of tcp_data struct, but at line server.c:5293, over there, it can be stored as data->query->variable = variable_to_save; and then in the query handler it can use query->variable to access the information. The query->variable is then added to struct query in query.h. Perhaps that can help have the information accessible that is needed.

Everything is ready and uploaded into this PR but in different commits.

1. a87e820
   Fixes a bug in client side where the first connection to server was without SSL. First load cert/key then connect

2. edeca99
   Adds a dedicated tls-auth-port where connections/XFRS must come from authenticated clients

3. 187f093
   Adds tls-auth-xfr-only option to allow XFRS only on tls-auth-port and only from authenticated clients. Works like "tls firewall"

4. a91e67c
   Adds tls-auth in provide-xfr ACL. Server validates the hostname of the connecting client according to tls-auth settings.
   This is to restrict valid certificates and not to accept any cert signed by a valid CA

Future work: These kind of protections can very easily be extended to other connections as well, such as notify, in order for all commucations between primaries/secondaries to be protected by TLS with authentication based on certificates.
This could also work for clients wanting normal traffic, irrelevant to zone transfers.

The changes are looking good! Is it alright if I merge this now, or are there more edits you want to make?

I'm OK.
I wanted to also implement regression tests for these changes, but this can come on a new PR
and probably after a few weeks.

Thanks for looking into this :)