Jan Žorž of Internet Society has written a text describing results of lab tests of DANE and SMTP at Go6Labs. The findings seems to be that if either of the MX or the TLSA records in the DNS is unsigned, then delivery of SMTP will not happen over the TLS protected connection if the cert is self-signed.
I think that is wrong.
For me SMTP delivery consists of two steps:
- Resolution of the MX
- Management of the SMTP connection
I think the two are separable from each other, and that seems to be where there is disagreement (and question on how it is actually implemented, for example in postfix).
Resolution of the MX takes the domain name of the target email address, and looks up MX record. The target of the MX record is the host that later one should open the SMTP connection to. This resolution to me uses normal DNSSEC rules for success. The record is to be either unsigned or signed, and if it is (to be) signed the validation is to succeed. Otherwise the result of the lookup is to be ignored, and delivery not possible. I do not think fallback to delivery to A/AAAA is correct if MX exists and validation of MX fails. Only if MX is missing fallback to A/AAAA is acceptable.
Second step is to deliver the email over an SMTP connection. A/AAAA is to be looked up (and validated if signed). On top of that TLSA record is to be looked up for _426._tcp.mail.example.com (if the target of the mx is mail.example.com). If the A/AAAA and TLSA records exists, and they are signed, then the DANE specification says the cert is to be trusted and can be used for the TLS connection.
I think that is something that is correct, and I think it should happen regardless of whether the MX record in the first step was signed or not. I think it should be enough that the MX in the first step is passing the requirements (either be signed and correctly validated, or not signed at all).