MTA-STS and TLS-RPT: the two records that stop your mail being downgraded to plain text

Mail between servers is encrypted by request, not by rule

Browsers solved this years ago. You type a URL, the server redirects to HTTPS, HSTS pins that choice, and a downgrade to plain HTTP produces a visible error. Server-to-server email never got that treatment.

When one mail server hands a message to another, encryption is negotiated with STARTTLS. The receiving server advertises that it supports TLS, the sender says yes, and the session is encrypted. That negotiation happens in the clear, and the fallback is silent. If the TLS handshake fails, if the certificate does not match, or if an attacker on the path strips the STARTTLS advertisement out of the greeting, the sending server shrugs and delivers your mail in plain text instead. Nobody gets an error. The message arrives. It just arrives readable to anyone on the wire.

That is the gap MTA-STS closes, and the gap TLS-RPT makes visible.

What MTA-STS actually does

MTA-STS (SMTP MTA Strict Transport Security) is defined in RFC 8461. It lets your domain publish a policy with three demands: mail sent here must use TLS, the certificate must be valid, and the certificate must match one of these hostnames. Sending servers that support MTA-STS fetch that policy, cache it, and refuse to deliver if the conditions are not met.

The refusal is the point. Without a policy, a failed TLS handshake becomes a plain text delivery. With an enforced policy, it becomes a deferred message that retries later. You trade a small amount of delivery availability for the guarantee that nothing leaves in the clear.

Gmail, Microsoft 365, and Yahoo all honor MTA-STS policies on outbound mail. Google documents its behavior in the Workspace admin help for MTA-STS. If your domain publishes a valid enforce policy, those senders will apply it.

The three pieces you have to publish

MTA-STS is unusual among email records because it is not DNS alone. It needs DNS and HTTPS together, and the HTTPS half is where most setups break.

1. A DNS TXT record at _mta-sts.yourdomain.com:

_mta-sts.example.com. IN TXT "v=STSv1; id=20261001000000"

The id is a version string you change every time you edit the policy file. Senders use it to decide whether their cached copy is stale. Use a timestamp so it always increases.

2. A policy file served over HTTPS at the exact path https://mta-sts.example.com/.well-known/mta-sts.txt:

version: STSv1
mode: enforce
mx: mail.example.com
mx: *.example-backup.com
max_age: 604800

3. A valid certificate on mta-sts.example.com. The policy is only trusted if it is fetched over HTTPS with a certificate that chains to a public root and covers that hostname. A self-signed cert, an expired cert, or a cert that only covers the apex domain means the policy is ignored entirely, and the sender falls back to opportunistic TLS without telling you.

That third requirement catches people. The subdomain usually points at a static host or a redirect, and it gets forgotten at renewal time. Run mta-sts.yourdomain.com through an SSL check after you set it up, then again whenever the certificate rotates.

Pick the right mode, and do not start with enforce

The mode field takes three values, and the order you use them in matters.

• none means the policy exists but imposes nothing. Use it to retire a policy cleanly rather than deleting the record, which would leave senders holding a cached enforce policy with nothing to refresh against.
• testing means senders evaluate the policy, do not act on failures, and report what they saw through TLS-RPT. Nothing is deferred. Nothing breaks.
• enforce means senders refuse to deliver when the policy is not satisfied.

Start in testing and leave it there for at least a week, ideally two. The reports will tell you whether every MX host in your record presents a valid certificate for the name it is listed under. Backup MX hosts, legacy relays, and anything run by a third party are the usual failures. Moving to enforce with a broken backup MX means mail deferring every time your primary is unavailable, which is exactly when you can least afford it.

max_age is how long senders cache the policy, in seconds. The RFC suggests a long value (604800 is one week) because a cached policy is what protects you against an attacker who can tamper with DNS. Short values weaken that. Set it long once you are confident, and remember that lowering it later takes a full cache cycle to take effect.

TLS-RPT is the half that tells you what happened

MTA-STS without TLS-RPT is a policy you cannot observe. RFC 8460 defines the reporting side, and it is one DNS record:

_smtp._tls.example.com. IN TXT "v=TLSRPTv1; rua=mailto:tlsrpt@example.com"

Sending servers that support it will post a daily JSON summary of every TLS session they attempted with your domain: how many succeeded, how many failed, and why. The failure types are specific and useful. starttls-not-supported means the advertisement was missing or stripped. certificate-expired and certificate-host-mismatch mean exactly what they say. validation-failure covers the rest of the chain problems.

A report looks roughly like this, trimmed:

{
  "organization-name": "Google Inc.",
  "policies": [{
    "policy": { "policy-type": "sts", "policy-domain": "example.com" },
    "summary": { "total-successful-session-count": 4812,
                 "total-failure-session-count": 7 },
    "failure-details": [{
      "result-type": "certificate-host-mismatch",
      "receiving-mx-hostname": "mail2.example.com",
      "failed-session-count": 7
    }]
  }]
}

Seven failures against one backup MX is the entire reason to run testing first. Fix that certificate, confirm a clean week of reports, then switch to enforce.

You can point rua at a mailbox you read, but the JSON is gzipped and the volume adds up. Most teams send it to a dedicated address and parse it, or use a hosted TLS-RPT processor. Either way, publish the record before you publish the policy. Reports from a testing policy are the data you need, and they only exist if something is listening.

Where this sits next to SPF, DKIM, and DMARC

These records answer different questions, and having one does not substitute for another.

SPF, DKIM, and DMARC answer "is this message really from the domain it claims?" They protect your recipients from spoofing, and they are what Gmail and Yahoo check against their bulk sender requirements.

MTA-STS and TLS-RPT answer "was the connection carrying this message encrypted and verified?" They protect the message in transit, to and from your domain, against passive interception and active downgrade.

A domain with a strict DMARC policy and no MTA-STS is still delivering readable mail any time a handshake fails. A domain with MTA-STS and a DMARC policy of none is encrypting mail that anybody can forge. You want both, and the authentication side is usually the one worth fixing first because it has the larger effect on whether your mail arrives at all.

Check what you are actually publishing

Three things are worth verifying after any change: that the TXT records resolve consistently everywhere, that the policy file is reachable over HTTPS with a valid certificate, and that the MX hostnames in your policy match what your DNS actually returns.

The DNS lookup tool queries 20 global DNS resolvers, which matters here because a record that propagated to your local resolver has not necessarily propagated everywhere a sending server might ask. The email check tool covers SPF, DKIM across 16 common DKIM selectors, DMARC, and your MX configuration, and screens your sending IPs against 77 verified active email blacklists.

For the whole picture, a full scan at acuityscan.com runs 350+ checks across 8 modules, including the email, DNS, and SSL modules that cover every piece described above in a single pass.

TL;DR

• STARTTLS fails open. A stripped or failed handshake means your mail is delivered in plain text, silently.
• MTA-STS (RFC 8461) needs three things: a _mta-sts TXT record, a policy file at https://mta-sts.yourdomain.com/.well-known/mta-sts.txt, and a valid certificate on that subdomain.
• Start in mode: testing. Run it for a week or two, confirm every MX host passes, then move to enforce.
• TLS-RPT (RFC 8460) is one TXT record at _smtp._tls and is what makes the testing period useful. Publish it first.
• Bump the policy id every time you edit the policy file, or senders keep serving the cached version.
• MTA-STS protects mail in transit. It does not replace SPF, DKIM, or DMARC, which protect against forgery.