Why do some ESPs require unnecessary SPF includes for DKIM, and what are the dangers of this practice?

Summary

Some ESPs require unnecessary SPF includes due to outdated practices, marketing tactics, or a misguided attempt at security. This practice is dangerous because it can lead to exceeding the SPF DNS lookup limit (10 lookups), causing SPF authentication to fail. This negatively impacts email deliverability, domain reputation, and DMARC compliance. In addition, lots of includes mean that it becomes harder to update, maintain and monitor your SPF records. While DKIM and SPF both authenticate email, unnecessary SPF includes don't directly improve DKIM. SPF flattening, by replacing includes with IPs, can mitigate this. It's recommended to keep the list of sending sources up-to-date and review who is sending mail on your behalf.

Key findings

  • Outdated Practices: Some ESPs rely on outdated information and misunderstandings about modern authentication practices.
  • Marketing Tactic: Some ESPs require includes for marketing purposes, marking their presence in the client's SPF record.
  • DNS Lookup Limit: The SPF standard has a 10 DNS lookup limit, easily exceeded by unnecessary includes, causing authentication failures.
  • Security Theatre: The inclusion of SPF records can sometimes be 'security theatre,' giving a false sense of enhanced security without any practical effect.
  • DMARC & Reputation Impact: SPF failures due to excessive includes can negatively impact DMARC compliance and damage the sender's domain reputation.
  • DKIM Independence: DKIM functions independently of SPF includes, making unnecessary SPF includes irrelevant for DKIM.
  • Authentication Issues: Ignorance or poor training at some ESPs leads to misunderstandings surrounding authentication.
  • Maintainability Concerns: Too many includes become difficult to update, maintain, and monitor.

Key considerations

  • Question ESP Requirements: Challenge the necessity of SPF include requests from ESPs and understand the reasoning behind them.
  • Flatten SPF Records: Use SPF flattening techniques (replacing includes with IPs) to reduce DNS lookups.
  • Regular Monitoring: Monitor your SPF records regularly to ensure they don't exceed the DNS lookup limit.
  • Review Sending Sources: Maintain a clear list of authorized sending sources. Avoid including domains that aren't actually sending mail on your behalf.
  • Validate Records: Use tools to validate the SPF records
  • Staying Informed: Stay informed on best practices and changes in the world of SPF records.

What email marketers say
10Marketer opinions

Some ESPs require unnecessary SPF includes, which is often due to outdated information, marketing tactics, or a desire to appear secure. This practice is dangerous because it can lead to exceeding the SPF DNS lookup limit, causing SPF authentication to fail. This failure can negatively affect email deliverability, domain reputation, and DMARC compliance. While DKIM and SPF work together for authentication, unnecessary SPF includes don't directly improve DKIM. The 'include' directive is useful when you have multiple senders, but there is risk of surpassing the DNS lookup limit. So you need to manage this efficiently, by keeping an eye on the DNS lookups.

Key opinions

  • Outdated Practices: Some ESPs require unnecessary SPF includes due to outdated information or a misunderstanding of current best practices.
  • Marketing Tactic: ESPs may require SPF includes as a way to mark their presence and boost market share, which benefits them more than the sender.
  • DNS Lookup Limit: Too many SPF includes can exceed the 10 DNS lookup limit, causing SPF authentication to fail and harming deliverability.
  • Security Theatre: Requesting unnecessary SPF includes is sometimes 'security theatre,' creating complexity without genuinely improving security.
  • DMARC Impact: Failing SPF authentication due to excessive includes can negatively affect DMARC compliance, harming email deliverability and brand reputation.
  • DKIM Independence: Unnecessary SPF includes do not improve DKIM authentication, as DKIM operates differently.

Key considerations

  • Monitor DNS Lookups: Regularly monitor your SPF record to ensure you're not exceeding the DNS lookup limit.
  • Flatten SPF Records: Consider flattening your SPF record by replacing 'include' statements with explicit IP addresses to reduce DNS lookups.
  • Evaluate ESP Requirements: Question the necessity of SPF include requests from your ESP and explore alternative authentication methods if possible.
  • Manage Multiple Senders: Be vigilant in managing your includes. Keep an eye on DNS lookups, and remove includes you don't need.
  • Domain Reputation: Ensure SPF passes to protect your domain reputation
Marketer view

Email marketer from Email Deliverability Experts Group, says that the 'include' directive is useful when you have multiple senders, but there is risk of surpassing the DNS lookup limit. So you need to manage this efficiently, by keeping an eye on the DNS lookups.

April 2021 - Email Deliverability Experts Group
Marketer view

Email marketer from Stack Overflow shares that too many SPF includes can exceed the 10 DNS lookup limit. This will cause SPF authentication to fail, impacting email deliverability. It’s recommended to flatten SPF records to avoid this.

July 2023 - Stack Overflow
Marketer view

Email marketer from Email Deliverability Blog responds that unnecessary SPF includes can indirectly affect DMARC compliance because DMARC relies on SPF and DKIM. If SPF fails due to too many lookups, it can cause DMARC to fail as well, negatively impacting email deliverability and brand reputation.

July 2021 - Email Deliverability Blog
Marketer view

Email marketer from Email Security Forum shares that sometimes the request for unnecessary SPF includes is 'security theatre' – it looks good but doesn't genuinely improve security. It can create unnecessary complexity and potential points of failure.

May 2021 - Email Security Forum
Marketer view

Marketer from Email Geeks complains about ESPs demanding a DMARC record (p=none) and absorbing all reports without informing the client, especially when enforcing a policy.

September 2023 - Email Geeks
Marketer view

Email marketer from Reddit responds that some ESPs might require unnecessary SPF includes due to outdated information or a desire to demonstrate 'security'. However, this can lead to SPF failing if limits are exceeded, and there are better ways to manage authentication.

May 2021 - Reddit
Marketer view

Email marketer from Email Marketing Tips Blog explains that while DKIM and SPF both authenticate email, they work differently. DKIM uses cryptographic signatures and does not rely on DNS lookups in the same way SPF does, so unnecessary SPF includes don't directly improve DKIM.

September 2023 - Email Marketing Tips Blog
Marketer view

Email marketer from EmailGeeks Forum explains that some ESPs require SPF includes as a way of marking their presence/market share. If they include a domain in your records, the ESP gets to include you as their client for reporting purposes.

August 2021 - EmailGeeks Forum
Marketer view

Marketer from Email Geeks believes that some ESP authentication issues arise from ignorance and poor internal training leading to misunderstandings.

February 2023 - Email Geeks
Marketer view

Email marketer from Email Marketing Community shares that failing SPF authentication can damage your domain's reputation. Mailbox providers may start filtering your emails to spam or block them altogether, negatively affecting your email program.

August 2024 - Email Marketing Community

What the experts say
4Expert opinions

Some ESPs mandate unnecessary SPF includes for DKIM due to outdated practices, marketing strategies, or a desire to show 'security'. This is a bad practice that can lead to exceeding the DNS lookup limit, causing SPF to fail, harming email delivery. Additionally, the growing number of includes makes maintenance difficult, and can lead to errors. It's also noted that Microsoft has shifted from breaking SPF to breaking DKIM.

Key opinions

  • Outdated Practices: Some ESPs follow old documentation, requiring SPF includes even though they are no longer necessary (e.g., Microsoft's old SPF lookup requirements).
  • Marketing Tactic: ESPs may require includes to market their service, indicating that they sent the email on behalf of the client.
  • DNS Lookup Limit: Using includes can easily exceed the DNS lookup limit in SPF records, causing authentication failure and harming email deliverability.
  • Maintenance Nightmare: More includes make SPF records harder to maintain, update, and monitor, increasing the risk of errors.
  • Authentication Shift: Microsoft moved from breaking SPF to breaking DKIM.

Key considerations

  • Question ESP Requirements: Carefully evaluate and question ESP's SPF inclusion requirements, as they might be unnecessary or detrimental.
  • Monitor SPF Records: Regularly monitor SPF records to ensure they remain within the DNS lookup limit.
  • Simplify SPF Records: Consider flattening SPF records to reduce the number of includes and DNS lookups.
  • Stay Updated: Be informed on changes to authentication standards, such as current requirements and Microsoft updates.
Expert view

Expert from Word to the Wise, Laura Atkins, explains that some ESPs require the inclusion because they are trying to market the fact that they sent the email. There is an impact as you get more and more includes in your SPF record. This becomes a maintenance nightmare, as it's difficult to update and monitor. It can also lead to errors.

November 2023 - Word to the Wise
Expert view

Expert from Email Geeks shares that Microsoft moved from breaking SPF to breaking DKIM by modifying the body content of emails.

February 2024 - Email Geeks
Expert view

Expert from Spam Resource, John Levine, explains that while using includes can be convenient for SPF records, they can quickly lead to exceeding the DNS lookup limit. This can cause SPF authentication to fail, harming email delivery.

February 2025 - Spam Resource
Expert view

Expert from Email Geeks explains about an ESP that won’t turn on custom DKIM signing for a client until the client publishes an SPF include in their root domain, indicating this is bad practice. She highlights that many still follow old documentation with this recommendation, even though Microsoft stopped requiring SPF lookups on the 5321.

November 2023 - Email Geeks

What the documentation says
5Technical articles

The SPF 'include' mechanism authorizes other domains to send mail on your behalf. However, overusing 'include' can lead to exceeding the SPF DNS lookup limit (10 lookups), causing SPF to fail. This negatively impacts deliverability, potentially causing mail to bounce or be marked as spam. SPF flattening, which replaces 'include' statements with actual IP addresses, can mitigate this risk. It's best practice to only include domains actually sending mail on your behalf, avoid unnecessary includes, and validate your SPF record to ensure proper authentication and syntax.

Key findings

  • SPF 'include' Purpose: The SPF 'include' mechanism authorizes other domains to send mail on behalf of your domain.
  • DNS Lookup Limit: RFC specifies a limit of 10 DNS lookups within an SPF record; exceeding this limit results in a 'PermError' and impacts deliverability.
  • SPF Flattening Solution: SPF flattening (replacing 'include' statements with IP addresses) reduces DNS lookups and avoids exceeding the limit.
  • Syntax Matters: Incorrect SPF syntax, including too many includes, causes authentication to fail.
  • Best Practices: Best practice is to only include necessary domains, avoid unnecessary includes, and flatten records to optimize deliverability.

Key considerations

  • Limit 'include' Usage: Be mindful of how many 'include' statements are in your SPF record.
  • Implement SPF Flattening: Consider SPF flattening to reduce DNS lookups.
  • Validate SPF Records: Use tools to validate your SPF record syntax and ensure proper authentication.
  • Regularly Review SPF: Regularly review and update your SPF record to ensure it only includes authorized sending sources.
Technical article

Documentation from dmarcian.com explains SPF flattening as a method to reduce the number of DNS lookups by replacing 'include' statements with the actual IP addresses. This avoids exceeding the lookup limit and ensures proper SPF authentication.

May 2023 - dmarcian.com
Technical article

Documentation from Mailjet shares that best practice is to only include the domains that are actually sending mail on your behalf. Avoid adding unnecessary 'include' statements and flatten your SPF record where possible to remain under the DNS lookup limit and optimize deliverability.

March 2023 - Mailjet
Technical article

Documentation from RFC Editor explains that the SPF 'include' mechanism is used to authorize other domains to send mail on behalf of your domain. While useful in some scenarios, overusing 'include' can lead to exceeding the SPF DNS lookup limit, causing SPF to fail and potentially harm deliverability.

November 2021 - RFC Editor
Technical article

Documentation from Valimail explains that the RFC specifies a limit of 10 DNS lookups within an SPF record. Going over this limit means that SPF will return a 'PermError' result, which will negatively impact deliverability and can cause mail to bounce or be marked as spam.

December 2021 - Valimail
Technical article

Documentation from EasyDMARC explains that incorrect SPF syntax, including too many includes, can cause authentication to fail. They recommend using tools to validate your SPF record and minimize includes to ensure proper authentication.

September 2021 - EasyDMARC