How can I accurately test and measure email deliverability and sender reputation?

Summary

Accurately testing and measuring email deliverability and sender reputation requires a multifaceted approach that combines proactive and reactive strategies. Traditional methods like probe-account testing are deemed inaccurate, emphasizing the need for comprehensive evaluation. Key strategies include proper email authentication (SPF, DKIM, DMARC), gradual IP warming, and maintaining a clean email list by removing unengaged subscribers. Monitoring key metrics such as open rates, click-through rates (CTR), spam complaints, and IP blacklists is essential for identifying potential deliverability issues. Utilizing tools like Google Postmaster Tools and Microsoft SNDS for reputation monitoring is highly recommended. It's also important to be cautious about using seed lists due to potential inaccuracies and to be aware of how ESPs handle seed addresses with low engagement. Sender reputation, directly impacting deliverability, is influenced by authentication, engagement, and complaint rates. Finally, understand IP and domain reputation aren't independent.

Key findings

  • Probe Inaccuracy: Probe-account based deliverability measures are no longer accurate.
  • Sequential Testing Issues: Sequential testing changes reputation filters, affecting accuracy.
  • Reputation Interdependence: Domain and IP reputation are not independent.
  • Tool Recommendations: Google Postmaster Tools and Microsoft SNDS are important for monitoring.
  • Authentication Importance: SPF, DKIM, and DMARC are essential for authentication.
  • Clean List Benefits: Maintaining a clean email list reduces bounce rates and spam complaints.
  • Reputation Impact: Sender reputation directly affects deliverability.
  • Gradual IP Warming: Gradual IP warming improves reputation.

Key considerations

  • Cautious Seed Use: Use seed lists with caution due to potential inaccuracies.
  • ESP Seed Handling: Be aware of how ESPs handle seed addresses.
  • Holistic Strategy: Use a holistic strategy combining monitoring and proactive practices.
  • Blacklist Monitoring: Regularly check and address IP blacklisting.
  • Reputation Management: Actively manage and protect sender reputation.
  • Suspicious Patterns: Avoid sending patterns that trigger alerts.
  • Industry Standards: Be aware that some ESPs do not adhere to general standards.

What email marketers say
11Marketer opinions

Accurately testing and measuring email deliverability and sender reputation involves a multi-faceted approach, incorporating both proactive and reactive measures. Proactive measures include using seed lists and deliverability testing tools to assess inbox placement across various providers. Ensuring proper email authentication (SPF, DKIM, DMARC), warming up new IP addresses gradually, and maintaining a clean email list by removing unengaged subscribers are vital. Reactive measures involve monitoring open rates, click-through rates, IP blacklists, feedback loops, and sender reputation using tools like Google Postmaster Tools and Microsoft SNDS. It's crucial to understand that a good sender reputation, influenced by engagement, authentication, and complaint rates, directly impacts deliverability. Seed lists, while helpful, should be used cautiously due to potential inaccuracies. Understanding ESP behavior concerning seeds is also important, specifically regarding low engagement and account shutdowns, and how they may be related to the seed provider rather than actual recipient behavior.

Key opinions

  • Testing Tools: Deliverability testing tools provide insights into inbox placement across different providers.
  • Email Authentication: Proper authentication (SPF, DKIM, DMARC) is crucial for legitimate email recognition.
  • IP Warming: Gradually warming up new IPs helps build a positive sender reputation.
  • List Hygiene: Maintaining a clean email list improves engagement and sender reputation.
  • Metric Monitoring: Monitoring open rates, CTR, and spam complaints provides insights into deliverability performance.
  • Reputation Monitoring: Sender reputation is a key factor influencing deliverability; monitor using provided tools.
  • Seed Lists: Seed lists provide indication on the placement of emails in inboxes, but can be inaccurate.

Key considerations

  • Seed List Accuracy: Seed list results may not accurately reflect real-world deliverability due to various factors.
  • ESP Seed Handling: ESPs may disable or bounce seed addresses due to low engagement or other concerns.
  • Holistic Approach: A comprehensive approach combining proactive and reactive measures yields the most accurate results.
  • Reputation Impact: Sender reputation is determined by engagement, authentication, and spam complaints.
  • IP Blacklists: Regularly check and address IP blacklistings to maintain deliverability.
  • Feedback Loops: Utilize feedback loops to identify and address problematic sending practices.
Marketer view

Email marketer from SendPulse shares to track email open rates and click-through rates (CTR) to gauge engagement levels. Low open rates might indicate deliverability issues, while low CTR could point to content problems. Monitoring these metrics over time will help you see if your deliverability and content is getting better or worse.

April 2022 - SendPulse
Marketer view

Email marketer from Email on Acid explains that regularly cleaning your email list by removing unengaged subscribers reduces bounce rates and spam complaints. Focusing on engaged subscribers improves your sender reputation and overall deliverability.

November 2023 - Email on Acid
Marketer view

Email marketer from Litmus shares to ensure your emails are properly authenticated using SPF, DKIM, and DMARC. These authentication protocols help verify that your emails are legitimate and reduce the chances of them being marked as spam. Correctly configured authentication also improves your sender reputation.

February 2024 - Litmus
Marketer view

Email marketer from StackOverflow shares to regularly check your IP address against known email blacklists. If your IP is blacklisted, take immediate action to get it removed. Being on a blacklist can significantly impact your email deliverability.

September 2023 - StackOverflow
Marketer view

Marketer from Email Geeks answers that shutdowns from ESP due to low engagement of seedlist, this might have more to do with the fact they are Glock seeds...although there is no way for us to know for sure

February 2023 - Email Geeks
Marketer view

Email marketer from TalosIntelligence says sender reputation is a score assigned to an IP address and domain that sends email, determining the likelihood that its messages are seen as spam. A good sender reputation leads to higher deliverability and is influenced by authentication, spam complaints, and engagement metrics.

February 2024 - TalosIntelligence
Marketer view

Email marketer from Reddit's r/emailmarketing explains that if you are using a new IP address, it's vital to warm it up gradually. Start by sending small volumes to your most engaged subscribers and slowly increase the volume over time to build a positive reputation with ISPs.

July 2021 - Reddit
Marketer view

Email marketer from Mailtrap Blog shares that using seed lists involves sending test emails to a list of email addresses owned by the tester. The placement of these emails in inboxes (or spam folders) indicates deliverability performance. However, seed lists should be used cautiously, as they may not accurately reflect real-world results due to factors like engagement and spam filtering.

August 2021 - Mailtrap Blog
Marketer view

Marketer from Email Geeks explains if the seed is disabled due to low engagement that has nothing to do with the ESP, that has to do with the sender. ESPs also don't bounce email to seeds for the hell of it, it would be due to the actual address bouncing or the address is globally suppressed due to concerns about the provider of the seed.

July 2023 - Email Geeks
Marketer view

Email marketer from GlockApps explains that using deliverability testing tools is the most efficient way to check where your emails land: inbox, spam, or missing. These tools send your email to real email addresses across different mailbox providers and give a comprehensive report on your email placement.

May 2022 - GlockApps
Marketer view

Email marketer from SparkPost explains feedback loops (FBLs) as a mechanism where ISPs forward complaints about spam to the sender. Monitoring and acting on FBL reports allows you to identify problematic sending practices and remove complainants from your list, improving your sender reputation and deliverability.

May 2024 - SparkPost

What the experts say
9Expert opinions

Accurately testing and measuring email deliverability and sender reputation requires a comprehensive understanding of various factors. Traditional probe-account methods are no longer reliable due to their limited scope and inability to replicate real-world sending patterns. Sequential testing is problematic, as initial tests can alter reputation filters. IP and domain reputations are intertwined, meaning actions affecting one influence the other. Maintaining a clean email list and consistent monitoring of IP and domain reputation using tools like Google Postmaster Tools and Microsoft SNDS are crucial for identifying deliverability issues. Shutting down entire accounts is a sign of highly suspicious sending patterns. Sender reputation impacts deliverability directly, with positive scores leading to better inbox placement, while negative scores result in spam filtering or blocking. Finally, addressing bounced addresses is critical.

Key opinions

  • Inaccurate Probes: Probe-account based deliverability measurements are no longer accurate.
  • Sequential Testing Issues: Sequential testing in email is unreliable due to reputation filter changes.
  • Intertwined Reputations: IP and domain reputations are not independent; they affect each other.
  • List Maintenance Crucial: Clean email lists are essential for measuring and maintaining good reputation.
  • Monitoring is Key: Consistent monitoring of IP and domain reputation is vital for identifying issues.
  • Sending Patterns Matter: Suspicious sending patterns can lead to account shutdowns.
  • Reputation Impact: Sender reputation directly affects email deliverability (inbox vs. spam).

Key considerations

  • Reputation Management: Focus on building and maintaining a positive sender reputation.
  • Blacklisting: Identify and resolve potential blacklisting issues promptly.
  • ESP Standards: Be aware that some ESPs might not adhere to general industry standards.
  • Holistic Approach: Employ a multi-faceted approach to testing and measuring deliverability.
  • Unusual Patterns: Avoid sending patterns that may trigger 'this is not a real customer' detectors.
Expert view

Expert from Email Geeks explains that probe-account based measures of deliverability are no longer accurate measures of actual deliverability. You can't measure deliverability with a small number of emails or recipients because it's a statistical thing and you're not providing a mailstream with a reputation.

May 2021 - Email Geeks
Expert view

Expert from Spamresource explains that sender reputation directly impacts deliverability, where ISPs assign reputation scores to senders based on factors like authentication, complaint rates, and sending volume. A positive reputation leads to better inbox placement, while a negative reputation results in emails being filtered to spam or blocked altogether.

May 2023 - Spamresource
Expert view

Expert from Email Geeks shares links to documents containing thoughts on reputation (<https://docs.google.com/document/d/1DsQTAcDt7u-UZd3vf7S_MWoiEtaxzmiScJgh75hMhEQ/edit#heading=h.lusqs66ahnq2>, <https://docs.google.com/document/d/1twW4H7nhn3gdiF5K6vObiWypUkPiVhCtC9xWN70GEkA/edit#heading=h.lusqs66ahnq2>, <https://docs.google.com/document/d/1UL7URneTI1UlxDOFq4Zo5vSJHWBOiPUmnURcQsx_IDo/edit#heading=h.lusqs66ahnq2>, and <https://docs.google.com/document/d/1f5SV0_nwT0Sz6WDz8K4vB84KlVRk8d_2h0NhxumwnaM/edit#heading=h.lusqs66ahnq2>).

February 2023 - Email Geeks
Expert view

Expert from Email Geeks shares that domains sending from low reputation IP addresses can damage their domain reputation, and vice versa.

October 2023 - Email Geeks
Expert view

Expert from Word to the Wise shares that consistent monitoring of your sending IP and domain reputation using tools like Google Postmaster Tools, Microsoft SNDS, and third-party services is crucial for identifying deliverability issues. Monitoring allows for early detection of blacklistings, spam complaints, or authentication failures that may affect inbox placement.

October 2021 - Word to the Wise
Expert view

Expert from Spamresource shares the importance of maintaining a clean email list to accurately measure deliverability. Regularly remove inactive subscribers, handle bounces promptly, and process unsubscribe requests to prevent low engagement from skewing deliverability metrics and negatively impacting sender reputation.

February 2024 - Spamresource
Expert view

Expert from Email Geeks shares shuts down the entire account - that tells her that they thought the sender was doing something totally hinky and they tripped their alerts and automated "this is not a real customer" detectors. Given they're trying to test 'deliverability' that tells her that their sending patterns are, ultimately, not 'normal.' That also contributes to the problematic nature of their claimed results.

July 2021 - Email Geeks
Expert view

Expert from Email Geeks shares that there is no way to sequentially test things in an email space because the first test will change the reputation filters, so the second test will not be starting with the same baseline. The underlying assumption that domain reputation and IP reputation are independent and independently measurable is not a good model of reputation.

July 2021 - Email Geeks
Expert view

Expert from Email Geeks says she has had discussions with industry groups and ESPs about removing addresses that bounce from one customer from all lists and thinks that's universally seen as a bad idea. She's unsure what ESPs Robert is using, but says they're not adhering to general standards.

November 2023 - Email Geeks

What the documentation says
4Technical articles

Accurately testing and measuring email deliverability and sender reputation involves using various tools and protocols. Google Postmaster Tools can monitor sending reputation with Gmail users through metrics like spam rate, feedback loop complaints, and authentication status. Microsoft SNDS provides data about sending IP addresses, including complaint rates and spam trap hits for Microsoft services like Outlook and Hotmail. Implementing DomainKeys Identified Mail (DKIM) verifies sender authenticity by associating a signature with email messages. Sender Policy Framework (SPF) prevents sender address forgery by specifying authorized mail servers for a domain.

Key findings

  • Google Postmaster Tools: Provides metrics for monitoring sender reputation with Gmail users.
  • Microsoft SNDS: Offers data on sending IP addresses for Microsoft email services.
  • DKIM Protocol: Verifies sender authenticity through digital signatures.
  • SPF Protocol: Prevents sender address forgery by specifying authorized mail servers.

Key considerations

  • Gmail Monitoring: Utilize Google Postmaster Tools to maintain a high sender reputation with Gmail.
  • Microsoft Monitoring: Monitor Microsoft SNDS data to identify and fix deliverability issues with Outlook/Hotmail.
  • DKIM Implementation: Implement DKIM to build trust and ensure message integrity.
  • SPF Implementation: Implement SPF to prevent sender address forgery and improve deliverability.
Technical article

Documentation from Microsoft SNDS explains the Sender Network Data Services (SNDS) program which provides data about your sending IP addresses, including complaint rates and spam trap hits. Monitoring SNDS data can help identify and fix deliverability issues when sending to Microsoft email services like Outlook and Hotmail.

March 2023 - Microsoft SNDS
Technical article

Documentation from Google Postmaster Tools describes how to use their tools to monitor your sending reputation with Gmail users. They provide metrics like spam rate, feedback loop complaints, and authentication status that are helpful to keep your sender reputation high with Gmail users.

July 2024 - Google Postmaster Tools
Technical article

Documentation from RFC-Editor defines Sender Policy Framework (SPF) as a mechanism to prevent sender address forgery, where domain owners specify which mail servers are authorized to send email on behalf of their domain. Email receivers can verify that emails claiming to be from a specific domain are sent from an approved server, improving deliverability and trust.

January 2023 - RFC-Editor
Technical article

Documentation from RFC-Editor defines DomainKeys Identified Mail (DKIM) as a mechanism that allows a domain to associate a signature with an email message, permitting a recipient to verify the sender's authenticity. This helps ensure the message wasn't altered in transit and builds trust.

November 2021 - RFC-Editor