How can I identify bot user agents in my email click data?
Summary
What email marketers say12Marketer opinions
Email marketer from Reddit recommends using JavaScript to check for human-like behavior (e.g., mouse movements, typing) and then send this data to the server for bot detection, which can complement user agent analysis.
Email marketer from Reddit suggests blocking known bot user agents at the server level to prevent them from accessing your site and skewing your analytics, and he also suggests blocking common headless browser user agents.
Email marketer from CDNetworks, suggest using a variety of methods to identify and block bots - including CAPTCHA, HTTP Headers, rate limiting, and Javascript challenges.
Email marketer from Email Geeks explains that the "Amazon CloudFront" user agent is often associated with domains that have MX records pointing to protection.outlook.com.
Email marketer from Distil Networks, now part of Imperva, shares a variety of user agents and bot characteristics, as well as how to block them using a variety of methods.
Email marketer from Email Geeks explains that the python-requests user agent indicates someone is running a script against your link, using the Requests package.
Email marketer from Cloudflare, suggests using a bot management system to identify bots. Bot management systems use machine learning, behavior detection and other techniques to identify likely bots.
Email marketer from Digital Authority Partners, suggests using a combination of user-agent analysis, HTTP request headers and frequency of user interactions. Also suggests using honeypot traps to catch bad bots.
Email marketer from Stack Overflow suggests using a combination of techniques like checking for headless browsers (e.g., PhantomJS), analyzing user agent strings (looking for known bot patterns), and monitoring behavior (e.g., rapid clicks or page visits) to identify bots.
Email marketer from Medium, suggests using techniques such as rate limiting, CAPTCHAs, user-agent analysis, anomaly detection, and IP reputation scoring to mitigate bot traffic.
Email marketer from Email Geeks shares links to resources for finding valid user agents: useragents.me, explore.whatismybrowser.com, and deviceatlas.com.
Email marketer from Email Geeks shares a list of user agents that don't start with `Mozilla` and are likely bots, based on data from one client over a few weeks.
What the experts say2Expert opinions
Expert from Spam Resource explains that you can monitor for suspicious activities like unusually high click rates, multiple clicks from a single IP in a short period, or clicks happening outside of normal business hours to discover bot activity.
Expert from Spam Resource suggests filtering out known bot user-agents and analyzing the data of the click source. User agents that are not valid or those from TOR networks can be deemed bots.
What the documentation says5Technical articles
Documentation from Google defines invalid traffic (IVT) as clicks or impressions generated by illegitimate means, including automated bots and crawlers, and discusses filtering IVT to improve data accuracy in analytics and advertising platforms.
Documentation from Amazon AWS explains how to prevent bot access to your websites, APIs, and mobile applications. Includes recommendations for using AWS WAF Bot Control.
Documentation from OWASP outlines various bot detection techniques, including analyzing HTTP headers (user agent, referer), JavaScript challenges (testing browser capabilities), CAPTCHAs, and behavioral analysis (e.g., request rates, navigation patterns).
Documentation from Imperva details advanced bot protection methods like behavioral analysis, device fingerprinting, and reputation analysis, noting that sophisticated bots can mimic human behavior, requiring multi-layered detection strategies.
Documentation from DataDome explains how to block bad bots using javascript bot detection. It explains how to differentiate bots from real human users to prevent malicious activity.