Tl;Dr:
As email moved from self-hosted to mailbox providers to white-label hosting with custom domains, both Open-Source MTAs and Commercial MTAs have implemented workarounds to identify and queue the different domains that are served by the same set of mail servers at a given provider, but have fallen short of a proper solution because they were all engineered around one queue per destination domain.
KumoMTA was built better, architected natively around the concept of MX rollups, allowing for admins to easily and automatically ensure that all messages destined for the same set of servers are treated as a single queue and throttling granularity.
Read on to learn about this challenge and how we've designed a better solution.
Early SMTP Was Simple
Anyone who has been working with email long enough will eventually pound their head against the wall of outdated systems and standards; email is ancient in Internet terms, and since there has always been a focus on backward compatibility, the standards and systems used to handle SMTP have not progressed as quickly as some other fundamental systems.
One example of this is the use of MX records: in the early days of email each organization hosted its own MTAs with their own MX records, and it was straightforward for systems to send to each other. Back then there was no concept of traffic shaping because email marketing hadn't been discovered, so at most a server would queue messages for each destination domain separately (this was also before the time of IP reputation and the need for an MTA to send across multiple IPs).
Most Open-Source MTAs were developed in this era, and even the early commercial MTA solutions operated on the assumption that each receiving entity ran its own MTAs and had unique MX records.
International Providers Changed the Landscape
Over time hosted mailbox services became more common, especially as the primary user base of the Internet moved from researchers to the general public, and those members of the general public needed to have an email address but did not have their own domain.
Suddenly we had large numbers of recipients at domains like AOL, Hotmail, and Gmail, but this did not add too much complexity to the queueing and throttling of messages, as each of the Mailbox Providers (MBPs) still used their own domains, and ran their own MTAs.
One early challenge created by the larger MBPs was around international operations: Some of the large MBPs were using country-level domains to help make their systems more friendly to international users, while still using a single set of MTAs for traffic across their entire networks. This meant that we started to see the previous assumption of one organization - one set of MTAs be broken, meaning that rather than having one queue and one set of traffic shaping rules to talk to a set of MTAs, without some kind of change the sending MTA might have a dozen queues and traffic shaping counters for the same remote MX group.
For example, let's say Hotmail is operating hotmail.com, hotmail.ca, hotmail.co.uk, and hotmail.jp: when there was only hotmail.com, an MTA would only have one queue and one set of traffic shaping counters, and if the admins of hotmail.com said to only open 20 connections at a time, the MTA would open 20 connections at most and send all the mail queued together for hotmail.com across those connections. Once the additional three international domains are introduced, the MTA will maintain four queues four sets of traffic shaping counters, one per domain, resulting in 80 connections to the Hotmail servers unless an administrator manually divides up the connection limits.
To address this development, the concept of MX Rollups was implemented by developers. An MX rollup (by whatever name is selected by the developer) will either duplicate or aggregate configuration and queues, depending on the implementation. When the rollup serves simply as an aliasing mechanism, it allows the MTA admin to define a regex or list of domains that all receive the same rules, but maintain their own queues and counters. When the rollup serves as an aggregation (the more common case), the regex or list results in a single queue, with a single set of traffic-shaping rules and counters. The latter solution was a great solution for addressing receiving systems that host across multiple domains, with a known set of domains.
Whitelabel Mailbox Providers Changed The Landscape Again
The next challenge to properly managing queues and throttles came when hosting providers started supporting white-label email hosting, where the user's domain is used for the email address rather than the provider's domain. Fundamentally the challenge is the same as the provider using multiple domains and country suffixes, except for one thing: there is no easy way to know which domains are hosted by which providers.
This presents a significant challenge for some MTA solutions: Not only do you have to know what all the top-level domains for a provider are and make sure to include them in your MX Rollup list, you somehow have to know what all the domains are that use a given provider.
Some MTA solutions have enabled a workaround for this: rather than define the traffic shaping rules for a domain, you create a list of all the MX hosts at the provider and write rules against it. The challenge is that it requires manual effort: the MailOps/Deliverability engineer has to gather up the hostnames and maintain the list. If for example, Microsoft adds new MX hosts or domains, you have to update the list to account for them.
Site Names Solve The MX Rollup Problem Automatically
When we created KumoMTA we knew that we wanted to find a better solution to certain common issues faced by users of existing Open-Source and Commercial MTA platforms, and MX Rollup was one of them.
Rather than queueing messages by destination domain, we designed KumoMTA to natively queue in a Ready Queue using a combination of the source (typically sending IP address) and the SiteName, an automatically generated representation of all the MX hosts that match the message's sending domain.
For example, at the time of writing gmail.com's MX records resolve to this list of names: gmail-smtp-in.l.google.com, and alt1.gmail-smtp-in.l.google.com through alt4.gmail-smtp-in.l.google.com. KumoMTA merges those names into a site name that looks similar to a regular expression: (alt1|alt2|alt3|alt4)?.gmail-smtp-in.l.google.com.
Any other domain that returns that same list of MX hosts with the same priority order will also produce the same site name string. For instance, googlemail.com also has that same set of MX hosts, so it will also produce the same site name. Many other domains that are hosted by Google will also expand to this same site name.
The site name, together with the source, is used to locate and/or construct the ready queue for that immediately-ready-to-send mail. As an administrator, you can easily apply your per-source-per-destination-site traffic shaping constraints to the ready queue.
This means that while you may be sending to 1,000 destination domains that all use GSuite as their provider, your KumoMTA instances will queue all of those messages together based on the fact that the SiteName that was automatically generated for those messages is the same.
Not only does this ensure that all messages destined for the same MX group can quickly and efficiently be delivered across an available connection, it allows for traffic shaping rules defined for the destination provider to be tracked and applied across all domains hosted by that provider, rather than individually for each destination domain.
Our Lua Helpers Make Traffic Shaping Configuration Easy
Because our configuration is handled using Lua policy scripts, users have extreme flexibility in accessing and defining configuration values, including those for traffic shaping. Each time a message is ready for egress, the server calls a get_egress_path_config event that is passed the configuration settings relevant to that path (the combination of source IP and destination site name). It's possible to gather that information from a number of sources, including filesystem access and database queries, then return the appropriate configuration.
To make things more straightforward, you can use the traffic_shaping.lua helper, which makes it easy to manage traffic shaping rules in either a TOML or JSON file formatted like this:
["default"]connection_limit = 10max_connection_rate = "100/min"max_deliveries_per_connection = 100max_message_rate = "100/s"idle_timeout = "60s"enable_tls = "Opportunistic"consecutive_connection_failures_before_delay = 100["gmail.com"]max_deliveries_per_connection = 50connection_limit = 5enable_tls = "Required"consecutive_connection_failures_before_delay = 5["example.com"]mx_rollup = falsemax_deliveries_per_connection = 100connection_limit = 3
By default, any domain defined is automatically an MX rollup target, so the rules can be defined for the provider's top-level domain and will be applied to any other domains hosted by the same set of MXes for that provider. In most cases, this will result in coverage for all mail hosted by that provider, but in some cases, the provider will have multiple MX sets defined for different countries or use cases. One example is yahoo.co.jp, which operates independently of the primary Yahoo! domains, and some providers use different MX sets based on whether the message is going to their consumer or hosted business domains.
In some cases you may want to override the traffic shaping rules for a domain that is part of a larger rollup, for example when you want to send slower to a specific destination for reasons other than deliverability, or because you suspect that tenant of the hosting provider implemented their own stricter rules. In that case, you can use the mx_rollup = false option to exempt the domain from the rollup rules in order to apply your own rules.
Give It a Try
We think that SiteNames represent a significant improvement to the existing solutions for MX sharing, and hope you'll give them a try. KumoMTA is Open-Source, you can start using it without a license. Install it today and join our community to learn more about how it can make your life easier and your sending more efficient!