Introduction
One of our early design goals with KumoMTA was to ensure it was well-behaved. This means following the applicable RFCs (even if we have to nudge the messages being relayed into conformity when necessary) and implementing support for the various traffic shaping tunables needed to keep it from overwhelming remote MTAs.
Our first beta release already had the initial traffic shaping options for things like connection count, message flow rate, retry interval, and the other various tunables a MailOps admin would expect, and in our second beta release, we added the shaping.lua helper, which took the power of our Lua-based configuration management and made it more accessible for those not looking to implement a deep integration with their management systems.
With our latest release, we're making it easy to adjust traffic shaping dynamically, allowing your KumoMTA instances to act on the feedback provided by the remote MTAs of the Mailbox Providers (MBPs) to ensure compliance with their policies.
The Need For Automation
While there are a variety of guidelines for how to configure the basic traffic shaping rules for your sending MTA (some provided by the MBPs themselves, some passed down as shared knowledge), those guidelines are never a one-size-fits-all; many remote MTAs, especially the ones operated by major MBPs, treat your various mail streams differently on the basis of reputation, audience engagement, and even just whether the MBP's MTAs are overloaded at the time of sending.
Think of having a conversation with someone: you're more willing to engage in conversations with people you've gotten to know over random strangers, especially those you assume you're not interested in speaking with, and if you're preoccupied you'll even tell your friends to call you back later or slow down while they are talking because you're dealing with too much at that exact moment. Now imagine those people never changed their behavior based on what you said and instead kept talking quickly without regard for whether you could keep up; you'd eventually just hang up the phone.
That's why successful senders need an MTA with the ability to automatically adhere to the feedback given by the remote MBP systems so that they don't get hung up on and can ultimately send the messages their subscribers are expecting, even if it takes a little longer because of the feedback from that system to pause or slow down.
Implementing Traffic Shaping Automation
While the details of how to install and configure KumoMTA are beyond the scope of a single blog entry, the process is fully documented, and we have a page specifically on how to configure Traffic Shaping Automation that covers how to enable the automation system.
After you install the automation capability, Traffic Shaping Automation is implemented through a set of straightforward rules like the following:
[["example.com".automation]]
regex = "250 2\\.0\\.0 Ok"
# There is no "trigger" here, so the action is taken immediately
# when a log record matches.
# This action causes delivery to be suspended for the pathway (source + site-name)
action = "Suspend"
# for 2 hours
duration = "2 hours"
[["example.com".automation]]
regex = "250 2\\.0\\.0 boop"
# sets max_connection_rate="100/s"
action = {SetConfig={name="max_connection_rate", value="100/s"}}
# if we see 2 or more matches in an hour. Unlike throttles, this
# doesn't divide down to per-second rates.
trigger = {Threshold="2/hr"}
# The config override will last for 2 hours
duration = "2 hours"
These enable you to define a remote response and how to handle it, either by temporarily suspending your sending from that IP to the target domain, or by temporarily changing traffic shaping rules. Which rule you implement will depend on the nature of the response from the MBP.
If you're wondering how you'll come up with all the necessary rules, we can help there too. As part of our community focus, we're developing a set of community-maintained resources for our users, including a default traffic-shaping rules file. The default rules file can be viewed on GitHub, and anyone in the community can submit updates. You're also able to maintain your own separate local rules file in order to add your own unique insights to your own servers.
Summary
Automation of traffic shaping rules and sending behaviors isn't just a good idea, it's mandatory for those looking to comply with the expectations of the major Mailbox Providers and key to being a responsible sender.
KumoMTA makes it easy to implement Traffic Shaping Automation thanks to our policy-based configuration engine and our shaping.lua helper. Give it a try and let us know your experiences!