Today's another major milestone in the evolution of KumoMTA with our release of Beta 2!
The Beta 2 release, officially known as 2023.05.10-e88ad036, helped achieve our goal of building an MTA that is made for the cloud (Kumo is Japanese for Cloud). In this release we've introduced support for both Webhooks and HAProxy, two key components for deploying an MTA in a public cloud environment.
New Features
The following new features are included in Beta 2:
HAProxy Forward Proxy
Most public cloud environments limit the number of IP addresses that can be connected to a single virtual machine. To overcome this limitation, senders can deploy a collection of nodes running HAProxy that allocate the IPs, then have their MTAs leverage the HAProxy nodes to send messages over those IPs. The nodes hosting the IPs do not even have to be in the same environment as the MTA nodes, so you can even leverage IPs on a separate provider without having to deploy additional MTAs.
HAProxy Documentation
Webhooks
While on-disk logs are useful for troubleshooting and recordkeeping, things get complicated when it comes to real-time event reporting into your backend systems. Webhooks make it possible to get every message event in real-time, with durability provided through our unique approach of storing the webhook messages in our internal queues until delivery.
Webhook Documentation
HTTP Message Delivery
The modular design of KumoMTA means that functionality built for one use case can be used in a myriad of ways. In this case that means that the HTTP delivery mechanism used for Webhooks can also be used with any message in the queue. Do you send on-prem but also use a SaaS email relay provider? You can route messages destined to the provider via an API call, removing the need to send via SMTP. Do you need to route a message as SMS? Use Lua to extract the SMS content and send it to your SMS provider via API.
HTTP Delivery Documentation
Traffic Shaping Helper
The Lua approach for configuration management makes things extremely flexible when it comes to things like data source integrations, and our automatic MX rollup helps ensure that hosted domains are properly queued and throttled, but it does make determining which rules to use more complex.
To help simplify things for those who want a straightforward traffic shaping config file, and to show an example of how data source-based configuration can work, we've introduced the shaping.lua helper and its associated shaping.toml file. Our hope is that shaping.toml file will be a community-maintained resource, enabling all new senders to have a solid starting point for their traffic-shaping rules. Users can define their own additional rules in either a TOML or JSON format, and can also use the shaping.lua file as a basis for developing their own data-source integrated configuration management system.
Traffic Shaping Documentation
In addition to these key features, we've also introduced additional per-queue information in our Metrics API and added the ability to force syncing on a per-message basis so that those who want the speed advantage of delayed spooling can still ensure the safety of critical messages.
Looking Forward to Beta 3
The Beta 3 release is intended to round out the functionality needed for a clustered installation in a public/private cloud environment.
SOCKS5 Proxy
While we were implementing HAProxy based on the requests of our community who were using similar functionality in their commercial MTAs of choice, we quickly realized it has a significant shortcoming: because HAProxy is designed to be an inbound proxy, it lacks certain important communications abilities that would be useful when implemented as an outbound proxy. When an MTA sends via an HAProxy instance, there's no mechanism for feedback when a connection failure occurs, meaning that the MTA instance won't know if the proxy failed to connect to the remote host, failed to use the assigned IP address, or if the remote host dropped the connection without communicating.
To overcome these shortcomings, we'll be implementing support for the SOCKS5 proxy protocol, which supports much richer communications between MTA and the proxy server when used as an outbound proxy. This will allow for a much more robust environment for those who need to leverage proxies.
RabbitMQ Publishing Support
While Webhooks are a great solution for bringing message events into a near real-time latency, a number of cloud environments are built around message queue technologies, with RabbitMQ being a popular example.
With publishing support, KumoMTA nodes will be able to act as publishers on RabbitMQ queues. While we anticipate this will be mostly used for log events, the implementation would also support publishing the actual messages to AMQP as well. Perhaps something useful for inbound message routing?
Clustered Traffic Shaping
With proxy support enabling multiple MTA nodes to use the same external IP addresses, traffic shaping needs to be cluster aware. We need to ensure that if multiple cluster nodes use the same IP address to send to the same MX that the collective traffic across all those nodes still conforms to the traffic shaping rules.
The current state of the tech is:
- We support throttles that can be optionally backed by Redis.
- We don't currently support tracking counters like total connection limits in Redis -> that's something we'll add in Beta 3.
Dynamic IPs and Domains
The current implementation for defining IPs (egress_source) and domains used for relaying, FBLs, and bounce processing is handled at server init, which means that changes to these settings require a server reload. In contrast, most other policy configuration is done using cached data in Lua events that are close to the time those settings are needed, such as DKIM keys and IP assignment logic.
We will be refactoring the policy flow within KumoMTA to move these configuration elements out of the init event and closer to their actual use, so that configuration of these settings can be adjusted at runtime, removing the need to restart the server for changes to take effect.
This should move all operational configuration changes outside of the init event, leaving only the more static elements such as spool, log, and listener IP configuration.
And More
We are also adding the ability to re-queue messages via an API call, for situations where messages reached the wrong queue and their assignment needs to be re-evaluated, and the ability to configure SMTP AUTH for outbound sending, useful when you're relaying to a service or host that requires authentication.
Questions?
We're here for your questions and feedback! Feel free to email us at info@kumomta.com, use the chat in the bottom-right, or come join the community at our Discord Server or Forum.