Dynamic XMPP Domains in MongooseIM

MongooseIM is a robust instant messaging server focused on scalability and performance. It makes use of XMPP (Extensible Messaging and Presence Protocol), an open technology used mainly to develop instant messaging solutions. The protocol is highly extensible and has a very active community supporting it, which results in a variety of possible use cases, be it one-to-one text messaging, mobile group chat or collecting data from IoT sensors.

MongooseIM is an XMPP server that is constantly evolving to meet the rapidly changing demands while remaining highly scalable to handle millions of messages per minute, which is confirmed by both our load tests and the existing production installations. Recently we have seen a growing demand for massive multi-tenancy, where one MongooseIM cluster would handle more and more independent XMPP domains. We have been working tirelessly for many months on this and the result is our latest 5.0 release, which implements a completely new concept of dynamic XMPP domains. This feature allows you to have literally thousands of XMPP servers in one. To see the difference it makes, let us start with the original concept of XMPP when one server used to equal one domain.

Single-domain setup

A typical use case of XMPP is for a mobile instant messaging app. Each user is identified by their JID (Jabber Identifier), which has a form similar to an email address, e.g. alice@example.com can communicate with bob@example.com by connecting to the server example.com. It is very easy to configure this in MongooseIM with the TOML configuration file. The default file already contains the basic configuration, but in this example, we will write it from scratch. Let’s start with the minimal general section with the domain example.com defined in the list of static hosts:

[general]
  hosts = ["example.com"]
  default_server_domain = "example.com"

The default_server_domain is the domain that appears as the sender of XMPP stream errors returned by the server when a user cannot connect and the XMPP domain of the client is not known yet. To make this example complete, let’s add the auth section to the file, enabling user authentication with their accounts stored in a relational database, e.g. PostgreSQL:

[auth]
  methods = ["rdbms"]
  [auth.rdbms]

We also need to define the default connection pool, so MongooseIM can connect to the database.

[outgoing_pools.rdbms.default]
  scope = "global"
  workers = 5

  [outgoing_pools.rdbms.default.connection]
    driver = "pgsql"
    host = "localhost"
    database = "mongooseim"
    username = "mongooseim"
    password = "mongooseim_secret"

Finally, we need to define a client-to-server (c2s) listener to allow the clients to connect:

[[listen.c2s]]
  port = 5222

Now you can start MongooseIM, create an account for alice@example.com and use an XMPP client app to connect to the server. This setup is very minimalistic and certainly not secure enough for production use (there is no TLS), please see the documentation for more details.

Static multi-domain setup

One service provider might maintain the XMPP servers for a few companies, each of them having their own XMPP domain, just like for email addresses. Similarly to email, these companies might share one server installation, which can be easier and cheaper than having one server per business. This is why several domains would be hosted on a single server. Let’s update the general section in the configuration file to introduce two more domains. ^[1]

[general]
  hosts = ["example.com", "example.org", "example.net"]
  default_server_domain = "example.com"

It is possible to configure each domain differently, e.g. example.com can have message archive management (MAM) enabled to allow the users to retrieve stored chat messages. To do this, let’s enable the mod_mam_meta extension module.^[2] The module should be specified in the host_config section to enable it only for one domain:

[[host_config]]
  host = "example.com"

  [host_config.modules.mod_mam_meta]
    backend = "rdbms"
    pm = {}

Dynamic multi-domain setup

Software as a service (SaaS) has become the standard way of providing IT services. In this scenario, we can imagine not just a few large companies, but thousands of small businesses (e.g. with up to 100 users each) using the same corporate chat solution built with MongooseIM. Each business would need their own XMPP domain, but they would neither want nor need their own MongooseIM installations, so instead they could pay for a hosted SaaS solution. The provider of such a solution would need to host hundreds or even thousands of domains on a single XMPP server.

To do this with a typical XMPP server (and with MongooseIM before version 5.0), one would have to edit the configuration file and restart the server for the changes to take effect. This is best done as a rolling upgrade, restarting one node at a time, but this procedure is quite tedious and takes some time. Another issue is that for each statically configured domain there are multiple resources allocated as all extension modules are started independently for each domain. Furthermore, the configuration file would become unmanageably large. To solve this problem, we could use the latest feature of MongooseIM 5.0.0: dynamic XMPP domains. Instead of defining thousands of hosts in the configuration file, we list only one host type – let’s call it basic. All we need to do is to modify the general section of the configuration file^[3]:

[general]
  host_types = ["basic"]
  default_server_domain = "example.com"

We still need a static default_server_domain to be able to respond with XMPP stream errors. To allow domain management, we need to enable a service called service_domain_db – by default it will reuse the globally defined default DB pool that we already defined. The domains will be stored in our PostgreSQL database.

[services.service_domain_db]

The last step is to define the HTTP listener that will handle the REST requests^[4]. Let’s set it up only on the loopback interface for localhost:

[[listen.http]]
  ip_address = "127.0.0.1"
  port = 8088

  [[listen.http.handlers.mongoose_domain_handler]]
    host = "localhost"
    path = "/api"

New domains can be added with a simple REST call:

curl -i -X PUT -H 'Content-Type: application/json' -d '{"host_type": "basic"}' \
  localhost:8088/api/domains/example.org

Such a request might be sent by a web server that would expose a GUI used to manage the domains. To cut off the inter-domain traffic we could separate them with an extension module called mod_domain_isolation. You can have multiple host types, which may correspond to different levels of service, e.g. when a distinction between standard and premium services is needed, we would add an advanced host type for the premium customers by editing the general section once more:

[general]
  host_types = ["basic", "advanced"]
  default_server_domain = "example.com"

Now we can enable the message archive only for the advanced host type:

[[host_config]]
  host_type = "advanced"

  [host_config.modules.mod_mam_meta]
    backend = "rdbms"
    pm = {}

You can use static and dynamic domains at the same time – for example, for a big company that would have its own unique set of configuration options, such as a separate database or other special features, the domain can be configured statically:

[general]
  hosts = ["big-company.example.com"]
  host_types = ["basic", "advanced"]
  default_server_domain = "example.com"

The diagram above summarizes the resulting setup, showing some client connections as well. Please refer to the documentation for more details regarding your MongooseIM configuration.

Performance of dynamic domains

When it comes to performance testing, we always push MongooseIM to the limits using amoc, our load testing tool, and amoc-arsenal-xmpp, a set of scenarios designed for testing XMPP servers. For dynamic domains we decided to run several scenarios targeted at different metrics, increasing the load to the point of failure. The number of users was up to 100 k for the one-to-one messaging test and these users were actively chatting, resulting in high message rates^[5]. We also decided that the system under load should be a three-node cluster of c5ad.xlarge AWS EC2 machines with an xlarge RDS instance of PostgreSQL, which is quite a small setup, to show that even this modest installation can handle a heavy load.

Initially, every test was executed for one static domain – the performance of version 5.0 was identical to the one of version 4.2. Then, the users got evenly distributed among 1,000 different domains, which did not result in any performance drop. Finally, the scenario was pushed to the extremes with as many domains as users. This meant up to 100,000 domains, but even that high number was not enough to cause any fall in performance other than a slight increase in memory usage. Domains were created on the fly at rates of up to 24 k / min without putting significant additional stress on the system. Selected test results are shown below. There were many more configurations tested, but they are omitted for the sake of simplicity. Results would vary with any difference in the setup, so if you need to determine the limits of your installation, please contact us.

Conclusion 

Some XMPP servers allow you to add virtual hosts – this is usually done with configuration files and does not allow grouping domains into host types. What sets MongooseIM 5.0 apart is that the dynamic domains are seamlessly integrated with almost all^[6] features and extensions, making it possible to easily set up and maintain thousands of domains without any performance penalty.

Load test results show that you can expect high performance from MongooseIM, no matter if you need to host one huge domain or thousands of smaller ones. The only thing to worry about is the design and implementation of your front-end application and MongooseIM will certainly take care of the traffic generated by the millions of connected devices.

If you would like to talk to us about how your project can benefit from using MongooseIM, you can contact us at general@erlang-solutions.com and one of our expert team will get right back to you.

^[1] default_server_domain can be one of the defined hosts, but it can be a different domain name as well.

 ^[2] Such modules enable optional features of MongooseIM and they usually implement XMPP extension protocols, e.g. mod_mam_meta implements XEP-0313. More information about it can be found in the documentation.

^[3] If you are following the examples, it is best to remove the host_config section as well, as it is no longer relevant.

^[4]  This listener is already enabled in the default configuration file.

^[5]  MongooseIM can handle many more connected users, especially on bigger instances, see our blog post on scalability. However, an extremely high number of users makes tests difficult to repeat, so it is better to have a lower number of users and a higher message rate per user. One could always add a lot of inactive users to the test, but there is little point in doing so.

^[6]  Please see the documentation for a complete list of exceptions.