How to use Lua for flexible configurations in Erlang and Elixir

When I need to configure something in a complicated way, I find myself reviewing the embedded language that provided the server to create a flexible configuration. In Redis, you can improve the performance of requests, in Nginx, you can improve the handling of incoming requests, FreeSwitch offers alternatives for performing the same tasks using different embedded languages. Even in a software like TheGimp, you can add your own code to make edit images.

Among the embedded languages, JavaScript and Lua are the most commonly used languages. JavaScript is very well known to the Erlang community because it was integrated (as a port, it is not implemented on top of Erlang) in popular products such as CouchDB and Riak. But I think the more exciting option, raised by Erlang Co-Creator, Robert Virding, is to implement Lua on top of Erlang, which can be used as an embedded language. Why? Let’s take a look.

Complex Configuration

Usually, when tasked with fitting the definition of a behaviour we would like to configure, we would create an algorithm in a simple language such as Lua. This saves us from performing activities like:

1. defining the configuration to fit with all of the cases,
2. reading and transmitting that information to be prepared for use,
3. writing a specific code to handle that standardised information.

The kind of implementation is used frequently. It is easy to think of examples you’re likely to come across in day-to-day life. For example, supermarket offers which have multiple dependencies, commissions for salespeople which might feature variable ranges and percentages based on the type of sale, amount of sale or tax brackets, even SMS, emails or HTTP requests could be considered examples.

To demonstrate this, let’s look at an example of developing a load balancer. This is a simple Erlang project using cowboy as a dependency, and depending on the headers and other information from the HTTP request, we can send it to the different web servers we have available and configured.

Based on the above premise we can write the configuration as follows:

{load_balancer, [
    {servers, [
        {odin, "1.1.1.1", [
            {in, method, [post]},
            {'>', <<"Content-size">>, 10000},
            {in, <<"Accept">>, [<<"json">>]}
        ]},
        {thor, "1.1.1.2", [
            {in, method, [get, post]},
            {'==', http_version, <<"2">>}
        ]},
        {balder, "1.1.1.3", [
            {in, method, [get, post]}
        ]}
    ]}
]}.

As you can see, we have to define a 3-tuple system with the operation in the first element and the two operators as the following elements inside of the tuple. In addition, we are occasionally handling the second element as the header name (if it is a binary), but at other times it’s the method we use to perform the request (using the atom “method”) and other times still, the HTTP version is used to gather the information.

The problem is that we have no closed specifications. We could add more elements or even change the meaning of them. What if we want to use logical modifiers like “and” and “or” to join the checks instead of assuming they are always using “and”? This change will add more complexity to our configuration and more complexity means more possibilities for making mistakes.

At the moment, if the configuration is wrong or is adding something that is not granted, it is up to us to trigger the corresponding error and point to where it is to make it easier to fix. As you can imagine, that is not an easy thing to do if you are handling it in the runtime.

Lua saves the day!

It’s not unconventional to think about configuration in terms of a specific code. At this point, Lua code could be put in charge of the definition because it is based on Lua semantics. We only need the information for the configuration and running of the snippet to give us the desired behaviour we want to plug into the correct place. For example, the previous configuration could be written as:

local odin = "1.1.1.1"
local thor = "1.1.1.2"
local balder = "1.1.1.3"

local method = http.method()
local size = http.header("Content-size")
local accept = config.split(http.header("Accept"), ", ")
local httpver = http.version()

if method == "post" and size > 10000 and member("json", accept) then
    return odin
elseif config.member(method, {"get", "post"}) and httpver == "2" then
    return thor
elseif config.member(method, {"get", "post"}) then
    return balder
end

As you can see, we are able to optimise and fix the code to suit our needs, it is shorter and clearer than the original configuration and, most importantly, we can now test and check to be sure it is compiling correctly.

The important thing to keep in mind is that the configuration code must include the functions which are going to be needed to handle the request. In the example above, we are using functions like http_version(), http_header("...") or even split(...) and member(...). These functions should be provided to the interpreter. Depending on the underlying language you are using, you can develop these functions in Erlang or Elixir.

Of course, the interpreter also has other functions available, we only need to provide the specific functions that are required for our business logic.

In addition to improving the performance, using these functions we are also improving the security because it has not been able to access functions which are not used. You can check the file luerl_sandbox.erl where it is removing the access to the functions which use the underlying operating system.

Where the code dwells?

Inserting the Lua code into the configuration can be a little tricky. To avoid this, I recommend putting these scripts into the priv directory as a normal Lua file (using the extension .lua) this could even be done inside of a database if we are handling the configuration in an automated way using a key/value storage configuration such as etcd.

The most important thing to keep in mind before running that code is to have a specific task which helps you to parse it and ensure the code is correct. One solution is to conduct a testing phase to ensure that the configuration is not breaking or negatively impacting other parts of the system.

For example, in the previous code, we would write a couple of libraries, one called utils for the functions needed for strings and tables and another called http needed for the HTTP functions. An example would be:

-module(luerl_lib_utils).
-export([load/1, install/1]).

-include_lib("luerl/include/luerl.hrl").

load(St) ->
    luerl:load_module([<<"utils">>], luerl_lib_utils, St).

install(St) ->
    luerl_heap:alloc_table(table(), St).

table() ->
  [
    {<<"split">>, #erl_func{code = fun split/2}},
    {<<"member">>, #erl_func{code = fun member/2}}
  ].

member([Entry, Table], St) ->
  #table{a = Array} = luerl_heap:get_table(Table, St),
  Result = array:foldl(fun
    (_, V, false) when V =:= Entry -> true;
    (_, _, Acc) -> Acc
  end, false, Array),
  {[Result], St}.

split([String, Sep], St) ->
  {[string:split(String, Sep, all)], St}.

As you can see, we are implementing a couple of functions which will be available on our Lua interface under the utils package. To load these functions, we have to run the function load which is exported in the previous module.

Then we can load the file:

{ok, Form, St2} = luerl:loadfile("config.lua", St1).

This step will help us obtain the forms that can be run in the following step. We will also obtain the modified state. To do this we just need to run the code:

luerl:eval(Form, St2).

The output for this code should be the return performed by the Lua code in an ok-tuple if everything was correct, or an error-tuple if something went wrong.

Scaling up

Lua scales up because it is built on top of Erlang. This means Lua is using the same processes as Erlang does, it also means we are not using ports to communicate with the Lua interpreter, we have the Lua interpreter running on Erlang.

This makes a great difference in comparison to JavaScript because if you are handling millions of requests and all of them require a JavaScript snippet, this can cause a bottleneck very quickly if you have to limit the number of ports or requests.

On the other hand, Lua is using native Erlang functions when it calls the functions we want to provide for the interpreter. That makes a clear improvement, saving us from performing data serialisation or transformation.

Conclusion

Using a language for flexible configuration gives us the possibility to create an easy interface to provide configurations, reduce the amount of code we need to write, improve the maintenance without jeopardising the performance of the system. At the moment, you can use Lua as we have explained during the article or you can jump into PHP if you need to process text or templates on top of Erlang or Elixir. Alternatively you can join the community and provide other solutions which help us build better fit-for-purpose software. Let us know if you need help building your system with one of these solutions. We look forward to hearing from you.