Can’t Live `with` It, Can’t Live `with`out It
- Brian Underwood
- 23rd Feb 2023
- 13 min of reading time
I’d like to share some thoughts about Elixir’s with
keyword. with
is a wonderful tool, but in my experience it is a bit overused. To use it best, we must understand how it behaves in all cases. So, let’s briefly cover the basics, starting with pipes in Elixir.
But like all tools, you should think about when it is best used…
Pipes are at their best when you expect your functions to accept and return basic values. But often we don’t have only simple values because we need to deal with error cases. For example:
region
|> Module.fetch_companies()
|> Module.fetch_departments()
|> Enum.map(& &1.employee_count)
|> calculate_average()
If our fetch_*
methods return list values there isn’t a problem. But often we fetch data from an external source, which means we introduce the possibility of an error. Generally in Elixir this means {:ok, _}
tuples for success and {:error, _}
tuples for failure. Using pipes that might become:
region
|> Module.fetch_companies()
|> case do
{:ok, companies} -> Module.fetch_departments(companies)
{:error, _} = error -> error
end
|> case do
{:ok, departments} ->
departments
|> Enum.map(& &1.employee_count)
|> calculate_average()
{:error, _} = error -> error
end
Not horrible, but certainly not beautiful. Fortunately, Elixir has with
!
But like all tools, you should think about when it’s best used…
with
is at it’s best when dealing with the happy paths of a set of calls which all return similar things. What do I mean by that? Let’s look at what this code might look like using with
?
with {:ok, companies} <- Module.fetch_companies(region),
{:ok, departments} <- Module.fetch_departments(companies) do
departments
|> Enum.map(& &1.employee_count)
|> calculate_average()
end
That’s definitely better!
{:error, _}
cases (in this case with
will return any clause values which don’t match the pattern before <-)
.But this is a great example of a happy path where the set of calls all return similar things. But where are some examples of where we might go wrong with with
?
What if Module.fetch_companies
returns {:error, _}
but `Module.fetch_departments` returns just :error
? That means your with
is going to return two different error results. If your with
is the end of your function call then that complexity is now the caller’s responsibility. You might not think that’s a big deal because we can do this:
else
:error -> {:error, "Error fetching departments"}
But this breaks to more-or-less important degrees because:
else
clause, you need to take care of every non-happy path case (e.g. above we should match the {:error, _}
returned by Module.fetch_companies
which we didn’t need to explicitly match before) 😤{:error, _, _}
) – there will be a WithClauseError
exception (again, because once you add an else
the fallback behavior of non-matching <-
patterns doesn’t work) 🤷♂️Module.fetch_departments
is later refactored to return {:error, _}
– we’ll then have an unused handler 🤷♂️:error
the message Error fetching departments
probably won’t be the right error 🙈with
clauses which call complex functions, it can become a nightmare 🙀So the first major thing to know when using with
is what happens when a clause doesn’t match it’s pattern:
else
is not specified then the non-matching clause is returned.else
is specified then the code for the first matching else
pattern is evaluated. If no else
pattern matches , a WithClauseError
is raised.As Stratus3D excellently put it: “with
blocks are the only Elixir construct that implicitly uses the same else
clauses to handle return values from different expressions. The lack of a one-to-one correspondence between an expression in the head of the with
block and the clauses that handle its return values makes it impossible to know when each else
clause will be used”. There are a couple of well known solutions to address this. One is using “tagged tuples”:
with {:fetch_companies, {:ok, companies} <- {:fetch_companies, Module.fetch_companies(region)},
{:fetch_departments, {:ok, departments} <- {:fetch_departments, Module.fetch_departments(companies)},
departments
|> Enum.map(& &1.employee_count)
|> calculate_average()
else
{:fetch_companies, {:error, reason}} -> ...
{:fetch_departments, :error} -> ...
end
Though tagged tuples should be avoided for various reasons:
else
is now being used, so we need to match all patterns that might occurelse
in sync when adding/removing/modifying clauses, leaving room for bugs.{:ok, _}
/ {:error, _}
tuples is that you can handle things generically without needing to worry about the sourceA generally better solution is to create functions which normalize the values matched in the patterns. This is covered well in a note in the docs for with and I recommend checking it out. One addition I would make: in the above case you could leave the Module.fetch_companies
alone and just surround the Module.fetch_departments
with a local fetch_departments
to turn the :error
into an {:error, reason}
.
We can even get unexpected results when with
succeeds! To start let’s look at the parse/1
function from the excellent decimal
library. It’s typespec tells us that it can return {Decimal.t(), binary()}
or :error
. If we want to match a decimal value without extra characters, we could have a with
clause like this:
with {:ok, value} <- fetch_value(),
{decimal, ""} <- Decimal.parse(value) do
{:ok, decimal}
But if value
is given as "1.23 "
(with a space at the end), then Decimal.parse/1
will return {#Decimal<1.23>, " "}
. Since that doesn’t match our pattern (string with a space vs. an empty string), the body of the with
will be skipped. If we don’t have an else
then instead of returning a {:ok, _}
value, we return {#Decimal<1.23>, " "}
.
The solution may seem simple: match on {decimal, _}
! But then we match strings like “1.23a”
which is what we were trying to avoid. Again, we’re likely better off defining a local parse_decimal
function which returns {:ok, _}
or {:error, _}
.
There are other, similar, situations:
{:ok, %{"key" => value}} <- fetch_data(...)
– the value inside of the {:ok, _}
tuple may not have a "key"
key.[%{id: value}] <- fetch_data(...)
– the list returned may have more or less than one item, or if it does only have one item it may not have the :id
keyvalue when length(value) > 2 <- fetch_data(...)
– the when
might not match. There are two cases where this might surprise you:
value
is a list, the length of the list being 2 or below will return the list.value
is a string, length
isn’t a valid function (you’d probably want byte_size
). Instead of an exception, the guard simply fails and the pattern doesn’t match.The problem in all of these cases is that the intermediate value from fetch_data
will be returned, not what the body of the with
would return. This means that our with
returns “uneven” results. We can handle these cases in the else
, but again, once we introduce else
we need to take care of all potential cases.
I might even go to the extent of recommending that you don’t define with
clause patterns which are at all deep in their pattern matching unless you are very sure the success case will be able to match the whole pattern. One example where you might take a risk is when matching %MyStruct{key: value} <- …
where you know that a MyStruct
value is going to be returned and you know that key
is one of the keys defined for the struct. No matter the case, dialyzer is one tool to gain confidence that you will be able to match on the pattern (at least for your own code or libraries which also use dialyzer).
One of the simplest and most standard ways to avoid these issues is to make sure the functions that you are calling return {:ok, variable}
or {:error, reason}
tuples. Then with
can fall through cleanly (definitely check out Chris Keathley’s discussion of “Avoid else in with blocks” in his post “Good and Bad Elixir”).
With all that said, I recommend using with
statements whenever you can! Just make sure that you think about fallback cases that might happen. Even better: write tests to cover all of your potential cases! If you can strike a balance and use with
carefully, your code can be both cleaner and more reliable.
We’ve helped 100’s of the world’s biggest companies achieve success with Elixir. From digital transformation, developing fit-for-purposes software for your business logic, to proof-of-concepts, right through to staff augmentation development and support. We’re here to make sure your system makes the most of Elixir to be scalable, reliable and easy to maintain. Talk to us to learn more.
Want to improve your Elixir skills? Our world-leading experts are here to help. Learn from the same team who architect, manage and develop some of the biggest in-production systems available. Head to our training page to learn more about our courses and tutorials.
Pawel Chrząszcz introduces MongooseIM 6.3.0 with Prometheus monitoring and CockroachDB support for greater scalability and flexibility.
Here's how machine learning drives business efficiency, from customer insights to fraud detection, powering smarter, faster decisions.
Phuong Van explores Phoenix LiveView implementation, covering data migration, UI development, and team collaboration from concept to production.