Custom assertions with ExUnit
Any unit testing library makes it easy to build your own custom assertions on top of the base assertions: just write functions. Often, though, you lose important information like appropriate line numbers, output that highlights structure differences, and so on. ExUnit makes it fairly easy to keep those, but you need a little background knowledge.
This article explains ExUnit's workings, and describes some helper functions that make it easier to create and test custom assertions.
ExUnit.AssertionError
ExUnit failure handling is built on the ExUnit.AssertionError structure. As of April 21, 02021, that structure looks like this:
@no_value :ex_unit_no_meaningful_value
defexception left: @no_value,
right: @no_value,
message: @no_value,
expr: @no_value,
args: @no_value,
doctest: @no_value,
context: :==
As far as I know, there's no guarantee this structure won't change in the future. But I'm willing to bet it won't, at least not in a way that breaks custom assertions.
Here's an assertion failure. I've annotated particular lines with the four AssertionError fields you'll need to understand:
If you're curious about the field values for any given assertion failure, an easy way to examine them is to take advantage of the fact that assert_raise returns the AssertionError structure it checks:
test "a sample failure" do
assert_raise(ExUnit.AssertionError,
fn -> assert f(1) == [1, 2, 3, 4] end)
|> IO.inspect
end
The result:
%ExUnit.AssertionError{
args: :ex_unit_no_meaningful_value,
context: :==,
doctest: :ex_unit_no_meaningful_value,
expr: {:assert, [line: 22],
[{:==, [line: 22], [{:f, [line: 22], [1]}, [1, 2, 3, 4]]}]},
left: [1, 3, 2, 4],
message: "Assertion with == failed",
right: [1, 2, 3, 4]
}
Working with AssertionError fields
Suppose you're testing a function that should return an error tuple. I'd use a plain old assert:
test "produces error tuple 1" do
actual = function_under_test_1(tag: 5)
assert actual == {:error, 5}
end
That's a perfectly fine test, and it plays nicely with ExUnit's colorizing of differences:
But let's have this simple example stand in for a more complicated check of just part of a data structure. That is, we want an assertion that's used like this:
test "produces error tuple" do
actual = function_under_test(tag: 5)
assert_error_content(actual, 5) ######## <<<<<
end
Its implementation might look like this:
def assert_error_content({:error, content}, expected) do
assert content == expected
end
def assert_error_content(input, _) do
flunk("The #{inspect input} value is not an error tuple")
end
And here's a test failure given by the second of those functions:
That's OK... but it grates to have the actual value crammed into the error message like that. It would stand out better in a "left:..." line:
That can be accomplished by having assert_error_content catch the AssertionError from flunk and change the :left field:
def assert_error_content(input, _) do
try do
flunk("Expected an error tuple")
rescue ex in ExUnit.AssertionError ->
reraise %{ex | left: input}, __STACKTRACE__
end
end
That's too much code to put in every custom assertion, so I generalized it to elaborate_flunk:
def assert_error_content(input, _) do
elaborate_flunk("Expected an error tuple", left: input)
end
You can import elaborate_flunk from the flow_assertions package, specifically this module.
Getting the "code:..." line right
Here's a test that will fail because the function's result is an error tuple that has the wrong content:
test "produces error tuple" do
actual = function_under_test(tag: 1)
assert_error_content(actual, 5)
end
The failure will come from the first implementation of assert_error_content, which, again, is this:
def assert_error_content({:error, content}, expected) do
assert content == expected
end
Here's the failure:
The line numbers in the failure are correct, but the code and error message don't have anything obvious to do with line 41:
assert_error_content(actual, 5)
We can add an error message to assert like this:
assert content == expected,
"Error tuple has the wrong content"
Surprisingly, the new message causes the "code:..." line to also change:
To see why that is, we can inspect the two AssertionErrors. Here's the one from the original assert, the one without an explicit error message:
%ExUnit.AssertionError{
expr: {:assert, [line: 82], [ {:==, [line: 82], [{:content, [line: 82], nil}, {:expected, [line: 82], nil}]} ]},
message: "Assertion with == failed",
...
}
Here's the one from the assert with the explicit message:
%ExUnit.AssertionError{
expr: :ex_unit_no_meaningful_value,
message: "Error tuple has the wrong content",
...
}
Adding a message also changes the expr field to ex_unit_no_meaningful_value. That instructs failure formatting code to look at the stack trace, find the topmost entry that belongs to the test module (in this case, CustomTest), read the file name (.../custom_test.exs) and line number (41) out of that entry, and then read the source at that line (assert_error_content(actual, 5)).
(Note: it's a little confusing that you change the value of the :expr field to change what's printed after "code:", but that's the way it is.)
Putting the fields back
I didn't call it out in the previous section, but adding an explicit message to assert also marks the :left and :right fields as having no value, so they're not printed.
In this case, that's not what we want, but it's easy to add them back by using elaborate_flunk or a new function, elaborate_assert, producing this:
Here's the version that uses elaborate_assert:
def assert_error_content({:error, content}, expected) do
elaborate_assert(content == expected,
"Error tuple has the wrong content",
left: content, right: expected)
end
Keeping the message, but not the "code: ..." line
Above, we overrode the :message and, as a side effect, changed what was printed as the source assertion (on the "code:" line). If we'd been OK with the message, we could have changed only the :expr field. You could do that with try/catch, but I prefer the more concise adjust_assertion_error:
def assert_error_content({:error, content}, expected) do
adjust_assertion_error(
fn -> assert content == expected end,
expr: ExUnit.AssertionError.no_value)
end
Notice that ExUnit code uses ExUnit.AssertionError.no_value/0 in preference to the literal ex_unit_no_meaningful_value, so you might as well too.
The meaning of :left and :right
The nice thing about failure output that has "left:" and "right:" lines is that it gives you the choice about which to use as the expected value and which as the actual value. (Many unit test frameworks are derived from 1989's SUnit, put the expected value as the first argument of an assert_equal function.)
"Left" and "right" become more awkward when your assertions aren't symmetric the way assert a == b is, and when you're only interested in part of the function-under-test's return value.
I suggest that you decide "left" means "the interesting part of the actual value" and "right" is always about what's expected. That's the opposite of the SUnit convention, but Elixir is a left-to-right language like English (think |> pipelines). In such languages, time flows from left to right. Creating a result happens before checking it, so the textual order should be the same.
Aside: flow-style assertions
Except for assert_raise, assertions aren't required to return any particular value. I like having them return their first argument so that they can be put in a pipeline:
VM.ServiceGap.accept_form(params, @institution)
|> ok_content
|> assert_valid
|> assert_changes(id: 1,
in_service_datestring: @iso_date_1,
out_of_service_datestring: @iso_date_2,
reason: "reason")
If you like that style, you can use defchain instead of def:
import FlowAssertions.Define.Defchain
defchain assert_fields(kvs, list_or_map) do
...
end
Testing custom assertions
If your assertions are complicated, you probably want to test them.
Testing cases where the code passes the assertion is easy:
test "..." do
my_assertion(1, 2)
end
Or, if you're using flow-style assertions:
test "..." do
assert my_assertion(1, 2) == 1
end
Using assert_raise to capture and check assertion failures is a bit cumbersome, so I offer assertion_fails:
describe "`assert_fields` with just a list of fields" do
test "how failure is reported" do
map = %{field1: 1}
checks = [:field1, :missing_field]
assertion_fails( #######
"Field `:missing_field` is missing",
[left: map, right: checks],
fn -> assert_fields(map, checks) end)
end
Tip 1: Although I'm usually a fan of test-driven design, I find I get better failure output from custom assertions by first trying them out as bare failing assertions, like:
test "how failure is reported" do
map = %{field1: 1}
checks = [:field1, :missing_field]
assert_fields(map, checks) ###########
...
... then examining the actual failure content to see if I like it:
... and only wrapping the assertion in assertion_fails when I'm happy. I'm not good at visualizing what an assertion failure will look like. I have to actually look at it.
Tip 2: As the above example shows, the "code:" line often won't contain literal values, but rather variable names. Err on the side of including :left and :right values. Sometimes, as with changeset assertions, I'll add the larger structure (the changeset) as a second :message line.
Tip 3: I will sometimes have complicated assertion failure messages, ones that span several lines. Exact string matches don't work well. For that reason, the first argument to assertion_fails can be a regular expression.
But even a regular expression is annoying if what you want to check is several small chunks of text embedded in a lot of other text. To avoid hard-to-read regular expressions that have to deal with line breaks, etc., take advantage of the fact that :message is just another field in an AssertionError and that the second argument to assertion_fails is a keyword list that allows duplicate keys. Like this:
assertion_fails(~r/First bit/,
[message: ~r/Second bit/, ##########
left: 5],
fn ->
refute_fields(%{"a" => [], "b" => 2}, ["a", "b"])
end)
I hope this has been useful.