Property based testing in C#

I like unit tests, I really do. But they have issues. The problem is twofold.
First, how to define a unit? From my experience, it's usually too granular, leads to over-specification, and stops us from refactoring.
My second concern is that they are based on examples. And those very few examples are derived from the implementation. That doesn't sound like thorough testing.

Our code has properties

Let's think of another approach. What if we define our unit by its behavior? Imagine a vending machine. You insert enough coins to buy a bag of chips and press the button. The bag drops into the tray, and you get the correct change. This is our behavior.
You can treat it as a function that for a given amount of money - x, gives you two outputs - bag of chips - y and correct change - z. Now we can call this function property, and of we go!

To write some property-based tests in C# we need FsCheck with xUnit adapter. The usual approach is to create a generator that will feed data into the property, validating its correctness.
Such a generator can look like so:

 private static readonly decimal[] AllowedCoins = {1m, 0.5m, 0.25m};     

 public static Arbitrary<(string,string)> MoreThan1D()     
   return Gen.Elements(AllowedCoins)     
             .Where(x => x.Sum() > 1.0m)     
             .Select(x =>      
                 (string.Join(", ", x), (x.Sum() - 1.0m).ToString()))     

This Arbitrary, in a FsCheck language, returns a tuple. Its first part is a string consisting of allowed coins, which sum is bigger than 1$, and the second part is an amount of change. Example input can look like this: ("1, 0.5", "0.5")

How to define a property?

var (coins, change) = generatedCoinsWithChange;
Func<bool> property = () =>
    var boughtColaWithChange = sut.GetCola();
    return boughtColaWithChange.Contains("Cola") 
          && boughtColaWithChange.Contains(change);

It's a function that returns true or false. After inserting coins, generated by our Arbitrary, into the machine, we buy a product. As the last step, we verify that the product and amount of change are correct. You can see the whole code on github.

And that's all. Now we can run this code as many times as we want. Each time our generator will give us a bit different data. This strategy leads to much better coverage than example-based tests.

Strange things can happen

Because the input is random and generated each time we run our test, there is a possibility that our suite will fail from time to time. And that's perfectly fine! Every time this happens, we get a report with parameters that proved our property false. With such knowledge, we can track the bug and fix it. And we get all that for free with property-based testing.

Another benefit of PB testing is that it forces us to think about our unit differently. We can't just think of examples looking at the implementation and just put them into test code. Before that, we have to define our inputs' domain and describe how our code behaves. And for me, that's the most significant advantage of property-based testing. It makes you think before coding.

No Comments Yet