Property - based tests are differing from a regular unit tests in a couple of ways. They are acting on rules instead of concrete values. They have randomized Arrange phase. They are execute multiple times. But why should we use Property - Based Testing instead of just using Random?

(In this example I use .NET 6, XUnit, FsCheck and FluentAssertions)

Let’s consider a simplest code, checking the equality of two integers

    public class NumbersComparer
    {
        public bool AreEqual(int x, int y)
        {
            return x == y;
        }
    }

Easiest class to be tested in the world. The test will be as simple as this

public class WhenComparingTwoNumbersInIntComparer
{
    [Fact]
    public void GivenSameNumbers_ShouldBeTrue()
    {
        var sut = new IntComparer();
        var result = sut.AreEqual(22, 22);
        result.Should().BeTrue();
    }
    
    [Fact]
    public void GivenDifferentNumbers_ShouldBeFalse()
    {
        var sut = new IntComparer();
        var result = sut.AreEqual(11, 22);
        result.Should().BeFalse();
    }
}

Okay, but what if we wanted to check more cases? What about comparing zero values? What about negatives? What about some huge number? We could change [Fact] into [Theory] in order to parametrize the test.

[Theory]
[InlineData(0, 0)]
[InlineData(-1, -1)]
[InlineData(-100, -100)]
[InlineData(int.MaxValue, int.MaxValue)]
public void GivenSameNumbers_ShouldBeTrue(int first, int second)
{
    var sut = new IntComparer();
    var result = sut.AreEqual(first, second);
    result.Should().BeTrue();
}

A bit better. But let’s imagine for a moment that the AreEqual() function has some business logic inside and it’s way more complicated. Product Owner came to us and asked to make some ‘slight changes’ in our business logic. He wants us to return false if the first number is between 1 and 10, without looking at the second value.

public class IntComparer
{
    public bool AreEqual(int x, int y)
    {
        if (x is >= 1 and <= 20)
            return false;
        return x == y;
    }
}

And we have a problem. Now we have pretty extended set of unnit tests, but we made a typo. Instead of 10, there is 20. You could say ‘easy, just add another [InlineData] with x = 1, x = 5, x = 20 with some random y value’. That is one way of solving it. Another one is to change our GivenSameNumbers_ShouldBeTrue to take two random numbers. I’ll limit it to 50 to make the test fail often, however in reality we could end up in a situation when test has 0.001% chance to fail, so it fails randomly once a week and finfing out why in this case would be close to impossible.

[Fact]
public void GivenSameNumbers_ShouldBeTrue()
{
    var numberToCompare = new Random().Next() % 50;
    var sut = new IntComparer();
    var result = sut.AreEqual(numberToCompare, numberToCompare);
    result.Should().BeTrue();
}

So now we have a test which generates random number from range (0, 50) and compares the number to itself. The chance of such test failing is close to 40%. Now, let’s take into consideration the new business case - when number is between 1 and 10, we expect false.

[Fact]
public void GivenSameNumbers_ShouldBeTrue()
{
    var numberToCompare = new Random().Next() % 50;
    var sut = new IntComparer();
    var result = sut.AreEqual(numberToCompare, numberToCompare);
    if(numberToCompare is >= 1 and <= 10)
        result.Should().BeFalse();
    else
        result.Should().BeTrue();
}

Now the test coverage is full, and it successfully manages to catch the typo we made, putting 20 instead of 10. But currently the test gives us no insight about what is wrong, since we don’t know the input.

PBT_1Tests.WhenComparingTwoNumbersInIntComparer.GivenSameNumbers_ShouldBeTrue

Xunit.Sdk.XunitException
Expected result to be true, but found False.
at FluentAssertions.Execution.XUnit2TestFramework.Throw(String message)
at FluentAssertions.Execution.TestFrameworkProvider.Throw(String message) 
(...)

We could debug the test a couple of times and finally we’d find out the issue. But what if the test would fail once every 100 runs? What about once every 10 000? Then things get a bit more complicated.

Here comes the Property - Based Testing. It will not solve all of the problems, but will help us a bit.

Here’s exactly the same test, but with input changed from Random to a number from generator

public class PropertyBasedWhenComparing
[Property]
public void GivenInteger_ItShouldReturnTrue(int numberToCompare)
{
    var sut = new IntComparer();
    var result = sut.AreEqual(numberToCompare, numberToCompare);
    if(numberToCompare is >= 1 and <= 10)
        result.Should().BeFalse();
    else
        result.Should().BeTrue();
}

This one also fails, but in a more gentle way - look at the test output

PBT_1Tests.PropertyBasedWhenComparing.GivenInteger_ItShouldReturnTrue

FsCheck.Xunit.PropertyFailedException:

FsCheck.Xunit.PropertyFailedException

Falsifiable, after 12 tests (0 shrinks) (13269095709155719581,15850512868929966471)
Last step was invoked with size of 13 and seed of (14449472247097790387,244200822284946969):
Original:
11

Exception doesn't have a stacktrace

Xunit.Sdk.XunitException
Expected result to be true, but found False.
at FluentAssertions.Execution.XUnit2TestFramework.Throw(String message)
at FluentAssertions.Execution.TestFrameworkProvider.Throw(String message)
(...)

First, and most important of all, we have input value. We can see that it ran 12 tests, and found that it failed for input value 11. So now we could just replace the input with 11 and debug it right away. For more complex scenarios where you’d generate complex tree of objects, there is this part Last step was invoked with size of 13 and seed of (14449472247097790387,244200822284946969). You can re-run test with the same seed for random generation, so that you’d get exactly the same state as the failed test, so that you can jump straight into debugging and fixing, skipping the part of looking for a good repro case.

You can find source code here GitHub