A Beginner’s Guide to Test/BDD Frameworks
When I first dived into the software testing world one of the most burning questions I had was ‘What is the difference between XUnit, NUnit, Specflow. JUnit, Selenium?’. Since then, I’ve worked with a few .NET testing frameworks and I still don’t believe I’m an expert I thought I would round up what I’ve learned in a short post.
I’ll go on to talk about differences between XUnit, NUnit, Specflow, and BDDfy as they are the ones I have most experience with. They are all test framework tools but each have pros and cons which make them better suited to certain testing situations. They are compatible with different tools so that may also affect which framework (or combinations of framework you use).
NUnit
An open source framework designed for writing and running unit tests in .NET.
Classes are known as Test Fixtures and methods are known as Tests. When tests are run, NUnit creates a new instance of the test class (Test Fixture) and then runs all of the test methods (Tests) from the same instance.
Knowledge pre-requisites: C#, Visual Studio, unit testing.
Compatible with: Visual studio, .NET and .NET core, BDDfy, Selenium, Specflow.
Used for: unit testing and functional testing in combination with BDDfy.
Con: not easily readable for those who do not code.
XUnit.net
Also an open source framework designed for writing and running unit tests in .NET. It supports the software development process test driven development (TDD), which allows for requirements to be written as short test cases which the program is developed against. Has a Java equivalent (JUnit) and these all come under an umbrella term of XUnit.
Facts are tests which are always true. They test invariant conditions. They do not take parameters and are independent of changing data.
Theories are tests which are only true for a particular set of data. Need inline data sent.
When tests are run, XUnit creates a new instance of the test class for each test method (fact or theory) which means tests are run in isolation (a fundamental difference from NUnit).
Knowledge pre-requisites: C#, Visual Studio, TDD, unit testing.
Compatible with: Visual studio, .NET and .NET core, BDDfy, Selenium, Specflow.
Used for: unit testing and functional testing in combination with BDDfy/Selenium/Specflow.
Con: not easily readable for those who do not code.
Specflow
A BDD framework which defines readable functional tests in .NET which support the software development process of behaviour driven development (BDD). Having such readable tests also allows them to double up as documentation for the program*.
*Personally, I also prefer having some sort of confluence or design document which outlines the technical requirements somewhere for complete documentation.
Tests are written in an human readable language in Feature files and they have C# test methods that drive them within corresponding class files known as step files.
Specflow requires a unit test runner in order to execute the tests: examples of which are NUnit or XUnit. It also has a good HTML template which outlines the scenarios executed, pass and failure rates with point of failures identified after tests have been executed.
Knowledge pre-requisites: C#, Behaviour driven development, Visual Studio, understanding of .NET automation test design models.
Compatible with: Selenium (for UI), NUnit, MSTest, .NET and .NET core.
Used for: API testing, microservice testing, and UI automation.
Con: Can be flaky when executing tests due to timings/race conditions, depends on how you write your tests and the execution method.
BDDfy
A BDD framework which defines readable functional tests in .NET which support the software development process of behaviour driven development (BDD).
A test class in BDDfy would be defined as a collection of scenarios, within this you would have information about the collection such as a story (as a user …, I want …, so that …). Similar to Specflow, the test class would also have a corresponding steps file which contains the C# code which drive the scenario methods. However, BBDfy uses a range of ‘scanners’. A Story scanner composes the story metadata and chooses the appropriate scenario scanner — fluent or reflective. The reflective scanner finds all the methods within the test class and turns them into execution steps, this allows the methods in the corresponding step file to be executed.
Unlike Specflow, BDDfy does not strictly require a test runner, it can be implemented against a POCO class on its own. However, it is compatible with test runners such as Nunit or XUnit (as shown in the example above).
Knowledge pre-requisites: C#, Behaviour driven development, Visual Studio, understanding of .NET automation test design models.
Compatible with: NUnit, XUnit, MSTest, .NET and .NET core.
Used for: API testing, microservice testing, and unit testing.
Con: Not as readable as Specflow but still easy to pick up with time.
In summary, XUnit and NUnit can be used to write unit tests whilst BDDfy or Specflow in conjunction with XUnit or NUnit can be used for functional testing. Specflow and BDDfy are more readable for those moving into automation testing or into the development world for the first time and are suitable for the behaviour driven development process.
Useful resources:
