Acceptance testing with SpecFlow and Selenium

I’m an avid believer in testing - TDD helps drive design, and having a test suite available to verify behaviour while maintaining an application is worth a million bucks. However, even the most complete unit test suite still doesn’t guarantee that the integration between different components is perfectly done, nor does it test the value a system delivers from a user perspective. Does a feature really do what a user expects it to do? Does your application fulfil it’s functional and non-functional requirements? Does it even run?

Acceptance testing tests features as if a user was interacting with them. This means testing the system through the user interface, a system driver layer, or a combination of the two. This could be useful in a couple of ways:

As a smoke test for a deployed application.
As part of your continuous integration build.
As part of your deployment pipeline strategy.

Acceptance testing can be implemented as a form of BDD - in a user story we should be able to express the requirements in a format that we can use to write an executable specification.

I quite liked the following simple distinction between what BDD and TDD aims for:

TDD implements and verifies details - did I build the thing right? BDD specifies features and scenarios - did I build the right thing? Armando Fox and David Patterson, Engineering Long-Lasting Software

To enable BDD, RSpec does a beautiful job in setting up context and Cucumber guides you down the given when then path. Both (as well as other testing frameworks) can be used to implement acceptance tests, but Cucumber only really makes sense if you have a business facing person looking at your specs. If not, writing your tests in English is a complete waste of time.

Unfortunately the .NET world does not quite have the same amount of tooling support for this kind of thing. This Stack Overflow question gives a nice summary on the current state of BDD libraries for .NET. For the xSpec (context / specification) flavour, we have NSpec, Machine.Specifications, and NSpecify. Quite frankly, because of the constraints of a statically typed language, the syntax of all of these libraries sucks a little. We could try and use RSpec with IronRuby, but it will add an extra, unneeded paradigm in a project.

For the xBehave family we are in luck - SpecFlow, is a BSD licensed library that allows us to write specifications like Cucumber in the popular Gherkin language on .NET. SpecFlow integrates beautifully with Visual Studio, providing support for auto completion in feature files. Features compile down to NUnit tests, which can be run with your stock standard testing tool set.

I’ve written a small web application that allows you to search through an in-memory list of fruit, to test out the tool set:

Sample application

This is what my feature looks like:

Feature: Search
	In order to avoid scanning through a long list
	As a user
	I want to be able to search for fruit by it's name


Background:

	Given I have the following fruit available:

	| Name      	|
	| Banana    	|
	| Pear		|
	| Kiwi		|
	| Watermelon	|
	| Orange	|
	| Naartjie	|
	| Lemon		|
	| Apple		|

	Given I am on the "Home Page"


Scenario: Browse search page
	Then I should see "Search Fruit"

@search
Scenario: Search for a specific fruit
	When I search for "apple"
	Then I should see "Apple"

@search
Scenario: Search for a partial fruit name
	When I search for "on"
	Then I should see "Watermelon"
	And I should see "Lemon"

@search
Scenario: No seach results
	When I search for "nonexistentfruit"
	Then I should see "No Results"

Behind the scenes, SpecFlow matches the Gherkin sentences to methods using regular expressions, which then drives the browser using Selenium backed by ChromeDriver for speed. For example, the following code drives navigation:

private static readonly Dictionary<string, string> paths = new Dictionary<string, string> { { "Home Page", "/" } };

private const string appUrl = "http://localhost:51553";

[Given(@"I am on the ""(.*)""")]
public void GivenIAmOnThe(string name)
{
    this.WhenIGoTo(name);
}

[When(@"I go to the ""(.*)""")]
public void WhenIGoTo(string name)
{
    string path;

    if (!paths.TryGetValue(name, out path))
    {
        ScenarioContext.Current.Pending();
    }
    else
    {
        SeleniumDriver.NavigateTo(appUrl + path);
    }
}

Note that if the path does not exist, yet the test is marked as pending and not failing - this allows us to write the specifications ahead of the actual code. We can of course also interact with the page:

[When(@"I set ""(.*)"" to ""(.*)""")]
public void WhenISetField(string field, string text)
{
    SeleniumDriver.SetTextBoxValue(field, text);
}

In the spirit of Capybara, the code backing this will attempt to find the element by its id or the text of the label referencing the input element. This style of locating elements shows how we can write our tests from a user point of view. Another such example is finding text on a page:

[Then(@"I should see ""(.*)""")]
public void ThenIShouldSee(string text)
{
    int textIndex = SeleniumDriver.PageSource.IndexOf(text, StringComparison.InvariantCulture);
    Assert.IsTrue(textIndex >= 0, "Could not find text \"" + text + "\" in the document.");
}

With these kind of tests, it’s very important to keep them DRY by composing bigger steps out of little steps. This also helps us to stay in sync with the ubiquitous language:

[When(@"I search for ""(.*)""")]
public void WhenISearchFor(string what)
{
    steps.WhenISetField("Search For", what);
    this.WhenIClickSearch();
}

[When(@"I click ""Search""")]
public void WhenIClickSearch()
{
    SeleniumDriver.ClickButton("Search", driver => driver.FindSingleElement("searchHeading").IsVisible());
}

For any kind of test to be repeatable, we need to have full control over the environment. This might entail setting up data, or even mocking out certain components in the application. Notice the setting up of the dataset for all of the tests:

Given I have the following fruit available:
    | Name          |
    | Banana        |
    | Pear          |
    | Kiwi          |
    | Watermelon    |
    | Orange        |
    | Naartjie      |
    | Lemon         |
    | Apple         |

In this example I’ve used the fantastic Deleporter library to reach into the running application and change its’ internals. This trick comes in useful when mocking dependencies that are difficult to control, like external web services.

[Given(@"I have the following fruit available:")]
public void GivenIHaveTheFollowingFruitAvailable(Table table)
{
    Fruit[] fruit = table.Rows.Select(x => new Fruit { Name = x["Name"] }).ToArray();
    Deleporter.Run(() => Fruit.SetFruit(fruit));
}

My initial impression of SpecFlow has been excellent. It’s also nice to bring a little of the Ruby tooling into my day to day .NET work . I’m now trying it out on a much bigger project - stay tuned for lessons learned.

Acceptance testing with SpecFlow and Selenium

Acceptance testing with SpecFlow and Selenium

Further Reading

See also