Our testing philosophy

Although CucumberStudio is a brand new platform, our team is in the development and testing business for more than 10 years. We have started our journey with waterfall process, adopted agile in 2006 and moved to devOps in 2011.

We deploy continuously new development into production. It means that we deploy between 1 to 10 small increments every day. It enables us to get early feedback from users, do A/B testing, react quickly when we have to deliver a bug fix… And during this journey, one of the most challenging transformation issue was the need to revolutionize our testing. In an era of continuous deployment and updates, there’s no time to have QA teams identify a problem and kick it back to the developers.

This is where we’ve found Behavior Driven Development (BDD) a very efficient approach over the past few years. It enables to align the team on what we should build and test earlier:

• Scenarios are requirement specifications that also behave like executable acceptance tests.

• Scenarios are the definition of “STOP” for developers.

• Scenarios are preferably written prior to development and based on a business domain language.

• Scenarios are confirmed with stakeholders and are automated.

• Scenarios should follow the DRY (don’t repeat yourself) principle.

• Scenarios should be reviewed and refactored continuously, just like code.

These good practices make our tests readable and maintainable: a valuable asset. But this is not enough. We also needed a very short feedback loop. And this is the reason why our acceptance tests are automated and integrated in the Continuous Integration process.

So far we have been successful at delivering a working software at the quality level required and at the speed and scale of DevOps. And among the key success factors we would pick: BDD, Refactoring and Continuous Integration .

This is a long journey. There is no shortcut and you can hardly get it right the first time. So like us keep trying, measuring and improving.

In our team, everyone acts as a developer and a tester (we always validate our colleagues’ work).

Usually the product owner has already written the acceptance tests with CucumberStudio before we start the development. The tests are used by the product owner and developers to discuss the feature. Anyone in the team can contribute to the CucumberStudio project by adding/editing scenarios and action words.

During development, we write unit and integration tests with RSpec and QUnit to test our Ruby and Javascript code.

To automate the acceptance tests, we use HipTest Publisher service to generate Ruby/Rspec code. All our acceptance tests are GUI tests, so we also integrate Selenium code via the Watir library.

All our tests (Unit, integration and acceptance) are integrated in a continuous deployment process. When a change is pushed to our git code repository, Shippable continuous integration runs all the tests to ensure everything works as expected. If any test fails, the team is notified so we can fix the issue. If all the tests pass, then the new version is deployed in production.

Shippable current build status can be seen in this badge:

Tests are not the only way we validate our features. Once the development process is done, another team member will validate the feature with exploratory testing and code review. This step of the process also leads sometimes to the creation of new tests (non regression tests if a bug is found or unit test if a special case was not tested).