Negative Testing

Author: SmartBear Software
Applies to: TestComplete 7.x, 8.x and 9

This article describes how to implement negative testing in TestComplete and provides an overview of negative testing with some common negative test scenarios.

The article contains the following sections:

• About Negative Testing?
• Typical Negative Testing Scenarios
• Negative Testing With TestComplete
• Conclusion

About Negative Testing

There are two main testing strategies in software testing: positive testing and negative testing.

Positive testing determines that your application works as expected. If an error is encountered during positive testing, the test fails.

Negative testing ensures that your application can gracefully handle invalid input or unexpected user behavior. For example, if a user tries to type a letter in a numeric field, the correct behavior in this case would be to display the “Incorrect data type, please enter a number” message. The purpose of negative testing is to detect such situations and prevent applications from crashing. Also, negative testing helps you improve the quality of your application and find its weak points.

The core difference between positive testing and negative testing is that throwing an exception is not an unexpected event in the latter. When you perform negative testing, exceptions are expected – they indicate that the application handles improper user behavior correctly.

It is generally considered a good practice to combine both the positive and the negative testing approaches. This strategy provides higher tested application coverage as compared to using only one of the specified testing methodologies.

Typical Negative Testing Scenarios

Negative testing is aimed at detecting possible application crashes in different situations. Below are several possible examples of such situations:

• Populating required fields – Some applications and web pages contain fields that are marked as required. To check an application’s behavior, create a test that leaves the required fields empty and analyzes the tested application’s response. For example, the application can show a message box requesting a user to populate an appropriate field. After that, your test must interpret the application’s behavior as correct when working with invalid data.
• Correspondence between data and field types – Typically, dialog boxes and forms contain controls that can accept data of a specific type (for example, numeric, date, text, etc.). To verify whether the application functions properly, you can create a test that enters incorrect data into a control, for example, a test that enters a letter into an UpDown edit box or the value of 13/30/2010 to a date field.
• Allowed number of characters – Some applications and web pages contain fields allowing a limited number of characters to be entered, for example, the value of the User Name field for applications and web pages must contain less than 50 characters. To check the behavior of the application, create a test that enters more characters into the field than is allowed.
• Allowed data bounds and limits – Applications can use input fields that accept data in a certain range. For example, there can be an edit box into which you enter an integer number from 10 to 50, or an edit box that accepts text of a specific length. To check the application’s behavior, create a negative test that enters a value smaller than the lower bound or greater than the upper bound of the specified field.
  Another example of this negative test case is entering data that exceeds the data type limits. For instance, an integer value can normally contain values in the range of -2,147,483,648..2,147,483,647 (the size is limited by the number of bytes in memory).
  To check the application’s behavior, you can create a negative test that enters a value exceeding the bounds. For instance, the test can enter a large number (100,000,000,000) into an integer field.
• Reasonable data – Some applications and web pages include fields that have a reasonable limit, for example, entering 200 or a negative number as the value for the “Your age:” field is not allowed. To check the application's behavior, create a negative test that enters invalid data into the specified field.
• Web session testing – Some web browsers require that you log in before the first web page is opened. To check that these browsers' function correctly, create a test that tries to open web pages in the tested application without logging in.

Negative Testing With TestComplete

TestComplete allows you to create negative tests using any possible variants of data input (both valid and invalid). This approach to tests creation will make your application more reliable and stable. Here, you can also specify certain values that will lead to an exception and assign error messages to them - this will warn users of possible exceptions.

Note that the response to a thrown exception depends on the tested application. For example, the application can show a message at the bottom of the window, highlight the field with invalid data, show a pop-up containing the name of the incorrectly populated field or, which is more often used, show an error message window containing a description of the exception’s cause.

To catch these windows, you need to add additional operations to your test.

To track error messages from a keyword test, use the If Object operation. For example, using this operation, you can create a test that will check whether the specified window exists. Insert the operation right after the operation with invalid data input and specify an error message window as the needed object. If the application handles input data correctly, the error message window appears. If the window does not appear, this means that an unexpected exception occurred, and you need to find its cause (see below).

The image below shows an example of using the If Object operation:

To track error messages from a script, add the WaitWindow method to your code. This method delays the script execution until the specified window appears on the screen. Also, you can specify a time period that TestComplete will wait for the window. If the timeout expires and the window does not appear, the WaitWindow method returns an empty stub object. For instance, the code below waits for the “Path not found” message box which the Save File dialog shows when you attempt to save a file to a non-existent drive:

Aliases.notepad.WaitWindow('#32770', 'Save As', 1, 10000);

Below is the list of our recommendations that will simplify your negative testing with TestComplete:

• If you load input data from a file, change its contents so that it contains both valid and invalid data values. Please keep in mind that when using invalid values, you need to build your test so that an error message is not considered as an unexpected window and if this message appears, the test passes successfully, which means that the tested application works correctly (see above).
• If you generate a list of testing input values automatically, change the generating conditions so that it contains both valid and invalid data values. For example, if your application contains a numeric field whose bounds are from 10 to 50, test your application with the correct data (for example, 15 and 45), the bound values (10 and 50) and create a negative test with values that are not allowed, for example, with 9 and 51. Unlike using only positive tests, this approach will improve the test coverage as it will test all possible conditions.
• If you input some data manually, create both a test with the correct input value and a test with invalid data. For example, try to create a file with an invalid name (for example, *NewFile.txt). In this case, the application must show an error message (for example, "Invalid file name. The file name must start with a letter, digit or underscore.") or fail to create the file.

Conclusion

As you can see, negative testing improves the testing coverage of your application. Using the negative and positive testing approaches together allows you to test your applications with any possible input data (both valid and invalid) and can help you make your application more stable and reliable.

We hope, this article contains fresh and useful information that will help you create thorough tests with TestComplete. If you haven’t used TestComplete yet, download and try it for free.