Vision AI

Vision AI is a plugin capability that uses artificial intelligence (AI) to identify user interface elements based on their visual appearance on the screen instead of internal application properties.

When Visual Object Detection is enabled, TestComplete can send screenshots and user interface metadata from the tested application to SmartBear-hosted AI services for processing. Restart TestComplete after you enable or disable Visual Object Detection at any time in Install Extensions dialog.

TestComplete displays recognized elements in the Object Browser and uses them during recording and playback.

Vision AI supports both text and images for object recognition, while Optical Character Recognition (OCR) supports text recognition.

TestComplete stores detected objects in the Object Browser.

To locate these detected objects:

1. In the Object Browser, expand the application under test.

2. Right-click the Identified VOD Objects, and then add them to the VOD identifier.

3. In the Keyword Test window, select the required test.

To use Vision AI, verify that the following prerequisites are met.

• Verify that you have a TestComplete Pro Bundle license or an Intelligent Quality (IQ) add-on license.

• Allow access to Vision AI service endpoints

  To use Vision AI, make sure that your environment can access the following endpoint:

  • https://ocr-service.prod.testcomplete.com

  If the endpoint is not accessible, work with your IT services team to add them to the allow list.

• Install and Enable Required Extensions

  1. Open TestComplete.

  2. Click File, and select Install Extensions.

  3. In the Install Extensions dialog, expand Intelligent Quality and verify that the Visual Object Detection checkbox is selected. If other check boxes within Intelligent Quality are also selected, this is an expected behavior.

  4. Click OK.

     Note:

     TestComplete may take a few moments to apply the extension changes.

  5. Restart TestComplete.

To enable VOD for a specific application, do the following:

1. Open your project in TestComplete.

2. At the project level, select Properties, and then click Visual Object Detection.

3. Click Add.

4. Enter the process name of the application that you want to prioritize for Vision AI, and then select the check box next to the process name to enable it.

5. To apply the priority setting to all applications, enter * as the process name.

6. Click OK to save the changes.

Prerequisites

Before you create a Vision AI test, verify the following:

• You have a TestComplete Pro Bundle license or an Intelligent Quality (IQ) add-on license.

• Visual Object Detection is enabled in the Install Extensions dialog box.

• Your tested application is running.

• Before you begin, make sure that Visual Object Detection (VOD) is enabled for the Application Under Test (AUT). Open the project properties, select Visual Object Detection, add the AUT to the list of applications, and then click Apply.

Capture the objects

To capture the objects your Vision AI test will use, do the following:

1. Open your project in TestComplete, and then select the Object Browser tab.

2. On the Object Browser toolbar, select the visual objects button to show detected objects in the tree.

3. In the object tree, expand Sys, expand the process for your tested application, and then expand the VODWindow node. TestComplete lists each detected element as a VODObject node named after its visible text.

4. Select the refresh button to update the object tree, and then wait for the update to complete.

5. Select a VODObject node. TestComplete shows the captured image of that element in the right pane.

6. Right-click the node, and then select Create VOD Object Identification Data.

7. In the Create VOD Object Identification Data dialog box, enter a name in the Object name box, and then select OK.

8. If the project has no Visual Object Identities item, the Create Project Item dialog box opens.

9. Select Visual Object Identities, confirm the values in the Names and Location boxes, and then select OK.

Review the captured identification data

To review or refine the captured data, do the following:

1. On the Project Explorer panel, select VisualObjectIdentities.

2. In the table, review each row. The columns record the process, window class, X, Y, W, and H coordinates, recognized text, a description, the element icon, and the context window image.

3. To rename an entry, select the value in the Name column, and then enter a new name.

4. To correct a captured value, select the cell, and then enter the new value. You can edit the coordinate, window class, and text values directly.

5. To replace an image, select the ellipsis button in the Icon or Context column, select an image in the Select Icon Image or Select Context Window Image dialog box, and then select Open.

6. To sort the table, select a column heading.

During a test run, TestComplete detects changes to the properties of a visual object, such as its position or icon. The self-healing feature then re-identifies the object and prompts you to confirm the new identity data before TestComplete saves it.

To review and apply a proposed object update, do the following:

1. Run a test that interacts with a visual object.

   If TestComplete re-identifies an object for which the properties have changed, the Confirm object update dialog box opens.

2. In the Confirm object update dialog box, review the required information.For example:

   • Object: The name of the re-identified object, for example, Stop Button.

   • Position: The stored coordinates and the new coordinates of the object, for example, (306, 52) and (75, 51).

   • Icon: The stored image and the new image of the object.

3. Click Update to save the new identity data for the object, or click Ignore to keep the existing stored identity data.

The test run continues.

If a recorded test does not run as expected:

1. Review the recorded steps in the test.

2. Remove or update incorrect or unnecessary actions.

3. Run the test again.

Vision AI identifies elements based on what appears on the screen. Because of this approach, the following limitations apply:

• Multi-monitor setups are not supported. We recommend that you use Vision AI on a single-monitor setup.

• Some actions may apply to a broader screen area instead of a specific element.

• Window actions depend on detection accuracy.

• Vision AI does not use Name Mapping.