Perfecto Mobile Cloud offers automated testing on real mobile devices, connected via Wi-Fi or cellular carriers to the Internet, spread in different locations around the world. AlertSite integrates with Perfecto Mobile Cloud, allowing you to schedule and monitor test runs on real devices. This way you can continuously monitor the key transactions on mobile devices and get proactively notified of any availability or performance issues.
Perfecto Mobile provides automated testing capabilities and real devices for monitoring, and AlertSite provides scheduling, alerting, and reporting. The following image shows the integration flow between the two systems.
User uploads an app and creates tests in Perfecto MobileCloud.
User schedules tests in AlertSite.
Tests run on real devices around the world in the Perfecto cloud.
Test results are sent to AlertSite.
AlertSite sends alerts if a test fails. User can also view test reports online in AlertSite.
Perfecto Mobile monitoring is an optional extra service to AlertSite Enterprise customers. Please contact your AlertSite Sales Manager for details. To view the contact information, select > AlertSite 1.0 from the top menu, then go to Support > Support Center.
Perfecto MobileCloud provides multiple copies of the same device model for redundancy. You use some devices for script development and debugging, and others for AlertSite monitoring. For example, if your Perfecto cloud includes three iPhone 5 devices, you use two of them for monitoring, and another one for scripting.
Monitoring devices must be paired, with each pair including two identical devices with the same model, OS version, network carrier, and geographical location. A pair of devices is used as a logical monitoring device, and they appear as a single device in AlertSite. A monitoring script runs automatically on both paired devices, one after another (in random order based on device availability). This improves fault tolerance: if a script fails on one device but succeeds on another, the run is reported as successful. Only errors detected on both devices during the same run may indicate an actual problem.
To designate devices for monitoring, you need to pick pairs of twin devices and specify a unique description for each device pair.
In Perfecto MobileCloud, select More > System Administration from the top menu.
Switch to the Devices tab.
Find two devices with the same model and OS version to use as a logical monitoring device.
Select one device and click Properties on the toolbar.
Enter the description for the device in the Description edit box. Use something that reflects the device’s business function or characteristics, such as model, carrier network, or location. For example, Amazon-Monitor-iPhone5, Android-ATT-Boston, and so on.
Add the same description for another paired device.
Repeat the procedure to specify descriptions for all pairs of your monitoring devices.
You create and maintain the monitoring scripts in the Perfecto MobileCloud IDE. Monitoring scripts usually contain these elements:
Device preparation and cleanup – application restart, returning the device to the Home screen, and so on.
User actions – button presses, link clicks, text input, and so on.
Transactions – operations whose availability and performance you want to measure, like the specific screens in your application. Transaction results will be sent to AlertSite. The next section discusses transactions in more details.
If you are targeting multiple platforms and devices, follow the Perfecto ScriptOnce methodology to create a device-agnostic script.
Save your monitoring scripts inside the Public or My (private) folder. Do not use group folders for monitoring scripts (group folders are named after the user group you are member of).
Perfecto monitoring scripts must include transactions to mark the actual things you want to monitor. A transaction is a group of commands with a transaction ID assigned. It succeeds if all of its commands succeed, and fails if at least one command fails.
Usually a transaction is a text checkpoint that measures your application’s response time and status (OK/error) in reaction to the preceding user action. This way you can monitor the application launch time, or the load time of specific screens in your application. To learn more about configuring Perfecto scripts to measure application responsiveness, see Measuring User Experience Timing in Perfecto documentation.
You can have up to 5 transactions in a Perfecto script.
For example, the following script includes two transactions, Site Load and What's new. Each transaction is a web page checkpoint that waits for a page to load, and verifies that is loaded successfully. The transaction results sent to AlertSite are the page load success/failure and the page load time.
To define a transaction:
Open your script in the Perfecto IDE.
To create a new group, select a command or several consecutive commands, then click the Group icon on the toolbar.
Double-click a group to open its properties.
Change the group name to something more descriptive. For example, Home page, Log in, Book a hotel, and so on. This name will appear in AlertSite reports.
Select the Transaction ID, from trans1 to trans5. The IDs define the order in which the script’s transactions will appear in AlertSite reports. It is best to assign the IDs sequentially from top to bottom of your script.
Click Close to save the changes.
Repeat the procedure to define other transactions, up to 5 in total.
|Note:||You can have as many groups as you wish, but only those with the transaction ID assigned will be reported to AlertSite.|
Commands in Perfecto scripts have error policies that specify how to handle failed commands, for example, whether to continue or stop the script. If later transactions in your script depend on earlier transactions, you may want to stop the script on error. In this case, use the Abort, Break, or Continue policy (see below). If the transactions are independent, you can use Ignore to ignore the errors and continue the script.
To specify the error policy for a command:
Open the script in the Perfecto IDE.
Select a command in the script. You can select several commands using SHIFT-clicks.
Click Edit on the toolbar.
Switch to the On error tab.
Select the error policy and click OK.
Ignore (default) – Continue the script. The transaction is marked failed.
Abort – Stop the script. The transaction is marked failed.
Break – Outside of a loop, works as Abort. When used inside a loop, exits the loop and continues from the next command following the loop. The transaction is marked as failed.
Continue – Outside of a loop, works as Abort. When used inside a loop, jumps to the next loop iteration. The transaction is marked as failed.
Catch – Use this in commands directly before a condition. Any error is ignored, and does not affect the transaction status.
For more information, see Error Policy in Perfecto documentation.
On the AlertSite Dashboard, hover over + Add New and select Monitor.
Select Mobile and click Real Device Monitor.
Specify your Perfecto account details and basic monitor information:
Enter the monitor name.
Select your mobile monitoring plan.
Fill in the Hostname (for example, mylab.perfectomobile.com), User ID, and Password fields.
Run Interval and Monitor Offset – The monitoring schedule: how often to run the script and which minute of an hour to start the run. For example, with the 60-minute interval and 10-minute offset, the script runs every hour at 8:10, 9:10, 10:10, and so on. With the 20-minute interval and 5-minute offset, the script runs every hour at 5, 25, and 45 minutes past hour. For 120-minute monitors, offsets 1-60 mean even hours (2 PM, 4 PM, and so on), offsets 61-120 – odd hours (3 PM, 5 PM, and so on)
|Devices in Perfecto MobileCloud can run only one script at a time. Configure the schedule to spread the monitoring over an hour between the various devices you have. Make sure that no two monitors try to access the same device at the same time.|
Collection/Notification Mode – If the script will run on multiple devices, choose whether to run on all devices at the same time, or on one device at a time, rotating through the selected devices.
Click Start Monitoring Now.
|Note:||This completes basic monitor configuration, but the monitor does not actually run yet.|
Click Add/Edit Script. Here you can select the script to run, and the mobile devices for monitoring.
Click Add Script and select the script to run.
Click Add a Device, select one or more devices to run this script, and click Continue.
To enable monitoring, select the Enabled check box on the Main tab of the monitor settings.
Click Save to save the monitor configuration.
AlertSite will now run the selected script in your Perfecto MobileCloud according to the schedule and collect the results.
AlertSite can send automated alerts when your mobile transactions fail, or the response time exceeds the configured limit. You need to enable alerts in the monitor settings and configure the alert recipients.
To enable alerts for a monitor:
Open the monitor settings. To do this, click on the monitor tile on the Dashboard and select Edit Configurations.
Click Monitor Alerts.
Select which alert types the monitor will send:
Availability alerts – Sent on an error condition.
Performance alerts – Sent when the response time exceeds a threshold. You can set thresholds for the entire script or individual transactions.
To configure the threshold:
Set Measurement Alert Type to Response Time.
Select the threshold type – Static (a fixed value) or Dynamic (compared to the historical performance).
Select the time period for analysis. Use a larger period to look for trends rather than single events.
Enter the error and warning thresholds.
Select the number of devices that should be in error or warning to trigger the alert.
To learn more about configuring performance alerts, see Performance Alerts.
Click in the top right to save the changes.
From the top menu, select Alerts > Alert Recipients.
To create a new recipient, click New Recipient and specify the contact information.
Select the recipient in the list and click Edit Recipient.
To receive alerts about errors, switch to the Availability Alerts tab and enable these alerts.
|For email recipients, also select Attach server response to e-mail alerts. This will include Perfecto execution reports in email alerts.|
To receive response time alerts, switch to the Performance Alerts tab and enable these alerts.
Configure other recipient settings as needed.
|Note:||If you have many recipients in AlertSite, but you want to send Perfecto monitor alerts only to specific recipients, you need to create a recipient group for the monitor and its alert recipients. For instructions, see Recipient Groups.|
You can see the status of your monitoring scripts on the AlertSite dashboard. The response time displayed is the total UX time of all transactions in the script. If a monitor discovers an error, it turns red and is moved to the top of the dashboard.
To view detailed results, you can either click the charts on the tiles or manually navigate the following dashboards:
Monitors > Availability – Shows the monitor status (OK/error) over time, as a total and per individual mobile devices.
Monitors > Performance – Shows the response time over time, as an average or per individual mobile devices.
Monitors > Runs – Shows all monitor runs and results as a list.
See Common Status Codes for status code descriptions and troubleshooting tips.
You can receive Perfecto reports attached to email alerts when your monitoring script completes with an error. Email recipients should have the Attach server response to e-mail alerts option selected to use this feature (see above).
You can also access Perfecto reports from AlertSite. First, go to Monitors > Runs, select your monitor and choose a time period at the top. You will see all the test results for that period. You can filter the runs by status (OK/error) or sort them by execution time to find the needed results. Expand the runs to see the results of individual transactions.
In the expanded run, click View Perfecto Report to view the detailed execution results with device screenshots and videos.
Failed runs with status 5300 may not have an associated Perfecto report. In this case, click Request Log on the Monitor Runs dashboard to view the AlertSite error log. The first error in the log indicates the reason why the test failed. See Common Status Codes below for some typical errors.
AlertSite provides a set of reports that you can download, print, or email.
To create a cumulative report:
Go to Reports > Performance Reports.
Select the report type, the monitor, and the report period.
Click Run Report.
To view AlertSite report for a specific test run, do any of the following:
In Monitors > Runs, expand a run and click Detail Report.
In Monitors > Summary > Availability History, click a time range, then click a specific run. In the tooltip, click View Detail Report.
This opens the Detail report for that run, with the status and response time of all transactions in your Perfecto script.
Reports contain test results as a table. You can expand a test to see the transactions performed in that test, their status and response time. Click the camera icon next to a test to view the Perfecto report. If the Perfecto report is not available, click to view the AlertSite error log.
This table contains some common error codes used by Perfecto monitors in AlertSite. For other status codes, please see AlertSite Status Codes.
|Status||Description||Possible Causes and Solutions|
|0||Site is OK now||The test completed successfully.|
|2||Test timed out||Status 2 is a failed time checkpoint. This usually means your application is busy and experiencing a slowdown. If the response time looks normal, review the timers in your script and increase the timeouts.|
|5||Validation failed||Status 5 is a failed checkpoint, meaning the expected text or image was not found in your application. You can see the checkpoint condition and results in the Perfecto execution report. If everything looks OK, review the checkpoint conditions in the script.|
|95||Unable to monitor||
Unable to run the script in Perfecto MobileCloud.
|96||Invalid XML||Unable to fetch script results from Perfecto MobileCloud. Please report this error to SmartBear Support.|
|5090||Processing exception||Status 5090 is a legacy status code that was used prior to October 23, 2015. It means no device was available to run the script. You may need to adjust the monitoring schedule (run interval and monitor offset) to avoid overlaps with other monitors.|
|5300||Service execution error||
There was a problem running your Perfecto script. To view the error message, go to Monitors > Runs, expand the failed run and click Request Log. Below are some common causes and solutions.
Handset unavailable, or Cannot open device
No devices are available to run the script. To avoid the problem, you can:
Transaction results missing from Perfecto
The script does not have any transactions defined. Transactions mark the operations to monitor, so no transactions means no performance data sent to AlertSite. To learn how to configure transactions in your Perfecto script, see Define Transactions to Monitor.
A failed script command means a problem with the script or the monitored application. You can review Perfecto report to identify the failed command and troubleshoot the script in the Perfecto IDE to fix the problem.
Empty variable or parameter:
Review the script in the Perfecto IDE and make sure all command parameters are specified.
If your script includes variables, make sure all variables except DUT (device under test) have default values and are marked as Runtime. The default values will be used when the script is run from AlertSite.
This happens if your Perfecto script does not contain any transactions – groups with the transaction ID assigned. AlertSite monitors transactions only rather than the entire script. To learn how to configure transactions, see Define Transactions to Monitor.
AlertSite can monitor only public and private scripts from your Perfecto repository, but not group scripts (scripts shared between members of a group). Please save your monitoring scripts to the Public or My (private) folder or subfolders.
This happens if the default primary location specified in your AlertSite account preferences is other than Fort Lauderdale or Atlanta. To avoid the issue, you need to temporarily change your default primary location:
From the top menu, select > Settings.
Switch to the Public Locations tab.
Select Fort Lauderdale or Atlanta as the Primary location.
Then, create your Perfecto monitor.
After that, you can revert to your previous location settings.