SLDN
|
Blog

iOS User Interface Testing

Automate, automate, automate...

Automation is part of our SoftLayer DNA. We automate not only our external processes and procedures, such as server provisioning and operating system reloads, but also internal processes such as our iOS mobile products testing.

Our mobile applications have short, 2-3 week release cycles, and growing feature sets. Regression tests on the applications can take several days to complete. In order to maintain high release quality standards with the small mobile teams, we decided to implement both User Interface and code level tests to exercise our daily builds. Failing test cases alert us when the product potentially becomes unstable. This article focuses on SoftLayer’s iOS products UI testing automation.

Apple’s developer tools include a product known as Instruments. Instruments, in turn, has a number modules that help when developing mobile device software. One of these modules is the Automation Instrument, a tool that enables user interface testing.

automate1.png
UI Automation Instrument In Action

The team enthusiastically started creating UI test scenarios using the Automation Instrument but quickly learned its limitations. The tool works well when it is run manually from the Mac OS X user interface environment, however running it from the command line script on a build machine inside the context of a Jenkins Continuous Integration process isn’t quite as effortless.

We would not be worthy of calling ourselves “SLayers” if we weren’t able to find solutions for several “minor” issues, right? So here’s how we got it working…

First we extended our build script so it accepts additional parameters that trigger execution of the automated test.

# Run the build script.
 
/usr/bin/ruby Scripts/build_script.rb --build_number ${BUILD_NUMBER} --run_automated_tests --run_unit_tests
 
BUILD_RESULT=${?}

When the test arguments are present, the following method gets called inside of the “build_script.rb”.

# ---------------------------------------------------------------
# Run UI Automation test suite
# ---------------------------------------------------------------
def run_test_suite(build_products_dir)
 
    unlock_login_keychain_command = "security -v unlock-keychain -p PasswordHere \"${HOME}/Library/Keychains/login.keychain\""
    if (!(system_debug(unlock_login_keychain_command)))
        raise "Error unlocking login keychain."
    end
    instruments_command = "/bin/sh ./Scripts/UIAutomationTestSuite/run_instruments.sh"
    if (!(system_debug(instruments_command)))
        raise "Error running instruments."
    end
end

Unfortunately, when starting Instruments from the command line, the user still must give Instruments permission to start and monitor processes. The method above executes a shell script on the build machine that then uses another command line tool, expect, to interact with Instruments on the users behalf.

#/bin/sh
/usr/bin/expect ./Scripts/UIAutomationTestSuite/interact_with_instruments.sh
wait
exit 0

"Expect" script:

#!/usr/bin/expect
 
set timeout -1
 
spawn instruments -t ./Scripts/UIAutomationTestSuite/Automation.tracetemplate /Users/build/workspace/MobileApplication_iPhone/SLMobileClient/TestProducts/Debug-iphonesimulator/SoftLayerMobile.app -e UIASCRIPT ./Scripts/UIAutomationTestSuite/iPhone_testSuite.js -e UIARESULTPATH /Users/build/workspace/MobileApplication_iPhone/SLMobileClient/TestProduct
 
expect "Name (build):"
send "\r"
expect "Password:"
send "PasswordHere\r"
interact
expect eof
catch wait result
exit [lindex $result 3]

When the credentials are provided, instruments start running test cases as defined in the UI Automation script. 

#import "./iPhone_loginAndLogout.js"
#import "./iPhone_launchAndNavigateBandwidthInfo.js"
#import "./iPhone_launchAndNavigateTickets.js"
#import "./iPhone_launchAndNavigateDevices.js"
#import "./iPhone_launchAndNavigateAbout.js"
#import "./iPhone_launchAndNavigateStorage.js"
#import "./iPhone_launchAndNavigateAccounting.js"
#import "./iPhone_launchAndNavigateNotifications.js"
#import "./iPhone_launchAndNavigateSupport.js"
 
var target = UIATarget.localTarget();
var host = target.host();
 
UIALogger.logMessage("Cleaning old tests results...");
host.performTaskWithPathArgumentsTimeout("/bin/sh", ["Scripts/UIAutomationTestSuite/clean_tests.sh"], 10);
 
UIALogger.logStart("Running iPhone UI test suite...");
 
run_testLaunchAndNavigateAbout(target);
run_testLoginAndLogout(target);
run_testLaunchAndNavigateTickets(target);
run_testLaunchAndNavigateDevices(target);
run_testLaunchAndNavigateBandwidthInfo(target);
run_testLaunchAndNavigateStorage(target);
run_testLaunchAndNavigateAccounting(target);
run_testLaunchAndNavigateNotifications(target);
run_testLaunchAndNavigateSupport(target);
 
UIALogger.logMessage("Stopped running iPhone UI test suite...");
 
var result = host.performTaskWithPathArgumentsTimeout("/bin/sh", ["Scripts/UIAutomationTestSuite/prepare_junit_test_report.sh"]);
logTaskResults(result, "false");
Automation test cases running on the simulator

When the instruments application finishes running the tests, their results are saved into a JUnit style XML file in a location where Jenkins can find them.

automate2.png
Test results reported by Jenkins

Jenkins also makes the test case results a part of the build artifacts.

Stay tuned for part 2: Automate, automate, automate… - How to automate Unit Testing on iOS

Pawel

Was this helpful?
Yes
No