Glad You're Ready. Let's Get Started!

Let us know how we can contact you.

Thank you!

We'll respond shortly.

iPhone UI Automation tests: a decent start

Apple’s inclusion of the UI Automation component in Instruments with iOS 4 is a definite step in the right direction. It’s the first reasonable way to write tests that externally exercise your actual app, rather than weirdly injecting test code into it. It’s also the only way to programmatically test lifecycle issues, such as how your app behaves when put in the background, when rotated, when the device locks, etc. Good stuff. Unfortunately, the current implementation of UI Automation also has some significant problems:

  1. There’s no way to run tests from the command line. The subtitle of the WWDC talk for UI Automation was “find bugs while you sleep;” unfortunately, you can’t find bugs while you sleep if you have to wake up to click the “Run” button.
  2. There’s no way to set up or reset state. The lack of fixtures which set up a known state at the beginning of iPhone tests has been a problem for unit testing (with OCUnit, Cedar, or what have you), particularly for apps that use CoreData. Now it’s worse than ever, because UI Automation manipulates the actual state of the app on the device, much like Selenium does in a browser. Sadly, UI Automation provides no method for reseting the device’s state, making it nigh impossible to prevent tests from affecting one another.
  3. Part of the previous problem is that UI Automation has no concept of discrete tests; it provides no form of organization for your test scripts. No test methods, no set up or tear down methods, just one big stream of consciousness line of execution. Obviously you can break this up into functions as you see fit, but why reinvent the wheel? Since the test script is JavaScript, I like the idea of using Jasmine for this.
  4. There’s no way to programmatically retrieve the results of the test run. You could debate the value of solving this issue at the moment, considering there’s no way to programmatically start the tests either. However, even if you were to write some clever AppleScript to kick off the tests automatically the only indication of the pass/fail status is in the Instruments UI, so you still have to wake up to check the results. I searched around a bit for information on deconstructing the protocol that UI Automation uses to talk to the device, but I came up empty.

I’ll definitely use UI Automation, particularly for app lifecycle testing. But, not being able to add those tests to a CI build definitely stings. I very much hope Apple keeps their momentum for automated testing and makes it more developer-friendly.

  1. Johan says:

    Can’t say much else other than I agree. UIAutomation seems like a moving target at the moment (good), but slightly slanted towards implementing repeatable performance tests (bad, if what you really want to do is integration-style testing)

    Have you submitted this to the radar ( If it’s not in there it’s generally not on Apple’s, uhm, radar…

  2. Nige says:

    I’ve been looking for a way to run my UIAutomation from the instruments command line so I can integrate it in with my Hudson build. So far no luck. I agree UIAutomation feels very 1.0 beta.

  3. Ryan Ausanka-Crues says:

    Additional problems:
    5. Calls are very slow to execute. In my testing on a MacBookPro in iPhone Simulator it took ~5 seconds to execute most of the methods on UIAElement.
    6. Spartan documentation and examples. How do you assert a table row’s accessoryType?
    7. Caching issues. If you are on a tableView and get the cells then switch to another tableView and request groups you get the groups of the new tableView but if you request cells, you get the old tableView cells. Since there no access to JavaScript source it’s hard to see what’s actually happening and why.
    8. UIAElement.logElement() is not very helpful and, since there is no support for breakpoints, debugging your script can be a pain in the ass.
    9. Doesn’t automatically terminate unless the app crashes. When all of the statements in your script have been executed you still have to manually stop Instrument execution.
    10. No way to get the cells within a table group.

    I really like the idea and am happy that it’s in JS instead of Objective-C but the implementation is more infuriating than it is helpful. I hope it grows into its potential.

  4. Adam Milligan says:

    Interesting. Ryan, you’ve clearly played around with UI Automation far more than I have. Thanks for the detail.

  5. Vijay Ganesh says:

    Hi Adam,

    I am currently working on iPhone automation testing. I wanted to know whether UI Automation can be used to automated the iPhone browser and test a particular website ?

  6. Adam Milligan says:

    Vijay, I don’t claim to be a UIAutomation expert just yet, but I don’t know of any way to test the content of a web view directly. I believe you can send touch events to particular locations on the device, but you’d have to know the screen coordinates of the elements on the web page you’d like to touch.

    Perhaps someone else will correct me and chime in with some better ideas.

  7. Vijay Ganesh says:

    Adam, you said that you have tried writing Applescript to automate UI Automation tool, i tried that too, but could go further except for opening “Instruments” . Is it possible to choose app to test, and test script and press “Start” button ?
    if yes, could plaza share the script ?

  8. Vijay Ganesh says:

    Adam, you said that you have tried writing Applescript to automate UI Automation tool, i tried that too, but couldn’t go further except for opening “Instruments” application. Is it possible to automate choosing app to test and test script, then press “Start” button ? if yes, could plz share the script ?

    *excuse my typo in the previous comment

  9. Adam Milligan says:

    Vijay, my comment about using AppleScript to start a UIAutomation test run was purely speculation; I haven’t actually tried it. If you get it working please post again and let us know how to go about it.

  10. Nghiem_huong says:

    I am using UI Automation to testing iPhone app.
    I want to test that next screen is true as follow of app. But I don’t know how to use?
    Ex: from Top Screen, when you click to Button “Help”, app will go to Help Screen. But app don’t work that same.

Post a Comment

Your Information (Name required. Email address will not be displayed with comment.)

* Copy This Password *

* Type Or Paste Password Here *