iOS Guy

Apple recently released this great framework for automated tests – integrated with Xcode – and our team finally adopted it. Now, we had some trouble with it at first (and later too :P), so I thought it would be interesting to write a bit about the experience I had and the framework itself, as right now there isn’t much content about XCTest out there.

But before we get to discuss about XCTest, it is really important we touch some basic topics about automated testing in general.

People like to label things and sometimes we end up labelling too much. Tests for instance have many categories. You probably heard a lot about many of them. Being the most commonly known, Unit tests. But you probably heard about Smoke Testing, Confidence Testing, Sanity Testing, Regression Testing, Functional Tests, Non-Functional tests, Integration Tests, Application Tests and so on. Truth is, most of those are the same thing and there is a lot of confusion (or different opinions) about how each one of those (as well as the others out there) are defined or differ.

Unfortunately I am one of those people too (there is no way around that :P) and I have my own opinion on what is what.

So below I provide a definition for these most common test types. That is the definition we will use on this post.

A Unit Test tests a unit. A unit is the smallest piece of software that you can test. Unit tests are supposed to be isolated, that is, each unit test runs independently in such a way they don’t interfere with each other.

However, Apple uses the term “Unit Test” also for tests which leverage the context of the application you are testing. That characteristic can be leveraged to write any kind of test you want, since it provides a more realistic environment for white box testing. The most common ones being “Integration Test” and “Functional Tests”.

The difference between these two is very clear in theory but not so much in practice. Briefly, the Integration Test aims to test how multiple units behave when working together while the Functional Test aims to exercise a functionality.

But the goal here is not to discuss what each kind of test is or whether they are the same or not. But rather provide an understanding that all those tests can be implemented with XCTest, provide you know how the framework works and how it approaches automated testing.

To simplify, we will look into automation testing considering 3 different types of tests (you will see those can be intuitively branched into Functional, Integration, Smoke, Regression Tests and so on):

1. Logic Tests: These are tests that verify the implementation of a given class or method in complete isolation. The idea is to validate that provided an input, the method/class will always result in a given output.
2. Application Tests: These tests are loaded into an existing application after it finishes loading. Meaning the full application environment is available. This is particularly useful if you are looking for more realistic tests, as the environment is not completely isolated.
3. UI Tests: These are black box tests. The only information you have available is what the user sees. In other words, you test the app in the user’s perspective. You don’t have access to the application’s context, that is, you can’t inspect the application state as you would in white box tests (Logic and Application Tests).

The first two types of tests are implemented by creating Unit Test Case files. The third, by creating UI Test Case files.

I think we are good regarding establishing nomenclatures.

API Overview

The API is as strait-forward as they come. The only complaint I have so far is that no sleep-like wait method is available out of the box. If you need that to wait for an animation to complete for instance, you need to implement it yourself.

The fastest way for you to get up to speed, in my opinion, is by watching the UI Testing in Xcode video from WWDC 2015. It is a one hour long video, but it will tell you how the XCTest framework works, give you an overview on the available APIs and present some usage examples.

I don’t think there is much value in discussing the API itself in depth here as there is such a great introductory video and in-depth documentation made available by Apple, but if you are like me, you would enjoy some code snippets, specially on the beginning. Fortunately Apple already provides lots of code snippets in their own documentation and also in the XCTest header files.

So instead, I will focus on making sure you are aware right away of some very useful APIs and list some Tips & Tricks that will certainly prove useful to you:

• waitForExpectationsWithTimeout:handler: creates a point of synchronization, blocking the test thread until an expectation is met or the timeout expires. Using this API you can create test cases which depend on asynchronous operations, like network requests.

• expectationForPredicate:evaluatedWithObject:handler: is one of the helper methods that create expectations. This one is special and appears very often on UI tests. It creates an expectation that is fulfilled if the predicate returns true when evaluated with the given object. You use this method when you need to wait for a view to appear or disappear for instance.

• containingType:identifier: returns a new query for finding elements that contain a descendant matching the specification. This method is very useful to find a cell in a table view based on it’s contents.

Tips & Tricks

By now you already know almost everything you need to know to get to writing your own automated tests (if you watched the video I recommended). There are a few more comments I would like to share before I let you go 🙂

• When you create your test target, you don’t need to account for all your application dependencies nor include all source files you want to test in the test target. All you need to do is add your application as a dependency and set the Enable Testability flag to YES in your application’s target. This flag doesn’t take effect on Release builds. If you create a new project, these settings will be set by default.
• The XCTest framework has calls to automate many user actions, but not all. You can simulate a tap (with multiple fingers), a double tap, long press, swipe and typing (in text fields). The UIView’s accessibilityActivationPoint property is used by the system to define where in your view (x,y coordinates) the interaction is going to happen. By default accessibilityActivationPoint points to the center of your view.
• You rarely need to write complex queries. That is, app.buttons[your button ID] will always succeed in finding the button with that ID even if that button is placed deep in the view hierarchy (i.e.: within a UITableViewCell displayed on a popover). Avoid complex queries to keep your code easy to understand and maintain.
• The UIView’s accessibilityIdentifier was created with the purpose of uniquely identifying an interface element in UI Automation tests. Use that property instead of the accessibilityLabel. That will help you keep the code simple, independent from the current language set and won’t affect the UX.
• Create as many XCTestCase extensions as you can. Once you start writing UI tests, you will notice lots of user actions can be re-used to create new test cases. By creating extensions to aid in test development, you can end up having very complex tests written with just a few very legible lines of code!
• As you write your own extensions, handle failures with recordFailureWithDescription so your test fails where the APIs are called instead of within the frameworks you wrote. This will help you a lot when a Regression test fails.
• UI Test Cases are a great way to introduce Swift in your professional day-to-day life, because you can learn, experiment and introduce Swift into your project without affecting your application code (your customers and schedule are safe ;).

There is something you should know…

Unfortunately Xcode 7 is very unstable in that regard. You will often see your tests fail, not because they actually failed, but because Xcode couldn’t launch or terminate application, or because the application was launched but the launch parameters were not delivered or simply because it couldn’t attach to the process – don’t forget infamous FBSOpenApplicationErrorDomain error 1 – .

So be prepared to have some headaches when working on UI testing. But more importantly, take that into consideration when planning a Continuous Integration set up with Automation Testing using XCTest.

About this post

If you have any comments, please share. Also, if you feel like I should have touched (or elaborated) a specific subject or topic, don’t hesitate to ask. I will consider writing about it!