Categorizing Tests
Overview
UITests may be grouped into logical categories by adorning them with the CategoryAttribute from the NUnit
framework. This provides some flexibility on what tests are to be run.
For example, an application may have one set of tests that are specifically for tablets, and another set that
are for phones. This makes it possible to run the tablet specific tests seperate from the phone tests.
Another popular scenario is to segregate slow tests from fast tests. The fast tests are run more frequently,
perhaps at each commit to source code control. The slow tests are run less frequently, for example at the
end of the day.
Requirements
Running tests by category is not supported in either Xamarin Studio or Visual Studio. Running tests
according to category can only be performed at the command line.
Categorizing Tests
UITests can be categorised either by text fixture or by test by adding the CategoryAttribute to the
class or method. It is possible to assign more than one category. The following class shows an example of
categorization:
[TestFixture]
[Category("nerp")]
public class Tests
{
iOSApp app;
[SetUp]
public void BeforeEachTest()
{
app = ConfigureApp.iOS.StartApp();
}
[Test]
[Category("derp")]
[Category("erp")]
public void CreditCardNumber_TooShort_DisplayErrorMessage()
{
app.WaitForElement(c=>c.Class("UINavigationBar").Marked("Simple Credit
Card Validator"));
app.EnterText(c=>c.Class("UITextField"), new string('9', 15));
app.Tap(c=>c.Marked("Validate Credit Card").Class("UIButton"));
app.WaitForElement(c => c.Marked("Credit card number is too
short.").Class("UILabel"));
}
[Test]
[Category("flerp")]
public void CreditCardNumber_TooLong_DisplayErrorMessage()
{
app.WaitForElement(c=>c.Class("UINavigationBar").Marked("Simple Credit
Card Validator"));
app.EnterText(c=>c.Class("UITextField"), new string('9', 17));
app.Tap(c=>c.Marked("Validate Credit Card").Class("UIButton"));
app.WaitForElement(c => c.Marked("Credit card number is too
long.").Class("UILabel"));
}
}
Xamarin Test Cloud does not honor the ExplicitAttribute; tests marked as Explicit will still be run.
Running Tests According to Category
There are two ways to run UITests according to category:
test-console.exe – this is the command line utility for submitting tests to Xamarin Test Cloud.
nunit-console.exe – this is the command line runner for NUnit tests. It is also used to run
UITests locally.
Submitting Tests to Test Cloud by Category
You can instruct Test Cloud to run a subset of your tests using the --category parameter.
test-cloud.exe <APK or IPA> <TEAM_API_KEY> --user <EMAIL> --category flerp
It may be specified multiple times to run the union of several test categories, for example:
test-cloud.exe <APK or IPA> <TEAM_API_KEY> --user <EMAIL> --category flerp -category erp
Running Tests Locally by Category
Running tests locally is accomplished using nunit-console.exe, NUnit's command line runner. The
command line switch -include identifies the test categories to run, while the switch -exclude specifies
the test categories to ignore.
nunit-console
./CreditCardValidator.iOS.UITests/bin/Debug/CreditCardValidator.iOS.UITests.dll
-include=flerp
Running tests locally is accomplished using nunit-console.exe, NUnit's command line runner. The
command line switch /include identifies the test categories to run, while the switch /exclude specifies
the test categories to ignore.
nunit-console
.\CreditCardValidator.iOS.UITests\bin\Debug\CreditCardValidator.iOS.UITests.dll
/include:flerp
Please the see the NUnit console documentation for more information on how to use category expressions
to include or exclude tests according to category combinations.
When running tests locally, IApp must be configured with the path to the application and the application
bundle.
Summary
In this guide we discussed how to categorize tests and run them by category at either the command line or
in Test Cloud.