Xcode 5 says tests failed but shows green checkmarks

Question

I have written a few unit tests in the new XCTest framework of Xcode 5.

The tests seem to pass successfully most of the time, but I've come across a strange behaviour where if I run the tests enough times consecutively, eventually I'll get a 'test failed' message and the console will show Program ended with exit code 0, but there are green checkmarks beside all the tests.

What's going wrong here?

Answer 1

This is definitely a bug. Submit a report via https://bugreport.apple.com/ feel free to reference my report (15251050).

Answer 2

Until the bug is fixed, close the project and re-open it.

Answer 3

Build Clean worked for me... sometimes anyways. Product -> Clean.

Answer 4

If you look at the output console, you should notice that Xcode starts running test suite:

Test Suite 'SomeClassTests' started at 2013-11-30 23:19:34 +0000

but suddenly exits without running test cases:

Program ended with exit code: 0

and shows a notification that test failed.

Restarting Xcode only helps temporarily as the problem quickly comes back.

I stopped executing tests in Xcode and run all tests by command line using xctool.

Answer 5

For me, I think this problem was linked to incorrect OCMock configuration. Removing OCMock (completely) and re-adding it to the project exactly as recommended by OCMock seems to have fixed it.

A couple specific things I did wrong:

1. I was using #import "OCMock.h instead of #import "OCMock/OCMock.h. Xcode autocompleted the former.
2. I had a bad link leftover in library search paths from previously trying to install OCMock with a different directory structure.
3. This is dumb, but in trying to debug this issue, I commented out the instantiation of my OCMockObject in my setup method.. and didn't remember to uncomment it. The nil OCMockObject caused similar behavior as described in the other answers here. This is odd.. In my current project I have two tests that use the OCMockObject that is instantiated in my setup. Both should fail. However, one of them passes, and the only difference that I can see between the two is that the one that is passing previously passes legitimately...

Anyways, with OCMock setup correctly my tests are working as they should

Answer 6

Having just run into this in my own updated XCode 5 thought I'd share my similar resolutions.

I notice if I hit any breakpoint when running the test and simply continue it reports as passing, very odd indeed and unacceptable for running your whole suite...

I also noticed adding a usleep as someone mentioned, as well as doing a clean build prior to running the tests seems to resolve the issue. However creating a new test often re-introduces the issue at least for that new test. Hope Apple fixes this soon!

Answer 7

This appears to have been fixed in XCode 5.1.1

Answer 8

This bug has been corrected in Xcode 5.1 :

Xcode 5.1 release notes

Notes specifically mention :

Test target [test name] encountered an error (Test process exited with code -1)

Attempt recovery by quitting and restarting the simulator. (15929053)

Answer 9

Despite the test I'm using is logic tests, Xcode wants to iPhone simulator to be running. Once in the agony of trying to get tests working again I've reset the iPhone Simulator contents and Settings from it's menu, and this helped.