Tests on Bitrise (some consistently, some inconsistently) failing with "AX action" errors

2
votes

I have a number of tests that I've written for an pretty substantial app (the app's been around for a few years longer than I have at this company) in the XCUITest framework. All of the tests pass consistently on my laptop and also the laptops of the other engineers.

When running tests on Bitrise, the first UI test fails every time on the setup phase with the following message:

testFixtureAttachment, UI Testing Failure - Failed to perform AX action for monitoring the animations of (app), error: Error -25204 performing AXAction 2043

Other tests usually pass but sometimes fail with errors such as:

UI Testing Failure - Failed to perform AX action for monitoring the event loop of (app), error: Error -25204 performing AXAction 2042

UI Testing Failure - Failed to scroll to visible (by AX action) TextField 0x7fe800f9fa20: traits: ... error: Error -25204 performing AXAction 2003

How can I resolve this so that, at a minimum, I don't have my first test always failing on setup?

2
iosxcode-ui-testingbitrise

2 Answers

3
votes

Per feedback from the XCUITest team, this is happening because the simulator is CPU-starved and AX Action is timing out when spinning up the app. I was able to reproduce this on my own machine (which did not normally exhibit the failure) by using Power Management to step down my CPU.

Increasing the CPU allocation in Bitrise is the obvious solution. However, there is another, bizarre solution!

Immediately before the setup function for the tests, I have the following line:

let app = XCUIApplication()

This allows my various tests to call app. instead of the longer, full syntax.

Removing this line was found to prevent the error from occurring. This was found via this Apple Developer forum thread:

https://forums.developer.apple.com/thread/4472

So... hopefully that fixes this for other people, but it fixed it for me.

1
votes

This is usually related to slower environments. Xcode is not really good to guarantee UI Test execution on older & slower machines. This is true for virtualized environments (like the one Bitrise.io uses), as well as for older machines or machines with HDD storage instead of SSD.

There are workarounds which might or might not help, depending on your project. You can find a list of related issues & possible solutions at: https://bitrise-io.github.io/devcenter/ios/known-xcode-issues/.

From the link, the solutions which work in most of the cases:

  • Try another Xcode version.
  • Try another Simulator device (e.g. instead of running the test in "iPhone 6" try it with "iPhone 6s Plus")
  • Some users had success with splitting the tests into multiple Schemes, and running those separately, with separate Test steps.
  • Others reported that if you add a delay after app.launch() it can leave enough time for Xcode / the iOS Simulator to initialize the Accessibility labels, so that UI Tests can properly find the elements by the Accessibility labels.

Related StackOverflow & other forum links: