Random failure of selenium test on test server

Question

I'm working on a project which uses nodejs and nighwatch for test automation. The problem here is that the tests are not reliable and give lots of false positives. I did everything to make them stable and still getting the errors. I went through some blogs like https://bocoup.com/blog/a-day-at-the-races and did some code refactoring. Did anyone have some suggestions to solve this issue. At this moment I have two options, either I rewrite the code in Java(removing nodejs and nightwatch from solution as I'm far more comfortable in Java then Javascript. Most of the time, struggle with the non blocking nature of Javascript) or taking snapshots/reviewing app logs/run one test at a time.

Test environment :-

• Server -Linux
• Display - Framebuffer
• Total VM's -9 with selenium nodes running the tests in parallel.
• Browser - Chrome

Type of errors which I get is element not found. Most of the time the tests fail as soon the page is loaded. I have already set 80 seconds for timeout so time can't be issue. The tests are running in parallel but on separate VM's so I don't know whether it can be issue or not.

Edit 1: - Was working on this to know the root cause. I did following things to eliminate random fails: - a. Added --suiteRetries to retry the failed cases. b. Went through the error screenshot and DOM source. Everything seems fine. c. Replaced the browser.pause with explicit waits

Also while debugging I observed one problem, maybe that is the issue which is causing random failures. Here's the code snippet

for (var i = 0; i < apiResponse.data.length; i++) {
  var name = apiResponse.data[i];
  browser.useXpath().waitForElementVisible(pageObject.getDynamicElement("@topicTextLabel", name.trim()), 5000, false);
  browser.useCss().assert.containsText(
    pageObject.getDynamicElement("@topicText", i + 1),
    name.trim(),
    util.format(issueCats.WRONG_DATA)
  );

}

I added the xpath check to validate if i'm waiting enough for that text to appear. I observed that visible assertion is getting passed but in next assertion the @topicText is coming as previous value or null.This is an intermittent issue but on test server happens frequently.

Answer 1

There is no magic bullet to brittle UI end to end tests. In the ideal world there would be an option set avoid_random_failures=true that would quickly and easily solve the problem, but for now it's only a dream.

Simple rewriting all tests in Java will not solve the problem, but if you feel better in java, then I would definitely go in that direction.

---

As you already know from this article Avoiding random failures in Selenium UI tests there are 3 commonly used avoidance techniques for race conditions in UI tests:

1. using constant sleep
2. using WebDriver's "implicit wait" parameter
3. using explicit waits (WebDriverWait + ExpectedConditions + FluentWait)

These techniques are also briefly mentioned on WebDriver: Advanced Usage, you can also read about them here: Tips to Avoid Brittle UI Tests

Methods 1 and 2 are generally not recommended, they have drawbaks, they can work well on simple HTML pages, but they are not 100% realiable on AJAX pages, and they slow down the tests. The best one is #3 - explicit waits.

---

In order to use technique #3 (explicit waits) You need to familiarize yourself and be comfortable with the following WebDriver tools (I point to theirs java versions, but they have their counterparts in other languages):

• WebDriverWait class
• ExpectedConditions class
• FluentWait - used very rarely, but very usefull in some difficult cases

ExpectedConditions has many predefinied wait states, the most used (in my experience) is ExpectedConditions#elementToBeClickable which waits until an element is visible and enabled such that you can click it.

How to use it - an example: say you open a page with a form which contains several fields to which you want to enter data. Usually it is enought to wait until the first field appears on the page and it will be editable (clickable):

By field1 = By.xpath("//div//input[.......]");
By field2 = By.id("some_id");
By field3 = By.name("some_name");
By buttonOk = By.xpath("//input[ text() = 'OK' ]");
....
....
WebDriwerWait wait = new WebDriverWait( driver, 60 ); // wait max 60 seconds

// wait max 60 seconds until element is visible and enabled such that you can click it
// if you can click it, that means it is editable
wait.until( ExpectedConditions.elementToBeClickable( field1 ) ).sendKeys("some data" );
driver.findElement( field2 ).sendKeys( "other data" );
driver.findElement( field3 ).sendKeys( "name" );
....
wait.until( ExpectedConditions.elementToBeClickable( buttonOK)).click(); 

The above code waits until field1 becomes editable after the page is loaded and rendered - but no longer, exactly as long as it is neccesarry. If the element will not be visible and editable after 60 seconds, then test will fail with TimeoutException.
Usually it's only necessary to wait for the first field on the page, if it becomes active, then the others also will be.