Random Selenium E2e Tests Fail because of timeouts on Azure DevOps but work locally and with remote Selenium (BrowserStack Automate)

Question

I\'ve got a suite of Selenium tests that work perfectly in my local environment and using Browserstack Automate, but fail on Azure DevOps.

There are no configurati

Answer 1

Here are some steps I would do :

1. What helped us in a similar case is to temporarily add a video recorder to tests then watch the test execution process on a VM from start to fail. There could be some clues on a video that help to see what is actually going wrong I was able to find this link for a C# example

2. Also, I would double check to make sure browser versions on Azure are exactly the same as in the run where it all works well. Making them the same is crucial to make sure there is no 'magic'. Same for the default browser window size.

3. I would do more detailed analysis of places where different tests fail.

   • is it possible to spot the similarities between different test failures. Does it always happen after clicks? after reloading pages? after anything else that is similar? If yes - try with the weirdest yet simple and sometimes lifesaving solution and add a 3-5 sec sleep before/after an action that is before the failure. (add sleeps with a condition to only happen when it's an Azzure run) (yes, sleeps are not recommended and {a lot of well-known info why they are not recommended could have been here} but... if they magically save your runs you can then replace them with some smart waits for sure)
   • is it possible that failures happen at some certain time? After same time after run start? At the same time during the day?

4. If you use Date/time APIs in your code, make sure the System time/locale/timezone settings are exactly the same. Or that the days do not change during the test runs. All in all - investigate around dates.

I know that the above is more like a general advice, but from my experience such "random failures" could be caused by literally anything that seems "not worth attention".