Intermittently failing tests with Protractor - promise order execution?

问题

I am facing intermittent protractor test failures across a range of test suites without any real pattern in the fail cases to indicate what could be going on, for example it's not the same tests that are failing. Sometimes I will get many failures and on other occasions just a single fail case.

I should point out that this tends to only happen when performing test runs on a Jenkins CI server we have configured (running under linux). Locally on Windows dev machines we may get a single fail case after 30-40 runs which I can live with!

The application we are testing is currently built with angular 1.5 and we are using angular material 1.1.3

Due to the animations used in angular material and the performance hit these can have, we have already tried disabling animations following this approach here which certainly make the tests quicker but dont help with the fail cases we are seeing/

I am at a point now where I am running one test suite over and over, after 5 successful runs it then failed on it's 6th attempt on our Jenkins CI environment\linux box, locally I have run this test many times now and no failures yet.

The test suite in question is detailed below along with a page object file snippet:

//test suite
describe('Operators View', function () {
  var operatorPage = require('./operators.po.js'),
    loginView = require('../login/login.po.js'),
    page = new operatorPage();

  describe('Large screen tests', function () {
    beforeAll(function () {
      loginView.login();
    });

    afterAll(function () {
      loginView.logout();
    });

    it('should create an operator', function () {
      page.settlementBtn.click();
      page.operatorsBtn.click();
      page.fabBtn.click();
      page.createOperator();
      expect(page.headline.getText()).toEqual('Operators');
    });
    });
});

// operators.po.js
var operatorsSection = function() {
    this.helper = new Helpers();
    this.headline = element(by.css('.md-headline'));
    this.settlementBtn = element(by.css('[ui-sref="settlement"]'));
    this.operatorsBtn = element(by.css('[ui-sref="operators"]'));
    this.fabBtn = element(by.css('.md-fab'));

    // Form Elements
    this.licenceNumber = element(by.model('vm.transportOperator.licenceNumber'));
    this.tradingName = element(by.model('vm.tradingName'));
    this.name = element(by.model('vm.name'));
    this.operatorAddressFirstLine = element(by.model('vm.transportOperator.address.line1'));
    this.operatorAddressCityTown = element(by.model('vm.transportOperator.address.line5'));
    this.operatorAddressPostCode = element(by.model('vm.transportOperator.address.postcode'));
    this.payeeAddressFirstLine = element(by.model('vm.transportOperator.payee.address.line1'));
    this.payeeAddressCityTown = element(by.model('vm.transportOperator.payee.address.line4'));
    this.payeeAddressPostCode = element(by.model('vm.transportOperator.payee.address.postcode'));
    this.opID = element(by.model('vm.transportOperator.fields.opID'));
    this.spID = element(by.model('vm.transportOperator.fields.spID'));
    this.schemeSelect = element(by.model('reference.scheme'));
    this.schemeOptions = element(by.exactRepeater('scheme in vm.schemes').row('0'));
    this.alias = element(by.model('reference.alias'));
    this.reference = element(by.model('reference.reference'));
    this.saveBtn = element(by.css('.md-raised'));


    this.createOperator = function() {
      this.licenceNumber.sendKeys(this.helper.getRandomId(10));
      this.tradingName.sendKeys('Protractor Trade Name LTD');
      this.name.sendKeys('Protractor Trade Name');
      this.operatorAddressFirstLine.sendKeys('Protractor Town');
      this.operatorAddressCityTown.sendKeys('Cardiff');
      this.operatorAddressPostCode.sendKeys('PT4 4TP');
      this.payeeAddressFirstLine.sendKeys('Protractor Town');
      this.payeeAddressCityTown.sendKeys('Cardiff');
      this.payeeAddressPostCode.sendKeys('PT4 4TP');
      this.opID.sendKeys('177');
      this.spID.sendKeys('Protractor Spid');
      this.schemeSelect.click();
      this.schemeOptions.click();
      this.alias.sendKeys('PTAlias');
      this.reference.sendKeys('Protractor');
      this.saveBtn.click();
    }

  };
module.exports = operatorsSection;

In this test suite after the call to createOperator from the PO file is invoked and the savteBtn is clicked, the application will transition to a state that shows a table of created entries (after successful creation of course). We are using angular ui-router also, currently on version 0.2.18

The expectation fails with:

Expected 'Create An Operator' to equal 'Operators'.

Yet the accompanying screenshot that was captured shows the table view with an 'Operators' heading, it seems the call to page.headline.getText() inside the expectation call is being invoked too soon, so before the database operation to create the item and the page change has had a chance to complete?

I have started wondering if this could be down to the order of promises executed by protractor. I have come across articles talking about control flow in protractor and why there may be occasions when you should hook into the result of a protractor call's promise using .then() - I found this

It got me wondering if I should move the call to my saveBtn.click(), that's called at the end of my page object's createOperator function, into the test suite, so doing something like:

it('should create an operator', function () {
      page.settlementBtn.click();
      page.operatorsBtn.click();
      page.fabBtn.click();
      page.createOperator();
      page.saveBtn.click().then(function(){
        expect(page.headline.getText()).toEqual('Operators');
      });
    });

I'm starting to clutch at straws here to be honest, so any thoughts\advice from the community here would be much appreciated.

Thanks!

回答1:

As requested, here is the function I use for waiting for URLs to be as they should.

public waitForUrlToBeLike (urlPart: string, timeout: number = 10000) {
    return browser.wait(() => {
        return browser.driver.getCurrentUrl().then((url) => {
            let regex = new RegExp(urlPart);
            return regex.test(url);
        });
    }, timeout);
}

I also use the following a lot to wait for elements to be present before making assertions on them:

public waitTillPresent (element: ElementFinder, timeout: number = 10000) {
    return browser.wait(() => {
        return element.isPresent();
    }, timeout);
}

来源：https://stackoverflow.com/questions/42902624/intermittently-failing-tests-with-protractor-promise-order-execution