Selenium Java (maven project): TestNG results differs from ReportNG

Question

I tried running testng.xml and the results were:

===============================================

Default test

Tests run: 14, Failures: 6, Skips: 0

Default suite

Total tests run: 14, Failures: 6, Skips: 0

===============================================

Now, I disabled deafult TestNG listener and added ReportNG listner in testng.xml. The testng.xml. looks like this:

<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE suite SYSTEM "http://testng.org/testng-1.0.dtd">
<suite name="Suite">
    <listeners>
        <listener class-name="org.uncommons.reportng.HTMLReporter" />
        <listener class-name="org.uncommons.reportng.JUnitXMLReporter" />
    </listeners>
    <test name="Test">
        <classes>
            <class name=".URL_Daily" />
        </classes>
    </test> <!-- Test -->
</suite> <!-- Suite -->

Following the steps, I added velocity, guice and reportng dependencies in maven's pom.xml.

Upon executing the testsuite testng.xml, following folders were created (marked in red box).

Upon opening index.html the results are like this:

As expected, an output folder should have been created by ReportNG which is not noticed in my scenario. Secondly, the results are different.

In addition, the report index.html doesn't look how actually it should be. Can somebody tell what's wrong?

Some details for your reference:

OS: Windows 7

Guice.jar version: guice-4.1.0

ReportNG Version: reportng-1.1.4

Velocity version: velocity-dep-1.4

TestNG Version: testng-6.11

Selenium Version: selenium-java-3.5.3

Eclipse: eclipse oxygen

My test case is as follows:

public class MwSites {
WebDriver driver;

@BeforeTest     
public void setup ()    
{
    System.setProperty("webdriver.chrome.driver", "F:\\Automation\\Drivers\\Selenium Drivers\\chromedriver_win32\\chromedriver.exe");
    ChromeOptions options = new ChromeOptions();
    options.addArguments("test-type");
    options.addArguments("start-maximized");
    options.addArguments("--js-flags=--expose-gc");  
    options.addArguments("--enable-precise-memory-info"); 
    options.addArguments("--disable-popup-blocking");
    options.addArguments("--disable-default-apps");
    options.addArguments("test-type=browser");
    options.addArguments("disable-infobars");
    driver = new ChromeDriver(options);
    driver.manage().timeouts().implicitlyWait(60, TimeUnit.SECONDS);
}

@AfterTest
public void Quit () throws InterruptedException
{
    driver.quit();
}

@Test(priority = 0)     
public void MI_Demo () throws InterruptedException
{
    driver.navigate().to("http://demo.movingwalls.com/mi/#/login");
    Assert.assertEquals("Login", driver.getTitle());
    if (driver.getTitle()=="Login"){
        System.out.println("Failed to access MI in demo environment");
    }
    else{
        System.out.println("MI is successfully accessed in demo environment");
    }
}

Answer 1

As your tests got skipped, you may want to check test results xml first to find out the root cause (exception).

On the other hand, I'm just wondering what's the initial intention of using a library, which is not maintained for 4 years? ReportNG is dead. It's a fact. And if you take a look at its sources, you'll see which TestNG version it uses (6.8). You could easily assume that its reporter is based on old TestNG API.

When you add TestNG 6.11 as a dependency, the probability of a jar hell occurrence is quite high. And I wouldn't be surprised, if it's a root cause of your issue.

If you haven't decided which reporting system to use yet, I'd recommend to take a look at a modern Allure 2 framework.

Otherwise, just check a stacktrace first to understand the root cause of your issue.

Answer 2

I was able to resolve this using the specified versions.

By default the ReportNG reports are generated under target/surefire-reports folder as configured by surefire plugin. This is something that can be changed via surefire configuration.

You can add any test classes, but for the sake of ease, I am just including pom file and two sample classes that I have used.

<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
         xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
    <modelVersion>4.0.0</modelVersion>
    <groupId>com.rationaleemotions</groupId>
    <artifactId>46923243</artifactId>
    <version>1.0-SNAPSHOT</version>
    <packaging>jar</packaging>

    <name>46923243</name>
    <url>http://maven.apache.org</url>

    <properties>
        <project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
        <maven.compiler.source>1.8</maven.compiler.source>
        <maven.compiler.target>1.8</maven.compiler.target>
    </properties>

    <dependencies>
        <!-- https://mvnrepository.com/artifact/velocity/velocity-dep -->
        <dependency>
            <groupId>org.seleniumhq.selenium</groupId>
            <artifactId>selenium-java</artifactId>
            <version>3.5.3</version>
        </dependency>
        <!-- https://mvnrepository.com/artifact/com.google.inject/guice -->
        <dependency>
            <groupId>com.google.inject</groupId>
            <artifactId>guice</artifactId>
            <version>3.0</version>
        </dependency>
        <dependency>
            <groupId>org.testng</groupId>
            <artifactId>testng</artifactId>
            <version>6.12</version>
            <scope>test</scope>
        </dependency>
        <!-- https://mvnrepository.com/artifact/org.uncommons/reportng -->
        <dependency>
            <groupId>org.uncommons</groupId>
            <artifactId>reportng</artifactId>
            <version>1.1.4</version>
            <scope>test</scope>
        </dependency>
    </dependencies>
    <build>
        <plugins>
            <plugin>
                <groupId>org.apache.maven.plugins</groupId>
                <artifactId>maven-surefire-plugin</artifactId>
                <version>2.20.1</version>
                <configuration>
                    <!-- Suite testng xml file to consider for test execution -->
                    <suiteXmlFiles>
                        <suiteXmlFile>src/test/resources/testng.xml</suiteXmlFile>
                    </suiteXmlFiles>
                    <properties>
                        <property>
                            <name>usedefaultlisteners</name>
                            <value>false</value>
                        </property>
                    </properties>
                </configuration>

            </plugin>
        </plugins>
    </build>
    <repositories>
        <repository>
            <id>jcenter</id>
            <name>jcenter</name>
            <url>http://jcenter.bintray.com</url>
        </repository>
    </repositories>
</project>

The test classes, that I used

package com.rationaleemotions;

import org.openqa.selenium.WebDriver;
import org.openqa.selenium.chrome.ChromeDriver;
import org.openqa.selenium.chrome.ChromeOptions;
import org.testng.Assert;
import org.testng.annotations.AfterClass;
import org.testng.annotations.BeforeClass;
import org.testng.annotations.Test;

import java.util.concurrent.TimeUnit;

public class MwSites {
    private WebDriver driver;

    @BeforeClass
    public void setup() {
        ChromeOptions options = new ChromeOptions();
        options.addArguments("test-type");
        options.addArguments("start-maximized");
        options.addArguments("--js-flags=--expose-gc");
        options.addArguments("--enable-precise-memory-info");
        options.addArguments("--disable-popup-blocking");
        options.addArguments("--disable-default-apps");
        options.addArguments("test-type=browser");
        options.addArguments("disable-infobars");
        driver = new ChromeDriver(options);
        driver.manage().timeouts().implicitlyWait(60, TimeUnit.SECONDS);
    }

    @AfterClass
    public void Quit() throws InterruptedException {
        driver.quit();
    }

    @Test
    public void MI_Demo() throws InterruptedException {
        driver.navigate().to("http://demo.movingwalls.com/mi/#/login");
        Assert.assertEquals("Login", driver.getTitle());
        if ("Login".equals(driver.getTitle())) {
            System.out.println("Failed to access MI in demo environment");
        } else {
            System.out.println("MI is successfully accessed in demo environment");
        }
    }
}

package com.rationaleemotions;

import org.testng.Assert;
import org.testng.Reporter;
import org.testng.annotations.Test;

public class TestClassWithMultipleResults {
    @Test
    public void testMethodWillPass() {
        Reporter.log("This test method will pass", true);
    }

    @Test
    public void testMethodWillFail() {
        Reporter.log("This test method will fail", true);
        Assert.fail();
    }

    @Test(dependsOnMethods = "testMethodWillFail")
    public void testMethodWillSkip() {
    }
}

Suite xml that I used:

<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE suite SYSTEM "http://testng.org/testng-1.0.dtd">
<suite name="Suite">
    <listeners>
        <listener class-name="org.uncommons.reportng.HTMLReporter"/>
        <listener class-name="org.uncommons.reportng.JUnitXMLReporter"/>
    </listeners>
    <test name="Test">
        <classes>
            <class name="com.rationaleemotions.MwSites"/>
            <class name="com.rationaleemotions.TestClassWithMultipleResults"/>
        </classes>
    </test> <!-- Test -->
</suite> <!-- Suite -->

Here are the screenshots from the ReportNG reports