Customized Execution status in Robot Framework

问题

In Robot Framework, the execution status for each test case can be either PASS or FAIL. But I have a specific requirement to mark few tests as NOT EXECUTED when it fails due to dependencies.

I'm not sure on how to achieve this. I need expert's advise for me to move ahead.

回答1:

There is nothing you can do, robot only supports two values for the test status: pass and fail. You can mark a test as non-critical so it won't break the build, but it will still show up in logs and reports as having been run.

The robot core team has said they will not support this feature. See issue 1732 for more information.

Even though robot doesn't support the notion of skipped tests, you have the option to write a script that scans output.xml and removes tests that you somehow marked as skipped (perhaps by adding a tag to the test). You will also have to adjust the counts of the failed tests in the xml. Once you've modified the output.xml file, you can use rebot to regenerate the log and report files.

回答2:

If you only need the change to be made for your log/report files you should take a look here for implementing a SuiteVisitor for the --prerebotmodifier option. As stated by Bryan Oakley, this might screw up your pass/fail count if you don't keep that in mind.

Currently it doesn't seem to be possible to actually alter the test-status before output.xml is created, but there are plans to implement it in RF 3.0. And there is a discussion for a skip status

Another more complex solution would be to create your own output file through implementing a listener to use with the --listener option that creates an output file as it fits your needs (possibly alongside with the original output.xml).

There is also the possibility to set tags during test execution, but im not familar with that yet so I can't really tell anything about that atm. That might be another possibility to account for those dependency-failures, as there are options to ignore certain tagged keywords for the log/report generation

回答3:

Until a SKIP status is implemented, you can use exitonfailure to stop further execution if a critical test failed, and then change the output.xml (and the tests results.html) to show those tests as "NOT_RUN" (gray color), rather than "FAILED" (red color).

Here's an example (Tested on RobotFramework 3.1.1 and Python 3.6):

First create a new class that extends the abstract class ResultVisitor:

class ResultSkippedAfterCritical(ResultVisitor):

    def visit_suite(self, suite):
        suite.set_criticality(critical_tags='Critical')
        for test in suite.tests:
            if test.status == 'FAIL' and "Critical failure occurred" in test.message:
                test.status = 'NOT_RUN'
                test.message = 'Skipping test execution after critical failure.'

Assuming you've already created the suite (for example with TestSuiteBuilder()), run it without creating report.html and log.html:

outputDir = suite.name.replace(" ", "_") 
outputFile = "output.xml"

logger.info(F"Running Test Suite: {suite.name}", also_console=True)
result = suite.run(output=outputFile, outputdir=outputDir, \
         report=None, log=None, critical='Critical', exitonfailure=True)

Notice that I've used "Critical" as the identifing tag for critical tests, and exitonfailure option.

Then, revisit the output.xml, and create report.html and log.html from it:

revisitOutputFile = os.path.join(outputDir, outputFile)
logger.info(F"Checking skipped tests in {revisitOutputFile} due to critical failures", also_console=True)
result = ExecutionResult(revisitOutputFile) 
result.visit(ResultSkippedAfterCritical())
result.save(revisitOutputFile)

reportFile = 'report.html'
logFile = 'log.html'
logger.info(F"Generating {reportFile} and {logFile}", also_console=True)
writer = ResultWriter(result)
writer.write_results(outputdir=outputDir, report=reportFile, log=logFile)

It should display all the tests after the critical failure with grayed status = "NOT_RUN":

回答4:

I solved it this way:

Run Keyword If ${blabla}==${True} do-this-task ELSE log to console ${PREV_TEST_STATUS}${yellow}| NRUN |

test not executed and marked as NRUN

回答5:

Actually, you can SET TAG to run whatever keyword you like (for sanity testing, regression testing...)

Just go to your test script configuration and set tags

And whenever you want to run, just go to Run tab and select check-box Only run tests with these tags / Skip tests with these tags

And click Start button :) Robot framework will select any keyword that match and run it.

Sorry, I don't have enough reputation to post images :(

来源：https://stackoverflow.com/questions/25422847/customized-execution-status-in-robot-framework

标签

python

robotframework