How can I run my performance tests more than ten times?

ぃ、小莉子 提交于 2020-12-29 05:11:28

问题


By default Xcodes performance tests are run ten times and my result is the average of those ten tests. The problem is the averaged result varies considerably each time I run it so I have to run the test at least five times to get a converged result. This is both tedious and time consuming; is there a way to configure either XCode or the unit test itself to run more than ten times?


回答1:


In the latest Xcode (11.0+) you don't need swizzling to change iterations count. Use the following function:

func measure(options: XCTMeasureOptions, block: () -> Void)

This will allow you to specify XCTMeasureOptions which has iterationCount property.

Interesting note from docs:

A performance test runs its block iterationCount+1 times, ignoring the first iteration and recording metrics for the remaining iterations. The test ignores the first iteration to reduce measurement variance associated with “warming up” caches and other first-run behavior.




回答2:


a class dump of XCTestCase exposes this method:

- (void)_recordValues:(id)arg1 forPerformanceMetricID:(id)arg2 name:(id)arg3 unitsOfMeasurement:(id)arg4 baselineName:(id)arg5 baselineAverage:(id)arg6 maxPercentRegression:(id)arg7 maxPercentRelativeStandardDeviation:(id)arg8 maxRegression:(id)arg9 maxStandardDeviation:(id)arg10 file:(id)arg11 line:(unsigned long long)arg12;

when this method is swizzled the first parameter (arg1) has the 10 durations:

["0.003544568",
"0.003456569",
"0.003198263",
"0.003257955",
"0.003508724",
"0.003454298",
"0.003461192",
"0.00423787",
"0.003359195",
"0.003335757"]

i added 4 new values (1.0, 2.0, 3.0, 4.0) to the end of this list before passing it back to the original implementation, but unfortunately a different class that observes, XCTestLog, has an internal sanity check that gets tripped:

Assertion failure in +[XCTestLog _messageForTest:didMeasureValues:forPerformanceMetricID:name:unitsOfMeasurement:baselineName:baselineAverage:maxPercentRegression:maxPercentRelativeStandardDeviation:maxRegression:maxStandardDeviation:file:line:]
caught "NSInternalInconsistencyException", "Performance Metrics must provide 10 measurements." 

once the XCTestLog method is also overridden so it doesn't assert, the additional 4 values can be added without any complaints. unfortunately the view still only shows the 10 results.

it does however update the total time + standard deviation values in the mini view.

Before Swizzling

After Swizzling and adding 4 values

in order to view more than 10 results one would probably have to tweak the XCode runtime to tell the table to show more items.




回答3:


The short answer: No, there is no current interface exposed to allow a measure block more than ten times.

The longer answer: No, but there is an interface exposed to modify certain metrics of the measure block. The default metrics are in a String array returned from defaultPerformanceMetrics. There appears to only be one metric supported right now: XCTPerformanceMetric_WallClockTime. This only specifies a performance metric that records the time in seconds between calls to a performance test's startMeasuring() and stopMeasuring() methods, and not the number of times the block is run.




回答4:


I typically expand the Standard Deviation value to include the range I'd accept.

One obvious workaround for you is to perform the operation 5 times within the measure block you are executing. (You'll want to change the expected time.)

    measure {
        for _ in [1...5] {
            // ...
        }
    }


来源:https://stackoverflow.com/questions/38672129/how-can-i-run-my-performance-tests-more-than-ten-times

易学教程内所有资源均来自网络或用户发布的内容,如有违反法律规定的内容欢迎反馈
该文章没有解决你所遇到的问题?点击提问,说说你的问题,让更多的人一起探讨吧!