Which JS benchmark site is correct?

问题

I created a benchmark on both jsperf.com and jsben.ch, however, they're giving substantially different results.

JSPerf: https://jsperf.com/join-vs-template-venryx
JSBench: http://jsben.ch/9DaxR

Note that the code blocks are exactly the same.

On jsperf, block 1 is "61% slower" than the fastest:

On jsbench, block 1 is only 32% slower than the fastest: ((99 - 75) / 75)

What gives? I would expect benchmark sites to give the same results, at least within a few percent.

As it stands, I'm unable to make a conclusion on which option is fastest because of the inconsistency.

EDIT

Extended list of benchmarks:

https://jsperf.com/join-vs-template-venryx
https://jsbench.me/f3k3g71sg9
http://jsbench.github.io/#7f03c3d3fdc9ae3a399d0f2d6de3d69f
https://run.perf.zone/view/Join-vs-Template-Venryx-1512492228976
http://jsben.ch/9DaxR

Not sure which is the best, but I'd skip jsben.ch (the last one) for the reasons Job mentions: it doesn't display the number of runs, the error margin, or the number of operations per second -- which is important for estimating absolute performance impact, and enabling stable comparison between benchmark sites and/or browsers and browser versions.

(At the moment http://jsbench.me is my favorite.)

回答1:

March 2019 Update: results are inconsistent between Firefox and Chrome - perf.zone behave anomalously on Chrome, jsben.ch behaves anomalously on Firefox. Until we know exactly why the best you can do is benchmark on multiple websites (but I'd still skip jsben.ch, the others give you a least some error margin and stats on how many runs were taken, and so on)

TL;DR: ~~running your code on perf.zone and on jsbench.github.io (see here and here), the results closely match jsperf.~~ Personally, and for other reasons than just these results, I trust these three websites more than jsben.ch.

Recently, I tried benchmarking the performance of string concatenation too, but in my case it's building one string out of 1000000+ single character strings (join('') wins for numbers this large and up, btw). On my machine the jsben.ch timed out instead of giving a result at all. Perhaps it works better on yours, but for me that's a big warning sign:

http://jsben.ch/mYaJk

http://jsbench.github.io/#26d1f3705b3340ace36cbad7b24055fb

https://run.perf.zone/view/join-vs-concat-when-dealing-with-very-long-lists-of-single-character-strings-1512490506658

(I can't be bothered to ever have to deal with jsperf's not all tests inserted again, sorry)

At the moment I suspect but can't prove that perf.zone has slightly more reliable benchmark numbers:

when optimising lz-string I used jsbench.github.io for a very long time, but at some point I noticed there were impossibly large error margins for certain types of code, over 100%.
running benchmarks on mobile is fine with jsperf.com and perf.zone, but jsbench.github.io is kinda janky and the CSS breaks while running tests.

Perhaps these two things are related: perhaps the method that jsbench.github.io uses to update the DOM introduces some kind of overhead that affects the benchmarks (they should meta-benchmark that...).

Note: perf.zone is not without its flaws. It sometimes times out when trying to save a benchmark (the worst time to do so...) and you can only fork your own code, not edit it. But the output still seems to be more in line with jsperf, and it has a really nice "quick" mode for throwaway benchmarking

回答2:

AFAIK one issue is that various JavaScript engines optimize vastly differently based on the environment.

I have a test of the exact same function that produces different results based on where the function is created. In other words, for example, in one test it's

const lib = {}
lib.testFn = function() {
   ....
}

And in other it's

const lib = {
 testFn: function() {
   ....
 },
};

and in another it's

function testFn() {
   ....
}

const lib = {}
lib.testFn = testFn

and there's a >10% difference in results for a non-trivial function in the same browser and different results across browsers.

What this means is no JavaScript benchmark is correct because how that benchmark runs it's tests, as in the test harness itself, affects the results. The harness for example might XHR the test script. Might call eval. Might run the test in a worker. Might run the test in an iframe. And the JS engine might optimize all of those differently.

来源：https://stackoverflow.com/questions/44612083/which-js-benchmark-site-is-correct

标签

javascript

performance

benchmarking

jsperf