Home > pleasantries > The case against benchmark testing in desktop computers

The case against benchmark testing in desktop computers

Nowadays, every computer review you read, every manufacturer and everyone else associated with computer testing, places emphasis on benchmarking. And I say that that is not good.

Benchmarks are defined as a series of standardized tests that are applied to the computer in question, and the results being quoted against a comparison chart.

Truth is, benchmarking is more incorrect than you know.

For one, benchmarking emulates ‘ideal’ conditions, such as those that are never there in real world situations. A normal benchmark testing for hard disk read speeds would not take into account the physical position of data placement, how many programs are trying to read to the disk at once, how fast the armatures move from one point to another, and so on.

Secondly, benchmarking is not very accurate for desktops. No two desktops are likely to be configured the same way, and a machine that is otherwise higher in the benchmarks may actually be slower during normal use than a machine with lower marks – simply because the loadout of the laptops is different, and each machine is optimized for different behavior.

Thirdly, it is more important for desktops to feel fast. If you are buying a laptop, it may perhaps be far better if you could do hands on testing, rather than have to look at benchmarks.

Fourthly, standard benchmarks often test individual units of a computer. 3DMark benchmarks the graphics, PCMark mainly checks the processor, etc. What needs to be tested is the whole computer as a whole, not one part at a time. And that can make a BIG difference.

Fifthly, computers are often used in ways different to what the benchmarks test them. One benchmark may test the processor for super number crunching, whereas that computer may normally be used for, say, multimedia. Another benchmark may test multitasking, when the computer’s main use may be Adobe Photoshop. That is where the problem lies: benchmarks are often portrayed – and accepted – as absolute ranks, when they are almost always relative to task specific requirement.

Sixthly, benchmarks may often be optimized for one specific kind of computer hardware. For example, nowadays, Intel processors of the Core 2 Duo breed are said to be triumphing over the AMD 64 X2 processors – based on benchmarks. But what if the benchmark algorithms are more optimized for Intel processors? They would then obviously be faster on the Intel! Even more complicated, I do know of a few benchmark results that run faster on an AMD Turion X2 than a Core 2 Duo Mobile of the same specifications, while other tests run faster on the Intel. Is it fair to say that the Intel is faster simply because it tops more benchmarks? After all, it all depends upon the hardware environment each processor is set in…

Endnote

Lastly, benchmarking does not make sense for regular desktop use. I am not interested in the speed score of the computer I am going to buy (or already have, but want to compare), I want to know if it gets the things I want done faster than other that compete. And in that kind of test, I have often found benchmarks to be awful liars: my laptop does the exact same sequence of Photoshop processes, video conversions, Matlab compilations and webserving faster than another computer with 3DMark, PCMark scores considerably than my own.

In other words, do not rely of benchmark scores: make your own decisions – after testing wisely.

Advertisements
  1. 11 November 2008 at 11:20 AM

    Nice information about buying new desktop computers… Thanks.

  1. No trackbacks yet.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

%d bloggers like this: