[lifehacker]

I am doing some tests on the performance of the ROMs available for the Dell Axim x51v. I will post the results here as they come in and will rearrange them in a clear fashion once I have all the results. All tests were done after a hard reset, on maximum processor speed, full battery at start, and with nothing installed other than PocketPCMark and SPB Benchmark (which were used for the tests).

Note that Spb benchmark is much less accurate than Anton Tomov's PocketPCMark. I suggest you follow the results of PocketPCMark. To sum the results up, Football's ROM had the best performance. The CPU performance is basically the same for all ROMs, but the memory benchmark (measures RAM speed) gave best results in Football's ROM. The File System Benchmark, which measures the read, write, seek and general file operations (create, delete, rename, etc.) speeds also had a clear winner- Football. The graphics benchmark, which measured the performance of the graphics API, was also more or less the same.

EDIT (24 May 2008): Today Football released his A06 version, which features a new build. True to his word, Football significantly increased the RAM's speed. Although all other factors remained more or less the same, Football's A05 version scored 784.84 points, while the A06 scored 805.25 points. Football's ROM is by far the best now if only system performance is taken into account.

ROMs tested:
- WM6.1 (Football)- CE OS 5.2.19209 build 19209.1.0.2- Test results, CE OS 5.2.19581 build 19581.1.1.1- Test results
- WM6.1 (Lennysh)- Lenny Edition CE OS 5.2.19209 Build 19209.1.0.2 A07- Test results, Bare Bones Edition CE OS 5.2.19209 Build 19209.1.0.2 A07- Test results
- WM6.1 (Makuu)- Pure Edition CE OS 5.2.19199 Build 19199.1.0.2 A06- Test results

[lifehacker]

Lennysh's Lenny Edition CE OS 5.2.19209 Build 19209.1.0.2 A07
CPU Dhrystone: 252.693 MIPS
- dhrystone run: 2.25 microseconds
- dhrystones per second: 443980.7
- VAX MIPS: 252.693

CPU Whetstone: 8.44 MWIPS
- time: 98. 36s

Memory: 783.88 points
- copy: 107.93
- move: 107.93
- set: 355.30
- bit operations: 139.40 mb/s
- random access: 7.27 mb/s
- allocation: 390625.00 blocks/sec

Memory Latency: 108.94 points
- memory bandwidth (avg): 503.27 mb/s
- cache bandwidth (avg): 518.13 mb/s
- memory latency (avg): 19.42 ns
- cache latency (avg): 0.02 CPU cycles

File Benchmark: 67.66 points (66.86)
- Read speed: 3.13 mb/s (3.12)
- Write speed: 2.23 mb/s (2.23)
- General operations: 139.58 operations/sec (132.6)
- Seek: 22526.04 positions/sec (22531.49)

Graphics: 50.33 points
- BitBlt: 10223.64 blits/s
- StretchBlt: 18.39 blits/s
- Ellipse: 252.23 ellipses/sec
- FillRect: 1422.29 fills/sec
- DrawIcon: 621.84 32x32 icons/sec
- LineTo: 697.49 32x32 lines/sec
- SetPoint: 4793.00 points/sec
- Text: 970.27 points/sec

The results of the file benchmark were very low so I tested them again. Those results are in the parenthesis, and are almost the same as the first results. Note that the test was done while quickmenu and other plugins were also running.

Spb Benchmark results:

[lifehacker]

Makuu's Pure Edition CE OS 5.2.19199 Build 19199.1.0.2 A06

CPU Dhrystone: 253.06 MIPS
- dhrystone run: 2.25 microseconds
- dhrystones per second: 444622.3
- VAX MIPS: 253.058

CPU Whetstone: 8.39 MWIPS
- time: 97.68s

Memory: 790.16 points
- copy: 107.95 mb/s
- move: 107.95 mb/s
- set: 335.42 mb/s
- bit operations: 139.41 mb/s
- sequential access: 12.18 mb/s
- random access: 7.28 mb/s
- allocation: 396825.41 blocks/sec

Memory Latency: 108.92 points
- memory bandwidth (avg): 503.36 mb/s
- cache bandwidth (avg): 519.71 mb/s
- memory latency (avg): 19.77 ns
- memory latency (avg): 0.06 CPU cycles
- cache latency (avg): 5.92 ns
- cache latency (avg): 0.02 CPU cycles

File Benchmark: 68.23 points
- Read speed: 3.12 mb/s
- Write speed: 2.22 mb/s
- General operations: 147.10 operations/sec
- Seek: 22655.87 positions/sec

Graphics: 50.42 points
- BitBlt: 10197.85 blits/s
- StretchBlt: 18.39 blits/s
- Ellipse: 253.30 ellipses/sec
- FillRect: 1422.02 fills/sec
- DrawIcon: 614.42 32x32 icons/sec
- LineTo: 795.03 32x32 lines/sec
- SetPoint: 4598.60 points/sec
- Text: 970.51 points/sec

Spb Benchmark results:

[lifehacker]

Football's CE OS 5.2.19209 build 19209.1.0.2

CPU Dhrystone: 252.69 MIPS
- dhrystone run: 2.25 microseconds
- dhrystones per second: 443980.7
- VAX MIPS: 252.693

CPU Whetstone: 8.44 MWIPS
- time: 98.36s

Memory: 793.57 points
- copy: 108.83 mb/s
- move: 108.89 mb/s
- set: 335.53 mb/s
- bit operations: 139.40 mb/s
- sequential access: 12.24 mb/s
- random access: 10.77 mb/s
- allocation: 384615.38 blocks/sec

Memory Latency: 108.94 points
- memory bandwidth (avg): 503.97 mb/s
- cache bandwidth (avg): 520.27 mb/s
- memory latency (avg): 19.37 ns
- memory latency (avg): 0.06 CPU cycles
- cache latency (avg): 5.92 ns
- cache latency (avg): 0.02 CPU cycles

File Benchmark: 73.99 points
- Read speed: 3.13 mb/s
- Write speed: 2.22 mb/s
- General operations: 203.52 operations/sec
- Seek: 24776.19 positions/sec

Graphics: 50.22 points
- BitBlt: 9947.54 blits/s
- StretchBlt: 18.39 blits/s
- Ellipse: 255.30 ellipses/sec
- FillRect: 1422.09 fills/sec
- DrawIcon: 620.37 32x32 icons/sec
- LineTo: 788.78 32x32 lines/sec
- SetPoint: 4691.54 points/sec
- Text: 964.80 points/sec

Spb Benchmark results:

[lifehacker]

Lennysh's Bare Bones Edition CE OS 5.2.19209 Build 19209.1.0.2 A07

CPU Dhrystone: 252.69 MIPS
- dhrystone run: 2.25 microseconds
- dhrystones per second: 443980.7
- VAX MIPS: 252.693

CPU Whetstone: 8.44 MWIPS
- time: 98.36s

Memory: 784.84 points
- copy: 108.45
- move: 108.45
- set: 335.42
- bit operations: 139.41 mb/s
- sequential access: 12.22 mb/s
- random access: 7.28 mb/s
- allocation: 390625.00 blocks/sec

Memory Latency: 108.90 points
- memory bandwidth (avg): 496.72 mb/s
- cache bandwidth (avg): 520.31 mb/s
- memory latency (avg): 19.47 ns
- memory latency (avg): 0.06 CPU cycles
- cache latency (avg): 5.92 ns
- cache latency (avg): 0.02 CPU cycles

File Benchmark: 68.05 points
- Read speed: 3.13 mb/s
- Write speed: 2.23 mb/s
- General operations: 143.74 operations/sec
- Seek: 22166.18 positions/sec

Graphics: 50.21 points
- BitBlt: 10186.35 blits/s
- StretchBlt: 18.39 blits/s
- Ellipse: 256.71 ellipses/sec
- FillRect: 1422.02 fills/sec
- DrawIcon: 621.70 32x32 icons/sec
- LineTo: 641.59 32x32 lines/sec
- SetPoint: 4716.90 points/sec
- Text: 976.75 points/sec

Spb Benchmark results:

excelent iniciative, Lifehacker!

I guess we should pay more attention on the results related to "memory access" to compare the roms - how they handle it and correlate it with the stability. I don't know how (if so) they differ in drives, for video and other hardware funcions.

It would be nice too some comparision on wi-fi, bluetooth, and irda. As well as power consumption.

That's a lot of stuff, but maybe somebody have already tested some.

By the way, have an idea what's the variance of this tests? I guess that +- 2% results can be considered the same, but it's just an opinion...

[lifehacker]

excelent iniciative, Lifehacker!

I guess we should pay more attention on the results related to "memory access" to compare the roms - how they handle it and correlate it with the stability. I don't know how (if so) they differ in drives, for video and other hardware funcions.

It would be nice too some comparision on wi-fi, bluetooth, and irda. As well as power consumption.

That's a lot of stuff, but maybe somebody have already tested some.

By the way, have an idea what's the variance of this tests? I guess that +- 2% results can be considered the same, but it's just an opinion...

I plan to do Wifi, bluetooth and power consumption afterwards. Right now Im just doing general statistics, I plan to draw conclusions in the end.

[lifehacker]

General performance tests completed and conclusions added to opening post.

interesting results...

but Football said on XDA's that is going to release a new WM6.1 rom (very) soon... He said also that this one have problems with IrDA, and that would be solved then. But I believe that it would not compromise the performance.

Wow amazing!Really cool test.In my A06 i will increase internal storage speed And i think results will be better than in A05 or other wm6.1 for dell

[lifehacker]

Wow amazing!Really cool test.In my A06 i will increase internal storage speed And i think results will be better than in A05 or other wm6.1 for dell

Waiting patiently for the release.

[lifehacker]

Football's CE OS 5.2.19581 build 19581.1.1.1 A06

Dhrystone: 252.69 MIPS
- Dhrystone run: 2.25 microseconds
- Dhrystones per second: 443980.7
- VAX MIPS: 252.693

Whetstone: 8.44 MWIPS
- time: 98.36s

Memory: 805.25 points
- copy: 108.46 mb/s
- move: 108.53 mb/s
- set: 335.42 mb/s
- bit operations: 139.41 mb/s
- sequential access: 12.22
- random access: 10.81 mb/s
- allocation: 396825.41 blocks/sec

Memory Latency: 108.92 points
- memory bandwidth (avg): 502.82 mb/s
- cache bandwidth (avg): 514.98 mb/s
- memory latency (avg): 19.68 ns
- memory latency (avg): 0.06 CPU cycles
- cache latency (avg): 5.92 CPU cycles
- cache latency (avg): 0.02 CPU cycles

File Benchmark: 72.58 points
- Read speed: 3.13 mb/s
- Write speed: 2.23 mb/s
- General operations: 188.41 operations/sec
- Seek: 26210.04 positions/sec

Graphics: 50.62 points
- BitBlt: 10395.60 blits/s
- StretchBlt: 18.38 blits/s
- Ellipse: 256.56 ellipses/sec
- FillRect: 1422.09 fills/sec
- DrawIcon: 621.29 32x32 icons/sec
- LineTo: 795.08 32x32 lines/sec
- SetPoint: 4714.91 points/sec
- Text: 973.52 points/sec

Conclusions added to opening post.

Here's mine with Football's A06:

cpu: 251

whetstone: 8.44 MWIPS

memory: 881.29

mem lat: 108.94

file sys: 54.83
read : 2.95MB/s
write: 0.75MB/s

gfx: 50.32

I don't know why my file system bench says write is 0.75 and yours 2.23, any ideas?

Also, free mem right after flashing and reset is 33.49MB, very nice!

[lifehacker]

What kind of conditions did you have during your test (hard reset, max processor, etc.)?

great overview lifehacker!

if somebody has the newest lennysh rom (A08), could they perhaps post its benchie in comparison to the latest football rom?

thanks