Kevin's Home

Browser Philters

Because every other test done has suggested IE7's phishing filter worked really quite well and some people have suggested that Firefox 2.0's system was a tad flawed that wholely independant bunch at the Mozilla Foundation commisioned a study to compare the two systems. Obviously to avoid people making comments about how independant they might be the Mozilla folks helpfully explain their efforts to avoid a testing bias:

Testers worked in teams of two, and would rotate testing from one browser to the next. Testers had to report results on all four browser modes before a test record was considered complete. Once this occurred, the completed test record dropped off the list and a new test record was added. Limiting the available test records to seven at a time ensured that all four modes were tested per URL in as short of a time window as possible.

Since time favors the second browser tested (it gives the phishing features more time to update their lists), the testing order between Firefox 2 and IE 7 was rotated to ensure that no one browser had a testing advantage over another. It should be noted that Firefox was tested first more times than IE 7 to discourage unfair advantages for Firefox.

Yes that second paragraph does read correctly. They actually say how careful they were to ensure that no setup was at an advantage, followed in the very next sentence with a claim that they basically crippled Firefox and put it at a massive disadvantage. So completely balanced, but we gave the other guy an advantage. Make up your minds.

Actually in the audit the whole thing is wordly a tad more sanely:

Browser testing order was rotated and randomized; however, it was not balanced. Firefox 2 was tested first more than half the time when compared to Internet Explorer 7, giving a slight advantage to Internet Explorer 7. Browser rotation is important as the last browser tested against a given phishing URL has more time to update its phishing database, allowing it to potentially show a positive result. The testing process should be more evenly distributed between browsers; however, since the testing window between browsers was 15 minutes of less, the lack even distribution probably did not have a significant affect on the results.

So the audit basically says that the data gathering was flawed and not balanced as it should be, however this imbalance wouldn't have made any real difference. Oh but wait, the Mozilla team say that IE7 had the non-existent advantage; and don't you lot forget it!

In short, much like most studies into how much better than your product our product is, the whole thing is a bit of a waste. My vote goes to the first browser to give me a Gumpf Filter which cuts out pointless studies with poor write ups.

Secret Email Addresses use only in an emergency!