Any studies on the security of different CMSs?

Question

I like to post links to Secunia search results to demonstrate (in numbers) how insecure a certain CMS (or blogging software) is.

See What are some of Drupal's shortcomings?

But there was an interesting comment to this answer:

Eaton:

It's also important to note that Secunia only publishes vulnerability reports that are explicitly announced. I've worked with other CMS packages that tuck important security fixes in minor releases with no announcements at all. Drupal has a 15 person secteam that reviews core and all 3500 addons and officially announces the security patches, no matter how minor, as a matter of policy.

Are there any studies or articles which take this into account when comparing Content Management Systems?

Answer 1

I have a small number of articles bookmarked (like this one by my coworker), but they're almost all by people defending their CMS of choice from accusations of poor security. (My own comment in your post included!) One of the difficulties is that I don't think anyone has ironed out what constitutes a 'reasonable comparison' -- everyone gets annoyed at a bad comparison, but wanders off before anyone can determine what a level playing field is.

A couple things stand out that most "quick overviews" miss:

• The security policy of the product's dev team
• The presence of a specific person or team (depending on the project's size) responsible for security. Everyone on the project should care, obviously
• Are there documented security best practices for third-party developers
• Comparison of vulnerabilities by type and severity

Perhaps this thread would be a good place to brainstorm what WOULD constitute a good comparison study?

Update - A colleague has had the opposite frustration with Secunia: inaccurate and erroneous reports filed by third-parties against an OSS project. Secunia refuses to update or amend them, apparently. It's a useful service or announcements, but everything I hear makes me cringe at using them for comparison.

Answer 2

The other major problem with using those Secunia searches is that they include all contributed modules along with Drupal Core even when the particular announcement even though a particular security announcement might be for a module that's used by about 30 people.

In addition to vulnerabilities by type and severity, you also need to take into account "core" vs. "add-on" modules and the practice of occasionally putting multiple vulnerabilities into a single announcement (happens often).

My feeling is that some of Eaton's measures on policy are more important than specific numbers or severity of vulnerabilities.

The last good measure I would add to that list is months in the past X years where a vulnerability was publicly disclosed without any fix from the project. That's rare, but is a sign of a truly failed security process.