Code Metrics - how useful are they?

Kevin Seal wrote an interesting article on the usefulness of code metrics. To summarize, it sounds like the code metrics he mentions (like NCSS, Cyclic Complexity etc.) aren't very useful. And I agree wholeheartedly with him based on my experience using them.

As I've said in previous entries - I love the automated code checkers out there, they help confirm my intuition / suspicions about the current state of the code, but the metrics I *really* care (worry!) about are as follows and most of them can't be measured solely from scanning the source code.

1) Getting Code Working

a) Bugs we know about: Especially the High and Medium priority bugs
- Are there metrics for this that can be measured using an automated tool suite? Yes, unit tests help but only up to a point. JUnit typically isn't set-up to spot system and integration testing issues (never mind performance / usability issues).
- Many of the true Sev-1 and Sev-2 bugs cannot be spotted by automated tools since, by their inherent nature, the tools lack the "semantic" understanding of the application
- So code-based metrics don't really help us here

b) Bugs we don't know about
Here again JUnit is helpful for finding bugs that no-one has spotted (yet!) e.g. checking parameter bounds on public methods ("Hey what happens if I pass in 'null' or 0"). Similarly automated code scanning tools are useful for finding a range of "bugs-in-waiting". For example

• empty exception blocks ("Yeah just ignore that error!" It amazes me how often I still see this - even from experienced developers - I just saw it three times last week!)
• catching "exception" or worse catching "throwable" ("Sure! We can recover when the JVM bombs out!")
• switch statements without a default clause ("So you're 100% sure that the input will never be any other value?")
• missing "break" in a switch statement (those can be nasty little bugs to find)
• inconsistent synchronization (sometimes your access to a data structure is synchronized, and sometimes not)
• Use of an unconditional wait "wait()" ("You're OK waiting forever for this resource?")

Perhaps we could score each of these problems, divide by the NCSS and come to some metric of "potential issues per line of code". That's a reasonable metric.

2) Keeping Code Working (Extensibility & Maintainability)

Automated tools are helpful here too; to spot issues that affect how easily maintained / extended the code is. For example

• Class too long (large classes are hard to maintain - typically because they are trying to do too much)
• Method too long (same reasoning)
• Too many fields (bad smell, again, that a class may be trying to do too much)
• Lack of Javadoc

The architecture and design themselves determine much of how easy the application is to extend and simple things such as "good variable naming" can impact how easy or hard code maintenance is - neither one is easy to determine through automation.

And beyond this, written Word/PDF documentation is also helpful in keeping everyone on the same page, as are unit tests (to make sure it still works when you extend/fix it).

Again - existing code metrics aren't really available currently to help us here.

3) Are our users happy? (in particular - usability)

Users don't just want bugs fixed - that's a default assumption. It doesn't help if you have no bugs, implemented the spec correctly and yet NO-ONE wants to use your system (and trust me I've worked on such applications!).

Just because you have a spec doesn't mean it's correct - user feedback is crucial and that's one of the reasons I love paper or HTML prototypes as they often find problems in the "idealized" or "in-theory-it-should-work-like-this" workflow.

Performance and scalability also come under this and here automated code checkers can help spot some of the low-level issues (e.g. using String concatentation "+" vs. StringBuffer / StringBuilder etc.) but none of them are going to realize that you're trying to load 5 TB of data into memory until you actually try and do it! :-)

Summary

Code metrics, IMHO, are useful for spotting code complexity hot-spots in your source. But frankly I don't get to worry about that until my bugs are well under control, the source is maintainable (note though that addressing maintainability problems such as "class too large", often helps reduce complexity) and best of all that our users are happy.

Metrics though are a good idea - but what should we measure?

Here's a good idea for a product - it does all the automated code checking (PMD etc.), integrates with your source code control system (e.g. Clearcase, Perforce etc.), your bug tracking system, and your QA tools (e..g. Segue, Mercury etc.) and then also polls your users for feedback too on each release!

Now *THAT* would be a metric we could all care about! Everyone from management to architects to developers and users loves working code, with easy, low-cost maintenance and happy end-users!