Rails test coverage: sometimes 100% is just right

DHH, the éminence grise of the Ruby on Rails world, took a swipe at the test-first cult with his provocative article "Testing like the TSA", saying in effect that 100% test coverage is mad, bad, crazy behaviour, worthless, and an overall affront to good taste and a crime against humanity. [I paraphrase.] Since we enforce 100% code coverage at all points through our development process, I want to explain how this does not necessarily make us time-wasting, genital-fondling idiots, how the needs of our business drive our quality strategy, and how this pays off for us.

At Sage our customers demand and deserve the best we can deliver. We are very quality focused because we build accounting solutions in which getting the right answer matters a great deal: perhaps some customers don't care about quality, but ours demonstrably do. Perhaps in some cases time-to-market is much more important than reliability or maintainability: it is a business decision, and there is no one-size-fits-all answer. However, if you're building for the future and want to avoid years of functional paralysis and a costly rewrite, building an application on a solid quality foundation makes a lot of economic sense.

Write less code

The most effective way to maintain 100% test coverage is by writing less code. We refactor like crazy, and we refactor our tests just as much as our code. We don't repeat ourselves. We spend time creating the right abstractions and evolving them. Having 100% test coverage makes it much easier for us to do this: it is a virtuous cycle.

We've been doing Rails development at Sage for five years now, and we've learned a few lessons. Even if you're writing unit tests with 100% code coverage, you're doing it wrong if:

• Generators are used to build untested code (i.e. using the default Rails scaffolds to build controllers and views)
• Partials are the most sophisticated method of generating views, and they look like PHP or ASP
• The tests are harder to understand than the code

What is the alternative? Well, if all of the controllers and views look pretty much the same, factor them out. The Rails generators create enormous amounts of crappy, unmaintainable boilerplate code – every bit as as much as a Visual Studio wizard. On the other hand, if the controllers and views are each completely different and unique flowers, is it for a good reason or is the code just a mess? Chances are, if the code looks like a mess, so does the app.

In my experience it's also basically useless to attempt to retrofit unit test code coverage onto a project that doesn't have it: the tests that wind up written are always written to pass, and they rarely help much. I haven't yet seen a project that could be rescued from this situation.

Whom do you trust?

When DHH says that the use of ActiveRecord associations, validations, and scopes (basic Rails infrastructure) shouldn't be tested, he's claiming that Rails is never wrong: not now, not in the future, not ever. It's his choice to make that promise, but it would be irresponsible of us to believe it:

• Rails changes all of the time. Sometimes there are even bugs! (Crazy talk, I know!) But active record associations and scopes are complex and ornery, and can easily be broken indirectly (through a change elsewhere in the code).
• Because we operate on the Internet, new security risks and fixes appear constantly: zero day attacks are real. We need to react to these threats quickly, and being able to prepare and deploy new versions of our apps based on updated components immediately is crucial. Having a robust test suite makes it much cheaper and less stressful to implement these changes, which drives down technical debt and makes development more responsive, and oh yeah, helps prevent a costly rewrite.
• We use components that extend and complement the behaviour of Rails. DHH calls out the example of testing validations to be particularly useless. Well, what about when the validations methods change in a rails upgrade? Or you want to adopt a new plugin that changes core Rails behaviour? Or you want to refactor an application to move validation to a more useful place? In all of those cases the tests on validation code would be useful.

Often this means a function in a spec mirroring a function in a model (but with enough difference in naming and syntax to be truly maddening). Yes, this feels stupid sometimes, but it is a very cheap insurance policy, and sometimes it pays off.

Time split

DHH says that you shouldn't be spending more than 1/3 of your time writing tests. This leads to a question: how are you characterizing your time? Is the person doing the implementation also the person making design decisions? If you are doing behaviour-driven development you are actually vetting the requirements at the time you write the tests, so is it a good idea to skip that part and move on to the coding? If you spend time refactoring tests to speed up the test process, should that be counted? Should the time spent writing tests before fixing bugs be counted? Have you decided to outsource quality to a bunch of manual testers? What is your deployment model? I'm reluctant to put a cap on the time writing tests. I find this metric as useful as dictating the time spent typing vs. reading, or the amount of time thinking vs. talking: my answer is not yours, and the end result is what matters.

Risk assessment

We enforce 100% test coverage because it ensures that no important line of code goes completely untested. One can decide to write tests for "important" code and ignore the "unimportant" code, but unfortunately a line of code only becomes "important" after it has failed and caused a major outage and data loss. Oops!

DHH avers that the criticality and likelihood of a mistake should be considered before deciding to write a test about something. However, this ignores the third criteria: cost. Is it cheaper to spend time deciding the criticality and likelihood of writing vs ignoring tests for every single line of code, or is cheaper to just write the stupid test and be done with it? Given the cost of doing a detailed long-term risk analysis on every line of code, does anybody ever really do it, or is the entire argument just an elaborate cop-out? The answer gets a lot clearer once you elect to write a lot less code, and it gets easier once you resign yourself to learning a new skill and changing your behaviour.

Closing

Code coverage is a great way to measure the amount of exposure you have to future changes, and depending on your business, it might be necessary to have 100% coverage. A highly respected figure speaking ex cathedra can be very wrong when it comes to the choices you need to make, and sometimes it shows. 100% code coverage may seem like an impossible goal, especially if you've never seen it done. I'm here to tell you it's not impossible: it's how we work, and in our case it makes a lot of sense.

Sympathy for the trolls: do everyone a favour and walk away from fights online

Everybody's been a troll at one point or another. Sometimes we know when we're trolling, but mostly we're just having a bad day (or week, month, year) and we take it out on someone else.

Sometimes we take it out on customer service representatives on the phone. That's the trolling I'm particularly guilty of, and the one I'm most ashamed of: for some reason I always find myself railing at the phone company. (Which phone company? Any of them.  All of them.)  Thankfully, although those conversations are recorded, they aren't (yet) transcribed and posted publicly.  (Now that would make people behave better on the phone.)

But mostly, it's a case of Duty Calls (when Someone is Wrong on the Internet) and we allow a disagreement to escalate.  I had one of these happen to me today, and I (for once) didn't make it worse.  I was proud of myself because I responded with grace and humour, attempted to defuse the situation, and didn't respond when it got truly nasty.

Instead of mixing it up and making things worse, I went for a walk with my husband and dog, where we saw flames peeking out of a big paper recycling bin.  My husband got someone to call the fire department while I ran to find a fire extinguisher, and then another when the police had used that one up and the fire had reignited.  Very exciting, but at least the fire didn't get out of control for very long before the firefighters arrived:

Later I was especially pleased that I didn't bite because I looked at the troll's Twitter account and saw that he was genuinely upset, having left the forum because of poor quality conversations (apparently a pattern he is experiencing). A woman I know once cried in frustration, "why is it that wherever I work, there's always some bitch who makes life miserable for me?"  Indeed.

Nothing I said could have made it better: he needed me to be the villain.  Okay then.  But that doesn't mean I had to feed the fire; I didn't need to correct him, I didn't need to have the last word, and I didn't need to humiliate him.  He took care of it himself.  And it turns out I really did have better things to do.

In closing: not the most mature choice of videos, but hey, gotta be me.

"Fire! Fire! Come in through the back door...
Fire! Fire! I want to be a fireman... and handle your hose."

Rails i18n translations in Yaml: translation tool support

With Rails 2.2 the i18n API was introduced with a new method for translations.  Instead of embracing the venerable gettext which had been the previous standard, the Rails team invented a new way using Yaml files.  The result is a particularly graceful, flexible and very Rubylike way of specifying translations.  It also is much more reliable than gettext, which had many inscrutable issues with locales and caching, and sometimes caused people to get things in the wrong language.  So: bravo, great job.

But to do this, they specified their own translation format, the very flexible Yaml file. There are already a lot of formats floating around, and translation tool vendors and open-source translation developers have been working for a long time on conversion tools between them.  The Translate Toolkit and Pootle emerged from South Africa (a country which groans beneath the weight revels in the glory of eleven official languages) which provide an excellent web-based tool for collaboration, centered around gettext PO files.  However, poor little Pootle started a migration from Python to Django, and we all know how rewrites go.  [Halfway. Badly.]  But Translate Toolkit supported a lot of formats:

• moz2po - Mozilla .properties and .dtd converter. Works with Firefox and Thunderbird
• oo2po - OpenOffice.org SDF converter (See also oo2xliff).
• odf2xliff - Convert OpenDocument (ODF) documents to XLIFF and vice-versa.
• prop2po - Java property file (.properties) converter
• php2po - PHP localisable string arrays converter.
• sub2po - Converter for various subtitle files
• txt2po - Plain text to PO converter
• po2wordfast - Wordfast Translation Memory converter
• po2tmx - TMX (Translation Memory Exchange) converter
• pot2po - initialise PO Template files for translation
• csv2po - Comma Separated Value (CSV) converter. Useful for doing translations using a spreadsheet.
• csv2tbx - Create TBX (TermBase eXchange) files from Comma Separated Value (CSV) files
• html2po - HTML converter
• ical2po - iCalendar file converter
• ini2po - Windows INI file converter
• json2po - JSON file converter
• web2py2po - web2py translation to PO converter
• rc2po - Windows Resource .rc (C++ Resource Compiler) converter
• symb2po - Symbian-style translation to PO converter
• tiki2po - TikiWiki language.php converter
• ts2po - Qt Linguist .ts converter
• xliff2po - XLIFF (XML Localisation Interchange File Format) converter

In its heels, Google introduced the Google Translate Toolkit, which lets you use the Google Translate engine to suggest translations (based on its own databases or translation memories you provide).  It also does the core of what Pootle does: collaboration, access, but without crashing and flakiness, and it works with:

• AdWords Editor Archive (.aea)
• Android Resource (.xml)
• Application Resource Bundle (.arb)
• Chrome Extension (.json)
• GNU gettext (.po)
• HTML (.html)
• Microsoft Word (.doc)
• OpenDocument Text (.odt)
• Plain Text (.txt)
• Rich Text (.rtf)
• SubRip (.srt)
• SubViewer (.sub)

But neither of them supports Yaml files.  Unfortunately, tooling support libraries have not embraced this format in the intervening two and a half years.  I did find one solution: i18n-translators-tools which supports conversion between Yaml and gettext PO files, but it's somewhat idiosyncratic, and it turns out there's a good reason why there isn't a straightforward Yaml ←→ PO converter: the PO format is consists of name-value pairs with metadata, and the Yaml format is a tree.

English source Yaml file	Spanish Yaml file produced by i18n-translators-tools from a PO file
page_info:   sales/credit_notes:     date: "Date"     title:       default: "Sales Credit Note"       new: "New Sales Credit Note"	page_info:   sales/credit_notes:     date: "Fecha"     title:       default:         default: "Sales Credit Note"         translation: "Crédito de venta"       new:         default: "New Sales Credit Note"         translation: "New Sales Credit Note"

There are some interesting things going on here: the Spanish Yaml file provides fallbacks so untranslated strings don't come through as blank.  The intermediate gettext PO file keeps the tree structure in the msgctxt metadata, and looks like this:

msgctxt "page_info.fuji_sales/sales_credit_notes.title.default"
msgid "Sales Credit Note"
msgstr "Crédito de venta"

msgctxt "page_info.fuji_sales/sales_credit_notes.title.new"
msgid "New Sales Credit Note"
msgstr "New Sales Credit Note"

So it's possible to use Google Translate Toolkit to translate your Rails Yaml files, provided you use the i18n-translators-tools library to do the conversions, and configure your Rails applications to support fallbacks.

Job satisfaction: the passionate dermatologist, the chair, and the metal hook

As a teenager I had terrible skin.  It was just average-bad until I was nineteen, at which point it went absolutely Vesuvian, requiring Accutane to tame it.  Each pill came in a nearly impregnable (hah) blister pack which required multiple steps to open: first, slide open the box and see the outline of a hydrocephalic fetus; second, remove a serrated paper tab with the silhouette of a pregnant woman and the red ban-symbol through it, and finally pop the liquid-filled gel pill through a rather tough plastic laminated foil surface.  (The packaging shown at right is nowhere near as extreme.) I would later give the paper tabs with the "no pregnancy" symbol to friends and co-workers to encourage contraception.  But I digress.

Rewind slightly: at sixteen I was taken to a dermatologist.  I went with my mother, who always tried very hard to get me to take care of my skin.  I can't count the number of doctors who looked at my face, looked again, mumbled uncomfortably, then handed me a bar of Purpose soap and gently indicated that I might try using it.  I didn't, of course - the only thing I was willing to do was take the erythromycin that would stain my teeth.  I went through a couple of dermatologists over the years.  But I digress.

So, the dermatologist: somewhere in the northern woods of Fulton County, a rather long drive from my parents' house in Marietta.  Picture me hurtling across the suburbs in a gray 1979 Ford Mustang: one time I recall taking a corner through a freshly red light, tires squealing as I accelerated through second gear, laying on the horn to make sure the other people didn't act too soon on their green light and get in my way.  But I digress: obviously I'm stalling.

The dermatologist was a woman in her mid-to-late thirties, average height, and sensible black hair.  Her skin had obviously been ravaged by acne.  She did the usual first visit: yes, here's your bar of Purpose soap, your prescription for Retin-A, your bottle of pills, whatever.  Then she put me in a reclining dentists' chair, fixed the spotlight on my face, and proceeded to squeeze out each and every one of my blackheads through use of a metal implement.  It hurt like hell: she pressed with that damn thing really hard, all across my nose, forehead and cheekbones.  Push down, scrape across, wipe on a tissue, repeat.  No small talk, no lectures, just intense concentration.

This took at least forty-five minutes, and at least a half hour on each of my subsequent visits.  Each time my face would eventually stop producing the goods and she'd reluctantly let me go.  This was back in the days of full insurance, no copays, and no referrals, when you could go to a specialist all you wanted, for anything you felt like doing.  I have no idea what this woman charged, and it probably wasn't cheap, but she was doing the whole thing herself: no receptionist, no assistants.  Just her versus the zits.

The last time I went she came out to the waiting room to get me.  I stood up and said hello to her, and she never looked me in the eyes: she mumbled hello as she started scanning my face.  I sat through that last agonizing session as she pushed, scraped, and wiped my throbbing face.  Yes, I stopped going because it was a long way from home, because it was painful, and because it didn't stop the pimples that actually bothered me, but mostly I stopped going because that woman creeped me out.

She obviously loved her job, but she loved it way too much: she was a sebum junky, a zit juice vampire, a woman on a mission of vengeance against the acne that had obviously scarred her for life.  To this day, when people talk about being "passionate" about their job I think of her.