Sunday, August 18, 2013

When writing about data, don't forget the context —or your own infographics

I was checking my RSS and Twitter feeds a few minutes ago and saw a link to an article at the Poynter Institute's website. This is its headline:

Holy alarming survey, Batman! More than a quarter is a lot, isn't it? Well, it turns out that we don't know. Why not? Because we lack enough context. There isn't a comparison with other majors, as a reader has pointed out. Moreover, the headline is accurate, but incomplete. A more adequate one, in my opinion, would have been: "The percentage of journalism grads who wish they'd chosen another career has remained stable for the past decade and a half." Not that catchy, I'll give you that, but it's more truthful. Here's the evidence, a graph showcased in the article:

Not that worrying, right? In fact my guess is that the slight variation in the time-series chart is due to simple random noise. Here's a a line that may support this conjecture (page 13 in the study):
"Sample error for the 2012 undergraduate data is 2.3%. Sample error terms for earlier surveys ranged from 1.8% (2004) to 3.7% (1988)."
Right after I tweeted about all the above, I double-checked the quotes extracted from the study. Here's a paragraph included in Poynter's story:
"“That it is a significant percent cannot be doubted,” the study’s authors write. "No standard from other fields exists for this question, however, and it seems likely that some graduates would be unhappy with their career choice regardless of which one they had selected. The match between expectation and actual job prospects is unlikely to ever be perfect. One in 20 of the journalism and mass communication graduates each year indicates that she or he had selected the field without ever intending to go into it."
But here's the line that precedes those in the study, which should have been there, too:
"(...) In 2012, about one in four of the graduates responded to one of those questions by saying that they regretted their career choice. The figure is relatively unchanged from a year earlier and about at the average across the 14 years the question has been asked."
In the past few years, I've grown quite concerned about how we, journalists and designers, deal with data, science, and evidence, and about how we explain them to readers. Examples like this one lead me to think that we still have a lot to learn. Why would you highlight the fact that around 25-30% of J-grads regret choosing journalism as a career? Wouldn't it be more newsworthy to report that this number has barely changed in spite of the deep crisis news media —and journalism itself, for that matter— is going through? That's the real story, I believe.

1 comment:

  1. Lewis Carroll's circa 1886 quote on the subject: “If you want to inspire confidence, give plenty of statistics – it does not matter that they should be accurate, or even intelligible, so long as there is enough of them.”

    A few decades ago "The Journal of Irreproducible Results" published the following brilliant satire:

    (Exercise in comparative logic and mathematical statistics)

    Cucumbers will ruin you! Each eaten cucumber approaches you to death. It is surprising, as thinking people till now have not distinguished lethality of this vegetative product and even resort to its name for comparison in positive sense («as small cucumber!»). And in spite of on anything, manufacture of tinned cucumbers grows.

    All main corporal illnesses are connected to cucumbers and all in general human misfortunes.

    1. Practically all people, suffering chronic diseases, ate cucumbers. Effect is cumulative obviously.

    2. 99.9% of all people died of a cancer, at a life ate cucumbers.

    3. 100% of all soldiers ate cucumbers.

    4. 99.7 % of all persons becoming victims of automobile and aviation accidents, ate cucumbers within two weeks previous to fatal accident.

    5. 93.1% of all juvenile criminals occur from families where cucumbers consumed constantly.

    There are data that harmful action of cucumbers affects very for a long time: among people born in 1839 and eating subsequently cucumbers, death rate is equal 100%. All persons of a birth of 1869-1879 have a flabby wrinkled skin, have lost almost all teeth, practically became blind (if the illnesses caused by consumption of cucumbers, have not sent them already to the grave). The result received by known collective of scientists - physicians is even more convincing: porpoises whom compulsorily fed on 20 pounds of cucumbers per day within a month, have lost any appetite!

    Unique way to avoid harmful action of cucumbers — to change a diet. Eat, for example, soup from marsh orchids. From it as far as we know, still nobody died."