Category: Metamedia

Metamedia: writing about journalism and the media generally

Good metrics vs bad measurement

My former colleague Chris Moran has lots of sensible things to say about what makes a good metric, as do the many people he’s enlisted in that linked post to talk about the characteristics they value in measurement. I wanted to build on one of them: the capacity for people to actually use the metric to change something.

Functionally, plenty of things aren’t easy to measure. Some are almost impossible — much as Chris says, I have lost count of the number of blisteringly smart people working out how to measure things like quality or impact when it comes to journalism. Anything that involves qualitative surveys is probably too high cost for a small project. Anything that requires you to implement completely new analytics software is unlikely to be valuable unless it’s genuinely transformative (and even then, you risk the business equivalent of redesigning your revision timetable rather than actually revising). Anything that relies on people giving an unbiased assessment of the value of their work — like asking editors to assign an “importance” score to a story, say, or Google’s now-defunct “standout” meta tag — is doomed to failure, because an individual can’t accurately assess the relative nature of their work in the context of the whole system. Key point from Chris’s post: if you were going to game your measure, how would you game it? Do you trust everyone involved to act purely in the interests of good data, even when that gets in the way of their own self-interest?

In one team I managed, I once ran an OKR that focused on making sure we were known and appropriately involved as internal experts by the rest of the organisation. We discussed how to measure that, and ended up deciding that we’d know if we were succeeding based on the number of surprises that happened to us in a week. We were accustomed to finding out about projects too late for us to really be helpful — and, to a lesser extent, we were finding that our work sometimes surprised other people who’d benefit from being involved earlier on.

How do you measure surprises? We could have spent weeks working that one out. But for the sake of just getting on with it, we built a Google form with three inputs: what’s the date, who was surprised, who did the surprising. Team leads took the responsibility of filling in the form when it happened. That’s all you really need in order to know roughly what’s going on, and in order to track the trajectory of a metric like that. But because we measured it — really, honestly, mostly because we talked about it every week as a measure of whether we were doing our best work, and that led to thinking about how we could change it, which led to action— it improved.

Conversely, if you don’t care about something you measure, it’s almost certainly not going to change at all. If you spend enormous organisational energy and effort agreeing and creating a single unified metric for loyalty, say, but then you don’t mention it in any meetings or use it to define success or make any decisions about your products or your output… why bother measuring it at all? Data in isolation is just noise. What matters is what you use it for.

So if you’re going to actually make decisions about quality, or impact, or loyalty, or surprises, the key isn’t to spend ages defining a perfect metric. It’s getting 80% of the way there with as little effort as you can pull off, and then doing the work. It means working out what information you (or your teams, or your editors, or your leaders) don’t have right now that they need in order to make those decisions. Then finding a reasonable, rational, most-of-the-way-there metric you can use that unblocks those decisions. Eventually you might find you need a better measure because the granularity or the complexity of the decisions has changed. But you might equally find that you don’t really need anything other than your first sketch, because the real value is in the conversations it prompts and the change in the output that happens as a result. Precision tends to be what data scientists naturally want to prioritise, but it’s usually missing the point.

Crossposted to Medium because there’s traffic and discussion over there.

13 things I learned from six years at the Guardian

… in which I went from SEO subeditor to executive editor for audience, via Sydney and New York.

This post is cross-posted from Medium for archival purposes.

I started at the Guardian in 2011 as an SEO subeditor, working out how to bring the Guardian’s journalism to the widest possible relevant audience; in 2013 I moved to Australia to launch the local edition, taking on a much broader audience development role. After nearly two years there, having built one of the most widely read news sites in the country, I moved to New York to do the same thing but with more resources (and a lot more news). Towards the end of 2015 I came home to take on the global challenge and bring a holistic approach to audience development to the broader Guardian. Now I’ve made the difficult decision to move on, and I’m leaving behind a brilliant set of people well equipped to take on the challenges of the future.

I’ve learned an enormous amount during my time and my travels, and I hope I’ve taught some people some useful things too. Here are 13 of the most important things I know now that I didn’t know six years ago.

1. Data isn’t magic, it’s what you do with it that counts.

There’s a tendency for news organisations (and a lot of other organisations) to get very excited and very suspicious around numbers. People who understand how linear regression works are clearly dangerous wizards, and getting involved with data at all used to be seen as something dirty?—?something that could taint you. This is patently daft, because numbers don’t remove people’s brains or their editorial sensibilities. We make better decisions when we’re better informed, and all data is is information.
The flip side of that is that data isn’t sufficient to make improvements in how we work or what we do. The only thing that matters is the decisions we take in response to the numbers. I’ve been lucky to be involved in the development of Ophan, the Guardian’s in-house live stats tool, and the most common misconception about it is that it’s just a data display. It’s never been that: it’s a cultural change tool. It’s not just about putting numbers into the hands of editorial people?—?it’s explicitly about getting them to change the way they make decisions, and to make them better. It’s a tool for enhancing journalistic instinct, and one of the reasons why we can be so cavalier about demonstrating it everywhere is that the commercial advantage it brings is not written on the screen. The advantage is in how we use it, and that’s a years-long project no other organisation will be able to imitate.

2. People are more important than stories.

You’d think this wasn’t controversial, but it is. Journalists have a tendency to work ourselves into the ground, to ignore our own needs and push ourselves incredibly hard to get stories. That’s part and parcel of the job, a lot of the time.
But if you’re a manager or an editor (or, more likely, both), you have to watch out for that tendency in others and in yourself. Good people who go above and beyond what’s asked of them for a story are worth protecting and supporting, and they are probably going to need some time to recover after massive events that take a lot out of them. They need to be able to take time out without feeling on edge about a story breaking that they might miss. Nothing is served by letting the best people burn out. Nothing is served by burning out yourself.

3. Management is a technology.

Management style is built, not intuited; it is actively and deliberately created, not naturally occurring. It is a technology, something that can be improved to make organisations more efficient or better, and that can be implemented in many different ways.

Making all managers within an organisation work out what management ought to be like for themselves is about as efficient as making every journalist design their own CMS. News organisations?—?especially on the editorial side?—?tend to have a healthy scepticism about management-speak and corporate bullshit, but that can’t be allowed to stand in the way of solid leadership approaches that can be universally understood and adopted.

4. Change is for everyone.

The news business has changed immeasurably in just the last decade, since I started. For those who started as journalists before the internet took hold, it can be almost unrecognisable. Change is constant, and innovation never ceases; there is a dramatic urgency about most news organisations’ efforts to change, and those on the cutting edge are often incredibly impatient for others to get on with it.

But if you find yourself thinking about how much everything needs to change, stop for a moment and look inwards at yourself. Chances are that you’re right?—?that everything does need to change, and that the folks around you are changing more slowly than you are. But that doesn’t mean that you don’t need to do your own work. You can’t always hurry things along, but often you can model the impact of those changes in your own way. Whether that’s altering your own newsgathering practices, implementing different techniques in your own team, or going out and getting the skills you think you might need tomorrow?—?you can probably make a bigger difference than you realise by working on yourself, not just the people around you.

5. Attention is the only thing that’s scarce on the internet.

You can get more of everything online except human attention. If you’re lucky enough to work in a business that aims to attract people’s attention for positive reasons?—?and good enough at what you do to succeed at it?—?then treat it with respect. The most important commodity most people have to spend online is their attention. If you want to gain their trust, don’t screw about with it.

6. Pivoting to video is not a strategy.

Video isn’t a strategy. “More video” isn’t a strategy. “More video with more video ads on it”: also not a strategy. What kinds of stories are you going to tell? Do people actually want those stories in that format? How are you going to reach people, how are you differentiating your work from all the other things on the internet, and why should anyone trust you in a market so crowded with terrible, useless video right now? Stop pivoting, start planning.

7. Platforms are not strategies, and they won’t save news.

Seriously. If someone else’s algorithm change could kill your traffic and/or your business model, then you’re already dead. Google and Facebook are never going to subsidise news providers directly, and nor should they. Stop waiting for someone to make it go back to the way it was before. If what you do is essential to your audience, so essential that their lives wouldn’t be the same without it, then you should be able to monetise that. If it’s not, your first priority should be to admit that and then get on with changing it.

8. Quality journalism can be a strategy.

Making good stuff that people want to read?—?or watch?—?is a valid strategy, if it also includes monetising that attention effectively. So is choosing which platforms to focus on based on where your intended audience is and what you can do with them there. Good journalism?—?especially good reportage?—?gives people something important for which there is no substitute. (So does good entertainment, of course.) Many people value it enormously and, if you’re known for providing it, they’ll come to expect it and trust you more as a result. There’s no law that says people will only read celebrity news or stuff you’ve nicked off the front page of Reddit.

The vast majority of the Guardian’s most read pieces of all time are high quality journalism on serious topics. Many of them are live blogs of breaking news. I remember very fondly launching a 7,000-word piece by the former prime minister of Australia at 10am on a Saturday, when the internet is basically empty, and watching it smash our local traffic records. I remember the day when a piece about the death of capitalism went viral. Not every big hit is a long read or a deeply serious bit of journalism, of course, but if you write for the audience you want, and you respect people’s attention and intelligence, you might be pleasantly surprised by the long term results.

9. The internet is made of humans.

You can’t predict the future, nor understand what scientific innovations might become dramatically important in the coming decades. You can maybe make some educated guesses about the next 18 months, but even that could be thrown out of the window by a major news event or a Zuckerbergian whim.

You can, however, understand a great deal about human motivations and behaviour, and filter your approach to new technologies based on what you know about people. A great deal of the work involved in predicting the future is really just understanding people and systems, and especially systems made up of people.

10. It’s often better to improve a system than develop one brilliant thing.

Making systems better is not particularly sexy work. It tends to be incremental, slow and messy, taking knotty problems and carefully unknitting them. In the time it takes to make a widely-used system very slightly better, you could probably make half a dozen gorgeous one-off pieces of journalism that the world would love.

But if you make the system better, you potentially make lots of people’s jobs easier, or you save dozens of person-hours in a month, or you make hundreds of pieces of journalism work slightly more effectively. It’s not flashy, and probably most people won’t even be aware of what you’ve done. Most organisations need people doing both, because without the brilliant beautiful one-off pieces, how would you know what the system needs to be able to do in the long run? But people who do the flashy things are plentiful, and people willing and able to graft on the stuff that just incrementally makes things better are in sadly short supply.

11. Radical transparency helps people work with complexity.

In a fast-moving environment where everything is constantly changing (eg: the internet, the news, and/or social media) you have no way of knowing what someone else might need to know in order to do their job well. The only way to deal with this is to be a conduit for information, and not bottle anything up or hide it unless it’s genuinely confidential. I can’t possibly know what information I come across might turn out to be helpful in a few months’ time, and I definitely don’t have the knowledge to do that for anyone else. People often need different data in order to get context for what they’re trying to achieve, and if you’re trying to communicate a specific message or a particular approach, you’re going to need to keep saying it over and over again. It’s basically impossible to communicate too much.

12. Most obvious dichotomies are false.

SEO isn’t dead; social isn’t pointless. Loyalty and reach both matter. Lifestyle journalism can exist alongside serious pieces. In fact, in both cases, the two apparent sides of the argument are interrelated in hugely positive ways, and elements of both will support the other. While we always need to be careful about what we prioritise and where we spend resources, it’s always worth thinking about the systemic ways that behaviours can reinforce each other and finding opportunities to efficiently do more than one thing.

13. What you say matters far less than what you do.

This should be obvious, but it probably isn’t. It doesn’t matter what you say you want, it’s what you do to make it happen that makes a difference in the world. You have so much power right now. It’s up to you to do something meaningful with it.

2017: it’s complex

It used to be possible to understand the inputs into the publishing business, and make plans based on what might happen. Don’t get me wrong – it’s always been very hard to see all the various levers that had to be pulled, to turn a story from a series of events and conversations into a piece of journalism and then get it in front of people. But it was possible, within the realms of imagination, that senior folks in news organisations could do it. Distribution and sales were pretty well-understood systems; news production likewise. Even the creation of news was, relatively speaking, straightforward: journalists could identify the likely places news would erupt from, and focus their efforts on cultivating sources in those areas. News generally came from journalists doing journalism, for one thing.

That understanding has been dying for a very long time now, but 2016 has thoroughly hammered the nails into its coffin. The ad business has imploded. Major platforms have launched surprise initiatives with massive impact on the news business – some in the form of products, others in the form of algorithm changes. Politics has gotten weird. Distribution is, more than ever, its own massively complex system where success is as much a function of luck and preparedness than effort. (It’s vital to optimise & commissioning with an eye to distribution helps, of course, but the runaway wins are often down to a complex mix of factors of which only a few are within a news organisation’s control.)

Next year, the media industry is going to have to embrace the idea that our work isn’t complicated, but that it is complex. Journalists will need to get comfortable with the concept that there’s no complete set of skills that will enable us to tell every story. Editors need to be OK with understanding that the range of ways to tell each story is huge, and that there is no digital decoder ring that can tell us how to make the best decisions for each one. Media execs will – more than ever before – need to understand that we can’t possibly know all of the details that would enable them to make the best decisions, and find ways to devolve power to the people who can. Information needs to flow freely within – and between – organisations, because that’s the only thing that can help us work with the complexity of the systems of which we’re now an inextricable part. Data will be crucial to our understanding, but it won’t lead us to the concrete, correct answers. Past performance is not indicative of future returns.

Change will not stop. There won’t be a moment where we all get to catch our breath and reflect, unless we create one for ourselves. On the one hand, as facts become harder to defend, it will be more crucial than ever that our reporting is clear, truthful, honest and consistent. On the other hand, our businesses will have to get used to nuance, to multiple options, and to the idea there might be more than one right answer. Both our journalism and our industry must reach a place where they can respond immediately to opportunities as they come up, and where they can take advantage of the butterfly effects that complex systems generate, without being ruined by them.

The only safe prediction for 2017 is that it won’t be normal.

Reddit meltdown: how not to build a community

Reddit is having a bit of a meltdown. Volunteer moderators have taken many of the site’s most popular and trafficked communities to private, making them impossible to read or participate in. Many others are staying open based on their purpose (to inform or to educate) but making clear statements that they support the issues raised.

The shutdown was triggered in protest at the sudden dismissal of Victoria Taylor, Reddit’s director of communications, who coordinated the site’s Ask Me Anything feature. But it’s more than that: the reason communities beyond r/IAmA are going dark is about longstanding issues with the treatment of moderators, communication problems and moderation tools, according to many prominent subreddit mods.

Really good community management matters. Communication matters. Being heard matters enormously to users, and the more work an individual is doing for the site, the more it matters to them personally.

Relying solely on volunteer moderators and community self-organisation limits what’s possible, because without the company’s support – both negative, in terms of banning and sanctioning, and positive in terms of tools, recognition and organisation – its users can’t effect significant change. What’s possible with buy-in from Reddit staff is far more interesting than what’s possible without – the AMAs Victoria supported are the prime example. It should be concerning for Reddit that there are so few others.

Communities grow and evolve through positive reinforcement, not just punishment when they contravene the rules. If the only time they get attention is when they push the boundaries, users will likely continue to push boundaries rather than creating constructively. They act out. Encouraging positive behaviour is vitally important if you want to shape a community around certain positive activities – say, asking questions – rather than focussing on its negatives.

That encouragement extends to offering the community leaders the tools they need to lead. The majority of moderators of Reddit’s default communities – the most popular ones on the site – use third-party tools because the site’s own architecture makes their work impossible. That should not be

And evolving communities need consistent procedures and policies, and those have to be implemented by someone with power as well as the trust and respect of the community. Power is relatively easy; any Reddit admin or employee has power, in the eyes of the community. Trust and respect is incredibly difficult. It has to be earned, piece by piece, often from individuals disinclined to trust or respect because of the power differential. That work doesn’t scale easily and can’t be mechanised; it’s about relationships.

Today’s meltdown isn’t just about u/chooter, though what’s happened to her is clearly the catalyst. It’s about the fact that she’s (rightly or wrongly) perceived to be the only Reddit admin to have both power and trust. She was seen as the sole company representative who listened, who worked with the community rather than above or around them. She was well-known and, crucially, well-liked.

Reddit needs more Victorias on its staff, not fewer. It needs more admins who are personally known within the community, more people who respond to messages and get involved on an individual level with the mods it relies on to do the hard work of maintaining its communities. It needs internal procedures to pass community issues up the chain and get work done for its super users and those who enable its communities to exist. It needs more positive reinforcement from those in power, especially in the light of increasing (and, I’d say, much-needed) negative reinforcement for certain behaviours; the community needs to see what ‘good’ looks like as well as ‘bad’. Not just spotlighting subreddits and blog posts about gift exchanges – actual, human engagement with the humans using the site.

Firing the figurehead for Reddit-done-right is not a good way to start.

Buzzfeed’s news numbers

Digiday has a piece today announcing that 17% of Buzzfeed’s web traffic goes to news. Here’s the key part:

Of BuzzFeed’s 76.7 million multiplatform unique visitors in April (comScore), 17 percent were coming for news. The publisher historically hasn’t broken out its content by vertical to comScore, like other top news sites including CNN, Yahoo and The Huffington Post do. But it started to on a limited basis as of last month, when it began breaking out its Entertainment and Life coverage (43.7 million and 20.1 million uniques, respectively) to comScore. Stripping out those verticals leaves 13 million uniques for the rest, including hard news.

Ignoring analysis for the moment, let’s just look at the reasoning here. Unique browsers can visit more than one section of a site, so it’s possible that there’s overlap, and that simply subtracting the known verticals from the known total traffic isn’t a useful way to start. (I’m not certain of comScore’s methodology for vertical breakouts, but would be surprised if it doesn’t let sites count a user twice in different verticals, given that audiences overlap.)

So that 17% could be higher than it appears at first. Then, that 17% includes hard news plus everything that doesn’t fit into Entertainment or Life, so the actual audience for Buzzfeed’s news could be smaller than it appears at first.

The other element here is that unique browsers are the broadest possible metric, and likely to show news in the brightest light. In March, again according to comScore, Buzzfeed averaged 4.9 visits per visitor and 2 views per visit in the US, for roughly 10 views per visitor. If, among the 17% who visited news at all, five of those views are to news, then news is very, very well read with an exceptionally loyal audience. If just one of those views is to news, then news is much less well read than “17% of traffic” might suggest.

This post has been brought to you by early morning web analytics pedantry.

Update: Digiday has now altered the headline of its post and the text of the paragraph posted above.

Reach and impact: news metrics you can use

I’ve been thinking about this Tow study for a while now. It looks at how stats are used in the New York Times, Gawker and Chartbeat; in the case of the latter, it examines how the company builds its real-time product, and for the former, how that product feeds in (or fails to feed in) to a newsroom culture around analytics. There’s lots to mull over if part of your work, like mine, includes communication and cultural work about numbers. The most interesting parts are all about feelings.

Petre says:

Metrics inspire a range of strong feelings in journalists, such as excitement, anxiety, self-doubt, triumph, competition, and demoralization. When devising internal policies for the use of metrics, newsroom managers should consider the potential effects of traffic data not only on editorial content, but also on editorial workers.

Which seems obvious, but probably isn’t. Any good manager has to think about the effects of making some performance data – the quantifiable stuff – public and easily accessible, on a minute-by-minute basis. The fears about numbers in newsrooms aren’t all about the data coming to affect decisions – the “race to the bottom”-style rhetoric that used to be very common as a knee-jerk reaction against audience data. Some of the fears are about how analytics will be used, and whether it will drive editorial decision-making in unhelpful ways, for sure, but the majority of newsroom fear these days seems to be far simpler and more personal. If I write an incredible, important story and only a few people read it, is it worth writing?

Even Buzzfeed, who seem to have their metrics and their purpose mostly aligned, seem to be having issues with this. Dao Nguyen has spoken publicly about their need to measure impact in terms of real-life effects, and how big news is by those standards. The idea of quantifying the usefulness of a piece of news, or its capacity to engender real change, is seductive but tricky: how do you build a scale between leaving one person better informed about the world to, say, changing the world’s approach to international surveillance? How do you measure a person losing their job, a change in the voting intention of a small proportion of readers, a decision to make an arrest? For all functional purposes it’s impossible.

But it matters. For the longest time, journalism has been measured by its impact as much as its ability to sell papers. Journalists have measured themselves by the waves they make within the communities upon which they report. So qualitative statements are important to take into account alongside quantitative measurements.

The numbers we report are expressions of value systems. Petre’s report warns newsrooms against unquestioningly accepting the values of an analytics company when picking a vendor – the affordances of a dashboard like Chartbeat can have a huge impact on the emotional attitude towards the metrics used. Something as simple as how many users a tool can have affects how something is perceived. Something as complex as which numbers are reported to whom and how has a similarly complex effect on culture. Fearing the numbers isn’t the answer; understanding that journalists are humans and react in human ways to metrics and measurement can help a great deal. Making the numbers actionable – giving everyone ways to affect things, and helping them understand how they can use them – helps even more.

Part of the solution – there are only partial solutions – to the problem of reach vs impact is to consider the two together, but to look at the types of audiences each piece of work is reading. If only 500 people read a review of a small art show, but 400 of those either have visited the show or are using that review to make decisions about what to see, that piece of work is absolutely valuable to its audience. If a story about violent California surfing subcultures reaches 20,000 people, mostly young people on the west coast of the US, then it is reaching an audience who are more likely to have a personal stake in its content than, say, older men in Austria might.

Shortly after I arrived in New York I was on a panel which discussed the problems of reach as a metric. One person claimed cheerfully that reach was a vanity metric, to some agreement. A few minutes later we were discussing how important it was to reach snake people (sorry, millennials) – and to measure that reach.

Reach is only a vanity metric if you fail to segment it. Thinking about which audiences need your work and measuring whether it’s reaching them – that’s useful. And much less frightening for journalists, too.

UsVsTh3m and Ampp3d are being cut back…

… and that’s a sad thing. Per Buzzfeed:

“UsVsTh3m and Ampp3d are more niche areas and have very high costs for the volume of traffic. It is therefore more sustainable for us to invest resource in an integrated digital team, focusing on the main areas of the Mirror site which have more mass-market appeal.

“We remain committed to digital and will continue to invest and innovate in this area, including with new roles both now and in the future. The sites will remain for the time being.”

Paul Bradshaw has an excellent overview of the legacy of those sites for British media – they have had a huge and surprising influence on the way the Mirror tackles the internet, for sure, but also on other legacy media in the UK, spawning copycats on both content and workflow terms – to which Buzzfeed’s Tom Philips has added an insightful comment:

“Quite where it went wrong is a matter for another day, but my guess would be that Trinity Mirror didn’t know what they had – they seemed to limit funding for it at exactly the point they should have aggressively expanded. TM’s current explanation (that they weren’t delivering the traffic given their supposedly high cost) may be true right now, I’ve no idea, but it certainly wasn’t back in late 2013 when a tiny team was delivering 10 million unique users a month. If the Mirror weren’t able to make something sustainable out of that, then I don’t think the blame can lie with the talented staff who produced it.”

And Adam Tinworth points out that the world doesn’t necessarily need more Mail clones, which seems to be the way the Mirror’s going, and notes that there are a lot of superb digital journalists about to leave the Mirror.

Both UsVsTh3m and Ampp3d did pioneering work – often silly, often irreverent, often light and bright, but always and unfailingly digital. Ampp3d created the single best interactive I have ever seen for mobile. UsVsTh3m had the strongest integration of games into journalism that I’ve seen, using them for satire and for commentary rather than for news delivery. Both hired fantastic new digital journalists and let them talk in their own voices, to the communities they came from; the result was probably the most successful legacy media project to build youth voices and young readership that the UK has ever seen.

Those voices were important, and they changed a great deal. They will be very much missed.

Ophan, the Guardian’s live stats tool

Digital audience editor Chris Moran, my former boss at Guardian UK and an all round top bloke, has explained Ophan to journalism.co.uk, and if you’re interested in knowing what I do or understanding how I do it, it’s an excellent primer on how we’re building analytics into the newsroom:

“We know everything about print, pretty much, there’s not many tricks left in the bag, we’ve done it for 200 years and we’re used to it. But the internet’s changing all the time, as much as anything else.”

An idea central to Ophan, said Moran, was for it to be useful to everyone working at the outlet, something he referred to as the “democratisation of data”.

This is at the absolute heart of what’s worked for us out here in Australia. We couldn’t have had the success we have out here without this feedback loop – not just the data, but also editors, subs and reporters all working with and caring about the data. Ophan’s transformed how we work, and will continue to do so as it adapts to the changing internet. There are no analytics tools on the market that do what it does, and building it into the heart of the newsroom is a crucial part of making it successful.

In other news, it’s been a little quiet round here as I gear up for leaving Australia; lots of small projects are on hiatus while I pack up life into boxes again, including Pocket Lint, my ongoing game design work on BOPTUB, and standard curmudgeonly blogging approach. Normal service will be resumed as soon as we are sure what is normal anyway.

No, good content is not enough for Facebook success

At the recent ONA conference, Liz Heron, who oversees Facebook’s news partnerships, came in for some questioning about how news organisations can do well on the platform – something that’s a cause of some consternation for many, as it becomes increasingly clear how important it is as a mass distribution service. This is one of her responses:

This is a familiar line from Facebook – I’ve been on panels with other employes who’ve said exactly the same thing. But while I have the greatest respect for Heron and understand that she has to present Facebook’s best side in public – and that a tweet may be cutting context away from a larger argument – this statement is demonstrably false. Even skimming the rather fraught question of what exactly “good” means in this context, it’s questionable whether quizzes and lists such as those that have brought Playbuzz its current success are in any meaningful way replicable for most news organisations.

It’s not that Playbuzz is “gaming the algorithm” necessarily, though it may be. It’s that the algorithm is not designed to promote news content. Facebook’s recent efforts to change that are, quite literally, an admission of that fact. Facebook itself knows that good – as in newsworthy, important, relevant, breaking, impactful, timely – is not sufficient for success on its platform; it sees that as a problem, now, and is moving to fix it.

In the mean time, creating “good” content will certainly help, but it won’t be sufficient. You can bypass that process completely by getting your community to create mediocre content that directly taps into questions of identity, like Playbuzz’s personality quizzes, and giving every piece absolutely superbly optimised headlines and sharing tools. You can cheerfully bury excellent work by putting it under headlines that don’t explain what on earth the story’s about, or are too long to parse, or are simply on subjects that people will happily read for hours but don’t want to associate themselves with publicly.

Time and attention are under huge pressure online. Facebook are split testing everything you create against everything else someone might want to see, from family photos to random links posted by people they’ve not met since high school, and first impressions matter enormously. “Good” isn’t enough for the algorithm, or for people who come to your site via their Facebook news feed. It never has been. Facebook should stop pretending that it is.

Further reading: Mathew Ingram has context and a longer discussion.

Do real names really make people nicer online?

Mathew Ingram at GigaOm has an interesting look at some Livefyre research suggesting that if you force people to use their real names to comment on your site, the vast majority will just stop commenting.

Most of those surveyed said that they responded anonymously (or pseudonymously) because they didn’t want their opinions to impact their work or professional life by being attached to their real names, or when they wanted the point of their comment to be the focus rather than their identity or background. And close to 80 percent of those surveyed said that if a site forced them to login with their offline identity, they would choose not to comment at all.

…

The bottom line is that by requiring real names, sites may decrease the potential for bad behavior, but they also significantly decrease the likelihood that many of their readers will comment.

That led me to an interesting question this morning: do real names really reduce the potential for bad behaviour in comments? It’s a popularly held belief, but there doesn’t seem to be a great deal of evidence out there to support the idea that meatspace identities are any more useful than persistent pseudonyms when it comes to holding people accountable for their actions online. On Twitter, Martin Belam points out:

…but there’s a big gap between “named staff members early in comment threads” and “real names for everyone”.

I can find some evidence that persistent pseudonymity is a positive thing. Disqus did a very large study in 2012 on their comments database; though their methodology is opaque, their results showed pseudonymous commenters posted both the largest number and the highest quality comments across their network. Pseudonyms make people more collaborative and more talkative in learning environments; they positively influence information sharing; they encourage people to share more about themselves in the relative safety of an identity disconnected from meatspace.

There’s also evidence that anonymity – a complete lack of identifiers and nothing to chain your interactions together to form a persona, as distinct from pseudonymity, where you pick your own identity then stick with it – is a negative influence on the civility of debate, and that it engenders more adversarial conversations in which fewer people’s minds are changed.

There’s a Czech study into the differences between anonymity and pseudonymity and how to design for reduced aggression, apparently, but the actual link 404s and the abstract isn’t detailed on the difference. There’s interesting research into the social cost of cheap, easily replaceable pseudonyms, which allow effective anonymity through the evasion of reputation consequences; Reddit and Twitter are great examples of communities where you can see this behaviour in action.

Investigating this issue isn’t helped by the fact that researchers have in the past conflated anonymous and pseudonymous behaviours, but there’s increasing awareness now that the two engender big community differences; it’s also skewed by places like 4chan and Reddit being prominently discussed while less adversarial communities like Tumblr or fandom communities are less often scrutinised. (Male-dominated communities are covered more than female-dominated communities, and widely seen as more typical: so it goes.)

I’ve found one study that suggests anonymity makes for less civil comments, but I can’t access the full study to find out what it says about pseudonymity. There are suggestions, like Martin’s, that moderation helps keep things civil; there’s evidence that genuine consequences for poor behaviour helps too. And there’s this:

“While evidence from South Korea, China, and Facebook is insufficient to draw conclusions about the long-term impacts of real name registration, the cases do provide insight into the formidable difficulties of implementing a real name system.”

But where’s the proof Facebook is right when it claims its real name policy is vital for civility on the site? Where’s the evidence that Facebook comments are more civil than a news site’s because of the identity attached, rather than because the news site goes largely unmoderated and Facebook’s comment plugin broadcasts your words to everyone you know on there? Or because most people just don’t comment any more, so arguments die faster? I am struggling to find one study that demonstrates a causal link.

So this is an open request: if you know of more studies that indicate that denying pseudonymity improves comment quality, let me know on Twitter @newsmary or share them in the comments here. If I’m wrong, and it makes a big difference, it’d be great to be better informed. And if I’m right, and it’s community norms, moderation and reputational consequences that matter, it’d be great to put the idea that real names are a magic bullet for community issues to bed once and for all.