The Apollo and Dionysus of digital evaluation

What's the digital dimension of non-profit evaluation? We started a discussion a couple of days ago at Charities Evaluation Services where myself and UnLtd World's Dan Lehner lobbed some digital stones in to a calm pool of nonprofit evaluation consultants.

Happily, I work somewhere where social media is embraced as a valid campaigning tool. But the righteous challenge is to hang that off the Planning Triangle in a way that answers the 'so what?' question. Facing that challenge means debating numbers versus stories and investment versus insight. This post lays out some of my thinking in the hope that others will pitch in with some ideas.

At Make Your Mark our specific objectives will be partly delivered by different kinds of digital activity, from buzz building to amplifying events to online community development. And each of these has some plausible metrics that can be drawn from lists (like this one for enterprise social media).

But for me the cross-cutting point of all that activity is engagement. As far as I can see, us digital types have been trying to visualise engagement for a couple of years and have been wrestling for just as long with how to make making numbers out of it. As Steve Bridger said "Measuring engagement is like eating an elephant: it’s a big job and you’re not sure where to start".

I wonder if I and others have become a bit lost in the chimera of measuring engagement. Even the latest thinking from the highly laudable Measurement Camp (if you haven't been yet, you're missing something) feels a bit like a post-reality justification for the purposes of satisfying digital buyers' spreadsheets.

Make Your Mark's purpose as a campaign is to change behaviour, so at the end of the day we need to influence people. The Edelman White Paper on Distributed Influence: Quantifying the Impact of Social Media(PDF) has some interesting pointers to measuring influence, ranging from the Social Media Index (uncannily similar to a spreadsheet we already developing for our own internal use) to the concepts of 'meme-starters' and 'meme-spreaders'. It finishes on the thought that traditional comms activities are amendable to metrics like metrics like impressions, conversations, in-bound links and friends, whereas activites that they call Open Collaboration "will adopt entirely new methodologies that measure based on outcomes".

But how do we track the outcomes. Anyone around Make Your Mark has had those experiences of seeing the light go on for a young person, that moment when they become inspired by the possibility of making their idea a reality. Or, equally as inspiring, has heard the enterprise message authentically expressed by someone like the young people from Moss Side who lack a lot of life's privileges. Social media is the story-engine that shares the feeling of what we do, the shared sense that 'this stuff makes a difference'. (There's a great series of NTEN webinars from the end of last year on Social Media and Storytelling). We can also go beyond this to track outcomes that come about because of social media, such as the schoolgirl who left a comment on the Make Your Mark blog which resulted in the team helping her to set up a MYM Club in her school.

So the trick must come in finding the right mashup of stories and numbers. The inimitable Beth Kanter explores this in depth in The ROI of Social Media where she riffs on the term 'Return on Insight'. What's the technique that converts the Apollonian distancing of neatly printed tables to the Dionysian celebration of shared sensations of change? I think one of the consultants at the CES session cracked it when she said that the lab coats of traditional expert evaluation were starting to give way to self-evaluation and user-led evaluation. Maybe what makes the difference is not just the social media but the people who's hands it's in - when the cameras are held by the young people (as they have been at some Make Your Mark events), where users are making the podcasts and the online communities are as self-managed as Savvy Chavvy - then, maybe, it'll be pretty clear what's working and what the impact really is.

Photo by smithmatt