I work at Delib and as part of my role I help governments around the world create and run online policy consultations. The challenges involved vary enormously, although a common thread is a concern about how to manage the analysis of the responses, particularly as a more open approach often encourages more people to respond. I’d argue that’s the whole point but I understand the trepidation given resources (i.e people) are becoming scarce (without even mentioning the pressure to perform driven by the mighty and much vaulted ‘Digital by Default’ agenda). It seems teams are expected to become experts at this sort of analysis, with little or no training.
I’ve known concerns about managing response analysis to sometimes block the entire idea of consulting online. This is a pity, because the efficiencies gained from being able to simply collect all responses into a dedicated online database, far outweigh the increased cost of analysing the additional responses you’d expect. Quite simply, running an online consultation by email, or dare I say it, post, is nuts. So, if we accept that running a policy consultation online removes the need for (and cost of) scanning, transcribing, creating and stamping unique IDs and other such horrors, then how can a policy team get better at grappling with the analysis of the results?
The first answer, which is more common that you might think given these frugal times, is simply to pay a company to do it for you. There’s lots of them out there, some worse than others, some more expensive than others, but once you find a good one, they can be an excellent partner to have. Some of the more notable companies in the field include: YouGov, GHK Consulting and Ipsos Mori, although smaller local companies can be just as effective.
Failing that, there are a number of simple things you can do to ease the pain of analysis.
1. Plan ahead.
Think about how you’re going to analyse your results at the start of the project – before you’ve decided which areas you’re focussing on, which questions you’re going to ask, in fact anything at all. Do you want to run an extremely in-depth analysis, for example are you going to assign a code to every word received? Will you be dealing with a lot of qualitative response or will more quantitative responses provide you with the information you need? If you’re coding, do you have a code book agreed upon? Will you be analysing by response or by question? Are you going to run an ongoing analysis while the consultation is still open or are you going to wait until the whole thing closes? There’s a lot to consider but by front loading the work you’ll save yourself some serious pain later. Be a good Scout – always be prepared.
2. Think about pragmatic ways to reduce the burden
There’s plenty of small things here that can make a big difference. One often discussed method is to try to reduce the amount of qualitative responses, and also to reduce the quantity of questions to the bear minimum. In a perfect world all you’d ask is ‘yes, no, maybe’ but that’s not necessarily that helpful from a research perspective, so the balance of response types should be considered early, (see above – Plan Ahead).
Some platforms, (e.g. Delib’s Citizen Space), allow for the inclusion of ‘analysis only questions’ which can greatly aid your analysis work flows. These are questions which can be added to a survey that aren’t publicly visible, allowing you to ask your analysis teams things like, ‘which team allocated this code’, ‘why was this code added’ or ‘why was this response rejected?’ Again these can be planned early and potentially remove several manual steps, thereby increasing your efficiency.
3. Stick to the plan
I’ve known people who do all the good work of preparing their analysis workflows well in advance, before deciding on a new ‘nice to have’ approach halfway through. Unless you have a minister shouting at you, which seems unlikely, stick to your original plan. Nothing makes analysis more painful, time consuming and expensive than changing your approach.
A recent example of a policy team finding a pragmatic way to reduce the burden of analysis comes from the Boundary Commission for England. In the words of Sam Hartley, “I’ve been involved in a number of policy consultation processes, the most recent of which combined the choice of a tool which offered workflow analysis features and good forward planning; making the process manageable. We saw the advantage of using a system with features like coding and analysis only questions”.
4. Get yourself trained
Perhaps this is a little idealistic, but I’d hope not. If you work in a policy team and you’ve found yourself responsible for the analysis of a particularly contentious consultation, it simply makes senses to get some basic analysis training. You can go to the private sector and pay for their time or why not seek out colleagues who know what they’re doing already? A couple of hours of training could save you days in the long run. If you are interested in getting trained, The Consultation Institute is worth a look.
As an aside to the training point, I’d like to see one of the departments (I’m looking at you Cabinet Office) arrange a training day/ morning to train as many policy teams as possible about basic analysis processes.
If you want to try out a system with analysis tools I luckily have one you can play with. It’s an example consultation within Citizen Space with a few responses that will allow you to try out data cleaning, coding/ tagging, running queries and so on, which you can access here, (username: firstname.lastname@example.org password: delib). If you want a hand getting used to the features, give me a shout and I can walk you through it.
I could talk about this a whole lot more but I won’t, I sense you’re bored; brevity is best. My point? If you run a proper policy consultation online, you’ll achieve greater efficiencies in terms of collecting responses, bring citizens/ stakeholders into decision making and ultimately create more effective and inclusive policy. Why let a simple thing like analysis stand in the way? Plan it extensively from the beginning, think of ways to streamline and if you need some training, go and seek it out. The prize is worth it. I hope you agree.