Methodical Snark critical reflections on how we measure and assess civic tech

2 week Roundup: civic space is good for trust, egov is bad for corruption, metrics for government culture change


So here’s two weeks’ worth, because summer frolic. There’ a lot to cover. Let’s start on a cheery note:


Trust and civic space.
The latest Democracy Perception Index (DPI) finds that “a majority of people around the world feel like they have no voice in politics and that their governments are not acting in their interest.” What to do about it? Well analysis of comparative indicators for 29 European countries offers more compelling evidence that “voice and accountability is a stronger correlate of trust in government than GDP growth.”  So yeah. There’s that. And though OGP suggests itself as a remedy, and notes that OGP member countries have healthier civic space than non-members, their recent analysis of civius and action plan data also finds that civic space is deteriorating generally across the globe, and that OGP action plans are not doing much to address the issue (less than 4% o 2,733 commitments analyzed were relevant).

Political communication
A German survey suggests that citizens are most drawn in city government facebook pages for interaction with government officials, not simply receiving information, while a Ugandan RCT (n=16,083) shows modest effect of budget and corruption related sms messages on voting behavior.

Social media and messaging.
Meanwhile, Twitter data analysis in the latest NBER shows that political bots affect social media behavior when they reiterate prior beliefs of social media users, which supports theories of online echo chambers. This might be less of a problem if people are increasingly turning to private messaging for news source instead of social media, as suggested by the recent Reuters Institute Digital News Report (DNR). Though the potential for rumours there is just as significant (see case studies below). In any case, Twitter activists still need to think about how to get global hashtags framed by national media, who are still exercise decisive control over national frames, according to  qualitative frame analysis of the #BringBackOurGirls campaign.

Political communication in periods of upheaval.
Survey data from the 2012 Wisconsin recall suggests that relationship between political communication and participation goes a little janky during heated partisan controversy, where “broadcast news consumption negatively predicts participation, whereas political conversation with coworkers and use of political social media positively predict participation,” and analysis of Thai social media protest suggest that the communications of organizations and loosely knit crowds align in times of political crisis.

Context, context, context. RCTs on the effects of social recognition of bureaucracy on public service delivery in Nigeria found markedly inexplicable outcomes in two ostensibly comparable states. “That an intervention was successful in one state, but not another, speaks to the importance of considering this variation when translating successful findings to a new context—even when the implementers in both cases are the same.” Similarly, a field experiment in Kyrgyz Republic demonstrates the effects of anonymous rating of tax inspectors on corruption, but only for firms with very specific market positions, depending on whether market elasticity allows the cost of bribes to be passed onto consumers.

Lastly, there’s yet more evidence that e-government has a positive effect on lowering corruption, this time based on analysis of sub-indices from prominent comparative global measures over a 15 year period. The study notes, among other things, that “the impact of ICT on citizens’ everyday life within the public sector is constantly increasing worldwide [while] the level of corruption is stable worldwide or the score slightly decreases (108).” There are also some fascinating methods discussions and insights on the impact of the financial crisis and other 2008 global events on the relationship between e-gov and corruption.

Useful Research

Oxfam has conducted a review of the contract disclosure policies of 40 oil, gas and mining companies, highlighting a gap with the emergent norm for open contracting policies by governments.

Political skill and credibility are the most important characteristics for “translators” who advance the use of evidence and research in policy process around the world. This and other useful insights confirm that the rigor of research is less important, in a fascinating report from @results4dev. Based on lit review and 2 case studies (Ghana and Buenos Aires).

Case studies

There were lots of articles and chapters on e-government, including information management in Bangladesh e-government, usability of the Philippines’ open data portal and Indonesia, e-government in Nepal. In Europe, there were cases on open evidence and collaborative government planning in the Basque Country, managing open government data in Swedish municipalities, and open health data privacy violations in Norway. You can also read about

There’s lots of activism cases in this round up, including analysis of how identity and self-representation in the #ILookLikeanEngineer campaign, and this masters thesis suggesting that contributors to /The_Donald sub-Reddit have “higher internal political efficacy than the general public, but appear to be drawn to online activism as a result of their unusually low external political efficacy.” Meanwhile, Duncan Green reviews Sarah Corbett’s Craftivist: the art of Gentle Protest, which is ostensibly more about civic needlepoint than civic tech, but looks like it has some great orthogonal thinking on how to craft and target information in advocacy.

Also check out these write ups of online fact-checking in Zimbabwe and social media fact-checking in Colombia, and this SSRN article outlining the history and Essentials of the Right of Access to Public Information.

Community & Curation

Public Administration Review has a special issue on Reduced‐Boundary Governance (by which they mean collaboration between government and others). It includes articles on capacity and cost efficiency issues related to co-production, the effectiveness of voluntary environmental programs and work release programs, motivational issues underpinning policy processes and “responsibility‐centered management” and methodological attention to of qualitative approaches and the “reliability of heavily utilized datasets.”

Open Data Roundup: The The Open Data Research Network has put out an update with 13 new research papers on open data that were published over the last six months.

Debate: There’s pushback on the critical 3ie evaluation of CDD programs highlighted on the my last roundup.

Get involved: OECD DAC is holding an open consultation on revising their evaluation criteria and @nesta_uk  is looking for feedback on it’s messaging for government that wants to go digital, and Sunlight Foundation is launching research on the relationship between cities’ open data policies and FOIA performance. Exciting stuff. And in true best practice, their asking for feedback on their methods and research design.

Lessons & Guides

In the Methodological Weeds

Whoa. @Nesta_UK has developed a metric for measuring culture change in government.

A review of 109 development impact evaluations was only able to reproduce the results of 27 studies, questioning the credibility of the approach. This might be a good time to revisit the approach, as other research shows that the wild expansion of impact evaluation since the 90s has begun to plateau.

Also:Not quite research and not quite methods, but ICTworks has a nice blogpost on “7 Reasons to Use Call Centers for Data Collection.” And I guess it’s not really tech either, since their general reason seems to be: humans.

Lastly, the GODAN project has published a review of methods and frameworks for evaluating open data impact. The 20 page report offers an excellent high-level view of the landscape, making useful distinctions between different types of approaches, drawn from academic and practitioner work. It might not be comprehensive, but if feels representative, and makes for an excellent introduction for anyone looking to situate themselves in the landscape. The report also makes a strong argument for a more user-centric approach to impact evaluation, which makes strong sense given GODAN’s advocacy position. The report concludes by noting that GODAN “will translate these lessons into a coherent conceptual and methodological framework to suit the goals and needs of the GODAN project.” Looking fwd.

Miscellanea and Absurdum

Add Comment

Methodical Snark critical reflections on how we measure and assess civic tech