Methodical Snark critical reflections on how we measure and assess civic tech

research links w 23-24 / 17



Research on nearly 3 decades of democratic innovation and e-participation in Latin America has some interesting findings (Brazil, Colombia, Mexico and Peru). According to an Open Democracy blogpost (the actual project’s website is down): civil society participation programming uses tech more often than not, smaller countries are less prolific than large countries in terms of tech-driven innovations, and tech driven innovations are just as common at the national level as they are at the sub-national level. Though digital innovations are widespread, they only rarely facilitate decision-making (30%)  or are formalized in legislation or policy (less than 50%).

University of Maryland research on anti-Trump protests finds digital media commonalities among an exceptionally diverse group, suggesting something that approximates a “movement.”

A review of research on government social media use finds that it is generally quantitative, ignoring both users and impacts, while a library study in the UK suggests that Open Data makes it hard to archive well in the NHS, and a study of service delivery in Kenya found that it was improved by decentralization, but that the mediating effects of e-government initiatives were insignificant (275 respondents, 8 county govts).

Things that work:

A review of original research from 2000-2015 (US) suggests that “nudge” programming by governments is a cost-benefit effective means of achieving social objectives when combined with other programming modalities. Open source pays, according to a recent report on seven years of strategic investment by the GFDRR (World Bank-hosted disaster response partnership), which showed a “200% return on investment in its open source software efforts.” There’s also new evidence that governance performance assessments influence country behavior most when they are comparative and participatory (my hot take here).

Not quite findings:

Last week saw papers calling government co-design magical thinking, presenting a a Do-it-yourself public visualization kit, describing two east African case studies on citizen-generated data, reviewing anglophonic standards for democratic consultations, arguing that deliberative democracy initiatives should forget being representative, proposing a model for measuring open government data success and then testing it in Indonesia, and finally, a new report from @billorme‏ assesses access to information in Mexico, Chile and Brazil under the SDG framework (#itscomplicated).

Community & Resources

A @NNIPHQ survey captures the state of play in tech training for US NGOs and government staff, resulting in recommendations for trainers and a catalogue of training offers for those who need it. Meanwhile, at the Open Knowledge blog, @sdopoku describes the challenges in training African CSOs to collect alternative data, including the importance of relationships to methodologically strong peers.

The 18th Annual International Conference on Digital Government Research presented ALL THE CONCEPTUAL FRAMEWORKS, including on government social media use, on open data competitive capability and on open budget data, each accompanied by empirical test runs.

Academics, what are they good for? Mentoring more than actual research outputs, suggests @fp2p‘s latest reflections. Nonetheless, @fp2p ‏ is asking for tips on case studies for upcoming research on adaptive programming in Nigeria, Pakistan, Myanmar and Mozambique, and GovLab is looking for researchers and practitioners to join the Data Collaboratives Research Network.

OGP launched it’s research strategy, which focuses on providing grants and convening spaces for researchers. It also includes a snapshot of how the OGP support unit sees the contemporary evidence base.


@canaryhaz ‏is the latest browser plugin to access scholarly articles, and it includes piping to burrow through your institutional subscriptions to find the best version of what you have access to #icanhazpdf. The Web Transparency and Accountability Project has released a set of jupiter notebooks and associated libraries with which to query their data (500 GB’s worth collected from the web’s top million websites every day), and @B_I_Tweets has released a tool for automating RCTs (no small claim). It’s designed for use in the field of tax collection, but looks flexible. You need to register interest to get started.

Most prominently, I’m kind of freaking out over @ODIHQ‘s Open Data Pathway (an online evaluation tool for organizational open data implementation), which has just added country-level comparison metrics. Whoa. Currently, this country level data is only associated with   179  of 345 countries, the remainders are waiting for users to “sign in and ‘add country’ to their accounts.” Still, this could be a tremendously interesting pile of data.

The new Handbook of Cyber-Development, Cyber-Democracy, and Cyber-Defense is a very weird collection of topics and authors, while the Digital Marketer is a book on “the grassroots circulation of opinion on digital platforms and beyond,” and Adam Fish has uploaded a free copy of his new book on how pro-democracy activists can use television (though you wouldn’t guess that’s what it’s about from the title).

In the Methodological Weeds

Based on our main results, Table 1 and Figure 1 show average marginal effects of corruption on political stability at different levels of youth bulge. We are 90% confident that a youth bulge beyond 20% of adult population, on average, combined with high levels of corruption can significantly destabilise political systems within specific countries when other factors described above also taken into account. We are 99% confident about a youth bulge beyond 30% levels.

That’s Mohammad Reza Farzanegan on using economic, demographic and corruption data to predict political uprisings. Unsurprisingly, I’d like to see information and communication capacities tested.

Otherwise, this blogpost argues for program evaluations that show no impact, and how they can and should impact policy, and the @fhi360research blog has more guidance on appropriate sample sizes.

Academic Opps

  • The OGP is offering grants for research on the impacts of open government (5pg proposals due 14 July)
  • The EU’s 2017 Ministerial Declaration on Digital Government is open for comment.
  • Webinar on ICT-mediated citizen engagement: voice or chatter (review of case studies supported by @allvoicescount, 5 July)


  • 2 Media/Comms PhDs @ Bournemouth University, UK, with special interest in Politics & media and Law, war & conflict, among many other areas (deadline 2 July)
  •  Interdisciplinary Lecturer or Senior Lecturer in Disasters (11 July, Univ Edinburgh)
  • Senior Research Analyst @ the Mo Ibrahim Foundation (application deadline 12 June)
  • Project Coordinator: Ethics and Governance of Artificial Intelligence (Berkman Klein Center)

Calls for proposals/papers:

Miscellanea & Absurdum

Add Comment

Methodical Snark critical reflections on how we measure and assess civic tech