Methodical Snark critical reflections on how we measure and assess civic tech

Nov-Jan Research Roundup: the many benefits of participation, tools for building civic engagement


So it’s been a minute. Sorry, that’s the problem with hobbies. Been busy with wrapping up my doctoral work and helping Rockefeller build a network to support digital transformation in US govt. More on both those soon. But for now, look out, it’s THREE MONTHS OF CIVIC TECH RESEARCH LINKS!!!!


Impact and outcomes:
Participation matters. A review of three separate studies suggests that online activism strengthens the relationship between individual’s gender identity and well-being, and that this effect is itself strengthened by toxic sexism online. There’s also evidence that the Austrians most active in online participation platforms are most satisfied with policy outcomes (transparency, participation, collaboration, trust), but repeated interaction in different activities doesn’t make a difference. This from a survey (n=73) of people using the MyLinz platform. There’s also more evidence that opengov builds trust. Specifically panel data from 29 European (2004 to 2015) shows “freedom of expression and citizen involvement in the democratic process” to be strong predictors.

Participatory budgeting initiatives that expand participation and deliberation are associated with broad health outcomes, such lower infant mortality rates, according to analysis of data from  114 Brazilian municipalities between 2009 and 2016.

On the other hand, a large-scale randomized health project evaluation in Uganda has failed to replicate  experimental evidence on the long-run impact of community-based monitoring (World Bank, 2014), which TAI reads as “casting doubt on the power of information to foster community monitoring or to generate improvements in health outcomes.”

Boosting citizen engagement:
First of all, how to get government about citizen engagement? According to a doctoral thesis on Montana, Idaho, and Wyoming, “the presence of institutions, collaboration, and leadership […] increase government officials’ perceptions that citizen engagement in the strategic planning process is both beneficial and impacts the public policy process.”

There’s also new experimental evidence that government responsiveness leads to increased citizen engagement, this time from two field experiments in which Ugandan community reporters on sanitation issues (90 and 97 administrative zones). Community nominations and public announcements did not increase citizen reporting, government responsiveness did, over time. Also, Indians know about m-government, but can’t be bothered to use it (online survey, n=81), and expectations about the efficiency, quality and accountability of e-services are demonstrated to drive citizen use in a Ghanian case study.

Inside government
Korea is a lot better at open government data than the US or Canada, according to comparative analysis by a Korean researcher. Though not all high value government data is high value, or at least, the cost-benefit ratio varies wildly across institutions and data sets, as demonstrated in an analysis of five dutch high value data sets.  In fact, a survey conducted by Open Data Watch (n=7 [!]) suggests that people are using national statistical web portals, but those websites aren’t well-configured.

Meanwhile, local governments participating in international processes have remarkably consistent conceptualizations of open government. This according to content analysis of 15 sub-national OGP action plans, which shows an emphasis on open data that has also been found in national action plans (Clark & Francoli, 2014).

Oh, and European policy labs are mostly interested in “public sector innovation” and significantly shaped by the administrative cultures of their home countries. This according to twitter analysis of the conversations between 64 such laboratories in Europe, which also finds that those conversations are dominated by the British and the French.

Field comment:
A global assessment of the ICTD field conducted by DIAL confirms what anyone working with ICT4D already knew: Funding does not support proven tech, “Open Source Software is valued, but not consistently used,” and “Capacity to use digital data and solutions is still low across all stakeholder groups.” The blogpost gives an excellent summary of the 32 pg report, which is based on desk research, survey (n=58), and key informant interviews (n=42).

Setting the stage:
How does open government data driven co-creation occur? That’s the title of a paper presenting 6 enabling factors on the basis of a Chicago case study: “motivated stakeholders, innovative leaders, proper communication, an existing OGD portal, external funding, and agile development.”

A review of Brazilian online deliberative platforms suggests that socio-political context is more important that structural design for determining the quality and character of deliberative processes, and digital activism is fundamentally motivated by individuals “experience and beliefs,” according to an Indonesian survey (n=52).

From the Duh Desk:

Useful Research

Open data and data portals: The ODW report on NSO websites mentioned above also produced a customized dashboard to help National Statistical Offices to monitor the use of their websites and data portals. IADB has a framework for assessing open data demand, and Dutch and Swedish researchers have proposed a framework for assessing the data ecosystems available to support countries’ SDG reporting. There’s also a framework for monitoring how citizens experience government documents, and feeding that into document design, is built on a review of 57 documents from the Chilean Agency for Quality in Education. Data stories, faceted menus and visualisations are some of the linked data viewing techniques that governments can use to increase citizen uptake of open data, and which this article validates in the Dutch open data context.

Civic apps and citizen voice: This framework for evaluating the democratic potential of civic tech applications focuses on six criteria (inclusiveness, deliberation, influence, publicity, mobilization, and knowledge production) and is validated on a Finnish hackathon. Meanwhile, the iGPS is a “Context-Driven Data Visualization Engine” for citizen input, that scrapes data on citizen complaints, then develops machine learning models and produces visualizations to give policy-makers actional insights for decision-making around service delivery. This article describes how the iGPS was recently validated on “six governmental datasets from 46 agencies.”

Crowdsourcing: Longitudinal data on Finish crowdlaw suggests that govt crowdsourcing projects should be designed to cater simultaneously to people’s self interest and desire to learn. Meanwhile, this case study of a DFID crowdsourcing platform shows how the best efforts might not yield any results.

Ombudsmen: The OECD has recommendations for how Ombudsmen Institutions can support open gov, based on a survey of 64 such orgs.

Procurement: Indian researchers have developed a data-visualization tool for collecting, collating and presenting e-Procurement process data for comparison and benchmarking. The tool is tested on Indian procurement data, but is built on the Open Contracting Data Standard, and uses a tabular web crawler, so should be widely applicable.

Data sharing: Dutch researchers have design recommendations for making data collaboratives sustainable in challenging environments. A survey of 9 US organizations suggests that there are four ways that data intermediary organizations can help communities use data, including (1) democratizing data, (2) adding  value to existing data, (3) enhancing communities’ data  literacy, and (4) building communities’ data capacity.

Cash transfers: A data visualization tool for policy-making on conditional cash transfers in the Philippines.

Digital Security:  Azerbaijanian researchers have proposed a framework for assessing the security threats that social networks pose to e-government.

AI: Mimi Onuoha and Mother Cyborg have written a “People’s Guide to AI” (80pg pdf), that spells out how norms and equity get smuggled into everyday life, and why it matters). It’s smart and easy to read (plus well designed).(h/t GovLab)

Smart cities: Performance assessment scheme for smart cities.

Case studies

Some case studies deserve special mention: Leader in e-government, laggard in open data: exploring the case of Estonia is a fascinating analysis of insider processes, suggesting that in this widely lauded hero of digital gov, digitization is in some ways obstructing open government.



  • Issues & Tools



  • Regions & Countries



  • Cities



Concepts and Frameworks

Community & Curation




  • The EC is reporting back on a project called The Future of Government 2030+. After holding consultations in seven European countries, they’ve suggested four scenarios for how digital govt might play out, in some ways updating Fung et al.’s (2013) Six models for the internet + politics. The scenarios are DIY Democracy, Private Algocracy, Super Collaborative Government, and Over-Regulatocracy.
  • The Stockholm Environmental Institute has a policy brief on how public participation can improve environmental decision-making, based on a wide review of policy studies research.
  • CSCW special proposes new methodologies to “explore power asymmetry, motivation, working conditions and community dynamics in crowdsourcing”.
  • There’s a research agenda for data collaboratives


Special issues:


In the Methodological Weeds

Miscellanea and Absurdum

Add Comment

Methodical Snark critical reflections on how we measure and assess civic tech


Get in touch

Suggest research to be reviewed or mini-lit reviews. Ask questions or tell me why I'm wrong.