I read this For you

What is it: Governing by rankings: How the Global Open Data Index helps advance the open data agenda, online report (I pdf’d it for you, at 25pgs)

Main point: GODI rankings are motivating government actors, but largely as a function of their simplicity, which can have perverse effects, and isn’t as useful to other stakeholders.

This is an important collection of anecdotal observations, which should confirm support and commitment to GODI from the international community. It should also motivate a more careful and thorough approach to the same research question, to produce useful insights for strategy and design.

Methods:

Analysis is based on a review of relevant policy and program documents in three countries, plus 12 interviews (distribution is unclear). This report seems to be based on causal information gathering, no methodology is described or apparent.

Where’s it coming from:

This is research by a PHD student, commissioned by Open Knowledge International, so presumably intended both to inform their own strategic thinking, and and to demonstrate impact to donors. It wants to find impact.

Should you care?

Yes, if you’re a donor or someone else who supports this type of work. This report provides useful preliminary evidence that rankings like GODI can make a difference. Support this.

But if you are concerned about the more nuanced issues surrounding how rankings like GODI motivate change, and what this means for their design, this report will leave you wanting. More and more rigorous research is needed to answer pressing questions about these types of initiatives motivate change and set policy agendas in different contexts.

Findings:

• There is lots of anecdotal evidence that government actors care about GODI rankings, and that this can mobilize a complex set of nuanced incentives within and between government institutions.
• There is less evidence that GODI rankings provoke rational behavior in line with the GODI theory of change, and some instances of perverse incentives.

Insights:

• The relevance of GODI rankings to government seems to derive largely from its simplicity (simple numbers make good soundbytes). That same simplicity makes it less useful to other stakeholder groups.
• There are several observations regarding the arbitrary nature of GODI rankings, and observations that this seems to be linked to lack of response by government (pg 15 & 19 in pdf), but this is not explored.
• Media in these countries treat GODI as the same soundbytes with which politician’s treat GODI, suggesting that impact is limited to internal government incentive structures, and that GODI is not opening meaningful civic spaces for policy debate.

Shortcomings:

• This is a great introduction to the issue. As a collection of anecdotes and casual observations from three countries, however, it doesn’t come anywhere near to answering the question implied by the report’s title, and reads like the opening chapter to an important book. Further research should take a structured approach to
  • explore the causal mechanisms that underpin government responses to rankings; and
  • compare the contextual factors at play in different contexts and how they influence government responses (typological theorizing would be a rigorous approach, mechanism mapping a more lightweight but still useful approach)
• Some of the most promising and compelling insights are given only casual reference and should be further explored. In addition to the insights listed above, the report notes that:
  • Interviewees in Argentina “believed that an ongoing evaluation capturing progress in open data publication over time would be more useful than running the assessment once a year and with a limited timespan.”
  • “GODI and other open data indices play an essential role in keeping the open data agenda alive politically. As an interviewee stated, if Ukraine gets visibility internationally through GODI or ODB, political commitment to open data is more likely to last longer.”