Agents, is there anything they can’t do?
Well, quite a lot, but as the discussion about agentic workflow and tool selection proceeds, I want to take a close look at what they can do to help analysis. This is prompted by some recent analyses I have seen posted related to my own research, where agentic analysis has clearly lowered the bar for analysing large and complex data sets.
Just after the UN General Assembly this September, ControlAI’s Jonathan Salter mapped the perspectives and positions of 102 actors on AI catastrophic risk, to show “who supports AI safety”, and Diplo’s Jovan Kurbalija released a UN GA AI Risk Matrix, mapping 89 existing, 37 exclusion and 12 existential AI risk mentions at the UN in New York, to make an argument about how risks should be diagnosed. These are deeply compelling contributions to understanding how global policy discourse around AI Safety is developing, but they are also noteworthy because they are clearly enabled by the analytical use of AI, though neither one gives a clear and detailed description of how (though Jovan did promptly and thoroughly answer my methodological questions in comments).
Lowering the threshold for this kind of research with AI is great. We need more rigorous analysis free from the glacial constraints of peer review. And the lack of methodological info is understandable, it’s the most boring part for most of us. But not having a methods section to interrogate makes it hard to assess whether the results of this kind of analysis make sense, or actually say what they claim to say.
Do we need a kind of standardized disclosure form for AI-supported analysis in hot takes and grey literature? Maybe, but for a first step, I’d like to ask if there is any emerging consensus on what best methodological practice for agentic and LLM enabled analysis looks like. So here’s this mini lit review.
What an agentic methodology might look like
Research on methodologies for using agents and LLMs in social science is not well established. Most of the cutting edge work seems to appear mostly in preprints, working papers, blog posts and lectures. Not a lot has gotten through peer review yet, and much of the work actually using AI in analysis only describe their methods in passing.
The digital methods tradition is the obvious place to build the methodology. Rogers and the DMI have spent two decades developing methods for web and platform data, and their summer and winter schools teach bleeding edge methods and reflexive use for both using and critiquing digital tools. So far this has produced a lot of interesting products and analysis, but aside from a couple of talks, there doesn’t seem to be any concrete guidelines on how to build AI-supported methods. So we need to pull from a broader pool.
Here’s what I was able to find: Haase and Pokutta (2025) set out a six-level continuum for ways of using AI, from stateless LLM tools (Level 0) to multi-agent and complex adaptive systems (Level 5), with each level adding memory, autonomy or coordination. The framework lets a reader say which kind of system a study uses, and so which risks apply. Its examples are mostly preprints. Xu et al. (2024) map where LLMs enter each stage of social research, though this was before most agentic capacity emerged, and they argue that simulated subjects have no gold standard for believability.
There’s quite a bit of light declaration of AI use in methods, including Atkinson’s (2024) use of Python scripted ChatGPT to collect policy PDFs and many DMI school outputs, though many of these also provide emerging guidelines. The 2026 DMI Winter School call, “Research-with-AI critique”, asks when outputs should be checked by hand or against other chatbots. An LT2D summer school at Université de Cergy-Pontoise reads like a syllabus: prompting strategies, research personas, reflexive annotation, auditing LLMs and synthetic data. Rogers’ 2025 Košice lecture follows a similar sequence, though only a faculty news item about it is public, so I wasn’t able to review the content.
Writing on strategic management research, Carlson and Burbano (2026) give the most detailed practical guidance so far. Testing five models and seven prompting strategies on 295,985 Kickstarter campaigns, they find that the most accurate configuration does not always recover the expected downstream effect. A Claude 3 Opus setup reaches 87.6% accuracy yet yields a negative coefficient. Their remedy is systematic sensitivity analysis. Their reporting guidance transfers to agents, though the study covers annotation.
Reviewing this suggests the following basic criteria could be considered to compose a methodology for using AI agents or LLMs in analysis for social science research. Each of these requires a deliberate design decision, testing, and documentation:
| Component | Description | Questions to ask | Resources for detailed discussion |
|---|---|---|---|
| Specification | Prompts, personas, tool permissions and system prompts logged as data, so the instructions behind an output can be read and rerun. | What exactly did the agent receive as instructions? Which tools could it call, and with what permissions? Were system prompts default or custom? Do personas bias the output in ways that matter for the question? | Carlson & Burbano (2026), Table 1 and Table 2, on documenting prompt components and system prompts (doi:10.1002/smj.70023). DMI Winter School 2026 call on persona development (AoIR list). |
| Model version and access | Model name, version, access route and archived outputs recorded, so a finding can be rerun when models change. | Which exact model and version produced the results? How was it accessed, and when? Are outputs stored, and can the model still be reached? What happens to replication if the provider deprecates it? | Carlson & Burbano (2026), Stage 2 and footnote 2 on deprecation and version documentation (doi:10.1002/smj.70023). Queen & Zou (2026), The shelf life of LLM findings, on claims that change with model releases. |
| Instrument audit | Labels and choices tested against a gold-standard subset built by humans before results are trusted. | How large and how representative is the gold set? Who labelled it, and what was their agreement? Where does the model fail, and are the errors systematic? | Carlson & Burbano (2026), Stage 1 and Section 3.4 on the 250-project gold set (doi:10.1002/smj.70023). Rogers’ 2025 lecture on auditing LLMs (UPJŠ news item). DMI-linked LT2D summer school module on auditing (programme). |
| Sensitivity analysis | Families of prompts and models run, with the spread of results reported rather than a single estimate. | How many prompt and model variants were tested? Does the conclusion hold across them? Which choice moves the result most? Is the spread reported alongside the headline finding? | Carlson & Burbano (2026), Section 3.5 and Figure 3 on 100 prompt variants and bounded estimates (doi:10.1002/smj.70023). |
| Traces | Each action, retrieval and decision logged, so a reader can follow the path from input to finding. | What did the agent do at each step, and in what order? Which sources did it retrieve? Where did it decide to stop or to try again? Can a reader reconstruct the path? | Haase & Pokutta (2025), on tool use and planning at Levels 2 to 3 (arXiv:2506.01839). AI sprints paper on intermediate objects (arXiv:2512.12371), to be verified. |
| Division of labour | The steps humans keep, check and verify, stated explicitly in the text. | Which steps did a human decide, check or override? Who verified the outputs, and how many were checked? Is the verification sample reported? | Kurbalija’s answers on the UN risk counts, in the LinkedIn exchange. Carlson & Burbano (2026), Discussion on continued human expertise (doi:10.1002/smj.70023). Haase & Pokutta (2025), on hybrid human–AI systems (arXiv:2506.01839). |
| Attribution | Identifying where the knowledge in outputs comes from, so credit and provenance can be traced. | Can outputs be traced to their sources? Does the method show whether a claim rests on retrieved material or on parametric memory? Who should be credited, and how is that decided? | Wührl, Ruckdeschel, Lo & Rogers (2026), FAccT ’26 (doi:10.1145/3805689.3812236; arXiv:2602.10995). ICLR 2026 blog post on the human knowledge loophole. |
Red lines
There seems to be alignment around the idea agreement that human control and oversight over both analysis and analytical workflows is essential. A blog post from the International Conference on Learning Representations (ICLR) offers a useful illustration, arguing that LLMs depend on human-generated data rather than computation alone, so humans remain essential to reliable analytical outcomes. Synthetic data can change the form of training examples but not their content, which underlines the continuing need for human knowledge and oversight.
In practice, this seems to imply that:
- Agents may execute, but humans must decide what counts as a finding.
- No result should rest on a single unreported configuration.
- Synthetic data cannot supply evidence about the population it imitates.
- Automated coding requires human-checked validation, with the check reported.
- Outputs must be traceable to inputs and sources.
To my mind, we should also add a red line for data protection (e.g. analysis of personal data must be run on sovereign platforms and only on GDPR-compliant services), and a declaration requirement (e.g. you must declare in detail how AI tools have been used). There’s likely more as well, but agreeing on these red lines, both for hot takes and for scientific production, would be a great start.
What it means for peer review
Most of the inspiration and hits for this mini-lit review exists outside of peer review, but these dynamics threaten to further cripple an already lumbering, clumsy dinosaur. If peer review succumbs to the general tendency among established academics to reject anything that carries a whiff of AI, we’ll get pushback against powerful methods, and disincentives to declare and improve those methods along the way.
We need to support the academics and social scientists who actually devote time and energy to reviewing peers’ work, to ask the right questions and to learn and support while doing so. The DMI community is a good place to begin elaborating these standards, and there might be a good opportunity to collaborate with Peer Review Week or the Committee on Publication Ethics (COPE) to create some easily accessible and usable guidelines. This is worth investing in.
What it means for research
Agents can now do in an afternoon what used to take months with a gaggle of research, and the findings can look convincing. This review suggests initial components for a rigorous and defensible agentic methodology: specification, model version and access, instrument audit, sensitivity analysis and traces, with a clear division of labour and attribution running through all of them. It also sets out red lines: uses of agents and LLMs that should disqualify an analysis from serious consideration. Human researchers need to stay not just in the loop, but in charge of designing and tracing it.
There are more questions still open than answered. How should researchers select and define personas and tools for reflexive quality control across multiple agents? Which models should be used, and how should that choice be justified and tested? How should data protection be handled when sensitive material passes through an agent? When and how can a reader tell a hot take from scientific analysis, and how do the standards for the second reach the first? I’ll inevitably come back to several of these. Let me know if you have any insights, questions or examples to fold into the mix.
