What worries me are pilots that begin with the outcome already decided. The licence has been signed, the roadmap assumes the tool will scale, and the evaluation is there to confirm the route. An experiment earns its name when it can come back with an unwelcome answer, and when that answer can still change what we do. A pilot that can only end in roll-out is a launch with a trial period.

One question belongs above every pilot, before it starts and all the way through: what are we actually doing here? Is this something our students and staff need? Is it something we want, enough to try at all? The answer can change halfway. It should be allowed to.

Listening to Margaret Bearman on a recent podcast, I picked up a distinction I like returning to. There are three ways of talking about AI. We can talk about what it is: a system that predicts the next word. We can talk about what it can do: summarise a reading, mark a quiz, run an errand through an agent. And we can talk about what it sets in motion, intended or not: the habits it changes, the roles it redraws, the things we stop doing because a machine now does them.

The third conversation is, to me, the most interesting, and the one we hold least often. When we do hold it, we keep it narrow and on the surface. Pilots are usually designed around the first two. The brief describes the tool and its functions; the evaluation measures whether it performed. However, to understand a pilot, we need all three. And the third, I am convinced, deserves the most care and attention.

It is also the easiest to lose. Ben Williamson and colleagues studied England’s £23 million AI tutoring testbed and call it experimental futuring (Williamson et al., 2026). A prototype, they show, is a working piece of a future someone has already chosen. Testing it changes things. Timetables move. Roles shift. Lesson plans and curriculum guidance are cut into tagged chunks, stored centrally, so AI tools can draw on them. By the time the verdict arrives, the school has been rebuilt around the tool. The pilot asked a question it had already answered.

Thomas Corbin sees the same pattern inside universities: when AI replaces a teaching interaction, the savings show up in this year’s budget and the losses years later, once the people who could rebuild it have gone (Corbin, 2026). Some pilots dissolve the very thing what you would need to go back pre-pilot. Whatever we were testing then becomes our practice, without the most important actors having decided that it should.

Margaret Bearman and colleagues found the same blind spot in the research itself. Reviewing thirteen studies on sustaining teaching innovations in higher education, they noticed that almost all of them framed innovation as good in itself, which left little room for innovations that had stopped being useful (Bearman et al., 2024). We ask how to keep new things going. We seldom ask what to let go of. Some pilots deserve to end, and deciding so should count as a result.

Nobody can wait for the evidence on AI in education to settle, and waiting has great costs of its own. So the task is to keep our experiments honest: open about what they might find, and clear about what they are for.

In practice, that means evaluating a pilot on two levels. The first asks whether the tool did what it promised. The second asks what it did to us: to how students study, how teachers spend their time, how a team talks about assessment, what we stopped doing. The first level fits neatly in a report. The second rarely makes it in, because nobody was asked to look for it. Yet the second level is where the third conversation lives, and it is usually the part worth keeping, whatever happens to the tool post-pilot.

Keeping the question open

So if these are real pitfalls, what might keep us from falling into them? A few habits help, and most of them cost little.

  • Ask the purpose question more than once. The Council of Europe’s new Compass for AI governance in education opens with it for a reason: “What are we trying to achieve?” (Council of Europe, 2026). Ask it at the start, again in month four when the licence is paid and the early results look promising, and once more before anyone decides the pilot becomes practice.
  • Name what the pilot will set in motion. Write down at the start what might outlast the tool: contracts, roles, data flows, curriculum formats, skills we stop practising, relationships that change. If the list is long, the pilot is already a decision, and it deserves to be treated as one. If it is short, let the pilot stay what it should be: an open question with an unknown answer, whose early results are a first reading and never a final verdict.
  • Decide in advance what would make you stop, and what would make you continue. Stopping is cheapest before anyone depends on the tool, so agree on the signals early, and on who gets to read them. That agreement cuts both ways. It keeps a pilot from sliding into practice unexamined, and it protects the pilot from being cancelled on a whim, a budget cycle, a change of leadership, or an administrative misstep that nobody wants to own. A pilot spends more than money. It spends people’s time, attention and goodwill, which are scarcer and slower to replenish. Staff who see their work ended without reasons will think twice before volunteering again. Whatever the decision, it should rest on what the pilot showed. If a pilot ends for other reasons, those reasons deserve to be named and owned, and the people who did the work should hear them.
  • Give the social view a team. What a pilot changes in how people learn, teach and talk rarely shows up by itself, and no single observer sees all of it. Bring together a few people who see the pilot from different places: a teacher, a couple of students, an educational developer, someone from the programme team. Give them time to study what they notice, to talk it through with the people involved, and to share it beyond the pilot, so the institution learns from it whatever happens to the tool. Overall, we are in the learning business.

A pilot judged only on whether the tool worked has made its most important choices out of sight. A pilot that keeps asking what it is for, and what it is changing, leaves us wiser and still free to choose, even when the tool fails. Sometimes especially then.

References & Further Readings


Curing the Curriculum. (2026, September 16). Learning is not an assembly line: AI, bias and the mess of learning. Margaret Bearman #CTC16 [Audio podcast episode]. Spotify. open.spotify.com

Bearman, M., Chandir, H., Mahoney, P., & Partridge, H. (2024). Sustaining teaching and learning innovations: A scoping review. Higher Education Research & Development. https://doi.org/10.1080/07294360.2024.2364096

Corbin, T. (2026, September 8). AI is becoming more common in uni courses. What can students gain and what might they lose? The Conversation. https://theconversation.com/ai-is-becoming-more-common-in-uni-courses-what-can-students-gain-and-what-might-they-lose-291164

Council of Europe. (2026). Compass for Governance of AI & Digital Transformation in Education. coe.int https://www.coe.int/en/web/education/-/compass-for-governance-of-ai-digital-transformation-in-education

Williamson, B., Facer, K., Pykett, J., Sriprakash, A., & Valladares-Celis, C. (2026). Experimental futuring: AI prototyping and pilot testing as anticipatory governance in education. Globalisation, Societies and Education. https://doi.org/10.1080/14767724.2026.2720476