Cognitive Helmets for the AI Bicycle Part 2: The Sometimes-Wrong Bot

The Sometimes-Wrong Bot

When I was a postdoc, reflective of the inability of the university to contain my multitudes, I was cross-appointed in three disciplines (Cognitive Science, Computer Science & Engineering) in an interdisciplinary center (The Design Lab at UC San Diego). That summer the Design Lab had only just opened, so it was more the idea of a center.  

I was the first postdoc in the lab, and although it grew quickly even in that first year, snowballing in the way that centers do by collecting undergrads and grads and various affiliated faculty, it is remarkable to look at the sprawling Design Lab now and remember when I was helping screw the legs into the chairs in a big room on the first floor of a building at the edge of campus. One day, our research planning was interrupted by the delivery of heavy desk mounts for monitors, which were arduous to set up and distracted us from research tasks for a good long afternoon.

But every integration of some new office prop carried salience because it felt like we were making this place real, and that planted a seed that I carried forward into founding my own teams and my own labs. When you screw in legs, set up people’s monitors, and do the dishes in the tiny shared kitchen while a place is barely an idea of a place, your fingerprints smudge all over it forever. You are inexorably part of making it real. That was a tactic for being valued that I stole from being even loosely affiliated with engineering, and never gave back.

As an older sister in both reality and emotional orientation I do tend to take on a lot of making this place real tasks. For instance, I held late night statistics learning hours with some younger labmates. Through the haze of many different disciplinary traditions, we sat around in our newly-assembled chairs and tried to figure out a joint language through which we could produce a research paper. I thought I couldn’t code, but they thought they couldn’t math; we were determined to prove each other wrong, and I think we mostly did. To me that time was more valuable than any scholarly output.  

Of course I knew the stakes, which for me at the time were still starkly framed as "succeed in this bizarre academic world or go back to serving coffee." I did scholarly output. I worked on speculative ideas for how we could improve peer to peer feedback in online edtech environments. I did research work on a project between UCSD, Duke University, Coursera and Google that ended up becoming a bridge into contracting with a team at Google on their online hiring experiments (it was MOOC era, if anyone remembers MOOC era; one of the people I met on that team would eventually become my co-founder as we worked on a small and ambitiously human developer tool startup, but that's a story for another day).

But I excavated tiny treasures from that time too, of the type that I was already realizing I valued more than academia did. I was concerned about how we judged online learners primarily on metrics of velocity and completion (sound familiar?), and published some open scales on learners’ perceptions of the value of learning for mastery (not just performance), the first time I had ever done that. At the time preprint servers like PsyArXiv were just small experiments I had only just begun hearing about too. When I published one small open scale about mastery learning on this brand-new preprint server thing, I worried that people would judge me for it! Little did I know that I was also planting seeds in my own motivation to share open, working research for the rest of my career. Now I can still access that little mastery scale paper, while I cannot access one of the big learning analytics papers I published while working in my tech jobs, because I could not afford to publish it open access. 

At this time, I had a long-running research study idea that I could not get out of my head. I wanted to give people an intentionally often-right but sometimes-wrong automated tutor to see if it would increase their metacognition.

I have to give you some context about how weird of an idea this was at the time. Smart tutors (sometimes called intelligent tutoring systems) and the idea of adaptive curriculum maps were around in my HCI and CS circles. Learning analytics and its many subfields was a large and roiling body of work, no doubt spurred by the growing libraries of data about students’ learning, practice, effort and testing on online education platforms. 

But my idea ran counter to the orienting mission of the vast majority of the experimental learning tools I saw examples of in HCI. One of their design assumptions was that tools had to be–of course!--right. Key concerns swirled around deriving signal despite the noise of student error, of automating feedback that we could trust, and of developing student concept inventories that could create carefully-determined steps through a curricula, every next choice weighted by top-down instructions. I’m not saying these designs were of no value; they provided a lot of value relative to questions about how to steer students along the railroad of a piece of curricula. 

I just had a different question. Could people strategically learn from a sometimes-wrong bot? Could they, by interrogating a supposed tutor who could also err, become more self-regulated and agentic learners themselves? Could they learn to assess the content itself and the proposed information structure, rather than defer to authority? It was a mischievous question, the kind of question you would ask if you had been a learner constantly disappointed by the hidden curricula and casual cruelty of schools, as I had been. But it was also the kind of question you would ask if you had a wildly ambitious belief in the capacity of human beings to learn, as I did then and still do.

The thing is, a lot of the stuff we have to deal with in the world sits somewhere between wrong and right. Refining and developing your own judgment about this is a key part of becoming a self-regulated learner, a person with both social and cognitive problems to solve.

I had the notion that being presented with the ways a supposed educational tool could be wrong, not just right, might actually help learners develop better metacognition. I had a lot of fun versions of this idea that I knew I couldn’t actually get working myself and didn’t know how to deploy ethically and consensually, but they were fun to think about…maybe Sometimes-Wrong-Tutor-Bot could start truthful and then start to lie to learners about more complex concepts, and we would see how long it would take learners to detect it. Maybe together learners would have better support for each other than alone, and Sometimes-Wrong-Tutor-Bot would trick solo learners but turn out to be something learners would pair up against and try to defeat. Maybe Sometimes-Wrong-Tutor-Bot would be designed to mimic a tenured professor (or at least learners’ stereotypes about them), all intimidating academic jargon, and we could test if people detecting the wrongness through the intimidation and trappings of elitism. Maybe Sometimes-Wrong-Tutor-Bot  would be warm and personable instead, and we could learn which of those two conditions was more misleading. 

Throughout all of these wild ideas was a very real insight I had that active learning was a key to understanding learning tools. The fact that our learning is constructive, not passive, is to me one of the more beautiful things about it. Contrary to many assumptions we are not simple wax tablets waiting for an imprint from the world. We are meaning-makers.  

In A Wrinkle in Time by Madeleine L’Engle, one of my favorite childhood books, on the sickening planet of Camazotz on a mission to rescue her father from forces she has only newly encountered and can hardly understand, protagonist Meg realizes that stepping through a solid wall is merely a matter of rearranging atoms. Learning is power: once she knows, she can do it, and the wall no longer has power over her. Identifying moments to learn and transform yourself in a world that doesn't always care about your learning is like that, maybe.

Reader, I have made the Sometimes-Wrong-Tutor-Bot from the science fiction classic, Do Not Build the Sometimes-Wrong-Tutor Bot. You can use it right now. It is a Claude Skill that I wrote called Learning-Opportunities. I wrote it in a fit of a weeklong experimentation I was doing to try to see if I could design for an active learning interaction, a set of basic instructions that would capitalize on what I know about learning science and try to wedge it into an AI coding workflow in a way that could be a useful set of instructions for developers about what works to help us learn. 

It is also somehow the first public repo I’ve ever shared and everybody was pretty nice about it. Just like that first preprint so many years ago on a scientific preprint server, I felt a sense of deep and profound disorientation around reaching my hand out expecting to hit a wall, and finding it malleable. I read a lot of papers in order to write it, but I was amazed at how well codification via a Skill translated the principles of papers into help without you having to read those papers. I tried it again and again and again, and sent it to critical friends. I had my very skeptical teaching professor wife try it. And it was still good, so I shared it.

And then something wild happened: people started using it, and because I encouraged it they started sending me a lot of messages about it. I even worked through pull requests! The Skill grew; we added choices and repo orienting guidance. It’s been used by probably more real-world people than ever read my HCI papers about peer to peer learning, certainly seen more usage in their real working days. I share this not as self-aggrandizement but as celebration of you, developers who want to learn, people who want to build their expertise even while they rise to the experimentation with AI. I got away from social media outrage channels for good long jam sessions with people about their actual learning in this moment. I felt free, because it is what it says on the tin. I did not make this Skill with the idea that the underlying technology implementing it was always going to be objectively right; I made this Skill with the orienting idea that you could bring meaning to how you interpreted what it gave you.

Calibration, Cognitive Science, and Learning with AI

In Part 1 of this series I wrote about the overarching goals of metacognition and a few techniques you can apply to your agentic coding.

The Learning-Opportunities Skill codifies a number of those techniques more thoroughly, especially using self-testing as an orienting principle for active learning. I thought this was a good place to start because there’s strong evidence that as a general design pattern to center on if you care about expertise and learning, you should care about calibration. Giving people the power to do better and more informative self-experiments about their learning measurably improves achievement and performance outcomes. Overconfidence can actually sabotage your learning, and self-testing is a pretty ruthless way to puncture overconfidence. Just forcing yourself to explain how and why something works can alert you to a calibration problem. 

To comfortingly remind us that we grappling with AI tooling in software development are not the first people to worry about learner calibration and poor strategies for comprehension, here is the introductory paragraph to a classic educational psychology paper by Zimmerman with whopping tens of thousands of citations:

Not gonna lie, I find Zimmerman a little harsh on hypothetical Tracy in this opener and a little bit in need of a culturally sensitive education perspective, and on math specifically, well, on our podcast I have recently done a research deep dive into the causal arguments we should make about where that comes from. But the core ideas in the paper remain helpful even if we had to wait about fifteen more years for gender-biased environment explanations of math attribution errors to come into the picture (sneering at teenagers for liking music videos never seems to go out of fashion).

These core ideas include: learners who are self-regulated:
set goals in accordance with their context,
select strategies that are likely to get them to those goals, and
evaluate outcomes.

This feels trite and sounds simple, but it’s hard to do in the first place and even harder to build consistent habits around. But these good habits predict outcomes. Self-regulated learners pay attention to what happens to them when they’re in “performance phase,” and conduct self-observations to challenge their assumptions about how good their strategies are. Experts seem to use self-regulation more than non-experts during their practice, too.

In fact, self-regulation can even compensate for individual differences in how easily learning comes to us. Self-regulated learners create this compensatory strategy by effectively monitoring their own comprehension. I have observed this in software teams: software teams with strong documentation practices, for instance, might highlight common misconceptions or mistakes that newcomers to their codebase make, which also serves to set a group norm that we will have misconceptions and you should pay attention to them. Expert developers self-monitor and when I ask them about it, have described myriad ways they monitor and switch strategies reasoning about what’s happening in a codebase or across a complex project, even if they wouldn’t use academic language for it. These strategies are often attributed to raw brainpower, but I hear them differently with behavioral science in my ear: I hear a lot of ways people have learned to transcend their individual limitations with strategies.

Here is a fun compensation strategy example in a completely different context: artists employ a wide range of strategies to compensate for limitations in visual perception; we literally can learn to see differently to draw. 

But self-regulation approaches aren’t just available to experts. I am going to say something controversial here, imho junior developers are surprisingly often more in tune with monitoring the metacognitive demands on their technical work than more senior counterparts than we might like to admit, who have sometimes been selected and promoted by bad hiring and poor technical cultures because they were at the right time and place to get lifted based on being fluent in a very niche thing they don’t even really have to do much anymore, and are not, themselves, particularly stellar examples of good cognitive work habits. 

Junior developers coming up now haven’t had that luxury: in the words of Cate Huston they’ve had to be Toronto raccoons. They have fluid intelligence, adaptive intuitions, and a lot of cognitive flexibility. They may not have all the content knowledge, but I have a strong working conviction that we need to listen to and learn from junior developers rather than just act like they are those impressionable wax tablets. I dislike the many pronouncements I see that say junior developers cannot effectively problem-solve with AI, and should be barred from the tools. Everything is changing and they’re already grappling with that; it feels remarkably cruel to say that junior developers shouldn’t be thinking about what they see every single peer in the industry thinking about. I am not convinced that the strategies of the past are always the right ones for the future. I need to do more thinking about this, but there's a glimpse into where I'm going.

Interviewing people about the Learning-Opportunities Skill

I decided to conduct interviews with folks who have used my Learning-Opportunities Skill. It was my hope that people would see it as an offer: you can wrest this agentic coding workflow into something that cares about your learning.

And across texts, emails, and a handful of qualitative interviews I have conducted with developers over the past month, I heard a lot of stories about learning. Of course, this is a self-selected group of people organized around my side passion project, not a census of developer behavior. But weaving together these anecdotes with self-regulated learning literature in mind presented a great contrast to the bluster on social media I have been trying to get past.

I have no idea how to communicate the output of this strange passion project, which is still in progress, but I wanted to give you an early access peek at it. I will just write it up here, casually, in my own voice and with some research notes about why I think it works. I’ve sketched out a few nascent themes around the success stories in people’s learning moments with AI: Enhancing Timing, Enhancing Goals, Enhancing Self-Efficacy, and Enhancing Element Interactivity. Each one of these feels like a rich barely-tapped vein for learning-with-AI design interactions.

A) Enhancing Timing

Four of my interviewees explicitly described learning to recognize the cadence of burnout in agentic coding, and how it would creep up on them. They successfully experimented with the time element of the Learning-Opportunities to intervene on their own over-production. One used the auto hook contributed by Ed Roppel to fire it at every commit, which they found “slowed me down in a meaningful way.” Another did Learning-Opportunities at the beginning of the day, which they found “useful for a reminder that I’m not just about lines of code.” Even though these are very different timing strategies, they shared the common insight that they needed a structured way to identify the crucial times that Zimmerman describes. Multiple folks besides this have described how my initial idea to experiment with a model-based trigger (e.g., instructions like “suggest a learning exercise immediately after completing major architectural work”) worked well for them.

This doesn’t surprise me because in order for people to really use them to change their behaviors, behavioral interventions are almost always dependent on being deployed at the right time. You need to offer a pattern that works not just in the abstract, but for when people are reminded of it. 

A potential design pattern for learning with AI: taking advantage of personalized workflows to inject more reflection and active learning into the moment when a user is most open to it, and most able to engage deeply with it.

B) Enhancing Goals

I wrote a second skill we haven’t even talked about yet: Learning-Goal. This was another experiment in helping people structure pre-planning before learning, and it is a real-life translation of an intervention exercise that I've actually tested in the field before with empirically measured results in people's persistence to learn. We think we know our goals, but most of us don't have clear plans for how to stick with them. Many people struggle with navigating the moment that an aspirational goal hits real obstacles. This skill uses MCII, a technique to once again force you to do more work: this time in committing to a battle plan for your most common obstacles.

I’ve only chatted with a few folks who used this Skill. But these developers shared unsurprising obstacles to me, all very understandable. Some emotional and psychological (“I doubt my boss cares if I learn,” “my organization won’t value this time”, “I don’t want to look like one of those dumb vibe coders to my team”), and some quite practical (“I struggle to prioritize this personal development over the output that’s going to really count,” “I feel exhausted on a friday, which is usually the time I have free time actually blocked for learning”). Every person I talked to changed the specific details of their goal after using the Skill, typically to make it smaller, more tangible, and (I hope) more resilient against their real obstacles. They all also expressed surprise at realizing that their previous learning goals were so vague. 

People have already been exploring the possibilities of AI interviewing the user when it comes to project planning, but I’m pleased that Learning-Goal showed a few folks we actually know a lot about why learners struggle to make good plans. 

A potential design pattern for learning with AI: taking advantage of the model’s ability to perform a structured interview and personalize multi-step interactive interventions, drawing from validated behavioral interventions.

C) Enhancing Self-Efficacy 

Learning-Opportunities can be incorrect. I was delighted to hear hints of the Sometimes-Wrong-Bot experiment emerge in people’s personal experiments. Several Learning-Opportunities Skill users described detecting subtle errors in Claude’s generated explanations and attributions after they went through a few exercises. But they were able to triage those errors and, as one developer told me, “It made me remember how much I really have learned about this language.” Self-efficacy boost achieved!

One of the reasons to become a self-regulated learner is that it helps you build more sustainable self-efficacy, i.e., your conviction that you will be able to achieve things and that momentary mistakes and upsets do not send a signal that you are incapable. Self-regulated learning strategies are more work up front, but they tune you into the subtle changes in yourself that you otherwise might dismiss, and in my experience, a lot of software folks are so brutally hard on themselves and do in fact miss this. There's a lot more to be probed here about model sychophany defaults and adversarial use of AI; I think challenging people's negative self-concepts is an interesting case that I haven't heard enough people talk about though, and it's a case that psychology has explored pretty thoroughly and has some good ideas about.

A potential design pattern for learning with AI: identifying and supportively presenting proof against users’ own misconceptions about their capability, their potential, and the possibilities of success.

D) Enhancing Our Ability to Deal with Element Interactivity 

You have probably heard the term cognitive load used about developers and their problem-solving. An originally fairly niche term that achieved escape velocity into the broader industry terminology, I want to give you a far less popular but deeply useful idea that came up in my interviews: element interactivity and how it helps us to think about transcending the limits of our individual moments of problem-solving to handle task complexity. 

A 2023 paper with Sweller described element interactivity this way: 

I bet you can immediately imagine many types of coding tasks that involve high element interactivity. Developers in my interviews described this, too: when self-testing on what they thought was a small concept pulled a thread on consequences they hadn’t thought about, or when they needed to navigate many different pieces of technology before the whole become clear and a problem become tractable. There are types of work that experts can do simply because they have a richer ability to handle element interactivity. I think it is really interesting to ask where AI can provide support here to technical workers.

The fundamental logic of cognitive load depends on recognizing that we are always problem-solving and learning with a limited working memory. This is why more efficiently retrieving stored information can improve our moment-by-moment working problem-solving – because long-term memory doesn’t consume working memory in the same way. Developers who have large amounts of knowledge and expert schemas to apply have cognitive architecture that helps them more efficiently retrieve the right information for the moment. 

Without thinking through this scaffolding and giving yourself aids for it, lord, are you putting a lot on yourself. This probably won’t surprise you: developers in my interviews have pointed out that in their first months using Claude Code in a more “raw” way, counting on themselves to manage and monitor every output for sustained hours, they have felt a creeping sense of fatigue. One person called this “over-monitoring,” and multiple people have used the metaphor of “becoming a manager.” (I’ll say another controversial thing, I don’t like this manager analogy at all. I don’t think that represents the whole possible spectrum of the cognitive science of agentic coding. But whatever) 

Tools have the potential to work with us as attention directing aids and as retrieval aids. Already, the Learning-Opportunities Skill sits within the context of your currently working files, which reduces extraneous load by embedding your self-testing in your actual workflow. Something I love about the Orient skill that Michael Mullarkey contributed is that it draws from research on codebase comprehension, in the same philosophy as the parent Learning-Opportunities Skill, to give people personalized starting places for making their effort moments higher value. Orient analyzes the structure and patterns in a codebase and suggests where a developer should focus their learning effort first.

If the particular context of the repo shapes what you should learn, we can pull out some useful features by running an orient session and giving your Learning-Opportunities an opinionated place for you to start self-testing, rather than an arbitrary one. This shaping of the potential learning context is lightweight and speculative (as far as I’ve found, top-down forcing too many bells and whistles penalizes model behavior), but it helped us think about the interactions between [user] * [learning-opportunities design] * [codebase/repo], each of which contribute meaningfully to a moment of learning. To get better at complex tasks, you can build deliberate practice strategies in the context of your actual tasks, and the structures that constrain your tasks can be made more visible to the AI agent to better its retrieval of learning-relevant content.

Michael and I recorded a conversation our experiences as two psychologists thinking together about this:

This is a more complex area, but I offer it because I think the story is a little bit more complicated than just, "are we cognitive offloading, or cognitively working hard." Sometimes we're cognitively working differently.

One junior developer who reached out to me shared that they had the uncomfortable experience of learning that the documentation around a complex piece of the codebase, originally authored by a very influential engineer in their org, was “quite wrong in a few big ways that everyone had missed.” They originally were certain it was their error, and ran Learning-Opportunities over that part of their files until they had built up enough different concept and construct knowledge to become certain that they were, in fact, right. This was strategically using scaffolding to gradually assemble a whole picture and do some advanced problem-solving, and I think it is an important reminder that real-world work can surprise us.

I think there are multiple design patterns in this one for learning with AI: 

• Asking the model to do pre-work that aligns to the specificity of what context shapes the learners’ goals 
• Surfacing organizational choices across repos and projects may reveal tacit practices for learners 
• Thinking about sequences of interactions that allow more learners to tackle high element interactivity tasks they previously weren't able to do

Finally, one of the things I think we who have been in tech longer should do is hold ourselves to a standard of caring about what is happening to juniors. Was the Learning-Opportunities only useful to experts? It doesn’t seem like it. As a sign of its use in this regard, I was really happy to hear that the Learning-Opportunities Skill is being used in more than five programming classrooms, as an example of an interaction design with AI in coding, and I’ve heard from more than a dozen junior developers about it. I also think that seeing professional developers actually share that they still care about learning was motivating to the earlier career folks I heard from, too. If our goal is to help our junior colleagues integrate into organizational goals to use this tooling, we also need to listen to them about their challenges and friction points, and believe in their potential for learning.  

Newcomers to our technical infrastructure are capable of remarkable problem-solving, and because they are less bogged down by pre-existing assumptions and stereotypes and attribution errors about why we do it this way, empowering them to go on their own learning journeys is central to innovation. I do not believe we should think about is as merely a thing we put up with, in the service of eventually getting experts. I believe it is a thing we need. Every developer in your organization should be treated as part of your thinking, problem-solving community. 

This can be hard to remember especially when the journey educational psychologists most classically pay most attention to is a clean linear story of moving from novice to expert. That linear journey from beginner to expert is pleasing to our minds. But in my work with software teams, I regularly hear developers tell me they have needed to start over again. To grapple with a new technology, to think about applying a previously obscure solution to a new context, and to orchestrate others’ innovations into novel and bespoke solutions, developers have always needed metacognitive strategies, and you can begin to apply these from the moment you are learning in a new area.

Instead of the clean linear beginner to expert journey, I would like to leave you on the provocation of the Sometimes-Wrong Bot. Encouraging active learning in our organizations might mean we must allow unexpected people to diagnose problems in unexpected ways: the project manager who was told they are “non-coding,” the junior who was told to “shut up and learn,” perhaps even the psychologist who said hm, I would like to work against these dehumanizing stereotypes about developers and use the pieces of educational psychology that I have found to be true, yet always hold these theories to account against the experiences of my friends in the real world. Perhaps there are new learning tools that can be built, yes, even out of the technology that people say is destroying learning, if you care hard enough about people's learning. Even on dark planets, knowledge from science coupled with letting yourself imagine the seemingly impossible, dissolves barriers.