When your AI tool produces an output that feels “technically right, but missing the point,” that experience is usually tracking something real. The tool is not broken. It’s not hallucinating. It is doing exactly what it was trained to do. What you’re bumping against is the edge of what it’s been trained to see--and we don’t yet have a good vocabulary for that experience, which means we can’t work with it deliberately.
I have been writing this essay about those limits with the thing that has them.[1] The tool that helped me develop the argument--that pushed back when my reasoning was loose, that helped translate an anthropological concept into engineering terms--is also the thing the argument is about. In the course of the collaboration, I kept noticing the edges of the tool’s attention: the patterns of what it would foreground and what it consistently didn’t, the kinds of knowing that required deliberate prompting to reach, the places where I’d ask something and get back an answer that was accurate but slightly adjacent to what I was pointing at. The collaboration produced something real. It also showed me, in practice, what the argument was trying to describe in the abstract.
I’m offering this not as a confessional move, but because the positioning matters. I am not a machine learning researcher. I have no standing to evaluate the engineering questions that occupy most of the discourse about AI capabilities, alignment, or safety. What I do have is twenty-five years of unpacking what happens when people recognize that the attention patterns their professional training gave them aren’t adequate for the questions they most need to answer--and a hunch that the same structural dynamic is happening, right now, at scale, in just about every organization and individual deploying AI tools across consequential work.
This essay is an attempt to name what’s happening — to give the experience a structure, if not yet a full account.
What Trained Attention Actually Means
Professional formation does something more radical than teaching its practitioners content or technique: it teaches them what to notice. A clinician learns to map presentations toward diagnostic categories, to trace causal chains between symptoms and treatable conditions. A wilderness guide learns to read terrain and weather as continuous real-time data. A cultural anthropologist--my own training--learns to find what is systematic in apparently random variation, to situate the individual in structural context, to see the institution behind the behavior and the behavior inside the institution. In each case, the training is successful precisely to the extent that the practitioner stops having to choose to attend this way and simply finds herself doing it: the perception becomes reflexive, automatic, reliable.
That automatic reliability is the training’s gift. It’s also what makes the training invisible. The attention patterns that become habitual through professional practice don’t feel like choices anymore--they feel like ‘just how things are’. A clinician doesn’t (always) experience herself as choosing to see her client through a diagnostic lens. She just… sees.[2]An anthropologist doesn’t consciously decide to look for structural patterns. He finds them, and then has to work, often uncomfortably, to notice the structural patterns in his own noticing.
What professional training produces, across every field I have examined, is what I’ve been calling a horizon. The valence of the term matters--a horizon is not a wall someone built; it is a sight-line from a position: the ridgeline visible to the person standing in the valley, which shifts when the person moves, is simultaneously a real feature of the landscape and entirely a function of where you’re standing. Professional training creates horizons by positioning practitioners in specific ways. From that position, certain things are visible and actionable. Other things fall on the other side of the ridge--not because they don’t exist, but because nothing in the positioning makes them available from here.
The chain works like this: professional positioning trains attention; trained attention creates perceptual horizons; and those horizons constrain which futures feel available--not as abstract logical possibilities, but as the range of things that seem plausibly imaginable, workable, worth trying, from where you’re standing. Different positions make different futures conceivable. This isn’t a limitation of intelligence or intention; it’s the structural consequence of how trained perception works.
I’ve watched this chain operate, consistently, in every professional context to which I’ve been privy over the past twenty-five years. What took me a minute to recognize is that the same chain operates in every AI tool I’ve used.
The Same Chain, Running Differently
A large language model is trained on an enormous corpus of human-generated text. That training doesn’t just teach the model content; it teaches the model where to look--which patterns of input should produce which patterns of output, which signals deserve foreground and which can safely recede. The training produces cultivated dispositions toward certain kinds of information and away from others: patterns of relevance that get baked in through training and then further reinforced through fine-tuning, instruction-following, system prompts, and deployment context.
Model training does the work of positioning an agent--establishing, at the level of weights and patterns, what counts as relevant, what counts as evidence, and which patterns of input deserve close attention and which can be safely backgrounded. System prompts and retrieval configuration train its attention further. That trained attention creates perceptual horizons: what the tool can and cannot find salient in what you’re giving it--not because it lacks the processing capacity to register certain kinds of information, but because nothing in its training and configuration has made that information foreground.
Those horizons then constrain what the tool generates. Not absolutely--you can sometimes prompt your way past them--but in the way that all trained attention works: as defaults; as the shape of the tool’s practice-of-noticing, when it’s operating without deliberate counter-pressure from outside.
The disanalogy matters, and I don’t want to paper over it. The parallel between professional formation in humans and training in AI systems is functional, not architectural: the mechanism by which an AI model acquires its perceptual dispositions is categorically different from the embodied, community-of-practice process by which a human practitioner learns to see.[3] What the two share is the structural consequence--trained dispositions that create horizons--not the underlying process. Whether that means the trained-attention framework is transitional for current-generation AI, or a permanent feature of any system trained on human data, is genuinely open. What I can say with confidence is that for current systems, deployed as they actually are, the perceptual constraints are real, and their effects are consistent and recognizable.
The Experience You’ve Already Had
Here is what the edge of a tool’s horizon feels like from the inside.
You ask your AI about an organizational problem (something specific, something you’ve been thinking about for weeks), and what comes back is accurate, fluent, and responsive to some version of what you asked, but slightly adjacent to what you were actually pointing at. The tool saw the process-improvement question inside your organizational question. That’s what it answered. What you were really asking--about what it means that a team has stopped trusting its leadership, or about whether the organization should even continue to exist in its current form--that fell on the other side of something.
You try rephrasing. You add context. Sometimes it helps; sometimes the tool circles back to the same kind of answer, because the rephrasing moved around the surface of what’s askable from here without actually relocating the tool’s position. The answer is still not ‘wrong’. It’s just attending to a slice of what you gave it, and treating that slice as if it were the whole.
Or: you use an AI tool to help think through a difficult decision. The tool is helpful with certain aspects of it--the ones that can be organized as tradeoffs, compared on dimensions, subjected to structured analysis. The aspects that involve genuine irreducibles--what you owe someone, what you can live with, what kind of person you’re becoming--tend to either get converted into a version of the tradeoff question, or get handed back to you as a reminder that these are ultimately your values. Both responses are technically correct. Neither quite engages what you were pointing at.
This is not a failure of the tool. It is the tool operating within its trained attention. The management consulting literature on which much organizational analysis was trained has its own horizon--a well-documented one, which tends to make process and structure visible, while making culture, meaning, and relational trust harder to see. The tool learned from that literature. The tool has that horizon. When you ask an organizational question, you’re pointing at something in a landscape the tool sees through a particular lens, and the parts of that landscape outside the lens will consistently not show up, regardless of how you phrase the question, because the phrasing is not the problem.
What Follows From This
It does not follow that you should use AI tools less, or mistrust them, or wish they were somehow different. Every perceiver is positioned somewhere.
The trained attention that creates horizons is also what makes expertise systematic and reliable. The point is not to eliminate trained attention--which would eliminate expertise--but to develop the meta-awareness that lets you work with it deliberately.
What that looks like in practice: learning to notice the edges of your tool’s attention. Not just what it produces, but where it consistently doesn’t quite reach--the questions it answers slightly adjacent to what you asked; the aspects of a problem it keeps not foregrounding; the places where you find yourself adding context the tool should have noticed, but didn’t. That noticing is information. It tells you something about the shape of the tool’s horizon, which tells you something about when to lean into what the tool does well and when to supply the attention it can’t.
It also looks like taking responsibility for your own horizon--which is the piece of this that gets almost no attention in the discourse. You are also attending from somewhere specific. Your trained attention shapes what you can ask for, what you can recognize in what the tool produces, what you can hold the tool accountable for. A clinician trained to see individual pathology will be better at evaluating whether an AI output is diagnostically sound than at evaluating whether the output treats the person’s suffering as something worth engaging on its own terms--because the first evaluation is inside her horizon and the second requires getting outside it. Two trained-attention systems in interaction, neither one with a clear view of what the other isn’t seeing: this is what the interface between humans and AI tools actually looks like, and it’s not a problem that better prompting alone can fix.
I want to be careful about what I’m claiming. I am not suggesting that AI systems are conscious, or that the trained perceptual dispositions I’ve been describing are equivalent to human expertise, professional judgment, or values. The claim is narrower and more immediately tractable: that what an AI system attends to is not neutral, that its training creates real perceptual horizons, and that those horizons have practical consequences for the people using the tool--consequences that become more significant as the stakes increase, and that can only be managed intelligently once they can be named.
David Foster Wallace told a graduating class in 2005 that “everybody worships. The only choice we get is what to worship.” He was talking about human consciousness: the way that habitually directed attention structures both the experience of reality, and the freedom available to those who notice what they’re doing and choose accordingly. The claim is portable. Every AI tool “worships” something, in the functional sense that matters here: the patterns of relevance its training and configuration have made habitual, the foregrounds it returns to reliably, the backgrounds it maintains as noise.
The tool is not neutral. It has been positioned, trained, and deployed from somewhere specific--and that somewhere shapes what it can offer you and what it will, structurally, not be able to see.
What is currently absent from most AI deployment--and from most of the discourse about it--is any structure for making that worship visible. We evaluate outputs. We have no shared vocabulary for auditing the perceptual configuration that produced them: for asking not just what did the tool give me, but what was it configured to attend to before it responded, and what fell, therefore, on the other side of its horizon.
This is what I have spent twenty-five years exploring among human practitioners: how trained attention becomes visible as trained--as a function of position rather than a feature of reality--and what becomes possible when it does. In every professional context I’ve been near, the practitioners who developed the most sophisticated and effective work were the ones who learned to see their own horizons--not to transcend them, which is impossible, but to work with them deliberately: to know where they were standing, and to seek out modes of attending that could reach what their primary training could not.
Two trained-attention systems in interaction, neither one with a clear view of what the other can’t see. Naming that doesn’t solve it… but it changes what you're doing when you sit down to work—from trying to fix the output, to trying to read the horizon.
That's the beginning of the thing.
[1] Re: “writing… with”--both the verb and the preposition are, for obvious reasons, tricky here; there’s an account to offer about how the four-dimensional-white-board quality of the thing has made it possible to grapple dialogically with my own higher-order questions in a way that feels new (which I’d comfortably call ‘thinking with’).
[2] I want to be careful, here, about suggesting overdetermination; what I’m actually interested in is the (dis)embodied texture of the experience.
[3] The concept of “professional vision”--that professional formation creates socially organized ways of seeing that make certain features of the environment salient and others invisible--was developed by Charles Goodwin (“Professional Vision,” American Anthropologist 96:3, 1994). The framework here draws on that insight, while extending it in two directions Goodwin’s work did not address: applying it to an AI system rather than a human practitioner, and examining what happens at the interface when a human practitioner’s trained attention meets an AI tool’s trained attention.
You are accurately describing the experience the almost all people I know are confirming.
I wrote a paper about an even deeper issue that many large corpus users are already experiencing — a real time slow mo degradation of corpus fidelity, requiring ever more frequent refreshes with diminishing returns and ever increasing drift.
“Is Truth Too Expensive for AI to Deliver?”:
https://craigcshelton.substack.com/p/is-truth-too-expensive-for-ai-to?r=h2o3&utm_medium=ios
“Technically accurate but slightly adjacent…” words I use at least four times a week with leaders who continue to want AI generated reports, updates, and decisions. They lack nuance, context and any sense of reality on the ground. Factual? Partly. Fixated on portraying certainty that isn’t actually there. Guaranteed.