Note: Hand over heart, I substantially drafted this post — including its core conceit — long before reading Josh Brake’s latest Substack post. No plagiarism here!

I have two sorts of problems with “AI” in general and Large Language Models (LLMs) in particular. One is the (infinitely ramifying) ethical problem. LLMs are built on deception. They are not human (and not “alive”), do not possess human cognitive faculties, and cannot “know” anything in the ordinary human sense of that word, and yet their model is built on — after vacuuming up an enormous amount of human-created linguistic “content” — mimicking human cognition and knowledge to such an effective degree that you spend all your time relying on GPT-4o or what have you, rather than other human beings. I take this to be a fairly straightforward form of deception, and because of the incommensurability of truth and falsehood, this first problem to be the most fundamental. What does constantly being deceived, and constantly self-deceiving, do to a human being? In what ways are we damaging, and might further damage, ourselves by using such a false tool? (See also: Mammon.) But that’s for another post.

The second is the education problem. Here my fears are well illustrated by an analogy from James C. Scott’s Seeing Like a State:

The principles of scientific forestry [TC: planting a single “crop,” in evenly-spaced rectangular grids, in place of the old ecologically diverse forests] were applied as rigorously as was practicable to most large German forests throughout much of the nineteenth century. The Norway spruce… became the bread-and-butter tree of commercial forestry. Originally [it] was seen as a restoration crop that might revive overexploited mixed forests, but the commercial profits from the first rotation were so stunning that there was little effort to return to mixed forests… Diverse old-growth forests, about three-fourths of which were broadleaf (deciduous) species, were replaced by largely coniferous forests in which Norway spruce or Scotch pine were the dominant or often only species. In the short run, this experiment in the radical simplification of the forest to a single commodity was a resounding success… the negative biological and ultimately commercial consequences of the stripped-down forest became painfully obvious only after the second rotation of conifers had been planted… An exceptionally complex process involving soil building, nutrient uptake, and symbiotic relations among fungi, insects, mammals, and flora—which were, and still are, not entirely understood—was apparently disrupted, with serious consequences. Most of these consequences can be traced to the radical simplicity of the scientific forest. … Apparently the first rotation of Norway spruce had grown exceptionally well in large part because it was living off (or mining) the long-accumulated soil capital of the diverse old-growth forest that it had replaced. Once that capital was depleted, the steep decline in growth rates began.

To apply the analogy: Maybe, just maybe, you can implement LLMs without too many problems in the first generation, among a population of adults who have already been educated. Their values have already been formed; they have already learned to read and write and think critically. (This already concedes far too much to the “AI” boosters, but for the sake of the argument, we will not pause overlong.) Perhaps they really could achieve the stunning productivity growth which we are constantly promised (though so far the results don’t seem great!). But even if that were true, can you expect those gains in the second generation, among children who are still being educated? Or would you rather expect systemic failure to ever form values, to learn critical thinking, essential reading comprehension, and basic writing skills? The adults who received pre-LLM educations have an existing store of cognitive and intellectual capital on which to draw as they encounter and learn to use LLMs. But children who never experience education without LLMs will never have the chance to develop that capital.

Furthermore, the broader environment in which this “first rotation” is encountering LLMs is not remotely the same as that in which the “second rotation” will encounter them. Indeed, the environments are being treated as if they are the same, when they should be different. My local school district is now integrating “AI” into primary and secondary education, because “universities and employers will expect AI literacy” — what tool is easier to learn to use than a natural language chatbot? Now, the workplace may appropriately demand certain kinds of efficiency from adult workers, and LLMs may just prove their usefulness in such cases (though in my view the jury is still out). Education, by contrast, should be inefficient, frictional, resistive. The mind is like a muscle: in order to grow, it must be repeatedly stretched to the limits of its capacity. The LLM chatbot is the ultimate anti-friction, super-efficient (except in, you know, water and energy) machine, which promises that you will never encounter resistance ever again; with the new “reasoning” modules, you’ll never have to think for yourself again. The implications for education hardly need to be spelled out.

Scott continues:

As pioneers in scientific forestry, the Germans also became pioneers in recognizing and attempting to remedy many of its undesirable consequences. To this end, they invented the science of what they called “forest hygiene.” In place of hollow trees that had been home to woodpeckers, owls, and other tree-nesting birds, the foresters provided specially designed boxes. Ant colonies were artificially raised and implanted in the forest, their nests tended by local schoolchildren. Several species of spiders, which had disappeared from the monocropped forest, were reintroduced. What is striking about these endeavors is that they are attempts to work around an impoverished habitat still planted with a single species of conifers for production purposes. In this case, “restoration forestry” attempted with mixed results to create a virtual ecology, while denying its chief sustaining condition: diversity.

I leave the resonances between this virtualized ecology and the state of education today as a trivial exercise for the reader.

(Scott’s remarks here of course have many parallels. Ivan Illich makes a remarkably analogous argument, with respect to medicine, in the opening of Tools for Conviviality; and Michael Polanyi offers a structurally similar observation about the Enlightenment “critical movement” that sought to banish belief from knowledge: “its incandescence had fed on the combustion of the Christian heritage in the oxygen of Greek rationalism, and when this fuel was exhausted the critical framework itself burnt away.")