From the latest story about how LLMs are destroying higher education:

Lee explained to me that by showing the world AI could be used to cheat during a remote job interview, he had pushed the tech industry to evolve the same way AI was forcing higher education to evolve. “Every technological innovation has caused humanity to sit back and think about what work is actually useful,” he said. “There might have been people complaining about machinery replacing blacksmiths in, like, the 1600s or 1800s, but now it’s just accepted that it’s useless to learn how to blacksmith.”

If only this 19-year-old Columbia suspendee had, at a minimum, done what he apparently did for every assignment and asked ChatGPT for information: “When were machines developed that could assist in metalworking, and have they made the crafts of smithing and metalwork obsolete?” But even asking that question — writing that prompt — would have required a measure of historical literacy, nay, a sliver of interest in history at all.

This (now former) student is an especially egregious offender, worthy indeed of becoming the framing device in a breathless New York Magazine story, but there is nothing remarkable about what he represents: it is the characteristic disease of “move fast and break things” culture. All that is prized is “innovation,” because innovation makes money fast and lets the innovator get out before he (and it is usually a he) is held accountable to clean up the wreckage. The destroyers do not understand, and do not want to understand, the things they are out to destroy.

Addendum: James Scott, of course, had the destroyers' number.

Note: Hand over heart, I substantially drafted this post — including its core conceit — long before reading Josh Brake’s latest Substack post. No plagiarism here!

I have two sorts of problems with “AI” in general and Large Language Models (LLMs) in particular. One is the (infinitely ramifying) ethical problem. LLMs are built on deception. They are not human (and not “alive”), do not possess human cognitive faculties, and cannot “know” anything in the ordinary human sense of that word, and yet their model is built on — after vacuuming up an enormous amount of human-created linguistic “content” — mimicking human cognition and knowledge to such an effective degree that you spend all your time relying on GPT-4o or what have you, rather than other human beings. I take this to be a fairly straightforward form of deception, and because of the incommensurability of truth and falsehood, this first problem to be the most fundamental. What does constantly being deceived, and constantly self-deceiving, do to a human being? In what ways are we damaging, and might further damage, ourselves by using such a false tool? (See also: Mammon.) But that’s for another post.

The second is the education problem. Here my fears are well illustrated by an analogy from James C. Scott’s Seeing Like a State:

The principles of scientific forestry [TC: planting a single “crop,” in evenly-spaced rectangular grids, in place of the old ecologically diverse forests] were applied as rigorously as was practicable to most large German forests throughout much of the nineteenth century. The Norway spruce… became the bread-and-butter tree of commercial forestry. Originally [it] was seen as a restoration crop that might revive overexploited mixed forests, but the commercial profits from the first rotation were so stunning that there was little effort to return to mixed forests… Diverse old-growth forests, about three-fourths of which were broadleaf (deciduous) species, were replaced by largely coniferous forests in which Norway spruce or Scotch pine were the dominant or often only species. In the short run, this experiment in the radical simplification of the forest to a single commodity was a resounding success… the negative biological and ultimately commercial consequences of the stripped-down forest became painfully obvious only after the second rotation of conifers had been planted… An exceptionally complex process involving soil building, nutrient uptake, and symbiotic relations among fungi, insects, mammals, and flora—which were, and still are, not entirely understood—was apparently disrupted, with serious consequences. Most of these consequences can be traced to the radical simplicity of the scientific forest. … Apparently the first rotation of Norway spruce had grown exceptionally well in large part because it was living off (or mining) the long-accumulated soil capital of the diverse old-growth forest that it had replaced. Once that capital was depleted, the steep decline in growth rates began.

To apply the analogy: Maybe, just maybe, you can implement LLMs without too many problems in the first generation, among a population of adults who have already been educated. Their values have already been formed; they have already learned to read and write and think critically. (This already concedes far too much to the “AI” boosters, but for the sake of the argument, we will not pause overlong.) Perhaps they really could achieve the stunning productivity growth which we are constantly promised (though so far the results don’t seem great!). But even if that were true, can you expect those gains in the second generation, among children who are still being educated? Or would you rather expect systemic failure to ever form values, to learn critical thinking, essential reading comprehension, and basic writing skills? The adults who received pre-LLM educations have an existing store of cognitive and intellectual capital on which to draw as they encounter and learn to use LLMs. But children who never experience education without LLMs will never have the chance to develop that capital.

Furthermore, the broader environment in which this “first rotation” is encountering LLMs is not remotely the same as that in which the “second rotation” will encounter them. Indeed, the environments are being treated as if they are the same, when they should be different. My local school district is now integrating “AI” into primary and secondary education, because “universities and employers will expect AI literacy” — what tool is easier to learn to use than a natural language chatbot? Now, the workplace may appropriately demand certain kinds of efficiency from adult workers, and LLMs may just prove their usefulness in such cases (though in my view the jury is still out). Education, by contrast, should be inefficient, frictional, resistive. The mind is like a muscle: in order to grow, it must be repeatedly stretched to the limits of its capacity. The LLM chatbot is the ultimate anti-friction, super-efficient (except in, you know, water and energy) machine, which promises that you will never encounter resistance ever again; with the new “reasoning” modules, you’ll never have to think for yourself again. The implications for education hardly need to be spelled out.

Scott continues:

As pioneers in scientific forestry, the Germans also became pioneers in recognizing and attempting to remedy many of its undesirable consequences. To this end, they invented the science of what they called “forest hygiene.” In place of hollow trees that had been home to woodpeckers, owls, and other tree-nesting birds, the foresters provided specially designed boxes. Ant colonies were artificially raised and implanted in the forest, their nests tended by local schoolchildren. Several species of spiders, which had disappeared from the monocropped forest, were reintroduced. What is striking about these endeavors is that they are attempts to work around an impoverished habitat still planted with a single species of conifers for production purposes. In this case, “restoration forestry” attempted with mixed results to create a virtual ecology, while denying its chief sustaining condition: diversity.

I leave the resonances between this virtualized ecology and the state of education today as a trivial exercise for the reader.

(Scott’s remarks here of course have many parallels. Ivan Illich makes a remarkably analogous argument, with respect to medicine, in the opening of Tools for Conviviality; and Michael Polanyi offers a structurally similar observation about the Enlightenment “critical movement” that sought to banish belief from knowledge: “its incandescence had fed on the combustion of the Christian heritage in the oxygen of Greek rationalism, and when this fuel was exhausted the critical framework itself burnt away.")

1. “Artificial general intelligence,” defined as “a computer able to do any cognitive task a human can do” — as envisioned for example in this new work of science fiction — is computationally impossible to achieve.

2. This is because “intelligence” — in the sense of “normal human intelligence,” which is presupposed by the above definition of “AGI” — is a) impossible to fully and simultaneously articulate (hereon inarticulable) and b) non-deterministic, and therefore in at least two senses strictly non-computable.

3. The inarticulability of intelligence has (at the very least) to do with its embodied and relational aspects. “Mind” is neither identical with nor even co-extensive with “brain activity”; rather, “mind” is (to crib from Dan Siegel’s definition) is an embodied and relational process. Emotion in particular seems, as far as the causality can be determined, to be body-first, brain-second, such that it is only articulable after the fact (and in a way that changes the emotional experience). Michael Polanyi’s great work demonstrates in a philosophical register what musicians, artists, and craftspeople have always known intuitively: that the “cognitive task” of playing an instrument or using a tool depends on integrating the instrument or tool into one’s bodily experience, in an inarticulable way. And relationship through interaction with other embodied minds is such a complex process, with so many emergent layers, that not only is it poorly theorized or modeled now, it may be impossible to exhaustively theorize or model — especially because it primarily seems to take place in and through the pre- and in-articulate dimensions of cognition.

4. Meanwhile, the non-determinism of intelligence has (at the very least) to do with quantum randomness effects in the brain, which at the mesoscale (the level at which daily human, and most complex organic, life takes place) emerge into relatively well-understood and predictable patterns, but at the nanoscale (the relevant level for a hypothetical deterministic model of cognition) are by definition impossible to predict, or even observe without altering them. I am unaware of any good reason to think the quantum effects in, say, an extremely large and inorganic GPU farm, would be interchangeable with or even meaningfully similar to those in a three-pound organic human neural system.

5. What is computationally possible, as far as I can tell, is a (relatively) high-fidelity simulation of one aspect of human cognition: the comparatively deterministic, hyper-articulated aspect of human cognition which Iain McGilchrist identifies as characteristic of the left hemisphere (hereon LH) of our brains (subject, of course, to obvious caveats from theses 2–4). Note: I am not saying, and I do not take McGilchrist to be saying, that a fully-computed model of the LH itself is possible; only that its characteristic thought-style can be simulated in high fidelity, precisely because that thought-style is comparatively deterministic and hyper-articulated.

6. In currently existing frontier Large Language Models (LLMs), I take it something like this has already been achieved. Commercially available LLMs are now (to use a technical term) pretty good at processing and reproducing both written and spoken natural language — albeit in such a sterile “voice” that it renders the phrase “natural language” almost meaningless — and quite good at analytically processing huge quantities of formally similar information. These are two of the characteristic specializations of LH cognition, and I expect the next generation of LLMs to be significantly better on both fronts. Notably, some of the persistent failure modes of LH cognition and of LLMs are startlingly similar: “hallucination” or fabrication of nonexistent supporting evidence, a predilection for lying or circumventing rules in order to achieve a desired result, an inability to attend to wholes at the expense of parts, and so forth.

7. Because much of contemporary Western life (as McGilchrist and others have extensively documented) is already organized to systematically advantage that aspect of human cognition, it is therefore no surprise or, in a sense, any remarkable accomplishment that frontier models now perform at the level of PhD students in solving advanced physics problems (albeit ones with solutions known to currently existing physics), or that some chatbots now “pass the Turing Test." This is the natural end result of reimagining science as “knowledge production” and credentialing scientists accordingly, or of technologically reducing the typical person’s normal experiences of and capacity for conversation to so great an extent that we now take what the LLMs offer to be “human” conversation. This — and all the attendant social/economic disruption (about which more below) — is all possible without “AGI” itself being computationally feasible.

8. The second strike against the possibility of “AGI” comes from limits in physical resources. Achievements in LLM development up to this point have been enabled by energy use, water depletion, and resource extraction on an already massive scale. The anticipated investments required for “AGI” (e.g., according to AI 2027, $2 quadrillion in new data centers over the next 10 years!!!) will require exponentially more energy, water, and mineral resources that we either simply do not have on this planet or cannot physically extract from it at the desired rate (unless we invent, say, cold fusion). This is to say nothing of the land required to build all of the new infrastructure. I therefore anticipate that “AI” development will, as a function of resource scarcity, fail to get anywhere close to the scale of investment theoretically required for “AGI.” This may only become clear to “AI” developers, however, after they have already inflicted genuinely ruinous and probably irreversible damage to the environment and to the communities that depend on it.

9. Considering all this, I find it probable that without ever achieving “artificial general intelligence” as imagined in science fiction, advances in “AI” over the next several years will make all but the top 1–5% of current “symbolic capitalists” functionally obsolete. This includes both high-status sectors such as consulting, finance, advertising, software development, law and legal services, etc., and lower-status (or at least lower-paying) sectors such as journalism, copywriting, teaching, administration, graphic design, the social sciences, etc. (Note that several of these lower-status professions are ones which the Internet revolution has already been destroying.) By “functionally obsolete” I mean that it will almost always be more cost-effective, and nearly as useful, to “employ” an “AI agent” for a task that previously required one to hire a human being.

10. Sectors that are symbolic-capitalism-adjacent but require long training in embodied skill — e.g., healthcare, the experimental sciences, mechanical engineering, war — will not be functionally obsoleted, at least not so thoroughly. An inorganic robot will never be able to perform skilled tasks in the real world with the same level of ability as a trained human being (see (3) above)… and “organic robots” capable of such skill would pretty much just be, well, lab-grown humans, with many of the same inefficiencies and time-delays as regular humans. (Only a conspiracy theorist would see current Silicon Valley investments in IVF, genetic selection and editing, and artificial wombs as an attempt to create the conditions of possibility for lab-grown humans… right???) But some current features of jobs in these sectors — the features, that is, which are most akin to “AI” core competencies — will be permanently outsourced to “AI agents.”

11. The “trades” and the “crafts,” on the other hand, will not become thoroughly automated, though they will be in various ways automation-directed and -augmented. Machine maintenance and repair, for instance: machine failure might be AI-diagnosable, but the intuitive skill necessary for actual repairs will remain the province of humans. To deal with water, you’ll always need a plumber. Reality has a surprising amount of detail, and fields like construction and mining will always require meaningful and skilled human attention to reckon with that detail. Agriculture represents an interesting test case: a field that is currently extremely mechanized, but as the lowest-skilled tier of human labor becomes (out of necessity) far cheaper to “buy,” one which may reabsorb much of that excess labor capacity. At the more humanistic end of the spectrum, traditional crafts might make a comeback of sorts (similar to the vinyl resurgence), and the performing arts will always be the province of human beings, though probably far fewer people will be performing artists in fifteen years than are right now; in both cases patronage will be the only economically viable model. For the ultra-wealthy, owning or sponsoring something evidently made only by humans will be a status symbol.

12. In sum: I believe we are headed neither for the existential-risk, civilization-ending disaster scenarios envisioned by the “AI Doomers,” nor for the golden era of peace and prosperity and universal basic income envisioned by the “AI optimists.” (Where, exactly, do the optimists think the value creation for UBI will come from in an era of mass human unemployment?) Rather, I suspect in the near-ish term we are headed for a poorer, less culturally vibrant, less highly educated world with much greater wealth inequality. This will be a world in which many more people, including some who might otherwise have been symbolic capitalists, work in various kinds of manual labor or “trades”: agriculture, mining, energy, construction, maintenance. Others will depend, one way or another, on the patronage of the few ultra-wealthy. The whole service-economy apparatus that depends on a large leisure class will be semi-permanently diminished in proportion. It might, in other words, look in certain ways remarkably like the period of transition into the Industrial Revolution.

Over the long run, I believe in the resilience of humanity, chiefly because I believe in the faithfulness of God to His wayward creatures. We will not be destroyed or superseded by a higher form of intelligence, nor will we manage to completely destroy ourselves. We are remarkably adaptable and creative: life always finds a way. But we will find that the remarkably widespread prosperity of the last few decades in particular and the last two centuries in general is not, once unlocked, a permanent and automatic feature of human existence. It has depended on our irretrievably consuming the planet’s resources at an ever-accelerating rate. What cannot go on indefinitely must eventually stop. The mechanization snake will finally eat its own tail. The only question is how soon.

[Discovering] things is much more gratifying if there has been some difficulty in the search for them. Those, after all, who never discover what they are looking for suffer from starvation, while those who do not have to look, because everything is ready to hand, often start wilting out of sheer boredom; in either case, a malady to be avoided.

— St. Augustine (tr. Edmund Hill, O.P.), Teaching Christianity (De Doctrina Christiana) 2.8. He is speaking about the interpretation of Scripture, and particularly of the “innumerable obscurities and ambiguities” (2.6.7); but there are many applications of this word.

Justin Smith-Ruiu: “[My] concern is not that we’re overestimating what machines might soon be able to do … but that we are systematically underselling the common understanding of what it is that human beings in fact do. We are now raising a generation of human beings who have come to believe of themselves that machines can do, or will soon be able to do, everything they as humans do, as well or better than themselves. This proves that they have accepted the model of themselves as essentially information systems. They don’t know, or can’t make any sense of the fact, that they are boiling over with affect, let alone that this is the dimension of them that they would do well to focus on if they wish to get some kind of handle on the human essence.”

If a government or major corporation wants to get serious about mitigating or reversing anthropogenic climate change, it should consider stopping research and development on generative “AI." Think about it:

• A ChatGPT query requires roughly 5x the electricity of a normal Google search — that is, a Google search from the days before Google’s “AI Overview” itself 5xed the energy use of every single Google search you perform. If this is the case for a purely text-based response, I don’t even want to know how much energy an image or video generator like DALL-E — or a “music” generator like Sora — requires per query.
• Furthermore, LLM chatbots — due to both their design and their inefficiency in completing the tasks for which they are designed — invite far more queries per day than your typical search engine. The teenagers who spend hours a day chatting with their LLM “therapists” or “AI girlfriends” may be the high tail of the bell curve, but there is no doubt that they are living right into the technology’s basic affordance. I challenge anyone to find a large number of people who literally spend all day Googling things (well, okay, actually, I do know at least one person like this). OpenAI wants everyone to use GPT-4o for everything, 24/7.
• To keep up with this insane and constantly-escalating scale of usage (think also of how every shopping website is suddenly integrating an AI-powered helper chatbot!), new data centers are being constructed every single day. Worldwide data center energy use probably doubled between 2022 and 2023, probably due in substantial part to the impact of rolling out generative AI tools, and seems on pace to keep doubling annually (in a sickening update to Moore’s Law).
• Note also that server farms and data centers are frequently built in regions like the American West which have plenty of available land but scarce water resources. In energy economics, there is a well-known tradeoff known as the “water-energy nexus:” the more water you are able to use (mostly for cooling), the less energy you have to use, and vice versa. In other words, there is ecologically no such thing as a free lunch: degrade your local ecosystem through water use, or pump carbon dioxide and other more noxious pollutants into the air somewhere else. Furthermore, by all accounts data centers significantly degrade quality of life and health outcomes (through, fascinatingly, steady noise pollution in both audible and inaudible frequency bands) for the people who are misfortunate enough to live near them.
• The easiest technologies to eliminate from the economy are, by definition, the ones which have not been integrated into the economy yet. Unlike the automobile, industrial agriculture, air travel, and the other technological revolutions to which we are constantly hearing “AI” compared, the “AI revolution” has not happened yet. It is incredibly unpopular — the American public views “AI” with something like 70% unfavorability the last time I looked into it, which is more unpopular than Donald Trump was at any point during his first term in office — and consistently becomes more unpopular as people learn more about it and have more experience with it. So why not simply say, with Bartleby the Scrivener, “I would prefer not to”? If you want to be serious about climate change, ban AI.

This, of course, will not actually happen. For one thing, it might not be legal (and certainly would not be legally practical) for, say, the US government to ban generative “AI” development. For another, all the incentive structures are aligned against it. To simply “not develop AI” is, clearly, a step that no currently existing tech company (and many not-yet-existing tech companies as well) is willing to countenance, for fear that they will be left behind by their AI-developing competitors — a classic race-to-the-bottom collective action problem. The incoming administration is filled with unapologetic cryptocurrency boosters (another infamously environmentally degradatory technology). And I should pause to say that I don’t quite wish to launch a Butlerian Jihad against all “AI” tools — I am very optimistic, for instance, about the improvements to weather forecasting which the new AI-based models seem to provide when used in conjunction with traditional computational physics-based models, and if AI tools can effectively replace human content moderators to keep porn off social media, all the better.

It’s also true that ending “AI” development would not come anywhere close to reversing anthropogenic climate change. Automobiles, industrial agriculture, and air travel are far larger contributors still to the problem, and there is no good replacement for fossil fuels in these domains (electric car boosters to the contrary). It is impossible to avoid the truism that if you want 18th-century emissions, you need an 18th-century lifestyle. Nobody in the 21st century is going to voluntarily revert to an 18th century lifestyle. What we need, rather, is a massive and non-fossil fuel source of energy that could not only, say, power AI, but also make planetary-scale carbon capture & storage economically viable. No solar or wind power technology is capable of providing this, for reasons of basic physics, and the ecological costs of resource extraction to make solar panels and their battery packs are so significant that it is not clear to me a solar panel will ever, environmentally speaking, “pay for itself” in emissions reductions. Hydropower sounds great if you have a massive river nearby (not the case everywhere!), but every time we check in on the maintenance requirements and ecological impacts of dams, the answer gets worse and worse. That is why I consider it enormously telling that AI developers such as Microsoft, recognizing that the new product they are shoving down all our throats requires an astounding quantity of energy which the current American grid is simply not ready to provide, are making quiet but massive investments in the future of nuclear energy.

The real proposal, then, might actually turn out to be: anthropogenic climate change, widespread generative “AI”, new nuclear energy — pick two.

Thesis: The appearance of effortless inhumanity is practically always dependent on the sacrifice or exploitation of hidden persons.

Why, then, the unscientific scorn for practical knowledge? There are at least three reasons for it, as far as I can tell. The first is the “professional” reason mentioned earlier: the more the cultivator knows, the less the importance of the specialist and his institutions. The second is the simple reflex of high modernism: namely, a contempt for history and past knowledge. As the scientist is always associated with the modern and the indigenous cultivator with the past that modernism will banish, the scientist feels that he or she has little to learn from that quarter. The third reason is that practical knowledge is represented and codified in a form uncongenial to scientific agriculture. From a narrow scientific view, nothing is known until and unless it is proven in a tightly controlled experiment. Knowledge that arrives in any form other than through the techniques and instruments of formal scientific procedure does not deserve to be taken seriously. The imperial pretense of scientific modernism admits knowledge only if it arrives through the aperture that the experimental method has constructed for its admission. Traditional practices, codified as they are in practice and in folk sayings, are seen presumptively as not meriting attention, let alone verification. And yet, as we have seen, cultivators have devised and perfected a host of techniques that do work, producing desirable results in crop production, pest control, soil preservation, and so forth. By constantly observing the results of their field experiments and retaining those methods that succeed, the farmers have discovered and refined practices that work, without knowing the precise chemical or physical reasons why they work. In agriculture, as in many other fields, “practice has long preceded theory.” And indeed some of these practically successful techniques, which involve a large number of simultaneously interacting variables, may never be fully understood by the techniques of science.

— James C. Scott, Seeing Like a State: Why Certain Schemes to Improve the Human Condition Have Failed, 305–06

Why you should not use ChatGPT, large language models, or other “artificial intelligence” (falsely so-called) tools in your research or work, for any of the synthetic tasks (summaries of data or information, etc.) for which it is proposed as a helpful time-saver:

1. The process of pattern recognition and synthetic integration is the basis of how human beings come to know and understand the world.
2. This process is an inextricably bodily process in humans. (This is true of human cognition in general: the whole of the human body, not just the brain, is involved in every act of thought — and in fact other bodies are involved, too, because thought is an intersubjective process. But I digress.)
3. ChatGPT and similar tools are, however, definitionally disembodied. Even if they are in fact “just pattern-recognition machines” (dubious), by virtue of being disembodied their pattern “recognition” is not the same as the real thing in humans.
4. In fact, insofar as ChatGPT exists in the physical world, it is under a very different sort of embodiment — a non-organic sort — which is antithetical to the human sort.
5. Therefore, ChatGPT and so forth cannot be trusted to faithfully simulate human knowing — and if the mechanism cannot be trusted neither can the results.
6. Additionally, by using such a tool, a human being forgoes the opportunity to practice and experience such knowing, kneecapping his or her capacity to learn from the experience.

In a nutshell: the promise of LLMs is “impersonal knowledge” — but no such thing exists. Relying on it is thus, in a meaningful sense, worse than nothing.