There is much hype afoot about LLMs now being suitable “research assistants” or “writing companions” for serious humanistic work, so I decided to take a temperature check by asking Claude (what I took to be) a fairly basic historical theology research question:

What is the best English edition of St. Augustine’s Enchiridion?

The Enchiridion, a short, late catechetical work, is perhaps one of that Father’s less well-known works today — it lacks the name recognition of, say, the Confessions or the City of God — but is nevertheless one of his most continuously influential works throughout church history. I had a hunch there might be a handful, but not a plethora, of translations. So I asked Claude to help me out.

You can read the ensuing hilarity here. Spoiler alert: It did not go well.

Disclaimers: I have not ponied up $20/month to Anthropic, so I ran this little experiment on Claude Sonnet 4 (rather than the supposedly Pro-level Opus 4, a “powerful, large model for complex challenges”; apparently I overestimated what a “complex challenge” this actually was). Now for the scorecard. Claude:

• got everything right about the NPNF translation by J. F. Shaw (which, being public domain, I imagine was actually part of its training data);
• misidentified the editor (Boniface Ramsey, O. P.) as the translator of the New City Press edition, and was unable to find the translator’s name (which I found in moments by following one of the links which Claude provided me);
• initially overlooked the Library of Christian Classics version translated by Albert Outler and published in 1955, and then required additional prompting to discover that the 1955 edition was, in fact, the same as the Outler edition;
• and completely invented not one but two(!) translations: one by Thomas Williams, who at least is a known translator of St. Augustine, and one by Mark Dever, who is not.

For now, I think, I will be sticking to my old and LLM-less research habits.

UPDATE: Ian Harber — who has ponied up for Claude Pro — tasked Sonnet 4 with the same prompt, but in Research mode, and it generated what appears to be a largely accurate and, I admit, pretty useful report; I learned from it about a couple of older translations which actually exist (though are largely unavailable). It does, however, offer some peculiar comments, e.g., “The market [for translations of the Enchiridion] shows stability around current major editions rather than competitive innovation” — a comment I would not expect to find in a research guide written by, you know, an actual human historian. Nevertheless, it’s fair to say that my humorous little experiment with the free, non-Research version did not give me an accurate sense of Claude’s current capabilities, and I stand appropriately corrected.

The principal question remains, of course, not whether one can but whether one should use even the more powerful and capable tools for humanistic research. About this we have much to say, and we may try to explain it later…