ChatGPT
vs Professor: The Good, Bad, and Bizarre of Machine Writing
The new ChatGPT
tool from OpenAI has rapidly garnered media attention and millions
of users. Although less effective sentence generators have existed
for years, ChatGPT's novelty has drawn both the public and the
reporters' interest, and its possibilities have led to it becoming
the fastest growing app in history. But when examined more closely,
its seemingly miraculous capabilities are ultimately disappointing,
at least when it comes to the academic writing of my discipline.
Like many of my peers, I wanted to test ChatGPT's capabilities
against the strictures and demands of my field. A physics professor
asked it thorny questions about cosmology, and a programmer used
it to generate code, but I was more interested in whether the
machine had been programmed carefully enough that it could be
used by my students to cheat on their papers.
I tried it
with several prompts from a list of typical essay topics, and
found the AI generated essays were summary-dependent, vague, fluent
in terms of diction but in substance nearly vacuous, and filled
with elementary compositional missteps as well as factual errors.
A decent command of diction and the ability to generate grammatically
correct sentences is not quite the same thing as producing meaning.
I tried the AI on three novels by H. G. Wells and Charlotte Perkins
Gilman's Herland, as well as several short stories by Thomas King
and others.
Although I
had been initially interested in how my students might use the
tool to plagiarize, I soon became intrigued by ChatGPT's mistakes.
Just as errors in meaning expose a student's facility with argument,
the AI's missteps tell us much more about its nascent capabilities
than the accuracy of its grammar or spelling.
It glibly
invented material it could not access, indulged in falsehood when
confronted, and wrote essays with a gleeful disregard for logic
and physics. Ultimately, to rework a quote from H. G. Wells, it
has developed in the most wonderful way the distinctive silliness
of a social-media-obsessed humanity without losing one jot of
the natural folly of a semi-informed monkey on a typewriter.
|