Seven months after its release, ChatGPT is being touted as a replacement for a range of occupations. Artificial Intelligence (AI) tools like ChatGPT (technically a Beta rollout) are being tested against humans in areas like accounting and early results are rolling in.
The early verdict? Stick with human accountants for the time being! ChatGPT not only performs significantly lower than real-life accountants – but when it doesn’t know the answer to a problem it completely fudges the numbers.
The American Accounting Association (AAA) compared how accounting students answered more than 28,000 questions on accounting assessments and textbook tests with ChatGPT. The human data came from 14 countries and 186 institutions (most of the schools are based in the U.S.)
ChatGPT provided correct answers for 56.5% of questions and partially correct answers for an additional 9.4% of questions. If ChatGPT were an accounting student, it might be able to talk its professor into a D- grade. (Or drop the course and take it again later!)
Human students did substantially better than the AI student, scoring 76.7%.
ChatGPT makes numbers out of thin air
The AAA researchers go out of their way to be fair to ChatGPT: “[It] performs better than the student average for 15.8% when we include partial credit [answers].”
We think they’re being too fair here. Consider that ChatGPT often:
- made up facts. “For instance, when providing a reference, it generates a real-looking reference that is completely fabricated … the work, and sometimes authors, don’t even exist.”
- suggested answers that weren’t included as multiple-choice options
- struggled to answer “multiple-choice questions that describe a situation [which] require accounting students to then select a concept illustrated by this situation and questions which require higher-order learning,” and
- failed to evaluate and analyze complex, nuanced assessments.
In ChatGPT’s favor, if it’s unable to answer a question, it’s good at providing “detailed instructions to complete a question. For instance, [it’s able to] provide steps on using a software tool or sample code to solve problems that require access to a specific database.”
In other words, the AI tool would make a promising accounting intern, albeit one that’s prone to make up numbers if not kept in check! Final score: Humans 1, AI 0.
