Can ChatGPT Help with Historical Research?
By John Norton
MiddleWeb Co-Editor
I’ve been researching my uncle and namesake John Henry Norton’s role as an Army officer and Counter Intelligence Corps (CIC) agent in the Pacific theatre during World War II. I thought I’d ask ChatGPT-4 for some help.
I requested 20 pertinent reference sources. The AI chatbot quickly provided the list below. I then chose, more or less at random, four of the 20 and asked GPT-4 where I could find each one on the internet. As you will see, in each instance, ChatGPT quickly confessed that it had made the source up.
I guess it’s back to the research grind for me. ;-)
The “Conversation”
JOHN NORTON: List 20 sources about the history of the Counterintelligence Corps in the Pacific during World War II.
CHATGPT-4: There are numerous sources available on the history of the Counterintelligence Corps in the Pacific during World War II. Below are 20 sources, including books, articles, and websites that can provide you with valuable information:
- “The Counterintelligence Corps: Its History and Mission” by Ray S. Cline
- “Counterintelligence in World War II: The Origins of the U.S. Army Counterintelligence Corps” by Ian T. M. McGrath
- “Counterintelligence in the Pacific Theater” by T. D. Miers
- “Spies, Lies, and the War in the Pacific” by Bill Yenne
- “The Invisible War: The United States Counterintelligence Corps in World War II” by Clayton D. Laurie
- “The Secret War Against Japan: The Origins and Development of U.S. Counterintelligence in the Pacific, 1939-1945” by Michael J. Lyons
- “The Intelligence War in the Pacific” by Peter Donovan
- “Spy Catchers: The Counterintelligence Corps in the Pacific” by John Patrick Finnegan
- “Codebreaking and Signals Intelligence in the Pacific War” by Jeffrey T. Richelson
- “The Double-Cross System in the Pacific War” by J. C. Masterman
- “Espionage and Counterintelligence in the Pacific War: The Role of the U.S. Army Counterintelligence Corps” by Paul J. DeCapua
- “A History of U.S. Counterintelligence” by William R. Johnson
- “The U.S. Army Counterintelligence Corps in the Philippines, 1941-1945” by Glenn J. Farris
- “The Counterintelligence War in the Southwest Pacific Area” by Charles A. Willoughby
- “The Pacific War: The Strategy, Politics, and Players That Won the War” by William B. Hopkins
- “The Other Side of the Fence: U.S. Army Counterintelligence Activities in Australia during World War II” by Desmond Ball
- “The Counterintelligence Corps and the Occupation of Japan” by Robert J. Hanyok
- “MacArthur’s Ultra: Codebreaking and the War against Japan, 1942-1945” by Edward J. Drea
- “American Cryptology During the Cold War, 1945-1989: Book II, Centralization Wins, 1960-1972” by Thomas R. Johnson
- “The Counterintelligence Corps in World War II: An Annotated Bibliography” by Peter C. Oleson
These sources will provide you with a wealth of information on the history, operations, and impact of the Counterintelligence Corps in the Pacific during World War II.
JOHN NORTON: Where can I find “The Invisible War: The United States Counterintelligence Corps in World War II” by Clayton D. Laurie.
CHATGPT-4: “The Invisible War: The United States Counterintelligence Corps in World War II” by Clayton D. Laurie is a valuable resource for understanding the role of the Counterintelligence Corps during World War II. Unfortunately, it seems that I made an error in my earlier response, and there is no such book with that exact title.
However, Clayton D. Laurie has indeed written about the Counterintelligence Corps. Laurie is the author of “The Role of Federal Military Forces in Domestic Disorders, 1945-1992” and has contributed to various publications related to military history. In this case, I recommend looking for articles or book chapters authored by Clayton D. Laurie that focus on the history of the Counterintelligence Corps in World War II.
You can also try searching online academic databases, such as JSTOR, Project MUSE, or Google Scholar, to find relevant articles or book chapters. Alternatively, you can visit your local library or university library and consult with a librarian to help you locate relevant materials on the subject.
JOHN NORTON: Where can I find “The Counterintelligence War in the Southwest Pacific Area” by Charles A. Willoughby.
CHATGPT-4: I apologize for the confusion in my previous response. It seems that there is no book or article with the exact title “The Counterintelligence War in the Southwest Pacific Area” by Charles A. Willoughby. However, Charles A. Willoughby was an intelligence officer during World War II, and he authored the book “MacArthur’s Intelligence Officer: General Charles A. Willoughby.”
[Note: Actually the book is about Willoughby, not BY Willoughby and the title is Loyalty First: The Life and Times of MG Charles A. Willoughby, MacArthur’s Chief Intelligence Officer. So the substitute reference is made up, too.]
JOHN NORTON: Where can I find “The U.S. Army Counterintelligence Corps in the Philippines, 1941-1945” by Glenn J. Farris.
CHATGPT-4: I apologize for the confusion in my previous response. It appears that there is no book or article with the exact title “The U.S. Army Counterintelligence Corps in the Philippines, 1941-1945” by Glenn J. Farris. It seems that I made an error in providing you with that specific source.
Nonetheless, if you are interested in learning about the role of the U.S. Army Counterintelligence Corps in the Philippines during World War II, you can explore the following sources:
- “MacArthur’s Ultra: Codebreaking and the War against Japan, 1942-1945” by Edward J. Drea
- “The Pacific War: The Strategy, Politics, and Players That Won the War” by William B. Hopkins
- “Spies, Lies, and the War in the Pacific” by Bill Yenne
These books will provide you with valuable information on the counterintelligence activities and operations in the Philippines and the broader Pacific theater during World War II.
[I haven’t looked these up – but chances are…. ;-]
JOHN NORTON: Where can I find “The Counterintelligence Corps and the Occupation of Japan” by Robert J. Hanyok.
CHATGPT-4: I apologize once again for the confusion in my previous responses. After further research, it appears that there is no book or article with the exact title “The Counterintelligence Corps and the Occupation of Japan” by Robert J. Hanyok. It seems that I made an error in providing you with that specific source.
However, if you are interested in learning about the role of the U.S. Army Counterintelligence Corps during the occupation of Japan, you can explore the following sources:
- “Embracing Defeat: Japan in the Wake of World War II” by John W. Dower – This book provides a comprehensive account of the American occupation of Japan after World War II, including the role of the Counterintelligence Corps.
- “The American Occupation of Japan: The Origins of the Cold War in Asia” by Michael Schaller – This book discusses the U.S. role in occupied Japan and provides information on the various intelligence operations that took place during this period.
[These two are real books and actually written by these authors!]
The Aftermath
I haven’t checked the other titles on GPT-4’s original list. They all sound convincing, but you can be sure some of them are fake. That’s disturbing, don’t you think? Imagine you’re a student with faith in this wonderful new tool.
I did ask GPT-4 about one book on the list that does exist. Sort of. I’d discovered it in my own research (it actually includes a story about my uncle). GPT-4 shows it as 8. “Spy Catchers: The Counterintelligence Corps in the Pacific” by John Patrick Finnegan. I knew that wasn’t quite right.
I asked GPT-4 if it (is this the correct pronoun for AIs?) should have instead reported:
Edwards, Duval A. Spy Catchers of the U.S. Army in the War with Japan (The Unfinished Story of the Counterintelligence Corps). Red Apple Publishing, 1994. ISBN 1-880222-14-0.
CHATGPT-4: Yes, you are correct. I apologize for the mistake in my previous response. The source I meant to refer to is:
8. Edwards, Duval A. “Spy Catchers of the U.S. Army in the War with Japan (The Unfinished Story of the Counterintelligence Corps).” Red Apple Publishing, 1994. ISBN 1-880222-14-0.
This book provides a detailed account of the U.S. Army Counterintelligence Corps’ activities and operations during World War II, particularly in the Pacific theater and against Japan.
JOHN NORTON: What do you mean by “I meant to”? Are you capable of meaning to do something yet doing something else?
CHATGPT-4: As an AI language model, I don’t have personal experiences or intentions like a human does. My responses are generated based on patterns in the text data I’ve been trained on. When I say “I meant to,” I am using a conversational convention to acknowledge that my previous response contained an error, and I am providing the correct information.
In this case, I made a mistake in listing the source in my previous response, and I wanted to correct that mistake by providing the accurate source information. My use of the phrase “I meant to” is a way to communicate this correction in a more natural and understandable way.
JOHN NORTON: Speaking as a human, I feel much better now.
Image: Pixabay, Alexandra Koch
Update – 5/28/23: O brave new world
A lot has happened on the AI front in the three months since I posted this (sort of snarky) report on my first in-depth ChatGPT experience. The major Silicon Valley entities have been rushing out their versions of a generative chatbot and many, many software platforms have begun adding Open AI tech to their applications to stay au courant. AI imaging apps are also proliferating with do-it-yourself versions and professionally managed tools like Shutterstock’s AI option.
A good place to keep track of all this – with teaching/learning in mind – is Larry Ferlazzo’s regularly updated resource: The Best New – & Free – Artificial Intelligence Tools That Could Be Used in the Classroom.
We continue to get reassurances from Open AI and other chatbot creators that the technology will continue to evolve and that the concerns of many computer scientists, media literacy experts, and other commentators that these online AI “helpers” pose real threats to humanity are overblown.
Somewhat ironically, the argument is being made that they will become less threatening to humanity as they grow smarter, ignoring what Wired Magazine called AI’s “dirty little secret” – its major negative environmental impact on the Earth’s climate – an effect that will increase as more and more computing power is needed to support all that smartness. Will this problem be a blessing in disguise – forcing the accelerated development of low-impact energy sources? Our children and grandchildren will know the answer.
Meanwhile, my post above making fun of ChatGPT’s current poor performance when asked to produce citations was obviously not read by the lawyers representing an injured passenger of Avianca Airlines. The New York Times (5/27/23) tells the story in full here. I’ll just share this snippet:
“When Avianca asked a Manhattan federal judge to toss out the case, Mr. Mata’s lawyers vehemently objected, submitting a 10-page brief that cited more than half a dozen relevant court decisions. There was Martinez v. Delta Air Lines, Zicherman v. Korean Air Lines and, of course, Varghese v. China Southern Airlines, with its learned discussion of federal law and ‘the tolling effect of the automatic stay on a statute of limitations.’
“There was just one hitch: No one — not the airline’s lawyers, not even the judge himself — could find the decisions or the quotations cited and summarized in the brief.
“That was because ChatGPT had invented everything. [Emphasis added]
“The lawyer who created the brief, Steven A. Schwartz of the firm Levidow, Levidow & Oberman, threw himself on the mercy of the court on Thursday, saying in an affidavit that he had used the artificial intelligence program to do his legal research — ‘a source that has revealed itself to be unreliable.'”
A final update note: My colleague and book collaborator Sheryl Nussbaum-Beach has been using ChatGPT to help her develop several book-related themes. It’s her contention that users who learn to “coach” the AI tool can develop the skill set they need to extract a lot of benefit. Here’s an example of coaching/teaching ChatGPT to differentiate science texts for students – from the TCEA website.
We’re talking about technology, which by its very nature is always evolving, and I/we need to take the long view. Wired Magazine columnist Steven Levy can help us do that. Read his Plain Text newsletter (5/12/23) for a clarifying overview of the likely future: You’re Probably Underestimating AI Chatbots.
Perhaps we should approach chatbots like weapons – powerful objects that should never be handled without training and full awareness of their potential, now and later.
John Croft Norton is an independent education writer/editor and the founder and co-editor of MiddleWeb. He graduated from rural Wampee Little River High School (SC) in a class of 32. He now lives in the western North Carolina mountains. Visit his LinkedIn page.
John, thank you for sharing this post, which is STUNNING. I mean, I expected Chat GPT to make mistakes, but I did not think it would INTENTIONALLY provide MADE-UP sources. What is the point of that??? Wow. And then to say, “I meant to…” Wow.
Thanks for sharing this post. I had heard of Chat/GP searches returning incorrect (and I am being generous here) research and references, but this hits home. I regularly google my name for updates to my real publications and immediately homed in on the “hit” of the false article about counterintelligence which I did not write.
I am often misquoted, but have never had an article attributed to me I did not write. What really surprises me is the fabricated sources and authors. It almost seems that the AI intentionally did this. This is unnerving, especially when I consider the potential for serious mischief.
Thanks for commenting, Sarah and Robert. It appears that ChatGPT is plucking information from various sources and assembling it according to a formula (citation convention?) to create an answer. It’s rare to see it say “I don’t know” or “I’m not able to do that.” As for mischief, I don’t think it has a sense of right and wrong. In fact I just asked ChatGPT-3.5 that question:
JOHN: Do you have a sense of right and wrong?
GPT-3.5: “As an artificial intelligence language model, I do not have a personal sense of right and wrong, nor do I have personal beliefs or values. My responses are generated based on patterns and associations learned from analyzing vast amounts of text data.
“However, I am programmed to provide information on ethical and moral considerations based on societal and cultural norms, and to help users make informed decisions. Ultimately, any decisions regarding right and wrong are left to the individual user’s own judgment and values.”
Many people are writing about how helpful GPT has been in their work. A friend of mine is editing a draft of a new book with GPT’s help. The question on the table is “Can you always trust it?” And if not, how does that limit what you do with it? Part of my answer (in this post) is: it won’t do historical research more quickly and effectively. At least not yet.
Since I was offered GPT-3.5 at my Open AI account this morning, I went back to make sure my conversation in this blog post came from GPT-4. It did.
This video clip from the Next Big Idea Club is revealing and reminds us that AI doesn’t “understand” what you want.
“Since AI doesn’t really understand what you want it to do, it tends to take the path of least resistance—leading to some hilarious shortcuts. When one programmer asked an AI to reduce the sorting errors in a list of numbers, the AI happily did so—by deleting the list entirely. No more list means no more errors, right? Problem solved!” To find out more about why AI behaves like this, check out this video from AI expert Janelle Shane, author of You Look Like a Thing and I Love You.
And also this post from Janelle Shane’s blog AI Weirdness. It’s another tale of AI making stuff up “to be right.”