The integration of generative AI directly into word processors will give rise to a new, acceptable form of synthesized AI-human writing and universities are not prepared.
Thanks for posting this piece. We have been having conversations at my own institution about this very thing for the past year or so. Same concern: GAI will become ubiquitous once it is seamlessly integrated into word processors and other standard office software; we need to prepare for this change.
I also signed up for Copilot Pro and integrated it with my personal Microsoft 365 account. Here are my observations of the product right now:
1. Copilot for Microsoft 365 is still very much a beta product. It is somewhat stapled onto the existing suite of Office applications (currently just Word, Excel, PowerPoint, OneNote, and Outlook). It has varying functionality and efficacy depending on what program you’re using (Excel’s version of Copilot is quite limited, especially with natural language data analysis and no where near as sophisticated or useful as ChatGPT Plus).
The Copilot integrations are extremely buggy. The writer function in Word breaks repeatedly, throws up error messages, and sometimes just quits mid-sentence. But if you hammer away at it with prompts, you can get it to go and do much of what you say. Here’s a video of me writing a 2000+ word essay on the development of the welfare state in Canada in less than 15 minutes: https://youtu.be/htaX9qZR_e8?si=cpaehJbp8HtjgHh1
It’s impressive, but still limited. With that said, this is the worst this technology is going to get. It will improve with time.
2. The references are unreliable. In my tests writing AI-generated history essays, I struggled to get Copilot in Word to generate one real citation. I got some real historical quotes, but no accurate references. There were real journal titles, but fake articles, and other citation details. This continues to be a severe limitation in this kind of software and I don’t know how soon the capabilities of GAI in generating footnotes will improve. Microsoft still hasn’t enabled the ability to add comments to footnotes so I’m not holding my breath just yet.
3. If you watch the video of my welfare state essay, you’ll see much of what you describe in your article. I can insert paragraphs, ask for particular details to be added, and expand this into a more complex piece of work. As I worked on this, a colleague asked me a key question: Would you be able to generate such effective prompts if you didn’t already have prior knowledge of the subject matter. I should probably try writing something on a topic I know nothing about. That would, at the very least, severely limit my ability to catch factual errors. For example, in an essay on the Fall of New France that I tried writing with Copilot, the first draft said that the Seven Years War was fought between Britain, France, and India.
4. Writing with Copilot in Word reminds me of vacuuming my living room with a robot vacuum. I have to move the furniture, clear cables and cords off the floor, check for cat toys that might have rolled under the couch, and then do a little sweeping after to catch areas the robot missed. At a certain point, writing with Copilot can start to feel more like just writing this thing myself. The line can get blurry, but from the perspective of academic misconduct, the amount of work I need to do to achieve higher order thinking and analysis in my writing with Copilot starts to become more work than just writing the essay without Copilot.
I think this is where GAI is driving those who teach in higher education to rethink assessment. As you say, we will need to have difficult conversations about writing as a method of assessment. There are types of assignments that may no longer make sense and we may have to push for assignments that ask for high order thinking and analysis. For now, we can start by asking for some research sources and citations.
Please keep up your writing on this subject. I’m reading with great interest.
Hi Mark (and Sean), thanks very much for this discussion.
I have a "newbie" question. If I were to write an original research paper in Word with Co-Pilot enabled, would that mean that my writing would be added to Co-Pilot's "memory" so to speak and then my own in-progress research findings would become publicly available?
Case in point, I downloaded Co-Pilot Pro this morning to test it out in Word, then later on started to draft a new research article in Word wherein I was going to put the results of my recent research, but that little "Co-Pilot" cursor was blinking at me. It certainly gave me pause.
Does enabling Co-Pilot in MS Office make our data less secure than ever? What are your thoughts? Thanks in advance.
Hi Mark!
Thanks for posting this piece. We have been having conversations at my own institution about this very thing for the past year or so. Same concern: GAI will become ubiquitous once it is seamlessly integrated into word processors and other standard office software; we need to prepare for this change.
I also signed up for Copilot Pro and integrated it with my personal Microsoft 365 account. Here are my observations of the product right now:
1. Copilot for Microsoft 365 is still very much a beta product. It is somewhat stapled onto the existing suite of Office applications (currently just Word, Excel, PowerPoint, OneNote, and Outlook). It has varying functionality and efficacy depending on what program you’re using (Excel’s version of Copilot is quite limited, especially with natural language data analysis and no where near as sophisticated or useful as ChatGPT Plus).
The Copilot integrations are extremely buggy. The writer function in Word breaks repeatedly, throws up error messages, and sometimes just quits mid-sentence. But if you hammer away at it with prompts, you can get it to go and do much of what you say. Here’s a video of me writing a 2000+ word essay on the development of the welfare state in Canada in less than 15 minutes: https://youtu.be/htaX9qZR_e8?si=cpaehJbp8HtjgHh1
It’s impressive, but still limited. With that said, this is the worst this technology is going to get. It will improve with time.
2. The references are unreliable. In my tests writing AI-generated history essays, I struggled to get Copilot in Word to generate one real citation. I got some real historical quotes, but no accurate references. There were real journal titles, but fake articles, and other citation details. This continues to be a severe limitation in this kind of software and I don’t know how soon the capabilities of GAI in generating footnotes will improve. Microsoft still hasn’t enabled the ability to add comments to footnotes so I’m not holding my breath just yet.
3. If you watch the video of my welfare state essay, you’ll see much of what you describe in your article. I can insert paragraphs, ask for particular details to be added, and expand this into a more complex piece of work. As I worked on this, a colleague asked me a key question: Would you be able to generate such effective prompts if you didn’t already have prior knowledge of the subject matter. I should probably try writing something on a topic I know nothing about. That would, at the very least, severely limit my ability to catch factual errors. For example, in an essay on the Fall of New France that I tried writing with Copilot, the first draft said that the Seven Years War was fought between Britain, France, and India.
4. Writing with Copilot in Word reminds me of vacuuming my living room with a robot vacuum. I have to move the furniture, clear cables and cords off the floor, check for cat toys that might have rolled under the couch, and then do a little sweeping after to catch areas the robot missed. At a certain point, writing with Copilot can start to feel more like just writing this thing myself. The line can get blurry, but from the perspective of academic misconduct, the amount of work I need to do to achieve higher order thinking and analysis in my writing with Copilot starts to become more work than just writing the essay without Copilot.
I think this is where GAI is driving those who teach in higher education to rethink assessment. As you say, we will need to have difficult conversations about writing as a method of assessment. There are types of assignments that may no longer make sense and we may have to push for assignments that ask for high order thinking and analysis. For now, we can start by asking for some research sources and citations.
Please keep up your writing on this subject. I’m reading with great interest.
Hi Mark (and Sean), thanks very much for this discussion.
I have a "newbie" question. If I were to write an original research paper in Word with Co-Pilot enabled, would that mean that my writing would be added to Co-Pilot's "memory" so to speak and then my own in-progress research findings would become publicly available?
Case in point, I downloaded Co-Pilot Pro this morning to test it out in Word, then later on started to draft a new research article in Word wherein I was going to put the results of my recent research, but that little "Co-Pilot" cursor was blinking at me. It certainly gave me pause.
Does enabling Co-Pilot in MS Office make our data less secure than ever? What are your thoughts? Thanks in advance.