Discussion about this post

User's avatar
Liam Kelley's avatar

That sentence ("The writing is not very good nor engaging and the findings are pretty boilerplate but it’s an A to A+ paper.") stood out for me as well, however, I have a different take. What I find with a lot of people who, like Mark, see the enormous transformation that is taking place and try to explain that to a readership that is not receptive, is that they (perhaps subconsciously) throw in phrases like that to "soften the (existential) blow" (perhaps to themselves as well).

There is no longer any reason for the writing to "not be very good." For a long time now, all you have needed to do is to prompt the LLM to write in a certain style, or to upload some text from someone and to instruct it to write in that style. And the savvy students already know this.

Now with Claude Cowork, you just add instructions. I haven't looked, but I'm sure that there must be "good-style-academic-writing.md" instructions that are circulating out there so students don't even need to think of what "good writing" might be. They can just drop those instructions into Claude and "voila"!

Further, from my own experience and experiments (still only using $20-a-month models), we're moving beyond the point of "boilerplate" conclusions. This has become particularly noticeable to me over the past month. I just had a skeptic the other day ask me to prompt an LLM about a rather niche topic regarding Vietnamese music (one that required figuring out different notation systems, etc.). According to my colleague, it produced a conclusion that no one had ever made before, and that he had never thought of before. And that was with zero follow-up prompts, etc. My own experiments on various topics produce the same outcomes.

Perhaps if the assignments are about well-worn topics, then yes, the LLM might produce boilerplate conclusions, but so would 99.99999% of undergraduate students in the pre-AI age have done the same.

I think we are entering a phase where there are people who know how to use LLMs to get the best results (who ask it to come up with the best prompt first, rather than writing it on their own or just dropping the prof's assignment in, and who add all kinds of instructions to Claude Cowork, etc.), and people who don't. Those who don't know will give the impression that LLMs are not capable of doing what historians can do (and that makes us feel like we still have a place/role). However, it's the papers written by the people who DO know what to do that we need to pay attention to.

It will be interesting to see on which side of the divide the AI-native students will fall. Just because they grew up with it does not mean that they are better at it (laziness/disengagement and procrastination also play important roles). But the fact that some cannot produce good papers doesn't mean that the LLMs are incapable. They are totally capable at this point.

As such, I completely agree with Mark that "To my mind, the traditional research paper as a tool of differential assessment is now dead."

And like Mark, I have no idea what to do next. . .

Donna Dickerson's avatar

Excellent! I would have given a master’s student a “B-“ and had her/him re-write after reading my lengthy critique. I would much rather a student re-think, re-write, etc. one paper until it earns an “A” than write 3 separate papers in a semester that aren’t good graduate work. It’s that iterative process that forces deeper thinking and mastery. It also moves the student away from dependence on AI in the latter stages of real scholarship. Yes, I even had doctoral students who hated that “polish and polish again” process, but many thanked me later for not giving up on them and for believing they were more capable than they themselves believed.

19 more comments...

No posts

Ready for more?