Check out an e-mail I just received today from a Pearson representative. I can get paid $250.00 and have two essay assignments graded for me, if I turn over my students’ papers for the benefit of Pearson’s computer grading scheme! I just learned about these scamtastic software packages last night via a comment that Indyanna left on my previous post. Professor Pushbutton, here we come!
I am emailing you regarding a class project that may be of interest to you. For this project, Pearson is looking for instructors of the U.S. History course who are interested in integrating writing assignments into their course.
. . . . .
Pearson is developing a computer-assisted grading program that will accurately auto-grade brief writing assignments. The program uses specific rubrics and writing prompts to achieve computer grading accuracy. For the program to work correctly, thousands of student essays are scored by hand and loaded into the system. By doing this, the system “learns” how to grade essay questions. This system has been successfully introduced into a number of course areas, and Pearson is now bringing this technology to the history market.
I am writing to see if you would be interested in helping us to build the bank of student essays needed to develop the product. Pearson will provide you with a choice of writing prompts and rubrics, two of which you would assign in your course during the spring 2012. The students will enter their essays through an online portal. These will be graded by a subject matter expert and the grades will be sent to you. These essays will also be used to create the auto-grading functionality mentioned above.
In exchange for your help and time, you will receive a $250 honorarium. In addition, you will have had 2 essay assignments graded for you.
If you are interested in working on this project, please let me know by return email. I will then send you the list of writing assignments for you to select the two assignments you wish to use in your course. This is a very simple way to include additional short essays into your Summer/Spring course.
I look forward to hearing from you.
Dear Pearson Person:
I’d rethink that sign-off if I were you. I don’t think you’re going to be happy when you’re through hearing from me! But don’t worry: no one reads my stupid blog anyway.
I do my best teaching at a dramatically underfunded public R-1, so you’d think that I’d leap at the chance to outsource my work and get paid $250.00. (After all, that’s $250.00 more than all of the raises I’ve not had for the past four years put together!) But seriously, friend: do you really think I’m as soulless and as cheap as an educrat? Even if you had, say, a decent sum of money, like $750,000 to offer me, I wouldn’t sell a single byte of my students’ intellectual property to you, because it’s their work, not mine. I also have serious qualms about shuffling off what I see as my responsibility to my students and colleagues to come up with my own essay topics and mark them myself. What can I say? I’m the only person who has to look at this mug in the mirror every morning.
Secondly, although I’m sure you know that I’m dramatically underpaid for the work that I do, my work has real value, and even if I’m the only person who believes this, at least I can sleep at night knowing that a human with a heart and a brain and 16 years of experience has evaluated my students’ writing. My students may not care if their grader has a Ph.D., a book, and several other publications to her name, but I’m pretty sure that I’ve thought longer and harder about evaluating student writing than any of your software designers. Call me a luddite, or a locolocascholastic–like a locolocavore, we think it’s more responsible and better for us to teach and grade locally.
I’ve got a question for you: who is your target audience for this software? Amherst College? Stanford? Yale? No??? How come?
Finally, from what I understand, these essay grading software packages are piss-poor, and are guaranteed in my view only to further degrade the quality of education by teaching students not how to write effective essays, but rather writerly-like tricks to fool your computers. I just don’t see where it’s in either my short-term or my long-term best interests to encourage you in the path that you’re on. The fact is that I can’t be replaced by “Professor Pushbutton,” and I see no reason to assist you in selling the lie than I can be.
Now git along now and don’t let the door hit you where the good Lord split you–
Historiann
Historiann 
Shockingly enough, I got that same e-mail. Thank you for covering it so that I don’t have to do it myself.
I’ll just add one thing, though: Suppose you’re a college freshman. You’re parents just took out a second mortgage so that you can go to college. How would you feel to know that your essays were being graded by a grade-bot? How would your parents feel?
LikeLike
Great point, Jonathan. But then, the educrats would say that that’s the price you pay for not being born into the ruling class. People who refinance their homes don’t deserve to have a *real professor* teach them and mark their work, right?
(I think of you every time I write about Professor Pushbutton!)
LikeLike
That NYT article clearly shows that these horrific programs can discern real facts. Very useful in a history paper. Oh, wait! That was a fragment — lower the score for me!
This is just one more push (along with that online fixation) to make professors obsolete. It makes me weep for the future of academia… I wonder how many profs responded in the positive?
And what about FERPA laws?
LikeLike
Can’t discern. Sorry.
LikeLike
Winerip also said yesterday that at least ETS allowed the MIT guy to use their system for a month, where he proved that the “auto grade functionality” could be gamed in all sorts of literally ludicrous ways, but that Pearson would not, because as its spokesman said “He only wants to prove we’re wrong” (paraphrase; don’t have the article in front of me). It sounded like a kid who had just had his lunch money taken on the playground.
I think all of this vendor-to-instructor solicitation– which includes various kinds of “selling the kids in the seats to the company” initiatives–ought to be turned over to the Justice Department as a form of suborning people with real fiduciary responsibilities (us) to commit all sorts of at least borderline criminality, which only begins with expropriation of intellectual property rights. [I once told an SSED colleague to scram when ze told me I had to give hir examples of my students best and worst papers because some acreditocrat needed to read them for the Ed School’s accreditation review]. If the regulatory community won’t intervene I can only think of a couple of alternatives: viz. Anonymous could target the vendor and take down their IT operation, or boycott: don’t order anything from them for any purpose.
LikeLike
Yee Haw. Crap, count me among the locoscholastics. What really irritates me is that these techies are too often the same ones that can’t speak a coherent sentence or look somebody in the eye and speak (where that is socially acceptable). No, they’d much rather do it all via e-technology: the speak bot. Twitter. Facebook. Anything to keep from having to deal with real live honest-to-goodness people. Administrators, again far too many of them, like this ‘replace troublesome faculty with tweak-able software/tech’ approach.
I’m hardly a Luddite; I use a lot of technology. But not to replace thinking, critical analysis, etc.. Grrrr.
LikeLike
Some chunks of healthy blasphemy: A lot of human research, deep thinking and quality individuals produced programs that can learn. We all know that the best chess players will lose to Deep Blue. Eventually, after large investments and way more experience, programs will be able to grade vanilla flavored history assignments. There are many examples of successful programs that perform tasks we always considered thought as solely human tasks.
Pearson’s program may not be around the corner, but it surely several blocks away.
LikeLike
I’m sure you’re probably right, koshembos, but there are enough of us out here to resist and slow down its approach. Furthermore: I’ll hold off on using software to mark my students’ essays until I know that the same programs are being used at Swarthmore, Colby, Williams, etc. I think the point you make about “plain vanilla” history essays is the right one. But most of us have adapted to the easy-plagiarism world of the internets pretty well, and most of us who assign essays are usually pretty careful to write original assignments and to require a specific set of sources or different readings. (That is, I think many humanities proffies are already writing assignments that can’t be realistically evaluated by software.)
Also: on proofreading, I think the terms I mean above are locascholastic, after locavore. LOCOscolastic = crazy scholar.
(Latinists: I’m surprised you haven’t corrected me yet or suggested a better term! Fire away, por favor.)
LikeLike
At the end of May I am going to have to grade over 250 essay questions in two weeks. We are not allowed to have our TAs help us grade, but there is nothing in the regulations banning machines. So please send me one of those grading machines. It would make my life so much easier. I wouldn’t mind grading so much if I made what tenured professors make in the US. But, it appears I make less than one fifth according to PhD comics.
LikeLike
Commentary in the mass media on this sort of nonsense slags off the teachers’ protests as being based merely on self-interest. The whole concept that teaching is a profession and maybe we know something about it just doesn’t exist.
Stuff (and nonsense) like this machine grading scheme is just more of the same ignorance about what teachers actually do. And what machines can do. Computers are fine at expert systems with limited numbers of options and defined goals, such as chess games or flying the shuttle.
A history paper is not like that. The writer should frame an interesting question, present an argument that shows judgment, show the ability to discern new patterns, and make connections that have perhaps never been made before. (Okay, maybe not in freshman History, but even there, that’s the general idea.) Artificial intelligence is not even in the ballpark for that kind of thinking. “Pearson’s program” is not several blocks away. It’s on a comet in the Oort cloud and we don’t even know how to land on it yet. Forget bringing anything back.
Just because any good teacher can do that kind of grading doesn’t mean it’s simple or easy. Trying to substitute something simple and easy (and CHEAP!) will lead to simple and easy minds. By the time it’s obvious, it’ll be way too late to do anything about it. The dumbing down will have made too many people who can’t fight their way out of wet tissue paper.
But what do I know? I’m just a prof with a job to protect, right?
LikeLike
(And, no, I’m not a Luddite either. I’m a biologist. I use automated sequencers. We machine grade multiple choice all the time. Technology in its place is great. Out of it, it’s Frankenstein’s monster.)
LikeLike
For $750,000 I bet you could give the students a cut they’d be willing to take in exchange for their papers and all come out ahead. Then you could hand-grade the essays. And maybe buy out a course or service or something. $250 not so lucrative.
LikeLike
If these theoretical machines are good enough to beat chess champs, why waste them on undergratuate essay exams? I see a whole new world here of machines writing referees’ reports for submitted article and book manuscripts; a machine that comments on papers at conference sessions (kept in check by another machine that heartlessly cuts off the papers themselves at the 20 minute mark); a choice between two well-pedigreed machines to be “president-elect” of the AHA or MLA–to be voted on, of course, by the hard-working machines down below. Us protoplasmic stakeholders could just do research that, while narcissistically-agreeable to do, would have no prospect of actually “appearing” anywhere, since who would want to get chewed out by a machine?
LikeLike
Brava! Also, it strikes me that it’s important not only to grade your own papers, but also, in British parlance, to set your own questions. I think Marc Bousquet’s post at the Chronicle has it right: if your essays can be graded by machine, it may well be that you’re not assigning complex enough essays. Personally, I think it’s part of my job to design assignments challenging enough to elicit answers that are, in turn, challenging to grade. Of course, sometimes I wish the grading were less challenging, but, in my better moments, I’m delighted to see them thinking, and coming up with ways of organizing their ideas that grow organically from their arguments, or even get tangled up in interesting ways as they wrestle with a new level of thinking.
LikeLike
@Indyanna: or they’d be sorting through the streams of information from various forms of surveillance so efficiently and effectively that we’d never have another terrorist attack, anywhere. Evidence suggests that, while surveillance has increased exponentially, even the best super-secret government machines haven’t reached that level of analysis.
LikeLiked by 1 person
Great post, heartening responses. They can have my red pen when they take it from my cold, dead hand. And not a moment sooner (fragment).
LikeLike
Yee-haw, cowgirl! This needs to be circulated far and wide.
LikeLike
Regarding the whole intellectual property issue, I remember there was a real dust-up at the University of British Columbia in the mid-2000s because the Political Science department required all essays to be submitted to turnitin.com (other professors were encouraged to use the service and could-and did–set their own policies, but the poli sci department required it). I just googled to try to find some links, but instead I’m just finding that apparently, at least as of Fall 2011, the university still subscribes to turnitin.com and encourages its use…
LikeLike
Well, they haven’t turned to European history yet, so I’ve been free of this silly request.
True story: I’m teaching our capstone seminar this semester, and in class today, one student asked if I read all of everyone’s paper. I think they were surprised when I said I did, and not only that, a few of them would be read by other faculty as part of our assessment work. It didn’t occur to me until now that someone had read about the robograders.
LikeLike
The very low price of $250 for two complete sets of essays makes me wonder who owns the essays that teachers send in for robo-grading. This must be in the contracts schools sign and my cynical self supposes that Pearson et al. own them.
LikeLike
Shoot! What about all that AP grading many of us have undertaken to repay our student loans?
LikeLike
They have indeed contacted European historians. I got the same query a couple of weeks ago. Great post, Historiann.
LikeLike
The whole idea is incredibly creepy.
LikeLike
Much as I grumble about grading, I hardly think it’s a step forward to put student writing into a machine for analysis. I’d rather experiment with peer-editing!
There are so many tasks we could be using technology for – allowing students to collect and analyze geo-located data, serving up micro-tutorials on key subjects (so that everyone can review that key lesson on how to create a useful footnote!) and more. Why try to say it will replace the helpful insight of the educator when it comes to marking? That’s simply laughable!
LikeLike
“This system has been successfully introduced into a number of course areas, and Pearson is now bringing this technology to the history market.”
“The history market” — that’s my favorite part. How many academics think of their department as a “market”? Oh, Pearson…
LikeLiked by 1 person
I’ll just add that computer evaluation implies that the purpose of marking papers is strictly about assessment, rather than about providing useful feedback to students in the form of comments or advice.
The value of marking papers is not only in assessing student performance.
LikeLike
Thanks, all. Great ideas here.
One thing I forgot to note, and I’m kind of surprised that none of the rest of you historians have made this point either: how can a software package possibly account for the appropriate use of primary souces, which frequently (at least in my period and field) are not in standard English nor use standard English spelling? So if students of mine (for example) quote a typically fragmentary and/or run-on 16th-18th source, as they must if they expect to get a decent grade on my assignments, then what will a computer grader do?
I agree with Contingent Cassandra and (apparently) Marc Bosquet: assigments that can be graded by a computer are worth about as much as that computer’s time, and little more. Complexity must be required and rewarded, in my view.
LikeLike
“Complexity must be required and rewarded, in my view.”
Pearson says: Passive voice. C-minus.
LikeLike
It should also be understood that this is also going to create one of those deliciously perverse “why am *I* not on the Nixon Enemies List?” situations. Viz’t: “I haven’t heard from Pearson yet; what do they think I am, chopped liver?” Appropriate to recall, I think, as yet another of the A-list Waterburglars has moved on into that great Watergate Apartment Complex in the sky.
LikeLike
Maybe your SPAM canner is just set at a higher level, Indyanna! I know Pearson hit colleagues yesterday at Northern Colorado and CSU-Pueblo, so maybe they’re taking a regional or state-by-state approach?
I liked your idea about thousands of servers doing all of our peer review for us. Except that that’s the kind of reading of other people’s work that I find really interesting and satisfying! Grading u/g essays: not so much, but I still think it’s important to evaluate essays for their content & not just their sentence structure.
LikeLike
Great comments! Grading essays is a method of teaching. As I tell my students, “good history IS good writing,” but beyond grammar I point out where they need to push their analysis to its logical end, how to see all the implications of an argument, where they contradict their own evidence without being aware of it, etc. This is, of course, hideously inefficient, labor-intensive, and costly. So we get back to rigid class boundaries: the rich can have my personal services, but the poor must content themselves with a machine.
LikeLike
Once we buy this hardware, though, it has to have work. So maybe we can kick it upstairs another notch and have it coordinate deanly, provostial, and presidential searches? Like put a hundred of these machines on chairs in an auditorium where “Candidate A” outlines hir vision-thingie and have the black boxes fill out response forms.
Other more serious thoughts, some of which may be implicitly included in previous comments, are that having assignments “graded *for* you” but necessarily submitted *by* you might have the same valences as taking an article from the Am. J. Sociology in the stacks and incorporating it into a keynote speech at the next big conference. Especially if we take this “scholarship of pedagogy” and “pedagogy [as] scholarship” hoo-ha seriously. And people should read their employment contracts carefully to make sure they are not precluded from taking supplementary compensation from external parties for work they are specifically assigned to do for the institution. (Which I would distinguish from professional functions or activities that you are expected to do but orchestrate yourself, such as writing a book and receiving modest royalties).
LikeLike
These soul dead assholes are of the same tribe that have reduced my day to endlessly punching radio buttons on a screen like some kind of lobotomized test chimp, producing 2 pages of anti literate gibberish that will never again see human eyes, when 2 well written sentences would more than suffice to justify being paid for the work done. Worse yet, I don’t even get the automated arm giving me a kibble of monkey chow as a reward for clicking enough radio buttons.
LikeLike
It’s all the wire monkey mother’s fault!!! She made the educrat class.
LikeLike
A Spanish prof. just told me that Pearson sells Skype language courses. A student enrolled in University of Somewhere will be taking a Skype course with a Spanish-language instructor in some completely different location and Pearson makes a profit.
LikeLike
???
What’s the point of preserving public higher ed if it’s just going to serve as a shell corporation rather than teaching its own courses?
This all seems super-scummy to me. (Super-scammy, too.) I keep thinking about those parents and families that Jonathan Rees mentioned in his first comment. They send their children to an accredited public institution, and find out that said institution can’t be bothered to offer a freaking SPANISH CLASS?
LikeLike
Great post, and great responses!
Quoth Tony Grafton:
Great post, heartening responses. They can have my red pen when they take it from my cold, dead hand. And not a moment sooner (fragment).
I’ll second this. I’ll gladly go down swinging over the value of my intellectual labor as a professor, as well as a researcher in a qualitative field.
LikeLike
Great post. I don’t have much to add, although I did include a few thoughts and a link to your post, Historiann, here:
http://www.medievalrobots.org/2012/04/brief-note-on-robo-grading.html
LikeLike
My fear is that students wouldn’t be bothered by this. They might actually believe a computer to be “more objective” as an evaluator of their work. I say this after watching too many commercials for on-line schools that actually pitch their product as an improvement to traditional higher ed institutions by saying things like “don’t you want to take charge of your education instead of sitting in some lecture?”
What concerns me is that the folks at Pearson think there is a market for this, and I think that is based on some research. It is the same market that they pitch all those textbooks with “instructor’s manuals” to, which are there to teach you how to teach your class. Which I find very problematic. But since they keep publishing them, I’m guessing someone is buying…
LikeLike
I agree with the sentiment Tom and Northern Barbarian raised; the time and effort I put into grading papers is about so much more than just checking the accuracy of the content. I give feedback for improving their writing, as well as point out how they could improve their arguments, amongst other things. I know most of them probably don’t pay any attention to it, but I hope some of them leave my class just a little more skilled as writers, scholars and (dare I say it?) critical thinkers.
LikeLike
Agree and agree again about the class division in institutions that this encourages, among other evils, Historiann. One civil disobedience possibility: historians hold a “plagiarize my blog” day in which everyone writes a wildly incorrect but perfect 6 essay and invites students to help themselves in submitting it.
LikeLike
ej wrote: “My fear is that students wouldn’t be bothered by this.”
You might be surprised. I spend a lot of time talking to students and asking them about what they think higher education should look like. Their answers are not (generally) different from mine. They desire knowledge. They want to be challenged. With only one or two exceptions over the years, they believe the best and truest education is in the classroom with a professor, and they want their professor to grade their assignments.
I started having this conversation with students when my institution moved away from its liberal arts mission. I couldn’t help it. I would attend the kill-liberal-arts meetings and would end up passionately arguing and passionately weeping. The meetings would end, and I would have five minutes to get into the classroom where students were curious about why I was so upset. So I told them. I have continued this conversation semester by semester, now with students who didn’t see me weeping my way into the classroom.
When I turned back a set of freshmen papers this semester, apologizing for the delay it took me to grade all 100 of them (two sections), a couple of students actually came up to me afterwards to thank me for the care I took in grading them and commenting on them. These two were not history majors and both, I believe, received Bs instead of the coveted As. And, OK, that is only 2 out of 100, but it gladdened my heart.
There certainly are plenty of students here who probably don’t care, but I am always pleased and heartened by those who do. Of course I am usually talking to (a) history majors and (b) students who are engaged in some form of social justice activism, so I do understand that my sample is not very good.
LikeLike
My experience is like yours, ProfSweddy: many students appreciate our work with their writing and they let us know. (Many probably just look at the grade and if it’s fine with them, we don’t hear from them.)
I find that my students for the most part have high expectations of the faculty who teach them. Whereas there is probably some level of acceptance of cheating or plagiarism among their fellow students, I think my students would be shocked and offended if they thought I’d just feed their essays into a scantron instead of reading and responding to them myself.
LikeLike
Historiann, have you seen this? The line from the Pearson vp is precious. http://www.nytimes.com/2012/04/23/education/robo-readers-used-to-grade-test-essays.html?_r=1
LikeLike
Historiann, I don’t usually comment here because I’m not an academic. But I went to college, and I remember my very first college writing assignment. It was jointly assigned by the English professors and the subject-matter professors in our core course (odd, yes, and I think the first time they’d tried this approach).
We students all did our research and wrote our papers. When we got them back, they were marked with minor spelling and grammar corrections, and a letter grade, and that was it.
We all complained. We’d been in honors classes in high school. We’d worked hard to qualify for college. We wanted to do well. Where was the feedback on ideas, scope of research, ways to improve?
It turned out that the English professors and subject-matter professors had each thought that the other group was going to address the substantive issues, so they’d confined their own evaluation to basic corrections. They went back and gave us in-depth evaluations.
If our papers had been graded by machine, we would probably have been up in arms.
LikeLike
Funny story, Wogglebug. I’m glad your faculty stepped up and held your class’s feet to the fire.
LikeLike
I got the same e-mail just now, only the stipend they offered me was only $200. Does that have to do with my status or the status of my university? Or are they haggling over price in the way Churchill was reported to?
LikeLike
Pingback: Lisa's (Online) Teaching Blog
An important part of the contribution of many modern historians is the verification or dismissal of earlier historical accounts through reviewing newly discovered sources and recent scholarship or through parallel disciplines like archaeology.
LikeLike
Pingback: What We’re Reading: May 3, 2012 - American Historical Association