mead cohen berger shevtsova garfinkle michta grygiel blankenhorn
The Rise of the Robo-Prof

Whe IBM’s “Watson” supercomputer made its debut on Jeopardy last year, few realized that it would mark the beginning of a sea change in the American workforce. Shortly after its debut Watson computers became available for diagnosing patients, and last Month IBM announced that Watson computers would begin a foray into the banking sector as well. Similar changes are underway in the law practices, where other powerful computers are increasingly used for data review of case documents.

As powerful new computer technologies move into white-collar workforces, one has remained relatively untouched—higher education. Yet a new report from Inside Higher Ed (h/t Marginal Revolution) suggests that this may be about to change. A new study at the University of Akron found that despite the doubts of English professors, a new computer program can grade student essays just as well as their human counterparts:

In the quantitative sense: yes, according to a study released Wednesday by researchers at the University of Akron. The study, funded by the William and Flora Hewlett Foundation, compared the software-generated ratings given to more than 22,000 short essays, written by students in junior high schools and high school sophomores, to the ratings given to the same essays by trained human readers.

The differences, across a number of different brands of automated essay scoring software (AES) and essay types, were minute. “The results demonstrated that over all, automated essay scoring was capable of producing scores similar to human scores for extended-response writing items,” the Akron researchers write, “with equal performance for both source-based and traditional writing genre.”

This is bad news for professors and TAs who thus far have managed to resist the forces of automation that have transformed manufacturing and are beginning to transform the professions. But it is good news for the rest of students and parents.

It is much too early to forecast how technologies like this will change the face of higher education. But cash strapped public universities, facing declining contributions by state governments and hammered by public discontent over rising tuition, are looking hard for ways to cut costs and improve efficiency. For profit companies are also looking for ways to compete more effectively with the existing model. And students and parents are looking for ways to get job credentials without being saddled with unbearable debt.

Change will come, and faster than you think.

Features Icon
show comments
  • Petar

    Reproducing the mean and standard deviation of grades is not the same as “can grade student essays just as well as their human counterparts”.

    I do not see why this is positive. If teaching can be automated (and it will be, just not in the following 2 decades), then all the jobs that teachings prepares you for can be automated as well. See where this is going?

  • Chase

    Wow, we are [lost]. Case closed.

  • Lorenz Gude

    When I went to school high school placed the emphasis on indoctrination while university education was was clearly more intended to produce adults who could think for themselves. The archetypal joke about this difference involved the Freshman who complained at year’s end that his professor had destroyed everything he believed in. The professor’s rejoinder was that Hercules was required to clean out Pegasus’ stall, but so far as he knew,
    was not required fill it up again.

    Since higher education has become largely indoctrination in politically correct memes I see no reason why machines can’t be used to check that only approved ideas are expressed in essays. I’m sure the Holy See will find these machines equally useful to expunge heresies. Pfui!

  • Mrs. Davis

    This is bad news for professors and TAs

    How so? I have met few of either who enjoyed teaching, let alone grading papers. That has to be one of the most time consuming and hated tasks in a teachers schedule. They should be overjoyed to have their time freed up to conduct more research. Or if they actually like teaching, to have time to consider who the outliers are and how to get them to improve their performance; gentle encouragement or kick in the pants.

    Where this goes is time for humans to spend being human instead of automatons. I don’t know a lot of farmers who wish they were standing behind a nag instead of riding in a quarter million dollar Deere. It’s time the “professions” learned the power of capital intensity. Look what it did for the US Army. Maybe they’ll even get rid of their slave labor, sorry, I mean TAs and FYPs.

  • Luke Lea

    I’d like to see a few concrete examples of machine grading. Frankly I’m having a hard time imagining a machine recognizing true quality in an essay. Semantics is key, and so far Google hasn’t figured out how to answer even simple questions.

  • Luke Lea

    [Quote] “If you go to a business school or an engineering school, they’re not looking for creative writers,” Shermis says. “They’re looking for people who can communicate ideas. And that’s what the technology is best at” evaluating.

    Sounds like the premise of a SNL routine.

  • Vilmos

    Fully agree with Petar.

    Also, if the scoring can be automated, then why not the writing?

    I don’t see a problem when tedious repetitive things are automated. But here a human touch is needed and this is what is eliminated.

    Yesterday I was at the playground with my daughter. Sometimes I do carry a newspaper or my ebook reader to read while she is playing. But whenever she wants me, I am always willing to put down the paper of the ereader. I saw a father with two children. The children were playing, and daddy was playing too. On his “smartphone”. His little daughter (5-6 years?) yelled excitedly “Daddy, I did it”, and daddy, without looking up and stopping playing with his toy yelled back “Way to go!”. I don’t have a smartphone (heck, not even a cellphone), and this is why I don’t want to have a smartphone. It ruins human relationships.

    Computers and automations are good things. But they turn bad when they are applied in places where the human touch is needed.

    I believe Einstein’s phrase covers this as well: Make everything as simple as possible but not simpler.


  • peter38a

    Fiddlesticks… Mmm, whatever they may be. “…can grade student essays… ” what does that mean exactly?

    Graded by what criteria? What great essays from the past and the future might have been, will be trashed by a computer? Shall the essayist write not to humans but to the computer? Will there be programs that teach you how to write for computer evaluation?
    I know a lady who “teaches” creative writing but she knows she doesn’t at all. She teaches “don’t do’s” and all the creative parts, all the enduring word smithing, etc, comes from the person.

    In 1917 a book, rather complete actually, was written that would help develop fictional stories: “Ploto” I believe was the name. I read some time ago that Hollywood had a program that would predict the “likeability” of a movie, i.e. heroine a blond instead of red haired, two gun fights instead of three… etc. And the outcome; naught!

    Are we to expect great literature, paintings and photographs to next emerge from a program someday and a machine that executes same? Will these computers be able to tell jokes?

    One of the first groups totalitarians attempt to coop or destroy are artists because they are driven to “tell” enduring human truths. When writing a computer program you have a goal. To write an “art” program wouldn’t the programmer have to know all the great truths of humanities past and future? One might wonder if such a person would want to invest time writing computer programs? I can already see the hammer descending on me for this last statement but I will be so delighted to defend it. LOL

  • mezzrow

    How would we recognize the ragged front edge of the “singularity” if we happened to be standing on it? The whole premise is the unimaginable nature of the pace and direction of change.

    Don’t blink.

  • Brett

    This educational computing doesn’t strike me as a replacement for teaching. It’s just a more sophisticated version of automated grading, like how computerized checking of scantrons has replaced a lot of the tedious grading-by-hand of multiple choice tests.

    Where these computers would really be useful, though, would be in rooting out plagiarism. There’s already a system that can check essays for similarities with older ones, if I recall correctly. This could just make it much, much faster.

  • Jim.

    Pop quiz, guys: What is the ultimate answer to life, the universe, and everything?

    (If you don’t know, ask Google. If the first fifty plus hits all come up with the same answer, it must be right, mustn’t it?)

    Bottom line: Computer software will NEVER be capable of doing discovering true creativity, quality, or value because that sort of software is impossible to test / validate.

    The good news? Computers can check your spelling. Computers can check your references. They can make sure the meaning of any given word you use a lot in your paper bears at least some relation to the other words you use in your paper.

    The trouble is, if computers can (and do) grade your papers, you’re going to get a pirated copy of that software code and dot the I’s and cross the T’s until the code tells you you get an “A”, whether you put any actual thought into the paper or not. Clever pirates will reverse-engineer the grading software and create scripts that will spew out A-grade papers that make syntactical sense, but may or may not be absolute gibberish from the point of view of actual ideas.

    That’s probably going to become a game amongst college students…. write the weirdest paper, the most obviously nonsensical paper, that your grading software will still rank with an “A”.

    Come to think of it, a lot of students already do this with their humanities courses, which could explain why the study Mead cites turns out like it does.

  • Vilmos

    I have the feeling that these kinds of automatizations are one of the reasons for the decline in the quality of education. I remember that at university (undergrad) we had to buy something called Scranton Paper for our multiple choice tests. We didn’t have much interaction with the instructors. We were just numbers who filled out multiple choice tests which was scored with machines. So much for “education”.

    My mother said that when she was a university chemistry student, the profs invited those students to their homes who were about to go abroad to study. The professor made sure that the students actually had table manners. She also said that her prof toured their labs twice a day. He knew exactly what was going on. Now that was a real prof who actually was an educator and not only a paycheck collector. Compare that to the profs I had for whom we were nothing but numbers (SSN) who filled out multiple choice papers. Barely an essay or even “worse”, oral exam.

    I think that the only way to test that a student actually knows the stuff is through oral exams. Written essays are fairly useful. But multiple choice tests are a sign of a lazy professor who doesn’t take the effort to engage with his/her students.

    When things like multiple choice tests, standardized tests, the lack of oral exams, go away, education will have a chance to recover.


  • Lorenz Gude

    I think there is a clear consensus in these comments that this ain’t gonna work. In so far as education teaches known knowns, we can do this sort of thing – maybe better than we think given the rather disappointing current state of computer ‘intelligence’. But ultimately education is a gift of love that one generation gives to the next. Without caring, without passionate caring, there is no way our posterity can stand on our shoulders. To merely stuff their heads with what we currently think are known knowns is to profoundly mistrust them and fails to recognize that each one of us, now and always, is a singularity.

  • Shu

    I don’t think there is a clear consensus that it won’t work. I think that people are pointing out strengths and weaknesses of AI grading, and I don’t think they are entirely wrong; Mrs Davis and Brett make excellent points. The fact is that these services are time savers that will allow teachers to do a better job. I’d argue that this statement isn’t accurate though: “The differences, across a number of different brands of automated essay scoring software (AES) and essay types, were minute.”

    MI’s PEG, Project Essay Grade, came out way ahead on every measurement in the study. That particular software is being marketed internationally for English Language Learners, but it’s available for k-12 and higher ed. It’s important to understand that it’s not being marketed as a teacher replacement. It augments and speeds up the writing process, and when it sees that consistent mistakes are being made, it offers tutorials.

© The American Interest LLC 2005-2016 About Us Masthead Submissions Advertise Customer Service