The American Interest
Analysis by Walter Russell Mead & Staff
Domo Arigato, Professor Roboto

professor roboto

Students, meet your robot professor. EdX, an educational consortium founded by Harvard and MIT we’ve written about before, has just released new software that grades student papers automatically. The NYT reports:

The EdX assessment tool requires human teachers, or graders, to first grade 100 essays or essay questions. The system then uses a variety of machine-learning techniques to train itself to be able to grade any number of essays or answers automatically and almost instantaneously.

The software will assign a grade depending on the scoring system created by the teacher, whether it is a letter grade or numerical rank. It will also provide general feedback, like telling a student whether an answer was on topic or not.

EdX is making this system available for free online to all schools that want it, and four states are already using a similar program in public high schools. Some like the new technology because it can provide students with instant feedback when human grading can take weeks. Others don’t:

One longtime critic, Les Perelman, has drawn national attention several times for putting together nonsense essays that have fooled software grading programs into giving high marks. He has also been highly critical of studies that purport to show that the software compares well to human graders.

“My first and greatest objection to the research is that they did not have any valid statistical test comparing the software directly to human graders,” said Mr. Perelman, a retired director of writing and a current researcher at M.I.T.

Via Meadia is withholding judgment on this specific implementation, as we haven’t had a chance to play with it. We’ll be enrolling a staffer in an EdX course to have them try to game the system so we can see what all the fuss is about.

And we understand Mr. Perelman’s skepticism. Machine learning has been an area of study for computer scientists for a long time now, and with computers getting ever faster, there have been huge strides made in areas such as data mining and pattern recognition. But these breakthroughs are a far cry from actual artificial intelligence, and as Mr. Perelman’s advocacy group correctly points out, “computers cannot ‘read’”.

Yet.

Nevertheless, tools like this have a definite place. The technology will probably never be a full substitute for a human teacher, but as a force multiplier, it could be quite useful. Today’s state-of-the-art algorithms could be good enough for grading short answers in online courses. And in a more traditional university setting, the technology could be used to free up professors for one-on-one mentoring by doing a first pass at grading a large stack of papers.

Either way, this is just another example of the technological disruption that is transforming many industries today, from law to medicine to manufacturing. Whether or not the EdX software itself catches on, innovations like this are going to become an important part of education’s future. There’s no use fighting the inevitable.

[Robot photo from Shutterstock]

Published on April 6, 2013 10:28 am
  • http://www.facebook.com/alexander.scipio Alexander Scipio

    Since we are lowering the workload of these profs, I’m sure they’ll agree it’s only fair to commensurately lower their salaries, right? Being the fair-minded liberal social justice elites that they are I’m certain they’ll agree. Right?

  • Jim Luebke

    Computers will not be able to detect creativity or original thought, and will in fact downgrade papers that contain such things as “inaccurate”.

    If you think that one of the problems in higher education is that untalented students cynically parrot what the professor and TA’s want to hear, putting a robot in the loop isn’t going to help matters.

  • Andrew Allison

    Perelman’s is a straw argument. While it would be possible, if the rules were know, to get a good grade from a nonsense essay, how likely is it that a student would do so? Assuming that the first 100 are human-graded each year, it seems unlikely that, even if they wished to, students would be able to beat the system.

  • JackLifton

    Here’s an example of what would happen:

    Student: I believe that we live in a 4-dimensional, not a three dimensional universe, and that the speed of light is a constant no matter what the relative frame of reference in which our laws of motion are used in calculation.

    Robot: Not correct; False premise

    Grade: F or “0″

    It is recommended that Mr Einstein be put back to an earlier class to learn classical mechanics or dismissed from the university entirely..

  • Bill Gryan

    Do you get a better grade if you sleep with it?

  • ROLYAT136

    From the Not-Yet-News Network:

    Graduate assistants, who helped develop this grading
    program, surreptitiously created a program to write college essays.

    Connection to the inter-net allows the algorithm to mine
    data for the essay based on inputs such as subject and desired length. Future
    upgrades are to include an add-on module that allows the writer’s class notes
    to help focus the essay towards the professor’s topics and ideology. A second
    add-on would turn audio recordings of the professor’s lectures into usable
    notes for the essay program.

    Senator Schumer held a press conference denouncing the
    writing program as a “threat” to the “substantive reputation of
    our institutions of higher learning”. Likening its effect to the impact
    that electronic calculators had on the “math I.Q.” of the nation,
    Schumer outlined his proposed regulations to prevent the dissemination of
    “high-capacity writing programs” which he suggested should include
    any program capable of creating an essay greater than 100 words and having
    features such as grammar checking, a thesaurus, or a bayonet mount. Senator
    Warren proposed regulations on the price of such programs and a program to “profit-share”
    the program’s revenues with the Federal Government, commenting that the alleged
    owners of the program had not experienced any material costs in its creation,
    and arguing that, in any event, they could not have built the program without
    the M.I.T. computers hardware purchased by the university with federal grants.
    President Soetoro has suggested that he is willing to be reasonable if the
    “pirates” offering this software agree to a reasonable scheme for
    comprehensive federal government profit sharing. ;-)