Jason Hutchens: loebner contest

Jason Hutchens: loebner contest

The Loebner Contest is version of the Turing Test which is held annually, and is open to anyone who cares to write a conversation simulator. The rules of the Loebner Contest can be found on the Loebner Prize Home Page.

The 1996 Loebner Contest

Rather suprisingly, my HeX program has won the 1996 Loebner contest, beating four other computer entries. The results are as below (you can read the transcripts of each program by clicking on the name of their author):

  1. Jason Hutchens, Centre for Intelligent Information Processing Systems
  2. Robby Garner, Robitron Software Research, Inc.
  3. Joseph Weintraub, Thinking Software
  4. Thad Crews, private entry
  5. Ed T. Toton III, NecroBones Enterprises

The 1997 Loebner Contest

The standard of Loebner Contest entries rose in 1997. The SEPO program didn't win, and since SEPO is a big improvement over HeX, I take that as a good sign for the future of the competition.

On second thoughts, having now taken a look at the transcripts I can see that the winning program was very clever because it's opening gambit was to ask the judges about something which had been in the news recently (the night before the contest, in fact). I personally feel that SEPO's conversation was more realistic, and that the judges were swayed to choosing David's entry because it could handle conversation about a recent news item rather well. I'm not disparaging his work at all, I just feel that this was the deciding factor, and I find that rather interesting.

The results of the 1997 contest are:

  1. David Levy, Intelligent Research Ltd.
  2. Jason Hutchens and Bruce Cooper, Centre for Intelligent Information Processing Systems
  3. Richard Gibbons, Electronic Games for Education in Mathematics and Science
  4. Robby Glen Garner, Robitron Software Research, Inc.
  5. Robert E. Medeksza, private entry

The 1998 Loebner Contest

In 1998, the Loebner Contest was held outside of the USA for the first time in its history. In fact, it was held in Australia, so I was able to attend in person. Four human beings and six computer programs made up the confederates. Ten judges had twelve minutes to chat to each of the confederates and rank them from most human to least human.

Rather surprisingly, MegaHAL was ranked as the second most human computer program. I entered the contest expecting it to come last, as it uses a technique quite different from normal conversation simulators.

The results of the 1998 contest are:

  1. Robby Glen Garner, Robitron Software Research, Inc.
  2. Jason Hutchens, Centre for Intelligent Information Processing Systems
  3. Joseph Strout, private entry
  4. Gerold Lee Gorman, private entry
  5. Denis Cordier and Veronique Bastin, private entry

The 1999 Loebner Contest

The Loebner Contest was once again held in Australia in 1999, this time in the "sleepy town" of Adelaide. Once again I attended in person. NonI, the program I entered, came a disappointing fourth, which I put down to a lack of training on my part. The program needed to be trained from scratch: it learnt everything during this period, including how to simulate typing. I just didn't put in enough of an effort.

The results of the 1999 contest are:

  1. Robby Glen Garner, Robitron Software Research, Inc.
  2. David Hamill, private entry.
  3. Chris Johnson and Sandy Johnson, private entry
  4. Jason Hutchens, Centre for Intelligent Information Processing Systems
  5. Bradley Custer, private entry
  6. Gerold Gorman, private entry

The 2000 Loebner Contest

I was livid when I saw the results for the 2000 Loebner Contest, which was held at Dartmouth College, the site of the famous summer workshop which initiated the field of Artificial Intelligence. My submission, eBrain (an improved version of SEPO) came second. However, the contest organisers deemed to use data files which I'd submitted several months before the contest, instead of the ones I'd submitted just before the contest, on the grounds that they hadn't had time to test them! The result is that my 2000 entry rather embarassingly resembles my 1997 entry, although that was not intended to be the case. Rather, I'd submitted the 1997 data files so the new executable could be tested by the organisers while new data files, which make use of the new eBrain features, were being developed.

The results of the 2000 contest are:

  1. Richard Wallace, private entry.
  2. Jason Hutchens, Amristar Pty. Ltd.
  3. Artificial Life, Artificial Life
  4. Michael Onofrio and Stephen Hildebrand, private entry
  5. Chris Johnson and Sandy Johnson, private entry
  6. Gerold Gorman, private entry

The 2001 Loebner Contest

In 2001, the Loebner Contest will be held at the London Science Museum on Saturday October 13.