Verbal IQ of a Four-Year Old Achieved in AI System

Tuesday, June 11, 2013


 Artificial Intelligence
Using an open-source common-sense reasoning-based AI system, researchers at the University of Illinois at Chicago have tested the ConceptNet system and found it to be at the verbal level of an average four-year old child.

R esearchers at the University of Illinois at Chicago have tested a rule-based artificial intelligence system and found it to be at the verbal level  of an average four-year old child.

The research has been published online as part of a submission to the 2013 Commonsense Reasoning Symposium.

One view of common-sense reasoning ability is that it is the ability to perform those tasks with verbal inputs and outputs that have traditionally been difficult for computer systems, but are easy for fairly young children. For the test, the researchers administered the verbal part of the Wechsler Preschool and Primary Scale of Intelligence (WPPSI-III, Third Edition) to the MIT-developed ConceptNet 4 system.

ConceptNet, currently on version 5.1, is a semantic network containing lots of things computers should know about the world, especially when understanding text written by people.

It was built from nodes representing concepts, in the form of words or short phrases of natural language, and labeled relationships between them. These are the kinds of things computers need to know to search for information better, answer questions, and understand people's goals.

For instance, ConceptNet contains everyday basic knowledge:
You would learn because you want to know more information.
Cultural knowledge:
A saxophone is used for jazz.
And scientific knowledge:
Plutinos can cross Neptune's orbit.
It would not adequately represent human knowledge if it didn't contain other languages besides English, as well:
本は紙でできている。 (A book is made of paper.)
To be precise, ConceptNet a hypergraph, meaning it has edges about edges. Each statement in ConceptNet has justifications pointing to it, explaining where it comes from and how reliable the information seems to be.

Related articles
Previous versions of ConceptNet has been distributed as idiosyncratic database structures plus some software to interact with them. ConceptNet 5 is not a piece of software or a database; it is a graph. It's a set of nodes and edges, which we can represent in multiple formats.

The IQ test’s questions like,“Why do we shake hands?” or “What do apples and bananas have in common,” were translated into ConceptNet 4 inputs using a combination of simple natural language processing tools that come with ConceptNet together with short programs they wrote. The question-answering primarily used the part of the ConceptNet system that represents the knowledge as a matrix based on spectral methods (AnalogySpace).

What the researchers found that the system has a Verbal IQ that is average for a four-year-old child, but below average for 5, 6, and 7 year olds. Large variations from subtest to subtest indicate potential areas of improvement. In particular, results were strongest for the Vocabulary and Similarities subtests, intermediate for the Information subtest, and lowest for the Comprehension and Word Reasoning.

Four Year Old

Comprehension is the subtest most strongly associated with 'common sense'.

According to the researchers, Stellan Ohlsson, Robert H. Sloan, György Turán and Aaron Urasky, children’s verbal IQ tests offer a new, objective, third-party metric for the evaluation and comparison of common-sense AI systems.

The researchers found that general why questions, including both the common-sense kind and factual why questions, such as IBM's Watson answered for Jeopardy!, are a known difficult problem in question answering, a field at the intersection of information retrieval, natural language processing and human-computer interaction.

The authors speculate that improvements in those areas could improve the results to average for a five or six year old child, but that something altogether new would be needed to answer Comprehension questions (from an age appropriate test) with the skill of a child of eight.  Some of these capabilities may already be worked into the current version of ConceptNet.

Still, it is remarkable that common-sense AI software has come this far.

Correction - 07.08.2013
This article originally said the research was conducted at the University of Chicago. It has since been updated and corrected to the University of Illinois at Chicago.

SOURCE  Commonsense 2013, Photo -

By 33rd SquareSubscribe to 33rd Square