The AI Revolution has outgrown The Turing Test — Introducing a new Framework.
Context
As AI becomes a transformative part of our technology landscape, a common vocabulary about the capabilities of each new tool and technique is essential. Common vocabularies create shared intellectual spaces allowing all stakeholders to accelerate understanding, increase adoption, facilitate collaboration, benchmark progress, and drive innovation.
So far, the most widely known tool for benchmarking AI is the Turning Test.
However, the field of artificial intelligence (AI) has come a long way since the inception of the Turing Test in 1950. As such, it is becoming increasingly clear that the Turing Test is insufficient for evaluating the full range of AI capabilities that are emerging today — or are likely to emerge in the future.
The Turing Test operates on a simplistic pass/fail basis and focuses heavily on chat/linguistic capability, which is only one aspect of human intelligence. This narrow focus on language ignores many other critical dimensions of intelligence, such as problem-solving, creativity, and social awareness. Additionally, the Turing Test presupposes a level of human-like intelligence that may not be relevant or useful for evaluating AI.
The Framework
To address these limitations, there is an urgent and critical need to develop a more nuanced and comprehensive framework for evaluating AI capabilities across multiple dimensions of intelligence.
This insight led me to develop the “AI Classification Framework”. The ACF is a new approach to evaluating AI capabilities based on the Theory of Multiple Intelligences.
The Theory of Multiple Intelligences was first proposed by psychologist Howard Gardner in 1983. Gardner argued that intelligence was not a single, unified entity but rather a collection of different abilities that could manifest in a variety of ways. Gardner identified eight different types of intelligence: According to Gardner, individuals may excel in one or more of these areas, and each type of intelligence is independent of the others. The theory challenged the traditional view of intelligence as a singular, fixed entity and opened up new avenues for exploring the diversity of human cognition. While the theory of multiple intelligences has been subject to some criticism and debate over the years, it has had a significant impact on the field of psychology and education, particularly in the development of alternative approaches to teaching and learning.
This seemed perfect as a basis for the AI Classification Framework. Following the theory, the framework supports evaluating AI tools across multiple dimensions of intelligence, including linguistic, logical-mathematical, musical, spatial, bodily-kinesthetic, interpersonal, and intrapersonal intelligence.
For each dimension of intelligence, the framework provides a scale from 1–5. 1 being “No Capability” or the equivalent of a human infant and 5 being “Self-agency” or capability that might be considered “Super Intelligence” — beyond human ability.
Examples
The framework itself is a detailed table of descriptions which can be found here. I’ve also created a simple visual representation for easy high-level reference. What follows are two simple examples of the segments that respond to the capabilities of ChatGPT and DALL-E 2.
Use-cases
For founders building AI startups, the Framework represents a methodology to find and communicate (both to potential users and potential investors) opportunities to break new ground and differentiate.
For operators looking to add AI capabilities to their products, the Framework represents a way to find and evaluate potential vendors against their internal requirements.
For investors looking to fund AI startups, the Framework can inform their investment thesis and ensure that they’re making informed bets across the field of possible intelligence and capability levels.
For end-users, the Framework is a tool to help them search for and choose products that might help them be more productive.
Next steps & Acknowledgements
The first version of the framework is intended to kick-start a conversation. I plan to update it as feedback is received from the community.
I also want to thank my friends Ben Parr and Jeremiah Owyang for their helpful feedback on this framework. With their input and the input of the larger AI community, I’m confident that this framework can be a valuable tool in evaluating the full range of AI tools and capabilities present and emerging.
Get the framework
Send feedback to me@chrissaad.com
See my other Startup and Product Management content
Let’s work together
At any given time, I’m working with a small portfolio of startups to help them avoid wasting precious time and money to fast-forward to the best high-growth answers. Let’s work together
In the spirit of using the tools that the framework is intended to classify, the first draft of this post was written by ChatGPT and then edited by me. What a time to be alive!
