OpenAI has unveiled SimpleQA, a new benchmark tool designed to measure how accurately artificial intelligence language models can answer factual questions. The tool aims to address one of artificial intelligence’s persistent challenges: ensuring AI systems provide factually correct information.
The Benchmark
SimpleQA contains more than 4,300 questions covering diverse topics from science and technology to entertainment and video games. The benchmark focuses specifically on short, fact-seeking queries that have clear, verifiable answers. This approach makes it easier to assess the accuracy of AI responses compared to evaluating longer, more complex text generations.
The development process involved multiple layers of verification. AI trainers created questions following strict criteria: each question needed to have an indisputable answer, remain valid over time, and be capable of triggering incorrect responses from existing AI models. A second independent AI trainer then answered each question without seeing the original response, and only questions where both trainers’ answers matched were included in the dataset.
To verify the benchmark’s reliability, OpenAI conducted additional testing using a third AI trainer who answered 1,000 randomly selected questions from the dataset. The results showed a 94.4% agreement rate with the original answers. Of the 5.6% disagreements, half were attributed to grader errors or human mistakes, while the other half stemmed from ambiguous questions or conflicting source information. This testing process established an estimated error rate of approximately 3% for the dataset.
Testing Advanced Language Models
OpenAI has used SimpleQA to evaluate several of its language models, including GPT-4o and o1-preview. The testing revealed that larger models generally performed better than smaller ones, which typically demonstrated less world knowledge. Interestingly, models designed to spend more time processing information, such as o1-mini and o1-preview, more frequently chose not to attempt answers rather than risk providing incorrect information.
The benchmark also measures model calibration – how well AI systems assess their own knowledge accuracy. This is done by having models state their confidence levels in their answers and comparing these to their actual performance. Testing showed that while larger models demonstrated better calibration, all models tended to overestimate their accuracy, indicating room for improvement in this area.
Limitations
While SimpleQA provides a structured way to evaluate AI factuality, it has limitations. The benchmark only tests responses to short, specific questions with single verifiable answers. Whether this ability correlates with accuracy in generating longer, fact-filled responses remains an open research question.
OpenAI has made SimpleQA available as an open-source tool, inviting researchers to use it for evaluating language models and to provide feedback for future improvements.
