📊 Full opportunity report: Minerva. The opposite path. on ThorstenMeyerAI.com — validation score, market gap, and execution plan.
TL;DR
Italy’s Minerva-3B, a sovereign LLM trained from scratch on half Italian data, achieved a low 4.9% score on Italian school exams. This challenges assumptions about the scale needed for effective language-specific models.
Italy’s Minerva-3B, a large-scale language model trained entirely from scratch on 2.5 trillion tokens with approximately 50% Italian content, scored just 4.9% on the INVALSI Italian school-exam benchmark, highlighting fundamental challenges in achieving country-specific language proficiency even with significant investment.
The Minerva project, led by Sapienza University of Rome and supported by Italy’s National Research Council and the CINECA supercomputing consortium, involved training models from 350 million to 7 billion parameters. Despite the large-scale training and open release of weights and data, Minerva-3B’s performance on academic content tests was near chance, with only a 4.9% score on the INVALSI benchmark, a standard measure of Italian student achievement.
Researchers emphasized that while the dataset composition and training methodology are important, the overall size of the dataset and the number of parameters are more crucial for handling complex language tasks. The results suggest that even substantial native-language investment may not be sufficient at current model scales to produce deep country-specific knowledge, raising questions about the effectiveness of current sovereign-LLM strategies.
Minerva.
The opposite
path.
Italy spent years building a European sovereign LLM from scratch. Then Minerva-3B scored 4.9% on the INVALSI Italian school exam.
Where AMÁLIA layered Portuguese specialization onto a multilingual foundation, Minerva trained from scratch on 2.5 trillion tokens with approximately 50% Italian content. Where AMÁLIA’s weights are not yet public, Minerva published weights, training data, and code as truly-open from day one. By every institutional measure, the Italian approach worked. But the empirical results contain a finding the press coverage has been quiet about — and it has implications that extend well beyond Italy.
Same problem. Opposite path.
European sovereign-LLM development has two primary architectural approaches. Italy chose from scratch with substantial native-language foundation. Portugal chose continuation pre-training of a multilingual model. The structural comparison surfaces what each commitment actually requires operationally.
The comparison is not “Italy did it better than Portugal.” Both projects respond to the same structural problem with different architectural strategies under different institutional and economic constraints. Italy’s national-AI investment is structurally larger by an order of magnitude — and Minerva is the visible artifact of that scale.
large language model training datasets
As an affiliate, we earn on qualifying purchases.
As an affiliate, we earn on qualifying purchases.
4.9% on INVALSI. The bitter lesson surfaces.
In June 2024, researchers evaluated Minerva-3B on the Italian school-exam benchmark. The result was unambiguous. This is not a critique of Minerva — it is a critique of the public discourse around what Minerva’s empirical results actually demonstrate.
Engineering a Small AI Language Model: Training, Evaluation, and Deployment Without Myth
As an affiliate, we earn on qualifying purchases.
As an affiliate, we earn on qualifying purchases.
350M to 7B. Four parameter scales, one architecture.
The Minerva model family covers four parameter tiers, each with specific training corpora. Each scale level reveals what the from-scratch path actually requires at different operating points.
Italian + English
100B English
~50% English
+ 200B code
AI model evaluation tools
As an affiliate, we earn on qualifying purchases.
As an affiliate, we earn on qualifying purchases.
Three answers. Same question.
Minerva, AMÁLIA, and OpenEuroLLM represent the three operational answers to the European sovereign-LLM question. Each makes different architectural and institutional bets. The strategic discourse benefits from treating all three as data points in the same empirical experiment.
AI research data management
As an affiliate, we earn on qualifying purchases.
As an affiliate, we earn on qualifying purchases.
Three standards the movement should adopt.
The structural critique generalizes beyond Minerva. The European sovereign-LLM movement benefits from internalizing these lessons across every subsequent national project. Italy modeled the openness standard; the movement should adopt it as norm.
Minerva is one valid answer to the European sovereign-LLM question. AMÁLIA is another. OpenEuroLLM is potentially a third. The strategic discourse benefits from treating all three as data points in the same empirical experiment rather than as competing national-prestige projects. More analysis like this is needed. Not less.
Implications of Scale for National Language Models
The results from Minerva challenge the assumption that training large models on native-language data alone guarantees deep language understanding. The low performance on academic benchmarks indicates that scale—both in data and parameters—is critical for achieving country-specific expertise. This has broad implications for European sovereign-LLM projects, suggesting they may need to reassess their investment levels and strategies to realize meaningful language proficiency and knowledge depth.
European Sovereign LLM Strategies and Challenges
Italy’s Minerva project represents a significant effort to build a European sovereign-language model from scratch, using extensive computational resources and open data. It contrasts with other approaches like Portugal’s AMÁLIA, which relies on continuation training of multilingual models with smaller amounts of native-language data. Despite the large-scale training, Minerva’s performance on complex language tasks remains limited, highlighting the ongoing debate about the optimal approach and necessary investment levels for effective country-specific LLMs.
“Minerva’s low academic benchmark score reveals that current investment levels may still be insufficient for true country-specific language mastery.”
— Thorsten Meyer, AI researcher
Unresolved Questions on Model Scaling and Effectiveness
It remains unclear whether increasing model size further or refining training data strategies will significantly improve Minerva’s performance on complex language tasks. The ongoing research aims to determine the thresholds necessary for achieving meaningful country-specific expertise, but definitive conclusions are still pending.
Next Steps in Sovereign-LLM Development and Evaluation
The Minerva team plans to continue iterating on training methodologies, including ongoing experiments with continual training and larger models. Future evaluations will focus on whether increased scale can bridge the performance gap on academic and complex language benchmarks. Broader discussions within the European AI community are expected to reassess investment strategies based on these findings.
Key Questions
Why did Minerva perform poorly on the Italian academic tests?
The low score suggests that even with large-scale training on native-language data, current model sizes may not be sufficient to develop deep language understanding and country-specific knowledge.
Does this mean training from scratch is ineffective?
Not necessarily. The results highlight the importance of scale, but ongoing research will determine whether larger models or different training strategies can improve performance.
How does Minerva compare to other European sovereign LLMs?
Minerva’s approach involved training from scratch with extensive native-language data, contrasting with models like Portugal’s AMÁLIA, which used continuation training. Performance disparities suggest scale and investment levels are critical factors.
What are the implications for European AI policy?
The findings indicate that achieving effective country-specific models may require substantial scaling investments, prompting policymakers to reconsider funding and infrastructure strategies.
Will increasing model size solve the performance issues?
It remains uncertain. While larger models tend to perform better, the current results suggest that scale alone may not be enough without careful data curation and training techniques.
Source: ThorstenMeyerAI.com
