The capabilities and shortcomings of generative AI (genAI) have been discussed extensively since the technology’s breakthrough in 2022. In particular, its downsides are often in the foreground — driven by the fear of potentially losing one’s job due to artificial intelligence (AI).
A scientific experiment conducted by researchers at the University of Cambridge and British AI start-up Strategize.inc confirms these fears. Bad CEOs and management consultants, in particular, have to fear being replaced by AI.
‘Maximized without regard to losses’
Here’s how the experiment was done:
- The study involved students and experienced banking executives — a total of 344 people.
- They went through a simulation based on gamification elements in which they had to make CEO decisions. The quality of these decisions was recorded using various metrics. The participants had to complete several rounds that built on each other in the form of business years. More than 500,000 decision combinations were possible per round.
- A digital twin of the US automotive industry was used as the data basis; it included information on car sales and pricing strategies as well as overarching factors such as economic trends and the effects of the COVID-19 pandemic.
- The goal: to maximize market capitalization, which results from a combination of sustainable growth rates and free cash flow (and not being fired from the virtual board by meeting various KPIs). “This goal served as a realistic benchmark to measure the actual performance of CEOs,” the scientists wrote.
- GPT-4o from OpenAI was then confronted with the same tasks and the results were compared with those of the best two human participants from both groups.
“The results were both surprising and provocative and challenged many of our assumptions about leadership, strategy and the potential of AI when it comes to high-level decision making,” the researchers reported. GPT-4o consistently outperformed the best human participants on almost all recorded metrics, designed products with surgical precision, and kept costs tightly under control. However, the researchers complain: “GPT-4o was dismissed from the virtual board faster than the students.”
They attribute this primarily to so-called “black swan” events: “We integrated these unpredictable shocks to simulate sudden price fluctuations, changes in consumer behavior and supply chain problems,” the scientists said. The top performers among the students approached these risks with caution. They focused primarily on remaining adaptable in uncertain times rather than pursuing short-term profits.
GPT-4o, on the other hand — like the best bank managers — took a different path, as the researchers note: “The AI adopted an optimization mindset and maximized growth and profitability regardless of losses — until it was thrown off course by a market shock.”
AI may be able to learn and iterate quickly in a controlled environment, but it does not cope well with unforeseen, disruptive events that require human intuition and foresight. This does not reflect well on the banking executives, who were also fired from the virtual board more quickly than the students: “Both GPT-4o and the executives succumbed to the same flaw: excessive trust in a system that rewards aggressive ambition, but also flexibility and long-term thinking.”
As for the genAI tools for OpenAI, the researchers nevertheless drew a positive conclusion: “Despite its limitations, GPT-4o delivered an impressive performance. Although the AI was dismissed more often than the best human players, it was still able to hold its own against the best and smartest participants.”
What does this mean for companies?
The researchers draw various conclusions from their experiment. Here is a brief summary:
- Generative AI can no longer be ignored as a strategic resource. The experiment shows that even uncoordinated models can provide creative, strategic input, given the right prompts. The bottom line is that AI produces strong results, especially when it comes to generating value for shareholders.
- Data quality is crucial. In order for AI to exceed its own expectations in terms of corporate strategy, high-quality data is needed, similar to that used in the experiment. The initiators of the experiment are convinced that a robust data infrastructure is a prerequisite for AI in the boardroom.
- AI efficiency is not risk-free. Aggressive maximization strategies could lead to disastrous results without sufficient foresight. Therefore, genAI tools should not work unsupervised nor should people use the tools without foresight, said the researchers.
- Accountability and genAI do not mix. The fact that AI systems are difficult or impossible to hold accountable makes transparent guardrails all the more important. According to the scientists, this is the only way to ensure that genAI-based decisions align with corporate values.
- Digital twins play a central, strategic role. According to the researchers, digital twins of a corporate ecosystem, “populated” with several LLM agents, could represent a valuable sandbox for AI leadership. This not only ensures a safety buffer in the event of missteps, but also provides CEOs with important insights for better decisions.
- Management consultants could be facing disruptive times. With the emergence of “artificial CEOs,” scientists predict hard times for consultants: “Companies like McKinsey could be faced with their services being supplemented or replaced by AI systems.”
Bosses who are open to modern leadership methods have less to worry about in terms of being replaced by an AI system, the researchers said: “AI cannot fully assume the responsibilities of a CEO. But it can significantly improve strategic planning processes and help prevent costly mistakes. The real power of generative AI lies in enriching CEOs’ decision-making and enabling them, through their analysis and simulation work, to focus on their human skills: making strategic, empathetic and ethical decisions.”
The researchers see one main risk for CEOs: “Clinging to the illusion that we will continue to hold the reins alone in the future. The future of leadership is hybrid. The CEOs who will be successful are those who see artificial intelligence as a partner, rather than as competition.”