How AI "Broke" Itself and What the Consequences Will Be

Why AI Is Still Imperfect

First, neural networks still hallucinate, and many have noticed this — for example, researchers at Duke University. This cannot be fixed, as it is a feature of artificial intelligence. AI experts — former SAP CTO Vishal Sikka and his son Varun Sikka — note that modern AI systems make mistakes and require human oversight.

This is because a typical ChatGPT strives to give the user an answer even in cases where the model lacks sufficient information. That is precisely why responses may contain errors, distorted facts, or even fabrications. Because of this, neural network responses have to be verified by humans. This means more and more staff are needed to correct AI inaccuracies. Many see progress in this development path — I do not.

Another problem is that neural networks have no transparency — no one knows how a particular answer was generated. The neural network simply selects what it considers the most appropriate set of words for a response. Therefore, errors can be passed on. Especially now, when neural networks are increasingly used to train other models.

Uncontrolled AI development is fraught not only with errors but also with uncertainty. In a few years, neural networks could become smarter than even the most outstanding people, writes Dario Amodei, CEO and co-founder of Anthropic, which developed the Claude family of language models, in his essay. What the consequences of such development will be is still unclear.

Another problem is security risks. A Cisco report states that attackers can intercept data during transmission between a neural network and other systems. According to one study mentioned in the report, 83% of organizations plan to implement agentic AI into their business processes, but only 29% believe they are ready to use such systems securely. This indicates that the risk of confidential information leakage remains high.

I believe that AI should not yet be implemented in sensitive areas — for example, in finance. A neural network can produce erroneous answers, and this is unacceptable. Added to this are other problems: AI can change filters set by humans, and the security level of neural networks is insufficient. For banks and brokerage companies, such risks are too high.

Where Neural Networks Could Lead Us

If everything continues to develop as it is now, AI could hit a technological dead end. Neural networks will continue to make mistakes, meaning they still cannot be used in areas where the cost of error is too high — in finance, for example.

Many today are enthusiastic about AI agents. They are expected to trade independently on the stock exchange, write code, manage logistics, and perform other complex tasks. But, in my view, they inherit the hallucinations of the underlying models because they rely on centralized data that may become outdated or distorted.

If AI is embedded into a familiar tool, such as a calculator, it would likely make mistakes even on simple calculations. Of course, developers are constantly training models to reduce the number of inaccurate responses. But, as I see it, it is impossible to eliminate them completely.

And this is the main risk of future neural networks. That is why large-capital owners will not entrust their finances to an investment fund that uses AI. Yes, there are already financial market participants actively implementing neural networks, but this could end in serious problems. An AI agent could successfully execute many operations and then make one mistake that leads to multi-million dollar losses.

As far as I can see, investment fund managers understand this. From the owners of large capital, there is a clear demand: zero tolerance for the use of public neural networks in operational processes. This is because no one wants to bear responsibility for AI errors.

How AI Can Be Improved

In my view, AI needs to be migrated to more resilient systems based on distributed ledger technology. One of its most well-known variants is blockchain.

Blockchain works as follows: information is not stored on a single computer but across many devices connected in a network. To change data, the consent of all network participants is required. Without this, changes do not take effect. At Otkritie, we used this approach when working with sensitive information.

Modern neural networks are structured differently. They do not use distributed ledger technology; information is stored centrally. Because of this, changes can occur without coordination with all interested parties. For some areas, this approach is unacceptable.

I believe that an upgraded AI built on a distributed ledger could play an important role in development — particularly in the Russian digital financial assets market. In this area, such solutions should be tested first. At the same time, such a neural network should only be used for part of the tasks, not to replace humans entirely.

One often hears fears that AI development will lead to mass layoffs. For example, many associate staff reductions in American technology companies with the implementation of neural networks. But I believe these events are not directly related. This happened for several reasons, including accumulated problems in corporate governance.

Moreover, in February, the US saw an increase in job openings in the financial sector. Banks and brokerage companies are looking for specialists who can develop new corporate strategies and assemble teams for the safe implementation of AI. This shows that neural networks will not replace people but rather change the requirements for specialists.

What Skills Will Be Useful for Working with AI in the Future

It is obvious to me: in ten years, AI will remain only for routine operations that require zero emotion and empathy. And even then, only neural networks combined with a distributed ledger will be used. Companies that hand over almost 99% of their processes to AI will ultimately face financial collapse. This collapse will manifest in different ways, but first and foremost — through a decline in market capitalization.

If this scenario materializes, demand for crisis managers who can restructure companies will increase. Therefore, it is necessary to learn the following:

Development and creation of new modifications of distributed ledger technology.

Building neural networks from scratch based on distributed ledger technology, particularly blockchain.

Development of new systems — not just AI on a distributed ledger, but genuine intelligence.

It is also necessary to be able to apply different variants of distributed ledger implementation. One corporate blockchain for everything is a path to nowhere. When we at the Innovation Factory within the Otkritie ecosystem studied different ways of implementing distributed ledger technology, we built a mix of different variants to achieve the goals that shareholders needed.

As I see it, distributed ledger modifications are evolving beyond the alternatives known to the general public. Some of them have much stronger characteristics than blockchain and are optimally structured for developing and implementing corporate neural networks using such a ledger.