What if AI could imagine the market's next move?

I run an AI hedge fund. With nearly two decades of experience applying AI across financial markets and other domains, I’m fortunate to lead a team with similarly deep expertise. Together, we design and operate sophisticated AI systems that trades fully autonomously, and most importantly, profitably.

It's sometimes frustrating to read about the latest developments in AI - the hype, the problems - and being unable to openly discuss the work we are doing. These blog posts are meant to change that somewhat, to communicate some of our thinking, some of what we've achieved and also to discuss where we think AI is going, including the (still changing) path to AGI - the term we use for human-level artificial intelligence.

And, like all good blog posts, I'm going to tell a story.

The story begins in 1943. In that year, Kenneth Craik, a philosopher and psychologist, published a book called, "The Nature of Explanation". In this book, he talks about "mental models" (although the idea appeared even earlier than this in other texts). What this proposes, is that we all form models of the world, of people, relationships, events and so on, in order to think, plan and act more effectively. These models are not perfect, but they are relevant. What we are in fact talking about is the ability to imagine possibilities.

Some of the best technology has come from attempts to mimic nature. And of course, AI is no exception. All AI systems are to some degree, attempts to mimic some part of the thinking processes we all take for granted every day. Whether that be making predictions about things like the weather, grouping similar things together, or working out how to move a (robot) arm.

And, not surprisingly, mental models are part of the AI 'toolbox' also. But we give them a different name, we instead call them World Models. The strict definition is not exactly the same, but is close enough. World Models also have a long history in computer science and AI. An example of a World Model is a chess playing system. Here the game is clearly structured and so rules can be explicity written in computer code. Other examples may be less clearly defined, there may be uncertainty, and so rules need to be discovered implicitly.

But anyway, back to trading. Years ago, when first designing our system, we understood that financial markets are uncertain. Regimes change, sometimes slowly, sometimes rapidly. Information is noisy. And so the system would have to be hugely adaptable.

Very deep and complex neural networks - at the heart of Large Language Models (LLMs), the systems dominating the press today - do not actually work well on this type of problem. The constantly changing nature of markets means whatever the models learnt before is not necessarily valid in the future.

So we again looked at nature and created a two-tier system. Those of you who have read "Thinking Fast and Slow" by Daniel Kahneman will know about System 1 (fast, intuitive) and System 2 (slow, analytical). This is, broadly speaking, how our system handles the immense challenges we throw at it.

One part of the system handles the constantly changing input data - this can be time or language/text based.

The other part takes that information stream and converts it into abstract rules and relationships. These are not fixed rules however. Just like a person's beliefs can change, our system will change its mind also. Some relationships change often, and some don't change much at all.

Readers might be wondering what is the point of all of this. Maybe it would have been better to just take a simpler model and train it somehow to predict financial market returns. While this certainly would have meant less work for us, it also gives you a system that is very narrow in focus, and less capable in terms of what it can do. We call such systems brittle. They are not robust to changes and don't generalise well to new situations.

On the other hand, a system that creates a World Model for financial markets (or a set of Mental Models), which are abstract, based on cause and effect, and handle uncertainty, is much more powerful and flexible in terms of what it can do:

1. The first is that it can imagine different possibilities. For example, what if X had changed, what would have happened to Y? This is called a counterfactual and is a key part of causal reasoning, something humans do very well, but LLMs struggle with.

2. The second is it can learn more with less data. LLMs are notoriously data hungry - the public discussion at the moment is on available data running out. Our system takes raw data in and compresses it into a more abstract, efficient, form, something that can be used to think about markets. For those of a more technical mindset, this process is related to entropy methods.

3. The third is it can generalise more effectively. This means what it has learnt for one domain or set of problems can be carried over to another. It is a more general purpose thinking machine.

The system we have been running - for years now - is indeed this World Model for financial markets. Something which we’ve noted, related to point 3, is that embracing uncertainty and using the above framework means our system is more stable, not less.

Also it's important to note that trading strategies are built on top of this engine (with a significant amount of additional work), while the general purpose reasoning and insights sit beneath them as part of a foundational layer. I guess you could call it a foundational model for markets.

I haven't described everything in the platform, like our own Generative Models for time and language for example - we give some brief descriptions of those at the end of the post together with other details.

Last but not least, and separate to the hedge fund, we have been preparing to make the reasoning platform more broadly available to different users - through chat interfaces and insights in a web application. This will happen through a sister company, called Cognisca, whose mission will be the continued development and distribution of a general purpose reasoning AI for finance and other business domains. Call it the fintech arm of our firm.

More details on this venture - and more posts - will be available soon. In the meantime feel free to subscribe for updates.

www.cognisca.com

For the Curious Minds

For those readers who are interested, we give some additional technical details here - but avoiding the use of too much technical jargon.

There is a particular name for the above system - it's called neuro-symbolic as it combines elements of statistical models (system 1) with elements of symbolic models (system 2). What we are trying to do with this architecture is to both mimic models of human thought, in a somewhat crude way, and to address the weaknesses of each part (statistical and symbolic). There is long-running debate about what kind of AI will achieve AGI or human-level intelligence. For some of the reasons laid out here (and some others), we believe neuro-symbolic approaches are likely to be a key part of achieving this.

The system includes so-called Generative Models for both time domain data and language data. Basically the system can update itself based on time series and on text inputs. It can also generate both of these forms of data. The language model learns grammatical structures and transforms these into causal understanding - updating our causal data structures directly. The techniques we use are different to an LLM - they are more efficient in their data usage, more explainable, and do not suffer from hallucinations. We intend to further develop these as part of the Cognisca product offering.

Explainability is a key consideration for our systems. We work in a highly regulated industry and trust is a related consideration. We'll probably write more about that in another post, but suffice to say, we tend to create systems from combinations of simpler building blocks (which in turn combine to give complexity), and provide for explainability at the micro and then macro level in the resultant architecture.

We've developed experience across much of the AI field - including unsupervised, supervised, and reinforcement learning, as well as deep learning architectures (like LLMs) and also less fashionable parts of the AI toolbox, like rule engines and ontologies. Our approach has always been to focus on the best techniques for a particular problem rather than on the most fashionable ones. LLMs and related technologies are very powerful machines, and have their place in performant architectures (including our own), but also have their limitations, which have already been published about extensively (hallucinations being one of them).

But, this is what makes the field of AI exciting, as no-one can really claim to yet know what the final AGI architecture will be.