This is the 1st chapter of the 11th 'Behind The Cloud' series:
The Data Engine - How AI Funds Sense Markets
Omphalos’ long-term development has reinforced one lesson: in live markets, robustness beats cleverness.
Data is where robustness begins.
This series continues the Behind The Cloud mission: to share research-based insights into what truly drives AI investing, beyond buzzwords, beyond demos, and always grounded in real-world constraints.
Trust good (!) data, not just AI.
The Market Sensor Stack - What an AI Fund Actually Observes
Most discussions about AI in investing start with models, forecasts, and trading decisions. But before any decision can be made, a fund must answer a simpler question.
What is it actually observing.
Markets are not a single stream of prices. They are an environment made of many signals, many delays, many distortions, and many partial truths. A professional AI system therefore does not “predict markets” in the abstract. It builds a view of reality from a set of sensors. It senses, then it decides.
But sensing is not only about what the system sees today. It is also about what it has seen before. Without coherent historical data, even the best models cannot learn stable relationships, stress behaviors, or regime transitions. In practice, the sensor stack only becomes useful when each input exists as a consistent, point-in-time history, clean enough to train on and repeatable enough to trust.
This chapter maps that sensor stack. Not as a list of datasets, but as a practical framework for how an AI hedge fund forms situational awareness, and why robust sensing matters as much as any model.
The First Sensor, Price Is the Simplest Truth, and Still Not the Whole Truth
Prices are the most accessible market sensor. Their real value for AI, however, comes from long, coherent histories, because learning market behavior is fundamentally a historical exercise, not a snapshot. They are continuous, standardized, and directly connected to tradable outcomes. For decades, quantitative finance treated them as the dominant source of truth.
But “price” is not a trivial concept. The same instrument can have multiple valid price representations depending on the use case: adjusted or unadjusted series, bid, ask, mid, last, official closes, indicative quotes, or OTC prints. Each choice changes the statistical properties of the time series, the apparent volatility, and sometimes even the direction of signals. If the definition shifts over time, models do not learn the market, they learn the dataset.
Prices are also not pure information. They are an output of market structure. They embed liquidity conditions, execution frictions, order flow, and the behavior of other participants. In stable regimes, prices behave as if they are a clean reflection of fundamentals. In unstable regimes, prices behave like the product of mechanics.
A data engine that only sees prices is not blind, but it is incomplete. It observes the market’s surface, not its internal state.
Volatility and the Volatility Surface, The Market’s Built-In Fear Gauge
If prices are the market’s surface, volatility is its nervous system.
Volatility signals how uncertain the market is about itself. It captures clustering behavior, regime transitions, and the tendency for calm to breed complacency. Options markets add another layer, the volatility surface, skew, and term structure represent how participants price tail risk and how that pricing changes across horizons.
But here too, “volatility” is not a single clean number. Options markets are fragmented and often less liquid than the underlying, bid-ask spreads can be wide, and the choice of marks, mid versus last versus theoretical, can materially change the inferred surface. Even the Greeks are not uniquely defined, they depend on the model assumptions and calibration choices used to compute them.
For an AI system, this is not just a forecasting input. It is context. It helps answer whether the environment is smooth or discontinuous, whether liquidity is stable or fragile, whether the market is in a regime that tends to trend, revert, or gap.
This is why many robust systematic systems treat volatility as a first-class sensor, not a derived statistic.
Microstructure and Execution Conditions, The Sensor Most Funds Ignore
A key difference between toy models and production systems is that production systems must trade.
Execution conditions matter. Spread, depth, order book dynamics, and the stability of price formation determine whether a signal is implementable. Markets can appear tradable in a backtest and become untradeable in reality when spreads widen or liquidity thins.
Microstructure is the sensor that tells you whether the market can absorb your decisions. It is the difference between a signal being correct in theory and profitable in practice. Just as important, microstructure data is only useful if it is available as a long, coherent history. Many microstructure feeds are short-lived, inconsistent across venues, or change their definition over time. Without stable historical coverage, models do not learn market behavior, they learn a temporary data artifact.
For AI funds, this sensor is critical because it helps distinguish between an information edge and an execution trap.
Macro and Fundamentals, Clean Data That Arrives Late
Macroeconomic releases, corporate fundamentals, and policy decisions are often viewed as clean data. They are published, standardized, and widely followed.
They are also delayed.
The challenge is not only lag, it is revision. Economic indicators are frequently updated after publication. Corporate fundamentals change through restatements and corporate actions. In a data engine, the question is always point in time. What was known when. What was revised later.
Just as important, methodology and coherence matter. Many macro series and accounting definitions are not stable across decades. Calculation methods change, baselines are rebased, classifications shift, and reporting standards evolve. If the system treats these series as perfectly consistent through time, it does not learn the economy or the company. It learns the changing measurement system.
Macro and fundamentals remain valuable sensors, but their value depends on disciplined handling. If point-in-time truth is not enforced, these sensors become a leakage channel.
Text and Language, The Most Powerful and Most Dangerous Sensor
Text is now one of the fastest growing inputs in systematic investing.
Earnings call transcripts, filings, central bank speeches, news streams, and corporate communications contain information that markets digest in messy, qualitative ways. Natural language processing allows this information to be structured into features and signals.
But text only becomes a reliable sensor when it exists as a long, coherent history. Many modern text feeds and vendor products simply did not exist ten years ago, or changed coverage, formatting, and availability over time. Without consistent historical depth and stable definitions, models do not learn market language. They learn the artifacts of a changing dataset.
And text is also a high-risk sensor. It is noisy, context-dependent, and vulnerable to narrative bias. It changes meaning across regimes. It can be manipulated. And it can mislead models if the input pipeline lacks governance.
Large language models amplify both the opportunity and the risk. They can summarize and interpret text at scale, but they can also produce fluent narratives that feel correct even when they are wrong. This is why LLM-driven workflows must be gated by source quality and retrieval integrity. Text is not just a sensor. It is an uncertainty amplifier if not controlled.
Positioning and Flows, Seeing the Market’s Behavior, Not Just Its Prices
Another sensor layer is market behavior. Positioning proxies, flow indicators, dealer inventory dynamics, and crowding measures help infer what other participants are doing.
These sensors are rarely perfect. Many are indirect. But they are valuable because they answer questions that price alone cannot. Are moves driven by fundamentals or forced unwinds. Are correlations compressing because risk is being reduced simultaneously. Is liquidity being provided or withdrawn.
Just as importantly, unified methodology is crucial. Many positioning datasets change definitions, coverage, and calculation rules over time. The COT reports (Commitments of Traders) are a good example, their methodology and classifications have evolved across decades. If a model treats such series as perfectly consistent history, it risks learning the measurement changes rather than true positioning dynamics.
In an AI-driven market structure, this layer becomes more important because it helps detect feedback loops before they fully unfold.
Alternative Data, A Sensor Layer With High Cost and High Fragility
Alternative data expands the sensor stack beyond traditional financial feeds. Web signals, satellite imagery, supply chain proxies, app activity, and other sources can add unique context.
But alternative data is not automatically edge. It is often expensive, difficult to validate, and prone to proxy leakage. Its reliability can decay quickly once a dataset becomes popular. Many alternative data projects fail not because the data is useless, but because it cannot be operationalized safely.
Just as importantly, alternative data only becomes useful when it exists as a long, coherent history with a stable methodology. Many datasets change coverage, collection methods, vendor definitions, or sampling frequency over time. Without consistency across the full history, models do not learn the underlying phenomenon. They learn the dataset’s changing measurement process.
In the sensor stack, alternative data is best treated as an optional layer, valuable when it improves context or risk control, dangerous when it is treated as magic.
The Core Challenge, Sensing Is Not Collecting
A modern AI hedge fund can observe many things. The real challenge is not collecting inputs. It is making them usable. Usable also means historically coherent. If a sensor’s definition, coverage, or quality changes over time, the model does not learn the market, it learns the dataset.
Every sensor comes with distortions. Some are fast but noisy. Others are clean but delayed. Some are continuous. Others are sparse. Some are stable. Others are regime dependent.
A robust system must therefore do three things at all times.
This is why a sensor stack is not a shopping list. It is an architecture.
Omphalos Perspective
At Omphalos, the sensor stack is treated as a living part of the platform, not a static library of datasets. We assume that every sensor can fail, drift, or become misleading. That applies in two dimensions: in real time, and across history. If a sensor’s historical record is not consistent, the system cannot build repeatable behavior, and repeatability is the foundation of trust. We therefore build the data engine around point-in-time discipline, continuous monitoring, and controlled fusion across sources.
We also take seriously the new role of language models. LLMs are powerful at structuring unstructured inputs, but they must be bounded by retrieval discipline and source governance. A model that speaks confidently is not a model that is correct. In investing, the cost of confident error is real.
The guiding principle remains unchanged.
Trust good (!) data, not just AI.
Supporting research & news
Next week we will publish the second chapter of this series: "Point-in-time reality - What the system could actually have known'
If you missed our former editions of "Behind The Cloud", please check out our BLOG.
Omphalos Fund won the "Funds Europe Awards 2025" in the category "European Thought Leader of the Year".
Omphalos Fund won the "EuroHedge Awards 2025"
© The Omphalos AI Research Team - June 2026
If you would like to use our content please contact press@omphalosfund.com