The paper focuses on a number of practical mistakes but overlooks some higher-level ones that I think are also important:
- Most ML algorithms perform well when there is some underlying phenomenon whose characteristics are being statistically inferred by the model. However, the stock market is now essentially a large network of computers trying to model one another. You can imagine how this might break some of the underlying assumptions that good ML results rely upon. We can see the results of this in the increasing frequency of "flash crashes" caused by over-leveraged quant hedge funds all tripping over sell triggers and/or getting margin calls at the same time.
- In practice, asset class/trading strategy correlations tend to change dramatically over time. Nowadays we talk about the market being 'risk-on' or 'risk-off' modes, since we're used to seeing market-wide selloffs and buyins. Your ML model can only do so much if the entire market is selling everything, unless you're going to dip your toes in derivatives or short selling to which I say: good luck! :)
- A major, major issue is execution. You can have the greatest and most accurate model in the world, but actually trading it is another beast entirely. Bid/ask spreads and market movements from your trading, particularly if you are dealing with a non-trivial amount of money or trading securities that have less than ideal liquidity, is usually going to eat up any alpha you might have. In my own backtests of fairly straightforward trading algorithms even a minor 0.05% spread on bid-ask spreads for a weekly trading algorithm can eat your lunch, nevermind if you're planning on doing intra-day trading or trading anything other than the most popular funds/stocks.
- Beyond any of these risks, you're going to have to inevitably suffer through downdrafts. I don't know about you, but watching my money disappear as it's being controlled by a trading algorithm/model that is subject to all kinds of mistakes and bugs is well beyond my own intestinal fortitude.
"Most ML algorithms perform well when there is some underlying phenomenon whose characteristics are being statistically inferred by the model. However, the stock market is now essentially a large network of computers trying to model one another."
This is one of the insights that I think deserves more ink than it gets. Historically system's analysis has focused on predominantly linear systems where underlying factors that were unseen but capable of being modeled as FSM combined into emergent behaviors which were more complex but ultimately still linear across the region of analysis.
That sort of analysis falls down however when the modelling system is a part of the system it is modelling. The result can easily become non-linear (or turbulent or chaotic depending on when you were introduced into systems analysis :-)) and the calculus for those conditions is a lot harder to tease out.
The feedback algorithms become signals to other feedback algorithms and you get hard to predict changes which don't track the measured data, they track the response to the measured data, which is itself changing in response to the response.
From a practical viewpoint (for me) its of interest in discovering entities that use ML techniques to game search engine rank which is itself being derived by ML algorithms.
Indeed -- it opens up a (scary) need for models to be "self-aware" of competitors' models and their effects on the training data. Above my pay grade, for sure.
Great point... I've been thinking lately how it must be possible to hack the systems now. E.g. there must be machine analysis of press coverage on equities, and the machines must deduce, quickly, "positive" or "negative". This means fake stories have a lot of power. Now this was always true, of course... but traditionally you had humans vet what seemed legit. Now I bet you can keyword stuff articles, among other things, to shift the market... etc who knows but sky's the limit.
- Most ML algorithms perform well when there is some underlying phenomenon whose characteristics are being statistically inferred by the model. However, the stock market is now essentially a large network of computers trying to model one another. You can imagine how this might break some of the underlying assumptions that good ML results rely upon. We can see the results of this in the increasing frequency of "flash crashes" caused by over-leveraged quant hedge funds all tripping over sell triggers and/or getting margin calls at the same time.
- In practice, asset class/trading strategy correlations tend to change dramatically over time. Nowadays we talk about the market being 'risk-on' or 'risk-off' modes, since we're used to seeing market-wide selloffs and buyins. Your ML model can only do so much if the entire market is selling everything, unless you're going to dip your toes in derivatives or short selling to which I say: good luck! :)
- A major, major issue is execution. You can have the greatest and most accurate model in the world, but actually trading it is another beast entirely. Bid/ask spreads and market movements from your trading, particularly if you are dealing with a non-trivial amount of money or trading securities that have less than ideal liquidity, is usually going to eat up any alpha you might have. In my own backtests of fairly straightforward trading algorithms even a minor 0.05% spread on bid-ask spreads for a weekly trading algorithm can eat your lunch, nevermind if you're planning on doing intra-day trading or trading anything other than the most popular funds/stocks.
- Beyond any of these risks, you're going to have to inevitably suffer through downdrafts. I don't know about you, but watching my money disappear as it's being controlled by a trading algorithm/model that is subject to all kinds of mistakes and bugs is well beyond my own intestinal fortitude.