Reinforcement Learning for Bitcoin Trading
A personal note on my research comparing PPO and DQN for Bitcoin trading across different market conditions.
In 2025, I had the opportunity to turn one of my interests in software and financial markets into an academic research paper.
Together with Sumanto, Indra Chaidir, and Adi Supriyatna, I published “Reinforcement learning for bitcoin trading: A comparative study of PPO and DQN” in Jurnal Mandiri IT, Vol. 14 No. 2, pages 159–169. The paper was published on August 22, 2025, and is available through DOI 10.35335/mandiri.v14i2.455.
What we studied
The main question was fairly straightforward: how do two deep reinforcement learning approaches, Proximal Policy Optimization (PPO) and Deep Q-Network (DQN), behave when applied to Bitcoin trading?
Bitcoin is an interesting environment for this kind of experiment because its price can move through very different market regimes. A strategy that behaves well during a strong uptrend does not necessarily behave the same way when the market becomes sideways or unstable.
Our experiments used hourly BTC/USDT data covering 2019 to early 2025. Instead of feeding the models raw price data alone, we built the trading state from several technical indicators, including RSI, MA20, volatility, MACD, volume trend, SMA200, and a weekly trend filter.
The goal was not to build a system that could magically predict Bitcoin's next move. The focus was on observing how different reinforcement learning agents learn to make trading decisions under changing market conditions.
PPO vs. DQN
One of the interesting parts of the research was seeing how differently the two algorithms behaved.
PPO tended to trade more aggressively. During bullish phases, this behavior could translate into stronger performance because the agent was more willing to participate in the market. The trade-off was greater exposure to unstable conditions.
DQN behaved more selectively. It generally made fewer decisions and showed more stable behavior when the market was sideways or choppy.
That difference was more important to me than simply asking which algorithm was better. The experiment suggested that the characteristics of an agent can interact heavily with the market regime it is operating in.
In other words, reinforcement learning for trading is not just about choosing a more sophisticated algorithm. It is also about understanding the environment in which that algorithm is expected to operate.
What I learned
This research changed the way I think about the intersection between software engineering, machine learning, and financial systems.
A trading system is not simply a prediction model wrapped in an API. There are many layers involved: data preparation, feature construction, environment design, reward formulation, training, evaluation, and finally the interpretation of results.
The work also reinforced an important lesson about backtesting. A model can appear impressive in one market condition and behave very differently in another. That makes it dangerous to interpret a single performance result as proof that a strategy will work in the future.
For me, the most interesting result was therefore the difference in behavior between PPO and DQN rather than a simple winner. PPO and DQN showed complementary characteristics across different market regimes, which is a useful perspective when thinking about adaptive trading systems.
The paper
This was my first published research work in this area, and it sits at an interesting intersection of several things I enjoy: programming, data, machine learning, and financial markets.
The full paper is available here:
Prasetyo, R. E., Sumanto, S., Chaidir, I., & Supriyatna, A. (2025). Reinforcement learning for bitcoin trading: A comparative study of PPO and DQN. Jurnal Mandiri IT, 14(2), 159–169.
This article is a personal summary of the research rather than a replacement for the full academic paper.