Deep Reinforcement Learning for Cryptocurrency Portfolio Management: An Application Using the PPO Algorithm

Abstract

Cryptocurrency markets are a challenging testbed for algorithmic trading due to high volatility and abrupt regime shifts. Using daily Bitcoin (BTC/USD) data for 2018–2024, this study applies an iterative scaling–based data-cleaning procedure to mitigate anomalies in raw exchange data and then trains a Deep Reinforcement Learning agent with Proximal Policy Optimization (PPO). The dataset is split chronologically: the first 80% is used for training and the last 20% for out-of-sample evaluation. In the test period, the PPO agent delivers a total return of 67.35%, while the buy-and-hold benchmark achieves 149.03%. From a risk-adjusted perspective, the agent exhibits positive Sharpe (1.31) and Sortino (1.90) ratios, and its maximum drawdown (-19.88%) is comparable to the benchmark (-20.06%). Overall, the results highlight that improving data quality through robust preprocessing is critical for the stability and generalization of DRL-based trading systems.

Keywords

• International Finance
• Deep Reinforcement Learning
• Algorithmic Trading
• Proximal Policy Optimization
• Portfolio Optimization

Derin Pekiştirmeli Öğrenme ile Kripto Para Portföy Yönetimi: PPO Algoritması ile Bir Uygulama

Abstract

Kripto para piyasaları, yüksek oynaklık ve ani rejim değişimleri nedeniyle algoritmik ticaret için zorlu bir test ortamıdır. Bu çalışmada Bitcoin (BTC/USD) günlük verileri (2018–2024) kullanılarak, ham veride görülen ölçek/anomali sorunlarını gidermek amacıyla yinelemeli ölçek düzeltme temelli bir veri temizleme prosedürü uygulanmış ve Proximal Policy Optimization (PPO) ile bir derin pekiştirmeli öğrenme (DRL) ajanı eğitilmiştir. Veri seti kronolojik olarak ayrılarak ilk %80’lik bölüm eğitim, son %20’lik bölüm ise out-of-sample test için kullanılmıştır. Test döneminde PPO ajanı toplam %67.35 getiri üretirken, buy-and-hold stratejisi %149.03 getiri sağlamıştır. Risk ayarlı performans metrikleri açısından ajan, Sharpe (1.31) ve Sortino (1.90) oranlarında pozitif bir profil sunmuş; maksimum düşüşü -%19.88 ile benchmark’a (-%20.06) yakın gerçekleşmiştir. Ajanın düşüşlerde pozisyon azaltıp nakde geçmesi, sermaye koruması odaklı bir davranışa işaret etmektedir. Bulgular, veri kalitesini iyileştiren ön işleme adımlarının DRL tabanlı ticaret sistemlerinin kararlılığı ve genellenebilirliği açısından kritik olduğunu göstermektedir.

Keywords

• Uluslararası Finans
• Derin Pekiştirmeli Öğrenme
• Algoritmik Ticaret
• Proksimal Politika Optimizasyonu
• Portföy Optimizasyonu

References

1. Baur, D. G., Hong, K., & Lee, A. D. (2018). Bitcoin: Medium of exchange or speculative assets? Journal of International Financial Markets, Institutions and Money, 54, 177–189. https://doi.org/10.1016/j.intfin.2017.12.004
2. DLR-RM. (2025). PPO — Stable Baselines3 documentation (Version 2.8.0a2). Retrieved January 7, 2026, from https://stable-baselines3.readthedocs.io/en/master/modules/ppo.html Stable Baselines3 Docs
3. Dwyer, G. P. (2015). The economics of Bitcoin and similar private digital currencies. Journal of Financial Stability, 17, 81–91. https://doi.org/10.1016/j.jfs.2014.11.006
4. Dyhrberg, A. H. (2016). Hedging capabilities of bitcoin. Is it the virtual gold? Finance Research Letters, 16, 139–144. https://doi.org/10.1016/j.frl.2015.10.025
5. Fama, E. F. (1970). Efficient capital markets: A review of theory and empirical work. The Journal of Finance, 25(2), 383–417. https://doi.org/10.1111/j.1540-6261.1970.tb00518.x
6. Fischer, T., & Krauss, C. (2018). Deep learning with long short-term memory networks for financial market predictions. European Journal of Operational Research, 270(2), 654–669. https://doi.org/10.1016/j.ejor.2017.11.054
7. Ji, Q., Bouri, E., Kristoufek, L., & Lucey, B. (2021). Realised volatility connectedness among Bitcoin exchange markets. Finance Research Letters, 38, 101391. https://doi.org/10.1016/j.frl.2019.101391
8. Jiang, Z., Xu, D., & Liang, J. (2017). A deep reinforcement learning framework for the financial portfolio management problem. arXiv. https://doi.org/10.48550/arXiv.1706.10059

9. Kahneman, D., & Tversky, A. (1979). Prospect theory: An analysis of decision under risk. Econometrica, 47(2), 263–291. https://doi.org/10.2307/1914185
10. Katsiampa, P. (2017). Volatility estimation for Bitcoin: A comparison of GARCH models. Economics Letters, 158, 3–6. https://doi.org/10.1016/j.econlet.2017.06.023
11. Katsiampa, P., Corbet, S., & Lucey, B. (2019). Volatility spillover effects in leading cryptocurrencies: A BEKK-MGARCH analysis. Finance Research Letters, 29, 68–74. https://doi.org/10.1016/j.frl.2019.03.009
12. Liu, Y., & Tsyvinski, A. (2021). Risks and returns of cryptocurrency. The Review of Financial Studies, 34(6), 2689–2727. https://doi.org/10.1093/rfs/hhaa113
13. Lo, A. W. (2004). The adaptive markets hypothesis: Market efficiency from an evolutionary perspective. The Journal of Portfolio Management, 30(5), 15–29. https://doi.org/10.3905/jpm.2004.442611 pm-research.com
14. Markowitz, H. (1952). Portfolio selection. The Journal of Finance, 7(1), 77–91. https://doi.org/10.1111/j.1540-6261.1952.tb01525.x
15. Mnih, V., Kavukcuoglu, K., Silver, D., et al. (2015). Human-level control through deep reinforcement learning. Nature, 518, 529–533. https://doi.org/10.1038/nature14236
16. Moody, J., & Saffell, M. (2001). Learning to trade via direct reinforcement. IEEE Transactions on Neural Networks, 12(4), 875–889. https://doi.org/10.1109/72.935097
17. Nakamoto, S. (2008). Bitcoin: A peer-to-peer electronic cash system. https://bitcoin.org/bitcoin.pdf
18. Raffin, A., Hill, A., Gleave, A., Kanervisto, A., Ernestus, M., & Dormann, N. (2021). Stable-Baselines3: Reliable reinforcement learning implementations. Journal of Machine Learning Research, 22(268), 1–8.
19. Schulman, J., Wolski, F., Dhariwal, P., Radford, A., & Klimov, O. (2017). Proximal policy optimization algorithms. arXiv. https://doi.org/10.48550/arXiv.1707.06347
20. Urquhart, A. (2016). The inefficiency of Bitcoin. Economics Letters, 148, 80–82. https://doi.org/10.1016/j.econlet.2016.09.019
21. Yi, S., Xu, Z., & Wang, G.-J. (2018). Volatility connectedness in the cryptocurrency market: Is Bitcoin a dominant cryptocurrency? International Review of Financial Analysis, 60, 98–114. https://doi.org/10.1016/j.irfa.2018.08.012
22. Zhang, Z., Zohren, S., & Roberts, S. (2020). Deep reinforcement learning for trading. The Journal of Financial Data Science, 2(2), 25–40. https://doi.org/10.3905/jfds.2020.1.030