RL traders that lose money on purpose, slowly

What it does

TensorTrade is a Python framework for building and training reinforcement-learning agents to trade financial markets. It provides composable pieces—environments, action schemes, reward functions, data feeds—that you wire together to create a custom trading system. The framework integrates with Ray RLlib for distributed training and includes scripts for hyperparameter tuning via Optuna.

The interesting bit

The authors actually ran the experiment and published the negative results. Their PPO agent on BTC/USD turned a profit only at zero commission; at 0.1% commission it lost $650 versus buy-and-hold’s $355 loss. The README openly flags “trading frequency reduction” and “commission-aware reward schemes” as priority contribution areas. This is unusually honest for a trading-ML project.

Key highlights

• Modular architecture: Observer → Agent → ActionScheme → Portfolio, with RewardScheme feeding back from portfolio state
• Default action scheme is BSH (Buy/Sell/Hold); default reward is PBR (Position-Based Returns)
• Training stack built on Ray RLlib; includes walk-forward validation and overfitting detection tutorials
• Extensive tutorial curriculum split by audience: trading-for-RL-practitioners, RL-for-traders, or both
• Apache 2.0 licensed, Python 3.11/3.12 required

Caveats

• The project appears to be in a research-heavy, pre-production state: the “best” configuration from experiments still underperforms once realistic costs are applied
• Troubleshooting section suggests NumPy version conflicts and TensorFlow CUDA issues are common enough to document

Verdict

Worth exploring if you’re researching RL in finance and want a structured baseline with honest benchmarks. Avoid if you’re looking for a profitable trading system to deploy; the authors themselves haven’t found one yet.