Background and Motivation

Cancer patients receiving similar doses of fluorouracil (5-FU) can have very different drug exposure because the rate at which they clear the drug varies. Initial doses are often calculated based on body surface area, which accounts for only part of this variability [1]. Therapeutic drug monitoring measures drug concentrations to help individualize later doses. For infusional 5-FU, exposure is commonly expressed as the area under the concentration–time curve (AUC), with approximately 20–30mg·h/L used as a target range [1, 3].

Because treatment consists of several cycles, the next dose can be chosen based on patient characteristics, previous doses, and measured exposure. Reinforcement learning (RL) learns a policy, or decision rule, for these repeated adjustments. Exploring potentially unsafe doses in patients during training is not feasible. Offline RL therefore learns from a fixed collection of treatment histories without testing new dose decisions in patients [2].

However, these data cover only a limited range of dose adjustments. A learned policy may recommend changes that were rarely or never observed, making their expected benefit difficult to estimate reliably [6, 7]. This study investigates whether a learned pharmacokinetic model can improve offline RL performance and whether restricting dose adjustments to the observed range leads to better exposure control.

Methods

Figure 1 outlines the study workflow, from simulated treatment data and policy training to the evaluation of dose recommendations.

Simulated treatment data

The comparison used three datasets, each containing trajectories from 15,000 virtual patients. A mechanistic pharmacokinetic and pharmacodynamic simulation represented repeated 24-hour or 46-hour infusions. The original dose adjustments followed the Kaldate algorithm [3]. Dataset A estimated AUC from noisy concentration measurements; Dataset B calculated cumulative AUC from model-predicted concentration profiles without residual error. Dataset C used the observation-based AUC with reduced variability in the measurement-noise component.

Pharmacometrics

The model-based methods used a separately trained multimodal pharmacokinetic scientific machine learning model (MMPK-SciML) [8]. It combines patient characteristics and treatment histories with a one-compartment pharmacokinetic model to estimate individual clearance and volume of distribution. The model remained fixed during RL training and predicted exposure under proposed dose adjustments.

For a concentration $C\left(t\right)$ the one-compartment dynamics can be written as

$$\frac{dC(t)}{dt}=\frac{R_{\text{in}}(t)}{V}-\left(\frac{CL}{V}\right)C(t)$$

where $R_{\text{in}}\left(t\right)$ is the infusion rate, $V$ the volume of distribution, and $CL$ the clearance. During infusion, $R_{\text{in}}\left(t\right)=\frac{D}{IT}$, where $D$ is the infusion dose and $IT$ is the infusion duration; after infusion it is zero. A bolus gives $C\left(0\right)=\frac{D_{\text{bolus}}}{V}$; without a bolus, $C\left(0\right)=0$.

Reinforcement Learning

Each treatment cycle formed a decision step. The policy used patient characteristics, cycle number, previous dose, and previous AUC to recommend a relative change to the next infusion dose. Bolus doses were not optimized. The reward favored AUC values in the target interval, with its maximum between 22 and 28 mg·h/L, as shown in Figure 2.

Four model-free methods learned directly from the recorded transitions: behavior cloning (BC) imitated recorded actions [9]; TD3 optimized reward without an explicit offline-data constraint [10]; TD3+BC combined reward optimization with imitation [7]; and conservative Q-learning (CQL) reduced optimistic value estimates for poorly supported actions [6]. MOPO and COMBO additionally used MMPK-SciML rollouts. MOPO penalized uncertain model predictions, while COMBO used a conservative learning objective [4, 5].

Model-based policies were evaluated both with and without action clamping: dose changes were limited to the observed range of [−0.3, 0.2]. Evaluation used five folds with patients separated between training, validation, and testing. Metrics covered target attainment, first entry into range, consecutive in-range cycles, and the direction and effect of dose adjustments.

Results

TD3+BC was the strongest model-free method across all three datasets. The best clamped model-based methods achieved slightly higher mean target attainment compared to no clamping. On Dataset A, the patient-wise mean proportion of cycles in the target range increased from 32.94 %  with the original dosing strategy to 47.50 % with TD3+BC and 48.79 % with clamped MOPO. Figure 3 illustrates how mean AUC changed across treatment cycles on Dataset A for the different dosing methods.

Table 1: Patient-wise mean proportion of treatment cycles with AUC in the range 20–30 mg·h/L. Values are means across five cross-validation folds.

The best model-based methods exceeded TD3+BC by 1.29, 1.85, and 0.20 percentage points on Datasets A, B, and C, respectively. Clamping improved both MOPO and COMBO compared with their unrestricted variants. Their best variants also reached the target range earlier and sustained longer sequences of in-range cycles. TD3 and CQL were less consistent; some configurations performed worse than the original strategy.

Conclusion and Outlook

The study shows that offline RL can improve 5-FU exposure control in simulated patients. TD3+BC provided a strong baseline, while model-based methods offered a modest additional benefit when dose adjustments were restricted to the range observed in the training data. However, these restrictions do not guarantee clinical safety. The policies have not yet been validated on real patient data, and achieving the target AUC range does not directly demonstrate reduced toxicity or improved treatment outcomes.

Future work should evaluate the policies using real 5-FU monitoring data and compare them with established model-informed precision dosing approaches. The next step will focus on extending the MMPK-SciML framework with pharmacodynamic modeling, particularly leukocyte dynamics as an indicator of treatment-related toxicity. This extension could enable the reward function to account for both drug exposure and predicted toxicity.

References

[1] Lee, J. J., Beumer, J. H., & Chu, E. (2016). Therapeutic drug monitoring of 5-fluorouracil. Cancer chemotherapy and pharmacology, 78(3), 447-464. https://doi.org/10.1007/s00280-016-3054-2

[2] Levine, S., Kumar, A., Tucker, G., & Fu, J. (2020). Offline reinforcement learning: Tutorial, review, and perspectives on open problems. arXiv preprint arXiv:2005.01643

[3] Kaldate, R. R., Haregewoin, A., Grier, C. E., Hamilton, S. A., & McLeod, H. L. (2012). Modeling the 5-fluorouracil area under the curve versus dose relationship to develop a pharmacokinetic dosing algorithm for colorectal cancer patients receiving FOLFOX6. The oncologist, 17(3), 296-302. https://doi.org/10.1634/theoncologist.2011-0357

[4] Yu, T., Thomas, G., Yu, L., Ermon, S., Zou, J. Y., Levine, S., ... & Ma, T. (2020). Mopo: Model-based offline policy optimization. Advances in neural information processing systems, 33, 14129-14142. 

[5] Yu, T., Kumar, A., Rafailov, R., Rajeswaran, A., Levine, S., & Finn, C. (2021). Combo: Conservative offline model-based policy optimization. Advances in neural information processing systems, 34, 28954-28967.

[6] Kumar, A., Zhou, A., Tucker, G., & Levine, S. (2020). Conservative q-learning for offline reinforcement learning. Advances in neural information processing systems, 33, 1179-1191.

[7] Fujimoto, S., & Gu, S. S. (2021). A minimalist approach to offline reinforcement learning. Advances in neural information processing systems, 34, 20132-20145.

[8] Valderrama, D., Teplytska, O., Koltermann, L. M., Trunz, E., Schmulenson, E., Fritsch, A., ... & Fröhlich, H. (2025). Comparing scientific machine learning with population pharmacokinetic and classical machine learning approaches for prediction of drug concentrations. CPT: Pharmacometrics & Systems Pharmacology, 14(4), 759-769. https://doi.org/10.1002/psp4.13313

[9] Hussein, A., Gaber, M. M., Elyan, E., & Jayne, C. (2017). Imitation learning: A survey of learning methods. ACM Computing Surveys (CSUR), 50(2), 1-35. https://doi.org/10.1145/3054912

[10] Fujimoto, S., Hoof, H., & Meger, D. (2018, July). Addressing function approximation error in actor-critic methods. In International conference on machine learning (pp. 1587-1596). PMLR