Risk-Averse Approximate Dynamic Programming with Quantile-Based Risk Measures
研究了有限时域马尔可夫决策过程中以分位数风险度量（如VaR和CVaR）为目标的风险厌恶问题，提出了数据驱动和基于模拟的近似动态规划算法，并通过重要性采样提高风险区域采样效率，在能源存储投标应用中验证了效果。
In this paper, we consider a finite-horizon Markov decision process (MDP) for which the objective at each stage is to minimize a quantile-based risk measure (QBRM) of the sequence of future costs; we call the overall objective a dynamic quantile-based risk measure (DQBRM). In particular, we consider optimizing dynamic risk measures where the one-step risk measures are QBRMs, a class of risk measures that includes the popular value at risk (VaR) and the conditional value at risk (CVaR). Although there is considerable theoretical development of risk-averse MDPs in the literature, the computational challenges have not been explored as thoroughly. We propose data-driven and simulation-based approximate dynamic programming (ADP) algorithms to solve the risk-averse sequential decision problem. We address the issue of inefficient sampling for risk applications in simulated settings and present a procedure, based on importance sampling, to direct samples toward the “risky region” as the ADP algorithm progresses. Finally, we show numerical results of our algorithms in the context of an application involving risk-averse bidding for energy storage. The online appendix is available at https://doi.org/10.1287/moor.2017.0872 .