Iterative Path Integral Method for Nonlinear Stochastic Optimal Control
Oct 16, 2012 6772 views 
So far, we have been studying nonlinear stochastic control. For example, in [1, 2, 3], we have proposed an asymptotically stabilization method based on properties of physical systems such as passivity and invariance for a class of nonlinear stochastic systems. Besides, in [4, 5], we have proposed a stochastic bounded stabilization controller, which renders the state of the plant system bounded in probability for given probability and bounds of the state. The main subject of this talk is nonlinear optimal control, and I would like to introduce our recent research with Prof. Bert Kappen, on extension of the path integral stochastic optimal control method. Nonlinear stochastic optimal control problem is reduced to solving the stochastic Hamilton- Jacobi-Bellman (SHJB) equation. However, it is generally quite difficult to solve the SHJB equation, because it is a second-order nonlinear PDE. The path integral method proposed by Kappen [6] provides an efficient solution for a SHJB equation corresponding to a class of nonlinear stochastic optimal control problems, based on statistical physics approach. Although this method is very useful, some assumptions required in this method restrict its application. To solve this problem, we have proposed an iterative solution for the path integral method in our report [7]. The proposed method solves the SHJB equation iteratively without imposing the assumptions, which are necessary in the conventional method. Consequently, it enables us to solve a wider class of stochastic optimal control problems based on the path integral approach. Since the proposed method reduces to the conventional method when the assumptions hold, it is considered to be a natural extension of the conventional result. Furthermore, we investigate a convergence property of the algorithm.\\ [1] S. Satoh and K. Fujimoto, "On passivity based control of stochastic port-Hamiltonian systems," in Proc. 47th IEEE Conf. on Decision and Control , 2008, pp. 4951-4956.\\ [2] --, "Passivity based control of stochastic port-Hamiltonian systems," Trans. the Society of Instrument and Control Engineers, vol. 44, no. 8, pp. 670-677, 2008, (in Japanese).\\ [3] --, "Stabilization of time-varying stochastic port-Hamiltonian systems based on stochastic passivity," in Proc. IFAC Symp. Nonlinear Control Systems, 2010, pp. 611-616.\\ [4] --, "Observer based stochastic trajectory tracking control of mechanical systems," in Proc. ICROS-SICE Int. Joint Conf. 2009, 2009, pp. 1244-1248. \\ [5] S. Satoh and M. Saeki, "Bounded stabilization of a class of stochastic port-Hamiltonian systems," in Proc. 20th Symp. Mathematical Theory of Networks and Systems, 2012, pp. (CD-ROM) 0150.\\ [6] H. J. Kappen, "Path integrals and symmetry breaking for optimal control theory," J. Statistical Mechanics: Theory and Experiment, p. P11011, 2005.\\ [7] S. Satoh, H. J. Kappen, and M. Saeki, "A solution method for nonlinear stochastic optimal control based on path integrals," in Proc. 12th SICE System Integration Division Annual Conf., 2012, p. P0194, (in Japanese).