Advanced Dynamic Programming

 

Instructor: Dr. Rajesh Ganesan

    

Eng Bldg. Room 2217

Phone: (703) 993-1693                                                    

Fax: (703) 993-1521                                                                                 

Email: rganesan at gmu dot edu

 

Week 1

Syllabus

Text Book: https://onlinelibrary.wiley.com/doi/book/10.1002/9781118029176

https://onlinelibrary.wiley.com/doi/book/10.1002/9780470544785 by Jennie Si et al.

Big picture

Examples for DP/Approx DP     Read chapter by Paul Werbos in Si. et al. pages 3-44

Weeks  2 - 3

DP refresher   Notes

DP example question  

Excel Example value iteration for MDP

 

 

Weeks 4 - 6

 

Chapters 1-3

ADP motivation

 

Chapter 4

ADP Dialect 1 - Asynchronous update with TPM - use of small case or little v, use pre decision states (Fig 4.2 page 120)

ADP Dialect 2 - Q- learning around pre decison state (no TPM). use of small case or little q, Robbins-Monroe Stochatic Approximation Scheme, Use of learning parameter alpha, use pre decision states

ADP Dialect 3 - Asynchronous update with TPM, use of small case or little v, Robbins-Monroe Stochatic Approximation Scheme, Use of learning parameter alpha, use pre decision states (Fig 4.4 page 128)

ADP Dialect 4 - Asynchronous update (No TPM), use of small case or little v, Robbins-Monroe Stochatic Approximation Scheme, Use of learning parameter alpha, use post decision states(Fig 4.7 page 141)

ADP Dialect 5 - Q- learning around post decison state (no TPM). use of small case or little q, Robbins-Monroe Stochatic Approximation Scheme, Use of learning parameter alpha, use post decision states

 

SARSA - evaluate a policy

RTDP- start from an optimistic value of a state (same as ADP Dialect 1)

 

Week 7

Summary of models and coding

MATLAB

DP- Value iteration Discounted cost criteria, DP, uses TPM, sync update so all possible next states are evaluated in each iteration

ADP Dialect 1 - Figure 4.2 page 120 value iteration_ADP, uses TPM, V=v, async update, TPM used for finding next state

ADP Dialect 3 - Figure 4.4 page 128 value iteration_ADP2, uses TPM, V = (1-alpha)V + alpha v, async update, TPM used for finding next state

ADP Dialect 4 - Figure 4.7 page 141 value iteration_ADP 3, no TPM, uses Post-decision state, V = (1-alpha)V + alpha v, async update, need some uncertainty model for finding next state

 

Week 8

Chapters 5 and 6 - Modeling and Policies

Week 9

Chapter 7 - Policy search