Dynamic Programming

Spring 2018

Instructor: Dr. Rajesh Ganesan

Class time: Mon 7:20 - 10:00 PM


Office: Engr Building, Room 2217

Phone: (703) 993-1693                                                    

Fax: (703) 993-1521                                                                                 

Email: rganesan@gmu.edu



Text: DP by Eric Denardo


Week 1: Introduction, DP Recursion vs IP and LP, Big picture

DP- Deterministic- Finite Horizon

 Notes 1,  Addendum to Notes 1


Slides from Ankit pdf

Week 2:  Applications: Shortest path, longest path problems, Dijkstra's Alg

Slides from Ankit pdf   Excel  Julia code

              Practice Problems: Problem 2, 4 from text, Group A and B problems from Winston's Book (Page 968-969)  

Excel solutions to some problems


I will update these files often. Always download the latest version if you plan on using them.

How to use these files

1. Save the long or short code.

2. Change the c and X matrices to suit your problem. For some problems these are available at the end of the code, which you can copy and paste to the top of the code. Use Cntl+R and Cntl+T to remove the comments % on the matrices.

2.a. Make sure that the c matrix has only one state in stage 1. This is to initialize. Consequently the first column will be all zeros.

2.b. If c is 0 on a feasible arc then use 0.0001 instead of zero. If an arc is infeasible put a large c value for the min problem or a large negative c value for a max problem.

2.c. If an arc does not exist at all then mark the c value as 0.

3. change max and min to suit your problem.

4. Save and run (green arrow icon). For the very first time you might get a pop up window asking you to add a path. Click on "Add to path".

Matlab short code: Finds the length and only 1 path (Generic)

Matlab long code: Finds the length and up to 12 paths (Generic)

matlab code for inventory problem to generate cost and action matrix (for inv control discussed in class page 969 in Winston)

matlab code for inventory problem to generate costand action matrix  (for pg 25 prob 1 in Denado's book)

matlab code for investment problem to generate costand action matrix  (investment in class problem)

*The following are used together*

matlab code for inventory problem to generate costand action matrix  (for pg 26 prob 3 in Denado's book)

Long code for  (for pg 26 prob 3 in Denado's book). The long code has been modified from the generic one by adding a few extra lines at the bottom

*The following are used together*

matlab code for inventory problem to generate cost and action matrix (for Sailco inv control - first problem from the first class )

matlab code for (for Sailco inv control - first problem from the first class)



Week 3, 4


DP formulation: Deterministic Inventory control, Resource Allocation: Investment and Knapsack, Min-max problems, Equipment replacement with bounded Horizon  Notes

               Practice Problems: 1 ,3 from text page 25, Prob 3 page 63 of the text, Prob 2 page 974, problem 2, 5 page 985 of Winston's


Refer to excel  and matlab solutions in the links under week 2.

Week 5


winstons https://archive.org/stream/OperationalResearchWinstonWayne/Operational-Research-Winston-Wayne#page/n1009/mode/2up



Capacity expansion, multiple resource allocation, Traveling salesman,  notes

     Practice Problems: Problem 7 -page 27 of text, Problem 6-page 1000 of Winston  (similar to one on page 998), Problem 5 page   1014 of Winston's



Week 6

Curse of dimensionality, gridding, cyclic graphs,  notes


Midterm Exam due April 16 (prob 2 corrected)

pdf to go with midterm

Submissin format:

Hard copy or if you have typed them up (Email submission) is fine too. I am looking for the DP formulation, definition of stage, state, action, contribution function, the cost and action matrices in the form of an Excel sheet (one sheet per problem)  and the final answer (final answer and the path, again only if you can solve it with numerical values). I donít need the matlab files. However, if the problem is solved using Excel then you may email me the excel file that already contains the cost/action matrices.

Week 7

DP- Deterministic- Infinite Horizon

Equipment replacement unbounded horizon analytical solution,   value iteration. For notes - see last 2 pages of week 6

Week 8 - spring break


Week 9



Stoochastic DP-Finite Horizon.types

(s,S) policy  notes

 Practice Problems: problem 4 page 1034 from Winston's book

 Practice: Problem 4 page 1023 and Problem 5 page 1035 of (Winstonísí book).

Project  Due on Final exam day.


Week 10

Examples of Stochatic DP -finite and Markov Chains   notes

Practice problem 6 and 7 page 130 Denardo.

Week 11- 12

Stochatic DP -infinite horizon

Project  Due on Final exam day.

Stochastic DP notes Limiting Probabilities and MDP, Exhaustive enumeration, LP solution to MDP - average cost/reward. 



MDP- Average Reward/cost- Policy and Value Iteration for machine replacement problem.

MDP- Discounted Reward/cost- Policy and Value Iteration for machine replacement problem.

policy iteration excel sheet   Machine replacement problem

Excel Example value iteration for MDP

Practice problem: 4,6,9,11 on page 1049


MATLAB value iteration matlab for machine replacement


eek 13

Water Resource example: see handout  value iteration using excel

Inventory control Example,  see hand out, excel for inventory control

Machine maintenance and gardner fertilization problem excel for both problems

Final exam Due May 14th along with the project. You may email or leave a hard copy in my dept mailbox.  1046 1049 1050


Week 14



SMDP summary

SMDP for Example B in the handout

SMDP questions. For solution click links below

SMDP for Machine replacement problem value iteration   

SMDP for pricing based investment/asset acquisition value iteration


Week 15

DP summary and way forward with ADP (OR 774)

Review DP

Please read