Temporal Difference (TD) Learning

• Recall Dynamic Programming value update:
  • Value(s) = Discount * Value(Policy(s)) + Reward(s)
• Temporal Difference value update:
  • Value(s) = Value(s) + Learning_Rate * (Discount * Value(Policy(s)) - Value(s) + Reward(s))
• Alternative formulations:
  • v(s) = v(s) + α * (γ * v(Policy(s)) - v(s) + r(s))
  • v(s) = (1 - α) * v(s) + α * (γ * v(Policy(s)) + r(s))
• Loop indefinitely
  • Perform Policy(s), yielding new state s'
  • v(s) = (1 - α) * v(s) + α * (γ * v(s') + r(s))