Many reinforcement learning (RL) algorithms are defined on Markov decision processes (or MDPs). More precisely, the problem that the RL agent is trying to solve is often formulated as an MDP. In this mini-post, I try to answer the following questions.

• What are MDPs? Or Who is Markov? and what exactly is a decision process?
• How are MDPs defined and do I really have to learn all of this if I am just interested in learning the basics of RL?
• When is a MDP formulation insufficient? What are the alternatives?

What are these MDPs?