The Mean-Squared Error of Double Q-Learning

          @misc{ scivideos_16826,
            doi = {},
            url = {https://simons.berkeley.edu/talks/tbd-246},
            author = {},
            keywords = {},
            language = {en},
            title = {The Mean-Squared Error of Double Q-Learning},
            publisher = {The Simons Institute for the Theory of Computing},
            year = {2020},
            month = {dec},
            note = {16826 see, \url{https://scivideos.org/Simons-Institute/16826}}
          }
          

Abstract

We establish a theoretical comparison between the asymptotic mean-squared error of Double Q-learning and Q-learning. Using prior work on the asymptotic mean-squared error of linear stochastic approximation based on Lyapunov equations, we show that the asymptotic mean-squared error of Double Q-learning is exactly equal to that of Q-learning if Double Q-learning uses twice the learning rate of Q-learning and outputs the average of its two estimators. We also present some practical implications of this theoretical observation using simulations.