Learning more with the same effort: how randomization improves the robustness of a robotic deep reinforcement learning agent

L. Güitta-López, J. Boal, A.J. López López

Applied Intelligence Vol. 53, nº. 12, pp. 14903 - 14917

Resumen:

The industrial application of Deep Reinforcement Learning (DRL) is frequently slowed down due to an inability to generate the experience required to train the models. Collecting data often involves considerable time and financial outlays that can make it unaffordable. Fortunately, devices like robots can be trained with synthetic experience through virtual environments. With this approach, the problems of sample efficiency with artificial agents are mitigated, but another issue arises: the need to efficiently transfer the synthetic experience into the real world (sim-to-real).
This paper analyzes the robustness of a state-of-the-art sim-to-real technique known as Progressive Neural Networks (PNNs) and studies how adding diversity to the synthetic experience can complement it.
To better understand the drivers that lead to a lack of robustness, the robotic agent is still tested in a virtual environment to ensure total control on the divergence between the simulated and real models.
The results show that a PNN-like agent exhibits a substantial decrease in its robustness at the beginning of the real training phase. Randomizing specific variables during simulation-based training significantly mitigates this issue. The average increase in the model’s accuracy is around 25% when diversity is introduced in the training process. This improvement can translate into a decrease in the number of real experiences required for the same final robust performance. Notwithstanding, adding real experience to agents should still be beneficial, regardless of the quality of the virtual experience fed to the agent. The source code is available at: https://gitlab.com/comillas-cic/sim-to-real/pnn-dr.git

Resumen divulgativo:

Los agentes de Aprendizaje por Refuerzo Profundo entrenados con experiencia sintética se enfrentan a cómo transferir experiencia virtual a la realidad. Este artículo analiza la robustez de las Redes Neuronales Progresivas y estudia como añadir diversidad con experiencia sintética las complementan.

 

Palabras Clave: Reinforcement Learning, Deep Learning, Sim-To-Real, Domain Randomization, Sample Efficiency, Robotics

Índice de impacto JCR y cuartil WoS: 5,300 - Q2 (2022)

Referencia DOI: https://doi.org/10.1007/s10489-022-04227-3

Publicado en papel: Junio 2023.

Publicado on-line: Noviembre 2022.

Cita:
L. Güitta-López, J. Boal, A.J. López López, Learning more with the same effort: how randomization improves the robustness of a robotic deep reinforcement learning agent. Applied Intelligence. Vol. 53, nº. 12, pp. 14903 - 14917, Junio 2023. [Online: Noviembre 2022]