Q-learning based energy management system on operating reserve and supply distribution

Due to their lower cost and performance predictability, traditional energy sources outperform renewables. Therefore, one of the most important concerns of energy management systems in 6G smart grids is to plan the distribution of energy supply and ensure adequate stability. In this study, we present a two-stage Q-learning based energy management system called 2QEMS. This system calculates the subsidies for each type of energy supply and a dynamic multiplier for the operating reserve by using Q-tables in the agents. To achieve the desired distribution of energy sources and maintain the operating reserve, the agent converses with its environment, which includes energy suppliers, energy demanders, and an auction system. The 2QEMS operates by using clear states and intuitive actions that provide a high degree of interpretability. Experiments show that the 2QEMS reduces the convergence time to 740 days, with a mean absolute error (MAE) of the supply distribution of 6.8%. This is 57% and 70% reduction, respectively, compared to the traditional system, as well as 23% and 20% reduction, respectively, compared to the state-of-the-art systems. The experiments demonstrate the effectiveness and resilience of the 2QEMS by showing excellent and consistent performance in a variety of scenarios.