การพัฒนาแบบจำลองแบบเรียลไทม์เพื่อเพิ่มประสิทธิภาพการควบคุมเรืออัตโนมัติด้วยอัลกอริทึม Twin Delayed Deep Deterministic Policy Gradient ในระบบการเรียนรู้เสริมกำลังเชิงลึก
Keywords:
Autonomous Shipping, Deep Reinforcement Learning (DRL), TD3 Algorithm, Energy Management, Collision Avoidance, Fuel EfficiencyAbstract
In accordance with the International Maritime Organization (IMO) regulations concerning greenhouse gas emission reduction and the transition toward autonomous shipping, developing a ship control system that integrates energy management efficiency with safety is imperative. This research proposes a real-time framework on the MATLAB/Simulink platform for the Ro-Pax vessel M/S Nils Holgersson. A comprehensive mathematical model was developed, encompassing ship dynamics, a multi-engine propulsion system, and environmental resistance utilizing simulated sensor data. Furthermore, an Application Programming Interface (API) was integrated with a marine meteorological database to retrieve real-time weather data for model training. This study applied Deep Reinforcement Learning (DRL) via the Twin Delayed Deep Deterministic Policy Gradient (TD3) algorithm to manage the main engines' load distribution and control the rudder angle. The primary objectives were to minimize fuel consumption and enhance collision avoidance success rates. Simulation results demonstrated that the proposed framework reduced total fuel consumption by 1.592% (equivalent to 73.56 kg per voyage) along a 43-nautical-mile route from Yachthafen Hohe Düne, Rostock, to Marina Baltica 1, Lübeck, Germany. This efficiency improvement consequently mitigated CO₂ emissions by 226.71 kg per voyage. Additionally, the system increased the collision avoidance success rate by 1.75% compared to conventional control systems (improving from 93.71% to 95.46%). This was evaluated based on the vessel's ability to maintain a safe distance and allocate adequate time margins for evasive maneuvers. These outcomes clearly demonstrate the proposed framework's high potential for commercial implementation, successfully achieving both safety and energy efficiency targets simultaneously. This research provides a significant foundation for the future development and advancement of autonomous marine vehicle control systems.
References
Sinay. How much does the shipping industry contribute to global CO2 emissions? [Internet]. 2023 [cited 2025 Sep 1]. Available from: https://sinay.ai/en/how-much-does-the-shipping-industry-contribute-to-global-co2-emissions/.
International Maritime Organization. IMO’s work to cut GHG emissions from ships [Internet]. 2023 [cited 2025 Sep 1]. Available from: https://www.imo.org/en/mediacentre/ hottopics/pages/cutting-ghg-emissions.aspx.
Kim SW. The development of route decision-making method based on tailor-made forecast 2d wave spectra due to the operation profile of the vessel. Ocean Engineering 2020;197:106907. doi: 10.1016/j.oceaneng.2019.106907.
McKie R. Maritime Autonomous Surface Ships (MASS) and SAR [Internet]. International Maritime Rescue Federation; 2023 [cited 2025 Sep 1]. Available from: https://www.international-maritime-rescue.org/news/maritime-autonomous-surface-ships-mass-and-sar#_ftn1.
Fortune Business Insights. Autonomous ships market size, share, industry analysis & Russia-Ukraine war analysis, by autonomy, by solution, by ship type, by end user, and regional forecast, 2026-2034 [Internet]. Pune: Fortune Business Insights; 2025 [cited 2025 Oct 9]. Report No.: FBI101797. Available from: https://www.fortunebusinessinsights.com/industry-reports/autonomous-ship-market-101797.
Xiao G, Pan L, Lai F. Application, opportunities, and challenges of digital technologies in the decarbonizing shipping industry: a bibliometric analysis. Frontiers in Marine Science 2025;12:1523267. doi: 10.3389/fmars.2025.1523267.
Heo S, Kang M, Choi J, Yun S, Park J. A preliminary study for development of digital twin ship technology using autonomous surface vehicle. In: 20th International Conference on Ubiquitous Robots (UR); 2023 Jun 25-28; Honolulu, HI, USA. Piscataway (NJ): IEEE; 2023. p. 255-60. doi: 10.1109/UR57808.2023.10202274.
MathWorks. Simulink: simulation and model-based design [Internet]. Natick, MA: The MathWorks, Inc.; c1994-2026 [cited 2026 Jul 21]. Available from: https://www.mathworks.com/products/simulink.html.
Ahmed YA, Hannan MA, Oraby MY, Maimun A. COLREGs compliant fuzzy-based collision avoidance system for multiple ship encounters. Journal of Marine Science and Engineering 2021;9(8):790. doi: 10.3390/jmse9080790.
Tengesdal T, Brekke EF, Johansen TA. On collision risk assessment for autonomous ships using scenario-based MPC. IFAC-PapersOnLine 2020;53(2):14509-16. doi: 10.1016/j.ifacol.2020.12.1454.
Sutton RS, Barto AG. Reinforcement learning: An introduction. 2nd ed. Cambridge, MA: MIT Press; 2018.
Wang W, Huang L, Liu K, Wu X, Wang J. A COLREGs-compliant collision avoidance decision approach based on deep reinforcement learning. Journal of Marine Science and Engineering 2022;10(7):944. doi: 10.3390/jmse10070944.
Xie W, Gang L, Zhang M, Liu T, Lan Z. Optimizing multi-vessel collision avoidance decision making for autonomous surface vessels: a COLREGs-compliant deep reinforcement learning approach. Journal of Marine Science and Engineering 2024;12(3):372. doi: 10.3390/jmse12030372.
Pan R, Zhang W, Wang S, Kang S. Deep reinforcement learning model for Multi-Ship collision avoidance decision making design implementation and performance analysis. Scientific Reports 2025;15:21250. doi: 10.1038/s41598-025-05636-3.
Zhang X, Sun T. Deep reinforcement learning for collision avoidance of autonomous ships in inland rivers. Proceedings of the Institution of Civil Engineers – Transport 2025;178(5):343-58. doi: 10.1680/jtran.24.00068.
Liu Z, Zhou Z, Zhang M, Liu J. A twin delayed deep deterministic policy gradient method for collision avoidance of autonomous ships. Journal of Transport Information and Safety 2022;40(3):60-74. doi: 10.3963/j.jssn.1674-4861.2022.03.007.
Kim S, Yun S, You Y. Eco-friendly speed control algorithm development for autonomous vessel route planning. Journal of Marine Science and Engineering 2021;9(6):583. doi: 10.3390/jmse9060583.
Moradi MH, Brutsche M, Wenig M, Wagner U, Koch T. Marine route optimization using reinforcement learning approach to reduce fuel consumption and consequently minimize CO2 emissions. Ocean Engineering 2022;259:111882. doi: 10.1016/j.oceaneng.2022.111882.
Latinopoulos C, Zavvos E, Kaklis D, Leemen V, Halatsis A. Marine voyage optimization and weather routing with deep reinforcement learning. Journal of Marine Science and Engineering 2025;13(5):902. doi: 10.3390/jmse13050902.
Alshareef MH, Alghanmi AF. Optimizing maritime energy efficiency: a machine learning approach using deep reinforcement learning for EEXI and CII compliance. Sustainability 2024;16(23):10534. doi: 10.3390/su162310534.
Fujimoto S, van Hoof H, Meger D. Addressing function approximation error in actor-critic methods. In: Dy J, Krause A, editors. Proceedings of the 35th International Conference on Machine Learning (ICML); 2018 Jul 10-15; Stockholm, Sweden. PMLR; 2018. p. 1587-96. doi: 10.48550/arXiv.1802.09477.
Hadi B, Khosravi A, Sarhadi P. Deep reinforcement learning for adaptive path planning and control of an autonomous underwater vehicle. Applied Ocean Research 2022;129:103326. doi: 10.1016/j.apor.2022.103326.
Yasukawa H, Yoshimura Y. Introduction of MMG standard method for ship maneuvering predictions. Journal of Marine Science and Technology 2015;20:37-52. doi: 10.1007/s00773-014-0293-y.
Alexandersson M. A study of methods to predict added resistance in waves [Master's thesis]. Stockholm: KTH Centre for Naval Architecture; 2009. doi: 10.13140/RG.2.2.18796.69767.
Park H, Lee P, Kim J, Kim H, Lee H, LeeY. Study on the estimation method of wind resistance considering self-induced wind by ship advance speed. Journal of Marine Science and Engineering 2024;12:1694. doi: 10.3390/jmse12101694.
Beji S. Formulation of wave and current forces acting on a body and resistance of ships. Ocean Engineering 2020;218:108121. doi: 10.1016/j.oceaneng.2020.108121.
Welch G, Bishop G. An Introduction to the Kalman Filter [Internet]. Chapel Hill, NC: Department of Computer Science, University of North Carolina at Chapel Hill; 1995 [cited 2025 Oct 9]. Available from: https://cis.temple.edu/~latecki/Courses/CIS750-03/Papers/KalmanFilterSIGGRAPH2001.pdf.
Lazarowska A. Review of collision avoidance and path planning methods for ships utilizing radar remote sensing. Remote Sensing 2021;13:3265. doi: 10.3390/rs13163265.
Open-Meteo. Marine weather API [Internet]. Open-Meteo; c2022-2026 [cited 2026 Jun 8]. Available from: https://open-meteo.com/en/docs/marine-weather-api.
Wang H, Zhou P, Wang Z. Reviews on current carbon emission reduction technologies and projects and their feasibilities on ships. Journal of Marine Science and Application 2017;16:129-36. doi: 10.1007/s11804-017-1413-y.
Downloads
Published
Issue
Section
License
Copyright (c) 2026 Kasem Bundit University
This work is licensed under a Creative Commons Attribution-NonCommercial-NoDerivatives 4.0 International License.
*Copyright
The article has been published in Kasem Bundit Engineering Journal (KBEJ) is the copyright of the Kasem Bundit University. Do not bring all of the messages or republished except permission from the university.
* Responsibility
If the article is published as an article that infringes the copyright or has the wrong content the author of article must be responsible.
