TrevorJD Posted August 14, 2022 Share Posted August 14, 2022 Hi folks, We run a modded server that has two serious problems. Firstly, it seems that we have a memory leak as the server will slowly creep up in RAM usage to the point where the server is running at almost 100% RAM usage. Secondly, the server is not always shutting down cleanly. The log shows it shutting down but it leaves a zombie process. That process still has hold of the game ports, so when the new instance starts up, it can't use them and the now-running server is invisible to players. Tonight it hit 15GB with only 7 people online. I told it to /stop and the server shut down but left the zombie process, according to Task Manager, 0% CPU but still 15GB RAM. Just an hour later, it was already back up to 10GB with the same 7 players. A few hours later it was time for the daily automatic reboot. Again, the server shut down but left a zombie process. I notice that the last line of the log is as follows: 2022-08-14 23-16-21| Error cleaning up lua::WorkerPool: shutdown() not called! I'm assuming both the memory leak and the inability to shut down cleanly are due to mods however perhaps some additional checks are needed to ensure that the server can never get in such a state. I would like to troubleshoot this but it would be quite difficult to replicate the conditions of an actively-used server. Further, looking at the logs, the shutdown problem is not consistent; sometimes it shuts down just fine. I could remove mods but that risks corrupting the save, and I couldn't convince several players to put in the effort on a test server. If you have any suggestions on how I might proceed with testing this further, please let me know. Thanks, Trev Link to comment Share on other sites More sharing options...
TrevorJD Posted August 15, 2022 Author Share Posted August 15, 2022 Further information: I'm working further to try to sort this out. Again, it's failed to shut down cleanly. The first part below is from the TCAdmin game console monitor. The second part is the tail end of the most recent log. Cleaning up entity transfers ... Server is shutting down. Saving settings... Saving script values... Saving factions... Saving sectors... Saving groups... Saving galaxy script data... Saving settings... waiting for unfinished sector generation jobs... waiting for unfinished sector save jobs... saving sector 0x19219060 saving sector 0x48c1c2b0 saving sector 0x178be6c0 saving sector 0x34432a40 src\tier1\fileio.cpp (5164) : m_vecRegisteredWriters.Count() == 0 src\tier1\fileio.cpp (5164) : m_vecRegisteredWriters.Count() == 0 src\tier0\threadtools.cpp (3628) : Assertion Failed: Illegal termination of worker thread 'CFileWriterThread' src\tier0\threadtools.cpp (3628) : Assertion Failed: Illegal termination of worker thread 'CFileWriterThread' 2022-08-15 16-11-03| waiting for unfinished sector generation jobs... 2022-08-15 16-11-03| waiting for unfinished sector save jobs... 2022-08-15 16-11-03| create save work for sector (-36:-52), 0x19219060, entities: 841 2022-08-15 16-11-03| create save work for sector (-35:-59), 0x48c1c2b0, entities: 818 2022-08-15 16-11-03| create save work for sector (-32:-56), 0x178be6c0, entities: 2927 2022-08-15 16-11-03| create save work for sector (-25:-58), 0x34432a40, entities: 294 2022-08-15 16-11-03| saving sector 0x19219060 2022-08-15 16-11-03| saving sector 0x19219060 2022-08-15 16-11-04| saving sector (-36:-52) 2022-08-15 16-11-04| sector (-36:-52) saved to "C:\TCAFiles\Users\staff\25\galaxies\avorion_galaxy\sectors\-36_-52" 2022-08-15 16-11-04| saving sector 0x48c1c2b0 2022-08-15 16-11-04| saving sector 0x48c1c2b0 2022-08-15 16-11-04| saving sector (-35:-59) 2022-08-15 16-11-04| sector (-35:-59) saved to "C:\TCAFiles\Users\staff\25\galaxies\avorion_galaxy\sectors\-35_-59" 2022-08-15 16-11-04| saving sector 0x178be6c0 2022-08-15 16-11-04| saving sector 0x178be6c0 2022-08-15 16-11-05| saving sector (-32:-56) 2022-08-15 16-11-06| sector (-32:-56) saved to "C:\TCAFiles\Users\staff\25\galaxies\avorion_galaxy\sectors\-32_-56" 2022-08-15 16-11-06| saving sector 0x34432a40 2022-08-15 16-11-06| saving sector 0x34432a40 2022-08-15 16-11-06| saving sector (-25:-58) 2022-08-15 16-11-09| Error cleaning up lua::WorkerPool: shutdown() not called! 2022-08-15 16-11-09| Thread [Lua WorkerPool Thread 4] with id 19 exiting... 2022-08-15 16-11-09| Thread [Lua WorkerPool Thread 4] with id 19 exited Link to comment Share on other sites More sharing options...
Recommended Posts
Create an account or sign in to comment
You need to be a member in order to leave a comment
Create an account
Sign up for a new account in our community. It's easy!
Register a new accountSign in
Already have an account? Sign in here.
Sign In Now