Jump to content

Welcome to the Forum!

The best place to exchange builds and ideas! Vote for the best ideas and suggestions here.

Join the Avorion Discord!

Connect with other passionate players and talk about the latest news.
Discord

DLC Avorion Into the Rift Out Now!

Now available on Steam!
Steam

Memory leak and shutdown fails, leaving a zombie process


TrevorJD
 Share

Recommended Posts

Hi folks,

We run a modded server that has two serious problems. Firstly, it seems that we have a memory leak as the server will slowly creep up in RAM usage to the point where the server is running at almost 100% RAM usage. Secondly, the server is not always shutting down cleanly. The log shows it shutting down but it leaves a zombie process. That process still has hold of the game ports, so when the new instance starts up, it can't use them and the now-running server is invisible to players.

Tonight it hit 15GB with only 7 people online. I told it to /stop and the server shut down but left the zombie process, according to Task Manager, 0% CPU but still 15GB RAM. Just an hour later, it was already back up to 10GB with the same 7 players. A few hours later it was time for the daily automatic reboot. Again, the server shut down but left a zombie process. I notice that the last line of the log is as follows:

2022-08-14 23-16-21| Error cleaning up lua::WorkerPool: shutdown() not called!

I'm assuming both the memory leak and the inability to shut down cleanly are due to mods however perhaps some additional checks are needed to ensure that the server can never get in such a state.

I would like to troubleshoot this but it would be quite difficult to replicate the conditions of an actively-used server. Further, looking at the logs, the shutdown problem is not consistent; sometimes it shuts down just fine. I could remove mods but that risks corrupting the save, and I couldn't convince several players to put in the effort on a test server. If you have any suggestions on how I might proceed with testing this further, please let me know.

Thanks,
Trev

Link to comment
Share on other sites

Further information: I'm working further to try to sort this out. Again, it's failed to shut down cleanly. The first part below is from the TCAdmin game console monitor. The second part is the tail end of the most recent log.

 

Cleaning up entity transfers ...
Server is shutting down.
Saving settings...
Saving script values...
Saving factions...
Saving sectors...
Saving groups...
Saving galaxy script data...
Saving settings...
waiting for unfinished sector generation jobs...
waiting for unfinished sector save jobs...
saving sector 0x19219060
saving sector 0x48c1c2b0
saving sector 0x178be6c0
saving sector 0x34432a40
src\tier1\fileio.cpp (5164) : m_vecRegisteredWriters.Count() == 0
src\tier1\fileio.cpp (5164) : m_vecRegisteredWriters.Count() == 0
src\tier0\threadtools.cpp (3628) : Assertion Failed: Illegal termination of worker thread 'CFileWriterThread'
src\tier0\threadtools.cpp (3628) : Assertion Failed: Illegal termination of worker thread 'CFileWriterThread'
 
 
 
2022-08-15 16-11-03| waiting for unfinished sector generation jobs...
2022-08-15 16-11-03| waiting for unfinished sector save jobs...
2022-08-15 16-11-03| create save work for sector (-36:-52), 0x19219060, entities: 841
2022-08-15 16-11-03| create save work for sector (-35:-59), 0x48c1c2b0, entities: 818
2022-08-15 16-11-03| create save work for sector (-32:-56), 0x178be6c0, entities: 2927
2022-08-15 16-11-03| create save work for sector (-25:-58), 0x34432a40, entities: 294
2022-08-15 16-11-03| saving sector 0x19219060
2022-08-15 16-11-03| saving sector 0x19219060
2022-08-15 16-11-04| saving sector (-36:-52)
2022-08-15 16-11-04| sector (-36:-52) saved to "C:\TCAFiles\Users\staff\25\galaxies\avorion_galaxy\sectors\-36_-52"
2022-08-15 16-11-04| saving sector 0x48c1c2b0
2022-08-15 16-11-04| saving sector 0x48c1c2b0
2022-08-15 16-11-04| saving sector (-35:-59)
2022-08-15 16-11-04| sector (-35:-59) saved to "C:\TCAFiles\Users\staff\25\galaxies\avorion_galaxy\sectors\-35_-59"
2022-08-15 16-11-04| saving sector 0x178be6c0
2022-08-15 16-11-04| saving sector 0x178be6c0
2022-08-15 16-11-05| saving sector (-32:-56)
2022-08-15 16-11-06| sector (-32:-56) saved to "C:\TCAFiles\Users\staff\25\galaxies\avorion_galaxy\sectors\-32_-56"
2022-08-15 16-11-06| saving sector 0x34432a40
2022-08-15 16-11-06| saving sector 0x34432a40
2022-08-15 16-11-06| saving sector (-25:-58)
2022-08-15 16-11-09| Error cleaning up lua::WorkerPool: shutdown() not called!
2022-08-15 16-11-09| Thread [Lua WorkerPool Thread 4] with id 19 exiting...
2022-08-15 16-11-09| Thread [Lua WorkerPool Thread 4] with id 19 exited
Link to comment
Share on other sites

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!

Register a new account

Sign in

Already have an account? Sign in here.

Sign In Now
 Share

×
×
  • Create New...