Agent-Piano

So far I have not seen a relationship between thread ratio, system hardware, and stability. This is based on tuning and troubleshooting a server deployment of around 10 active players.

Great post, thanks for the information!

I guess your whole hardware is too old, also it is only virtualized. Can you reproduce your issues on a I7 or I9 CPU, using a dedicated machine and not a VM? Dedicated machines usely work better for hosting an Avorion server, also we still talk about a game - your Xeon CPU may result in a bad performance. You could try a benchmark and compare to your desktop computer or so... I just watched your CPU... it is 10 years old, you really should get better hardware. Also I recommand a Desktop CPU, not a server CPU. I have no issues on my server with 20 people online, but you need a high-end server for a really good performance. First of all, thank you continuing to stay engaged in the conversation to provide feedback. My hope is to work through understanding what kind of CPU requirements the dedicated server actually needs as they are not particularly well documented on the wikipedia. My observation in troubleshooting this issue is that the dedicated server engine needs a faster single-thread speed and that this is not a multi-core/socket issue. I have tried several configurations where I have added and reduced the CPU configurations for the Virtual Machine and the server engine simply does not use the extra processing power. The same is true with memory as I have added several GBs of additional memory and the server engine simple does not use it (even with adjustments to sectors in memory, etc.). With that said, my suspicion is that the limitation is more around CPU frequency (the speed of single threaded operations). I am in the process of implementing a new VMware host which has much higher CPU clock speeds and will test my theory, but having a grasp of what other folks are running without issue would be helpful. There is no reason that a Xeon class processor would not be able to run this server engine, but I do suspect that it's going to be looking for a minimum clock speed to run well and is not going to take advantage of having a significantly wider thread/CPU layout. Some i7 and i9 processors have much higher clock speeds, but that is not true for all of them. Accordingly, simple moving to an i7 or i9 isn't inherently going to lead to better performance. There is going to be some sort of coefficient of what the actual performance of the processor is versus the performance of the server engine. With that said, can you share what CPU configurations you're running on the system that hosts your server? I'm specifically looking for the model of CPU, memory speeds, and ideally chip-set. Being able to study the FSB speeds, cache speeds, and frequency will help us put some scientific data behind what is fast and slow If you have a post where you've already explained this I'd be happy to read it, just need some kind of cross-link. Thanks again for your time.

6 vCPU (3 Sockets x 2 Cores) - 1400MHz Reservation, High Share Priority 16GB Memory 6x 1400MHz is just not enaugh, as I said above this is no good setting for hosting Avorion. Depending on number of players online 16bg may also not be enaugh. You should upgrade. The system isn't running 6 x 1400MHz. It's running at 6 x 2.4GHz and there is a cpu reservation for 14000* MHz on the host. The original number of 1400 was a typo, my apologies for that. So, the system running it has 6 x 2.4GHz processors which totals out to 14400MHz. Of that, 14000MHz is reserved on the host for only that VM so it is not competing for resources. While the wikipedia articles and FAQs seem to refer a lot to core and socket allocation I have not found any details which explain single thread speeds and what the requirements are. Based on that information, can you elaborate on your comments that my CPU configuration may not be adequate?

We are also now seeing issues where player owned ships and stations are sometimes duplicating or cloning after players are disconnected from the server. Players are getting disconnected somewhat frequently. Sometimes the clone items disappear and sometimes they remain permanent. In other instances, ships are jumping to the incorrect system and to random coordinates far out of the range of their jump range (full quadrants, etc.). Hardware performance for the system continues to be very strong with no evidence of limitations. Still unsure as to why this application is struggling as much as it is.

After continued performance testing, we're still seeing a large amount of issues. It seems to be that once the server goes over a population of about 5-6 players the server becomes very unstable. Recently we experienced an evening where the server mass-disconnected all players three times in a fifteen minute period. Reviewing performance charts, there are not utilization or latency issues related to CPU, Memory, Disk, or Network. While it is operating in a shared environment there isn't any information that supports a hardware related problem. Any other thoughts on how to troubleshoot or remediate the problem?

Thanks for the response, over the weekend I adjusted the configuration from 3 x 2 to 1 x 6 which seems to have led to some stability. I have also made the adjustment you recommend to provide hardware concurrency. From a CPU/RAM standpoint, neither the VMware host or the VM are tensioned, I could look at expanding the vCPUs laterally but based on what I'm seeing in performance data it looks as though those resources would not be used. Will be interesting to see how the game server performs with these changes.

Hello! I've been running a dedicated server for about one week that is averaging between 5-10 clients. In the last few days we started seeing issues where client "ping" goes to about 5s. This is then resolved by restarting the game server. I have been able to pin down when this starts by way of CPU performance charting for the system. Using those timestamps I've stepped back through server logs in attempt to identify any trace data. With profiling enabled, I'm seeing this entries which definitely "look" pretty bad. Thoughts? Also, the log file is too large for pastebin, alt. upload link below... https://gofile.io/?c=5oQHuB Server Hardware Windows Server 2019 VMware Hardware V11 6 vCPU (3 Sockets x 2 Cores) - 14000MHz Reservation, High Share Priority 16GB Memory SSD Based Storage VMware Host HP Proliant DL360 G7 - ESXi 6.0 15517548 2 x Intel Xeon E5620 @ 2.4GHz (16 Logical) - (Aprox 25% utilization) 120GB Memory (Aprox 40% utilization) 2020-04-22 13-18-07| saving sector (177:41) 2020-04-22 13-18-07| Player kevin moved to sector (119:219)from (119:220) server time taken for change: 0ms S2020-04-22 13-18-07e| ctoSrSeDecactttooarrbDDaaattsaaebb:aas seeL:: oLLaoodaaiddniignn ggS eSSceetccottroo rr( (1(1112720:0:2:1229021)3) 2020-04-22 13-18-07| )2020-04-22 13-18-07| 2020-04-22 13-18-08| sector (177:41) saved to "D:\SteamDDSuite\Build\Active\save\BORED_Space_Bois\sectors\177_41" 2020-04-22 13-18-09| saving sector (177:35) 2020-04-22 13-21-13| saving sector (108:175) 2020-04-22 13-21-14| sector (108:175) saved to "D:\SteamDDSuite\Build\Active\save\BORED_Space_Bois\sectors\108_175" 2020-04-22 13-21-14| Player Hewitt moved to sector (178:38)from (161:81) server time taken for change: 124ms SectorDatabase: Loading Sec2020-04-22 13-21-14to| r (S17e7c:t41o)r 2020-04-22 13-21-14| Database: Loading Sector (177:35) 2020-04-22 13-21-14| scheduled save for sector (178:38), 0x3859f7d0, entities: 579 2020-04-22 13-21-14| saving sector (178:38) 2020-04-22 13-24-58| scheduled save for sector (280:384), 0x3dea5410, entities: 1275 2020-04-22 13-24-58| saving sector (280:384) 2020-04-22 13-24-59| Player kevin moved to sector (120:191)from (120:190) server time taken for change: 0ms Sect2020-04-22 13-24-59o| rDSaetcatboarsDeat:a bLasoea:d Lionagdi nSge cSteocrt or( 1(2132:11:8189)1 2020-04-22 13-24-59| ) 2020-04-22 13-24-59| sector (280:384) saved to "D:\SteamDDSuite\Build\Active\save\BORED_Space_Bois\sectors\280_384" 2020-04-22 13-24-59| scheduled save for sector (293:340), 0x7d699470, entities: 1579 2020-04-22 13-24-59| saving sector (293:340) 2020-04-22 13-24-59| Server frame took over 1 second. 2020-04-22 13-25-57| Sector[120:191]::buildUpdateChain: 0.0008684 2020-04-22 13-25-57| Sector[113:174] deferred update Hangar t_0 elements: 10: 0.0008665 2020-04-22 13-25-57| Sector[119:219]::buildUpdateChain: 0.0008634 2020-04-22 13-25-58| Player Handalaan moved to sector (219:79)from (223:71) server time taken for change: 0ms SectorDa2020-04-22 13-25-58t| abaSsee:c tLooraDdaintga bSaecsteo:r (Lo2a2d0i:n7g8 )S2020-04-22 13-25-58| ecto r (218:88) 2020-04-22 13-25-58| scheduled save for sector (219:79), 0x7d69ef10, entities: 1808 2020-04-22 13-25-59| saving sector (219:79) 2020-04-22 13-25-59| Server frame took over 1 second. 2020-04-22 13-25-59| [ void Server::update(float): 1178.5 ms; 100% ] 2020-04-22 13-29-24| scheduled save for sector (117:227), 0xfdf8caf0, entities: 732 2020-04-22 13-29-24| saving sector (117:227) 2020-04-22 13-29-25| Player kevin moved to sector (110:228)from (117:220) server time taken for change: 0ms Se2020-04-22 13-29-25| ctSeoctroDrDaattaababsea: sLoead:in gL Soecatodr i(n11g3: 2S26e)c 2020-04-22 13-29-25| tor (106:227) 2020-04-22 13-29-25| sector (117:227) saved to "D:\SteamDDSuite\Build\Active\save\BORED_Space_Bois\sectors\117_227" 2020-04-22 13-29-25| scheduled save for sector (119:219), 0xc2f8a530, entities: 447 2020-04-22 13-29-25| saving sector (119:219) I have also included my server.ini for review in case that helps you guys out. Thanks in advance for any support. Happy to get additional details around the issue to help work through it!