Jump to content
Agent-Piano

Dedicated Server - CPU/Client Ping spike after sector generation/load

Recommended Posts

Hello!

 

I've been running a dedicated server for about one week that is averaging between 5-10 clients. In the last few days we started seeing issues where client "ping" goes to about 5s. This is then resolved by restarting the game server. I have been able to pin down when this starts by way of CPU performance charting for the system. Using those timestamps I've stepped back through server logs in attempt to identify any trace data. With profiling enabled, I'm seeing this entries which definitely "look" pretty bad. Thoughts?

 

Also, the log file is too large for pastebin, alt. upload link below...

 

https://gofile.io/?c=5oQHuB

 

Server Hardware

Windows Server 2019

VMware Hardware V11

6 vCPU (3 Sockets x 2 Cores) - 14000MHz Reservation, High Share Priority

16GB Memory

SSD Based Storage

 

VMware Host

HP Proliant DL360 G7 - ESXi 6.0 15517548

2 x Intel Xeon E5620 @ 2.4GHz (16 Logical) -  (Aprox 25% utilization)

120GB Memory (Aprox 40% utilization)

 

 

2020-04-22 13-18-07| saving sector (177:41)

2020-04-22 13-18-07| Player kevin moved to sector (119:219)from (119:220) server time taken for change: 0ms

S2020-04-22 13-18-07e| ctoSrSeDecactttooarrbDDaaattsaaebb:aas seeL::  oLLaoodaaiddniignn ggS  eSSceetccottroo rr(  (1(1112720:0:2:1229021)3)

2020-04-22 13-18-07| )2020-04-22 13-18-07|

2020-04-22 13-18-08| sector (177:41) saved to "D:\SteamDDSuite\Build\Active\save\BORED_Space_Bois\sectors\177_41"

2020-04-22 13-18-09| saving sector (177:35)

 

2020-04-22 13-21-13| saving sector (108:175)

2020-04-22 13-21-14| sector (108:175) saved to "D:\SteamDDSuite\Build\Active\save\BORED_Space_Bois\sectors\108_175"

2020-04-22 13-21-14| Player Hewitt moved to sector (178:38)from (161:81) server time taken for change: 124ms

SectorDatabase: Loading Sec2020-04-22 13-21-14to| r (S17e7c:t41o)r

2020-04-22 13-21-14| Database: Loading Sector (177:35)

2020-04-22 13-21-14| scheduled save for sector (178:38), 0x3859f7d0, entities: 579

2020-04-22 13-21-14| saving sector (178:38)

 

2020-04-22 13-24-58| scheduled save for sector (280:384), 0x3dea5410, entities: 1275

2020-04-22 13-24-58| saving sector (280:384)

2020-04-22 13-24-59| Player kevin moved to sector (120:191)from (120:190) server time taken for change: 0ms

Sect2020-04-22 13-24-59o| rDSaetcatboarsDeat:a bLasoea:d Lionagdi nSge cSteocrt or( 1(2132:11:8189)1

2020-04-22 13-24-59| )

2020-04-22 13-24-59| sector (280:384) saved to "D:\SteamDDSuite\Build\Active\save\BORED_Space_Bois\sectors\280_384"

2020-04-22 13-24-59| scheduled save for sector (293:340), 0x7d699470, entities: 1579

2020-04-22 13-24-59| saving sector (293:340)

2020-04-22 13-24-59| Server frame took over 1 second.

 

2020-04-22 13-25-57| Sector[120:191]::buildUpdateChain: 0.0008684

2020-04-22 13-25-57| Sector[113:174] deferred update Hangar t_0 elements: 10: 0.0008665

2020-04-22 13-25-57| Sector[119:219]::buildUpdateChain: 0.0008634

2020-04-22 13-25-58| Player Handalaan moved to sector (219:79)from (223:71) server time taken for change: 0ms

SectorDa2020-04-22 13-25-58t| abaSsee:c tLooraDdaintga bSaecsteo:r  (Lo2a2d0i:n7g8 )S2020-04-22 13-25-58| ecto

r (218:88)

2020-04-22 13-25-58| scheduled save for sector (219:79), 0x7d69ef10, entities: 1808

2020-04-22 13-25-59| saving sector (219:79)

2020-04-22 13-25-59| Server frame took over 1 second.

2020-04-22 13-25-59| [ void Server::update(float): 1178.5 ms; 100% ]

 

2020-04-22 13-29-24| scheduled save for sector (117:227), 0xfdf8caf0, entities: 732

2020-04-22 13-29-24| saving sector (117:227)

2020-04-22 13-29-25| Player kevin moved to sector (110:228)from (117:220) server time taken for change: 0ms

Se2020-04-22 13-29-25| ctSeoctroDrDaattaababsea: sLoead:in gL Soecatodr i(n11g3: 2S26e)c

2020-04-22 13-29-25| tor (106:227)

2020-04-22 13-29-25| sector (117:227) saved to "D:\SteamDDSuite\Build\Active\save\BORED_Space_Bois\sectors\117_227"

2020-04-22 13-29-25| scheduled save for sector (119:219), 0xc2f8a530, entities: 447

2020-04-22 13-29-25| saving sector (119:219)

 

I have also included my server.ini for review in case that helps you guys out.

 

[Game]

Seed=O3sWV4jQNd

Difficulty=0

HardcoreEnabled=false

InfiniteResources=false

CollisionDamage=1

SafePlayerInput=false

PlayerToPlayerDamage=true

LogoutInvincibility=true

LogoutInvincibilityDelay=30

DevMode=false

ExplicitCallables=true

BigWreckageDespawnTime=1800

SmallWreckageDespawnTime=900

MaximumFightersPerSectorAndPlayer=-1

MaximumBlocksPerCraft=-1

MaximumVolumePerShip=-1

MaximumVolumePerStation=-1

MaximumPlayerShips=-1

MaximumPlayerStations=-1

MaximumBlocksPerTurret=250

PlayerInventorySlots=1000

AllianceInventorySlots=1000

Version=1.0

sameStartSector=true

startUpScript=data/scripts/server/server.lua

startSectorScript=startsector

[system]

saveInterval=300

sectorUpdateTimeLimit=600

emptySectorUpdateInterval=0.5

workerThreads=32

generatorThreads=16

scriptBackgroundThreads=8

aliveSectorsPerPlayer=10

weakUpdate=true

profiling=true

sendCrashReports=true

hangDetection=true

backups=true

backupsPath=

simulateHighLoadServer=false

sendSectorDelay=2

placeInShipOnDeathDelay=7

[Networking]

port=27322

broadcastInterval=5

isPublic=true

isListed=true

isAuthenticated=true

sendStatsToAdmins=true

useSteam=true

rconIp=

rconPassword=PASSWORD

rconPort=27323

[Administration]

maxPlayers=69

name=SERVERNAME

description=SERVERDESCRIPTION

password=PASSWORD

pausable=false

accessListMode=Blacklist

steamIdOverride=0

 

Thanks in advance for any support. Happy to get additional details around the issue to help work through it!

Share this post


Link to post
Share on other sites

Im also seeing this issue. The ping spikes are a pretty noticeable 2-3 seconds

 

Again with garbled sector database lines;

 

2020-04-23 18-52-01| Player Xeigho moved to sector (39:462)from (90:178) server time taken for change: 58ms

SectorDatabas2020-04-23 18-52-01e| : SLeocatdoirnDga tSaebcatsoer:  (L4o0a:d4i6n3g)

2020-04-23 18-52-01| Sector (38:461)

2020-04-23 18-52-01| scheduled save for sector (39:462), 0x7f88dc001d20, entities: 560

2020-04-23 18-52-01| saving sector (39:462)

2020-04-23 18-52-01| sector (39:462) saved to "/home/steam/avion/galaxies/xeigho/sectors/39_462"

2020-04-23 18-52-01| scheduled save for sector (90:178), 0x7f88dfda66f0, entities: 1935

2020-04-23 18-52-01| saving sector (90:178)

2020-04-23 18-52-02| sector (90:178) saved to "/home/steam/avion/galaxies/xeigho/sectors/90_178"

2020-04-23 18-52-04| scheduled save for sector (38:461), 0x7f885696c5a0, entities: 1191

2020-04-23 18-52-04| saving sector (38:461)

2020-04-23 18-52-05| sector (38:461) saved to "/home/steam/avion/galaxies/xeigho/sectors/38_461"

2020-04-23 18-52-05| scheduled save for sector (40:463), 0x7f87fe263840, entities: 1645

2020-04-23 18-52-05| saving sector (40:463)

2020-04-23 18-52-06| sector (40:463) saved to "/home/steam/avion/galaxies/xeigho/sectors/40_463"

2020-04-23 18-52-40| SectorDatabase: Loading Sector (43:447)

2020-04-23 18-52-43| scheduled save for sector (43:447), 0x7f885d67f750, entities: 1622

2020-04-23 18-52-43| saving sector (43:447)

SectorData2020-04-23 18-52-44ba| se: LoaSdiengc StecotrorD a(t43a:b44a4)s

2020-04-23 18-52-44| e: Loading Sector (42:456)

 

 

Share this post


Link to post
Share on other sites

Your logfiles look fine, just the logfile itself is a lil bugged, a harmless issue.

 

You should change your server.ini:

 

workerThreads=32
generatorThreads=16
scriptBackgroundThreads=8

CHANGE TO:

workerThreads=6
generatorThreads=3
scriptBackgroundThreads=3

 

Further information regarding multithreading on servers:

https://avorion.gamepedia.com/Setting_up_a_server#Multithreading

 

But I guess you are having performance issues cause hosting Avorion on a shared server usely does not work very well... Maybe try to add more cores and a lil more ram to the VM....

 

Share this post


Link to post
Share on other sites

Your logfiles look fine, just the logfile itself is a lil bugged, a harmless issue.

 

You should change your server.ini:

 

workerThreads=32
generatorThreads=16
scriptBackgroundThreads=8

CHANGE TO:

workerThreads=6
generatorThreads=3
scriptBackgroundThreads=3

 

Further information regarding multithreading on servers:

https://avorion.gamepedia.com/Setting_up_a_server#Multithreading

 

But I guess you are having performance issues cause hosting Avorion on a shared server usely does not work very well... Maybe try to add more cores and a lil more ram to the VM....

 

Thanks for the response, over the weekend I adjusted the configuration from 3 x 2 to 1 x 6 which seems to have led to some stability. I have also made the adjustment you recommend to provide hardware concurrency. From a CPU/RAM standpoint, neither the VMware host or the VM are tensioned, I could look at expanding the vCPUs laterally but based on what I'm seeing in performance data it looks as though those resources would not be used.

 

Will be interesting to see how the game server performs with these changes.

Share this post


Link to post
Share on other sites

Your logfiles look fine, just the logfile itself is a lil bugged, a harmless issue.

 

You should change your server.ini:

 

workerThreads=32
generatorThreads=16
scriptBackgroundThreads=8

CHANGE TO:

workerThreads=6
generatorThreads=3
scriptBackgroundThreads=3

 

Further information regarding multithreading on servers:

https://avorion.gamepedia.com/Setting_up_a_server#Multithreading

 

But I guess you are having performance issues cause hosting Avorion on a shared server usely does not work very well... Maybe try to add more cores and a lil more ram to the VM....

 

After continued performance testing, we're still seeing a large amount of issues. It seems to be that once the server goes over a population of about 5-6 players the server becomes very unstable. Recently we experienced an evening where the server mass-disconnected all players three times in a fifteen minute period.

 

Reviewing performance charts, there are not utilization or latency issues related to CPU, Memory, Disk, or Network. While it is operating in a shared environment there isn't any information that supports a hardware related problem. Any other thoughts on how to troubleshoot or remediate the problem?

Share this post


Link to post
Share on other sites

We are also now seeing issues where player owned ships and stations are sometimes duplicating or cloning after players are disconnected from the server. Players are getting disconnected somewhat frequently. Sometimes the clone items disappear and sometimes they remain permanent.

 

In other instances, ships are jumping to the incorrect system and to random coordinates far out of the range of their jump range (full quadrants, etc.).

 

Hardware performance for the system continues to be very strong with no evidence of limitations. Still unsure as to why this application is struggling as much as it is.

 

 

Share this post


Link to post
Share on other sites

We are also now seeing issues where player owned ships and stations are sometimes duplicating or cloning after players are disconnected from the server. Players are getting disconnected somewhat frequently. Sometimes the clone items disappear and sometimes they remain permanent.

 

In other instances, ships are jumping to the incorrect system and to random coordinates far out of the range of their jump range (full quadrants, etc.).

 

Hardware performance for the system continues to be very strong with no evidence of limitations. Still unsure as to why this application is struggling as much as it is.

 

6 vCPU (3 Sockets x 2 Cores) - 1400MHz Reservation, High Share Priority
16GB Memory

 

6x 1400MHz is just not enaugh, as I said above this is no good setting for hosting Avorion. Depending on number of players online 16bg may also not be enaugh. You should upgrade.

 

Share this post


Link to post
Share on other sites

We are also now seeing issues where player owned ships and stations are sometimes duplicating or cloning after players are disconnected from the server. Players are getting disconnected somewhat frequently. Sometimes the clone items disappear and sometimes they remain permanent.

 

In other instances, ships are jumping to the incorrect system and to random coordinates far out of the range of their jump range (full quadrants, etc.).

 

Hardware performance for the system continues to be very strong with no evidence of limitations. Still unsure as to why this application is struggling as much as it is.

 

6 vCPU (3 Sockets x 2 Cores) - 1400MHz Reservation, High Share Priority
16GB Memory

 

6x 1400MHz is just not enaugh, as I said above this is no good setting for hosting Avorion. Depending on number of players online 16bg may also not be enaugh. You should upgrade.

 

The system isn't running 6 x 1400MHz. It's running at 6 x 2.4GHz and there is a cpu reservation for 14000* MHz on the host. The original number of 1400 was a typo, my apologies for that.

 

So, the system running it has 6 x 2.4GHz processors which totals out to 14400MHz. Of that, 14000MHz is reserved on the host for only that VM so it is not competing for resources. While the wikipedia articles and FAQs seem to refer a lot to core and socket allocation I have not found any details which explain single thread speeds and what the requirements are.

 

Based on that information, can you elaborate on your comments that my CPU configuration may not be adequate?

Share this post


Link to post
Share on other sites

Based on that information, can you elaborate on your comments that my CPU configuration may not be adequate?

I guess your whole hardware is too old, also it is only virtualized. Can you reproduce your issues on a I7 or I9 CPU, using a dedicated machine and not a VM? Dedicated machines usely work better for hosting an Avorion server, also we still talk about a game - your Xeon CPU may result in a bad performance. You could try a benchmark and compare to your desktop computer or so...

I just watched your CPU... it is 10 years old, you really should get better hardware. Also I recommand a Desktop CPU, not a server CPU. I have no issues on my server with 20 people online, but you need a high-end server for a really good performance.

 

Share this post


Link to post
Share on other sites

Based on that information, can you elaborate on your comments that my CPU configuration may not be adequate?

I guess your whole hardware is too old, also it is only virtualized. Can you reproduce your issues on a I7 or I9 CPU, using a dedicated machine and not a VM? Dedicated machines usely work better for hosting an Avorion server, also we still talk about a game - your Xeon CPU may result in a bad performance. You could try a benchmark and compare to your desktop computer or so...

I just watched your CPU... it is 10 years old, you really should get better hardware. Also I recommand a Desktop CPU, not a server CPU. I have no issues on my server with 20 people online, but you need a high-end server for a really good performance.

 

First of all, thank you continuing to stay engaged in the conversation to provide feedback. My hope is to work through understanding what kind of CPU requirements the dedicated server actually needs as they are not particularly well documented on the wikipedia.

 

My observation in troubleshooting this issue is that the dedicated server engine needs a faster single-thread speed and that this is not a multi-core/socket issue. I have tried several configurations where I have added and reduced the CPU configurations for the Virtual Machine and the server engine simply does not use the extra processing power. The same is true with memory as I have added several GBs of additional memory and the server engine simple does not use it (even with adjustments to sectors in memory, etc.).

 

With that said, my suspicion is that the limitation is more around CPU frequency (the speed of single threaded operations).

 

I am in the process of implementing a new VMware host which has much higher CPU clock speeds and will test my theory, but having a grasp of what other folks are running without issue would be helpful. There is no reason that a Xeon class processor would not be able to run this server engine, but I do suspect that it's going to be looking for a minimum clock speed to run well and is not going to take advantage of having a significantly wider thread/CPU layout. Some i7 and i9 processors have much higher clock speeds, but that is not true for all of them. Accordingly, simple moving to an i7 or i9 isn't inherently going to lead to better performance. There is going to be some sort of coefficient of what the actual performance of the processor is versus the performance of the server engine.

 

With that said, can you share what CPU configurations you're running on the system that hosts your server? I'm specifically looking for the model of CPU, memory speeds, and ideally chip-set. Being able to study the FSB speeds, cache speeds, and frequency will help us put some scientific data behind what is fast and slow If you have a post where you've already explained this I'd be happy to read it, just need some kind of cross-link.

 

Thanks again for your time.

Share this post


Link to post
Share on other sites

With that said, can you share what CPU configurations you're running on the system that hosts your server? I'm specifically looking for the model of CPU, memory speeds, and ideally chip-set. Being able to study the FSB speeds, cache speeds, and frequency will help us put some scientific data behind what is fast and slow If you have a post where you've already explained this I'd be happy to read it, just need some kind of cross-link.

 

Thanks again for your time.

I dont know memory clock or chipset. I rented a dedicated root server, guess its running at default settings, google the CPU specs.

 

I9 9900k

64G ram

SSD

Share this post


Link to post
Share on other sites
Guest
Reply to this topic...

×   Pasted as rich text.   Paste as plain text instead

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.


×
×
  • Create New...