Jump to content

BloodyIron

Members
  • Posts

    20
  • Joined

  • Last visited

Everything posted by BloodyIron

  1. Avorion is a fun game, in my opinion, that's a reality. Back in March, when the game launched 1.0, and the Pandemic stuff started being a thing, I took it upon myself to mobilise myself and my org's staff to whip up an updated Avorion server and promote it to try and build a community, plus maybe help some people keep sanity in this madness. That was March. I didn't realise that the server software was going to perform as abysmally as it did. We have seen drastic performance issues with only 5-12 people, and that's with absurdly over-provisioned specs. I posted and reported on this matter as early as March 22nd : https://www.avorion.net/forum/index.php/topic,6443.0.html Between now and then we've seen a few Beta updates (which we haven't tested, because beta in production is bad practice), and a few aspects of those updates have mentioned server performance. I don't know if those would address the issues we've experienced, but the optimist in me says there's a good chance it would. Fastforward, it is now May 26th. Yes, I know, there's a literal global pandemic going on, but I'm not going to wait around forever running my server with the devs not releasing something that is literally crippling my server, and many other servers. Nobody wants to play on servers with the lag we've experienced, and it most certainly is not anything we're doing wrong, nor how many resources we're allocating. So when exactly are we going to get any real updates that fix server performance? It's been months, and supposedly code is already written. I'm a patient person, but at the end of the day, I'm not going to wait around forever. If the devs aren't going to fix the performance issues, I'm eventually going to nuke my server and probably move onto other games. I want to run a massive server, and with the way the server code works now, that's literally impossible. This game could be awesome in scale. So devs, when? I don't think we've seen a single update since release that wasn't beta.
  2. Not sure what you're asking for about anything else in particular, so unsure how to answer that question. In regards to the storage, I've monitored it from within the system itself (and yes, this is inside a VM) but also at the storage level. i do not see IOPS contention at the storage level, and I have not observed iowait on the Avorion side (in the VM). The amount of storage traffic that is happening is nowhere near saturation of the network links, nor the IOPS that the storage can serve. I may not have mentioned that I used that server config link, was more pointing out that I exhausted multiple resources that I dug up online (including that one) before I posted here, as I would have preferred to solve this issue by myself, however the devs do need to get involved here.
  3. 1. We were already having performance issues with sectors at 5, and as we upped it to 7, and then later to 12, the problems stayed about the same. I only upped it because our player base was requesting it, and we observed no change in how bad the server was performing as we gradually upped it. 2. ZFS is on a dedicated NAS, not local to the hypervisor itself. FreeNAS is running the ZFS zpool, so yeah. 3. I take it you didn't read the model of CPUs we're using. I guarantee it has HT ;) 4. We had a much higher worker count, but saw zero benefit from it. When trying to profile it and look at the worker thread graphics that were generated, a lot of worker threads were doing nothing, so we lowered the count in an effort to increase efficiency. Either way, nothing changed. 5. At this point, I am 100% confident this is bad code, and I'm very optimistic about the next patch helping, as the beta patch notes already talk about server performance fixes. 6. Also, I turned profiling off to try and improve performance and reduce spam to the console (so I could actually read the console). Not sure why you're recommending it to be on in a situation that involves poor performnace. 7. I had already, multiple times, referenced the link you mention, as we were building the server.
  4. One of the gamers on my server is requesting I whitelist a client-side only mod (or so it looks), but I'm not really finding how to whitelist a clientside-only mod, only how to enable server-side mods. Can someone please point me in the right direction? :)
  5. How is your server load so much lower than mine? How many CPU cores did you throw at it, and what is the model of the CPU? Is your setup bare metal or a VM, or what? What is the type of storage backing your Avorion Server?
  6. Can't wait for server performance improvements. CHUG CHUG CHUG :(
  7. The server performance improvements can't come fast enough :(
  8. Also would really like it if borderless worked more appropriately. I'm on Ubuntu 19.10 using the Gnome DE. When I go borderless there's always something cut-off on the top or bottom, no matter where I place the window. The window really needs to resize to the usable space for whatever DE/OS/GUI is used. I have to use fullscreen to see everything and alt-tabbing minimises, which is pain.
  9. I really would like a better way to queue (and save) ordered commands for AI ships. Accidentally not pressing shift when issuing a command, wiping complex commands, is extremely frustrating, and there's better ways to do this. Such as a dedicated interface, or something like that. If I have to go do something to one of my ships doing loops, having to remake those loops is painful now, and as soon as I enter a ship, it gets wiped.
  10. I don't know what sectors people are in while logged off though, more hoping for a baked-in feature for the server here. But I like your attitude here :P
  11. Until I hear from the devs I'm going to hold off on doing that. What you propose is logical, however I don't know if there are abnormal outcomes of that, such as if players are in that space while I regen/wipe it, what happens then?
  12. So my alerting thresholds weren't quite ideal (95% usage, with a few percentage of inaccuracy, UGH) and my server ran out of space over night. I've since grown the disk plenty, and free disk space is readily available. Looks like a few factions were generated as new, and on startup it complains saying "reading of file ... fialed, moving on to the next instance, error: unexpected end of file", which are completely reasonable error messages, as the factions probably weren't fully written to disk. In scenarios like this, as an option for an admin, I'd like to be able to force re-gen (or complete generation of remaining) of the problematic factions. Since they are new factions, I would be okay with regenning them as people haven't really become reliant on them for very long (3 hours now?). This is an infrequent scenario indeed, but it would be nice to have an option (or it just do it) to do this, so we can reduce errors at the console log going forward, instead of increase them. Thoughts?
  13. The Server load didn't seem to increase until a particular player logged on, destroyed 5-6 ships and started salvaging them. So that seems like a probable candidate for a cause of the drastic server load increase. And an area where efficiency increases probably would help lots. Not 100% sure, but IMO worth looking into, devs. ;P
  14. So, curious situation, increased "aliveSectorsPerPlayer" to 12 (was 7), decreased "workerThreads" to 16 (was 32) & "generatorThreads" to 8 (was 16), leaving "scriptBackgroundThreads" at 16. And now the server seems to be running better. Right now just two players online, so we'll have to see how it holds up when more players jump back on, but the "server 100% load" alert for players is not currently happening (again, 2 players). Still think this game needs actual programmatic tuning, but figured I'd share this possible improvement.
  15. Can we please have the Trading Overview (when you have the module installed, of course) able to be bound to a button, so we can press that button instead of always having to click on the icon? That'd be hella useful.
  16. Well I certainly can understand that. I just want to make sure the right eyes are seeing this is all, so we can help get it addressed, by doing bug reports, debug filings, etc. :P So if this is the right forum, great. If it isn't, well I can post elsewhere? Hmmmm.
  17. Is this the wrong section here? Should I be posting elsewhere? The performance really is a big issue here. :(
  18. Also, the server load looks like this a lot : https://i.imgur.com/1dUy9ln.png As of that picture, we have only 8 gamers on the server.
  19. With the whole Pandemic thing going on and Avorion 1.0 releasing, we wanted to put together an event where we wiped our Galaxy and tried to get more gamers engaged on our server ( https://www.lanified.com/News/2020/Join-Our-Galaxy-Adventures ). We have been successful at getting more gamers to play regularly, however we have had performance issues with the Avorion dedicated server software just not scaling well at all. We cannot identify a hardware or resource bottleneck at this point in our environment, so we believe confidently (but would love to be proved wrong) that the issue lies with inefficiencies in the game server itself. Before we start uploading server logs and the like, I want to describe the resources allocated and what we've done so far, and stuff like that. Since we launched the new Galaxy we've had a steady stream of gamers, having anywhere from 4-13 gamers connected simultaneously (we have the limit currently set to 50), with people connecting from around the globe. We're in Canada. Pretty much the whole time the server has been going since we started this Friday the gamers keep seeing "Server load is over 100%" in the top left, and the server frame graphs (what I assume they are) in-game continually are either yellow or red, rarely green. When we see lots of red, hit registration goes out the window, and other wonkiness ensues, until it clears up for whatever reason. We can't yet find a pattern around this in regards to what the players are doing. I don't believe anyone has fighters yet, and most certainly there aren't any wars going on. As for the resourcing... the Avorion dedicated server runs in an Ubuntu 16.04 VM with regular updates. We may upgrade to 18.04 in the near future, but not right now as it's being used regularly. The VM resides on a striped-mirror array of SSD storage that runs ZFS, which is very fast. When we look at the ZFS IO stats we do not see any concerning level of IOPS or throughput that could be bottlenecking so we are very confident this is not a storage issue. We have, however, scaled up the CPU, RAM and workers for the server several steps to try to address the issue, and while it has helped, it has not properly eliminated the issue. First step: CPU: 6x cores RAM: 4GB Second step: CPU: 12x cores RAM: 8GB Third step: CPU: 16x cores RAM: 24GB As we upped the CPU we upped the workers (current values I'll list below), and the earlier steps did help, but we feel we have plateaued from the CPU/worker aspect at this point. The RAM upping certainly was necessary, as we are now using 12GB of RAM out of our 24GB. We can add more RAM if need be, but I see no reason to do that. Here are the current worker parameters: workerThreads=32 generatorThreads=16 scriptBackgroundThreads=16 We have tried 16/8/8 when we had 16x CPU cores, however we have not seen a difference between 16/8/8 and 32/16/16, so for now I am leaving it at 32/16/16 as I have to restart the server each time to change that (don't want to scare away/frustrate our gamers). I have also had profiling on multiple times, and took a look at the worker map html for the 32/16/16 configuration, generating that map when a "red storm" happened (red graphs), and I did not see anything stick out as "oh we need more workers" or something is problematic, etc. Additionally, when profiling is off, the server console frequently spits out "Server Frame took over 1 second" and then shows the frame chart kind of thing. The VM runs on Proxmox VE, effectively LinuxKVM at the core of the hypervisor, a very fast and good hypervisor. The VM currently is on a host that is a Dell R720 with the CPUs in it being 2x intel Xeon e5-2650 v0's. The Linux OS reports a load average of 6/6/6, so while we have lots of worker threads, I do not see any sort of thread queueing being an issue here. As for the upload bandwidth, the connection is 15mbps and we're using about 4-5mbps for the Avorion server, and nothing else is pushing us close to our upload limit. The gateway is a very reliable and fast pfSense box, and the CPU on that is not getting pinned, so it is not a routing performance issue (I see these issues on LAN too btw). So, at this point, I am very confident this is a software efficiency issue. I can provide server logs privately to the developers and other debug info where possible. I'm not comfortable posting that publicly. If there are any areas that I may have overlooked, I'm all ears, because I've ran out of online resources that I could dig up to help me with this problem. I read through all of the release notes for the last 2 years, and could not find anything to help me with this case. We really want to be able to scale this up to 20-30+ gamers, but based on what I'm seeing, the experience is going to suffer more and more as we scale up. Please help!
×
×
  • Create New...