Dev Progress II: Out of sector updates & server performance

[koonschi]

DISCLAIMER: I will speak about what happened, progress and performance optimization over the last few days and since it's mostly been coding, it will get kind of technical. I could post some images but there really is nothing new on them that I could show you, so I don't see the point of doing it. This will most likely also be more like a blog post or something. We'll see.

 

Alliances & Progress

Over the last weeks we finished up on the alliances mechanics, meaning: Alliances are basically finished (feature-wise)! They need more testing, but the features are all in.

 

However, there's a lot of other features that we want to ship alongside alliances with the next update, the most important ones being Steam Workshop support for ship and station models and out-of-sector-updates for player ships and stations. We want to ship them all at once because we know that if we ship them one by one they will break their own database structures (ie. corrupting your saves) or we'd have to add a converter each single time (which is actually not an option if we have to do it for lots of small updates) since we're doing a lot of work in that area right now.

 

I also improved how player ships are saved and, most importantly, restored when there are database corruptions. Right now ships are always stored in the sector files, since technically speaking, they belong to a sector. But this lead to annyoing ship losses in a few cases when the server crashes before a sector can be written to disk. With the new system, which will be shipped with the update, player ships and stations will be saved in the player's file. In addition to that, the server will now save multiple instances of the player's file, meaning even if a player file gets corrupted due to lack of HDD space, BSOD, a power outage or other reasons, there are 5 other fallback saves which you can access so at max only the progress of the last 5 minutes is lost. This also means that you can now transfer entire players, including their ships (!) to another server by copying that player's file. It also means that sector and player files no longer need to be written out every time a player changes sectors, which should improve performance of multiplayer servers dramatically.

 

This also allowed me to fix an issue where the entire database would get written out in some cases to ensure a consistent state. This has been identified as the major cause for those huge lags that tended to occur on large servers (like the public test server) whenever a player logged in or whenever a new faction was created (ie. when a player changed sectors into a region of a new faction).

 

Steam Workshop and Out-Of-Sector Performance Improvements

Steam Workshop implementation should be quite straightforward. If its ease of implementation is similar to the other Steam API features that I've used in the past it should be realized quite quickly. Performance optimization for holding lots of sectors in memory at once, however, has proven quite the problem (but a solvable one, phew) over the last few days.

 

When we're talking about keeping lots of sectors alive at once, The most important things are update performance and memory consumption of the server. Now, sectors without players are actually really lightweight when it comes to update performance, since the game is using a simplified update variant for these, however they can still easily use 100 - 150 MB of RAM per sector while in memory. And players tend to visit the populated sectors more often since they're a lot more interesting than empty space. So we have to improve on the memory consumption most importantly.

 

So we profiled what was actually causing the high memory consumption. At first we thought "obvious, it's all the geometry of the objects". After measuring memory consumption before and after stripping down all the plans of all the objects in several highly populated sectors we realized that those only took up 2 - 5 MB of RAM, out of 130MB. So that clearly wasn't the cause for the high RAM consumption. Which was to be honest both scary and a relief. A relief because if it were the block structures, we'd have to strip them every time a player leaves a sector and re-add them every time a player reenters the sector, which would lead to a myriad of special cases where components and scripts would always have to check if a plan was actually loaded or not (have fun, modders, haha). But it was also scary since it had to be something that we hadn't thought of before, and that's never good.

 

After further investigation it turned out to be the lua scripts. In highly populated sectors we saw that there were on average around 220 scripts active (basically nearly every single station or ship interaction is its own script). But it wasn't the scripts themselves, it was the fact that we were using a single lua VM for each unique script. This might sound ludicrous, but at the time it seemed like a good idea since these VMs are actually rather lightweight, and each script could be cleaned up easily when it's being detached or deleted. Well, they were lightweight until we started adding the Avorion Scripting API, which blows each VM's memory consumption from 5 - 20KB (depending on which built-in lua libs we offered) to at least 130KB. 130KB times 220 is 28.6 MB, so we had at least 28.6 MB per sector. And we're not yet counting any initialization or actual scripts, this is just the absolute minimum basis per script. Initialized scripts could take 500KB to 2.5MB. After profiling some more we saw that the scripts used up to 80 MB per sector - quite the amount. The big issue here is that when using a separate VM for each script, there will have to be a lot of redundant data available to each VM, since the VMs themselves don't know that there are other VMs.

 

The improved Scripting API

So what I've come up with now is simple: Use a single lua VM for each entity, and no longer for each script. We were forced to make a few script API changes in order to do this, but the good news is: We managed to do it in a way that will NOT kill current mods! I'd still strongly advise modders to adjust their scripts to make use of the improved memory consumption, though. The changes are as follows:

• 
  Since every script defines functions that will be called from the main game, and these functions have to have the same names, I've added modules or namespaces (I'll decide on which term we'll use later). Each script can define a namespace/module in which its functions will live that it'll expose to the main program. By separating them into different modules or namespaces they will no longer have the same name in the VM and they won't get overridden when another script is loaded into the VM.
  
• When loading a script, Avorion will check if it's using a namespace, and if yes and if there are no duplicates it will be injected into the existing lua VM of the object.
  If no namespace is defined or the namespace exists already (basically whenever there's a potential for methods being overridden by the newly loaded script) the script will be run in a new VM. This is basically exactly the same as the old model that we've been using so far.
  
• We've profiled this change and so far it's been looking very promising. Script memory consumption has been more than halved and it looks like we might even be able to reduce it to 20% of its previous amount, which is actually crazy good. We only adjusted the major scripts and we already got the memory consumption down from 130MB to ~40MB for four loaded sectors. We'll be on the lookout for more of these optimizations, but that's what we've got so far.
  

Another great feature of the new system is that you can now easily access other scripts in the same entity - provided that they're loaded in the same VM (which they should be, by default): Just check and access the namespace of that script. With these changes we also paved the way for improved modding support where you won't have to merge files manually or with tools but should be able to just copy them into, let's say, a specific folder and they'll work.

 

Alright that's it for now!

If you made it here, congratulations and thanks for reading! Hopefully I managed to give you an insight on what we're doing right now.

 

OMG I DON'T CARE UPDATE WHEN!?

I won't give another date right now because this stuff is really (and I mean REALLY) hard to predict, but I want you all to know that we're working hard on new features and improving the game and we've been making some great progress over the last weeks.

 

As I said in the last announcement: Software engineering is a complex thing and there can always be unforeseen issues (these issues I just talked to you about are basically the proof) so telling an exact date is hard if we want to ship with as little issues as possible. There's a reason why lots of companies just say "when it's done".

 

But to give you at least some feedback: About 50% of the work of the update is done, and hopefully the rest will be more straightforward so let that be an indicator for you.

 

Oh, and there will be more new screenshots of nice new features in the next update, promise :)

 

Have fun and see you all soon!

[Ohm is Futile]

Thanks a lot for this this. It's very nice of you to bother telling us the details of what's going on "backstage". It's also great that you care about preserving existing saves and ships given this is early access. Most of all, I appreciate the fact that you're working on optimizing features before releasing them given how tempting it might be to just keep pushing out stuff to keep the hype train going and quiet down the "we need more content" crowd.

 

It relieves me, too, because that means I got time to do things before everything changes, so whatever I do now doesn't feel like it will go to waste (especially since I've got limited time).

 

Anyways, I'm thrilled about what's coming, but I can wait. Keep doing what you're doing, you're probably my favourite game dev right now, one of the few projects that I've seen the potential for that doesn't feel like it will get ruined for all the worst reasons.

 

<3

[Fox]

Lots of things happening, and nice to see how stuff gets on its way... ty 4 the keeping us up to date!

...

Performance optimization for holding lots of sectors in memory at once, however, has proven quite the problem (but a solvable one, phew) over the last few days.

 

When we're talking about keeping lots of sectors alive at once, ... without players ... they can still easily use 100 - 150 MB of RAM per sector ...

Fascinating read! Not that i can say i have lots of knowledge in programming ("Hello World!"-esque), but getting a glimpse on what exactly is happening behind the scenes of pretty pics and helpful Interfaces was very interesting, good to hear the (from my Point of View) hardest part is comming along!

 

... you can now transfer entire players, including their ships (!) to another server ...

 

... we also paved the way for improved modding support where you won't have to merge files manually or with tools but should be able to just copy them into, let's say, a specific folder and they'll work. ...

\o/

 

... Hopefully I managed to give you an insight on what we're doing right now.

I'd say yes, Mission accomplished, effort much appreciated! Cant speak for others, but at least me had a good time reading 8)

 

Cheers & keep up the nice work, Devs!

 

EDIT:

"Some things i forget to mention!"

+1

[Tchey]

Always a little frustrating when a game is not getting quick updates when on Early Access.

Always a little releasing when the devs are well active and communicative.

Always my pleasure to read about game development process and what's going on when where how and all.

[chronoss2008]

nice looking forward

 

:)

[unbekannt1984]

Would it be possible to put those Performance-Optimizations into a Beta-Build? That way we could try those, report Bugs (well, testing)...

[cy414]

uh, i think we broke koonschi. he thinks he's a we now.

 

unless of course i missed something and he finally has some help.

 

:P

[koonschi]

We've been a "we" since the Kickstarter, it's just that over the last couple of months Philipp had to focus on his graduation (which, of course, has absolute priority). Which is why you've mostly seen me. But these investigations and lots of the decision making was done by the both of us.

[cy414]

ah, ok. only got the game when it went early access on steam and was under the impression it was a one man show.

while that'd have been impressive, good to know you arent going to go insane quite so soon.

[Jack_Miron]

Would it be possible to put those Performance-Optimizations into a Beta-Build? That way we could try those, report Bugs (well, testing)...

 

Sure thing

[Super_Chicken]

With the amount of performance upgrades and optimization updates as well as the whole new implementation of Alliances I do believe the one question on a lot of people's minds is that if this update is going to cause a mandatory reset of already established servers?

 

I do understand that this game is in Beta and that you should expect resets and server wipes. I just think it would be nice to know ahead of time instead of getting this whole production line set up for this update and then get a "Oh, by the way..."

 

Otherwise I am VERY excited for this update and cant wait. Great job Gents

[AikawaKizuna]

Very detailed explanation of current progress.Thank you.

You are doing a better job than those devs who said,"fixed many mysterious bugs" ;)

 

 

[ErthParadine]

Your updates continue to impress: showing consistent progress, confidence, and competency. Can't decide if I'm looking more forward to the next dev report, or the next release. Thanks for the wonderful communications!

[chronoss2008]

wait what...there was an update to the game....no just what hes up too....and im not impressed

 

its like someone saying they will meet you in a few hours then shows up a few weeks later....i get it koonshi lots to do more then you prolly thought ....try saying it....anyhow ive logged into the game once ina  week the crew costs are so much i wont till i hear of a real update

[CaptPatrick01]

Huh... I wasn't aware the crew salary wasn't actually supposed to be as high as it is. I always assumed the auto-payment system just deferred the payments 3 times for some reason (as dumb as that sounds). Then again, that's probably because I have never gotten very far in the game before the dreaded SP inventory loading crash happens (because Research Stations are just too darn rare). So I never experienced roadblockingly huge salary payments firsthand, just 150,000 at most, which I easily made up through pirate-hunting, or selling force-turrets.