Chain crashing server

[Hammelpilaw]

We have a chain crash problem on our server, that seems to be related to a specific player (Baridor). When the player is not online and sectors with his ships and stations are not active, there is no issue.

 

I thought there might be an issue in an entity or sector. So we deleted most of his entities and regenerated sectors, but nothing seems to fix it.

 

2017-11-25 16-16-00| Error: EXCEPTION_ACCESS_VIOLATION
2017-11-25 16-16-00| [1] ?? 6751232
2017-11-25 16-16-00| [2] ?? 6107660
2017-11-25 16-16-00| [3] ?? 6045273
2017-11-25 16-16-00| [4] ?? 6082818
2017-11-25 16-16-00| [5] ?? 7323042
2017-11-25 16-16-00| [6] ?? 10804158
2017-11-25 16-16-00| [7] ?? 12387688
2017-11-25 16-16-00| [8] ?? 13052367
2017-11-25 16-16-00| [9] ?? 10032548
2017-11-25 16-16-00| [10] beginthreadex 140704628716218
2017-11-25 16-16-00| [11] endthreadex 140704628716428
2017-11-25 16-16-00| [12] BaseThreadInitThunk 140704629162852

 

The error looks always like this. Does anybody have another idea what could be wrong?

 

The server is modded, but the modpack used to work, and it still seems to work when Baridor is not online, so I dont thik the issue is related to our mods.

serverlog_2017-11-29_19-17-04.zip

[WSY]

If the player is offline and you go into a sector where his ships are will it crash?

 

When you Regen the sectors, does that kill off fighter entities? Had a buddy of mine manage to build a rapid fire cannon with cool down, and built a fighter with it and the fighter was bugged which crashed the sector when it fired.

 

Got rid of the fighter and that cleared it up.

 

Just a thought.

[Hammelpilaw]

Once I had some ships in his main sector (with ships and factories) to keep it active, then it crashed. But I do not remember wich entities was active in his main sector. But there is no sector that crashes the server as soon as it is active. Sometimes it crashes when any specific sector gets active, but sometimes it does not crash when this sector is active for a longer time. I already regenerated a sector that seems to crash (not sure if it did), but it did not solve the issue.

 

When I regenerate a sector all fighers are deleted.

 

Up to now this issue only happened, when the server setting "aliveSectorsPerPlayer" was higher then 0...

 

His figherts should work at all. I use the same type of figher (created of the same type of weapon).

 

Could a bugged ship-xml cause those issues?

[WSY]

A bugged ship xml in a specific sector seems like it could, but this seems (from the way you describe it), that the sectors that are having an issue sometimes work for a while.

 

Maybe script related? 

 

Or a ship performing a certain action in those sectors, so the ship, or the xml it uses is buggy when a specific situation presents itself?

 

Do the sectors in question have several stations close to each other? Maybe the game trying to spawn cargo vessels causing an issue? This of course beings me back to thinking a possible script issue.

 

What mods do you have?

 

 

 

[Hammelpilaw]

It may be script related, we use lot of mods...

 

Complex3s

Out of sector production

DockBuilder

ExtraminesM

AutoResearch

MineCorp

GuardianScout

CarrierCommander

 

This are only the most important mods. Ofc all mods are compatible or merged to the current Avorion version and working well. Cause there are no error messages in the logs I think they should be ok... but cant say for sure. Oh and i forgot to say that the server used beta branch.

 

Yesterday it crashed once when the related player was not online. This happened after I let 2 ships (controlled by captains) launch 240 fighters to salvage a scrapyard. Baridor uses even more fighters.

Maybe this is the issue. I will try to reproduce this by using even more fighters.

[Hammelpilaw]

Ok now I am able to reproduce this issue with vanilla scripts. The galaxy is still created by modded scripts, and I cant say for sure it is not bugged by mods, but when I load the savestate (no matter if modded or vanilla sripts are used) and calculate a jumping route into some specific sector, the server crashes. It is not only one sector that causes crashing, i found at least 2 different sectors that causes a server crash when I calculate jumping route as soon as i logged in.

 

This does not help to find the issue yet... but maybe anybody got an idea what to do to find the issue?

[WSY]

Are you able to analyze the sector files to find similar entrees?

 

Thought for sure you had it with the fighters. With carrier command scripts maybe getting confused.

 

Can you make a copy of sectors somewhere else on hard drive, then delete so the server regenerates them?

 

Are you able to use gates to access those sectors?

[WSY]

Hey Hammel did you ever figure this out?

 

 

[Hammelpilaw]

Hey there, im not 100% sure yet, but I think that this issue is really related to our fighters. Tried to reproduce in a fresh vanilla galaxy, but without any success. At least our server did not crash since we stopped using our best salvage fighter. When there were 240 or more of them active in one sector the server seemed to crash sometimes.... however, cant say anything for sure, thats why I did not post any update.

 

I do not think it is related to the carrier commander mod. Also our sectors with 240+ fighters, wich may have cause the crashes, did not use this mod.

 

So... I figured out a lot of stuff, but nothing really helped to find the issue. I dont know what else to do to find it. At least it seems to work right now.

[Hammelpilaw]

I just managed to reproduce this issue in a fresh 100% vanilla galaxy. It must be some kind of issue with entities when using many fighters. Could still not find out though what exactly it is... In this test galaxy I got 120 salvage fighter in a scrapyard. Reducing the active sectors per player to 0 seems to reduces the issue, but it does not fix it.

 

Download the galaxy

 

All the entities I have are in these sectors: 0:2, 0:1, -1:1.

 

So it is not related to any special fighter. Few minutes ago our live server seemed to crash by cargo fighters.

 

Edit: Since we got a windows server all crashes happened on Windows. Maybe it is a OS related issue.

[WSY]

Hey Hammel, over the next day or two I will experiment with this save as I am very curious. I am planning on reloading my server into a much harder difficulty with a few mods and I am hoping to make fighters play more of a role. Especially with PD coming, so would be great to find out what causes it.

 

Have you messed with the max fighter per person/sector for the Galaxy?

 

 

[Hammelpilaw]

Have you messed with the max fighter per person/sector for the Galaxy?

 

Did not set max fighter, its still on default value (-1). Settings file is included in the save state.

The same I used to do in our live galaxy. Yesterday I changed this setting on our live server and hopefully this helps to get it working stable until this issue is fixed.

[WSY]

Hey Hammel, sorry have not had time to check on this, holiday time is family time, so next week I will try to experiment with it.

 

 

 

 

[Hammelpilaw]

Hi WSY, thanks for your reply. But dont hurry and enjoy christmas ;)

Our server did not crash since i limited max fighter per player/sector to 50. So currently I think we just gotta wait until this issue get fixed by Koon.

[Hammelpilaw]

Any news about this bug? Has it been fixed or is it going to be fixed soon?

[Hammelpilaw]

In version 0.16.0 the crash logging got improved. Today our server crashed again and console printed out:

 

11:48:32:235 An exception occurred: boost::filesystem::status: Zugriff verweigert: "C:\GameServer\Avorion\galaxy\Galaxy-Gamers\sectors\6_7~"
11:49:49:270 threadtools.cpp (3294) : Assertion Failed: Illegal termination of worker thread 'Thread(0x000000005DF3B5E0/0x000'

 

Server log:

2018-02-18 11-48-32| An exception occurred: boost::filesystem::status: Zugriff verweigert: "C:\GameServer\Avorion\galaxy\Galaxy-Gamers\sectors\6_7~"

 

I'm think this is related to the issue of this topic.