brunobf

Hello Devs, I decided to attempt to setup the backup feature on my server again, and I'm still getting crashes when it's enabled. This has been happening for a long time and I just gave up last time after a lot of debugging. For reference, I'm using the server inside of a docker container (specifically this image: rfvgyhn/avorion:stable). I've mapped both the galaxy path to a volume, and the backup path to another volume. I've ensured that both paths have exactly the same user ownership (uid 1000) as well as permissions: drwxr-xr-x 11 1000 1000 4096 Feb 14 04:29 avorion drwxr-xr-x 2 1000 1000 6 Feb 14 00:43 avorion_backup During file server startup, I can see that it's setup to write to the correct path. I tried both leaving backupsPath empty or hardcoded to the same path. I can see from the initialization log that both are mapping to the same path: 2023-02-14 04-38-48| Backup creation enabled. Path: "/home/steam/.avorion/backups" 2023-02-14 04-38-48| Please ensure that the server has write-access to that path. What I also see in the log is the following: 2023-02-14 04-39-24| Creating emergency backup, waiting for database ... (0s); collecting data ... (1.56s); saving backup ... 2023-02-14 04-39-25| Server frame took over 1 second. 2023-02-14 04-39-25| 2023-02-14 04-39-36| Caught signal 15. Initiating shutdown. 2023-02-14 04-39-37| Cleaning up entity transfers ... If I search for the thread that exits last, and which I'm assuming is the one dumping the stack trace, I see the following: 2023-02-14 04-38-48| Starting thread [Game WorkerPool Thread 1]... 2023-02-14 04-38-48| Started thread [Game WorkerPool Thread 1] with id 140224147474176 ... 2023-02-14 04-39-43| Thread 140224147474176 Received signal SIGSEGV 2023-02-14 04-39-43| Total memory in use by process: 5.88GB 2023-02-14 04-39-43| Thread Name [140224172652288]: Save Sectors WorkerPool Thread 0 2023-02-14 04-39-43| Workpacket [140224147474176]: BackupModule::createBackup() ... 2023-02-14 04-39-43| Factions In Memory: 0 2023-02-14 04-39-43| Sectors Loaded: 0 2023-02-14 04-39-43| Total memory in use (whole system): 45.84GB 2023-02-14 04-39-43| Runtime: 50 seconds 2023-02-14 04-39-43| 2023-02-14 04-39-43| === STACKTRACE ===================================== 2023-02-14 04-39-43| #1: ?? [0xd01781] 2023-02-14 04-39-43| #2: [ext] ?? [0x7f88b7624d60] 2023-02-14 04-39-43| #3: [ext] ?? [0x7f88b774d30e] 2023-02-14 04-39-43| #4: ?? [0xc646bc] 2023-02-14 04-39-43| #5: ?? [0xc6503c] 2023-02-14 04-39-43| #6: ?? [0x3e3d0f] 2023-02-14 04-39-43| #7: ?? [0xc655c0] 2023-02-14 04-39-43| #8: ?? [0xc6608c] 2023-02-14 04-39-43| #9: ?? [0xc655c0] 2023-02-14 04-39-43| #10: ?? [0xc5a5e8] 2023-02-14 04-39-43| #11: ?? [0x484f6] 2023-02-14 04-39-43| #12: ?? [0x460d5] 2023-02-14 04-39-43| #13: ?? [0x46c10] 2023-02-14 04-39-43| #14: ?? [0xd19456] 2023-02-14 04-39-43| #15: [ext] ?? [0x7f88b79f01de] 2023-02-14 04-39-43| #16: [ext] ?? [0x7f88b7d56ea7] 2023-02-14 04-39-43| #17: [ext] ?? [0x7f88b76e8a2f] 2023-02-14 04-39-43| =================================================== 2023-02-14 04-39-43| 2023-02-14 04-39-43| Sending crash report ... 2023-02-14 04-39-44| Crash report sent successfully! 2023-02-14 04-39-44| Thread 140224147474176 Received signal SIGSEGV With regards to the standard info requested to be included in the posts: Operating System: debian:bullseye seems to be the base image OS (cm2network/steamcmd) https://github.com/CM2Walki/steamcmd/blob/master/bullseye/Dockerfile CPU: AMD Ryzen 9 3900X GPU: None being used I have included 3 examples logs of the crashes (the server pretty much crash loops when it's enabled) Let me know if any other information can help on this. serverlog 2023-02-14 04-38-46.txt serverlog 2023-02-14 04-36-44.txt serverlog 2023-02-14 04-32-41.txt

<removed> EDIT: Sure, although you are in another timezone and I rarely see you online when I'm not working :) Feel free to reach out to me as well, even at work sometimes I check Discord for notifications. I'm BBF#3471

I haven't tried force enabling on the client, but if you setup a server and set "forceEnabling=true", the server does load on 25.2. No one is able to join though, because apparently that property is not replicated to the client side. Since some investigation might be done in the topic of force enabling on the client side due to the OP, would it also be possible to consider reflecting the forceEnabling property of the server on the client side? The idea of having a "max" dependency on Avorion for mods seems great on principle, but has been very frustrating. Even more frustrating that if you modify to remove the "max" property, it gets added automatically back during mod submission. For my mods I've just set the max to 10.0 :)

One way to do it is like this: if onClient() then local metaTable = getmetatable(UIRenderer) local old_renderEntityArrow = metaTable["renderEntityArrow"] metaTable["renderEntityArrow"] = function(self, entity, width, length, visibilityThreshold, ...) -- The original miningsystem uses these parameters: -- renderer:renderEntityArrow(tuple.asteroid, 30, 10, 250, tuple.material.color); if width == 30 and length == 10 and visibilityThreshold == 250 then old_renderEntityArrow(self, entity, 5, 5, visibilityThreshold, ...) old_renderEntityArrow(self, entity, 5, 10, visibilityThreshold, ...) else old_renderEntityArrow(self, entity, width, length, visibilityThreshold, ...) end end end Because I don't know if there will ever be other calls to renderEntityArrow from that lua VM, I only did the override for that very specific set of params, based on the original code. If the devs ever change that, this will break. You could instead override all calls, which right now would also work, except that if a new call is added it would also break. Pick your poison :)

I can't say what happens with new players on the current version since I'm stuck at running the previous version. Unfortunately the errors with the broken scripts aren't being fixed by the server because the server still crashes before it can save its state. It also doesn't seem that a minidump is being generated and uploaded with the crash.

Hi koonschi, In some of those logs I had temporarily removed some of the mods from modconfig.lua because they were the one that were causing the warnings in the logs. After I discovered that the references (absolute) to the file paths were stored directly on the entities, I left all them back on my modconfig.lua. I've updated to your latest build, 0.23.4.16335, and the server still crashes. On the previous build, I had noticed on the logs that following pattern: .... (snip) .... Avorion server Beta 0.23.4 r16262 79f7b3aecb15 running on Windows 10 starting up in "H:/Games/Avorion\GalaxyModSeason_alt" .... (snip) .... Trying to load player 76561197993124810 [bBFz0r] Warning: Found an absolute path while loading script information: "H:\Games\Avorion\GalaxyModSeason\workshop\content\445220\1722652757\data/scripts/player/azimuthlib-clientdata.lua" === STACKTRACE ===================================== #1: ?? [0xafb41c] #2: ?? [0xb732a9] .... (snip) .... Trying to load player 76561197993124810 [bBFz0r] .... (snip) .... Trying to load player 76561197993124810 [bBFz0r] .... (snip) .... FactionDataBase failed to load player 76561197993124810 [bBFz0r], error: 12ServerPlayer: Script 'H:\Games\Avorion\GalaxyModSeason\workshop\content\445220\1722652757\data/scripts/player/azimuthlib-clientdata.lua' not found. Tried the following paths: .... (snip) .... New Player registered: 76561197993124810 [bBFz0r] index: 81 Player's were getting a new profile when logging in because of their corrupted profile, which after a few retries (3?) the engine would just give up. That was happening for Players and Sectors , and they are being wiped whenever they could not be loaded. Please note that the galaxy for the server is actually under H:\Games\Avorion\GalaxyModSeason_alt, and the reference above is to the original location when the server was first started (which I had to rename to make sure it wasn't actually loading content from that path). I noticed on the new build / log a few entries of the following line: Script will be removed from this scriptable object to avoid further errors. I still could not find an entry that logged the successful load of the player profile, or a new one being created, before the crash though. I've attached an updated server log below. Let me know if there's anything else that could also help. Would a sample player.dat help ? :) serverlog_2019-07-22_20-14-32.zip

It doesn't seem like I will be able to fix this using a mod. I think the engine just throws an exception when it detects an absolute path attached to an entity. It would be great if instead it either: A) Removed the offending script from the entity automatically (Alliance, Entity, EntityDescriptor, Player, Sector) B) Ignored the script entry from the entity Option A would be best since it would fix the corrupted saves. Option B would at least allow me to write a script to clean up the save :) (although while I can check for colon on Windows saves and fix that, I would not be able to test a Linux/Mac path using lua file system restrictions, or assuming that we can never have /data/scripts twice) if onServer() then local p = Player() for sid, script in pairs(p:getScripts()) do printlog("Player() sid %d script %s", sid, script) if string.find(script, ':') then -- <driver>:<path> <-- Absolute on Windows printlog("Found invalid char in script, deleting due to absolute path: %s", script) p:removeScript(script) end end end Update: Apparently running on the previous version of the executable, when I list all the scripts in the player, they are all relative paths. So I don't think Option B is viable at all. The only way I can expect to recover the save if I want to upgrade is if some fix is implemented in the engine. :( Thanks

So I noticed the new manifests in the Steam depots, and decided to try out the new build in hopes the fixes were included. And apparently they were :) Unfortunately though, once I did that the server begun crashing. Initially I thought I would get away by just disabling the offending mods, which did not work. I noticed the logs had references even to the disabled mods. So I renamed the workshop download folders, and that made some difference, but not enough to fix it. Since lua scripts are attached to game objects/entities, and they contain absolute paths, the server is very unhappy with it's current state on this new build. Are there any asserts() for invalid state? Player profiles are not being loaded correctly, and blank ones are been attached to existing players. It's not a happy sight. I had to rollback the galaxy save to before updating, and then manually download the previous depots to keep players on the server playing while something can be done. At this point I'm considering if I should write a mod that removes scripts from the entities when it detects absolute paths on them, and even if that will be enough to prevent the latest crashes. I'm attaching the logs from the crashes in case anyone has a better idea on how to fix the state in the saves. Thanks Server_Logs.zip

Hi koonschi, Sorry for not noticing your previous requests for more info, I never received any email from the forum that I had replies on this thread. Although the bug has been fixed, to address your previous questions, I had created a brand new galaxy when providing the reproduction steps. On the client side I had also unsubscribed from all mods after I noticed the issue, and the issue did not manifest on all mods, only 4 of them. Anyways, thank you for fixing the bug. Is the fix coming out on the 0.23.x stable branch, or only eventually on 0.24.x?

Hello, I have some ideas for mods that I would like to implement, and to do them properly, I would like to create custom blocks that could be added to ships. I did a few searches on the script folders and could not find any definition of the core game blocks. At this point I'm assuming that they can currently only be defined in C++ on the engine. If that is really the case, are there any plans to change that? Or I shouldn't hold my breath, and instead implement my ideas in different ways? Thanks

By the way, I noticed that there is some log corruption happening on the server. It seems like calls to the logger are not thread safe. Example: Call to 'sectorOpener.initialize' in '"H:\GameFilesecurer destination content size: s\A649274vor ion\GalaxyModSeason\workshop\content\445220\1694550170\data/scripts/player/sectorOpener.lua"' successful. I think this is supposed to be: Filesecurer destination content size: 649274 Call to 'sectorOpener.initialize' in '"H:\Games\Avorion\GalaxyModSeason\workshop\content\445220\1694550170\data/scripts/player/sectorOpener.lua"' successful.

Hello all, I'm trying to run a dedicated server which is enabling both server and client side mods. For reference, here's a copy of my modconfig.lua: modLocation = "" forceEnabling = false -- Docs: https://avorion.gamepedia.com/Using_Mods_on_Dedicated_Servers mods = { {workshopid = "1731575231"}, -- Auto Research By: Rinart73 {workshopid = "1722652757"}, -- AzimuthLib - Library for modders By: Rinart73 {workshopid = "1722261398"}, -- Compass-like Gate Pixel Icons By: Rinart73 {workshopid = "1756645965"}, -- Crewboard Tweaks By: Rinart73 {workshopid = "1765619430"}, -- Galaxy Map QoL By: Rinart73 {workshopid = "1734791653"}, -- Gate Founder By: Rinart73 {workshopid = "1723188393"}, -- Improved Energy Suppressor By: Rinart73 {workshopid = "1728695059"}, -- NPC Respawn By: Rinart73 {workshopid = "1769379152"}, -- Resource Display By: Rinart73 {workshopid = "1722256729"}, -- Sector Overview By: Rinart73 {workshopid = "1776283892"}, -- Trading Overview Fixes By: Rinart73 {workshopid = "1754114031"}, -- Trading Tweaks By: Rinart73 {workshopid = "1722263986"}, -- Transfer Cargo Tweaks By: Rinart73 {workshopid = "1691591293"}, -- claim.lua claim() hook By: Laserzwei {workshopid = "1695671502"}, -- Laserzwei's Advanced Shipyard By: Laserzwei {workshopid = "1691539727"}, -- Laserzwei's Move Asteroids By: Laserzwei {workshopid = "1694550170"}, -- Laserzwei's Sector Manager By: Laserzwei {workshopid = "1692998037"}, -- Laserzwei's simple asteroid respawn mod By: Laserzwei {workshopid = "1745027189"}, -- Sell Asteroid - Extended By: Finite {workshopid = "1746985221"}, -- IFTL Library [Current v1.3] By: IFoundTheLight {workshopid = "1747690472"}, -- IFTL Loot Display [Current v1.3] By: IFoundTheLight {workshopid = "1746985158"}, -- IFTL Player Gate [Current v1.3] By: IFoundTheLight {workshopid = "1751636748"}, -- Detailed Turret Tooltips By: MassCraxx {workshopid = "1751540023"}, -- Fleet Jump through Gate Command By: MassCraxx {workshopid = "1751798934"}, -- System Scanner Upgrade By: MassCraxx } -- these are allowed mods, that can run on the client only (ie. UI mods) allowed = { } On the server, I don't see any issues, and the console doesn't produce any errors or warnings. When I try to connect a client to the server though, I see the following issues: Warning: Couldn't add script from stream: 12ClientPlayer: Script 'H:\Games\Avorion\GalaxyModSeason\workshop\content\445220\1722652757\data/scripts/player/azimuthlib-clientdata.lua' not found. Tried the following paths: data/scripts/player/H:\Games\Avorion\GalaxyModSeason\workshop\content\445220\1722652757\data/scripts/player/azimuthlib-clientdata.lua data/scripts/player/H:\Games\Avorion\GalaxyModSeason\workshop\content\445220\1722652757\data/scripts/player/azimuthlib-clientdata.lua.lua data/scripts/playerH:\Games\Avorion\GalaxyModSeason\workshop\content\445220\1722652757\data/scripts/player/azimuthlib-clientdata.lua data/scripts/playerH:\Games\Avorion\GalaxyModSeason\workshop\content\445220\1722652757\data/scripts/player/azimuthlib-clientdata.lua.lua data/scripts/quest/H:\Games\Avorion\GalaxyModSeason\workshop\content\445220\1722652757\data/scripts/player/azimuthlib-clientdata.lua data/scripts/quest/H:\Games\Avorion\GalaxyModSeason\workshop\content\445220\1722652757\data/scripts/player/azimuthlib-clientdata.lua.lua data/scripts/questH:\Games\Avorion\GalaxyModSeason\workshop\content\445220\1722652757\data/scripts/player/azimuthlib-clientdata.lua data/scripts/questH:\Games\Avorion\GalaxyModSeason\workshop\content\445220\1722652757\data/scripts/player/azimuthlib-clientdata.lua.lua H:\Games\Avorion\GalaxyModSeason\workshop\content\445220\1722652757\data/scripts/player/azimuthlib-clientdata.lua H:\Games\Avorion\GalaxyModSeason\workshop\content\445220\1722652757\data/scripts/player/azimuthlib-clientdata.lua.lua data/scripts/H:\Games\Avorion\GalaxyModSeason\workshop\content\445220\1722652757\data/scripts/player/azimuthlib-clientdata.lua data/scripts/H:\Games\Avorion\GalaxyModSeason\workshop\content\445220\1722652757\data/scripts/player/azimuthlib-clientdata.lua.lua data/scriptsH:\Games\Avorion\GalaxyModSeason\workshop\content\445220\1722652757\data/scripts/player/azimuthlib-clientdata.lua data/scriptsH:\Games\Avorion\GalaxyModSeason\workshop\content\445220\1722652757\data/scripts/player/azimuthlib-clientdata.lua.lua Modders: Make sure to enable Dev-Mode while writing scripts, or files you created while the game was already running might get ignored. Warning: Couldn't add script from stream: 12ClientPlayer: Script 'H:\Games\Avorion\GalaxyModSeason\workshop\content\445220\1694550170\data/scripts/player/sectorOpener.lua' not found. Tried the following paths: data/scripts/player/H:\Games\Avorion\GalaxyModSeason\workshop\content\445220\1694550170\data/scripts/player/sectorOpener.lua <...snip....> data/scriptsH:\Games\Avorion\GalaxyModSeason\workshop\content\445220\1694550170\data/scripts/player/sectorOpener.lua.lua Modders: Make sure to enable Dev-Mode while writing scripts, or files you created while the game was already running might get ignored. Warning: Couldn't add script from stream: 12ClientPlayer: Script 'H:\Games\Avorion\GalaxyModSeason\workshop\content\445220\1734791653\data/scripts/player/gatefounder.lua' not found. Tried the following paths: <...snip....> Modders: Make sure to enable Dev-Mode while writing scripts, or files you created while the game was already running might get ignored. Warning: Couldn't add script from stream: 12ClientPlayer: Script 'H:\Games\Avorion\GalaxyModSeason\workshop\content\445220\1765619430\data/scripts/player/map/galaxymapqol.lua' not found. Tried the following paths: <...snip....> Modders: Make sure to enable Dev-Mode while writing scripts, or files you created while the game was already running might get ignored. <Server> Player BBFz0r joined the galaxy Player BBFz0r joined the galaxy Warning: Couldn't add script from stream: 12ClientPlayer: Script 'H:\Games\Avorion\GalaxyModSeason\workshop\content\445220\1722652757\data/scripts/player/azimuthlib-clientdata.lua' not found. Tried the following paths: data/scripts/player/H:\Games\Avorion\GalaxyModSeason\workshop\content\445220\1722652757\data/scripts/player/azimuthlib-clientdata.lua <...snip....> The server is being spawned with the following command: bin\AvorionServer.exe --galaxy-name GalaxyModSeason --seed ModSeason234 --datapath H:/Games/Avorion --admin <...snip...> --collision-damage 0.25 --use-steam-networking 1 -t all So the save for the galaxy is located at: H:\Games\Avorion\GalaxyModSeason And the location that is being used for the workshop mods is: H:\Games\Avorion\GalaxyModSeason\workshop\content\445220 Somehow, that absolute server path is being sent to the client side, and being expanded by package.path. Am I'm missing something in my configuration, or is this a bug?