🐛 Server crash upon creating new char (ON DUPLICATE KEY UPDATE query latency / watchdog crash)

[Mortalelite]

I affirm:

• I understand that if I do not agree to the following points by completing the checkboxes my issue will be ignored.
• I have read and understood the Contributing Guide and the Code of Conduct.
• I have searched existing issues to see if the issue has already been opened, and I have checked the commit log to see if the issue has been resolved since my server was last updated.

OS / platform the server is running (if known)

Windows 10

Branch affected by issue

base

Steps to reproduce

Create a new char, upon logging in to get CS, Map Server will crash. Help someone setup a new server and they got the same results. It only crashed when logging into a new char. Once char created able to log in as normal and get the CS. Also happens even with CS turned off.

#5342

Expected behavior

Not crash upon first time logging in

[WinterSolstice8]

Char creation cutscene was enabled, had a few socket debugs left over from previous development but I didn't crash
What nation/race/face was chosen to crash? do you have a crash log? traceback?

Not entirely impossible linux is immune but that seems unlikely...

[Mortalelite]

Was on Male Hume, Sandy Face 4, Hair A

xi_map.exe_14-10_18-50-26.log

I tested again with Female Elv, Bastok, Face 4 Hair A and crashed again
xi_map.exe_14-10_22-14-23.log

Got this right before the crash
[10/14/24 22:14:19:993][map][debug] Player logging in to zone <235> (charutils::LoadChar:718)
[10/14/24 22:14:23:381][map][critical] Process main tick has taken 2000ms or more. (main::<lambda_1>::operator ():256)
[10/14/24 22:14:23:381][map][critical] Watchdog thread time exceeded. Killing process. (main::<lambda_1>::operator ():266)
SQL query took 1861ms: INSERT INTO char_equip SET charid = 4, equipslotid = 7 , slotid = 2, containerid = 0 ON DUPLICATE KEY UPDATE slotid = 2, containerid = 0

[WinterSolstice8]

It looks like the watchdog is tripping. We don't know why yet. You can disable it for now (please don't do this permanently)

in settings/main.lua you can set DISABLE_INACTIVITY_WATCHDOG to true

If things continue working as normal and there's no other issues I'm not sure what's going on. I'll give it a quick look on my laptop though.

[WinterSolstice8]

SQL query took 1861ms: INSERT INTO char_equip SET charid = 4, equipslotid = 7 , slotid = 2, containerid = 0 ON DUPLICATE KEY UPDATE slotid = 2, containerid = 0

huh? how?

[Mortalelite]

SQL query took 1861ms: INSERT INTO char_equip SET charid = 4, equipslotid = 7 , slotid = 2, containerid = 0 ON DUPLICATE KEY UPDATE slotid = 2, containerid = 0

huh? how?

honestly not sure, just seen that in the map log when looking back to see if showed anything.
But honestly char can log in as normal after the crash.

[WinterSolstice8]

Do you use a spinning disk HDD by any chance? I suppose if a head parked and it needed to unpark to write to the db this could happen... but it wasn't THAT long ago that the db was just written to by the login server

[Mortalelite]

Yea this system is on spinning drive, not had chance moving over to SSD

[Mortalelite]

No crash with Watchdog disable. did get message about the Process main tick has taken 2000ms or more. So is it possible that drive could be going bad and not able to keep up and when trying to load causing to crash?

[WinterSolstice8]

I am not sure the drive is necessarily going bad. Hard drives these days have aggressive power management features and can/will spin down the disk and park heads. There may be software from your HDD manufacturer to disable these features.

It takes time to spin the disk back up, and perhaps that's what is happening here

[WinterSolstice8]

Changing this setting might help, but the HDD might ignore it. Worth trying

[Mortalelite]

Oh I turn that stuff off for the server to give all available power it needs. I will test another drive and see if get same results.

[zach2good]

Linking to this issue for my own reference: #4577

[Mortalelite]

Do you use a spinning disk HDD by any chance? I suppose if a head parked and it needed to unpark to write to the db this could happen... but it wasn't THAT long ago that the db was just written to by the login server

Just wanted to let you know putting on SSD made no difference with the crash. Same results.

[WinterSolstice8]

That's unfortunate... I do wonder what's going on with that, fortunately you can disable the watchdog (or just increase it's timeout) and it's not a show stopper. Still very weird that it seems to happen on character creation

[zach2good]

How big is chars and char_equip?

[Mortalelite]

This was tested on fresh install of windows to the server setup, nothing added in DB besides default stuff.
char_equip 0 spots and char 0 spot cause it was all fresh

But on the first crash was just one main char and like level 12 equipment, But when making that char would been second one.
1st crash char_equip 17 spots and char 1 spot this was only char on the server