🔨 "New Architecture" Tracker Issue (2024 Refresh)

[zach2good]

I affirm:

• I understand that if I do not agree to the following points by completing the checkboxes my issue will be ignored.
• I have read and understood the Contributing Guide and the Code of Conduct.
• I have searched existing issues to see if the issue has already been opened, and I have checked the commit log to see if the issue has been resolved since my server was last updated.

Describe the feature

This whole writeup is TODO. I just felt motivated to write things down after a particularly good coffee!

LOL at me pushing towards a 2024 refresh at the end of September. As always: I speak only for myself, and the others might have differing opinions from me. Nothing is binding/promised. I'm a human person with a real life, a day job, and other hobbies. My (hobby/open-source time) priority is to this project. If that happens to help others and other projects out: great!

The Build, CI, & Tooling

General themes:

• We're still sitting on very incremental and cargo-culted versions of the build that I rewrote years ago.
• Flags and options need auditing, we need to look at how we build and link the exes to try and get the time down.
• We need to break out the build into smaller libs that we link together into final exes. The final exe main should just be a simple wrapper around Application
• CMake configuration and VSCode debug jobs are slow and wasteful. They can for sure be faster and rebuild less.
• We need CI to be even more stringent, including modules and tooling.
• We need to completely dogfood dbtool and the other parts of the setup guide in CI on all platforms.
• It would be cool to make HeadlessXI better.
• Compiler and library upgrades.
• I need to port Eden's unit test suite. Oh my god. I've been saying it for years.
• dbtool needs to evolve into xit, to handle more things and erase an entire family of setup issues.
• SAN runs in CI

Issues:

• 🐛 ENV64BIT is not set on 64-bit OSX build #6235
• 🔨 Store "most recent migration run" as a DB_VER of sorts #5907
• 🐛 "Full" CI checks only take into account Lua modules, not SQL or CPP #5347
• 🔨 CI should be using dbtool.py for db setup #4955
• 🐛 All external libs are linked to all exes #4902
• 🔨 dbtool should handle default collation settings of xidb #4900
• 🔨 [C++] Provide Windows Service plumbing so that nssm.exe doesn't need to be used #4755
• 🔨 Ubuntu 24.04 LTS (Noble Numbat) Upgrade (eta. April 25, 2024) #4670
• 🔨 Add CI check for valid license text in .h & .cpp files #4077
• 🔨 Negative testing #2360
• 🔨 dbtool improvements  #2240

Core

General themes:

• We've made huge strides in the last few years, with some systems getting improved - but not always fully adopted.
• SQLConnection needs to die and be replaced with db:: everywhere.
• We keep coming back to the same problem of stale entity pointers being held and causing problems. This can be solved easily with my EntityId proposal. Huge project though.
• World server concept fell on its face because it's hard to use. It needs to be easy to use.
• As above, IPC/RPC is hard to use, so nobody does it, and if they do it goes poorly. It needs to be a "pit of success".
• After very careful consideration, I think it's finally time to introduce 😱multithreading😱 - but hidden behind sufficient plumbing that it's hard for people to fuck up.
• BIG CONCEPT CHANGE: We move network and packet handling to its own thread that is always running and not tick-tocking against the entity tick logic. The incoming packets are received, validated, packed up into events that are then shipped off to the relevant zone executor to be processed alongside the zone/entity ticks.
• As above: my behind the scenes work designing how we would use coroutines for this in a human-readable way.
• LockToken primitive: an RAII resource locker.
• Queues and Mailboxes concepts for thread communication, as lock-free as possible.
• EDIT: God damn I'd like to get full startup time down. If we could start accepting connections as soon as everything that isn't a zone is ready, and then prioritise loading zones that have pending players waiting to zone in - we could get the development loop time down to <15 seconds.

Issues:

• 🔨 Exceptions should be exceptional #4024
• Core: Make pushPacket use unique_ptr<T> #6261
• 🔨 EntityId tracking structure #6167
• 🔨 Death to raw mutex and lock usage #5172
• 🔨 Death to raw operator new #5171
• 🔨 Death to ZMQ? (IPC & RPC refresh) #6165
• 🔨 Spatial partitioning of entities / octree implementation #5196
• 🔨 Splitting out networking workload #5173
• 🔨 Refactor onMobDeath and onMobDeathEx #6094 (optParams, etc.)
• 🔨 Add binding to force zones to stay away for arbitrary amount of time #5910
• 🔨 Remove mixins special handling in core, replace with pure Lua globals  #5374
• 🔨 New API (and performance improvements) for mixins #4493
• 🔨 New mariadb-connector-cpp shortcomings (tracker) #5361
• 🔨 Make id-space maths human readable/usable #5065
• 🔨 We need more performance metrics #4951
• 🔨 Add benchmark target in CMake to capture C++ vs Cached Lua vs Uncached Lua performance #4944
• 🐛 Regular map log doesn't indicate when there has been a crash #4797
• 🔨 [C++] Death to keyword extern #4745
• 🔨 THE WAR ON BAD CASTING 💀  #4641
• 🔨 Support "hot-delete" of scripting files from the cache #4494
• 🔨 ONGOING: Keep chipping away at clang-tidy rules #4169
• 🐛 Possible memory leak in xi_search #3561
• 🔨 Wrap ts::task_system to make it blocking #3550
• 🔨 Long distance pathing support #3522
• 🔨 Smarter "Module Stacking" and reload #3300
• 🔨 Improve logging in Search server #2072
• [chore] Proper backtrace support for Linux, consolidate logic into common/debug.h/.cpp #530
• [build] CMake compiler/platform/settings configuration is a mess #170
• db: Add ResultSetWrapper type #6300
• Core: Add generate_ipc_stubs workflow #6359

"Content"

General themes:

• For me personally, I tend to prefer writing systems and not content.
• I would like to introduce some modules that bypass broken or missing content so that we can help out the folks who "just want to experience the story"
• I think it's fair for us to start accepting "mostly good" work and slapping it behind experimental flags (like we have with BCNMs), so it's obvious to onlookers that it isn't complete, balanced, or possibly playable - but it allows us to make forward progress on certain content while having it not be confused as "this is great, nobody needs to look at this ever again". THIS DOES NOT APPLY TO ROV MISSIONS, OH MY GOD, WHY DID SE MAKE ROV SUCH A NIGHTMARE!
• Eden has many goodies we could port, but who has the time? Amazing that they have things, amazing that they're available to us, but porting other people's code is really not fun or rewarding.

Issues:

• Module idea: Print warnings before known problem CS's, in relation to FastCS etc. #646

Server Operation, Docs, Repo, & Meta

General themes:

• We remain a FFXI Retail Server Emulator project. Sometimes we individually become interested in things that server operators would like/benefit from, but that's on an individual level.
• LSB is largely a black box to the wider community. It will stay this way. Some of us spend a lot of time in other Discords, but that's our choice. We're still not going to have a public Discord because GitHub remain the best way for us to track work, host discussions, and have issues reported to us. It's also a necessary small point of friction that forces people to think and respect our time instead of pinging us in public Discords asking us why their build isn't working because they haven't added Python to their PATH.
• We all work independently on whatever we choose to work on.
• Wiki and docs in general could do with a refresh. Quick-start guide, contributor guide, CoC, "What Works", post-install guide, etc.
• By virtue of the work we're doing to consolidate different systems and make them human-readable/move them out of core: the number of customisation points for era/custom is going up all the time. This isn't an end goal, it's just a happy (for some) symptom of good and necessary work.
• (Especially for me, mini-rant:) We need to put the "build it and they will come" mentality to rest. They never do. Every system we've written or expanded to help downstream further their own goals in the hopes that it'll result in more reports/fixes/content/development effort coming back to us has failed to do so. The amount that comes back is laughable in comparison to the effort it took to set up. This is especially egregious for bug fixes not coming back to us (even as reports) unless downstream can't fix them on their own. We should be focusing on changes that make OUR development easier, more pleasant, and more productive.

Issues:

• 🔨 Login: Block repeated incorrect login attempts & rate limit #5898
• 🔨 wiki: Write article about what is dictated by the server vs the client #4550
• 🔨 wiki: Specify OpenSSL and G++ versions required #4503
• 🔨 Update contributors guide and rules  #2314

Bonus: The Game Client

General themes:

• I will never advocate for the server needing a copy of the relevant client at runtime! Nor a user needing a copy of the client to configure/compile the server. But optionally integrating the relevant client to the development process doesn't give me the heebie jeebies.
• I can't find where I saw it now, but the WoW community has libraries that some emulator projects use that completely expose the content of the game client to the server. They use these (to my observation) to pull out navmesh data etc.
• Wouldn't it be wonderful if we had something similar and didn't rely on other projects and groups for black-boxed and largely unloved tools to get what we need from the client?
• I'd love to have libffxi, libvanadiel, libgalka, libtaru or something, but who has the time?! If someone was interested in working on it, in C or C++, I think the LSB org would happily host and support it. Unfortunately, all of my dev time is taken up by the server, so I'd only be able to review and advise, and help with CI and integrations, etc.

[cocosolos]

I think it's fair for us to start accepting "mostly good" work and slapping it behind experimental flags (like we have with BCNMs), so it's obvious to onlookers that it isn't complete, balanced, or possibly playable - but it allows us to make forward progress on certain content while having it not be confused as "this is great, nobody needs to look at this ever again". THIS DOES NOT APPLY TO ROV MISSIONS, OH MY GOD, WHY DID SE MAKE ROV SUCH A NIGHTMARE!

How would you feel about setting the RESTRICT_CONTENT and ENABLE_x settings to disable later expansions and their associated content by default, so you have to go out of your way to enable likely incomplete content? Could turn RESTRICT_CONTENT off again in CI to make sure it's all loaded for checks.

[zach2good]

I don't really want to mix the things we're (already) marking as experimental with the kinda cludgey RESTRICT_CONTENT and ENABLE_x settings. The gimmick with experimental content is that it outs itself as being incomplete by jacking up the levels (and maybe stats) of the mobs in the fight - making them unwinnable. I'd rather there be friction for re-enabling or unbuffing experimental content, so it remains obvious and annoying to get it back to being faceroll mode and then people don't address or complete it.