MPF Website + Docs site overhaul

[toomanybrians]

A lot of my focus with MPF now is to simplify things. Clean up loose ends, get out of some of the tech debt, and get everything cleaned up and easier to maintain.

Here are my thoughts on what I want to change for the main missionpinball.org website, and docs sites.

How they work today

The main website is in the missionpinball-website repo. It's mostly hand-written HTML (!) with a bootstrap theme. There's a static blog engine, but the last post was almost 2 years ago. There are a bunch of static pages which are also never really updated. Basically it's a simple static site based on 10 year old CSS and web themes. The site is hosted as a GitHub static site.

The docs site docs.missionpinball.org is based on Sphinx and is hosted on ReadTheDocs. The content comes from the mpf-docs repo. I recently collapsed the various versions since they were pain to maintain, they confused users, and there's not a lot of activity happening with MPF dev, so things are more stable. There are a lot of custom scripts that run which scan the MPF source code and extract the doc content for the config file reference, event reference, variable reference, etc. Also there's code which runs and verifies all the config file snippets, and some other things. All those must be run locally and the content they create checked in to the repo. The doc format is RST, which is also old now and not common (compared to MD) and generally a pain to work with. The whole thing is brittle and looks and feels old and not fun to work on.

The developer docs site also runs on RTD and uses Sphinx. It runs against the core MPF and MPF-MC repos to extract docstrings to create the API reference, as well as having some static pages to explain how MPF does some things. It doesn't seem to have been updated since 0.54 which is 3 years old. I don't think anyone uses it.

New plan

My plan is to drastically simplify, streamline, and modernize the MPF documentation. Here are my thoughts & plans:

• Migrate everything to mkdocs with the material for mkdocs theme.
• Combine all three MPF websites into a single site, which will be at missionpinball.org. This means it will be one repo, one hoster, one build process, etc.
• Use GitHub for hosting, and set up proper GitHub actions & workflows for proper cloud-based CICD builds.
• Simplify and decide what to keep/remove for all the automations and script stuff. And then move those to the cloud workflows. (e.g. the config file reference that auto builds from the config_spec is great.) But we might not need all of them, especially ones for things that don't change that often.
• Migrate all the RST formatted docs to MD.

The focus will be to keep everything as turnkey and "out of the box" as possible. That will make it as easy as possible to keep current with the newest tools and platforms, and make it easy for new people to contribute. It will modernize and freshen everything. Obviously its still all open source, so anyone can contribute, and I can say after using mkdocs & the material theme package for the past year (this is what I use for the FAST Pinball website, working in this new environment is about 100X more pleasant than using Sphinx, RSTs, and all our old scripts. Material for Mkdocs has a fantastic site search feature, supports dark and light modes, works on mobile, and it's still a static site generator.

Part of the new site will be to have tables which show which platforms, devices, and other features are current and supported, and which ones need help, and which ones are stale. (e.g. for FAST Pinball or CobraPin, those can have a "green" status since they work with the latest versions of MPF and Python and they are actively supported by their vendors.) There are some other platforms which haven't been touched in years, so we can say things like "last updated for MPF 0.54, might work after, untested." etc. Same for devices, e.g. score reels which might still work but haven't been tested."

So that's the general plan, I don't have a timeframe but wanted to get my thoughts out there.

[atummons]

I like it. The easier it is to maintain, the better. The easier it is to contribute the better. And most importantly it has to be easy for people to pick up and use out of the box. It seems like this accomplishes all of that.

As long as it's a net neutral or increase in available details. There are some nuanced parts, but have that one detail there. Also things like how to call placeholders for MC that are nearly impossible to find. I'm nice we get a good structure, we can build out the whole process better.

[ralfit0]

Brian,

make it easy for new people to contribute

I think that would be key, and motivation alone to make that shift. 10 minutes ago I was pondering over how to improve existing docs, but it is a hurdle, especially for people like me, computer literate, but not expert enough to enjoy a Github pull request for a simple text improvement.
my 2 cents
Ralf

[worldpeace-germany]

Couldn't agree more to what everyone of you said. Looking forward to it some day in the future. If I can contribute to that effort somehow, then I am happy to help.

[toomanybrians]

Progress Update:

Over the weekend I started doing this work. Here’s the progress:

• Create a brand new mkdocs site with the Material for MkDocs theme. (5 min)
• Research how to convert RST to MD (Pandoc), play around with the settings, have ChatGPT write a script which bulk migrates the fils (2 hours)
• Have ChatGPT write scripts that are essentially “very fancy find and replace with lots of Regex” to fix / clean up all the weird things that didn’t migrate properly. (3 days) 🤣

But I have everything running and looking good. I have a couple more hours of things to clean up, and then I’ll roll out the new site.

I’m going to drop this new site as a single PR into the mpf-docs repo (since that’s the one with the full history of contributors, etc.) The order of operations will be:

1. Disable ReadTheDocs hooks so the current docs.missionpinball.org site doesn’t get messed up during this process.
2. Get the new mpf-docs site building and publishing via GitHub actions
3. Once the new site is running, cut over the DNS so docs.missionpinball.org points to the new site on GitHub rather than the current RTD site.
4. Change the URL of the new site from docs.missionpinball.org to missionpinball.org (the new site will be THE site, combining the stub missionpinball.org site, as well as docs., developer., hardware., etc.
5. Set up all the forwarders so docs.missionpinball.org points to the proper URLs. I’ll also update so the old links which are still in google with versions embedded (e.g. /0.55/tutorial/…) get forwarded properly.

I intend on doing all of this today. We’ll see how far I get, assuming nothing weird happens with all the DNS and pointing repos to the right places and all that …. fingers crossed! 🤞

Here are a few photos of the work in progress:

[ralfit0]

Brian,
this is so awesome, thanks a lot!
Let us know about how support.
Ralf

[worldpeace-germany]

Hi,

could it be that this change broke some of the documentation? I am missing some example config files here https://missionpinball.org/mechs/lights/#fully-working-example-2-light_stripes Some other pages gave me an odd feeling as well, but would have to investigate more if on these pages something is really wrong.

[toomanybrians]

Yeah I think you're right.

I think what's happening here is the old documentation would pull a bunch of examples in based on the config files for MPF tests. (The internal tests MPF uses to test itself.) The idea there is those configs are always up to date by definition since they're current tests, versus when we were putting config files in docs directly and they'd get old and never updated and people were confused.

But all the scripts that pulled all the things into the docs were hard to maintain and confusing people too (since now you had to update some things in mpf source code or test source in order to change the docs). So my intention was just to link to the test config files from the docs and let the user view the test configs directly.

I made a page explaining that here: https://missionpinball.org/examples/tests/

But as you mention, there may be several instances where we should link directly to the test config files from other spots in the docs. For example, that light_stripes reference example on the page you mentioned should like to here: https://github.com/missionpinball/mpf/blob/9f90c8b1586363b65340017bfa3af5d56d32c6d9/mpf/tests/machine_files/light/config/light_groups.yaml

So maybe this is just some manual changes here and there? Or if there are a lot of them, a quick cleanup script could bulk add them.

[worldpeace-germany]

Hi,

I don't mind to double check the documentation I have added in the last months and do the fixes manually. So you are saying all config file examples should end in the test folder? I can do that for my edits, question is if documentation is still readable if you have to open the example additionally and don't have it on the same page as the explanations. But I guess that is probably as well personal preference.

My biggest fear is that we miss too many occurrences where something went wrong and people stay away because they get confused from the documentation.

There is more wrong than only the mpf code examples. Take the same chapter I pointed out in my previous post, you have quite at the top of that page five bullet points saying "/tutorial/17_add_lights_leds" and each text points to a different link. I would claim that wasn't there before (I would have fixed that when I made my additions earlier the year in that chapter).

Can you figure out where something went wrong that at least we have a list what we need to fix? Happy to help where I can.

[toomanybrians]

Yes, I think all config file examples should come from the actual repos themselves, which is the easiest way to guarantee we always point readers to valid configs.

The exact URL / folder will vary based on what repo it is. e.g.:

• https://github.com/missionpinball/mpf/tree/dev/mpf/tests/machine_files
• https://github.com/missionpinball/mpf-mc/tree/dev/mpfmc/tests/machine_files

Also I want to update the template stuff for every page to also link to the config file spec, which is what MPF / MPF-MC uses to valid the configs. If you want the absolutely 100% real config details, they are here:

• https://github.com/missionpinball/mpf/blob/dev/mpf/config_spec.yaml

Here's an example of what's in the config spec:

achievements:
    __valid_in__: mode
    __type__: device
    show_tokens: dict|str:str|None
    restart_after_stop_possible: single|bool|true
    restart_on_next_ball_when_started: single|bool|false
    enable_on_next_ball_when_enabled: single|bool|true
    start_enabled: single|bool|None

I think it would be cool to add details about how to read this file into the docs. So if someone is ever confused about what sections are valid, or what type of entries are valid for a specific setting, this is where they could look.

As for your fears about something going wrong in the conversion, that is certainly possible. I used a tool (pandoc) to do the main RST to MD conversion of the files themselves. That took about 30 seconds for the whole site. Then I spent the next 5 days going through and finding the things that got broken or mangled, and in most cases I then manually wrote a script (using ChatGPT's help) to crawl all the files looking for that specific type of mangled conversion and fixing it. So there are probably more like that which I just missed, but it would be easy to write another script to bulk-update them.

So that's what we need to do now, is to look at the types of things that are broken and collect a few examples, and if they are one-offs then we can just fix them, and if it seems like there are multiple examples of things being broken in the same way, then I'll write a script to fix them.

I just created a new issue to track this: missionpinball/mpf-docs#482

If you find any things like this that are messed up, post it there so we have everything in one place.