YAML library support developer discussion

[toomanybrians]

I'm starting a thread to capture and share thoughts around MPF's YAML support.

MPF currently uses ruamel.yaml for YAML support. This third party library has several advantages over the default PyYaml library. The main reason we chose it for MPF was that had a round trip YAML reader / writer that could preserve the white space and comments in YAML files, which was critical in the early days when MPF would change the config file formats and needed a migrator utility that would update those files for users.

Nowadays I don't think we need that, for several reasons:

• The MPF config file format is more stable. Any changes can be handled with find and replace or ChatGPT.
• The migrator is not maintained.
• Users tend to stick with the version of MPF they started with, rather than constantly upgrading. (And as MPF transitions from "actively developed" to "barely maintained," new versions will not be major or frequent.

Also, MPF's usage of YAML diverges from the published YAML spec in a few ways. This required making some changes to the YAML parser (by subclassing and customizing some of the ruamel.yaml components) for our own needs. These changes were mostly done for "ease of use" reasons, though in hindsight they probably just created more work for developers.

Examples of changes that MPF YAML parsing adds:

• Process any item beginning with a plus sign (+) as a string
• Process any 3+ digit number with a leading zero as a string
• Process yes/no values as strings, not bool

MPF's config interface enforces its own typing, so if the YAML parser brings in 123 as an int and MPF wants it to be a string, MPF will change it. That's fine. The problem is sometimes things are lost at the YAML level. e.g. the YAML library might see an RGB color entry 050505 as an int of 50505, which even if MPF changes it back to a string, the leading zero is lost. Same with +20 that might be imported as the int of 20, losing the leading plus which has a specific meaning for MPF.

That said, the easy fix for all these is just to put quotes around the values in the config file. So it's quite possible we could just make the change to go back to all the defaults and then in the few cases where it breaks, the user will sort of figure it out (since it broke) and add quotes.

I'll update this message later with the complete list of MPF YAML changes. I need to look at the files a bit more.

MPF currently uses ruamel.yaml 0.15.x while the latest version is 0.17.x. Some changes were made to ruamel.yaml in between those versions, which need to be adopted by MPF. This is part of the reason MPF doesn't support anything newer than Python 3.9 since that's the last support for ruamel.yaml 0.15. I started the work here, I think the yaml portions are done:

https://github.com/fastpinball/mpf/tree/python311-support

If we decide to not support the YAML customizations, I think we can remove the various subclassing and then use ruamel.yaml without customization which make it easier for to MPF support future versions of Python and/or ruamel.yaml.

If we remove the config file migrator (which, since it's not supported and I don't think it's used, we probably should), then that's a big reason we went to ruamel.yaml and maybe a reason to stop using that library altogether. That said, This probably doesn't matter though, because Python doesn't have a built-in YAML parser. The de facto default is PyYaml, but that's still a third party library that needs to be installed via pip, and it doesn't seem to be updated any more or less often than ruamel.yaml, so there's probably no reason to switch.

I'll investigate some of these things over the next few weeks. Most likely it will be removing our subclassing to see what happens. Our test coverage is pretty good, so I think we should be able to find all the things it breaks fairly easily and then make a decision as to whether we want to make that change official or not.

[atummons]

Brian, I will certainly defer to your knowledge of these things, but that seems fairly straight forward. I don't know if your comment about people staying on the version is true, but certainly not out of the realm of possibilities. I would guess tinkerers/people that have the functionality they need will never upgrade. Those who upgrade know a bit more and what they are in for, so would be able to go through and find the issues and get them fixed. Might take a while, or trial and error for less used modes for your string examples.

All that said, my only concern is that we can't outpace the platforms that interface. I know we already have P-ROC at this point, which gets brought up at least once a week. Maybe that's the only volatile one, but would hate to move faster and start to alienate users on a historical version for which they can't upgrade. This is compounded based on the docs pages not housing historical as easy as they used to. I would like to get the person working on packages and Quinn together to go over the pypinproc stuff to see if we can get the files updated to support newer than 3.8. If we can get steps written for how to do it, I'm sure people of my technical abilities can support it in the future.

[toomanybrians]

I hear you on the code outpacing the platforms. However, at some point we need the individual platform people (whether from the company, or people from their community) to step up and keep their platforms up to date if they want the latest versions of MPF to work with their hardware. If they don't want to, that's fine, since their users can continue to use the existing versions of MPF.

That said, MPF python support is really driven by Kivy. Kivy 2.1.0 (the latest) supports Python 3.7 - 3.10, so that's ideally what MPF would support. (I say "ideally" since we're not to 3.10 yet.) But we definitely can't support older versions.

For the MPF core and MPF Monitor, I will always try for newer versions too. (I have them both running on 3.11 locally.) Lots of people don't use the MC, so I like the idea of supporting the newest current Python release, but anything older than Kivy support has no chance for MPF.

[atummons]

Correct, and I totally agree that we can't wait. Just need to figure out how to get Gerry and Hugh up to speed, I guess. With you from Fast and someone on their ends (or users assisting) we have it covered and can keep it going forward.

[toomanybrians]

Yeah makes sense. Hugh and Thomas have been active in keeping the OPP/CobraPin platform up to date, and they've also been active supporting users and contributing to MPF. So that's awesome. Hopefully we can get someone representing Multimorphic to do the same, and if not, we can at least update the docs to be more clear about what is supported where.

In fact (this should be a separate conversation), I was thinking yesterday that as we march towards a "1.0" release, we can mark the various features as having different levels of testing. (I don't want to call it "support" since this is open source and free and really nothing is supported unless a vendor is involved.)

For example, when I was making the changes to get MPF running on Python 3.11, all the tests for the score reels were failing. (This would be for driving old school switch and coil based score reels, not the modern stepper versions.) I think the score reels would've worked fine in real life, but I don't have any so I can't test. I spent an hour or two messing with the test code, but wasn't really getting anywhere before finally realizing, "Why am I stressing over this super old very obscure code that probably no one even uses? This should not be a blocker in getting to Python 3.10 and 3.11."

So I don't think we need to abandon a device type like the score reels, rather we just update a status which says, "This code is still there, it has not been tested since 0.56, but maybe it still works? If you want to volunteer to maintain and test it, we would love that."

I figure we can create a status chart with green/yellow/red for all the platforms, device types, features, etc.

[atummons]

Love it! We can discuss in a different topic further down the road as we get there to keep it isolated.

[toomanybrians]

I spent some time looking at the MPF custom YAML processing.

First, I should update my previous statements about how this works. What I described was how it used to work, where you had to create a class instance of the YAML resolver class and then override the regex for how it figures out what it's dealing with. In the more modern ruamel.yaml library, there's a specific function to add your own custom resolvers. So we only need to add the 3-4 that we need, rather than overriding everything. This means what we're doing is not really a "hack" and would probably be fine moving forward.

That said, I'm still playing with the idea of just removing all customizations altogether. The good news is that some of these can be removed at the YAML level, but MPF itself can still be updated to work properly. For example, "yes" and "no" are strings in YAML world, not bools. But MPF allows them to be used as bools. It might not be the end of the world for us to just say "if you have yes/no in a config file, change it to true/false." However, MPF has the config file validator which is used to parse the values of config files (versus the YAML engine which just checks type). So the MPF config file validator, when it's looking for a bool, could just do the regex there to say, "if this value is a string, but it's true, false, yes, no, etc." then process it as a bool. That would be simple and let us use ruamel.yaml out of the box with no customizations while also not requiring user changes.

This technique won't work for everything, though. For example, the YAML spec allows for exponents (processed as floats), so a color value in MPF like 123e45 would be processed as 123 to the 45th power rather than a string. But that scenario is pretty narrow (e.g. only if you have a color with some digits, the single letter e, and some more digits). And in that case, if the YAML processor resolved that as a float, MPF would give an error like, "line 45 char 4 float value 1.111e45 is not a valid color" or something like that, and the solution would be for the user to just put it in quotes. I feel like that's something we can address in the docs and or error messages.

So I'll still play with this some more. I have a 12 hour plane ride on Thurs so lots of focus time. But it's looking like we ought to be able to go to the non-customized out of the box yaml processor fairly easily. (Again this is assuming we completely remove the YAML round trip processing which we don't need anymore.

[toomanybrians]

As an experiment, I have completely removed all customizations / hacks to ruamel.yaml so we are now using pure YAML. Then I updated all the tests and some real config files to see what the actual changes would be. It seems like they're not too bad, really just a few types of values that will need to be wrapped in quotes. For example:

• Incremental values with the leading plus sign, like time: +2 will need to be time: "+2". This is since YAML will read this as a regular int. However this is an easy find and replace.
• color values that all numbers with only a letter "e" in the middle like 12e345 will be read as exponents unless they're in quotes. This should be super rare and somewhat obvious to fix. (I added an explanatory error message to help.)
• Color values which are only numeric and have leading zeros, like color: 006400 would be read as ints. I attempted to add functionality to the color validation that would left-pad the color with zeros if a color comes in as an int, but it turns out that's not so easy to do as there are a lot of edge cases and processing paths so it doesn't always work. So those would also need quotes, like color: "006400".

I was able to fix the boolean (minor in the grand scheme), but a nice win:

• Boolen "yes" / "no" values will be read as strings by YAML, but the config validator knows they should be bools so it will regex match them and turn them into proper bools.

So all-in-all, I think moving to stock YAML is not too big of a deal. This will keep things easier to maintain, since overriding certain YAML types required us to override everything, which was brittle. So now we can use the latest version of ruamel.yaml, which also opens up getting MPF to run on Python 3.10 and 3.11.

Since I started that work already, I went back to that today, and now I have the tests are passing on GitHub for all three platforms from Python 3.7 - 3.11. (A bunch of FAST tests fail since I'm rewriting that platform and it's still a work in progress, so no green checks yet, but everything else is working.)