Using PyInstaller for MPF packaging

[ericselk2018]

In another forum we have been talking a lot about issues that people are having trying to install MPF on Windows (most Mac users aren't having as many issues). Since I have a good amount of experience with installed applications on Windows, I've been trying to help. My knowledge of Python, MPF, and Python Packages is almost 0 though -- I am an NPM/TypeScript dev today, and before that C++/EXE/DLL native Windows apps.

I've put together detailed notes of my "clean install on Windows 10" in case it might help or start some discussion. It should be publicly available here:
https://docs.google.com/document/d/1Z8QHu7pB-2hzHaD4zLK4NsKw9J4VwKIWOpJmpkL_q2U/edit?usp=sharing

TLDR; It seems for me, I needed to check the "install for all users" option in the Python installer to have success installing MPF. Without that, the pipx MPF install step was unable to locate pip. Since it creates the venv using the --without-pip flag, maybe that is why it can't find pip unless it is installed globally "for all users". If this does seem like a valid requirement, I can try updating the Windows installer docs and submit a PR for that (never done it, but I understand the basic concepts). If anyone sees anything else useful in my notes, I'd be happy to discuss or recreate my steps to get more info from them.

At least until the end of May, and maybe June... I can't work on pinball, so happy to help in other ways I can. After that I may go dark for a while, heads up.

[toomanybrians]

This is awesome, thanks!

Looks like there are definitely some nuggets in here for the docs.

This is making me wonder about something. Dunno if this is the right place for it, but I'm wondering if we should not tell people to use pipx. I get that pipx is probably technically better, but the whole concept of a venv is super confusing to people, and I suspect that 99% of MPF users don't have other python things, so just having one native python environment might greatly simplify things?

And if people do have another python app or environment, we can guide them to pipx, but also they probably know more about what they're doing?

[ericselk2018]

Hard to say. I could try going through the steps using pip instead, but it would probably work just like pipx does now that I know the right options to use when installing Python.

With an app based on Python and Python packages -- is it feasible to create/maintain a more bundled "single click" type installer? I think apps that bundle all the binaries they need in one, and run and load everything from a single folder, without any environment variables, registry settings, or other "system dependencies", tend to have less install issues. For about 20 years I worked for a company that distributed apps to 1000s of users, and we found way less problems when we started installing everything like that -- single folder, not installing anything to "system" folders, no path variables needed to locate things because it was all "in this folder". This was all C/C++ stuff though, so not sure if Python or their package system supports that kind of environment. For the first 10 years that wasn't an option because of the huge download size of the bundle, but these days I don't think that is really a problem.

[toomanybrians]

Oh yeah, I think this could definitely be done. MPF is simple, and MPF-MC is essentially a Kivy app, and Kivy has lots of instructions for creating packages like this:

https://kivy.org/doc/stable/guide/packaging-windows.html
https://kivy.org/doc/stable/guide/packaging-osx.html

(Also Linux here, though Linux is pretty straightforward and most people using Linux probably can figure out how to install everything.)

The main issue is just finding people who want to take on the ownership of doing this. But definitely doable.

BTW if you do pull at this thread, the MPF-MC has some components written in Cython that must be compiled as part of the package creation process.

Also BTW #2, MPF Monitor uses PyQt6 for graphics, not Kivy (since it needs a more standard forms-based desktop app, not raw graphics like the MC). Also not impossible, and prob would be easy to build in (I think MPF Monitor is simple to install also and usually "just works" for people.)

[ericselk2018]

I'll see what I can do with PyInstaller. Kivy recommends that, and it seems to be the most used, compared to say PyVan which I think had 8 total downloads.

[ericselk2018]

So far it appears PyInstaller only works smoothly for projects that are really designed a certain way, maybe that started using PyInstaller early on or only using packages designed for it. I'll probably keep working on this, but a couple examples of why it will not be trivial:

ruamel doesn't work well: pyinstaller/pyinstaller#5338 -- PyInstaller group isn't going to change their side, and so far looks like ruamel hasn't been updated (or MPF is not using the latest version?). I did the manual hack stated here, editing the ruamel file in my local packages folder, which did at least work.

PyInstaller doesn't work well with dynamic imports, or anything using importlib. Even after doing the easy/documented things to get the "hidden packages" listed so PyInstaller will bundle them, there are still issues getting them to load correctly when included in the PYZ -- packages/modules being inside the EXE instead of loading from the file system as normal.

On the plus side -- I did get mpf --version to "work". But, only when building in a mode where all of the packages/modules are included in the package as subfolders and inside the EXE, which isn't really a good usable solution or option.

[toomanybrians]

This is great information. I think we can make some changes to MPF to help support this too.

For the ruamel.yaml support, I started the work a few months ago to update MPF to work with Python 3.11. (This should also work with 3.12 now.) It's mostly updating our customizations to ruamel to work with the latest version, as well as a few simpler asyncio changes. MPF's customizations only work on ruamel.yaml 0.15.x, and the latest is 0.17.x. I made those changes in this branch:

https://github.com/fastpinball/mpf/tree/python311-support

All the tests pass locally, but they all fail on the github action in the cloud. Dunno why (I didn't really investigate), but I assume this is close.

Second, for the dynamic imports, I feel like this is something we can remove also. I think they're only used for the run commands, and also for dynamic loading / importing of the various things like platforms, config players, devices, etc which maintain their lists in YAML files.

I think 10 years ago, I thought that MPF would have this big dynamic ecosystem, with 3rd parties writing plugins, etc. And so it would be important to have an external way to add modules and stuff. But that's not the case, so changing all of these imports to standard imports should be fine.

So, dunno, I would be willing to remove the dynamic imports too in order to give this a shot? All my work is in MPF which maybe could be the starting point, and then if that works I could do the same for MPF-MC? (The MC has other issues at the moment, like that the sound doesn't work if it's compiled via the Github workflow but if I build it locally it works? 🤷‍♂️🤦‍♂️ No idea why, but that's why those windows tests are disabled in the Github workflow in the MC.)

[ericselk2018]

All good info, thank you. For now I will try to get what I can working the fast way. For example, I changed this line in main:
mpf.commands.run_from_command_line(args)
to:
mpf.commands.game.Command('','c:/pinball/mc_demo',None)

To just mostly bypass the commands area that was causing problems.

I then got the game to try to run but got errors about config_spec.yaml and mpfcofig.yaml. I used the --add-files option of PyInstaller to add those files to the EXE bundle, and to my joy MPF was able to load those -- PyInstaller must do a good job of hooking the os file related calls to get them to load from the bundle, nifty.

That got me to a new failure in utility_functions. Trying to load mpf.devices.daul_wound_coil using import. Obviously, more of the dynamic import stuff we discussed.

Until I have more confidence that this route will work/help, I'll just keep plugging away at testing, not really rewriting any code in a way that we might keep. Good to know that it will at least be a valid option to get rid of dynamic imports if that ends up being a blocker to a better deployment solution. I think I can get by most of these by just keep adding modules to the "hidden module" import list as they fail to load, or if that gets too painful, figure out how to add entire folders like mpf/devices.

[ericselk2018]

Got a lot further, MPF loads the system attract and game YAML files now, and a bunch of devices.

inside text_ui.py it imports psutil -- this seems to be different than the other import related issues I fixed. Something special about psutil, I'll do some searching tomorrow to see why the same fix for other missing modules isn't working with this one.

[ericselk2018]

Issue with psutil was just that I didn't have it installed on my machine, and PyInstaller will only bundle something if installed.

Now I have a single EXE (15 MB) that I can place in mc_demo folder and run it. It works, in that I see the asciimatic UI loaded as expected, and no errors or warnings in the log files.

Next steps:

1. Test this EXE on a Windows PC without Python or anything MPF related on it -- probably use my wife's laptop for this.
2. Once step 1 proves out, move on to MPF MC -- mc_demo isn't much fun to look at/etc without the MC.

[toomanybrians]

Whoa, awesome! I’ll be home in a week and will be happy to test it out too on a Windows VM. A lot of people will be very excited for the work you’re doing!

[ericselk2018]

My test package worked on my wife's laptop, running Windows 10 and having nothing Python related or MPF preinstalled. Probably will be better if I provide some instructions to build the package from source, and maybe push the minor changes (to one file) to a branch somewhere... but for now I just created this to transfer to my wife's laptop to run it, so sharing this for now:

https://drive.google.com/drive/folders/1L6U9YC50xksD0QZOHP6iZT9q-ssYJXNM

There is an mc_demo.zip file in here. It is just a copy of the mc_demo as-is (no changes), with an extra run.exe in it, which is the file I made from the MPF code with PyInstaller. I just unzipped the file on my wife's laptop and ran the run.exe file in it, and I saw the expected MPF console window.

I haven't made the minor changes needed to let you pass command-line args, so I'll probably do that soon, and then push this code somewhere, maybe my private github repo for now. I've only made changes to main.py, basically added a bunch of import statements so that PyInstaller would package all of the stuff that uses dynamic imports in the real code, and changed one line of code to bypass the "commands" system, just always runs "game". Maybe I'll just attach the file to this thread, we'll see.

[toomanybrians]

If you need to make changes to MPF to support this, I'd suggest forking mpf and doing it in your repo (it would be public then), and we can work on it there and then merge them back into the main one when done. Even if they're ugly hacks, that's fine, that will give me (or whomever) a target to do them in a way that's more permanent.

Also I have a lot of experience with GitHub workflows and the CICD pipeline stuff. MPF makes decent use of this already for the builds and testing processes, so we can absolutely get the PyInstaller creation process integrated into this. Then we can host the generated binaries on GitHub as proper releases. I'm also happy to help with this or to do it if you can figure out the pyinstaller process.

[ericselk2018]

Fork created here: https://github.com/ericselk2018/mpf/tree/pyinstaller

I added some notes to top of readme.md file, for the simple command-line and test steps I have been doing. I've only been testing on Windows, but according to PyInstaller docs, it will build an executable for whatever platform it is run on -- so in theory, the same steps done on Mac will produce a Mac compatible package (seems too good to be true, surely some extra work needed).

[ericselk2018]

Mostly some notes to self, but also sharing in case I don't get to follow up on this.

I did some work on MC today. Started out trying to run PyInstaller from the MC repo, but that didn't work very well, probably because MC isn't designed to be a console application, but also for circular dependency issues (MPF imports MC and MC imports MPF). So then I added a static import for mpfmc.commands.mc to main.py in MPF code, and called the Command() method on that. This will likely be the way to go and keep the commands interface the same -- for example, a user can still run "mpf mc" instead of something new like mpf-mc.exe.

I need to look at the code in commands/both.py to run MPF and MC concurrently, currently I'm just testing by running one of these commands to toggle between the two in main.py:
mpfmc.commands.mc.Command('', '', args)
mpf.commands.game.Command('', '', args)

The MC tries to run with what I have now, but stops because mpfmc/mcconfig.yaml does not exist. I need to figure out why PyInstaller isn't including this file, I think it might need to be added as a required dependency in the toml file in MPF MC maybe. It is in my Python site packages (pip installed folder), but PyInstaller isn't copying it to the compiled PYZ bundle.

Edit: Was able to resolve the mcconfig.yaml issue by adding --collect-all=mpfmc option. There are other related commands that might be better than "all" to collect just data files, just binary files, or other options. Now getting error about a missing audio related DLL (aka progress).

[toomanybrians]

This is great, thanks for the progress update. Making permanent changes to remove the circular and dynamic imports is definitely doable. I'd be happy to take a look at that when the time is right.