Pickling & decorators

[emptierset]

We are using this library for an implementation of a board game.

We need the ability to pickle and unpickle games. We are using dill for this because my initial attempts to pickle with basic pickle saw failures[0] when it tried to pickle certain things inside StateMachine. We noticed an interesting bug when unpicking saved games and I was wondering if you might have any input.

Essentially, we have a transition list like this:

end_game = s.TURN_END.to(s.GAME_END)

@end_game.cond
def game_is_over(self) -> bool:
   ...

advance_round =  end_game | s.TURN_END.to(s.GAME_PLAY)

When playing a game after it was pickled and unpickled, sometimes it goes to GAME_END at surprising times, even if manually calling game_is_over via REPL shows that it is false. After some digging, I identified that the cond isn't actually running, so the "game over" transition is always getting picked.

Do you have any idea why pickling and/or unpickling causes cond to "fall off"?

Is your expectation that state machines are serializable?

Do you think switching our codebase to a non-decorator approach would help?

---

[0]: Specifically, it failed with AttributeError: Can't pickle local object 'SignatureAdapter.wrap.<locals>.signature_adapter'.

[fgmacedo]

Hi @emptierset, how's it going? Thanks for reporting this!

I noticed there aren't automated tests specifically for pickle. I did some manual tests in the past, and it worked fine back then. However, a lot has changed internally, so this might be a regression.

State machines are meant to be serializable, but I'd love to understand your use case better. Could you share what you're trying to achieve by serializing the state machine? I'm asking because maybe an approach like this could be good enough.

I'll set up an automated test to reproduce the issue. Do you happen to have a complete example that demonstrates the problem?

Looking forward to your reply!

[emptierset]

Ah, got it. Thanks for the time.

[emptierset]

I think it worked! I changed my toy example to def __init__(self, *args, **kwargs): to let it work trivially with however your code was trying to use it, and the test passed.

[emptierset]

OK, after messing with it a bit more, I think this change makes it so a lot of basic state doesn't get retained. I reworked my custom __init__ to not take any parameters (and instead "inject" them via the event that moves the machine out of INIT) to allow the unpickling to work, but here is the difference between the pre-pickle Game object and post-unpickle Game object:

https://www.diffchecker.com/nFU2mDR0/

Basically, almost none of state specific to my application is still there, presumably because my class subclasses StateMachine and does not override __reduce__, so dill just uses StateMachine.__reduce__. I think it would be ideal if the default pickling behavior is retained, but StateMachine somehow also reregisters all of the callbacks. Not sure if that's possible.

[emptierset]

I was fiddling with the statemachine code and this makes my test pass:

    def __getstate__(self):
        state = self.__dict__.copy()
        return state

    def __setstate__(self, state):
        self.__dict__.update(state)
        self._register_callbacks([])

I'm not sure if it's too minimal. Might want self.add_listener(*self._listeners.keys()) in there, but my code doesn't use listeners. Also, might want a self._callbacks.clear() in there, since I assume the callbacks registry is full of junk that can never matter post-pickling.

[emptierset]

Oops, yes, you need that self._callbacks_registry.clear() if you want this to also work for in-memory pickling. Otherwise, there will be non-junk duplicate callbacks that happen.