Monitor how often each binary constraint rejects a proposal

[maxhully]

We could have an alternative Validator that checks all the constraints and keeps Counters of how often each constraint is true/false.

[maxhully]

This would be good for benchmarking/troubleshooting weird chain behavior.

[rdbliss]

Implemented in #182, but creates big log files for lengthy runs. My plan was to instead track things with a dictionary in Validator since that should be pretty inexpensive. I haven't started this yet.

@apizzimenti Are you still working on this? If so, this morning's master has the first implementation. There was a little bit of talk on gitter about this as well. (I'll stay away from this unless you've moved on!)

[pizzimathy]

(Accidentally unassigned myself – oops)

I was going to take a look at it after work today, as it seemed reasonably low-priority and I have some stuff to tackle back in Iowa right now. I can take your idea on improving #182 and run with it a bit, though.

[rdbliss]

All yours then - try whatever you like! Just ping in here if you want someone else to jump on it.

…

On Tue, Jul 17, 2018, 15:37 Anthony Pizzimenti ***@***.***> wrote: (Accidentally unassigned myself – oops) I was going to take a look at it after work today, as it seemed reasonably low-priority and I have some stuff to tackle back in Iowa right now. I can take your idea on improving #182 <#182> and run with it a bit, though. — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#166 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AMyLFzZ1hUmVBH0QM09FdLnd1I45N0zlks5uHj0HgaJpZM4VNeSv> .

[zschutzman]

Was just flipping through chain stuff and I saw this issue -- I have something kind of like this working. Currently, Validator checks against constraints in the order they're added to the constructor. This is good, because it lets me reject things which violate population balance (fast) before checking whether they're contiguous (kinda fast) before computing a bunch of eigenvalues (slow!). If you instead want to count the number of steps on which any of the checks fail, you need to run all of them, which slows things down if you have expensive constraints.

The implementation for sequential checks is really basic. I added
self.constraint_fails = {c : 0 for c in constraints}
to the constructor of Validator and replaced
return False
in __call__() with
self.constraint_fails[constraint] +=1
return False

The keys of the dictionary are the constraint function objects themselves, so there's no need to try to come up with key names for every constraint that might get passed through.

[maxhully]

Awesome! That seems like the right way to do this. I can add it and merge it in sometime soon.