BitSet not serialized correctly (1.6+?)

[eheimburg]

This is a tricky one for me to report - I'm upgrading our massive project from 0.111 to 1.6. That's as far as I can easily upgrade today, because I'm not on Java 12 quite yet. So this problem occurs in 1.6, and does not occur in 0.111. I don't know if it occurs in 2.3, and would like to know!

BitSet gets serialized correctly, but after loading it, it's not correctly initialized in memory: the internal transient variable BitSet.wordsInUse is not initialized. This causes the results of most BitSet functions to be incorrect.

I can fix it by setting/clearing a bit manually in the @PostLoad. Setting a bit internally calls the private function BitSet.recalculateWordsInUse(). But morphia isn't doing it automatically during re-serialization anymore. (I say "anymore" because it worked in 0.111)

I have a lot of BitSets to fix manually, but I can. I'm more concerned about why it broke and if it's fixed.

Is this fixed in a newer version? Is there a test case for BitSet that we know works? (If I need to keep upgrading the project to Java 12 to fix this, then I will, but that's a lot of work if it won't actually help.)

Thanks for the insight, and the excellent library. We've been using it for almost a decade now!

[evanchooly]

Wow, a decade! That's pretty amazing. Thanks for being such a loyal user. Now to your questions.

As of Morphia 2.0, the baseline is Java 11 and not 12. I'm not sure if that makes a difference or not. Java 11 is an LTS for what it's worth. I'm not sure what would have change in 1.6 vs 0.111. Both are rather ancient at this point so my memory is hazy. Have you tried with an earlier 1.x version to see if the regression shows up earlier? I don't recall if it's the driver or Morphia doing that mapping in 0.11 but I'd be surprised if it was Morphia. It doesn't appear there's a BitSet codec in 2.0, either in Morphia or the driver itself, so the support there is utterly lacking. Writing your own codec is not terribly difficult. If you'd like to attempt that and register it, I can help walk you through it. Otherwise we can look at Morphia providing it though that may or may not mean a 2.4 release (depending on how pedantic everyone is feeling about semver).

[eheimburg]

Ah, Java 11 is actually where we're stuck at, so I can upgrade to that! That's great news. (I can't find the link that said it required Java 12, but I know I read it somewhere on the internet...)

Speaking of not finding things -- I remember reading in the patch notes years ago (when upgrading to some ancient version prior to 0.111) that the new version supported BitSet thanks to a user contribution by someone. But I can't find any trace of that. We were using 0,99.1-SNAPSHOT.jar before 0.111, and that also supports BitSet. I haven't tried versions between 0.111 and 1.6 yet.

It may also be related to the MongoDB driver change -- we were using the 2.11.3 mongo driver for forever, so that could be what changed. (from the "legacy" to the new driver?)

But anyway, it sounds like the right answer is a custom codec! That seems doable. Can you give me some pointers on where to start? Thanks!

[evanchooly]

As luck would have it, I do have an example of a custom codec. For various reasons, I needed to test something involving a ZonedDateTime but didn't want to commit to having it in core. (Largely because there are competing, valid choices on how to store such types.) At any rate, you can find that codec here. Once you have a codec, you need a provider that you can find here. Once you have those two pieces, you simply need to register them when setting up Morphia. You can see that happening here.

This should get you started, at least. If you have any questions, let me know.

[eheimburg]

Thanks for the assistance on this. Is a BitSet codec worth adding to the library? If so, I can put up a PR for it.

The codec I wrote saves/loads the bits as an array of 64-bit ints (which is how the class stores them internally). For backwards compatibility with the legacy driver, it can also read a BitSet as a nested document containing a variable "words" which contains the array of longs.

As well as some basic unit tests.

[evanchooly]

If you have it, it wouldn't hurt to include it here, I don't think. I'm not sure the legacy support is necessary at this point, though, but maybe it's worth having.