Allow to provide a fallback serializer to avoid throwing an exception for non serializable objects

[gabrielgiussi]

Intro

We are writing a logback layout to write structured logs in JSON using Jackson (through cheshire currently but also testing alternatives like jsonista) but we have to fallback to the unstructured logging if the event contains non-serializable objects.
The goal is to make the serialization succeed even in the presence of non-serializable objects, serializing those with a special placeholder or an empty string.

Alternatives

Avoid adding non-serializable objects to the log event

One alternative is to make the user responsible of passing only serializable arguments to the log event, however this is quite cumbersome since in a concrete case is a big map containing different configuration properties, and this is injected from user code, while the logging is happening in a framework, so the framework would have to be explicit on which properties to log, which is (a) verbose and (b) discards information that could be serialized but, since the framework only logs known properties, it is discarded.
This alternative looks like this in our codebase

(log/info logger {:log    ::starting-producer
                  :config (select-keys config [ProducerConfig/BOOTSTRAP_SERVERS_CONFIG
                                               ProducerConfig/ENABLE_IDEMPOTENCE_CONFIG
                                               ProducerConfig/VALUE_SERIALIZER_CLASS_CONFIG
                                               ProducerConfig/KEY_SERIALIZER_CLASS_CONFIG
                                               ProducerConfig/MAX_IN_FLIGHT_REQUESTS_PER_CONNECTION
                                               ProducerConfig/TRANSACTION_TIMEOUT_CONFIG
                                               ProducerConfig/RETRIES_CONFIG
                                               ProducerConfig/ACKS_CONFIG
                                               ProducerConfig/TRANSACTIONAL_ID_CONFIG])})

Custom serializers

The jsonista library allows to configure the ObjectMapper, but we don't want to configure a Serializer for each one of the objects that are non-serializable, even if we do that for the ones we know today, new objects could be included in the config and the logs will fallback to the unstructured format.

(require '[jsonista.core :as j])
(def mapper
  (j/object-mapper
    {:modules [(doto
                 (SimpleModule.)
                 (.addSerializer AFunction (proxy [JsonSerializer] []
                                             (serialize [_obj jsonGenerator serializer]
                                               (.writeString jsonGenerator "<non-serializable>")))))]}))
(j/write-value-as-string {:a (fn [] 1) :b "a"} mapper)
; => "{\"a\":\"<non-serializable>\",\"b\":\"a\"}"

Tweak ObjectMapper features

Configuring an ObjectMapper like this

(def mapper (doto (j/object-mapper)
              (.setVisibility PropertyAccessor/ALL
                              JsonAutoDetect$Visibility/NONE)
              (.disable SerializationFeature/FAIL_ON_EMPTY_BEANS)))


(j/write-value-as-string {:a 123 :b (atom {:a 123})} mapper)
; => "{\"a\":123,\"b\":{}}"

will do te trick of not failing when, for example, an attempt to serialize an clojure.lang.Atom is made. The visibility prevents Jackson from using the getters and the FAIL_ON_EMPTY_BEANS is what makes the serialization succeed writing an empty json object instead of failing.
However, this has the disadvantage of affecting the expected behavior for POJOs which are serialized out of the box by using their getters.

Feature request

Add a SerializationFeature that will allow to fallback to a general serializer when an object can't be serialized, instead of throwing the exception.

Related issues

• Fallback serializers? metosin/jsonista#48
• Support some kind of fallback encoder dakrone/cheshire#161

[pjfanning]

I don't think a SerializationFeature is good for this - a feature is an on/off flag. You probably need to be able to provide a Serialization Provider instance - different people would want to have different fallback behaviours. Little point hardcoding one behaviour - the fallback could be Java serialization, Java toString or any host of alternatives.

Honestly, I don't see any of this providing much better support than users having try/catch blocks around the Jackson serialization call and then having the fallback serialization in the catch.

If this is because Clojure users hate exceptions, Jackson is a Java lib and exceptions are ok in Java. Chesire can add its own support for providing a fallback function and having the internal code catch the Jackson exception and call the fallback function.

[pjfanning]

If are interested in doing a POC, then Jackson has a DeserializationProblemHandler and an equivalent SerializationProblemHandler might be useful.

[gabrielgiussi]

You probably need to be able to provide a Serialization Provider instance - different people would want to have different fallback behaviours

Agreed

having try/catch blocks around the Jackson serialization call and then having the fallback serialization in the catch.

I don't think this covers my use case since I'm trying to serialize an heterogeneous map of objects and some of them might be non-serializable, but I don't want the whole serialization to fail, just skip (or provide a default serialization) the ones that couldn't be serialized. If I use a try catch then I will have to deal with serializing the whole map.
I don't think cheshire and jsonista can do much without having that support in Jackson, of course this may seems insufficient to introduce a new feature to Jackson, but I think it may be beneficial for Jackson itself.

then Jackson has a DeserializationProblemHandler and an equivalent SerializationProblemHandler might be useful.

This was the thing I was looking for, I'll try to see if a SerilizationProblemHandler could be introduced. Thanks.

[pjfanning]

Jackson has good support for custom serialisation. Maybe you would be best writing a custom serialiser for Maps that does what you want.

[cowtowncoder]

Right, we'd probably need a way to register fallback JsonSerializer. But realistically it probably should be a callback of some kind... and it all does depend on specific reason why something is not serializable.
Handling of "no properties" found is not in a convenient place so not sure how tricky this would be.

In a way, existing interface Serializers would be one place to add the new method.

[cowtowncoder]

One other thing to note: serializers cannot prevent value from being serialized; that is, at the point when serializer gets called, a value MUST be produced (mostly since Object's property name has been already written).
It is possible to filter out values using other means but no by custom serializers; either by preventing inclusion of properties of POJOs when constructing BeanSerializer, or by various other filter mechanisms (used by @JsonView, @JsonFilter).

[gabrielgiussi]

and it all does depend on specific reason why something is not serializable.
Handling of "no properties" found is not in a convenient place so not sure how tricky this would be.

I was thinking in catching InvalidDefinitionException, which is the exception being thrown for example when no properties are found

(j/write-value-as-string {:a (Object.)})
; Execution error (InvalidDefinitionException) at com.fasterxml.jackson.databind.exc.InvalidDefinitionException/from (InvalidDefinitionException.java:77).
; No serializer found for class java.lang.Object and no properties discovered to create BeanSerializer (to avoid exception, disable SerializationFeature.FAIL_ON_EMPTY_BEANS) (through reference chain: clojure.lang.PersistentArrayMap[":a"])

Are you saying that wouldn't be a reasonable decision to trigger the fallback serialization?

a value MUST be produced (mostly since Object's property name has been already written).

Yes, I'm fine with that, since the main goal is to avoid the serialization to fail.

[cowtowncoder]

Catching exceptions is a rather challenging design for most things, due to couple of reasons:

1. Constructing Exceptions is VERY expensive (or rather filling in stack trace)
2. Due to exceptions short-cutting handling the state of classes is not guaranteed to be consistent: it's an error condition and may or may not recoverable (in this case it would probably be safe tho, fwtw)
3. There are many reasons for InvalidDefinitionException and catching that could hide issues (sort of like logic on what kind of request failures may be retried safely)
4. Handling would need to be done in many places, not at root level to handle "closest" serializer failure; probably all through the codebase

So I don't think that would be something I wanted to do within core databind handling.

Having said that it can certainly be something that code outside of jackson-databind could do. But I realize that it may not be feasible (as handling would need to be close to actual failure).