External commands do no handle non ascii characters

[jccomputing]

Hello

I found that when I send an external command that contains non ascii characters, it's not taken into account. I tested this with Thruk and Nagvis with French accents.

The external tools do send a proper UTF-8 encoded string (verified with Firebug for the http part and tcpdump for the livestatus tcp socket). The broker receives the external command, but it's never transmitted to the scheduler.

Here is the reason why:
The string is received as a Python byte string. Then it's decoded to a proper Python unicode string. The representation of this object contains escaped Unicode characters and some are mapped to the latin1 character set. When Shinken tries to send this Unicode string, it serializes it with Pickle, then convert the output to JSON with json.dumps. The JSON dumps function expects UTF-8 input, but it finds latin1 in the Pickle dump and throws a UnicodeDecodeError exception. The external never reaches it's destination.

Here is how to reproduce this behavior in the Python interpreter:

Python 2.7.3 (default, Mar 13 2014, 11:03:55) 
[GCC 4.7.2] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import locale
>>> locale.getdefaultlocale()[1]
'UTF-8'
>>> import json
>>> import cPickle
>>> s = "[1403598279] ACKNOWLEDGE_HOST_PROBLEM;dummy-host-crit;2;1;0;thrukadmin;test é"
>>> u = s.decode('utf8')
>>> u
u'[1403598279] ACKNOWLEDGE_HOST_PROBLEM;dummy-host-crit;2;1;0;thrukadmin;test \xe9'
>>> cp = cPickle.dumps(u)
>>> json.dumps(cp)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/lib/python2.7/json/__init__.py", line 231, in dumps
    return _default_encoder.encode(obj)
  File "/usr/lib/python2.7/json/encoder.py", line 194, in encode
    return encode_basestring_ascii(o)
UnicodeDecodeError: 'utf8' codec can't decode byte 0xe9 in position 77: invalid continuation byte
>>>

We have two solutions to this problem:

1. Drop the bytestring to unicode conversion. This solution affects the Livestatus module itself. To do this we should delete this line.
2. Ensure that the data are 7 bits safe. This solution affects Shinken's core. We should use for external commands what's done in other parts: cPickle, then zlib compression, then base64 encoding.

For information, the named-pipe module doesn't decode incoming data to unicode so it works, but the webui module does, so it has the same problem.

JCC