[Gateway] insecure string pickle

[jihoonl]

[INFO] [WallTime: 1402386399.358722] Gateway : disengaged connection with the hub [Cafe Concert][192.168.10.151:6380]
Traceback (most recent call last):
  File "/home/yujin/turtlebot/turtlebot/src/rocon_multimaster/rocon_gateway/scripts/gateway.py", line 22, in <module>
    gateway.spin()
  File "/home/yujin/turtlebot/turtlebot/src/rocon_multimaster/rocon_gateway/src/rocon_gateway/gateway_node.py", line 78, in spin
    self._gateway.spin()
  File "/home/yujin/turtlebot/turtlebot/src/rocon_multimaster/rocon_gateway/src/rocon_gateway/gateway.py", line 78, in spin
    self.watcher_thread.start()
  File "/home/yujin/turtlebot/turtlebot/src/rocon_multimaster/rocon_gateway/src/rocon_gateway/watcher_thread.py", line 75, in start
    self._gateway.update_flipped_interface(connections, remote_gateway_hub_index)
  File "/home/yujin/turtlebot/turtlebot/src/rocon_multimaster/rocon_gateway/src/rocon_gateway/gateway.py", line 184, in update_flipped_interface
    status = hub.get_flip_request_status(flip.remote_rule)
  File "/home/yujin/turtlebot/turtlebot/src/rocon_multimaster/rocon_gateway/src/rocon_gateway/gateway_hub.py", line 717, in get_flip_request_status
    status = self.get_multiple_flip_request_status([remote_rule])
  File "/home/yujin/turtlebot/turtlebot/src/rocon_multimaster/rocon_gateway/src/rocon_gateway/gateway_hub.py", line 748, in get_multiple_flip_request_status
    rule_status, source, connection_list = utils.deserialize_request(flip)
  File "/home/yujin/turtlebot/turtlebot/src/rocon_multimaster/rocon_gateway/src/rocon_gateway/utils.py", line 218, in deserialize_request
    deserialized_list = deserialize(request_str)
  File "/home/yujin/turtlebot/turtlebot/src/rocon_multimaster/rocon_gateway/src/rocon_gateway/utils.py", line 182, in deserialize
    return pickle.loads(str_msg)
ValueError: insecure string pickle

[stonier]

Do you have any other details on this. Was this a robot, or was it the concert? Did it happen on startup, runtime or on shutdown?

[jihoonl]

It was on the robot,and runtime. The concert was up and running more than 5mins.

Though it happened, public interface connections didn't get broke and were able to receive all commands.

[stonier]

Yeah, it would bring down the gateway node, but all existing socket connections would remain.

[jihoonl]

Happened in the robot.

[INFO] [WallTime: 1418812976.291401] Gateway : disengaged connection with the hub [xavier][192.168.10.32:6380]
Traceback (most recent call last):
  File "/home/yujin/ros/turtlebot/src/rocon_multimaster/rocon_gateway/scripts/gateway.py", line 22, in <module>
    gateway.spin()
  File "/home/yujin/ros/turtlebot/src/rocon_multimaster/rocon_gateway/src/rocon_gateway/gateway_node.py", line 78, in spin
    self._gateway.spin()
  File "/home/yujin/ros/turtlebot/src/rocon_multimaster/rocon_gateway/src/rocon_gateway/gateway.py", line 78, in spin
    self.watcher_thread.start()
  File "/home/yujin/ros/turtlebot/src/rocon_multimaster/rocon_gateway/src/rocon_gateway/watcher_thread.py", line 75, in start
    self._gateway.update_flipped_interface(connections, remote_gateway_hub_index)
  File "/home/yujin/ros/turtlebot/src/rocon_multimaster/rocon_gateway/src/rocon_gateway/gateway.py", line 132, in update_flipped_interface
    status = hub.get_flip_request_status(flip.remote_rule)
  File "/home/yujin/ros/turtlebot/src/rocon_multimaster/rocon_gateway/src/rocon_gateway/gateway_hub.py", line 717, in get_flip_request_status
    status = self.get_multiple_flip_request_status([remote_rule])
  File "/home/yujin/ros/turtlebot/src/rocon_multimaster/rocon_gateway/src/rocon_gateway/gateway_hub.py", line 748, in get_multiple_flip_request_status
    rule_status, source, connection_list = utils.deserialize_request(flip)
  File "/home/yujin/ros/turtlebot/src/rocon_multimaster/rocon_gateway/src/rocon_gateway/utils.py", line 218, in deserialize_request
    deserialized_list = deserialize(request_str)
  File "/home/yujin/ros/turtlebot/src/rocon_multimaster/rocon_gateway/src/rocon_gateway/utils.py", line 182, in deserialize
    return pickle.loads(str_msg)
ValueError: insecure string pickle

[piyushk]

According to http://stackoverflow.com/questions/1746825/python-valueerror-insecure-string-pickle, it's probably a corrupted pickle. We should catch this error, and discard the original request being deserialized.

[jihoonl]

Do you have any suggestion to chase up this issue? where would be the best place to start to add debug message?

[piyushk]

I don't have a particularly good solution for this problem.

1. We should catch this error in the deserialize function, print a debug message, and return a deserialization error.
2. Everything in gateway_hub.py should catch the deserialize error and deal with it appropriately. Unfortunately, we'll have to look at each deserialize call one by one and see how we want to handle it if something goes wrong.
3. We then need to establish that when this error happens, have the values on the redis hub gotten corrupted (i.e. the debug message is printed over and over again or not). If the value on the redis hub has gotten corrupted, we'll need to discuss how we want to fix that. Maybe we can figure out why the value got corrupted in the first place. If the error message is only printed one off, then we should be fine as things will normalize in the next update.

[jihoonl]

Another log

[INFO] [WallTime: 1433148471.084004] Gateway : lost connection with hub, attempting to disconnect...
Traceback (most recent call last):
  File "/home/yujin/ros/turtlebot/src/rocon_multimaster/rocon_gateway/scripts/gateway.py", line 22, in <module>
    gateway.spin()
  File "/home/yujin/ros/turtlebot/src/rocon_multimaster/rocon_gateway/src/rocon_gateway/gateway_node.py", line 81, in spin
    self._gateway.spin()
  File "/home/yujin/ros/turtlebot/src/rocon_multimaster/rocon_gateway/src/rocon_gateway/gateway.py", line 78, in spin
    self.watcher_thread.start()
  File "/home/yujin/ros/turtlebot/src/rocon_multimaster/rocon_gateway/src/rocon_gateway/watcher_thread.py", line 75, in start
    self._gateway.update_flipped_interface(connections, remote_gateway_hub_index)
  File "/home/yujin/ros/turtlebot/src/rocon_multimaster/rocon_gateway/src/rocon_gateway/gateway.py", line 184, in update_flipped_interface
    status = hub.get_flip_request_status(flip.remote_rule)
  File "/home/yujin/ros/turtlebot/src/rocon_multimaster/rocon_gateway/src/rocon_gateway/gateway_hub.py", line 717, in get_flip_request_status
    status = self.get_multiple_flip_request_status([remote_rule])
  File "/home/yujin/ros/turtlebot/src/rocon_multimaster/rocon_gateway/src/rocon_gateway/gateway_hub.py", line 748, in get_multiple_flip_request_status
    rule_status, source, connection_list = utils.deserialize_request(flip)
  File "/home/yujin/ros/turtlebot/src/rocon_multimaster/rocon_gateway/src/rocon_gateway/utils.py", line 218, in deserialize_request
    deserialized_list = deserialize(request_str)
  File "/home/yujin/ros/turtlebot/src/rocon_multimaster/rocon_gateway/src/rocon_gateway/utils.py", line 182, in deserialize
    return pickle.loads(str_msg)
ValueError: insecure string pickle

[jihoonl]

Looking at the three different logs, insecure string pickle error raised while deserializing flip_in request in all cases. Thus, flip_ins keys in hub is corrupted probably.