Skip to content
This repository has been archived by the owner on Apr 18, 2018. It is now read-only.

Depend on simplejson for Python < 2.7 #47

Open
wants to merge 1 commit into
base: develop
Choose a base branch
from
Open

Depend on simplejson for Python < 2.7 #47

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 0 additions & 1 deletion examples/top_urls/requirements.txt
View file
Open in desktop

This file was deleted.

7 changes: 1 addition & 6 deletions examples/top_urls/top_urls/requests_generator.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -5,12 +5,7 @@
from random import choice
import time

try:
import simplejson as json
_ = json # pyflakes
except ImportError:
import json

from pyleus.compat import json
from pyleus.storm import Spout


Expand Down
11 changes: 10 additions & 1 deletion pyleus/compat.py
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,6 +1,8 @@
import sys

if sys.version_info[0] < 3:
python_version = (sys.version_info[0], sys.version_info[1])

if python_version < (3, 0):
from cStringIO import StringIO
BytesIO = StringIO
else:
Expand All @@ -9,3 +11,10 @@

_ = BytesIO # pyflakes
_ = StringIO # pyflakes

if python_version < (2, 7):
import simplejson as json
else:
import json
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

In this way we are preventing the use of simplejson for python 2.7 and 3.x, right? I know that a try/catch import here misses the point of this change (even though it would still simplify all other modules), but I feel we are losing something that might be useful (e.g. from my measurements, simplejson seemed to achieve lower latencies than json while processing tuples).

Nevertheless, I don't want to block the pull request for such an issue, since we ship messagepack as default encoding format and simplejson has problems too (https://code.google.com/p/simplejson/issues/detail?id=40). I just wanted to point it out.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Did you do that test on 2.7? My understanding was that the real performance benefit was on 2.6, but that come 2.7 it was negligible since the relevant improvements had been mainlined.


_ = json # pyflakes
9 changes: 2 additions & 7 deletions pyleus/json_fields_bolt.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -3,13 +3,8 @@

import logging

try:
import simplejson as json
_ = json # pyflakes
except ImportError:
import json

from .storm import SimpleBolt
from pyleus.compat import json
from pyleus.storm import SimpleBolt

log = logging.getLogger(__name__)

Expand Down
7 changes: 1 addition & 6 deletions pyleus/storm/component.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -11,12 +11,7 @@ class around Storm configurations.
import sys
import traceback

try:
import simplejson as json
_ = json # pyflakes
except ImportError:
import json

from pyleus.compat import json
from pyleus.storm import DEFAULT_STREAM
from pyleus.storm import StormTuple
from pyleus.storm.serializers.msgpack_serializer import MsgpackSerializer
Expand Down
7 changes: 1 addition & 6 deletions pyleus/storm/serializers/json_serializer.py
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,11 +1,6 @@
"""JSON implementation of Pyleus serializer"""

try:
import simplejson as json
_ = json # pyflakes
except ImportError:
import json

from pyleus.compat import json
from pyleus.storm import StormWentAwayError
from pyleus.storm.serializers.serializer import Serializer

Expand Down
4 changes: 4 additions & 0 deletions setup.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -65,6 +65,10 @@ def run(self):
# argparse is in the standard library of Python >= 2.7
extra_install_requires.append("argparse")

# simplejson has better beformance in Python < 2.6 than the built-in json
# module
extra_install_requires.append("simplejson")


setup(
name="pyleus",
Expand Down
7 changes: 1 addition & 6 deletions tests/storm/serializers/json_serializer_test.py
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,11 +1,6 @@
import mock

try:
import simplejson as json
_ = json # pyflakes
except ImportError:
import json

from pyleus.compat import json
from pyleus.compat import StringIO
from pyleus.storm.serializers.json_serializer import JSONSerializer
from testing.serializer import SerializerTestCase
Expand Down