Remove requests dependency?

[zzbennett]

I'm trying out the collectd-cloudwatch plugin and it seems like it is just what I need. I've run into a number of frustrating setup pains however, particularly around installing requests, which is the only external dependency (disclaimer: I am not a python developer so I'm kind of stumbling through this)

Some excepts:

[root@ip-10-0-203-249 ec2-user]# python ./setup.py
Installing dependencies ... OK
Installing python dependencies ... OK
Downloading plugin ... OK
Extracting plugin ... OK
Moving to collectd plugins directory ... OK
Copying CloudWatch plugin include file ... OK
Traceback (most recent call last):
  File "./setup.py", line 658, in <module>
    main()
  File "./setup.py", line 651, in main
    install_plugin()
  File "./setup.py", line 576, in install_plugin
    supply_config()
  File "./setup.py", line 592, in supply_config
    _prepare_plugin_config(config)
  File "./setup.py", line 606, in _prepare_plugin_config
    InteractiveConfigurator(plugin_config, metadata_reader, COLLECTD_INFO).run()
  File "./setup.py", line 313, in run
    self._configure_region()
  File "./setup.py", line 341, in _configure_region
    region = self.metadata_reader.get_region()
  File "./setup.py", line 201, in get_region
    region = self._get_metadata(MetadataReader._REGION_METADATA_REQUEST)
  File "./setup.py", line 220, in _get_metadata
    from requests import Session, codes
  File "/usr/local/lib/python2.7/site-packages/requests/__init__.py", line 74, in <module>
    raise RuntimeError('Requests dependency \'chardet\' must be version >= 3.0.2, < 3.1.0!')
RuntimeError: Requests dependency 'chardet' must be version >= 3.0.2, < 3.1.0!

Doing sudo pip install chardet=3.0.4 yields this:

[ec2-user@ip-10-0-203-249 ~]$ sudo pip install chardet==3.0.4
Requirement already satisfied: chardet==3.0.4 in /usr/local/lib/python2.7/site-packages

So the right version of chardet is installed in the directory /usr/local/lib/python2.7/site-packages, however, there is another version installed to /usr/lib/python2.7/dist-packages which is version 2.0.1.

The setup.py script is not loading the correct version because the sys.path is listing dist-packages first. Now, I think I could do sudo easy_install -U chardet=3.0.4 and try to upgrade the version that is in /usr/lib/python2.7/dist-packages but I don't know if that will break other python scripts in my system that depend on that specific version of chardet.

I ran the setup.py script in a virtualenv to try to get the initial setup working, which worked pretty well until the setup.py script tried to restart collectd.

Plugin configuration written successfully.
Creating backup of the original configuration ... OK
Replacing collectd configuration ... OK
Replacing whitelist configuration ... OK
Stopping collectd process ... NOT OK
Starting collectd process ... NOT OK
Installation cancelled due to an error.
Executed command: '/usr/sbin/collectd'.
Error output: '[2017-06-13 01:04:17] plugin_load: plugin "logfile" successfully loaded.
Error: Reading the config file failed!
Read the logs for details.'.

I saw this error in the collectd.log file:

[2017-06-13 00:26:39] [error] python plugin: Error importing module "cloudwatch_writer".
[2017-06-13 00:26:39] [error] Unhandled python exception in importing module: ImportError: No module named requests
[2017-06-13 00:26:39] [error] Traceback (most recent call last):
[2017-06-13 00:26:39] [error]   File "/opt/collectd-plugins/cloudwatch_writer.py", line 10, in <module>
    from cloudwatch.modules.configuration.confighelper import ConfigHelper
[2017-06-13 00:26:39] [error]   File "/opt/collectd-plugins/cloudwatch/modules/configuration/confighelper.py", line 4, in <module>
    from metadatareader import MetadataReader
[2017-06-13 00:26:39] [error]   File "/opt/collectd-plugins/cloudwatch/modules/configuration/metadatareader.py", line 2, in <module>
    from requests import Session, codes
[2017-06-13 00:26:39] [error] ImportError: No module named requests
[2017-06-13 01:04:17] [info] plugin_load: plugin "cpu" successfully loaded.
[2017-06-13 01:04:17] [info] plugin_load: plugin "df" successfully loaded.
[2017-06-13 01:04:17] [info] plugin_load: plugin "memory" successfully loaded.
[2017-06-13 01:04:17] [info] plugin_load: plugin "swap" successfully loaded.

So whatever path collectd is using to run python doesn't even have requests installed.

Okay so long story short, I think the setup process would be easier and less error prone if users didn't need to fuss with external dependencies and python paths to get this working. This is especially true because this python script and the collectd plugin depend on root's python environment.

[zzbennett]

Alternatively, rather than installing requests in the root python environment, it could be installed in the path where the signalfx collectd plugin lives, e.g. /opt/collectd-plugins. I just copied the requests library there and that solved all my issues.