Merge #1396: Use python3's venv module instead of virtualenv

11ddec7 docs: Remove now-confusing mentions of virtualenv (Carl Dong)
6e3b6ec Dockerfiles: Use Python venv instead of virtualenv (Carl Dong)
8bbca60 install: Use python3's venv module when available (Carl Dong)

Pull request description:

  Python 3.3 and above provide a standard venv module which provides all the functionality that install.sh needs from virtualenv.

  Detect its existence and use it, fallback to virtualenv if not.

Top commit has no ACKs.

Tree-SHA512: 426ff4397e9142f991c3f091f6a17974a9f2fb694b652e40972a2323992d97c5817528d05dc1a17f7928ac7874ed7c96c076ae1818af2cd3def02cbae71ad512

docs/INSTALL.md

@@ -24,7 +24,7 @@ is actually newer in version number, than what was there already.
 
 To install everything (client and server), install these packages:
 
-    sudo apt-get install python3-dev python3-pip git build-essential automake pkg-config libtool libffi-dev libssl-dev
+    sudo apt-get install python3-dev python3-pip python3-venv git build-essential automake pkg-config libtool libffi-dev libssl-dev
 
 (+ `libsodium-dev` if you can find it, else build after)
 
@@ -46,8 +46,7 @@ Then install this repo:
 
 Then:
 
-    sudo pip install virtualenv
-    virtualenv --python=python3 jmvenv
+    python3 -m venv jmvenv
     source jmvenv/bin/activate
 
 **At this point you should see `(jmvenv)` at the beginning of your command prompt.**
@@ -72,7 +71,7 @@ Then build and install a local copy of libsecp256k1 for python-bitcointx:
 
 #### Installing packages to run everything in-one:
 
-> *NOTE*: It is very important to have activated virtualenv before running this step. Otherwise, `pip install` will fail, you may be tempted to re-run it with `sudo pip install` which will cause problems in the future.
+> *NOTE*: It is very important to have activated the virtual environment before running this step. Otherwise, `pip install` will fail, you may be tempted to re-run it with `sudo pip install` which will cause problems in the future.
 
     pip install -r requirements/base.txt
 
@@ -112,10 +111,9 @@ If you have installed this "full" version of the client, you can use it with the
     git clone https://github.com/Joinmarket-Org/joinmarket-clientserver
     cd joinmarket-clientserver
     ```
-6) Create virtualenv "jmvenv"
+6) Create virtual environment "jmvenv"
     ```sh
-    sudo pip3 install virtualenv
-    virtualenv jmvenv
+    python3 -m venv jmvenv
     source jmvenv/bin/activate
     ```
     At this point you should see `(jmvenv)` at the beginning of your command prompt.
@@ -235,7 +233,7 @@ If you installed using WSL, the following configuration is necessary:
 > note: you need to have installed JoinMarket with Qt support (see [this](../README.md#joinmarket-qt) section in the readme file)
 1. In Ubuntu, install additional dependencies `sudo apt install libgl1-mesa-glx`.
 2. Download and install [MobaXterm](https://mobaxterm.mobatek.net). This program needs to be running before you can start JoinMarket-Qt. It requires no additional configuration.
-3. Open WSL-Ubuntu session in MobaXTerm. Go to JoinMarket directory and run `source jmvenv/bin/activate` to activate Python virtualenv.
+3. Open WSL-Ubuntu session in MobaXTerm. Go to JoinMarket directory and run `source jmvenv/bin/activate` to activate the Python virtual environment.
 4. You can now start JoinMarket-Qt as described [here](JOINMARKET-QT-GUIDE.md).
 If you find that the program crashes with `qt.qpa.plugin: Could not load the Qt platform plugin`, you can add Qt5 dependencies with `sudo apt install qtbase5-dev` and try again.
 
@@ -286,7 +284,7 @@ do not run the `python setupall.py` commands above. Instead run:
     python setupall.py --develop
 
 The normal installation (`--daemon` or `--client-bitcoin`) would install the JoinMarket
-packages to the virtualenv's `site-packages` directory. This would mean any changes you make to
+packages to the virtual environment's `site-packages` directory. This would mean any changes you make to
 the local code would not have effect until the packages are reinstalled.
 
 Using `--develop` causes a `.egg-link` file to be added to `site-packages` for each package.

docs/TESTING.md

@@ -1,6 +1,6 @@
 ### Test instructions (for developers):
 
-Work in your `jmvenv` virtualenv as for all Joinmarket work. Make sure to have [bitcoind](https://bitcoin.org/en/full-node) 0.18 or newer installed. Also need miniircd installed to the root (i.e. in your `joinmarket-clientserver` directory):
+Work in your `jmvenv` virtual environment as for all Joinmarket work. Make sure to have [bitcoind](https://bitcoin.org/en/full-node) 0.18 or newer installed. Also need miniircd installed to the root (i.e. in your `joinmarket-clientserver` directory):
 
     (jmvenv)$ cd /path/to/joinmarket-clientserver
     (jmvenv)$ git clone https://github.com/Joinmarket-Org/miniircd

docs/onion-message-channels.md

@@ -156,7 +156,7 @@ but in any case, very high uptime. For reliability it also makes sense to config
 
 The currently suggested way to run a directory node is to use the [`start-dn.py` script](https://github.com/JoinMarket-Org/custom-scripts/blob/master/start-dn.py); you can place it in your `joinmarket-clientserver/scripts` directory and run it with a message argument to be sent as part of the connection handshake, for example: 'Greetings from Directory Node' and one option flag: `--datadir=/your/chosen/datadir` (as you'll see below).
 
-This slightly unobvious approach is based on the following ideas: we run a Joinmarket script, with a Joinmarket python virtualenv, so that we are able to parse messages; this means that the directory node *can* be a bot, e.g. a maker bot, but need not be - and here it is basically a "crippled" maker bot that cannot do anything. This 'crippling' is actually very useful because (a) we use the `no-blockchain` argument (it is forced in-code; you don't need to set it) so we don't need a running Bitcoin node (of whatever flavour), and (b) we don't need a wallet either.
+This slightly unobvious approach is based on the following ideas: we run a Joinmarket script, with a Joinmarket python virtual environment, so that we are able to parse messages; this means that the directory node *can* be a bot, e.g. a maker bot, but need not be - and here it is basically a "crippled" maker bot that cannot do anything. This 'crippling' is actually very useful because (a) we use the `no-blockchain` argument (it is forced in-code; you don't need to set it) so we don't need a running Bitcoin node (of whatever flavour), and (b) we don't need a wallet either.
 
 #### Joinmarket-specific configuration
 

install.sh

@@ -67,9 +67,9 @@ deps_install ()
         'python3-dev' \
         'python3-pip' \
         'python3-setuptools' \
+        'python3-venv' \
         'libltdl-dev' )
 
-    if [ "$with_jmvenv" == 1 ]; then debian_deps+=("virtualenv"); fi
     if [ "$with_sudo" == 1 ]; then debian_deps+=("sudo"); fi
 
     darwin_deps=( \
@@ -152,21 +152,6 @@ dar_deps_install ()
     if ! brew install ${dar_deps[@]}; then
         return 1
     fi
-
-    if ! which virtualenv >/dev/null; then
-        sudo_command=''
-        if [ "$with_sudo" == 1 ]; then
-            echo "
-                sudo password required to run :
-
-                \`sudo pip3 install virtualenv\`
-                "
-            sudo_command="sudo"
-        fi
-        if $with_jmvenv && ! $sudo_command pip3 install virtualenv; then
-            return 1
-        fi
-    fi
 }
 
 check_skip_build ()
@@ -193,7 +178,7 @@ venv_setup ()
     else
         reinstall='true'
     fi
-    virtualenv -p "${python}" "${jm_source}/jmvenv" || return 1
+    "${python}" -m venv "${jm_source}/jmvenv" || return 1
     source "${jm_source}/jmvenv/bin/activate" || return 1
     pip install --upgrade pip
     pip install --upgrade setuptools
@@ -623,7 +608,7 @@ main ()
     fi
     if [ "$with_jmvenv" == 1 ]; then
         if ! venv_setup; then
-            echo "Joinmarket virtualenv could not be setup. Exiting."
+            echo "Joinmarket Python virtual environment could not be setup. Exiting."
             return 1
         fi
         source "${jm_root}/bin/activate"
@@ -667,7 +652,7 @@ main ()
 
         \`source jmvenv/bin/activate\`
 
-        from this directory, to activate virtualenv."
+        from this directory, to activate the virtual environment."
     fi
 }
 main ${@}

jmclient/jmclient/wallet_utils.py

@@ -1095,7 +1095,7 @@ def f(r, deposits, deposit_times, now, final_balance):
             jmprint('(as if yield generator was a bank account)')
         except ImportError:
             jmprint('scipy not installed, unable to predict accumulation rate')
-            jmprint('to add it to this virtualenv, use `pip install scipy`')
+            jmprint('to add it to this virtual environment, use `pip install scipy`')
 
     # includes disabled utxos in accounting:
     total_wallet_balance = sum(wallet.get_balance_by_mixdepth(

scripts/obwatch/ob-watcher.py

@@ -36,7 +36,7 @@
     import matplotlib
 except:
     log.warning("matplotlib not found, charts will not be available. "
-                "Do `pip install matplotlib` in the joinmarket virtualenv.")
+                "Do `pip install matplotlib` in the joinmarket virtual environment.")
 
 if 'matplotlib' in sys.modules:
     # https://stackoverflow.com/questions/2801882/generating-a-png-with-matplotlib-when-display-is-undefined

test/Dockerfiles/bionic-py3.Dockerfile

@@ -4,7 +4,7 @@ SHELL ["/bin/bash", "-c"]
 # dependencies
 RUN apt-get update
 RUN apt-get install -y \
-    python3-dev python3-pip virtualenv libsodium23
+    python3-dev python3-pip libsodium23 python3-venv
 
 # curl is a better tool
 RUN apt-get install -y curl
@@ -30,5 +30,5 @@ RUN bitcoind --version | head -1
 
 # install script
 WORKDIR ${repo_name}
-RUN virtualenv --python=python3 jmvenv
+RUN python3 -m venv jmvenv
 RUN source jmvenv/bin/activate && ./test/run_tests.sh

test/Dockerfiles/centos7-py3.Dockerfile

@@ -6,7 +6,7 @@ RUN yum -y groups install 'Development tools'
 RUN yum -y install epel-release && \
     yum -y update
 RUN yum -y install \
-    python3-devel python3-pip python3-virtualenv libsodium
+    python3-devel python3-pip libsodium
 
 RUN useradd --home-dir /home/chaum --create-home --shell /bin/bash --skel /etc/skel/ chaum
 ARG core_version
@@ -29,5 +29,5 @@ RUN bitcoind --version | head -1
 
 # install script
 WORKDIR ${repo_name}
-RUN virtualenv-3 --python=python3 jmvenv
+RUN python3 -m venv jmvenv
 RUN source jmvenv/bin/activate && ./test/run_tests.sh

test/Dockerfiles/fedora27-py3.Dockerfile

@@ -4,7 +4,7 @@ SHELL ["/bin/bash", "-c"]
 # dependencies
 RUN dnf -y groups install 'Development tools'
 RUN dnf -y install \
-    python3-devel python3-pip python3-virtualenv libsodium
+    python3-devel python3-pip libsodium
 
 # needed for build time
 # https://stackoverflow.com/questions/34624428/g-error-usr-lib-rpm-redhat-redhat-hardened-cc1-no-such-file-or-directory
@@ -31,5 +31,5 @@ RUN bitcoind --version | head -1
 
 # install script
 WORKDIR ${repo_name}
-RUN virtualenv-3 --python=python3 jmvenv
+RUN python3 -m venv jmvenv
 RUN source jmvenv/bin/activate && ./test/run_tests.sh

test/Dockerfiles/stretch-py3.Dockerfile

@@ -7,7 +7,7 @@ RUN apt-get install -y build-essential
 RUN apt-get install -y \
     automake pkg-config libtool
 RUN apt-get install -y \
-    python3-dev python3-pip virtualenv python3-pyqt4 python3-sip
+    python3-dev python3-pip python3-pyqt4 python3-sip python3-venv
 
 # curl is a better tool
 RUN apt-get install -y curl

test/Dockerfiles/xenial-py3.Dockerfile

@@ -7,7 +7,7 @@ RUN apt-get install -y build-essential
 RUN apt-get install -y \
     automake pkg-config libtool
 RUN apt-get install -y \
-    python3-dev python3-pip virtualenv python3-pyqt4 python3-sip
+    python3-dev python3-pip python3-pyqt4 python3-sip python3-venv
 
 # curl is a better tool
 RUN apt-get install -y curl

test/run_tests.sh

@@ -165,7 +165,7 @@ run_jm_tests ()
     fi
 
     if [[ -z "${VIRTUAL_ENV}" ]]; then
-        echo "Source JM virtualenv before running tests:
+        echo "Source JM virtual environment before running tests:
 
         \`source ./jmvenv/bin/activate\`"
         return 1