install_via_pip.pl might downgrade already installed packages

[tkilias]

Background

• if you call pip multiple times with different packages and packages of subsequent calls require a different version of packages of previous calls, they get changed.
• the following article suggests using constraint files to solve this https://stackoverflow.com/questions/43121005/can-i-prevent-pip-from-downgrading-packages-implicitly

  pip freeze | grep == | sed 's/==/>=/' >constraints.txt
  pip install -c constraints.txt whatever-you-want-to-install
  

• Downgrading has two issues for us:
  1. We want to pin the exact version we deliver
  2. For some packages, this can render them unusable, e.g. numpy might not work properly afterward
  • we got, for example, the following error message: ImportError: numpy.core.multiarray failed to import
  • the likely reason for this issue, because we link against numpy

Acceptance Criteria

• Add constraint file mechanism to install_via_pip.pl
• Find a test case for this and add it

[tkilias]

To test the solution for this bug, we need to rework the test framework for the package management scripts #797. To accomplish this, it is probably the best to extract the scripts from this repository. #242

For this reason, this issue is blocked.

[tomuben]

The proposed solution (using the constraint files) has two problems:

1. If using | grep == | sed 's/==/>=/' as suggested, new version of transitive dependencies will be installed, even if it would break compatibility of an already installed package which needs an the old (pinned) version
2. If we would remove | grep == | sed 's/==/>=/' => we would require always the exact same version of all transitive dependencies to be installed. This could be possibly a problem for customers, for who it would not be obvious why they can't install a specific version

[tomuben]

Our new proposal is to pin only the packages which are defined in a previous build step, but ignore all other Python packages.
For that we would parse all pip package files in the image info folder on the docker image, and generate one (huge) requirements file, where all the version are pinned with ==. Transitive dependencies will be ignored.

[tomuben]

Here the required scenarios for testing.
Note:
azure-batch==1.0.0 depends on azure-common 1.1.4
azure-batch1.0.0 depends on azure-common1.1.4

Same Package different Version

Step 1: azure-common 1.1.4
Step 2: azure-common 1.1.28

=> Error

Dependency with correct version already installed

Step 1: azure-common 1.1.4
Step 2: azure-batch==1.0.0

Success

Dependency with incorrect version already installed

Step 1: azure-common 1.1.28
Step 2: azure-batch==1.0.0

Error

##Two packages with same dependency but different version and second step needs newer version of dependency
Step 1: azure-batch1.0.0 -> azure-common1.1.4
Step 2: azure-storage-queue==1.1.0 -> azure-common>=1.1.28

Error

Two packages with same dependency but different version and second step needs older version of dependency

Step 1: azure-storage-queue==1.1.0 -> azure-common>=1.1.28
Step 2: azure-batch1.0.0 -> azure-common1.1.4

Error