Merge for 2.12.0 release

.gitmodules

@@ -8,3 +8,6 @@
 	path = modules/pmi1
 	url = https://github.com/salilab/pmi.git
 	branch = legacy-pmi1
+[submodule "modules/bayesianem"]
+	path = modules/bayesianem
+	url = https://gitlab.pasteur.fr/rpellari/bayesianem.git

ChangeLog.md

@@ -1,6 +1,63 @@
 ChangeLog {#changelog}
 =========
 
+# 2.12.0 - 2019-12-06 # {#changelog_2_12_0}
+- The Windows .exe installer now supports Python 3.8, and has dropped support
+  for Python 3.4.
+- Packages are no longer provided for Ubuntu 14.04 (Trusty Tahr) since
+  it has reached end of life.
+- The IMP distribution now includes the IMP::bayesianem module developed at
+  Institut Pasteur, Paris, for Bayesian handling of cryo-electron microscopy
+  density map data. See Bonomi et al. at https://doi.org/10.1101/113951 for
+  more information.
+- The old IMP::pmi::representation::Representation class has been removed
+  from IMP.pmi. New applications should use IMP::pmi::topology::System instead.
+- The IMP::pmi::restraints::crosslinking::ISDCrossLinkMS class for handling
+  crosslinking has been removed. Use
+  IMP::pmi::restraints::crosslinking::CrossLinkingMassSpectrometryRestraint
+  instead.
+- The `rg` tool (part of the IMP::saxs module, used to compute radius of
+  gyration from a SAXS profile) is now called `compute_rg` for consistency
+  with other SAXS tools and to avoid conflicts with other packages.
+- We now provide RPM packages for RedHat Enterprise Linux 8 (or compatible
+  operating systems such as CentOS 8).
+- The RPM packages for RedHat Enterprise Linux 8 and for Fedora now use
+  Python 3 by default. If you need Python 2, install the IMP-python2 package
+  as well.
+- IMP::algebra::Rotation3D::get_derivative(),
+  IMP::algebra::Rotation3D::get_gradient(),
+  IMP::algebra::get_gradient_of_composed_with_respect_to_first(), and
+  IMP::algebra::get_gradient_of_composed_with_respect_to_second() have been
+  deprecated and are superseded by
+  IMP::algebra::Rotation3D::get_gradient_of_rotated(),
+  IMP::algebra::Rotation3D::get_jacobian_of_rotated(),
+  IMP::algebra::get_jacobian_of_composed_wrt_first(), and
+  IMP::algebra::get_jacobian_of_composed_wrt_second(), respectively. By
+  default, the derivatives are now computed with respect to the unnormalized
+  quaternion and do not include the normalization operation.
+- New methods are added to compute adjoint derivatives (reverse-mode
+  sensitivities) for compositions and actions of IMP::algebra::Rotation3D and
+  IMP::algebra::Transformation3D upon 3D vectors.
+- Fixed a bug in nested rigid body derivative accumulation, where derivatives
+  with respect to quaternions were incorrectly projected to be orthogonal to
+  the quaternion.
+- Reimplemented rigid body derivative accumulation to use the new adjoint
+  methods. The many-argument versions of
+  IMP::core::RigidBody::add_to_derivatives(),
+  IMP::core::RigidBody::add_to_rotational_derivatives(), and
+  IMP::core::NonRigidMember::add_to_internal_rotational_derivatives(), which
+  previously pulled adjoints from member global reference frame to member
+  local reference frame and parent global reference frame are now deprecated.
+  Pullback functionality is now handled by
+  IMP::core::RigidBody::pull_back_members_adjoints().
+- IMP::isd::Weight is now constrained to the unit simplex, and methods were
+  added for adding to its derivatives. IMP::isd::Weight::add_weight() no longer
+  resets all the weights to the barycenter of the unit simplex and instead
+  initializes the new weight to 0. IMP::isd::Weight::get_number_of_states()
+  and IMP::isd::Weight::get_nstates_key() were deprecated and superseded by
+  IMP::isd::Weight::get_number_of_weights() and
+  IMP::isd::Weight::get_number_of_weights_key(), respectively.
+
 # 2.11.1 - 2019-07-18 # {#changelog_2_11_1}
 - Bugfix: fix build system failures with CMake 3.12 and 3.13, and on Windows.
 - Bugfix: IMP::atom::create_clone() now always copies mass, even of particles
@@ -31,7 +88,7 @@ ChangeLog {#changelog}
 
 # 2.10.1 - 2019-02-26 # {#changelog_2_10_1}
 - Add support for OpenCV 4.
-- Fix IMP::isd `create_gmm.py` script to handle command line options correctly. 
+- Fix IMP::isd `create_gmm.py` script to handle command line options correctly.
 - Command line tools in the Mac and Ubuntu packages should now use system
   Python (/usr/bin/python), not the first Python (e.g. Anaconda Python)
   found in PATH (which might not be compatible with IMP's Python libraries).

README.md

@@ -1,5 +1,7 @@
 [![nightly build](https://integrativemodeling.org/nightly/results/?p=stat)](https://integrativemodeling.org/nightly/results/)
 [![coverity scan](https://img.shields.io/coverity/scan/8505.svg)](https://scan.coverity.com/projects/salilab-imp)
+[![DOI](https://zenodo.org/badge/DOI/10.5281/zenodo.3256565.svg)](https://doi.org/10.5281/zenodo.3256565)
+
 
 For full installation and usage instructions, see the
 [documentation](https://integrativemodeling.org/nightly/doc/manual/).

VERSION

@@ -1 +1 @@
-2.11.1
+2.12.0

doc/manual/cmake_config.md

@@ -71,7 +71,7 @@ which control the build. For example:
 ## Python binary/header mismatch {#cmake_python}
 
 In order to build %IMP Python extensions, CMake needs to find the Python header
-and library files that match the `python3` or `python` binary. If using a
+and library files that match the `python3`, `python2` or `python` binary. If using a
 recent version of CMake (3.14 or later) it should have no issues in doing so.
 However, old versions of CMake might get confused if you have multiple versions
 of Python installed (for example on a Mac with [Homebrew](https://brew.sh/)),

doc/manual/installation.md

@@ -114,7 +114,7 @@ procedure we use.
   with something like:
 
         git clone -b master https://github.com/salilab/imp.git
-        (cd imp && ./setup_git.py)
+        (cd imp && git submodule update --init && ./setup_git.py)
 
   (the `master` branch tracks the most recent stable
   release; alternatively you can use `develop` to get the most recent code,

modules/algebra/include/Rotation3D.h

@@ -22,6 +22,11 @@
 
 IMPALGEBRA_BEGIN_NAMESPACE
 
+typedef Vector4D Rotation3DAdjoint;
+typedef std::pair<Vector3D, Rotation3DAdjoint> RotatedVector3DAdjoint;
+typedef std::pair<Rotation3DAdjoint, Rotation3DAdjoint>
+    ComposeRotation3DAdjoint;
+
 #if !defined(IMP_DOXYGEN) && !defined(SWIG)
 class Rotation3D;
 Rotation3D compose(const Rotation3D &a, const Rotation3D &b);
@@ -175,6 +180,24 @@ class IMPALGEBRAEXPORT Rotation3D : public GeometricPrimitiveD<3> {
     return Vector3D(o * matrix_[0], o * matrix_[1], o * matrix_[2]);
   }
 
+#ifndef SWIG
+  //! Get adjoint of inputs to `get_rotated` from adjoint of output
+  /** Compute the adjoint (reverse-mode sensitivity) of input vector
+      to `get_rotated` and this rotation from the adjoint of the
+      output vector.
+   */
+  void get_rotated_adjoint(const Vector3D &v, const Vector3D &Dw,
+                           Vector3D *Dv, Rotation3DAdjoint *Dr) const;
+#endif
+
+  //! Get adjoint of inputs to `get_rotated` from adjoint of output
+  /** Compute the adjoint (reverse-mode sensitivity) of input vector
+      to `get_rotated` and this rotation from the adjoint of the
+      output vector.
+   */
+  RotatedVector3DAdjoint
+  get_rotated_adjoint(const Vector3D &v, const Vector3D &Dw) const;
+
   //! Get only the requested rotation coordinate of the vector
   double get_rotated_one_coordinate(const Vector3D &o,
                                     unsigned int coord) const {
@@ -229,36 +252,46 @@ class IMPALGEBRAEXPORT Rotation3D : public GeometricPrimitiveD<3> {
     return *this;
   }
 
-  //! Return the derivative of rotated vector wrt the ith quaternion element.
+  //! Return the gradient of rotated vector wrt the ith quaternion element.
   /** Given the rotation \f$x = R(q) v\f$, where \f$v\f$ is a vector,
       \f$q=(q_0,q_1,q_2,q_3)\f$ is the quaternion of the rotation with
       elements \f$q_i\f$, and \f$R(q)\f$ is the corresponding rotation matrix,
-      the function returns the derivative \f$\frac{d x}{d q_i}\f$.
+      the function returns the gradient \f$\nabla_{q_i} x\f$.
 
-      This function just returns a single column from get_gradient(), so it is
+      This function just returns a single column from `get_jacobian()`, so it is
       more efficient to call that function if all columns are needed.
 
       \param[in] v vector to be rotated by rotation \f$R(q)\f$
-      \param[in] projected project derivative onto tangent space to \f$q\f$.
-                           Equivalent to differentiating wrt
-                           \f$\frac{q_i}{\|q\|}\f$ instead of \f$q_i\f$.
+      \param[in] wrt_unnorm Gradient is computed wrt unnormalized quaternion.
+                            Rotation includes a normalization operation, and
+                            the gradient is projected to the tangent space at
+                            \f$q\f$.
   */
+  Vector3D get_gradient_of_rotated(const Vector3D &v, unsigned int i,
+                                   bool wrt_unnorm = false) const;
+
+  IMPALGEBRA_DEPRECATED_METHOD_DECL(2.12)
   Vector3D get_derivative(const Vector3D &v, unsigned int i,
-                          bool projected = true) const;
+                          bool wrt_unnorm = true) const;
 
-  //! Return the gradient of rotated vector wrt the quaternion.
+  //! Return the Jacobian of rotated vector wrt the quaternion.
   /** Given the rotation \f$x = R(q) v\f$, where \f$v\f$ is a vector,
       \f$q\f$ is the quaternion of the rotation, and \f$R(q)\f$ is the
       corresponding rotation matrix, the function returns the 3x4 matrix
-      \f$M = \nabla_q x\f$ with elements \f$M_{ij}=\frac{d x_i}{d q_j}\f$.
+      \f$J\f$ with elements \f$J_{ij}=\frac{\partial x_i}{\partial q_j}\f$.
       
       \param[in] v vector to be rotated by rotation \f$R(q)\f$
-      \param[in] projected project gradient onto tangent space to \f$q\f$.
-                           Equivalent to differentiating wrt
-                           \f$\frac{q_i}{\|q\|}\f$ instead of \f$q_i\f$.
+      \param[in] wrt_unnorm Jacobian is computed wrt unnormalized quaternion.
+                            Rotation includes a normalization operation, and
+                            the columns are projected to the tangent space at
+                            \f$q\f$.
    */
+  Eigen::MatrixXd get_jacobian_of_rotated(
+    const Eigen::Vector3d &v, bool wrt_unnorm = false) const;
+
+  IMPALGEBRA_DEPRECATED_METHOD_DECL(2.12)
   Eigen::MatrixXd get_gradient(
-    const Eigen::Vector3d &v, bool projected = true) const;
+    const Eigen::Vector3d &v, bool wrt_unnorm = true) const;
 
   /** Return true is the rotation is valid, false if
       invalid or not initialized (e.g., only initialized by
@@ -277,32 +310,46 @@ IMP_VALUES(Rotation3D, Rotation3Ds);
     \f$q\f$ are quaternions, the quaternion of the composed rotation
     \f$R(s)=R(q) R(p)\f$ can be expressed through the Hamilton product of the
     two quaternions \f$s(q,p) = q p\f$. This function returns the matrix
-    \f$M = \nabla_q s(q, p)\f$ with elements \f$M_{ij}=\frac{d s_i}{d q_j}\f$.
+    \f$J\f$ with elements \f$J_{ij}=\frac{\partial s_i}{\partial q_j}\f$.
 
     \param[in] q rotation corresponding to first quaternion
     \param[in] p rotation corresponding to second quaternion
-    \param[in] projected project derivative onto tangent space to \f$q\f$.
-                         Equivalent to differentiating wrt
-                         \f$\frac{q_i}{\|q\|}\f$ instead of \f$q_i\f$.
+    \param[in] wrt_unnorm Jacobian is computed wrt unnormalized quaternion.
+                          Rotation includes a normalization operation, and
+                          the columns are projected to the tangent space at
+                          \f$q\f$.
  */
-IMPALGEBRAEXPORT Eigen::MatrixXd get_gradient_of_composed_with_respect_to_first(
-  const Rotation3D &q, const Rotation3D &p, bool projected = true);
+IMPALGEBRAEXPORT Eigen::MatrixXd get_jacobian_of_composed_wrt_first(
+  const Rotation3D &q, const Rotation3D &p, bool wrt_unnorm = false);
+
+IMPALGEBRA_DEPRECATED_FUNCTION_DECL(2.12)
+IMPALGEBRAEXPORT Eigen::MatrixXd
+  get_gradient_of_composed_with_respect_to_first(
+    const Rotation3D &q, const Rotation3D &p, bool wrt_unnorm = true);
 
-//! Get gradient of quaternion product with respect to second quaternion.
+
+//! Get Jacobian of quaternion product with respect to second quaternion.
 /** Given the rotation \f$R(p)\f$ followed by \f$R(q)\f$, where \f$p\f$ and
     \f$q\f$ are quaternions, the quaternion of the composed rotation
     \f$R(s)=R(q) R(p)\f$ can be expressed through the Hamilton product of the
     two quaternions \f$s(q,p) = q p\f$. This function returns the matrix
-    \f$\nabla_p s(q, p)\f$ with elements \f$M_{ij}=\frac{d s_i}{d p_j}\f$.
+    \f$J\f$ with elements \f$J_{ij}=\frac{\partial s_i}{\partial p_j}\f$.
 
     \param[in] q rotation corresponding to first quaternion
     \param[in] p rotation corresponding to second quaternion
-    \param[in] projected project derivative onto tangent space to \f$p\f$.
-                         Equivalent to differentiating wrt
-                         \f$\frac{p_i}{\|p\|}\f$ instead of \f$p_i\f$.
+    \param[in] wrt_unnorm Jacobian is computed wrt unnormalized quaternion.
+                          Rotation includes a normalization operation, and
+                          the columns are projected to the tangent space at
+                          \f$p\f$.
  */
-IMPALGEBRAEXPORT Eigen::MatrixXd get_gradient_of_composed_with_respect_to_second(
-  const Rotation3D &q, const Rotation3D &p, bool projected = true);
+IMPALGEBRAEXPORT Eigen::MatrixXd get_jacobian_of_composed_wrt_second(
+  const Rotation3D &q, const Rotation3D &p, bool wrt_unnorm = false);
+
+IMPALGEBRA_DEPRECATED_FUNCTION_DECL(2.12)
+IMPALGEBRAEXPORT Eigen::MatrixXd
+  get_gradient_of_composed_with_respect_to_second(
+    const Rotation3D &q, const Rotation3D &p, bool wrt_unnorm = true);
+
 
 //! Return a rotation that does not do anything
 /** \see Rotation3D */
@@ -456,6 +503,23 @@ inline Rotation3D compose(const Rotation3D &a, const Rotation3D &b) {
                         a.v_[3] * b.v_[0]);
 }
 
+#ifndef SWIG
+//! Get adjoint of inputs to `compose` from adjoint of output
+/** Compute the adjoint (reverse-mode sensitivity) of input rotations
+    to `compose` from the adjoint of the output rotation.
+ */
+IMPALGEBRAEXPORT void
+compose_adjoint(const Rotation3D &A, const Rotation3D &B, Vector4D DC,
+                Rotation3DAdjoint *DA, Rotation3DAdjoint *DB);
+#endif
+
+//! Get adjoint of inputs to `compose` from adjoint of output
+/** Compute the adjoint (reverse-mode sensitivity) of input rotations
+    to `compose` from the adjoint of the output rotation.
+ */
+IMPALGEBRAEXPORT ComposeRotation3DAdjoint
+compose_adjoint(const Rotation3D &A, const Rotation3D &B, const Rotation3DAdjoint &DC);
+
 /** \name Euler Angles
     There are many conventions for how to define Euler angles, based on choices
     of which of the x,y,z axis to use in what order and whether the rotation

modules/algebra/include/Transformation3D.h

@@ -23,6 +23,11 @@ class Transformation3D;
 Transformation3D compose(const Transformation3D &a, const Transformation3D &b);
 #endif
 
+typedef std::pair<Vector4D, Vector3D> Transformation3DAdjoint;
+typedef std::pair<Vector3D, Transformation3DAdjoint> TransformedVector3DAdjoint;
+typedef std::pair<Transformation3DAdjoint, Transformation3DAdjoint>
+    ComposeTransformation3DAdjoint;
+
 //! Simple 3D transformation class
 /** The rotation is applied first, and then the point is translated.
     \see IMP::core::Transform
@@ -44,6 +49,25 @@ class IMPALGEBRAEXPORT Transformation3D : public GeometricPrimitiveD<3> {
   Vector3D get_transformed(const Vector3D &o) const {
     return rot_.get_rotated(o) + trans_;
   }
+
+#ifndef SWIG
+  //! Get adjoint of inputs to `get_transformed` from adjoint of output
+  /** Compute the adjoint (reverse-mode sensitivity) of input vector
+      to `get_transformed` and this transformation from the adjoint of the
+      output vector.
+   */
+  void get_transformed_adjoint(const Vector3D &v, const Vector3D &Dw,
+                               Vector3D *Dv, Transformation3DAdjoint *DT) const;
+#endif
+
+  //! Get adjoint of inputs to `get_transformed` from adjoint of output
+  /** Compute the adjoint (reverse-mode sensitivity) of input vector
+      to `get_transformed` and this transformation from the adjoint of the
+      output vector.
+   */
+  TransformedVector3DAdjoint
+  get_transformed_adjoint(const Vector3D &v, const Vector3D &Dw) const;
+
   //! Apply transformation (rotate and then translate)
   Vector3D operator*(const Vector3D &v) const { return get_transformed(v); }
   /** Compose two rigid transformation such that for any vector v
@@ -127,6 +151,25 @@ inline Transformation3D compose(const Transformation3D &a,
                           a.get_transformed(b.get_translation()));
 }
 
+#ifndef SWIG
+//! Get adjoint of inputs to `compose` from adjoint of output
+/** Compute the adjoint (reverse-mode sensitivity) of input transformations
+    to `compose` from the adjoint of the output transformation.
+ */
+IMPALGEBRAEXPORT void
+compose_adjoint(const Transformation3D &TA, const Transformation3D &TB,
+                const Transformation3DAdjoint &DTC,
+                Transformation3DAdjoint *DTA, Transformation3DAdjoint *DTB);
+#endif
+
+//! Get adjoint of inputs to `compose` from adjoint of output
+/** Compute the adjoint (reverse-mode sensitivity) of input transformations
+    to `compose` from the adjoint of the output transformation.
+ */
+IMPALGEBRAEXPORT ComposeTransformation3DAdjoint
+compose_adjoint(const Transformation3D &TA, const Transformation3D &TB,
+                const Transformation3DAdjoint &DTC);
+
 class Transformation2D;
 
 //! Build a 3D transformation from a 2D one.