diff --git a/.gitmodules b/.gitmodules index e6a109a536..c2703274d7 100644 --- a/.gitmodules +++ b/.gitmodules @@ -8,3 +8,6 @@ path = modules/pmi1 url = https://github.com/salilab/pmi.git branch = legacy-pmi1 +[submodule "modules/bayesianem"] + path = modules/bayesianem + url = https://gitlab.pasteur.fr/rpellari/bayesianem.git diff --git a/ChangeLog.md b/ChangeLog.md index ab50d2a81b..0323285b53 100644 --- a/ChangeLog.md +++ b/ChangeLog.md @@ -1,6 +1,63 @@ ChangeLog {#changelog} ========= +# 2.12.0 - 2019-12-06 # {#changelog_2_12_0} +- The Windows .exe installer now supports Python 3.8, and has dropped support + for Python 3.4. +- Packages are no longer provided for Ubuntu 14.04 (Trusty Tahr) since + it has reached end of life. +- The IMP distribution now includes the IMP::bayesianem module developed at + Institut Pasteur, Paris, for Bayesian handling of cryo-electron microscopy + density map data. See Bonomi et al. at https://doi.org/10.1101/113951 for + more information. +- The old IMP::pmi::representation::Representation class has been removed + from IMP.pmi. New applications should use IMP::pmi::topology::System instead. +- The IMP::pmi::restraints::crosslinking::ISDCrossLinkMS class for handling + crosslinking has been removed. Use + IMP::pmi::restraints::crosslinking::CrossLinkingMassSpectrometryRestraint + instead. +- The `rg` tool (part of the IMP::saxs module, used to compute radius of + gyration from a SAXS profile) is now called `compute_rg` for consistency + with other SAXS tools and to avoid conflicts with other packages. +- We now provide RPM packages for RedHat Enterprise Linux 8 (or compatible + operating systems such as CentOS 8). +- The RPM packages for RedHat Enterprise Linux 8 and for Fedora now use + Python 3 by default. If you need Python 2, install the IMP-python2 package + as well. +- IMP::algebra::Rotation3D::get_derivative(), + IMP::algebra::Rotation3D::get_gradient(), + IMP::algebra::get_gradient_of_composed_with_respect_to_first(), and + IMP::algebra::get_gradient_of_composed_with_respect_to_second() have been + deprecated and are superseded by + IMP::algebra::Rotation3D::get_gradient_of_rotated(), + IMP::algebra::Rotation3D::get_jacobian_of_rotated(), + IMP::algebra::get_jacobian_of_composed_wrt_first(), and + IMP::algebra::get_jacobian_of_composed_wrt_second(), respectively. By + default, the derivatives are now computed with respect to the unnormalized + quaternion and do not include the normalization operation. +- New methods are added to compute adjoint derivatives (reverse-mode + sensitivities) for compositions and actions of IMP::algebra::Rotation3D and + IMP::algebra::Transformation3D upon 3D vectors. +- Fixed a bug in nested rigid body derivative accumulation, where derivatives + with respect to quaternions were incorrectly projected to be orthogonal to + the quaternion. +- Reimplemented rigid body derivative accumulation to use the new adjoint + methods. The many-argument versions of + IMP::core::RigidBody::add_to_derivatives(), + IMP::core::RigidBody::add_to_rotational_derivatives(), and + IMP::core::NonRigidMember::add_to_internal_rotational_derivatives(), which + previously pulled adjoints from member global reference frame to member + local reference frame and parent global reference frame are now deprecated. + Pullback functionality is now handled by + IMP::core::RigidBody::pull_back_members_adjoints(). +- IMP::isd::Weight is now constrained to the unit simplex, and methods were + added for adding to its derivatives. IMP::isd::Weight::add_weight() no longer + resets all the weights to the barycenter of the unit simplex and instead + initializes the new weight to 0. IMP::isd::Weight::get_number_of_states() + and IMP::isd::Weight::get_nstates_key() were deprecated and superseded by + IMP::isd::Weight::get_number_of_weights() and + IMP::isd::Weight::get_number_of_weights_key(), respectively. + # 2.11.1 - 2019-07-18 # {#changelog_2_11_1} - Bugfix: fix build system failures with CMake 3.12 and 3.13, and on Windows. - Bugfix: IMP::atom::create_clone() now always copies mass, even of particles @@ -31,7 +88,7 @@ ChangeLog {#changelog} # 2.10.1 - 2019-02-26 # {#changelog_2_10_1} - Add support for OpenCV 4. -- Fix IMP::isd `create_gmm.py` script to handle command line options correctly. +- Fix IMP::isd `create_gmm.py` script to handle command line options correctly. - Command line tools in the Mac and Ubuntu packages should now use system Python (/usr/bin/python), not the first Python (e.g. Anaconda Python) found in PATH (which might not be compatible with IMP's Python libraries). diff --git a/README.md b/README.md index c95af92d7e..fa11fef053 100644 --- a/README.md +++ b/README.md @@ -1,5 +1,7 @@ [![nightly build](https://integrativemodeling.org/nightly/results/?p=stat)](https://integrativemodeling.org/nightly/results/) [![coverity scan](https://img.shields.io/coverity/scan/8505.svg)](https://scan.coverity.com/projects/salilab-imp) +[![DOI](https://zenodo.org/badge/DOI/10.5281/zenodo.3256565.svg)](https://doi.org/10.5281/zenodo.3256565) + For full installation and usage instructions, see the [documentation](https://integrativemodeling.org/nightly/doc/manual/). diff --git a/VERSION b/VERSION index 6ceb272eec..d8b698973a 100644 --- a/VERSION +++ b/VERSION @@ -1 +1 @@ -2.11.1 +2.12.0 diff --git a/doc/manual/cmake_config.md b/doc/manual/cmake_config.md index 3d6733fed8..78751f071e 100644 --- a/doc/manual/cmake_config.md +++ b/doc/manual/cmake_config.md @@ -71,7 +71,7 @@ which control the build. For example: ## Python binary/header mismatch {#cmake_python} In order to build %IMP Python extensions, CMake needs to find the Python header -and library files that match the `python3` or `python` binary. If using a +and library files that match the `python3`, `python2` or `python` binary. If using a recent version of CMake (3.14 or later) it should have no issues in doing so. However, old versions of CMake might get confused if you have multiple versions of Python installed (for example on a Mac with [Homebrew](https://brew.sh/)), diff --git a/doc/manual/installation.md b/doc/manual/installation.md index 9399a76f54..81162bf74e 100644 --- a/doc/manual/installation.md +++ b/doc/manual/installation.md @@ -114,7 +114,7 @@ procedure we use. with something like: git clone -b master https://github.com/salilab/imp.git - (cd imp && ./setup_git.py) + (cd imp && git submodule update --init && ./setup_git.py) (the `master` branch tracks the most recent stable release; alternatively you can use `develop` to get the most recent code, diff --git a/modules/algebra/include/Rotation3D.h b/modules/algebra/include/Rotation3D.h index 233aefbbad..4e835aee1a 100644 --- a/modules/algebra/include/Rotation3D.h +++ b/modules/algebra/include/Rotation3D.h @@ -22,6 +22,11 @@ IMPALGEBRA_BEGIN_NAMESPACE +typedef Vector4D Rotation3DAdjoint; +typedef std::pair RotatedVector3DAdjoint; +typedef std::pair + ComposeRotation3DAdjoint; + #if !defined(IMP_DOXYGEN) && !defined(SWIG) class Rotation3D; Rotation3D compose(const Rotation3D &a, const Rotation3D &b); @@ -175,6 +180,24 @@ class IMPALGEBRAEXPORT Rotation3D : public GeometricPrimitiveD<3> { return Vector3D(o * matrix_[0], o * matrix_[1], o * matrix_[2]); } +#ifndef SWIG + //! Get adjoint of inputs to `get_rotated` from adjoint of output + /** Compute the adjoint (reverse-mode sensitivity) of input vector + to `get_rotated` and this rotation from the adjoint of the + output vector. + */ + void get_rotated_adjoint(const Vector3D &v, const Vector3D &Dw, + Vector3D *Dv, Rotation3DAdjoint *Dr) const; +#endif + + //! Get adjoint of inputs to `get_rotated` from adjoint of output + /** Compute the adjoint (reverse-mode sensitivity) of input vector + to `get_rotated` and this rotation from the adjoint of the + output vector. + */ + RotatedVector3DAdjoint + get_rotated_adjoint(const Vector3D &v, const Vector3D &Dw) const; + //! Get only the requested rotation coordinate of the vector double get_rotated_one_coordinate(const Vector3D &o, unsigned int coord) const { @@ -229,36 +252,46 @@ class IMPALGEBRAEXPORT Rotation3D : public GeometricPrimitiveD<3> { return *this; } - //! Return the derivative of rotated vector wrt the ith quaternion element. + //! Return the gradient of rotated vector wrt the ith quaternion element. /** Given the rotation \f$x = R(q) v\f$, where \f$v\f$ is a vector, \f$q=(q_0,q_1,q_2,q_3)\f$ is the quaternion of the rotation with elements \f$q_i\f$, and \f$R(q)\f$ is the corresponding rotation matrix, - the function returns the derivative \f$\frac{d x}{d q_i}\f$. + the function returns the gradient \f$\nabla_{q_i} x\f$. - This function just returns a single column from get_gradient(), so it is + This function just returns a single column from `get_jacobian()`, so it is more efficient to call that function if all columns are needed. \param[in] v vector to be rotated by rotation \f$R(q)\f$ - \param[in] projected project derivative onto tangent space to \f$q\f$. - Equivalent to differentiating wrt - \f$\frac{q_i}{\|q\|}\f$ instead of \f$q_i\f$. + \param[in] wrt_unnorm Gradient is computed wrt unnormalized quaternion. + Rotation includes a normalization operation, and + the gradient is projected to the tangent space at + \f$q\f$. */ + Vector3D get_gradient_of_rotated(const Vector3D &v, unsigned int i, + bool wrt_unnorm = false) const; + + IMPALGEBRA_DEPRECATED_METHOD_DECL(2.12) Vector3D get_derivative(const Vector3D &v, unsigned int i, - bool projected = true) const; + bool wrt_unnorm = true) const; - //! Return the gradient of rotated vector wrt the quaternion. + //! Return the Jacobian of rotated vector wrt the quaternion. /** Given the rotation \f$x = R(q) v\f$, where \f$v\f$ is a vector, \f$q\f$ is the quaternion of the rotation, and \f$R(q)\f$ is the corresponding rotation matrix, the function returns the 3x4 matrix - \f$M = \nabla_q x\f$ with elements \f$M_{ij}=\frac{d x_i}{d q_j}\f$. + \f$J\f$ with elements \f$J_{ij}=\frac{\partial x_i}{\partial q_j}\f$. \param[in] v vector to be rotated by rotation \f$R(q)\f$ - \param[in] projected project gradient onto tangent space to \f$q\f$. - Equivalent to differentiating wrt - \f$\frac{q_i}{\|q\|}\f$ instead of \f$q_i\f$. + \param[in] wrt_unnorm Jacobian is computed wrt unnormalized quaternion. + Rotation includes a normalization operation, and + the columns are projected to the tangent space at + \f$q\f$. */ + Eigen::MatrixXd get_jacobian_of_rotated( + const Eigen::Vector3d &v, bool wrt_unnorm = false) const; + + IMPALGEBRA_DEPRECATED_METHOD_DECL(2.12) Eigen::MatrixXd get_gradient( - const Eigen::Vector3d &v, bool projected = true) const; + const Eigen::Vector3d &v, bool wrt_unnorm = true) const; /** Return true is the rotation is valid, false if invalid or not initialized (e.g., only initialized by @@ -277,32 +310,46 @@ IMP_VALUES(Rotation3D, Rotation3Ds); \f$q\f$ are quaternions, the quaternion of the composed rotation \f$R(s)=R(q) R(p)\f$ can be expressed through the Hamilton product of the two quaternions \f$s(q,p) = q p\f$. This function returns the matrix - \f$M = \nabla_q s(q, p)\f$ with elements \f$M_{ij}=\frac{d s_i}{d q_j}\f$. + \f$J\f$ with elements \f$J_{ij}=\frac{\partial s_i}{\partial q_j}\f$. \param[in] q rotation corresponding to first quaternion \param[in] p rotation corresponding to second quaternion - \param[in] projected project derivative onto tangent space to \f$q\f$. - Equivalent to differentiating wrt - \f$\frac{q_i}{\|q\|}\f$ instead of \f$q_i\f$. + \param[in] wrt_unnorm Jacobian is computed wrt unnormalized quaternion. + Rotation includes a normalization operation, and + the columns are projected to the tangent space at + \f$q\f$. */ -IMPALGEBRAEXPORT Eigen::MatrixXd get_gradient_of_composed_with_respect_to_first( - const Rotation3D &q, const Rotation3D &p, bool projected = true); +IMPALGEBRAEXPORT Eigen::MatrixXd get_jacobian_of_composed_wrt_first( + const Rotation3D &q, const Rotation3D &p, bool wrt_unnorm = false); + +IMPALGEBRA_DEPRECATED_FUNCTION_DECL(2.12) +IMPALGEBRAEXPORT Eigen::MatrixXd + get_gradient_of_composed_with_respect_to_first( + const Rotation3D &q, const Rotation3D &p, bool wrt_unnorm = true); -//! Get gradient of quaternion product with respect to second quaternion. + +//! Get Jacobian of quaternion product with respect to second quaternion. /** Given the rotation \f$R(p)\f$ followed by \f$R(q)\f$, where \f$p\f$ and \f$q\f$ are quaternions, the quaternion of the composed rotation \f$R(s)=R(q) R(p)\f$ can be expressed through the Hamilton product of the two quaternions \f$s(q,p) = q p\f$. This function returns the matrix - \f$\nabla_p s(q, p)\f$ with elements \f$M_{ij}=\frac{d s_i}{d p_j}\f$. + \f$J\f$ with elements \f$J_{ij}=\frac{\partial s_i}{\partial p_j}\f$. \param[in] q rotation corresponding to first quaternion \param[in] p rotation corresponding to second quaternion - \param[in] projected project derivative onto tangent space to \f$p\f$. - Equivalent to differentiating wrt - \f$\frac{p_i}{\|p\|}\f$ instead of \f$p_i\f$. + \param[in] wrt_unnorm Jacobian is computed wrt unnormalized quaternion. + Rotation includes a normalization operation, and + the columns are projected to the tangent space at + \f$p\f$. */ -IMPALGEBRAEXPORT Eigen::MatrixXd get_gradient_of_composed_with_respect_to_second( - const Rotation3D &q, const Rotation3D &p, bool projected = true); +IMPALGEBRAEXPORT Eigen::MatrixXd get_jacobian_of_composed_wrt_second( + const Rotation3D &q, const Rotation3D &p, bool wrt_unnorm = false); + +IMPALGEBRA_DEPRECATED_FUNCTION_DECL(2.12) +IMPALGEBRAEXPORT Eigen::MatrixXd + get_gradient_of_composed_with_respect_to_second( + const Rotation3D &q, const Rotation3D &p, bool wrt_unnorm = true); + //! Return a rotation that does not do anything /** \see Rotation3D */ @@ -456,6 +503,23 @@ inline Rotation3D compose(const Rotation3D &a, const Rotation3D &b) { a.v_[3] * b.v_[0]); } +#ifndef SWIG +//! Get adjoint of inputs to `compose` from adjoint of output +/** Compute the adjoint (reverse-mode sensitivity) of input rotations + to `compose` from the adjoint of the output rotation. + */ +IMPALGEBRAEXPORT void +compose_adjoint(const Rotation3D &A, const Rotation3D &B, Vector4D DC, + Rotation3DAdjoint *DA, Rotation3DAdjoint *DB); +#endif + +//! Get adjoint of inputs to `compose` from adjoint of output +/** Compute the adjoint (reverse-mode sensitivity) of input rotations + to `compose` from the adjoint of the output rotation. + */ +IMPALGEBRAEXPORT ComposeRotation3DAdjoint +compose_adjoint(const Rotation3D &A, const Rotation3D &B, const Rotation3DAdjoint &DC); + /** \name Euler Angles There are many conventions for how to define Euler angles, based on choices of which of the x,y,z axis to use in what order and whether the rotation diff --git a/modules/algebra/include/Transformation3D.h b/modules/algebra/include/Transformation3D.h index 37b3105da6..d0c30ab679 100644 --- a/modules/algebra/include/Transformation3D.h +++ b/modules/algebra/include/Transformation3D.h @@ -23,6 +23,11 @@ class Transformation3D; Transformation3D compose(const Transformation3D &a, const Transformation3D &b); #endif +typedef std::pair Transformation3DAdjoint; +typedef std::pair TransformedVector3DAdjoint; +typedef std::pair + ComposeTransformation3DAdjoint; + //! Simple 3D transformation class /** The rotation is applied first, and then the point is translated. \see IMP::core::Transform @@ -44,6 +49,25 @@ class IMPALGEBRAEXPORT Transformation3D : public GeometricPrimitiveD<3> { Vector3D get_transformed(const Vector3D &o) const { return rot_.get_rotated(o) + trans_; } + +#ifndef SWIG + //! Get adjoint of inputs to `get_transformed` from adjoint of output + /** Compute the adjoint (reverse-mode sensitivity) of input vector + to `get_transformed` and this transformation from the adjoint of the + output vector. + */ + void get_transformed_adjoint(const Vector3D &v, const Vector3D &Dw, + Vector3D *Dv, Transformation3DAdjoint *DT) const; +#endif + + //! Get adjoint of inputs to `get_transformed` from adjoint of output + /** Compute the adjoint (reverse-mode sensitivity) of input vector + to `get_transformed` and this transformation from the adjoint of the + output vector. + */ + TransformedVector3DAdjoint + get_transformed_adjoint(const Vector3D &v, const Vector3D &Dw) const; + //! Apply transformation (rotate and then translate) Vector3D operator*(const Vector3D &v) const { return get_transformed(v); } /** Compose two rigid transformation such that for any vector v @@ -127,6 +151,25 @@ inline Transformation3D compose(const Transformation3D &a, a.get_transformed(b.get_translation())); } +#ifndef SWIG +//! Get adjoint of inputs to `compose` from adjoint of output +/** Compute the adjoint (reverse-mode sensitivity) of input transformations + to `compose` from the adjoint of the output transformation. + */ +IMPALGEBRAEXPORT void +compose_adjoint(const Transformation3D &TA, const Transformation3D &TB, + const Transformation3DAdjoint &DTC, + Transformation3DAdjoint *DTA, Transformation3DAdjoint *DTB); +#endif + +//! Get adjoint of inputs to `compose` from adjoint of output +/** Compute the adjoint (reverse-mode sensitivity) of input transformations + to `compose` from the adjoint of the output transformation. + */ +IMPALGEBRAEXPORT ComposeTransformation3DAdjoint +compose_adjoint(const Transformation3D &TA, const Transformation3D &TB, + const Transformation3DAdjoint &DTC); + class Transformation2D; //! Build a 3D transformation from a 2D one. diff --git a/modules/algebra/include/UnitSimplexD.h b/modules/algebra/include/UnitSimplexD.h new file mode 100644 index 0000000000..f0b536e7e3 --- /dev/null +++ b/modules/algebra/include/UnitSimplexD.h @@ -0,0 +1,200 @@ +/** + * \file IMP/algebra/UnitSimplexD.h + * \brief Simple unit simplex class. + * + * Copyright 2007-2019 IMP Inventors. All rights reserved. + * + */ + +#ifndef IMPALGEBRA_UNIT_SIMPLEX_D_H +#define IMPALGEBRA_UNIT_SIMPLEX_D_H + +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include + +IMPALGEBRA_BEGIN_NAMESPACE + +//! Base class for a unit simplex embedded in D-dimensional real space. +/** See UnitSimplexD for more information. */ +template +class UnitSimplexBaseD : public GeometricPrimitiveD { + public: + //! Get dimension D of embedded real space. + virtual int get_dimension() const = 0; + + //! Get center of mass on simplex in embedded coordinates. + /** This is equivalent to the ones vector divided by D. */ + VectorD get_barycenter() const { + double ci = 1.0 / static_cast(get_dimension()); + return get_ones_vector_kd(get_dimension(), ci); + } + + //! Get whether the point is on the unit simplex + /** + \param[in] p Point + \param[in] atol Absolute tolerance of error for each element. Total + tolerance is atol*d. + */ + bool get_contains( + const VectorD &p, + double atol = std::numeric_limits::epsilon()) const { + int d = p.get_dimension(); + if (d != get_dimension()) return false; + double nrm = 0; + for (int i = 0; i < d; ++i) { + if (p[i] < 0) return false; + nrm += p[i]; + } + if (std::abs(nrm - 1) > d * atol) return false; + return true; + } +}; + +//! Represent a unit simplex embedded in D-dimensional real space. +/** The unit simplex (also known as standard simplex or probability simplex) + is the manifold of vectors of probabilities that sum to 1: + \f$ \Delta^{D-1} = \{ (t_1,\dots,t_D) \in \mathbb{R}^D \mid \sum_{d=1}^D t_d = 1, t_d \ge 0 \} \f$ + + \geometry + */ +template +class UnitSimplexD : public UnitSimplexBaseD { + public: + UnitSimplexD() { IMP_USAGE_CHECK(D > 0, "Dimension must be positive."); } + + //! Get dimension D of embedded real space. + int get_dimension() const IMP_OVERRIDE { return D; } + + IMP_SHOWABLE_INLINE(UnitSimplexD, { out << "UnitSimplex" << D << "D"; }); +}; + +//! Represent a unit simplex embedded in d-dimensional real space. +/** This version's dimension is not known at compile time and can operate with + variable-dimension vectors. + */ +template <> +class UnitSimplexD<-1> : public UnitSimplexBaseD<-1> { + public: + //! Construct from dimension d of embedded real space. + UnitSimplexD(int d = 1) : d_(d) { + IMP_USAGE_CHECK(d > 0, "Dimension must be positive."); + } + + //! Get dimension of embedded real space. + int get_dimension() const IMP_OVERRIDE { return d_; } + + // FIXME: SWIG doesn't seem to use this. + IMP_SHOWABLE_INLINE(UnitSimplexD<-1>, + { out << "UnitSimplexKD(" << d_ << ")"; }); + + private: + int d_; +}; + +typedef UnitSimplexD<1> UnitSimplex1D; +IMP_VALUES(UnitSimplex1D, UnitSimplex1Ds); +typedef UnitSimplexD<2> UnitSimplex2D; +IMP_VALUES(UnitSimplex2D, UnitSimplex2Ds); +typedef UnitSimplexD<3> UnitSimplex3D; +IMP_VALUES(UnitSimplex3D, UnitSimplex3Ds); +typedef UnitSimplexD<4> UnitSimplex4D; +IMP_VALUES(UnitSimplex4D, UnitSimplex4Ds); +typedef UnitSimplexD<5> UnitSimplex5D; +IMP_VALUES(UnitSimplex5D, UnitSimplex5Ds); +typedef UnitSimplexD<6> UnitSimplex6D; +IMP_VALUES(UnitSimplex6D, UnitSimplex6Ds); +typedef UnitSimplexD<-1> UnitSimplexKD; +IMP_VALUES(UnitSimplexKD, UnitSimplexKDs); + +//! Return a list of the vertices (bases) of the unit simplex. +template +inline Vector > get_vertices(const UnitSimplexD &s) { + Vector > ret; + int d = s.get_dimension(); + for (int i = 0; i < d; ++i) + ret.push_back(get_basis_vector_kd(d, i)); + return ret; +} + +inline algebra::Triangle3D get_triangle_3d(const UnitSimplex3D &s) { + Vector3Ds ps = get_vertices(s); + return Triangle3D(ps[0], ps[1], ps[2]); +} + +//! Convert point on simplex from embedded to increasing coordinates. +/** Increasing coordinates are defined as the vector whose elements contain the + cumulative sum of all embedded coordinates of lower indices. + */ +template +inline VectorD get_increasing_from_embedded(const UnitSimplexD &s, + const VectorD &p) { + int d = s.get_dimension(); + IMP_USAGE_CHECK(static_cast(p.get_dimension()) == d, + "Dimension of point must match dimension of simplex."); + IMP_INTERNAL_CHECK(s.get_contains(p), "Input point is not on simplex"); + VectorD q = get_zero_vector_kd(d); + std::partial_sum(p.begin(), p.end(), q.begin()); + return q; +} + +//! Convert point on simplex from increasing to embedded coordinates. +/** \see get_increasing_from_embedded */ +template +inline VectorD get_embedded_from_increasing(const UnitSimplexD &s, + const VectorD &p) { + int d = s.get_dimension(); + IMP_USAGE_CHECK(static_cast(p.get_dimension()) == d, + "Dimension of point must match dimension of simplex."); + VectorD q = get_zero_vector_kd(d); + std::adjacent_difference(p.begin(), p.end(), q.begin()); + IMP_INTERNAL_CHECK(s.get_contains(q), "Output point is not on simplex"); + return q; +} + +// Perform Euclidean projection of p onto the unit simplex s. +/** + Any negative weights are set to 0, and the unit l1-norm is enforced. The + algorithm used to project to the simplex is described in + arXiv:1309.1541. It finds a threshold below which all weights are set to + 0 and above which all weights shifted by the threshold sum to 1. + */ +template +inline VectorD get_projected(const UnitSimplexD &s, const VectorD &p) { + int d = s.get_dimension(); + IMP_USAGE_CHECK(d == static_cast(p.get_dimension()), + "Dimension of point must match dimension of simplex."); + + if (s.get_contains(p)) return p; + + VectorD u(p); + std::sort(u.begin(), u.end(), std::greater()); + + Floats u_cumsum(d); + std::partial_sum(u.begin(), u.end(), u_cumsum.begin()); + + int rho = 1; + while (rho < d) { + if (u[rho] + (1 - u_cumsum[rho]) / (rho + 1) < 0) break; + rho += 1; + } + double lam = (1 - u_cumsum[rho - 1]) / rho; + + for (int i = 0; i < d; ++i) { + double ui = p[i] + lam; + u[i] = ui > 0 ? ui : 0.0; + } + + return u; +} + +IMPALGEBRA_END_NAMESPACE + +#endif /* IMPALGEBRA_UNIT_SIMPLEX_D_H */ diff --git a/modules/algebra/include/VectorD.h b/modules/algebra/include/VectorD.h index 638b490ed8..908ce3eade 100644 --- a/modules/algebra/include/VectorD.h +++ b/modules/algebra/include/VectorD.h @@ -231,7 +231,16 @@ inline VectorD get_basis_vector_d(unsigned int coordinate) { return VectorD(vs, vs + D); } +//! Return the basis vector for the given coordinate +template +inline VectorD get_basis_vector_kd(int Di, unsigned int coordinate) { + IMP_USAGE_CHECK(D == Di, "D must be equal"); + IMP_UNUSED(Di); + return get_basis_vector_d(coordinate); +} + //! Return a dynamically sized basis vector +template <> inline VectorD<-1> get_basis_vector_kd(int D, unsigned int coordinate) { IMP_USAGE_CHECK(D > 0, "D must be positive"); IMP_USAGE_CHECK(coordinate < static_cast(D), @@ -246,6 +255,11 @@ inline VectorD<-1> get_basis_vector_kd(int D, unsigned int coordinate) { return VectorD<-1>(vs.get(), vs.get() + D); } +//! Return a dynamically sized basis vector +inline VectorD<-1> get_basis_vector_kd(int D, unsigned int coordinate) { + return get_basis_vector_kd<-1>(D, coordinate); +} + //! Return a vector of zeros template inline VectorD get_zero_vector_d() { @@ -260,18 +274,24 @@ inline VectorD get_zero_vector_d() { //! Return a dynamically sized vector of zeros template inline VectorD get_zero_vector_kd(int Di) { - IMP_USAGE_CHECK(D == Di, "D must be positive"); + IMP_USAGE_CHECK(D == Di, "D must be equal"); IMP_UNUSED(Di); return get_zero_vector_d(); } //! Return a dynamically sized vector of zeros +template<> inline VectorD<-1> get_zero_vector_kd(int D) { IMP_USAGE_CHECK(D > 0, "D must be positive"); Floats vs(D, 0); return VectorD<-1>(vs.begin(), vs.end()); } +//! Return a dynamically sized vector of zeros +inline VectorD<-1> get_zero_vector_kd(int D) { + return get_zero_vector_kd<-1>(D); +} + //! Return a vector of ones (or another constant) template inline VectorD get_ones_vector_d(double v = 1) { @@ -293,7 +313,8 @@ inline VectorD get_ones_vector_kd(unsigned int Di, double v = 1) { } //! Return a vector of ones (or another constant) -inline VectorD<-1> get_ones_vector_kd(unsigned int D, double v = 1) { +template <> +inline VectorD<-1> get_ones_vector_kd(unsigned int D, double v) { IMP_USAGE_CHECK(D > 0, "D must be positive"); boost::scoped_array vv(new double[D]); for (unsigned int i = 0; i < D; ++i) { @@ -302,6 +323,11 @@ inline VectorD<-1> get_ones_vector_kd(unsigned int D, double v = 1) { return VectorD<-1>(vv.get(), vv.get() + D); } +//! Return a dynamically sized vector of zeros +inline VectorD<-1> get_ones_vector_kd(unsigned int D, double v = 1) { + return get_ones_vector_kd<-1>(D, v); +} + #ifndef SWIG /** \name Norms diff --git a/modules/algebra/include/internal/internal_vector_generators.h b/modules/algebra/include/internal/internal_vector_generators.h index cd991b732b..a81519d0f4 100644 --- a/modules/algebra/include/internal/internal_vector_generators.h +++ b/modules/algebra/include/internal/internal_vector_generators.h @@ -10,6 +10,7 @@ #include "../VectorD.h" #include "../SphereD.h" #include "../SphericalVector3D.h" +#include "../UnitSimplexD.h" #include "../utility.h" #include "utility.h" #include @@ -18,8 +19,9 @@ #endif #include #include -#include #include +#include +#include IMPALGEBRA_BEGIN_INTERNAL_NAMESPACE template @@ -86,6 +88,24 @@ inline VectorD<3> get_random_vector_on_unit_sphere() { } while (true); } +//! returns a random vector on a unit simplex +template +inline VectorD get_random_vector_on(const UnitSimplexD &s) { + boost::exponential_distribution<> dist(1.0); + boost::variate_generator > + randexp(random_number_generator, dist); + int d = s.get_dimension(); + VectorD p = get_zero_vector_kd(d); + double psum = 0; + for (int i = 0; i < d; ++i) { + p[i] = randexp(); + psum += p[i]; + } + p /= psum; + return p; +} + //! returns a random vector on the surface of the sphere s in 3D //! with implementation optimized for the 3D case inline VectorD<3> get_random_vector_on(const SphereD<3> &s) { diff --git a/modules/algebra/include/internal/quaternion_derivatives.h b/modules/algebra/include/internal/quaternion_derivatives.h index fe59f56c82..352b0fe24d 100644 --- a/modules/algebra/include/internal/quaternion_derivatives.h +++ b/modules/algebra/include/internal/quaternion_derivatives.h @@ -14,10 +14,12 @@ IMPALGEBRA_BEGIN_INTERNAL_NAMESPACE -//! Get 4x4 matrix that projects a matrix to the tangent space of q. -inline Eigen::Matrix4d get_projection( +//! Get Jacobian of l2-normalization of vector. +inline Eigen::Matrix4d get_normalization_jacobian( const Eigen::Vector4d &q) { - return Eigen::Matrix4d::Identity() - q * q.transpose(); + double qnorm = q.norm(); + Eigen::Vector4d x = q / qnorm; + return (Eigen::Matrix4d::Identity() - x * x.transpose()) / qnorm; } //! Get the skew-symmetric matrix that is equivalent to the cross product. @@ -33,15 +35,15 @@ inline Eigen::Matrix3d get_cross_matrix( return vcross; } -//! Get gradient of rotated vector wrt quaternion of rotation. +//! Get Jacobian of rotated vector wrt quaternion of rotation. /** \param[in] Q quaternion of rotation (assumed to be normalized) \param[in] v vector to be rotated by rotation R(Q) - \param[in] projected Project gradient onto tangent space to Q. - Equivalent to differentiating wrt Q/||Q|| - instead of Q. + \param[in] wrt_unnorm Jacobian is computed wrt unnormalized quaternion. + Rotation includes a normalization operation, and + the columns are projected to the tangent space at Q. */ -inline Eigen::Matrix get_gradient_of_rotated( - const Eigen::Vector4d& Q, const Eigen::Vector3d& v, bool projected = true) { +inline Eigen::Matrix get_jacobian_of_rotated( + const Eigen::Vector4d& Q, const Eigen::Vector3d& v, bool wrt_unnorm = false) { Eigen::Matrix dRv_dq; Eigen::Matrix3d vcross; Eigen::Vector3d q = Q.tail(3); @@ -54,49 +56,49 @@ inline Eigen::Matrix get_gradient_of_rotated( + q * v.transpose() - q0 * vcross); - return (!projected) ? dRv_dq : dRv_dq * get_projection(Q); + return (!wrt_unnorm) ? dRv_dq : dRv_dq * get_normalization_jacobian(Q); } -//! Get gradient of Hamilton product of two quaternions wrt first quaternion. +//! Get Jacobian of Hamilton product of two quaternions wrt first quaternion. /** The combined quaternion is S = Q P. \param[in] Q first quaternion of rotation (assumed to be normalized) \param[in] P second quaternion of rotation (assumed to be normalized) - \param[in] projected Project gradient onto tangent space to Q. - Equivalent to differentiating wrt Q/||Q|| - instead of Q. + \param[in] wrt_unnorm Jacobian is computed wrt unnormalized quaternion. + Rotation includes a normalization operation, and + the columns are projected to the tangent space at Q. */ inline Eigen::Matrix4d - get_gradient_of_composed_wrt_first( + get_jacobian_of_composed_wrt_first( const Eigen::Vector4d& Q, const Eigen::Vector4d& P, - bool projected = true) { + bool wrt_unnorm = false) { Eigen::Matrix4d dqp_dq; dqp_dq.leftCols(1) = P; dqp_dq.topRightCorner(1, 3) = -P.tail(3).transpose(); dqp_dq.bottomRightCorner(3, 3) = P[0] * Eigen::Matrix3d::Identity() - get_cross_matrix(P.tail(3)); - return (!projected) ? dqp_dq : dqp_dq * get_projection(Q); + return (!wrt_unnorm) ? dqp_dq : dqp_dq * get_normalization_jacobian(Q); } -//! Get gradient of Hamilton product of two quaternions wrt second quaternion. +//! Get Jacobian of Hamilton product of two quaternions wrt second quaternion. /** The combined quaternion is S = Q P. \param[in] Q first quaternion of rotation (assumed to be normalized) \param[in] P second quaternion of rotation (assumed to be normalized) - \param[in] projected Project gradient onto tangent space to P. - Equivalent to differentiating wrt P/||P|| - instead of P. + \param[in] wrt_unnorm Jacobian is computed wrt unnormalized quaternion. + Rotation includes a normalization operation, and + the columns are projected to the tangent space at P. */ inline Eigen::Matrix4d - get_gradient_of_composed_wrt_second( + get_jacobian_of_composed_wrt_second( const Eigen::Vector4d& Q, const Eigen::Vector4d& P, - bool projected = true) { + bool wrt_unnorm = false) { Eigen::Matrix4d dqp_dp; dqp_dp.leftCols(1) = Q; dqp_dp.topRightCorner(1, 3) = -Q.tail(3).transpose(); dqp_dp.bottomRightCorner(3, 3) = Q[0] * Eigen::Matrix3d::Identity() + get_cross_matrix(Q.tail(3)); - return (!projected) ? dqp_dp : dqp_dp * get_projection(P); + return (!wrt_unnorm) ? dqp_dp : dqp_dp * get_normalization_jacobian(P); } IMPALGEBRA_END_INTERNAL_NAMESPACE diff --git a/modules/algebra/include/vector_generators.h b/modules/algebra/include/vector_generators.h index 0e2e503304..ee5f9a2a28 100644 --- a/modules/algebra/include/vector_generators.h +++ b/modules/algebra/include/vector_generators.h @@ -137,6 +137,15 @@ inline Vector > get_uniform_upper_hemisphere_cover( false); } +//! Generate a random vector on a unit simplex with uniform density +/** \see VectorD + \see UnitSimplexD + */ +template +inline VectorD get_random_vector_on(const UnitSimplexD &s) { + return internal::get_random_vector_on(s); +} + //! Generate a grid of 3d points on a cylinder surface /** \see Vector3D \see Cylinder3D diff --git a/modules/algebra/pyext/IMP_algebra.vector.i b/modules/algebra/pyext/IMP_algebra.vector.i index 3ddc877634..812414f9e1 100644 --- a/modules/algebra/pyext/IMP_algebra.vector.i +++ b/modules/algebra/pyext/IMP_algebra.vector.i @@ -34,14 +34,9 @@ namespace IMP { } } +%feature("python:maybecall", "0") IMP::algebra::VectorD::__cmp__; +%feature("python:maybecall", "0") IMP::algebra::VectorD::__eq__; %extend IMP::algebra::VectorD { - double __getitem__(unsigned int index) const { - if (index >= D) throw IMP::IndexException(""); - return self->operator[](index); - } - void __setitem__(unsigned int index, double val) { - self->operator[](index) = val; - } /* Ignore C++ return value from inplace operators, so that SWIG does not generate a new SWIG wrapper for the return value (see above). */ void __iadd__(const IMP::algebra::VectorD &o) { self->operator+=(o); } @@ -70,11 +65,59 @@ namespace IMP { }; %enddef +%define IMP_ALGEBRA_FIXED_SIZE_VECTOR(D) +IMP_ALGEBRA_VECTOR(D); + +%extend IMP::algebra::VectorD { + double __getitem__(int index) const { + if (index >= 0 && index < D) { + return self->operator[](index); + } else if (index <= -1 && index >= -(D)) { + return self->operator[](index + D); + } else { + throw IMP::IndexException("VectorD index out of range"); + } + } + void __setitem__(int index, double val) { + if (index >= 0 && index < D) { + self->operator[](index) = val; + } else if (index <= -1 && index >= -(D)) { + self->operator[](index + D) = val; + } else { + throw IMP::IndexException("VectorD assignment index out of range"); + } + } +} +%enddef + IMP_ALGEBRA_VECTOR(-1); -IMP_ALGEBRA_VECTOR(1); -IMP_ALGEBRA_VECTOR(2); -IMP_ALGEBRA_VECTOR(3); -IMP_ALGEBRA_VECTOR(4); -IMP_ALGEBRA_VECTOR(5); -IMP_ALGEBRA_VECTOR(6); +IMP_ALGEBRA_FIXED_SIZE_VECTOR(1); +IMP_ALGEBRA_FIXED_SIZE_VECTOR(2); +IMP_ALGEBRA_FIXED_SIZE_VECTOR(3); +IMP_ALGEBRA_FIXED_SIZE_VECTOR(4); +IMP_ALGEBRA_FIXED_SIZE_VECTOR(5); +IMP_ALGEBRA_FIXED_SIZE_VECTOR(6); IMP_SWIG_ALGEBRA_VALUE_D(IMP::algebra, Vector); + +%extend IMP::algebra::VectorD<-1> { + double __getitem__(int index) const { + int D = self->get_dimension(); + if (index >= 0 && index < D) { + return self->operator[](index); + } else if (index <= -1 && index >= -(D)) { + return self->operator[](index + D); + } else { + throw IMP::IndexException("VectorD index out of range"); + } + } + void __setitem__(int index, double val) { + int D = self->get_dimension(); + if (index >= 0 && index < D) { + self->operator[](index) = val; + } else if (index <= -1 && index >= -(D)) { + self->operator[](index + D) = val; + } else { + throw IMP::IndexException("VectorD assignment index out of range"); + } + } +} diff --git a/modules/algebra/pyext/include/IMP_algebra.types.i b/modules/algebra/pyext/include/IMP_algebra.types.i index b00d9116f6..1cfea928cf 100644 --- a/modules/algebra/pyext/include/IMP_algebra.types.i +++ b/modules/algebra/pyext/include/IMP_algebra.types.i @@ -25,6 +25,8 @@ IMP_SWIG_SEQUENCE_TYPEMAP_IMPL(Namespace::Namebase##D<5>, IMP::Vector, IMP::Vector >,); IMP_SWIG_SEQUENCE_TYPEMAP_IMPL(Namespace::Namebase##D<6>, IMP::Vector >,const&); IMP_SWIG_VALUE_TEMPLATE(Namespace, Namebase##D); +%feature("python:maybecall", "0") Namespace::Namebase##D::__cmp__; +%feature("python:maybecall", "0") Namespace::Namebase##D::__eq__; %extend Namespace::Namebase##D { int __cmp__(const Namebase##D &) const { IMP_UNUSED(self); @@ -51,6 +53,7 @@ IMP_SWIG_SEQUENCE_TYPEMAP_IMPL(Namespace::Namebase##D<-1>, IMP::Vector; } } + %include "IMP/algebra/VectorD.h" %include "IMP/algebra/Vector2D.h" %include "IMP/algebra/Vector3D.h" @@ -117,10 +119,23 @@ namespace IMP { %include "IMP/algebra/Transformation2D.h" %include "IMP/algebra/SphereD.h" %include "IMP/algebra/Sphere3D.h" +%include "IMP/algebra/UnitSimplexD.h" +namespace IMP { + namespace algebra { + %template(_UnitSimplexBaseKD) UnitSimplexBaseD<-1>; + %template(_UnitSimplexBase1D) UnitSimplexBaseD<1>; + %template(_UnitSimplexBase2D) UnitSimplexBaseD<2>; + %template(_UnitSimplexBase3D) UnitSimplexBaseD<3>; + %template(_UnitSimplexBase4D) UnitSimplexBaseD<4>; + %template(_UnitSimplexBase5D) UnitSimplexBaseD<5>; + %template(_UnitSimplexBase6D) UnitSimplexBaseD<6>; + } +} IMP_SWIG_ALGEBRA_TEMPLATE_D(IMP::algebra, Vector); IMP_SWIG_ALGEBRA_TEMPLATE_D(IMP::algebra, BoundingBox); IMP_SWIG_ALGEBRA_TEMPLATE_D(IMP::algebra, Sphere); +IMP_SWIG_ALGEBRA_TEMPLATE_D(IMP::algebra, UnitSimplex); %include "IMP/algebra/ReferenceFrame3D.h" @@ -206,6 +221,7 @@ IMP_SWIG_ALGEBRA_FUNCTION_TEMPLATE_D(get_unit_sphere); IMP_SWIG_ALGEBRA_FUNCTION_N_DD(bool, get_interiors_intersect, Sphere, Sphere); IMP_SWIG_ALGEBRA_FUNCTION_N_DD(bool, get_interiors_intersect, BoundingBox, BoundingBox); IMP_SWIG_ALGEBRA_FUNCTION_D_D(Vector, get_random_vector_on, Sphere); +IMP_SWIG_ALGEBRA_FUNCTION_D_D(Vector, get_random_vector_on, UnitSimplex); IMP_SWIG_ALGEBRA_FUNCTION_D_D(Vector, get_random_vector_on, BoundingBox); IMP_SWIG_ALGEBRA_FUNCTION_D_D(Vector, get_random_vector_in, Sphere); IMP_SWIG_ALGEBRA_FUNCTION_D_D(Vector, get_random_vector_in, BoundingBox); @@ -214,6 +230,10 @@ IMP_SWIG_ALGEBRA_FUNCTION_D_DD(BoundingBox, get_union, BoundingBox, BoundingBox) IMP_SWIG_ALGEBRA_FUNCTION_D_D(BoundingBox, get_bounding_box, Sphere); IMP_SWIG_ALGEBRA_FUNCTION_DS_DN(Vector, get_uniform_surface_cover, Sphere, unsigned int); IMP_SWIG_ALGEBRA_FUNCTION_DS_DN(Vector, get_grid_interior_cover_by_spacing, BoundingBox, double); +IMP_SWIG_ALGEBRA_FUNCTION_D_DD(Vector, get_projected, UnitSimplex, Vector); +IMP_SWIG_ALGEBRA_FUNCTION_DS_D(Vector, get_vertices, UnitSimplex); +IMP_SWIG_ALGEBRA_FUNCTION_D_DD(Vector, get_increasing_from_embedded, UnitSimplex, Vector); +IMP_SWIG_ALGEBRA_FUNCTION_D_DD(Vector, get_embedded_from_increasing, UnitSimplex, Vector); namespace IMP { namespace algebra { @@ -221,6 +241,13 @@ namespace IMP { %template(get_transformation_aligning_first_to_second) get_transformation_aligning_first_to_second >, IMP::Vector > >; // rotation operations + + %template(_RotatedVector3DAdjoint) ::std::pair,IMP::algebra::VectorD<4> >; + %template(_ComposeRotation3DAdjoint) ::std::pair,IMP::algebra::VectorD<4> >; + + %template(_Transformation3DAdjoint) ::std::pair,IMP::algebra::VectorD<3> >; + %template(_TransformedVector3DAdjoint) ::std::pair,std::pair,IMP::algebra::VectorD<3> > >; + %template(_ComposeTransformation3DAdjoint) ::std::pair,IMP::algebra::VectorD<3> >,std::pair,IMP::algebra::VectorD<3> > >; } } diff --git a/modules/algebra/src/Rotation3D.cpp b/modules/algebra/src/Rotation3D.cpp index 358387269b..7ce9fc50c4 100644 --- a/modules/algebra/src/Rotation3D.cpp +++ b/modules/algebra/src/Rotation3D.cpp @@ -105,45 +105,139 @@ Rotation3D get_rotation_from_matrix(double m11, double m12, double m13, return ret; } -Vector3D Rotation3D::get_derivative(const Vector3D &v, - unsigned int i, - bool projected) const { +void Rotation3D::get_rotated_adjoint(const Vector3D &x, const Vector3D &Dy, + Vector3D *Dx, Rotation3DAdjoint *DQ) const { + // Convert w=R(r)v to y=R(Q)x + double q0 = v_[0]; + const Vector3D q(v_[1], v_[2], v_[3]); + double qDy = q * Dy; + + if (Dx) { + Vector3D qcrossDy = get_vector_product(q, Dy); + double cos_theta = 2 * q0 * q0 - 1; + *Dx = cos_theta * Dy + 2 * (qDy * q - q0 * qcrossDy); // R(q)^T Dy + } + + if (DQ) { + double xDy = x * Dy; + Vector3D xcrossDy = get_vector_product(x, Dy); + Vector3D Dq = 2 * (-xDy * q + q0 * xcrossDy + q * x * Dy + qDy * x); // J_q(R(q0,q)x)^T Dy + (*DQ)[0] = 2 * (q0 * xDy + q * xcrossDy); + std::copy(Dq.begin(), Dq.end(), DQ->begin() + 1); + } +} + +RotatedVector3DAdjoint +Rotation3D::get_rotated_adjoint(const Vector3D &v, const Vector3D &Dw) const { + Vector3D Dv; + Rotation3DAdjoint Dr; + get_rotated_adjoint(v, Dw, &Dv, &Dr); + return RotatedVector3DAdjoint(Dv, Dr); +} + +Vector3D Rotation3D::get_gradient_of_rotated(const Vector3D &v, + unsigned int i, + bool wrt_unnorm) const { IMP_USAGE_CHECK(i < 4, "Invalid derivative component."); Eigen::Vector4d q(v_.get_data()); Eigen::Vector3d V(v.get_data()); - Eigen::Matrix dRv_dq = internal::get_gradient_of_rotated( - q, V, projected); + Eigen::Matrix dRv_dq = internal::get_jacobian_of_rotated( + q, V, wrt_unnorm); Vector3D dRv_dqi; Eigen::VectorXd::Map(&dRv_dqi[0], 3) = dRv_dq.col(i); return dRv_dqi; } -Eigen::MatrixXd Rotation3D::get_gradient( - const Eigen::Vector3d &v, bool projected) const { +Vector3D Rotation3D::get_derivative(const Vector3D &v, + unsigned int i, + bool wrt_unnorm) const { + IMPALGEBRA_DEPRECATED_METHOD_DEF( + 2.12, + "Use get_gradient_of_rotated(args) instead." + ); + return get_gradient_of_rotated(v, i, wrt_unnorm); +} + +Eigen::MatrixXd Rotation3D::get_jacobian_of_rotated( + const Eigen::Vector3d &v, bool wrt_unnorm) const { Eigen::Vector4d q(v_.get_data()); - return internal::get_gradient_of_rotated(q, v, projected); + return internal::get_jacobian_of_rotated(q, v, wrt_unnorm); } -Eigen::MatrixXd get_gradient_of_composed_with_respect_to_first( - const Rotation3D &q, const Rotation3D &p, bool projected) { +Eigen::MatrixXd Rotation3D::get_gradient( + const Eigen::Vector3d &v, bool wrt_unnorm) const { + IMPALGEBRA_DEPRECATED_METHOD_DEF( + 2.12, + "Use get_jacobian_of_rotated(args) instead." + ); + return get_jacobian_of_rotated(v, wrt_unnorm); +} + +void compose_adjoint(const Rotation3D &RA, const Rotation3D &RB, Vector4D DC, + Rotation3DAdjoint *DA, Rotation3DAdjoint *DB) { + const Vector4D A = RA.get_quaternion(); + const Vector4D B = RB.get_quaternion(); + + // account for compose() canonicalizing rotation + if ((A[0] * B[0] - A[1] * B[1] - A[2] * B[2] - A[3] * B[3]) < 0) + DC *= -1; + + Eigen::Map Dc(DC.get_data() + 1); + + if (DA) { + Eigen::Map b(B.begin() + 1); + Eigen::Map Da(DA->begin() + 1); + (*DA)[0] = B[0] * DC[0] + b.dot(Dc); + Da = -DC[0] * b + B[0] * Dc + b.cross(Dc); + } + + if (DB) { + Eigen::Map a(A.begin() + 1); + Eigen::Map Db(DB->begin() + 1); + (*DB)[0] = A[0] * DC[0] + a.dot(Dc); + Db = -DC[0] * a + A[0] * Dc - a.cross(Dc); + } +} + +ComposeRotation3DAdjoint +compose_adjoint(const Rotation3D &A, const Rotation3D &B, const Rotation3DAdjoint &DC) { + Rotation3DAdjoint DA, DB; + compose_adjoint(A, B, DC, &DA, &DB); + return ComposeRotation3DAdjoint(DA, DB); +} + +Eigen::MatrixXd get_jacobian_of_composed_wrt_first( + const Rotation3D &q, const Rotation3D &p, bool wrt_unnorm) { Eigen::Vector4d Q(q.get_quaternion().get_data()); Eigen::Vector4d P(p.get_quaternion().get_data()); if (Q[0] * P[0] - Q.tail(3).dot(P.tail(3)) < 0) { // account for compose() canonicalizing rotation P *= -1; } - return internal::get_gradient_of_composed_wrt_first(Q, P, projected); + return internal::get_jacobian_of_composed_wrt_first(Q, P, wrt_unnorm); } -Eigen::MatrixXd get_gradient_of_composed_with_respect_to_second( - const Rotation3D &q, const Rotation3D &p, bool projected) { +Eigen::MatrixXd get_gradient_of_composed_with_respect_to_first( + const Rotation3D &q, const Rotation3D &p, bool wrt_unnorm) { + IMPALGEBRA_DEPRECATED_FUNCTION_DEF(2.12, "Use get_jacobian_of_composed_wrt_first(args) instead."); + return get_jacobian_of_composed_wrt_first(q, p, wrt_unnorm); +} + +Eigen::MatrixXd get_jacobian_of_composed_wrt_second( + const Rotation3D &q, const Rotation3D &p, bool wrt_unnorm) { Eigen::Vector4d Q(q.get_quaternion().get_data()); Eigen::Vector4d P(p.get_quaternion().get_data()); if (Q[0] * P[0] - Q.tail(3).dot(P.tail(3)) < 0) { // account for compose() canonicalizing rotation Q *= -1; } - return internal::get_gradient_of_composed_wrt_second(Q, P, projected); + return internal::get_jacobian_of_composed_wrt_second(Q, P, wrt_unnorm); +} + +Eigen::MatrixXd get_gradient_of_composed_with_respect_to_second( + const Rotation3D &q, const Rotation3D &p, bool wrt_unnorm) { + IMPALGEBRA_DEPRECATED_FUNCTION_DEF(2.12, "Use get_jacobian_of_composed_with_respect_to_second(args) instead."); + return get_jacobian_of_composed_wrt_second(q, p, wrt_unnorm); } Rotation3D get_random_rotation_3d() { diff --git a/modules/algebra/src/Transformation3D.cpp b/modules/algebra/src/Transformation3D.cpp index a4be3dc4ff..8c3de87976 100644 --- a/modules/algebra/src/Transformation3D.cpp +++ b/modules/algebra/src/Transformation3D.cpp @@ -12,6 +12,50 @@ IMPALGEBRA_BEGIN_NAMESPACE Transformation3D::~Transformation3D() {} + +void Transformation3D::get_transformed_adjoint( + const Vector3D &v, const Vector3D &Dw, + Vector3D *Dv, Transformation3DAdjoint *DT) const { + if (!DT) return; + rot_.get_rotated_adjoint(v, Dw, Dv, &(DT->first)); + std::copy(Dw.begin(), Dw.end(), DT->second.begin()); +} + +TransformedVector3DAdjoint +Transformation3D::get_transformed_adjoint( + const Vector3D &v, const Vector3D &Dw) const { + Vector3D Dv; + Transformation3DAdjoint DT; + get_transformed_adjoint(v, Dw, &Dv, &DT); + return TransformedVector3DAdjoint(Dv, DT); +} + +void compose_adjoint(const Transformation3D &TA, const Transformation3D &TB, + const Transformation3DAdjoint &DTC, + Transformation3DAdjoint *DTA, + Transformation3DAdjoint *DTB) { + bool has_DTB = DTB; + TA.get_transformed_adjoint(TB.get_translation(), DTC.second, + has_DTB ? &(DTB->second) : nullptr, DTA); + if (DTA) { + Rotation3DAdjoint DA; + compose_adjoint(TA.get_rotation(), TB.get_rotation(), DTC.first, &DA, + has_DTB ? &(DTB->first) : nullptr); + DTA->first += DA; + } else { + compose_adjoint(TA.get_rotation(), TB.get_rotation(), DTC.first, nullptr, + has_DTB ? &(DTB->first) : nullptr); + } +} + +ComposeTransformation3DAdjoint +compose_adjoint(const Transformation3D &TA, const Transformation3D &TB, + const Transformation3DAdjoint &DTC) { + Transformation3DAdjoint DTA, DTB; + compose_adjoint(TA, TB, DTC, &DTA, &DTB); + return ComposeTransformation3DAdjoint(DTA, DTB); +} + Transformation3D Transformation3D::get_inverse() const { Rotation3D inv_rot = rot_.get_inverse(); return Transformation3D(inv_rot, -(inv_rot.get_rotated(trans_))); diff --git a/modules/algebra/test/standards_exceptions b/modules/algebra/test/standards_exceptions index 56314af320..4020a147b1 100644 --- a/modules/algebra/test/standards_exceptions +++ b/modules/algebra/test/standards_exceptions @@ -11,7 +11,7 @@ plural_exceptions=['FixedXYZ', 'FixedZYZ', 'Matrix2D', 'UnboundedGridStorage3D', 'AxisAnglePair'] show_exceptions=[] -function_name_exceptions=['compose', 'reversed_read', 'reversed_write']+\ +function_name_exceptions=['compose', 'compose_adjoint', 'reversed_read', 'reversed_write']+\ matrix_function_name_exceptions value_object_exceptions=['DenseDoubleGrid3D', 'DenseFloatGrid3D', @@ -24,4 +24,4 @@ value_object_exceptions=['DenseDoubleGrid3D', 'SparseUnboundedIntGridKD', 'Matrix2D', 'Matrix3D', 'LinearFit', 'ParabolicFit'] -spelling_exceptions=["xyz", "zyz", "zxz", "kd", "connolly", "rasterized", "rmsd"] +spelling_exceptions=["xyz", "zyz", "zxz", "kd", "connolly", "rasterized", "rmsd", "wrt"] diff --git a/modules/algebra/test/test_rotation_3d.py b/modules/algebra/test/test_rotation_3d.py index e1b66857ee..0f9c1d27d9 100644 --- a/modules/algebra/test/test_rotation_3d.py +++ b/modules/algebra/test/test_rotation_3d.py @@ -2,6 +2,7 @@ import IMP import IMP.test import IMP.algebra +import numpy as np import random import math @@ -28,7 +29,7 @@ def __call__(self, v): def get_analytic_deriv(self): uv = self.q.get_unit_vector() r = IMP.algebra.Rotation3D(uv[0], uv[1], uv[2], uv[3]) - return r.get_derivative(self.x, self.qi)[self.xi] + return r.get_jacobian_of_rotated(self.x, True)[self.xi][self.qi] class TransformFunct2: @@ -49,8 +50,8 @@ def __call__(self, v): def get_analytic_deriv(self): uv = self.q.get_unit_vector() q = IMP.algebra.Rotation3D(uv[0], uv[1], uv[2], uv[3]) - return IMP.algebra.get_gradient_of_composed_with_respect_to_first( - q, self.p)[self.pi][self.qi] + return IMP.algebra.get_jacobian_of_composed_wrt_first( + q, self.p, True)[self.pi][self.qi] class TransformFunct3: @@ -71,8 +72,8 @@ def __call__(self, v): def get_analytic_deriv(self): uv = self.p.get_unit_vector() p = IMP.algebra.Rotation3D(uv[0], uv[1], uv[2], uv[3]) - return IMP.algebra.get_gradient_of_composed_with_respect_to_second( - self.q, p)[self.pi][self.qi] + return IMP.algebra.get_jacobian_of_composed_wrt_second( + self.q, p, True)[self.pi][self.qi] class Tests(IMP.test.TestCase): @@ -197,6 +198,29 @@ def test_deriv_quaternion_product_second(self): print(ad) self.assertAlmostEqual(d, ad, delta=.05) + def test_get_rotated_adjoint(self): + rot = IMP.algebra.get_random_rotation_3d() + x = np.random.normal(size=3) + Dy = np.random.normal(size=3) + J = rot.get_jacobian_of_rotated(x) + Dq = np.dot(np.array(J).T, Dy) + Dx = rot.get_inverse() * Dy + adjoints = rot.get_rotated_adjoint(x, Dy) + self.assertSequenceAlmostEqual(list(adjoints[0]), list(Dx)) + self.assertSequenceAlmostEqual(list(adjoints[1]), Dq.tolist()) + + def test_compose_adjoint(self): + p = IMP.algebra.get_random_rotation_3d() + q = IMP.algebra.get_random_rotation_3d() + Dr = np.random.normal(size=4) + Jp = IMP.algebra.get_jacobian_of_composed_wrt_first(p, q) + Jq = IMP.algebra.get_jacobian_of_composed_wrt_second(p, q) + Dp = np.dot(np.array(Jp).T, Dr) + Dq = np.dot(np.array(Jq).T, Dr) + adjoints = IMP.algebra.compose_adjoint(p, q, Dr) + self.assertSequenceAlmostEqual(list(adjoints[0]), Dp.tolist()) + self.assertSequenceAlmostEqual(list(adjoints[1]), Dq.tolist()) + def test_rotation_between_vectors(self): """Check that the rotation between two vectors is correct""" axis = [] diff --git a/modules/algebra/test/test_transformation_3d.py b/modules/algebra/test/test_transformation_3d.py index f192776186..8abe943e4d 100644 --- a/modules/algebra/test/test_transformation_3d.py +++ b/modules/algebra/test/test_transformation_3d.py @@ -2,6 +2,7 @@ import IMP import IMP.test import IMP.algebra +import numpy as np import math import random @@ -42,6 +43,50 @@ def test_transformation3d_from2d(self): print("V1: " + str(v1)) self.assertAlmostEqual(v1[2], 0.0, delta=.01) + def test_get_transformed_adjoint(self): + r = IMP.algebra.get_random_rotation_3d() + t = np.random.normal(size=3) + T = IMP.algebra.Transformation3D(r, t) + x = np.random.normal(size=3) + Dy = np.random.normal(size=3) + + rot_adjoints = r.get_rotated_adjoint(x, Dy) + Dx = rot_adjoints[0] + Dr = rot_adjoints[1] + adjoints = T.get_transformed_adjoint(x, Dy) + + self.assertSequenceAlmostEqual(list(adjoints[0]), list(Dx)) + self.assertSequenceAlmostEqual(list(adjoints[1][0]), list(Dr)) + self.assertSequenceAlmostEqual(list(adjoints[1][1]), list(Dy)) + + def test_compose_adjoint(self): + A = IMP.algebra.get_random_rotation_3d() + B = IMP.algebra.get_random_rotation_3d() + a = np.random.normal(size=3) + b = np.random.normal(size=3) + TA = IMP.algebra.Transformation3D(A, a) + TB = IMP.algebra.Transformation3D(B, b) + DC = np.random.normal(size=4) + Dc = np.random.normal(size=3) + DTC = IMP.algebra._Transformation3DAdjoint(DC, Dc) + + DADB = IMP.algebra.compose_adjoint(A, B, DC) + DA = np.array(DADB[0]) + DB = DADB[1] + DbDTA = TA.get_transformed_adjoint(b, Dc) + Db = DbDTA[0] + DTA = DbDTA[1] + DA += np.array(DTA[0]) + Da = DTA[1] + + adjoints = IMP.algebra.compose_adjoint(TA, TB, DTC) + TAadjoints = adjoints[0] + TBadjoints = adjoints[1] + self.assertSequenceAlmostEqual(list(TAadjoints[0]), list(DA)) + self.assertSequenceAlmostEqual(list(TAadjoints[1]), list(Da)) + self.assertSequenceAlmostEqual(list(TBadjoints[0]), list(DB)) + self.assertSequenceAlmostEqual(list(TBadjoints[1]), list(Db)) + def test_transformation_between_two_reference_frames(self): """Check calculating a transformation between two reference frames""" # define a first reference frame diff --git a/modules/algebra/test/test_unit_simplex.py b/modules/algebra/test/test_unit_simplex.py new file mode 100644 index 0000000000..00c7fa0509 --- /dev/null +++ b/modules/algebra/test/test_unit_simplex.py @@ -0,0 +1,168 @@ +from __future__ import print_function +import math + +import numpy as np + +try: + import scipy.special +except ImportError: + scipy = None + +import IMP +import IMP.test +import IMP.algebra + + +class UnitSimplexDTests(IMP.test.TestCase): + + types = [ + (1, IMP.algebra.UnitSimplex1D, (), IMP.algebra.Vector1D), + (2, IMP.algebra.UnitSimplex2D, (), IMP.algebra.Vector2D), + (3, IMP.algebra.UnitSimplex3D, (), IMP.algebra.Vector3D), + (4, IMP.algebra.UnitSimplex4D, (), IMP.algebra.Vector4D), + (5, IMP.algebra.UnitSimplex5D, (), IMP.algebra.Vector5D), + (6, IMP.algebra.UnitSimplex6D, (), IMP.algebra.Vector6D), + ] + + types += [ + (d, IMP.algebra.UnitSimplexKD, (d,), IMP.algebra.VectorKD) + for d in range(1, 11) + ] + + @staticmethod + def _get_normal_thresh(tailprob, sigma=1, dim=1): + alpha = (1 - tailprob) ** dim + return sigma * np.sqrt(2) * scipy.special.erfinv(alpha) + + def test_construction(self): + """Check that fixed-dimension simplices are constructed correctly""" + for d, st, args, vt in self.types: + s = st(*args) + self.assertEqual(s.get_dimension(), d) + + def test_construct_kd_with_wrong_dimension_raises_error(self): + self.assertRaisesUsageException(IMP.algebra.UnitSimplexKD, 0) + self.assertRaisesUsageException(IMP.algebra.UnitSimplexKD, -1) + + def test_get_barycenter(self): + for d, st, args, vt in self.types: + s = st(*args) + v = s.get_barycenter() + self.assertIsInstance(v, vt) + self.assertSequenceAlmostEqual(list(s.get_barycenter()), [1.0 / d] * d) + + def test_get_contains(self): + for d, st, args, vt in self.types: + s = st(*args) + for i in range(10): + if isinstance(vt, IMP.algebra.VectorKD): + v = -np.log(np.random.uniform(size=d + 1)) + v /= np.sum(v) + self.assertFalse(s.get_contains(vt(v))) + + v = -np.log(np.random.uniform(size=d)) + self.assertFalse(s.get_contains(vt(v))) + + v /= np.sum(v) + self.assertTrue(s.get_contains(vt(v))) + + def test_get_vertices(self): + for d, st, args, vt in self.types: + s = st(*args) + vs = IMP.algebra.get_vertices(s) + I = np.eye(d) + self.assertEqual(len(vs), d) + for i, v in enumerate(vs): + self.assertIsInstance(v, vt) + self.assertSequenceAlmostEqual(list(v), list(I[i, :])) + + def test_get_increasing_from_embedded(self): + for d, st, args, vt in self.types: + s = st(*args) + for i in range(10): + v = -np.log(np.random.uniform(size=d)) + v /= np.sum(v) + + inc = IMP.algebra.get_increasing_from_embedded(s, vt(v)) + self.assertIsInstance(inc, vt) + self.assertSequenceAlmostEqual(list(inc), list(np.cumsum(v))) + + def test_get_embedded_from_increasing(self): + for d, st, args, vt in self.types: + s = st(*args) + for i in range(10): + v = -np.log(np.random.uniform(size=d)) + v /= np.sum(v) + + inc = np.cumsum(v) + v2 = IMP.algebra.get_embedded_from_increasing(s, vt(inc)) + self.assertIsInstance(v2, vt) + self.assertSequenceAlmostEqual(list(v2), list(v)) + + def test_get_projected(self): + for d, st, args, vt in self.types: + s = st(*args) + v = np.random.normal(size=d) + v_proj = IMP.algebra.get_projected(s, vt(v)) + self.assertIsInstance(v_proj, vt) + v_proj = np.array(v_proj, dtype=np.double) + + pos_inds = v_proj != 0.0 + vshift = v[pos_inds] - v_proj[pos_inds] + + self.assertTrue(np.all(v_proj >= 0)) + self.assertAlmostEqual(np.sum(v_proj), 1) + + # projection has cut point + if len(v[~pos_inds]) > 0: + min_pos = np.amin(v[pos_inds]) + max_zero = np.amax(v[~pos_inds]) + self.assertGreater(min_pos, max_zero) + + # projection is rigid shift + self.assertSequenceAlmostEqual( + list(vshift), [vshift[0]] * len(vshift) + ) + + def test_get_random_vector_on(self): + for d, st, args, vt in self.types: + s = st(*args) + for i in range(10): + v = IMP.algebra.get_random_vector_on(s) + self.assertIsInstance(v, vt) + self.assertEqual(v.get_dimension(), d) + print(v) + print(np.sum(list(v))) + self.assertAlmostEqual(np.sum(v), 1) + + @IMP.test.skipIf(scipy is None, "Requires SciPy") + def test_get_random_vector_on_is_uniform(self): + """Test that result of get_random_vector_on is uniform on simplex. + + Checks that each component of the Monte Carlo estimate of the mean + follows the central limit theorem. + """ + n = 1000 + fail_prob = 1e-3 # Probability of all tests failing. + each_fail_prob = 1 - (1 - fail_prob) ** (1.0 / len(self.types)) + + for d, st, args, vt in self.types: + s = st(*args) + bary_vs = [] + c = s.get_barycenter() + for i in range(n): + v = IMP.algebra.get_random_vector_on(s) + bary_vs.append(np.array(v - c, dtype=np.double)) + + if scipy: + mean_bary_vs = np.mean(bary_vs, axis=0) + mcse = ((d - 1.0) / (d + 1.0) / n) ** 0.5 / d + mean_thresh = self._get_normal_thresh( + each_fail_prob, dim=d, sigma=mcse + ) + for i in range(d): + self.assertLessEqual(mean_bary_vs[i], mean_thresh) + + +if __name__ == "__main__": + IMP.test.main() diff --git a/modules/algebra/test/test_vector3d.py b/modules/algebra/test/test_vector3d.py index e43d7603ec..6afd5e19e4 100644 --- a/modules/algebra/test/test_vector3d.py +++ b/modules/algebra/test/test_vector3d.py @@ -55,8 +55,29 @@ def test_component(self): self.assertEqual(v[0], 1.0) self.assertEqual(v[1], 2.0) self.assertEqual(v[2], 3.0) + + self.assertEqual(v[-3], 1.0) + self.assertEqual(v[-2], 2.0) + self.assertEqual(v[-1], 3.0) v[0] = 10.0 + v[-1] = 30.0 self.assertEqual(v[0], 10.0) + self.assertEqual(v[2], 30.0) + self.assertRaises(IndexError, lambda: v[3]) + self.assertRaises(IndexError, lambda: v[-4]) + def test_set(ind): + v[ind] = 0. + self.assertRaises(IndexError, test_set, 3) + self.assertRaises(IndexError, test_set, -4) + + def test_to_list(self): + """Check conversion of Vector3D to list""" + v = IMP.algebra.Vector3D(1.0, 2.0, 3.0) + l = list(v) + self.assertEqual(len(l), 3) + self.assertAlmostEqual(l[0], 1.0, delta=1e-6) + self.assertAlmostEqual(l[1], 2.0, delta=1e-6) + self.assertAlmostEqual(l[2], 3.0, delta=1e-6) def test_len(self): """Check Vector3D length""" diff --git a/modules/algebra/test/test_vectorkd.py b/modules/algebra/test/test_vectorkd.py index ee9235443f..ae6faef5e0 100644 --- a/modules/algebra/test/test_vectorkd.py +++ b/modules/algebra/test/test_vectorkd.py @@ -32,8 +32,29 @@ def test_component(self): self.assertEqual(v[2], 3.0) self.assertEqual(v[3], 4.0) self.assertEqual(v[4], 5.0) + + self.assertEqual(v[-3], 3.0) + self.assertEqual(v[-2], 4.0) + self.assertEqual(v[-1], 5.0) v[0] = 10.0 + v[-1] = 30.0 self.assertEqual(v[0], 10.0) + self.assertEqual(v[4], 30.0) + self.assertRaises(IndexError, lambda: v[5]) + self.assertRaises(IndexError, lambda: v[-6]) + def test_set(ind): + v[ind] = 0. + self.assertRaises(IndexError, test_set, 5) + self.assertRaises(IndexError, test_set, -6) + + def test_to_list(self): + """Check conversion of VectorKD to list""" + v = IMP.algebra.VectorKD([1.0, 2.0, 3.0]) + l = list(v) + self.assertEqual(len(l), 3) + self.assertAlmostEqual(l[0], 1.0, delta=1e-6) + self.assertAlmostEqual(l[1], 2.0, delta=1e-6) + self.assertAlmostEqual(l[2], 3.0, delta=1e-6) def test_len(self): """Check VectorKD length""" diff --git a/modules/atom/include/Residue.h b/modules/atom/include/Residue.h index 183304a4e2..58cd9e7bea 100644 --- a/modules/atom/include/Residue.h +++ b/modules/atom/include/Residue.h @@ -112,13 +112,14 @@ IMPATOMEXPORT extern const ResidueType DCYT; IMPATOMEXPORT extern const ResidueType DGUA; //! thymine (DNA) IMPATOMEXPORT extern const ResidueType DTHY; - // All further residues (including user-added residues) are ligands //! water molecule IMPATOMEXPORT extern const ResidueType HOH; //! heme IMPATOMEXPORT extern const ResidueType HEME; +//! Phosphitidyl Choline +IMPATOMEXPORT extern const ResidueType POP; #endif /*@}*/ diff --git a/modules/atom/src/Residue.cpp b/modules/atom/src/Residue.cpp index dde30dcbaf..b22ee80df1 100644 --- a/modules/atom/src/Residue.cpp +++ b/modules/atom/src/Residue.cpp @@ -73,6 +73,7 @@ RNAME_ALIAS(DTHY, DADE_T, "DTHY"); RNAME_DEF(HOH); RNAME_DEF(HEME); +RNAME_DEF(POP); RNAME_ALIAS(HEME, HEME_OLD, "HEM"); void Residue::show(std::ostream &out) const { @@ -179,6 +180,7 @@ double get_mass(ResidueType c) { residue_type_to_mass[atom::DTHY] = 482.2; residue_type_to_mass[atom::DCYT] = 467.2; residue_type_to_mass[atom::DGUA] = 507.2; + residue_type_to_mass[atom::POP] = 686.97; } return residue_type_to_mass[c]; } diff --git a/modules/bayesianem b/modules/bayesianem new file mode 160000 index 0000000000..4bc766e465 --- /dev/null +++ b/modules/bayesianem @@ -0,0 +1 @@ +Subproject commit 4bc766e465cea1a8b8722498b85b8f0a83104b19 diff --git a/modules/cgal/src/internal/bounding_sphere.cpp b/modules/cgal/src/internal/bounding_sphere.cpp index 7ac096b9b5..3aea268d42 100644 --- a/modules/cgal/src/internal/bounding_sphere.cpp +++ b/modules/cgal/src/internal/bounding_sphere.cpp @@ -5,6 +5,14 @@ * Copyright 2007-2019 IMP Inventors. All rights reserved. */ #include + +/* Workaround for a Clang parser bug; + https://bugs.llvm.org/show_bug.cgi?id=43266 + */ +#ifdef __clang__ +# include +#endif + IMP_CLANG_PRAGMA(diagnostic ignored "-Wc++11-extensions") #include #include diff --git a/modules/core/dependency/python-ihm/.travis.yml b/modules/core/dependency/python-ihm/.travis.yml index 013233386a..e47c213df1 100644 --- a/modules/core/dependency/python-ihm/.travis.yml +++ b/modules/core/dependency/python-ihm/.travis.yml @@ -5,6 +5,7 @@ python: - 2.7 - 3.6 - 3.7 + - 3.8 matrix: include: - dist: trusty diff --git a/modules/core/dependency/python-ihm/ChangeLog.rst b/modules/core/dependency/python-ihm/ChangeLog.rst index d03aa3446c..2080c0ce78 100644 --- a/modules/core/dependency/python-ihm/ChangeLog.rst +++ b/modules/core/dependency/python-ihm/ChangeLog.rst @@ -1,3 +1,38 @@ +HEAD +==== + - :func:`ihm.reader.read` has a new optional ``reject_old_file`` argument. + If set, it will raise an exception if asked to read a file that conforms + to too old a version of the IHM extension dictionary. + - Definitions for the DHSO and BMSO cross-linkers are now provided in the + :mod:`ihm.cross_linkers` module. + +0.12 - 2019-10-16 +================= + - :class:`ihm.restraint.ResidueFeature` objects can now act on one or + more :class:`Residue` objects, which act equivalently to + 1-residue ranges (:class:`AsymUnitRange` or :class:`EntityRange`). + - The new :class:`ihm.dataset.GeneticInteractionsDataset` class and the + ``mic_value`` argument to :class:`ihm.restraint.DerivedDistanceRestraint` + can be used to represent restraints from genetic interactions, such as + point-mutant epistatic miniarray profile (pE-MAP) data. + +0.11 - 2019-09-05 +================= + - :class:`ihm.Assembly` objects can now only contain :class:`AsymUnit` + and :class:`AsymUnitRange` objects (not :class:`Entity` or + :class:`EntityRange`). + - Bugfix: ensembles that don't reference a :class:`ihm.model.ModelGroup` + no longer cause the reader to create bogus empty model groups. + +0.10 - 2019-07-09 +================= + - Features (:class:`ihm.restraint.AtomFeature`, + :class:`ihm.restraint.ResidueFeature`, and + :class:`ihm.restraint.NonPolyFeature`), which previously could select part + or all of an :class:`ihm.AsymUnit`, can now also select parts of an + :class:`Entity`. A restraint acting on an entity-feature is assumed + to apply to all instances of that entity. + 0.9 - 2019-05-31 ================ - Add support for the latest version of the IHM dictionary. diff --git a/modules/core/dependency/python-ihm/MANIFEST.in b/modules/core/dependency/python-ihm/MANIFEST.in index b1f6fbd25f..900988ce88 100644 --- a/modules/core/dependency/python-ihm/MANIFEST.in +++ b/modules/core/dependency/python-ihm/MANIFEST.in @@ -1,3 +1,3 @@ include src/ihm_format.h include src/ihm_format.i -include src/ihm_format_wrap_0.9.c +include src/ihm_format_wrap_0.12.c diff --git a/modules/core/dependency/python-ihm/docs/cross_linkers.rst b/modules/core/dependency/python-ihm/docs/cross_linkers.rst index 10346ec5b7..78df5dcf6e 100644 --- a/modules/core/dependency/python-ihm/docs/cross_linkers.rst +++ b/modules/core/dependency/python-ihm/docs/cross_linkers.rst @@ -12,6 +12,11 @@ The :mod:`ihm.cross_linkers` Python module DSS cross-linker that links a primary amine with another primary amine (non-water-soluble). +.. data:: dsg + + DSG cross-linker that links a primary amine with another primary amine + (non-water-soluble). + .. data:: bs3 BS3 cross-linker that links a primary amine with another primary amine @@ -26,3 +31,15 @@ The :mod:`ihm.cross_linkers` Python module .. data:: edc EDC cross-linker that links a carboxyl group with a primary amine. + +.. data:: dhso + + DHSO (dihydrazide sulfoxide) MS-cleavable cross-linker that links + carboxyl groups, described in + `Gutierrez et al, 2016 `_. + +.. data:: bmso + + BMSO (bismaleimide sulfoxide) MS-cleavable cross-linker that links + cysteines, described in + `Gutierrez et al, 2018 `_. diff --git a/modules/core/dependency/python-ihm/docs/dataset.rst b/modules/core/dependency/python-ihm/docs/dataset.rst index fa92165715..e78a5b4d76 100644 --- a/modules/core/dependency/python-ihm/docs/dataset.rst +++ b/modules/core/dependency/python-ihm/docs/dataset.rst @@ -54,3 +54,6 @@ The :mod:`ihm.dataset` Python module .. autoclass:: YeastTwoHybridDataset :members: + +.. autoclass:: GeneticInteractionsDataset + :members: diff --git a/modules/core/dependency/python-ihm/docs/flr.rst b/modules/core/dependency/python-ihm/docs/flr.rst index 1a2100acb5..3ced68a6c6 100644 --- a/modules/core/dependency/python-ihm/docs/flr.rst +++ b/modules/core/dependency/python-ihm/docs/flr.rst @@ -40,12 +40,27 @@ The :mod:`ihm.flr` Python module .. autoclass:: Instrument :members: -.. autoclass:: ExpSetting +.. autoclass:: InstSetting :members: +.. autoclass:: ExpCondition + :members: + .. autoclass:: FRETAnalysis :members: +.. autoclass:: LifetimeFitModel + :members: + +.. autoclass:: RefMeasurementGroup + :members: + +.. autoclass:: RefMeasurement + :members: + +.. autoclass:: RefMeasurementLifetime + :members: + .. autoclass:: FRETDistanceRestraintGroup :members: diff --git a/modules/core/dependency/python-ihm/docs/reader.rst b/modules/core/dependency/python-ihm/docs/reader.rst index 88ba35d7f4..d5c69b4119 100644 --- a/modules/core/dependency/python-ihm/docs/reader.rst +++ b/modules/core/dependency/python-ihm/docs/reader.rst @@ -13,6 +13,8 @@ The :mod:`ihm.reader` Python module .. autoexception:: UnknownKeywordWarning +.. autoexception:: OldFileError + .. autoclass:: Handler :members: diff --git a/modules/core/dependency/python-ihm/ihm/__init__.py b/modules/core/dependency/python-ihm/ihm/__init__.py index 94479c53da..0415b23105 100644 --- a/modules/core/dependency/python-ihm/ihm/__init__.py +++ b/modules/core/dependency/python-ihm/ihm/__init__.py @@ -20,7 +20,7 @@ import urllib2 import json -__version__ = '0.9' +__version__ = '0.12' class __UnknownValue(object): # Represent the mmCIF 'unknown' special value @@ -170,7 +170,7 @@ def __init__(self, title=None, id='model'): #: See :class:`~ihm.model.Ensemble`. self.ensembles = [] - #: All ordered processes + #: All ordered processes. #: See :class:`~ihm.model.OrderedProcess`. self.ordered_processes = [] @@ -238,7 +238,7 @@ def _all_chem_descriptors(self): """Iterate over all ChemDescriptors in the system. Duplicates may be present.""" return itertools.chain(self.orphan_chem_descriptors, - (restraint.linker for restraint in self.restraints + (restraint.linker for restraint in self._all_restraints() if hasattr(restraint, 'linker') and restraint.linker), (itertools.chain.from_iterable( @@ -332,8 +332,9 @@ def _all_assemblies(self): if step.assembly), (step.assembly for step in self._all_analysis_steps() if step.assembly), - (restraint.assembly for restraint in self.restraints - if restraint.assembly)) + (restraint.assembly + for restraint in self._all_restraints() + if restraint.assembly)) def _all_dataset_groups(self): """Iterate over all DatasetGroups in the system. @@ -364,7 +365,7 @@ def _all_datasets_in_groups(): _all_datasets_in_groups(), (sm.dataset for sm in self._all_starting_models() if sm.dataset), - (restraint.dataset for restraint in self.restraints + (restraint.dataset for restraint in self._all_restraints() if restraint.dataset), (template.dataset for template in self._all_templates() if template.dataset)) @@ -415,20 +416,23 @@ def _all_geometric_objects(self): Duplicates may be present.""" return itertools.chain( self.orphan_geometric_objects, - (restraint.geometric_object for restraint in self.restraints - if hasattr(restraint, 'geometric_object') - and restraint.geometric_object)) + (restraint.geometric_object + for restraint in self._all_restraints() + if hasattr(restraint, 'geometric_object') + and restraint.geometric_object)) def _all_features(self): """Iterate over all Features in the system. This includes all Features referenced from other objects, plus any referenced from the top-level system. Duplicates may be present.""" - return itertools.chain( - self.orphan_features, - (restraint.feature for restraint in self.restraints - if hasattr(restraint, 'feature') - and restraint.feature)) + def _all_restraint_features(): + for r in self._all_restraints(): + if hasattr(r, '_all_features'): + for feature in r._all_features: + if feature: + yield feature + return itertools.chain(self.orphan_features, _all_restraint_features()) def _all_software(self): """Iterate over all Software in the system. @@ -454,7 +458,7 @@ def _all_citations(self): return _remove_identical(itertools.chain( self.citations, (restraint.fitting_method_citation_id - for restraint in self.restraints + for restraint in self._all_restraints() if hasattr(restraint, 'fitting_method_citation_id') and restraint.fitting_method_citation_id))) @@ -469,22 +473,18 @@ def _all_entity_ranges(self): (sm.asym_unit for sm in self._all_starting_models()), (seg.asym_unit for seg in self._all_segments()), (comp for a in self._all_assemblies() for comp in a), + (comp for f in self._all_features() + for comp in f._all_entities_or_asyms()), (d.asym_unit for d in self._all_densities()))) def _make_complete_assembly(self): - """Fill in the complete assembly with all entities/asym units""" + """Fill in the complete assembly with all asym units""" # Clear out any existing components self.complete_assembly[:] = [] # Include all asym units - seen_entities = {} for asym in self.asym_units: self.complete_assembly.append(asym) - seen_entities[asym.entity] = None - # Add all entities without structure - for entity in self.entities: - if entity not in seen_entities: - self.complete_assembly.append(entity) class Software(object): @@ -566,7 +566,7 @@ def __init__(self, pmid, title, journal, volume, page_range, year, authors, @classmethod def from_pubmed_id(cls, pubmed_id): """Create a Citation from just a PubMed ID. - This is done by querying NCBI's web api, so requires network access. + This is done by querying NCBI's web API, so requires network access. :param int pubmed_id: The PubMed identifier. :return: A new Citation for the given identifier. @@ -671,7 +671,9 @@ def __get_weight(self): return None return weight - formula_weight = property(__get_weight, doc="Formula weight") + formula_weight = property(__get_weight, + doc="Formula weight (dalton). This is calculated automatically from " + "the chemical formula and known atomic masses.") # Equal if all identifiers are the same def __eq__(self, other): @@ -906,7 +908,7 @@ class Residue(object): :meth:`AsymUnit.residue`. """ - __slots__ = ['entity', 'asym', 'seq_id'] + __slots__ = ['entity', 'asym', 'seq_id', '_range_id'] def __init__(self, seq_id, entity=None, asym=None): self.entity = entity @@ -923,6 +925,8 @@ def _get_auth_seq_id(self): auth_seq_id = property(_get_auth_seq_id, doc="Author-provided seq_id; only makes sense " "for asymmetric units") + # Allow passing residues where a range is requested (e.g. to ResidueFeature) + seq_id_range = property(lambda self: (self.seq_id, self.seq_id)) class Entity(object): @@ -991,7 +995,9 @@ def __get_weight(self): else: return None return weight - formula_weight = property(__get_weight, doc="Formula weight") + formula_weight = property(__get_weight, + doc="Formula weight (dalton). This is calculated automatically " + "from that of the chemical components.") def __init__(self, sequence, alphabet=LPeptideAlphabet, description=None, details=None, source=None): @@ -1121,10 +1127,8 @@ class Assembly(list): :param str description: Longer text that describes this assembly. This is implemented as a simple list of asymmetric units (or parts of - them) and/or entities (or parts of them), i.e. a list of - :class:`AsymUnit`, :class:`AsymUnitRange`, - :class:`Entity`, and :class:`EntityRange` objects. An Assembly is - typically assigned to one or more of + them), i.e. a list of :class:`AsymUnit` and/or :class:`AsymUnitRange` + objects. An Assembly is typically assigned to one or more of - :class:`~ihm.model.Model` - :class:`ihm.protocol.Step` diff --git a/modules/core/dependency/python-ihm/ihm/analysis.py b/modules/core/dependency/python-ihm/ihm/analysis.py index ac588865fd..dee8e6e8b2 100644 --- a/modules/core/dependency/python-ihm/ihm/analysis.py +++ b/modules/core/dependency/python-ihm/ihm/analysis.py @@ -23,17 +23,19 @@ class Step(object): script used in this step (usually a :class:`~ihm.location.WorkflowFileLocation`). :type script_file: :class:`~ihm.location.Location` + :param str details: Additional text describing this step """ type = 'other' def __init__(self, feature, num_models_begin, num_models_end, assembly=None, dataset_group=None, software=None, - script_file=None): + script_file=None, details=None): self.assembly, self.dataset_group = assembly, dataset_group self.feature, self.software = feature, software self.num_models_begin = num_models_begin self.num_models_end = num_models_end self.script_file = script_file + self.details = details class FilterStep(Step): diff --git a/modules/core/dependency/python-ihm/ihm/cross_linkers.py b/modules/core/dependency/python-ihm/ihm/cross_linkers.py index 24bb5356c8..eb4492d78b 100644 --- a/modules/core/dependency/python-ihm/ihm/cross_linkers.py +++ b/modules/core/dependency/python-ihm/ihm/cross_linkers.py @@ -13,6 +13,12 @@ '3-1-2-4-6-16(24)26-18-13(21)9-10-14(18)22/h1-10H2', inchi_key='ZWIBGKZDAWNIFC-UHFFFAOYSA-N') +dsg = ihm.ChemDescriptor('DSG', chemical_name='disuccinimidyl glutarate', + smiles='C1CC(=O)N(C1=O)OC(=O)CCCC(=O)ON2C(=O)CCC2=O', + inchi='1S/C13H14N2O8/c16-8-4-5-9(17)14(8)22-12(20)2-1-3-' + '13(21)23-15-10(18)6-7-11(15)19/h1-7H2', + inchi_key='LNQHREYHFRFJAU-UHFFFAOYSA-N') + bs3 = ihm.ChemDescriptor('BS3', chemical_name='bissulfosuccinimidyl suberate', smiles='C1C(C(=O)N(C1=O)OC(=O)CCCCCCC(=O)ON2C(=O)CC(C2=O)S(=O)' '(=O)O)S(=O)(=O)O', @@ -32,3 +38,18 @@ smiles='CCN=C=NCCCN(C)C', inchi='1S/C8H17N3/c1-4-9-8-10-6-5-7-11(2)3/h4-7H2,1-3H3', inchi_key='LMDZBCPBFSXMTL-UHFFFAOYSA-N') + +dhso = ihm.ChemDescriptor('DHSO', + chemical_name='dihydrazide sulfoxide', + smiles='NNC(=O)CC[S](=O)CCC(=O)NN', + inchi='1S/C6H14N4O3S/c7-9-5(11)1-3-14(13)4-2-6(12)10-8' + '/h1-4,7-8H2,(H,9,11)(H,10,12)', + inchi_key='XTCXQISMAWBOOT-UHFFFAOYSA-N') + +bmso = ihm.ChemDescriptor('BMSO', + chemical_name='bismaleimide sulfoxide', + smiles='O=C(CC[S](=O)CCC(=O)NCCN1C(=O)C=CC1=O)NCCN2C(=O)C=CC2=O', + inchi='1S/C18H22N4O7S/c23-13(19-7-9-21-15(25)1-2-16(21)26)5-' + '11-30(29)12-6-14(24)20-8-10-22-17(27)3-4-18(22)28/h1-' + '4H,5-12H2,(H,19,23)(H,20,24)', + inchi_key='PUNDHDZIOGBGHG-UHFFFAOYSA-N') diff --git a/modules/core/dependency/python-ihm/ihm/dataset.py b/modules/core/dependency/python-ihm/ihm/dataset.py index 51f2becc59..cc131e89ad 100644 --- a/modules/core/dependency/python-ihm/ihm/dataset.py +++ b/modules/core/dependency/python-ihm/ihm/dataset.py @@ -141,3 +141,8 @@ class FRETDataset(Dataset): class YeastTwoHybridDataset(Dataset): """Yeast two-hybrid data""" data_type = 'Yeast two-hybrid screening data' + + +class GeneticInteractionsDataset(Dataset): + """Quantitative measurements of genetic interactions""" + data_type = 'Quantitative measurements of genetic interactions' diff --git a/modules/core/dependency/python-ihm/ihm/dumper.py b/modules/core/dependency/python-ihm/ihm/dumper.py index 6a348da28b..e007aa208b 100644 --- a/modules/core/dependency/python-ihm/ihm/dumper.py +++ b/modules/core/dependency/python-ihm/ihm/dumper.py @@ -1,4 +1,4 @@ -"""Utility classes to dump out information in mmCIF format""" +"""Utility classes to dump out information in mmCIF or BinaryCIF format""" import re import os @@ -54,8 +54,8 @@ class _AuditConformDumper(Dumper): def dump(self, system, writer): with writer.category("_audit_conform") as l: # Update to match the version of the IHM dictionary we support: - l.write(dict_name="ihm-extension.dic", dict_version="0.137", - dict_location=self.URL % "7ea672a") + l.write(dict_name="ihm-extension.dic", dict_version="1.04", + dict_location=self.URL % "3a8e0b9") class _StructDumper(Dumper): @@ -536,12 +536,9 @@ def dump(self, system, writer): class _AssemblyDumper(Dumper): def finalize(self, system): - # Sort each assembly by entity/asym id/range + # Sort each assembly by entity id/asym id/range def component_key(comp): - if hasattr(comp, 'entity'): # asymmetric unit or range - return (comp.entity._id, comp._ordinal, comp.seq_id_range) - else: # entity or range - return (comp._id, 0, comp.seq_id_range) + return (comp.entity._id, comp._ordinal, comp.seq_id_range) for a in system._all_assemblies(): a.sort(key=component_key) @@ -616,6 +613,7 @@ class _LocalFiles(object): reference = None refers_to = 'Other' url = None + details = None def __init__(self, top_directory): self.top_directory = top_directory @@ -655,13 +653,13 @@ def dump_repos(self, writer): with writer.loop("_ihm_external_reference_info", ["reference_id", "reference_provider", "reference_type", "reference", "refers_to", - "associated_url"]) as l: + "associated_url", "details"]) as l: for repo in self._repo_by_id: l.write(reference_id=repo._id, reference_provider=repo.reference_provider, reference_type=repo.reference_type, reference=repo.reference, refers_to=repo.refers_to, - associated_url=repo.url) + associated_url=repo.url, details=repo.details) def dump_refs(self, writer): with writer.loop("_ihm_external_files", @@ -802,7 +800,7 @@ def dump_details(self, system, writer): "entity_asym_id", "entity_poly_segment_id", "model_object_primitive", "starting_model_id", "model_mode", "model_granularity", - "model_object_count"]) as l: + "model_object_count", "description"]) as l: for r in system._all_representations(): for segment in r: entity = segment.asym_unit.entity @@ -817,7 +815,8 @@ def dump_details(self, system, writer): else None, model_mode='rigid' if segment.rigid else 'flexible', model_granularity=segment.granularity, - model_object_count=segment.count) + model_object_count=segment.count, + description=segment.description) ordinal_id += 1 @@ -846,7 +845,7 @@ def dump_details(self, system, writer): "starting_model_source", "starting_model_auth_asym_id", "starting_model_sequence_offset", - "dataset_list_id"]) as l: + "dataset_list_id", "description"]) as l: for sm in system._all_starting_models(): l.write(starting_model_id=sm._id, entity_id=sm.asym_unit.entity._id, @@ -856,7 +855,8 @@ def dump_details(self, system, writer): starting_model_source=source_map[sm.dataset.data_type], starting_model_auth_asym_id=sm.asym_id, dataset_list_id=sm.dataset._id, - starting_model_sequence_offset=sm.offset) + starting_model_sequence_offset=sm.offset, + description=sm.description) def dump_computational(self, system, writer): """Dump details on computational models.""" @@ -981,7 +981,8 @@ def dump_details(self, system, writer): "step_name", "step_method", "num_models_begin", "num_models_end", "multi_scale_flag", "multi_state_flag", "ordered_flag", - "software_id", "script_file_id"]) as l: + "software_id", "script_file_id", + "description"]) as l: for p in system._all_protocols(): for s in p.steps: l.write(id=ordinal, protocol_id=p._id, @@ -998,7 +999,8 @@ def dump_details(self, system, writer): multi_scale_flag=s.multi_scale, software_id=s.software._id if s.software else None, script_file_id=s.script_file._id - if s.script_file else None) + if s.script_file else None, + description=s.description) ordinal += 1 @@ -1023,7 +1025,7 @@ def dump(self, system, writer): "type", "feature", "num_models_begin", "num_models_end", "struct_assembly_id", "dataset_group_id", "software_id", - "script_file_id"]) as l: + "script_file_id", "details"]) as l: for p in system._all_protocols(): for a in p.analyses: for s in a.steps: @@ -1039,7 +1041,8 @@ def dump(self, system, writer): software_id=s.software._id if s.software else None, script_file_id=s.script_file._id - if s.script_file else None) + if s.script_file else None, + details=s.details) class _RangeChecker(object): @@ -1250,7 +1253,7 @@ def dump_atoms(self, system, writer): "label_atom_id", "label_alt_id", "label_comp_id", "label_seq_id", "label_asym_id", "Cartn_x", - "Cartn_y", "Cartn_z", "label_entity_id", + "Cartn_y", "Cartn_z", "occupancy", "label_entity_id", "auth_asym_id", "B_iso_or_equiv", "pdbx_PDB_model_num", "ihm_model_id"]) as l: @@ -1272,6 +1275,7 @@ def dump_atoms(self, system, writer): auth_asym_id=atom.asym_unit._id, Cartn_x=atom.x, Cartn_y=atom.y, Cartn_z=atom.z, B_iso_or_equiv=atom.biso, + occupancy=atom.occupancy, pdbx_PDB_model_num=model._id, ihm_model_id=model._id) ordinal += 1 @@ -1313,7 +1317,7 @@ def dump(self, system, writer): "num_ensemble_models", "num_ensemble_models_deposited", "ensemble_precision_value", - "ensemble_file_id"]) as l: + "ensemble_file_id", "details"]) as l: for e in system.ensembles: l.write(ensemble_id=e._id, ensemble_name=e.name, post_process_id=e.post_process._id if e.post_process @@ -1324,7 +1328,8 @@ def dump(self, system, writer): num_ensemble_models=e.num_models, num_ensemble_models_deposited=e.num_models_deposited, ensemble_precision_value=e.precision, - ensemble_file_id=e.file._id if e.file else None) + ensemble_file_id=e.file._id if e.file else None, + details=e.details) class _DensityDumper(Dumper): @@ -1484,11 +1489,10 @@ def dump_transformations(self, writer): def dump_generic(self, writer): with writer.loop("_ihm_geometric_object_list", ["object_id", "object_type", "object_name", - "object_description", "other_details"]) as l: + "object_description"]) as l: for o in self._objects_by_id: l.write(object_id=o._id, object_type=o.type, object_name=o.name, - object_description=o.description, - other_details=o.details) + object_description=o.description) def dump_sphere(self, writer): with writer.loop("_ihm_geometric_object_sphere", @@ -1564,12 +1568,19 @@ def dump(self, system, writer): def dump_list(self, writer): with writer.loop("_ihm_feature_list", - ["feature_id", "feature_type", "entity_type"]) as l: + ["feature_id", "feature_type", "entity_type", + "details"]) as l: for f in self._features_by_id: l.write(feature_id=f._id, feature_type=f.type, - entity_type=f._get_entity_type()) + entity_type=f._get_entity_type(), + details=f.details) def dump_poly_residue(self, writer): + def _get_entity(x): + return x if isinstance(x, ihm.Entity) else x.entity + def _get_asym_id(x): + return (x._id if isinstance(x, (ihm.AsymUnit, ihm.AsymUnitRange)) + else None) ordinal = 1 with writer.loop("_ihm_poly_residue_feature", ["ordinal_id", "feature_id", "entity_id", "asym_id", @@ -1579,9 +1590,10 @@ def dump_poly_residue(self, writer): if not isinstance(f, restraint.ResidueFeature): continue for r in f.ranges: - seq = r.entity.sequence + entity = _get_entity(r) + seq = entity.sequence l.write(ordinal_id=ordinal, feature_id=f._id, - entity_id=r.entity._id, asym_id=r._id, + entity_id=entity._id, asym_id=_get_asym_id(r), seq_id_begin=r.seq_id_range[0], comp_id_begin=seq[r.seq_id_range[0]-1].id, seq_id_end=r.seq_id_range[1], @@ -1598,10 +1610,12 @@ def dump_poly_atom(self, writer): continue for a in f.atoms: r = a.residue - if r.asym.entity.is_polymeric(): - seq = r.asym.entity.sequence + entity = r.entity if r.entity else r.asym.entity + if entity.is_polymeric(): + seq = entity.sequence l.write(ordinal_id=ordinal, feature_id=f._id, - entity_id=r.asym.entity._id, asym_id=r.asym._id, + entity_id=entity._id, + asym_id=r.asym._id if r.asym else None, seq_id=r.seq_id, comp_id=seq[r.seq_id-1].id, atom_id=a.id) ordinal += 1 @@ -1615,32 +1629,35 @@ def dump_non_poly(self, writer): if isinstance(f, restraint.AtomFeature): for a in f.atoms: r = a.residue - if not r.asym.entity.is_polymeric(): - seq = r.asym.entity.sequence + entity = r.entity if r.entity else r.asym.entity + if not entity.is_polymeric(): + seq = entity.sequence l.write(ordinal_id=ordinal, feature_id=f._id, - entity_id=r.asym.entity._id, - asym_id=r.asym._id, + entity_id=entity._id, + asym_id=r.asym._id if r.asym else None, comp_id=seq[r.seq_id-1].id, atom_id=a.id) ordinal += 1 elif isinstance(f, restraint.NonPolyFeature): _ = f._get_entity_type() # trigger check for poly/nonpoly - for a in f.asyms: - seq = a.entity.sequence + for a in f.objs: + entity = a if isinstance(a, ihm.Entity) else a.entity + asym_id = a._id if isinstance(a, ihm.AsymUnit) else None + seq = entity.sequence l.write(ordinal_id=ordinal, feature_id=f._id, - entity_id=a.entity._id, - asym_id=a._id, comp_id=seq[0].id, + entity_id=entity._id, + asym_id=asym_id, comp_id=seq[0].id, atom_id=None) ordinal += 1 def dump_pseudo_site(self, writer): with writer.loop("_ihm_pseudo_site_feature", ["feature_id", "Cartn_x", "Cartn_y", - "Cartn_z", "radius", "description"]) as l: + "Cartn_z", "radius"]) as l: for f in self._features_by_id: if not isinstance(f, restraint.PseudoSiteFeature): continue l.write(feature_id=f._id, Cartn_x=f.x, Cartn_y=f.y, - Cartn_z=f.z, radius=f.radius, description=f.description) + Cartn_z=f.z, radius=f.radius) class _CrossLinkDumper(Dumper): @@ -1707,7 +1724,7 @@ def dump_list(self, system, writer): "entity_description_2", "entity_id_2", "seq_id_2", "comp_id_2", "linker_chem_comp_descriptor_id", "linker_type", - "dataset_list_id"]) as l: + "dataset_list_id", "details"]) as l: for r, xl in self._ex_xls_by_id: entity1 = xl.residue1.entity entity2 = xl.residue2.entity @@ -1724,7 +1741,8 @@ def dump_list(self, system, writer): comp_id_2=seq2[xl.residue2.seq_id-1].id, linker_chem_comp_descriptor_id=r.linker._id, linker_type=r.linker.auth_name, - dataset_list_id=r.dataset._id) + dataset_list_id=r.dataset._id, + details=xl.details) def dump_restraint(self, system, writer): with writer.loop("_ihm_cross_link_restraint", @@ -1843,7 +1861,7 @@ def dump(self, system, writer): with writer.loop("_ihm_derived_distance_restraint", ["id", "group_id", "feature_id_1", "feature_id_2", "restraint_type", "distance_lower_limit", - "distance_upper_limit", "probability", + "distance_upper_limit", "probability", "mic_value", "group_conditionality", "dataset_list_id"]) as l: for r in self._restraints_by_id: l.write(id=r._id, feature_id_1=r.feature1._id, @@ -1852,7 +1870,7 @@ def dump(self, system, writer): restraint_type=r.distance.restraint_type, distance_lower_limit=r.distance.distance_lower_limit, distance_upper_limit=r.distance.distance_upper_limit, - probability=r.probability, + probability=r.probability, mic_value=r.mic_value, group_conditionality=condmap[r.restrain_all], dataset_list_id=r.dataset._id if r.dataset else None) @@ -2065,29 +2083,42 @@ def all_experiments(): def dump(self, system, writer): with writer.loop('_flr_experiment', - ['ordinal_id', 'id', 'instrument_id', 'exp_setting_id', - 'sample_id', 'details']) as l: + ['ordinal_id', 'id', 'instrument_id', 'inst_setting_id', + 'exp_condition_id','sample_id', 'details']) as l: ordinal = 1 for x in self._experiments_by_id: for i in range(len(x.sample_list)): l.write(ordinal_id=ordinal, id=x._id, instrument_id=x.instrument_list[i]._id, - exp_setting_id=x.exp_setting_list[i]._id, + inst_setting_id=x.inst_setting_list[i]._id, + exp_condition_id=x.exp_condition_list[i]._id, sample_id=x.sample_list[i]._id, details=x.details_list[i]) ordinal +=1 -class _FLRExpSettingDumper(Dumper): +class _FLRInstSettingDumper(Dumper): + def finalize(self, system): + def all_inst_settings(): + return itertools.chain.from_iterable(f._all_inst_settings() + for f in system.flr_data) + self._inst_settings_by_id = _assign_all_ids(all_inst_settings) + + def dump(self, system, writer): + with writer.loop('_flr_inst_setting', ['id', 'details']) as l: + for x in self._inst_settings_by_id: + l.write(id=x._id, details=x.details) + +class _FLR_ExpConditionDumper(Dumper): def finalize(self, system): - def all_exp_settings(): - return itertools.chain.from_iterable(f._all_exp_settings() + def all_exp_conditions(): + return itertools.chain.from_iterable(f._all_exp_conditions() for f in system.flr_data) - self._exp_settings_by_id = _assign_all_ids(all_exp_settings) + self._exp_conditions_by_id = _assign_all_ids(all_exp_conditions) def dump(self, system, writer): - with writer.loop('_flr_exp_setting', ['id', 'details']) as l: - for x in self._exp_settings_by_id: + with writer.loop('_flr_exp_condition', ['id', 'details']) as l: + for x in self._exp_conditions_by_id: l.write(id=x._id, details=x.details) @@ -2337,6 +2368,90 @@ def dump(self, system, writer): gamma=x.gamma, delta=x.delta, a_b=x.a_b) +class _FLRLifetimeFitModelDumper(Dumper): + def finalize(self, system): + def all_lifetime_fit_models(): + return itertools.chain.from_iterable(f._all_lifetime_fit_models() + for f in system.flr_data) + self._lifetime_fit_models_by_id = _assign_all_ids(all_lifetime_fit_models) + + def dump(self, system, writer): + with writer.loop('_flr_lifetime_fit_model', + ['id', 'name', 'description', + 'external_file_id', 'citation_id']) as l: + for x in self._lifetime_fit_models_by_id: + l.write(id = x._id, name = x.name, + description = x.description, + external_file_id = None if x.external_file is None + else x.external_file._id, + citation_id = None if x.citation is None + else x.citation._id) + + +class _FLRRefMeasurementDumper(Dumper): + def finalize(self, system): + def all_ref_measurement_groups(): + return itertools.chain.from_iterable(f._all_ref_measurement_groups() + for f in system.flr_data) + self._ref_measurement_groups_by_id = _assign_all_ids(all_ref_measurement_groups) + + def _all_ref_measurements(): + return itertools.chain.from_iterable(f._all_ref_measurements() + for f in system.flr_data) + self._ref_measurements_by_id = _assign_all_ids(_all_ref_measurements) + + def _all_ref_measurement_lifetimes(): + return itertools.chain.from_iterable(f._all_ref_measurement_lifetimes() + for f in system.flr_data) + self._ref_measurement_lifetimes_by_id = _assign_all_ids(_all_ref_measurement_lifetimes) + + def dump(self, system, writer): + self.dump_ref_measurement_group(system, writer) + self.dump_ref_measurement_group_link(system, writer) + self.dump_ref_measurement(system, writer) + self.dump_ref_measurement_lifetimes(system, writer) + + def dump_ref_measurement_group(self, system, writer): + with writer.loop('_flr_reference_measurement_group', + ['id', 'num_measurements','details']) as l: + for x in self._ref_measurement_groups_by_id: + l.write(id = x._id, + num_measurements = len(x.ref_measurement_list), + details = x.details) + + def dump_ref_measurement_group_link(self, system, writer): + with writer.loop('_flr_reference_measurement_group_link', + ['group_id', 'reference_measurement_id']) as l: + for x in self._ref_measurement_groups_by_id: + for m in x.ref_measurement_list: + l.write(group_id=x._id, + reference_measurement_id = m._id) + + def dump_ref_measurement(self, system, writer): + with writer.loop('_flr_reference_measurement', + ['id', 'reference_sample_probe_id', + 'num_species', 'details']) as l: + for x in self._ref_measurements_by_id: + l.write(id = x._id, + reference_sample_probe_id = x.ref_sample_probe._id, + num_species = len(x.list_of_lifetimes), + details = x.details) + + def dump_ref_measurement_lifetimes(self, system, writer): + with writer.loop('_flr_reference_measurement_lifetime', + ['ordinal_id', 'reference_measurement_id', + 'species_name', 'species_fraction', 'lifetime']) as l: + ordinal = 1 + for x in self._ref_measurements_by_id: + for m in x.list_of_lifetimes: + l.write(ordinal_id = ordinal, + reference_measurement_id = x._id, + species_name = m.species_name, + species_fraction = m.species_fraction, + lifetime = m.lifetime) + ordinal += 1 + + class _FLRAnalysisDumper(Dumper): def finalize(self, system): def all_analyses(): @@ -2345,27 +2460,67 @@ def all_analyses(): self._analyses_by_id = _assign_all_ids(all_analyses) def dump(self, system, writer): + self.dump_fret_analysis_general(system, writer) + self.dump_fret_analysis_intensity(system, writer) + self.dump_fret_analysis_lifetime(system, writer) + + def dump_fret_analysis_general(self, system, writer): with writer.loop('_flr_fret_analysis', - ['id', 'experiment_id', 'sample_probe_id_1', - 'sample_probe_id_2', 'forster_radius_id', - 'calibration_parameters_id', 'method_name', - 'chi_square_reduced', 'dataset_list_id', + ['id', 'experiment_id', 'type', + 'sample_probe_id_1', 'sample_probe_id_2', + 'forster_radius_id', 'dataset_list_id', 'external_file_id', 'software_id']) as l: for x in self._analyses_by_id: l.write(id=x._id, experiment_id=x.experiment._id, + type = x.type, sample_probe_id_1=x.sample_probe_1._id, sample_probe_id_2=x.sample_probe_2._id, forster_radius_id=x.forster_radius._id, - calibration_parameters_id=x.calibration_parameters._id, - method_name=x.method_name, - chi_square_reduced=x.chi_square_reduced, dataset_list_id=x.dataset._id, external_file_id=None if x.external_file is None else x.external_file._id, software_id=None if x.software is None else x.software._id) + def dump_fret_analysis_intensity(self, system, writer): + with writer.loop('_flr_fret_analysis_intensity', + ['ordinal_id', 'analysis_id', + 'calibration_parameters_id', 'donor_only_fraction', + 'chi_square_reduced', 'method_name', 'details']) as l: + ordinal = 1 + for x in self._analyses_by_id: + ## if it is an intensity-based analysis. + if 'intensity' in x.type: + l.write(ordinal_id = ordinal, + analysis_id = x._id, + calibration_parameters_id = None if x.calibration_parameters is None else x.calibration_parameters._id, + donor_only_fraction = x.donor_only_fraction, + chi_square_reduced = x.chi_square_reduced, + method_name = x.method_name, + details = x.details) + ordinal += 1 + + def dump_fret_analysis_lifetime(self, system, writer): + with writer.loop('_flr_fret_analysis_lifetime', + ['ordinal_id', 'analysis_id', + 'reference_measurement_group_id', 'lifetime_fit_model_id', + 'donor_only_fraction', 'chi_square_reduced', + 'method_name', 'details']) as l: + ordinal = 1 + for x in self._analyses_by_id: + ## if it is a lifetime-based analysis + if 'lifetime' in x.type: + l.write(ordinal_id = ordinal, + analysis_id = x._id, + reference_measurement_group_id = x.ref_measurement_group._id, + lifetime_fit_model_id = x.lifetime_fit_model._id, + donor_only_fraction = x.donor_only_fraction, + chi_square_reduced = x.chi_square_reduced, + method_name = x.method_name, + details = x.details) + ordinal += 1 + class _FLRPeakAssignmentDumper(Dumper): def finalize(self, system): @@ -2635,7 +2790,8 @@ def _check_restraint_groups(system): "Restraints. Due to limitations of the underlying dictionary, " "all objects in a RestraintGroup must be of the same type, " "and only certain types (currently only " - "DerivedDistanceRestraint) can be grouped." % g) + "DerivedDistanceRestraint or PredictedContactRestraint) " + "can be grouped." % g) def write(fh, systems, format='mmCIF', dumpers=[]): """Write out all `systems` to the file handle `fh`. @@ -2681,12 +2837,14 @@ def write(fh, systems, format='mmCIF', dumpers=[]): _EnsembleDumper(), _DensityDumper(), _MultiStateDumper(), _OrderedDumper(), - _FLRExperimentDumper(), _FLRExpSettingDumper(), + _FLRExperimentDumper(), _FLRInstSettingDumper(), + _FLR_ExpConditionDumper(), _FLRInstrumentDumper(), _FLREntityAssemblyDumper(), _FLRSampleConditionDumper(), _FLRSampleDumper(), _FLRProbeDumper(), _FLRSampleProbeDetailsDumper(), _FLRPolyProbePositionDumper(), _FLRConjugateDumper(), _FLRForsterRadiusDumper(), _FLRCalibrationParametersDumper(), + _FLRLifetimeFitModelDumper(), _FLRRefMeasurementDumper(), _FLRAnalysisDumper(), _FLRPeakAssignmentDumper(), _FLRDistanceRestraintDumper(), _FLRModelQualityDumper(), _FLRModelDistanceDumper(), _FLRFPSModelingDumper(), diff --git a/modules/core/dependency/python-ihm/ihm/flr.py b/modules/core/dependency/python-ihm/ihm/flr.py index 1706042a71..f757143720 100644 --- a/modules/core/dependency/python-ihm/ihm/flr.py +++ b/modules/core/dependency/python-ihm/ihm/flr.py @@ -244,59 +244,66 @@ class Experiment(object): :param instrument: The instrument. :type instrument: :class:`Instrument` - :param exp_setting: The experimental setting. - :type exp_setting: :class:`ExpSetting` + :param inst_setting: The instrument setting. + :type inst_setting: :class:`InstSetting` + :param exp_condition: The experimental conditions. + :type exp_condition: :class:`ExpCondition` :param sample: The sample. :type sample: :class:`Sample` :param details: Details on the experiment. """ - def __init__(self, instrument=None, exp_setting=None, sample=None, - details=None): + def __init__(self, instrument=None, inst_setting=None, exp_condition=None, + sample=None, details=None): """The Experiment object can either be initiated with empty lists, or with an entry for each of them. In this way, an experiment object is created and filled with one entry. """ self.instrument_list = [] - self.exp_setting_list = [] + self.inst_setting_list = [] + self.exp_condition_list = [] self.sample_list = [] self.details_list = [] - if instrument != None and exp_setting != None and sample != None: - self.add_entry(instrument=instrument, exp_setting=exp_setting, + if instrument != None and inst_setting != None and exp_condition != None and sample != None: + self.add_entry(instrument=instrument, inst_setting=inst_setting, exp_condition=exp_condition, sample=sample, details=details) - def add_entry(self, instrument, exp_setting, sample,details=None): + def add_entry(self, instrument, inst_setting, exp_condition, sample,details=None): """Entries to the experiment object can also be added one by one. """ self.instrument_list.append(instrument) - self.exp_setting_list.append(exp_setting) + self.inst_setting_list.append(inst_setting) + self.exp_condition_list.append(exp_condition) self.sample_list.append(sample) self.details_list.append(details) def get_entry_by_index(self, index): - """Returns the combination of :class:`Instrument`, :class:`ExpSetting`, - :class:`Sample`, and details for a given index. + """Returns the combination of :class:`Instrument`, :class:`InstSetting`, + :class:`ExpCondition`, :class:`Sample`, and details for a given index. """ return (self.instrument_list[index], - self.exp_setting_list[index], self.sample_list[index], + self.inst_setting_list[index], + self.exp_condition_list[index], + self.sample_list[index], self.details_list[index]) def __eq__(self, other): -# return self.__dict__ == other.__dict__ return ((self.instrument_list == other.instrument_list) - and (self.exp_setting_list == other.exp_setting_list) + and (self.inst_setting_list == other.inst_setting_list) + and (self.exp_condition_list == other.exp_condition_list) and (self.sample_list == other.sample_list) and (self.details_list == other.details_list)) - def contains(self, instrument, exp_setting, sample): + def contains(self, instrument, inst_setting, exp_condition, sample): """Checks whether a combination of :class:`Instrument`, - :class:`ExpSetting`, :class:`Sample` is already included in - the experiment object. + :class:`InstSetting`, :class:`ExpCondition`, + :class:`Sample` is already included in the experiment object. """ ## TODO: possibly extend this by the details_list? for i in range(len(self.instrument_list)): if ((instrument == self.instrument_list[i]) - and (exp_setting == self.exp_setting_list[i]) + and (inst_setting == self.inst_setting_list[i]) + and (exp_condition == self.exp_condition_list[i]) and (sample == self.sample_list[i])): return True return False @@ -316,22 +323,35 @@ def __eq__(self, other): return self.__dict__ == other.__dict__ -class ExpSetting(object): - """Description of the experimental settings. +class InstSetting(object): + """Description of the instrument settings. *Currently this is only text, but will be extended in the future.* - :param str details: Description of the experimental settings used for - the measurement (e.g. temperature, or size of observation + :param str details: Description of the instrument settings used for + the measurement (e.g. laser power or size of observation volume in case of confocal measurements). """ - def __init__(self, details=None): self.details = details def __eq__(self, other): return self.__dict__ == other.__dict__ +class ExpCondition(object): + """Description of the experimental conditions. + + * Currently this is only text, but will be extended in the future.* + + :param str details: Description of the experimental conditions (e.g. + the temperature at which the experiment was carried out). + """ + def __init__(self, details=None): + self.details = details + + def __eq__(self,other): + return self.__dict__ == other.__dict__ + class FRETAnalysis(object): """An analysis of FRET data that was performed. @@ -346,12 +366,19 @@ class FRETAnalysis(object): :type sample_probe_2: :class:`SampleProbeDetails` :param forster_radius: The Förster radius object for this FRET analysis. :type forster_radius: :class:`FRETForsterRadius`. + :param str type: The type of the FRET analysis (intensity-based or lifetime-based). :param calibration_parameters: The calibration parameters used for - this analysis. + this analysis (only in case of intensity-based analyses). :type calibration_parameters: :class:`FRETCalibrationParameters` + :param lifetime_fit_model: The fit model used in case of lifetime-based analyses. + :type lifetime_fit_model: :class:`LifetimeFitModel` + :param ref_measurement_group: The group of reference measurements + in case of lifetime-based analyses. + :type ref_measurement_group: :class:`LifetimeRefMeasurementGroup` :param str method_name: The method used for the analysis. :param float chi_square_reduced: The chi-square reduced as a quality measure for the fit. + :param float donor_only_fraction: The donor-only fraction. :param dataset: The dataset used. :type dataset: :class:`ihm.dataset.Dataset` :param external_file: The external file that contains (results of) @@ -361,16 +388,26 @@ class FRETAnalysis(object): """ def __init__(self, experiment, sample_probe_1, sample_probe_2, - forster_radius, calibration_parameters, method_name=None, - chi_square_reduced=None, dataset=None, - external_file=None, software=None): + forster_radius, type, calibration_parameters=None, + lifetime_fit_model = None, + ref_measurement_group = None, + method_name=None, details = None, + chi_square_reduced=None, donor_only_fraction=None, + dataset=None, external_file=None, software=None): + if type not in ['lifetime-based', 'intensity-based', None]: + raise ValueError('FRETAnalysis.type can be \'lifetime-based\' or \'intensity-based\'. The value is %s'%(type)) self.experiment = experiment self.sample_probe_1 = sample_probe_1 self.sample_probe_2 = sample_probe_2 self.forster_radius = forster_radius + self.type = type self.calibration_parameters = calibration_parameters + self.lifetime_fit_model = lifetime_fit_model + self.ref_measurement_group = ref_measurement_group self.method_name = method_name + self.details = details self.chi_square_reduced = chi_square_reduced + self.donor_only_fraction = donor_only_fraction self.dataset = dataset self.external_file = external_file self.software = software @@ -379,6 +416,86 @@ def __eq__(self, other): return self.__dict__ == other.__dict__ +class LifetimeFitModel(object): + """A lifetime-fit model used for lifetime-based analysis. + + :param str name: The name of the fit model. + :param str description: A description of the fit model. + :param external_file: An external file that contains additional information on the fit model. + :param citation: A citation for the fit model. + :type citation: :class:`ihm.Citation` + """ + def __init__(self, name, description, external_file=None, citation=None): + self.name = name + self.description = description + self.external_file = external_file + self.citation = citation + + def __eq__(self, other): + return self.__dict__ == other.__dict__ + + +class RefMeasurementGroup(object): + """A Group containing reference measurements for lifetime-based analysis. + + :param str details: Details on the Group of reference measurements. + """ + def __init__(self, details=None): + self.details = details + self.ref_measurement_list = [] + self.num_measurements = len(self.ref_measurement_list) + + def add_ref_measurement(self, ref_measurement): + """Add a lifetime reference measurement to a ref_measurement_group.""" + self.ref_measurement_list.append(ref_measurement) + self.num_measurements = len(self.ref_measurement_list) + + def get_info(self): + return self.ref_measurement_list + + def __eq__(self, other): + return self.__dict__ == other.__dict__ + + +class RefMeasurement(object): + """A reference measurement for lifetime-based analysis. + + :param ref_sample_probe: The combination of sample and probe used for the reference measurement. + :type ref_sample_probe: :class:`SampleProbeDetails` + :param str details: Details on the measurement. + :param list_of_lifetimes: A list of the results from the reference measurement. + :type list_of_lifetimes: List of :class:`RefMeasurementLifetime` + """ + def __init__(self, ref_sample_probe, details=None, list_of_lifetimes=None): + self.ref_sample_probe = ref_sample_probe + self.details = details + self.list_of_lifetimes = list_of_lifetimes if list_of_lifetimes is not None else [] + self.num_species = len(self.list_of_lifetimes) + + def add_lifetime(self, lifetime): + """Add a lifetime to the list_of_lifetimes.""" + self.list_of_lifetimes.append(lifetime) + self.num_species = len(self.list_of_lifetimes) + + def __eq__(self,other): + return self.__dict__ == other.__dict__ + + +class RefMeasurementLifetime(object): + """Lifetime for a species in a reference measurement. + :param float species_fraction: The species-fraction for the respective lifetime. + :param float lifetime: The lifetime (in ns). + :param str species_name: A name for the species. + """ + def __init__(self, species_fraction, lifetime, species_name=None): + self.species_fraction = species_fraction + self.lifetime = lifetime + self.species_name = species_name + + def __eq__(self, other): + return self.__dict__ == other.__dict__ + + class FRETDistanceRestraintGroup(object): """A collection of FRET distance restraints that are used together. """ @@ -828,7 +945,8 @@ def __init__(self): ## The following dictionaries are so far only used when reading data self._collection_flr_experiment = {} - self._collection_flr_exp_setting = {} + self._collection_flr_inst_setting = {} + self._collection_flr_exp_condition = {} self._collection_flr_instrument = {} self._collection_flr_entity_assembly = {} self._collection_flr_sample_condition = {} @@ -842,6 +960,10 @@ def __init__(self): self._collection_flr_fret_forster_radius = {} self._collection_flr_fret_calibration_parameters = {} self._collection_flr_fret_analysis = {} + self._collection_flr_lifetime_fit_model = {} + self._collection_flr_ref_measurement_group = {} + self._collection_flr_ref_measurement = {} + self._collection_flr_ref_measurement_lifetime = {} self._collection_flr_peak_assignment = {} self._collection_flr_fret_distance_restraint = {} self._collection_flr_fret_distance_restraint_group = {} @@ -884,13 +1006,40 @@ def _all_forster_radii(self): def _all_calibration_parameters(self): """Yield all FRETCalibrationParameters objects""" for a in self._all_analyses(): - yield a.calibration_parameters + if a.type == 'intensity-based': + yield a.calibration_parameters + + def _all_lifetime_fit_models(self): + """Yield all LifetimeFitModel objects""" + for a in self._all_analyses(): + if a.type == 'lifetime-based': + yield a.lifetime_fit_model + + def _all_ref_measurement_groups(self): + """Yield all RefMeasurementGroup objects""" + for a in self._all_analyses(): + if a.type == 'lifetime-based': + yield a.ref_measurement_group + + def _all_ref_measurements(self): + """Yield all RefMeasurement objects""" + for rg in self._all_ref_measurement_groups(): + for x in rg.ref_measurement_list: + yield x + + def _all_ref_measurement_lifetimes(self): + """Yield all RefMeasurementLifetime objects""" + for r in self._all_ref_measurements(): + for x in r.list_of_lifetimes: + yield x def _all_sample_probe_details(self): """Yield all SampleProbeDetails objects""" for r in self._all_distance_restraints(): yield r.sample_probe_1 yield r.sample_probe_2 + for r in self._all_ref_measurements(): + yield r.ref_sample_probe def _all_samples(self): """Yield all Sample objects""" @@ -907,10 +1056,16 @@ def _all_poly_probe_positions(self): for s in self._all_sample_probe_details(): yield s.poly_probe_position - def _all_exp_settings(self): - """Yield all ExpSetting objects""" + def _all_inst_settings(self): + """Yield all InstSetting objects""" + for e in self._all_experiments(): + for s in e.inst_setting_list: + yield s + + def _all_exp_conditions(self): + """Yield all ExpCondition objects""" for e in self._all_experiments(): - for s in e.exp_setting_list: + for s in e.exp_condition_list: yield s def _all_instruments(self): diff --git a/modules/core/dependency/python-ihm/ihm/geometry.py b/modules/core/dependency/python-ihm/ihm/geometry.py index a661344fa5..1ea7495cb8 100644 --- a/modules/core/dependency/python-ihm/ihm/geometry.py +++ b/modules/core/dependency/python-ihm/ihm/geometry.py @@ -45,13 +45,10 @@ class GeometricObject(object): :param str name: A short user-provided name. :param str description: A brief description of the object. - :param str details: Additional details about the object (particularly - useful for generic objects). """ type = 'other' - def __init__(self, name=None, description=None, details=None): + def __init__(self, name=None, description=None): self.name, self.description = name, description - self.details = details class Sphere(GeometricObject): @@ -65,14 +62,13 @@ class Sphere(GeometricObject): :type transformation: :class:`Transformation` :param str name: A short user-provided name. :param str description: A brief description of the object. - :param str details: Additional details about the object. """ type = 'sphere' def __init__(self, center, radius, transformation=None, - name=None, description=None, details=None): - super(Sphere, self).__init__(name, description, details) + name=None, description=None): + super(Sphere, self).__init__(name, description) self.center, self.transformation = center, transformation self.radius = radius @@ -91,13 +87,12 @@ class Torus(GeometricObject): :type transformation: :class:`Transformation` :param str name: A short user-provided name. :param str description: A brief description of the object. - :param str details: Additional details about the object. """ type = 'torus' def __init__(self, center, major_radius, minor_radius, transformation=None, - name=None, description=None, details=None): - super(Torus, self).__init__(name, description, details) + name=None, description=None): + super(Torus, self).__init__(name, description) self.center, self.transformation = center, transformation self.major_radius, self.minor_radius = major_radius, minor_radius @@ -110,16 +105,15 @@ class HalfTorus(GeometricObject): :param thickness: The thickness of the surface. :param inner: True if the surface is the 'inner' half of the torus (i.e. closer to the center), False for the outer surface, or - None for some other section (described in `details`). + None for some other section (described in `description`). See :class:`Torus` for a description of the other parameters. """ type = 'half-torus' def __init__(self, center, major_radius, minor_radius, thickness, - transformation=None, inner=None, name=None, description=None, - details=None): - super(HalfTorus, self).__init__(name, description, details) + transformation=None, inner=None, name=None, description=None): + super(HalfTorus, self).__init__(name, description) self.center, self.transformation = center, transformation self.major_radius, self.minor_radius = major_radius, minor_radius self.thickness, self.inner = thickness, inner @@ -134,13 +128,11 @@ class Axis(GeometricObject): :type transformation: :class:`Transformation` :param str name: A short user-provided name. :param str description: A brief description of the object. - :param str details: Additional details about the object. """ type = 'axis' - def __init__(self, transformation=None, name=None, description=None, - details=None): - super(Axis, self).__init__(name, description, details) + def __init__(self, transformation=None, name=None, description=None): + super(Axis, self).__init__(name, description) self.transformation = transformation @@ -177,13 +169,11 @@ class Plane(GeometricObject): :type transformation: :class:`Transformation` :param str name: A short user-provided name. :param str description: A brief description of the object. - :param str details: Additional details about the object. """ type = 'plane' - def __init__(self, transformation=None, name=None, description=None, - details=None): - super(Plane, self).__init__(name, description, details) + def __init__(self, transformation=None, name=None, description=None): + super(Plane, self).__init__(name, description) self.transformation = transformation diff --git a/modules/core/dependency/python-ihm/ihm/location.py b/modules/core/dependency/python-ihm/ihm/location.py index 1ac33f3239..248e229d34 100644 --- a/modules/core/dependency/python-ihm/ihm/location.py +++ b/modules/core/dependency/python-ihm/ihm/location.py @@ -6,6 +6,7 @@ class Location(object): """Identifies the location where a resource can be found. + Do not use this class itself, but one of its subclasses. Typically the resource may be found in a file (either on the local disk or at a DOI) - for this use one of the subclasses of :class:`FileLocation`. Alternatively the resource may be found in @@ -257,6 +258,7 @@ class Repository(object): at the URL or DOI (for example, GitHub repositories archived at Zenodo get placed in a subdirectory named for the repository and git hash). + :param str details: Additional text describing this repository """ reference_type = 'DOI' @@ -267,10 +269,12 @@ def __eq__(self, other): def __hash__(self): return hash((self.doi, self.url)) - def __init__(self, doi, root=None, url=None, top_directory=None): + def __init__(self, doi, root=None, url=None, top_directory=None, + details=None): # todo: DOI should be optional (could also use URL, local path) self.doi = doi self.url, self.top_directory = url, top_directory + self.details = details if root is not None: # Store absolute path in case the working directory changes later self._root = os.path.abspath(root) diff --git a/modules/core/dependency/python-ihm/ihm/metadata.py b/modules/core/dependency/python-ihm/ihm/metadata.py index 2d3c396e7d..a5532c4779 100644 --- a/modules/core/dependency/python-ihm/ihm/metadata.py +++ b/modules/core/dependency/python-ihm/ihm/metadata.py @@ -68,7 +68,7 @@ def parse_file(self, filename): def _get_emdb(self, filename): """Return the EMDB id of the file, or None.""" - r = re.compile(b'EMDATABANK\.org.*(EMD\-\d+)') + r = re.compile(b'EMDATABANK\.org.*(EMD\-\\d+)') with open(filename, 'rb') as fh: fh.seek(220) # Offset of number of labels num_labels_raw = fh.read(4) @@ -321,12 +321,12 @@ def _get_templates_script(self, pdbname, target_dataset): template_path_map = {} alnfile = None script = None - alnfilere = re.compile('REMARK 6 ALIGNMENT: (\S+)') - scriptre = re.compile('REMARK 6 SCRIPT: (\S+)') - tmppathre = re.compile('REMARK 6 TEMPLATE PATH (\S+) (\S+)') - tmpre = re.compile('REMARK 6 TEMPLATE: ' - '(\S+) (\S+):(\S+) \- (\S+):\S+ ' - 'MODELS (\S+):(\S+) \- (\S+):\S+ AT (\S+)%') + alnfilere = re.compile(r'REMARK 6 ALIGNMENT: (\S+)') + scriptre = re.compile(r'REMARK 6 SCRIPT: (\S+)') + tmppathre = re.compile(r'REMARK 6 TEMPLATE PATH (\S+) (\S+)') + tmpre = re.compile(r'REMARK 6 TEMPLATE: ' + r'(\S+) (\S+):(\S+) \- (\S+):\S+ ' + r'MODELS (\S+):(\S+) \- (\S+):\S+ AT (\S+)%') template_info = [] with open(pdbname) as fh: @@ -382,7 +382,7 @@ def _handle_template(self, info, template_path_map, target_dataset, # Assume a code of 1abc, 1abc_N, 1abcX, or 1abcX_N refers # to a real PDB structure - m = re.match('(\d[a-zA-Z0-9]{3})[a-zA-Z]?(_.*)?$', template_code) + m = re.match(r'(\d[a-zA-Z0-9]{3})[a-zA-Z]?(_.*)?$', template_code) if m: template_db_code = m.group(1).upper() l = location.PDBLocation(template_db_code) diff --git a/modules/core/dependency/python-ihm/ihm/model.py b/modules/core/dependency/python-ihm/ihm/model.py index 1dd4deca88..bd9f13fdee 100644 --- a/modules/core/dependency/python-ihm/ihm/model.py +++ b/modules/core/dependency/python-ihm/ihm/model.py @@ -43,19 +43,21 @@ class Atom(object): :param float z: z coordinate of the atom :param bool het: True for HETATM sites, False (default) for ATOM :param float biso: Temperature factor or equivalent (if applicable) + :param float occupancy: Fraction of the atom type present (if applicable) """ # Reduce memory usage __slots__ = ['asym_unit', 'seq_id', 'atom_id', 'type_symbol', - 'x', 'y', 'z', 'het', 'biso'] + 'x', 'y', 'z', 'het', 'biso', 'occupancy'] def __init__(self, asym_unit, seq_id, atom_id, type_symbol, x, y, z, - het=False, biso=None): + het=False, biso=None, occupancy=None): self.asym_unit = asym_unit self.seq_id, self.atom_id = seq_id, atom_id self.type_symbol = type_symbol self.x, self.y, self.z = x, y, z self.het, self.biso = het, biso + self.occupancy = occupancy class Model(object): @@ -182,15 +184,17 @@ class Ensemble(object): for the entire ensemble, for example as a DCD file (see :class:`DCDWriter`). :type file: :class:`ihm.location.OutputFileLocation` + :param str details: Additional text describing this ensemble """ def __init__(self, model_group, num_models, post_process=None, clustering_method=None, clustering_feature=None, name=None, - precision=None, file=None): + precision=None, file=None, details=None): self.model_group, self.num_models = model_group, num_models self.post_process = post_process self.clustering_method = clustering_method self.clustering_feature = clustering_feature self.name, self.precision, self.file = name, precision, file + self.details = details #: All localization densities for this ensemble, as #: :class:`LocalizationDensity` objects diff --git a/modules/core/dependency/python-ihm/ihm/protocol.py b/modules/core/dependency/python-ihm/ihm/protocol.py index 79f611c81d..67a5aee627 100644 --- a/modules/core/dependency/python-ihm/ihm/protocol.py +++ b/modules/core/dependency/python-ihm/ihm/protocol.py @@ -22,11 +22,12 @@ class Step(object): :param bool multi_scale: Indicates if the modeling is multi-scale :param bool multi_state: Indicates if the modeling is multi-state :param bool ordered: Indicates if the modeling is ordered + :param str description: Additional text describing the step """ def __init__(self, assembly, dataset_group, method, num_models_begin=None, num_models_end=None, software=None, script_file=None, multi_scale=False, multi_state=False, ordered=False, - name=None): + name=None, description=None): self.assembly = assembly self.dataset_group = dataset_group self.method = method @@ -35,6 +36,7 @@ def __init__(self, assembly, dataset_group, method, num_models_begin=None, self.multi_scale, self.multi_state = multi_scale, multi_state self.software, self.ordered, self.name = software, ordered, name self.script_file = script_file + self.description = description class Protocol(object): diff --git a/modules/core/dependency/python-ihm/ihm/reader.py b/modules/core/dependency/python-ihm/ihm/reader.py index b76635bf8e..dca0b0a8b4 100644 --- a/modules/core/dependency/python-ihm/ihm/reader.py +++ b/modules/core/dependency/python-ihm/ihm/reader.py @@ -1,4 +1,4 @@ -"""Utility classes to read in information in mmCIF format""" +"""Utility classes to read in information in mmCIF or BinaryCIF format""" import ihm.format import ihm.format_bcif @@ -21,6 +21,13 @@ except ImportError: _format = None + +class OldFileError(Exception): + """Exception raised if a file conforms to too old a version of the + IHM extension dictionary. See :func:`read`.""" + pass + + def _make_new_entity(): """Make a new Entity object""" e = ihm.Entity([]) @@ -215,8 +222,8 @@ def _update_old_object(self, obj, newcls=None): and not hasattr(obj, 'atoms'): obj.atoms = [] elif newcls is ihm.restraint.NonPolyFeature \ - and not hasattr(obj, 'asyms'): - obj.asyms = [] + and not hasattr(obj, 'objs'): + obj.objs = [] elif newcls is ihm.restraint.PseudoSiteFeature \ and not hasattr(obj, 'x'): obj.x = obj.y = obj.z = None @@ -560,10 +567,15 @@ def __init__(self, model_class, starting_model_class): #: Mapping from ID to :class:`ihm.flr.FLRData` objects self.flr_data = IDMapper(self.system.flr_data, ihm.flr.FLRData) - #: Mapping from ID to :class:`ihm.flr.ExpSetting` objects - self.flr_exp_settings = _FLRIDMapper('_collection_flr_exp_setting', + #: Mapping from ID to :class:`ihm.flr.InstSetting` objects + self.flr_inst_settings = _FLRIDMapper('_collection_flr_inst_setting', None, self.flr_data, - ihm.flr.ExpSetting) + ihm.flr.InstSetting) + + #: Mapping from ID to :class:`ihm.flr.ExpCondition` objects + self.flr_exp_conditions = _FLRIDMapper('_collection_flr_exp_condition', + None, self.flr_data, + ihm.flr.ExpCondition) #: Mapping from ID to :class:`ihm.flr.Instrument` objects self.flr_instruments = _FLRIDMapper('_collection_flr_instrument', @@ -625,7 +637,31 @@ def __init__(self, model_class, starting_model_class): #: Mapping from ID to :class:`ihm.flr.FRETAnalysis` objects self.flr_fret_analyses = _FLRIDMapper('_collection_flr_fret_analysis', None, self.flr_data, ihm.flr.FRETAnalysis, - *(None,)*10) + *(None,)*9) + + #: Mapping from ID to :class:`ihm.flr.LifetimeFitModel` objects + self.flr_lifetime_fit_models = _FLRIDMapper( + '_collection_flr_lifetime_fit_model', + None, self.flr_data, ihm.flr.LifetimeFitModel, + *(None,)*4) + + #: Mapping from ID to :class:`ihm.flr.RefMeasurementGroup` objects + self.flr_ref_measurement_groups = _FLRIDMapper( + '_collection_flr_ref_measurement_group', + None, self.flr_data, ihm.flr.RefMeasurementGroup, + *(None,)) + + #: Mapping from ID to :class:`ihm.flr.RefMeasurement` objects + self.flr_ref_measurements = _FLRIDMapper( + '_collection_flr_ref_measurement', + None, self.flr_data, ihm.flr.RefMeasurement, + *(None,)*3) + + #: Mapping from ID to :class:`ihm.flr.RefMeasurementLifetime` objects + self.flr_ref_measurement_lifetimes = _FLRIDMapper( + '_collection_flr_ref_measurement_lifetime', + None, self.flr_data, ihm.flr.RefMeasurementLifetime, + *(None,)*3) #: Mapping from ID to :class:`ihm.flr.PeakAssignment` objects self.flr_peak_assignments = _FLRIDMapper( @@ -784,6 +820,12 @@ def end_save_frame(self): """Called at the end of each save frame.""" pass + def _get_asym_or_entity(self, asym_id, entity_id): + """Return an :class:`AsymUnit`, or an :class:`Entity` + if asym_id is omitted""" + asym = self.sysr.asym_units.get_by_id_or_none(asym_id) + return asym if asym else self.sysr.entities.get_by_id(entity_id) + def copy_if_present(self, obj, data, keys=[], mapkeys={}): """Set obj.x from data['x'] for each x in keys if present in data. The dict mapkeys is handled similarly except that its keys are looked @@ -809,6 +851,24 @@ def __call__(self, title, entry_id): mapkeys={'entry_id': 'id'}) +class _AuditConformHandler(Handler): + category = '_audit_conform' + + def __call__(self, dict_name, dict_version): + # Reject old file versions if we can parse the version + if dict_name == 'ihm-extension.dic': + try: + major, minor = [int(x) for x in dict_version.split('.')] + if (major, minor) < (1, 0): + raise OldFileError( + "This version of python-ihm only supports reading " + "files that conform to version 1.0 or later of the " + "IHM extension dictionary. This file conforms to " + "version %s." % dict_version) + except ValueError: + pass + + class _SoftwareHandler(Handler): category = '_software' @@ -1121,12 +1181,14 @@ def __init__(self, *args): self.type_map = {'doi':ihm.location.Repository, 'supplementary files':_LocalFiles} - def __call__(self, reference_id, reference_type, reference, associated_url): + def __call__(self, reference_id, reference_type, reference, associated_url, + details): ref_id = reference_id typ = 'doi' if reference_type is None else reference_type.lower() repo = self.sysr.repos.get_by_id(ref_id, self.type_map.get(typ, ihm.location.Repository)) self.copy_if_present(repo, locals(), + keys=('details',), mapkeys={'reference':'doi', 'associated_url':'url'}) def finalize(self): @@ -1253,24 +1315,26 @@ def __call__(self, id, name, details): self.copy_if_present(rep, locals(), keys=('name', 'details')) -def _make_atom_segment(asym, rigid, primitive, count, smodel): +def _make_atom_segment(asym, rigid, primitive, count, smodel, description): return ihm.representation.AtomicSegment( - asym_unit=asym, rigid=rigid, starting_model=smodel) + asym_unit=asym, rigid=rigid, starting_model=smodel, + description=description) -def _make_residue_segment(asym, rigid, primitive, count, smodel): +def _make_residue_segment(asym, rigid, primitive, count, smodel, description): return ihm.representation.ResidueSegment( asym_unit=asym, rigid=rigid, primitive=primitive, - starting_model=smodel) + starting_model=smodel, description=description) -def _make_multi_residue_segment(asym, rigid, primitive, count, smodel): +def _make_multi_residue_segment(asym, rigid, primitive, count, smodel, + description): return ihm.representation.MultiResidueSegment( asym_unit=asym, rigid=rigid, primitive=primitive, - starting_model=smodel) + starting_model=smodel, description=description) -def _make_feature_segment(asym, rigid, primitive, count, smodel): +def _make_feature_segment(asym, rigid, primitive, count, smodel, description): return ihm.representation.FeatureSegment( asym_unit=asym, rigid=rigid, primitive=primitive, - count=count, starting_model=smodel) + count=count, starting_model=smodel, description=description) class _ModelRepresentationDetailsHandler(Handler): @@ -1285,7 +1349,8 @@ class _ModelRepresentationDetailsHandler(Handler): def __call__(self, entity_asym_id, entity_poly_segment_id, representation_id, starting_model_id, model_object_primitive, - model_granularity, model_object_count, model_mode): + model_granularity, model_object_count, model_mode, + description): asym = self.sysr.ranges.get( self.sysr.asym_units.get_by_id(entity_asym_id), entity_poly_segment_id) @@ -1298,7 +1363,7 @@ def __call__(self, entity_asym_id, entity_poly_segment_id, count = self.get_int(model_object_count) rigid = self._rigid_map[self.get_lower(model_mode)] segment = self._segment_factory[gran](asym, rigid, primitive, - count, smodel) + count, smodel, description) rep.append(segment) @@ -1309,7 +1374,7 @@ class _StartingModelDetailsHandler(Handler): def __call__(self, starting_model_id, asym_id, entity_poly_segment_id, dataset_list_id, starting_model_auth_asym_id, - starting_model_sequence_offset): + starting_model_sequence_offset, description): m = self.sysr.starting_models.get_by_id(starting_model_id) asym = self.sysr.ranges.get( self.sysr.asym_units.get_by_id(asym_id), @@ -1317,6 +1382,7 @@ def __call__(self, starting_model_id, asym_id, entity_poly_segment_id, m.asym_unit = asym m.dataset = self.sysr.datasets.get_by_id(dataset_list_id) self.copy_if_present(m, locals(), + keys=('description',), mapkeys={'starting_model_auth_asym_id':'asym_id'}) if starting_model_sequence_offset is not None: m.offset = int(starting_model_sequence_offset) @@ -1375,7 +1441,8 @@ class _ProtocolDetailsHandler(Handler): def __call__(self, protocol_id, step_id, num_models_begin, num_models_end, multi_scale_flag, multi_state_flag, ordered_flag, struct_assembly_id, dataset_group_id, - software_id, script_file_id, step_name, step_method): + software_id, script_file_id, step_name, step_method, + description): p = self.sysr.protocols.get_by_id(protocol_id) nbegin = self.get_int(num_models_begin) nend = self.get_int(num_models_end) @@ -1391,7 +1458,8 @@ def __call__(self, protocol_id, step_id, num_models_begin, method=None, num_models_begin=nbegin, num_models_end=nend, multi_scale=mscale, multi_state=mstate, ordered=ordered, - software=software, script_file=script) + software=software, script_file=script, + description=description) s._id = step_id self.copy_if_present(s, locals(), mapkeys={'step_name':'name', 'step_method':'method'}) @@ -1413,7 +1481,7 @@ def __init__(self, *args): def __call__(self, protocol_id, analysis_id, type, id, num_models_begin, num_models_end, struct_assembly_id, dataset_group_id, - software_id, script_file_id, feature): + software_id, script_file_id, feature, details): protocol = self.sysr.protocols.get_by_id(protocol_id) analysis = self.sysr.analyses.get_by_id(analysis_id) if analysis._id not in [a._id for a in protocol.analyses]: @@ -1423,6 +1491,7 @@ def __call__(self, protocol_id, analysis_id, type, id, num_models_begin, step = self.sysr.analysis_steps.get_by_id(id, self.type_map.get(typ, ihm.analysis.Step)) analysis.steps.append(step) + step.details = details if typ == 'none': # If this step was forward referenced, feature will have been set @@ -1520,9 +1589,10 @@ class _EnsembleHandler(Handler): def __call__(self, ensemble_id, model_group_id, post_process_id, ensemble_file_id, num_ensemble_models, ensemble_precision_value, ensemble_name, - ensemble_clustering_method, ensemble_clustering_feature): + ensemble_clustering_method, ensemble_clustering_feature, + details): ensemble = self.sysr.ensembles.get_by_id(ensemble_id) - mg = self.sysr.model_groups.get_by_id(model_group_id) + mg = self.sysr.model_groups.get_by_id_or_none(model_group_id) pp = self.sysr.analysis_steps.get_by_id_or_none(post_process_id) f = self.sysr.external_files.get_by_id_or_none(ensemble_file_id) @@ -1533,6 +1603,7 @@ def __call__(self, ensemble_id, model_group_id, post_process_id, # model group anyway) ensemble.post_process = pp ensemble.file = f + ensemble.details = details self.copy_if_present(ensemble, locals(), mapkeys={'ensemble_name':'name', 'ensemble_clustering_method':'clustering_method', @@ -1660,12 +1731,13 @@ class _AtomSiteHandler(Handler): def __call__(self, pdbx_pdb_model_num, label_asym_id, b_iso_or_equiv, label_seq_id, label_atom_id, type_symbol, cartn_x, cartn_y, - cartn_z, group_pdb, auth_seq_id): + cartn_z, occupancy, group_pdb, auth_seq_id): # todo: handle fields other than those output by us # todo: handle insertion codes model = self.sysr.models.get_by_id(pdbx_pdb_model_num) asym = self.sysr.asym_units.get_by_id(label_asym_id) biso = self.get_float(b_iso_or_equiv) + occupancy = self.get_float(occupancy) # seq_id can be None for non-polymers (HETATM) seq_id = self.get_int(label_seq_id) group = 'ATOM' if group_pdb is None else group_pdb @@ -1675,7 +1747,7 @@ def __call__(self, pdbx_pdb_model_num, label_asym_id, b_iso_or_equiv, type_symbol=type_symbol, x=float(cartn_x), y=float(cartn_y), z=float(cartn_z), het=group != 'ATOM', - biso=biso) + biso=biso, occupancy=occupancy) model.add_atom(a) auth_seq_id = self.get_int_or_string(auth_seq_id) @@ -1723,41 +1795,51 @@ def __call__(self, starting_model_id, db_seq_id, seq_id, db_comp_id, class _PolyResidueFeatureHandler(Handler): category = '_ihm_poly_residue_feature' - def __call__(self, feature_id, asym_id, seq_id_begin, seq_id_end): + def __call__(self, feature_id, entity_id, asym_id, seq_id_begin, + seq_id_end): f = self.sysr.features.get_by_id( feature_id, ihm.restraint.ResidueFeature) - asym = self.sysr.asym_units.get_by_id(asym_id) + asym_or_entity = self._get_asym_or_entity(asym_id, entity_id) r1 = int(seq_id_begin) r2 = int(seq_id_end) - f.ranges.append(asym(r1,r2)) + f.ranges.append(asym_or_entity(r1,r2)) + + +class _FeatureListHandler(Handler): + category = '_ihm_feature_list' + + def __call__(self, feature_id, details): + if details: + f = self.sysr.features.get_by_id(feature_id) + f.details = details class _PolyAtomFeatureHandler(Handler): category = '_ihm_poly_atom_feature' - def __call__(self, feature_id, asym_id, seq_id, atom_id): + def __call__(self, feature_id, entity_id, asym_id, seq_id, atom_id): f = self.sysr.features.get_by_id( feature_id, ihm.restraint.AtomFeature) - asym = self.sysr.asym_units.get_by_id(asym_id) + asym_or_entity = self._get_asym_or_entity(asym_id, entity_id) seq_id = int(seq_id) - atom = asym.residue(seq_id).atom(atom_id) + atom = asym_or_entity.residue(seq_id).atom(atom_id) f.atoms.append(atom) class _NonPolyFeatureHandler(Handler): category = '_ihm_non_poly_feature' - def __call__(self, feature_id, asym_id, atom_id): - asym = self.sysr.asym_units.get_by_id(asym_id) + def __call__(self, feature_id, entity_id, asym_id, atom_id): + asym_or_entity = self._get_asym_or_entity(asym_id, entity_id) if atom_id is None: f = self.sysr.features.get_by_id( feature_id, ihm.restraint.NonPolyFeature) - f.asyms.append(asym) + f.objs.append(asym_or_entity) else: f = self.sysr.features.get_by_id( feature_id, ihm.restraint.AtomFeature) # todo: handle multiple copies, e.g. waters? - atom = asym.residue(1).atom(atom_id) + atom = asym_or_entity.residue(1).atom(atom_id) f.atoms.append(atom) @@ -1806,7 +1888,8 @@ class _DerivedDistanceRestraintHandler(Handler): def __call__(self, id, group_id, dataset_list_id, feature_id_1, feature_id_2, restraint_type, group_conditionality, - probability, distance_lower_limit, distance_upper_limit): + probability, mic_value, distance_lower_limit, + distance_upper_limit): r = self.sysr.dist_restraints.get_by_id(id) if group_id is not None: rg = self.sysr.dist_restraint_groups.get_by_id(group_id) @@ -1819,6 +1902,7 @@ def __call__(self, id, group_id, dataset_list_id, feature_id_1, self.get_float) r.restrain_all = self._cond_map[group_conditionality] r.probability = self.get_float(probability) + r.mic_value = self.get_float(mic_value) class _PredictedContactRestraintHandler(Handler): @@ -1883,15 +1967,13 @@ class _GeometricObjectHandler(Handler): if issubclass(x[1], ihm.geometry.GeometricObject) and ihm.geometry.GeometricObject in x[1].__bases__) - def __call__(self, object_type, object_id, object_name, object_description, - other_details): + def __call__(self, object_type, object_id, object_name, object_description): typ = object_type.lower() if object_type is not None else 'other' g = self.sysr.geometries.get_by_id(object_id, self._type_map.get(typ, ihm.geometry.GeometricObject)) self.copy_if_present(g, locals(), mapkeys={'object_name': 'name', - 'object_description': 'description', - 'other_details': 'details'}) + 'object_description': 'description'}) class _SphereHandler(Handler): @@ -2081,7 +2163,7 @@ def _get_linker_by_name(self, name): def __call__(self, dataset_list_id, linker_chem_comp_descriptor_id, group_id, id, entity_id_1, entity_id_2, seq_id_1, seq_id_2, - linker_type): + linker_type, details): dataset = self.sysr.datasets.get_by_id_or_none(dataset_list_id) if linker_chem_comp_descriptor_id is None and linker_type is not None: linker = self._get_linker_by_name(linker_type) @@ -2101,6 +2183,7 @@ def __call__(self, dataset_list_id, linker_chem_comp_descriptor_id, xl_group.append(xl) xl.residue1 = self._get_entity_residue(entity_id_1, seq_id_1) xl.residue2 = self._get_entity_residue(entity_id_2, seq_id_2) + xl.details = details def _get_entity_residue(self, entity_id, seq_id): entity = self.sysr.entities.get_by_id(entity_id) @@ -2247,25 +2330,37 @@ class _FLRExperimentHandler(Handler): category = '_flr_experiment' def __call__(self, ordinal_id, id, instrument_id, - exp_setting_id, sample_id, details): + inst_setting_id, exp_condition_id, + sample_id, details): # Get the object or create the object experiment = self.sysr.flr_experiments.get_by_id(id) # Fill the object instrument = self.sysr.flr_instruments.get_by_id(instrument_id) - exp_setting = self.sysr.flr_exp_settings.get_by_id(exp_setting_id) + inst_setting = self.sysr.flr_inst_settings.get_by_id(inst_setting_id) + exp_condition = self.sysr.flr_exp_conditions.get_by_id(exp_condition_id) sample = self.sysr.flr_samples.get_by_id(sample_id) - experiment.add_entry(instrument=instrument, exp_setting=exp_setting, - sample=sample, details=details) + experiment.add_entry(instrument=instrument, inst_setting=inst_setting, + exp_condition=exp_condition, sample=sample, details=details) + + +class _FLRInstSettingHandler(Handler): + category = '_flr_inst_setting' + + def __call__(self, id, details): + # Get the object or create the object + cur_inst_setting = self.sysr.flr_inst_settings.get_by_id(id) + # Set the variables + self.copy_if_present(cur_inst_setting, locals(), keys=('details',)) -class _FLRExpSettingHandler(Handler): - category = '_flr_exp_setting' +class _FLRExpConditionHandler(Handler): + category = '_flr_exp_condition' def __call__(self, id, details): # Get the object or create the object - cur_exp_setting = self.sysr.flr_exp_settings.get_by_id(id) + cur_exp_condition = self.sysr.flr_exp_conditions.get_by_id(id) # Set the variables - self.copy_if_present(cur_exp_setting, locals(), keys=('details',)) + self.copy_if_present(cur_exp_condition, locals(), keys=('details',)) class _FLRInstrumentHandler(Handler): @@ -2446,27 +2541,111 @@ def __call__(self, id, phi_acceptor, alpha, alpha_sd, gg_gr_ratio, beta, class _FLRFretAnalysisHandler(Handler): category = '_flr_fret_analysis' - def __call__(self, id, experiment_id, sample_probe_id_1, sample_probe_id_2, - forster_radius_id, calibration_parameters_id, method_name, - chi_square_reduced, dataset_list_id, external_file_id, - software_id): + def __call__(self, id, experiment_id, type, + sample_probe_id_1, sample_probe_id_2, + forster_radius_id, dataset_list_id, + external_file_id, software_id): f = self.sysr.flr_fret_analyses.get_by_id(id) f.experiment = self.sysr.flr_experiments.get_by_id(experiment_id) + f.type = type f.sample_probe_1 = self.sysr.flr_sample_probe_details.get_by_id( sample_probe_id_1) f.sample_probe_2 = self.sysr.flr_sample_probe_details.get_by_id( sample_probe_id_2) f.forster_radius = self.sysr.flr_fret_forster_radius.get_by_id( forster_radius_id) - f.calibration_parameters \ - = self.sysr.flr_fret_calibration_parameters.get_by_id( - calibration_parameters_id) f.dataset = self.sysr.datasets.get_by_id(dataset_list_id) f.external_file = self.sysr.external_files.get_by_id_or_none( external_file_id) f.software = self.sysr.software.get_by_id_or_none(software_id) + + +class _FLRFretAnalysisIntensityHandler(Handler): + category = '_flr_fret_analysis_intensity' + + def __call__(self, ordinal_id, analysis_id, + calibration_parameters_id, donor_only_fraction, + chi_square_reduced, method_name, details): + f = self.sysr.flr_fret_analyses.get_by_id(analysis_id) + f.type = 'intensity-based' + f.calibration_parameters = \ + self.sysr.flr_fret_calibration_parameters.get_by_id(calibration_parameters_id) + f.donor_only_fraction = self.get_float(donor_only_fraction) + f.chi_square_reduced = self.get_float(chi_square_reduced) f.method_name = method_name + f.details = details + + +class _FLRFretAnalysisLifetimeHandler(Handler): + category = '_flr_fret_analysis_lifetime' + + def __call__(self, ordinal_id, analysis_id, + reference_measurement_group_id, lifetime_fit_model_id, + donor_only_fraction, chi_square_reduced, method_name, details): + f = self.sysr.flr_fret_analyses.get_by_id(analysis_id) + f.type = 'lifetime-based' + f.reference_measurement_group = self.sysr.flr_ref_measurement_groups.get_by_id(reference_measurement_group_id) + f.lifetime_fit_model = self.sysr.flr_lifetime_fit_models.get_by_id(lifetime_fit_model_id) + f.donor_only_fraction = self.get_float(donor_only_fraction) f.chi_square_reduced = self.get_float(chi_square_reduced) + f.method_name = method_name + f.details = details + + +class _FLRLifetimeFitModelHandler(Handler): + category = '_flr_lifetime_fit_model' + + def __call__(self, id, name, description, + external_file_id, citation_id): + f = self.sysr.flr_lifetime_fit_models.get_by_id(id) + f.name = name + f.description = description + f.external_file = \ + self.sysr.external_files.get_by_id_or_none(external_file_id) + f.citation = \ + self.sysr.citations.get_by_id_or_none(citation_id) + + +class _FLRRefMeasurementHandler(Handler): + category = '_flr_reference_measurement' + + def __call__(self, id, reference_sample_probe_id, + num_species, details): + r = self.sysr.flr_ref_measurements.get_by_id(id) + r.ref_sample_probe = self.sysr.flr_sample_probe_details.get_by_id(reference_sample_probe_id) + r.details = details + + +class _FLRRefMeasurementGroupHandler(Handler): + category = '_flr_reference_measurement_group' + + def __call__(self, id, num_measurements, details): + g = self.sysr.flr_ref_measurement_groups.get_by_id(id) + g.details = details + + +class _FLRRefMeasurementGroupLinkHandler(Handler): + category = '_flr_reference_measurement_group_link' + + def __call__(self, group_id, reference_measurement_id): + g = self.sysr.flr_ref_measurement_groups.get_by_id(group_id) + r = self.sysr.flr_ref_measurements.get_by_id(reference_measurement_id) + g.add_ref_measurement(r) + + +class _FLRRefMeasurementLifetimeHandler(Handler): + category = '_flr_reference_measurement_lifetime' + + def __call__(self, ordinal_id, reference_measurement_id, + species_name, species_fraction, lifetime): + l = self.sysr.flr_ref_measurement_lifetimes.get_by_id(ordinal_id) + l.species_name = species_name + l.species_fraction = self.get_float(species_fraction) + l.lifetime = self.get_float(lifetime) + + ## Add the lifetime to the reference measurement + r = self.sysr.flr_ref_measurements.get_by_id(reference_measurement_id) + r.add_lifetime(l) class _FLRPeakAssignmentHandler(Handler): @@ -2652,10 +2831,11 @@ def __call__(self, ordinal_id, fps_modeling_id, mpp_id, def read(fh, model_class=ihm.model.Model, format='mmCIF', handlers=[], warn_unknown_category=False, warn_unknown_keyword=False, read_starting_model_coord=True, - starting_model_class=ihm.startmodel.StartingModel): - """Read data from the mmCIF file handle `fh`. + starting_model_class=ihm.startmodel.StartingModel, + reject_old_file=False): + """Read data from the file handle `fh`. - Note that the reader currently expects to see an mmCIF file compliant + Note that the reader currently expects to see a file compliant with the PDBx and/or IHM dictionaries. It is not particularly tolerant of noncompliant or incomplete files, and will probably throw an exception rather than warning about and trying to handle such files. @@ -2670,7 +2850,8 @@ def read(fh, model_class=ihm.model.Model, format='mmCIF', handlers=[], so is used if built. The BinaryCIF reader needs the msgpack Python module to function. - :param file fh: The file handle to read from. + :param file fh: The file handle to read from. (For BinaryCIF files, + the file should be opened in binary mode.) :param model_class: The class to use to store model information (such as coordinates). For use with other software, it is recommended to subclass :class:`ihm.model.Model` and override @@ -2696,6 +2877,10 @@ def read(fh, model_class=ihm.model.Model, format='mmCIF', handlers=[], is recommended to subclass :class:`ihm.startmodel.StartingModel` and override :meth:`~ihm.startmodel.StartingModel.add_atom` and/or :meth:`~ihm.startmodel.StartingModel.add_seq_dif`. + :param bool reject_old_file: If True, raise an + :exc:`ihm.reader.OldFileError` if the file conforms to an + older version of the dictionary than this library supports + (by default the library will read what it can from the file). :return: A list of :class:`ihm.System` objects. """ systems = [] @@ -2735,6 +2920,7 @@ def read(fh, model_class=ihm.model.Model, format='mmCIF', handlers=[], _EM3DRestraintHandler(s), _EM2DRestraintHandler(s), _EM2DFittingHandler(s), _SASRestraintHandler(s), _SphereObjSiteHandler(s), _AtomSiteHandler(s), + _FeatureListHandler(s), _PolyResidueFeatureHandler(s), _PolyAtomFeatureHandler(s), _NonPolyFeatureHandler(s), _PseudoSiteFeatureHandler(s), _DerivedDistanceRestraintHandler(s), @@ -2746,7 +2932,8 @@ def read(fh, model_class=ihm.model.Model, format='mmCIF', handlers=[], _CrossLinkListHandler(s), _CrossLinkRestraintHandler(s), _CrossLinkResultHandler(s), _StartingModelSeqDifHandler(s), _OrderedEnsembleHandler(s), _FLRChemDescriptorHandler(s), - _FLRExpSettingHandler(s), + _FLRInstSettingHandler(s), + _FLRExpConditionHandler(s), _FLRInstrumentHandler(s), _FLRSampleConditionHandler(s), _FLREntityAssemblyHandler(s), @@ -2762,6 +2949,13 @@ def read(fh, model_class=ihm.model.Model, format='mmCIF', handlers=[], _FLRFretForsterRadiusHandler(s), _FLRFretCalibrationParametersHandler(s), _FLRFretAnalysisHandler(s), + _FLRFretAnalysisIntensityHandler(s), + _FLRFretAnalysisLifetimeHandler(s), + _FLRLifetimeFitModelHandler(s), + _FLRRefMeasurementHandler(s), + _FLRRefMeasurementGroupHandler(s), + _FLRRefMeasurementGroupLinkHandler(s), + _FLRRefMeasurementLifetimeHandler(s), _FLRPeakAssignmentHandler(s), _FLRFretDistanceRestraintHandler(s), _FLRFretModelQualityHandler(s), @@ -2773,6 +2967,8 @@ def read(fh, model_class=ihm.model.Model, format='mmCIF', handlers=[], _FLRFPSMPPHandler(s), _FLRFPSMPPAtomPositionHandler(s), _FLRFPSMPPModelingHandler(s)] + [h(s) for h in handlers] + if reject_old_file: + hs.append(_AuditConformHandler(s)) if read_starting_model_coord: hs.append(_StartingModelCoordHandler(s)) if uchandler: diff --git a/modules/core/dependency/python-ihm/ihm/representation.py b/modules/core/dependency/python-ihm/ihm/representation.py index 5ba39205fc..fd8a5a841e 100644 --- a/modules/core/dependency/python-ihm/ihm/representation.py +++ b/modules/core/dependency/python-ihm/ihm/representation.py @@ -21,15 +21,17 @@ class AtomicSegment(object): :param starting_model: initial coordinates used for the segment (or None). :type starting_model: :class:`~ihm.startmodel.StartingModel` + :param str description: Additional text describing this segment. """ primitive = 'atomistic' count = None granularity = 'by-atom' - def __init__(self, asym_unit, rigid, starting_model=None): + def __init__(self, asym_unit, rigid, starting_model=None, description=None): self.asym_unit = asym_unit self.starting_model, self.rigid = starting_model, rigid + self.description = description class ResidueSegment(object): @@ -46,15 +48,18 @@ class ResidueSegment(object): :param starting_model: initial coordinates used for the segment (or None). :type starting_model: :class:`~ihm.startmodel.StartingModel` + :param str description: Additional text describing this segment. """ count = None granularity = 'by-residue' - def __init__(self, asym_unit, rigid, primitive, starting_model=None): + def __init__(self, asym_unit, rigid, primitive, starting_model=None, + description=None): self.asym_unit = asym_unit self.primitive = primitive self.starting_model, self.rigid = starting_model, rigid + self.description = description class MultiResidueSegment(object): @@ -71,15 +76,18 @@ class MultiResidueSegment(object): :param starting_model: initial coordinates used for the segment (or None). :type starting_model: :class:`~ihm.startmodel.StartingModel` + :param str description: Additional text describing this segment. """ count = None granularity = 'multi-residue' - def __init__(self, asym_unit, rigid, primitive, starting_model=None): + def __init__(self, asym_unit, rigid, primitive, starting_model=None, + description=None): self.asym_unit = asym_unit self.primitive = primitive self.starting_model, self.rigid = starting_model, rigid + self.description = description class FeatureSegment(object): @@ -97,14 +105,17 @@ class FeatureSegment(object): :param starting_model: initial coordinates used for the segment (or None). :type starting_model: :class:`~ihm.startmodel.StartingModel` + :param str description: Additional text describing this segment. """ granularity = 'by-feature' - def __init__(self, asym_unit, rigid, primitive, count, starting_model=None): + def __init__(self, asym_unit, rigid, primitive, count, starting_model=None, + description=None): self.asym_unit = asym_unit self.primitive, self.count = primitive, count self.starting_model, self.rigid = starting_model, rigid + self.description = description class Representation(list): diff --git a/modules/core/dependency/python-ihm/ihm/restraint.py b/modules/core/dependency/python-ihm/ihm/restraint.py index b539bea2d8..7087e5774d 100644 --- a/modules/core/dependency/python-ihm/ihm/restraint.py +++ b/modules/core/dependency/python-ihm/ihm/restraint.py @@ -1,6 +1,9 @@ """Classes for handling restraints on the system. """ +import ihm + + class Restraint(object): """Base class for all restraints. See :attr:`ihm.System.restraints`. @@ -206,7 +209,8 @@ def __init__(self, dataset, linker): #: restraint.experimental_cross_links.append([xl2, xl3]) self.experimental_cross_links = [] - #: All cross-links used in the modeling + #: All cross-links used in the modeling, as a list of + #: :class:`CrossLink` objects. self.cross_links = [] @@ -217,9 +221,11 @@ class ExperimentalCrossLink(object): :type residue1: :class:`ihm.Residue` :param residue2: The second residue linked by the cross-link. :type residue2: :class:`ihm.Residue` + :param str details: Additional text describing the cross-link. """ - def __init__(self, residue1, residue2): + def __init__(self, residue1, residue2, details=None): self.residue1, self.residue2 = residue1, residue2 + self.details = details class DistanceRestraint(object): @@ -293,8 +299,9 @@ def __init__(self, distance_lower_limit, distance_upper_limit): class CrossLink(object): """Base class for all cross-links used in the modeling. - See :class:`ResidueCrossLink`, :class:`AtomCrossLink`, - :class:`FeatureCrossLink`.""" + Do not use this class directly, but instead use a subclass: + :class:`ResidueCrossLink`, :class:`AtomCrossLink`, + or :class:`FeatureCrossLink`.""" pass @@ -431,16 +438,27 @@ class Feature(object): :class:`NonPolyFeature`, and :class:`PseudoSiteFeature`. Features are typically assigned to one or more - :class:`~ihm.restraint.GeometricRestraint` objects. + :class:`~ihm.restraint.GeometricRestraint` or + :class:`~ihm.restraint.DerivedDistanceRestraint` objects. """ - pass + details = None + def _all_entities_or_asyms(self): + # Get all Entities or AsymUnits referenced by this object + return [] class ResidueFeature(Feature): """Selection of one or more residues from the system. - :param sequence ranges: A list of :class:`AsymUnitRange` and/or - :class:`AsymUnit` objects. + Residues can be selected from both :class:`AsymUnit` and + :class:`Entity` (the latter implies that it selects residues + in all instances of that entity). Individual residues can + also be selected by passing :class:`Residue` objects. + + :param sequence ranges: A list of :class:`AsymUnitRange`, + :class:`AsymUnit`, :class:`EntityRange`, :class:`Residue`, + and/or :class:`Entity` objects. + :param str details: Additional text describing this feature. """ # Type is 'residue' if each range selects a single residue, otherwise @@ -452,59 +470,81 @@ def __get_type(self): return 'residue' type = property(__get_type) - def __init__(self, ranges): - self.ranges = ranges + def __init__(self, ranges, details=None): + self.ranges, self.details = ranges, details _ = self._get_entity_type() + def _all_entities_or_asyms(self): + return self.ranges + def _get_entity_type(self): - if any(not r.entity.is_polymeric() for r in self.ranges): + def _get_entity(x): + return x if isinstance(x, ihm.Entity) else x.entity + if any(not _get_entity(r).is_polymeric() for r in self.ranges): raise ValueError("%s cannot select non-polymeric entities" % self) else: - return self.ranges[0].entity.type if self.ranges else None + return _get_entity(self.ranges[0]).type if self.ranges else None class AtomFeature(Feature): """Selection of one or more atoms from the system. Atoms can be selected from polymers or non-polymers (but not both). + Atoms can also be selected from both :class:`AsymUnit` and + :class:`Entity` (the latter implies that it selects atoms + in all instances of that entity). For selecting an entire polymer or residue(s), see :class:`ResidueFeature`. For selecting an entire non-polymer, see :class:`NonPolyFeature`. :param sequence atoms: A list of :class:`ihm.Atom` objects. + :param str details: Additional text describing this feature. """ type = 'atom' - def __init__(self, atoms): - self.atoms = atoms + def __init__(self, atoms, details=None): + self.atoms, self.details = atoms, details _ = self._get_entity_type() def _get_entity_type(self): - types = frozenset(a.residue.asym.entity.type for a in self.atoms) + def _get_entity(residue): + return residue.entity if residue.entity else residue.asym.entity + types = frozenset(_get_entity(a.residue).type for a in self.atoms) if len(types) > 1: raise ValueError("%s cannot span both polymeric and " "non-polymeric entities" % self) elif types: - return self.atoms[0].residue.asym.entity.type + return tuple(types)[0] class NonPolyFeature(Feature): """Selection of one or more non-polymers from the system. To select individual atoms from a non-polymer, see :class:`AtomFeature`. - :param sequence asyms: A list of :class:`AsymUnit` objects. + Features can include both :class:`AsymUnit` and + :class:`Entity` (the latter implies that it selects non-polymers + in all instances of that entity). + + :param sequence objs: A list of :class:`AsymUnit` and/or + :class:`Entity` objects. + :param str details: Additional text describing this feature. """ type = 'ligand' - def __init__(self, asyms): - self.asyms = asyms + def __init__(self, objs, details=None): + self.objs, self.details = objs, details _ = self._get_entity_type() + def _all_entities_or_asyms(self): + return self.objs + def _get_entity_type(self): - if any(r.entity.is_polymeric() for r in self.asyms): + def _get_entity(x): + return x if isinstance(x, ihm.Entity) else x.entity + if any(_get_entity(r).is_polymeric() for r in self.objs): raise ValueError("%s can only select non-polymeric entities" % self) else: - return self.asyms[0].entity.type if self.asyms else None + return _get_entity(self.objs[0]).type if self.objs else None class PseudoSiteFeature(Feature): @@ -514,15 +554,15 @@ class PseudoSiteFeature(Feature): :param float y: Cartesian Y coordinate of this site. :param float z: Cartesian Z coordinate of this site. :param float radius: Radius of the site, if applicable. - :param str description: Textual description of this site. + :param str details: Additional text describing this feature. """ type = 'pseudo site' - def __init__(self, x, y, z, radius=None, description=None): + def __init__(self, x, y, z, radius=None, details=None): self.x, self.y, self.z = x, y, z self.radius = radius - self.description = description + self.details = details def _get_entity_type(self): return 'other' @@ -555,6 +595,7 @@ def __init__(self, dataset, geometric_object, feature, distance, self.geometric_object, self.feature = geometric_object, feature self.distance, self.restrain_all = distance, restrain_all self.harmonic_force_constant = harmonic_force_constant + _all_features = property(lambda self: (self.feature,)) class CenterGeometricRestraint(GeometricRestraint): @@ -596,15 +637,19 @@ class DerivedDistanceRestraint(object): :type distance: :class:`DistanceRestraint` :param float probability: Likelihood that restraint is correct (0. - 1.) :param bool restrain_all: If True, all distances are restrained. + :param float mic_value: Value of the Maximal Information Coefficient + (MIC) for this interaction, if applicable. """ assembly = None # no struct_assembly_id for derived distance restraints def __init__(self, dataset, feature1, feature2, distance, - probability=None, restrain_all=None): + probability=None, restrain_all=None, mic_value=None): self.dataset = dataset self.feature1, self.feature2 = feature1, feature2 self.distance, self.restrain_all = distance, restrain_all self.probability = probability + self.mic_value = mic_value + _all_features = property(lambda self: (self.feature1, self.feature2)) class PredictedContactRestraint(object): diff --git a/modules/core/dependency/python-ihm/ihm/startmodel.py b/modules/core/dependency/python-ihm/ihm/startmodel.py index fb134a866b..8d68e2c339 100644 --- a/modules/core/dependency/python-ihm/ihm/startmodel.py +++ b/modules/core/dependency/python-ihm/ihm/startmodel.py @@ -107,14 +107,17 @@ class StartingModel(object): script used to generate the starting model (usually a :class:`~ihm.location.WorkflowFileLocation`). :type script_file: :class:`~ihm.location.Location` + :param str description: Additional text describing the starting model. """ def __init__(self, asym_unit, dataset, asym_id, templates=None, offset=0, - metadata=None, software=None, script_file=None): + metadata=None, software=None, script_file=None, + description=None): self.templates = templates if templates is not None else [] self.metadata = metadata if metadata is not None else [] self.asym_unit = asym_unit self.dataset, self.asym_id, self.offset = dataset, asym_id, offset self.software, self.script_file = software, script_file + self.description = description self._atoms = [] self._seq_difs = [] diff --git a/modules/core/dependency/python-ihm/make-release.sh b/modules/core/dependency/python-ihm/make-release.sh index d1595b0b7c..0cb9486cbc 100755 --- a/modules/core/dependency/python-ihm/make-release.sh +++ b/modules/core/dependency/python-ihm/make-release.sh @@ -3,7 +3,10 @@ # First, do # - Update ChangeLog.rst with the release number # - Update release number in ihm/__init__.py, MANIFEST.in, and setup.py -# - Commit and tag +# - Commit, tag, and push +# - Make release on GitHub +# - Upload the release tarball from +# https://github.com/ihmwg/python-ihm/releases to Zenodo as a new release # Make SWIG wrapper so users don't need SWIG rm -rf build src/ihm_format_wrap.c diff --git a/modules/core/dependency/python-ihm/setup.py b/modules/core/dependency/python-ihm/setup.py index 80508f5196..0a140a4455 100755 --- a/modules/core/dependency/python-ihm/setup.py +++ b/modules/core/dependency/python-ihm/setup.py @@ -7,7 +7,7 @@ import sys import os -VERSION = "0.9" +VERSION = "0.12" copy_args = sys.argv[1:] diff --git a/modules/core/dependency/python-ihm/test/test_dataset.py b/modules/core/dependency/python-ihm/test/test_dataset.py index 58853dbbc1..a778252758 100644 --- a/modules/core/dependency/python-ihm/test/test_dataset.py +++ b/modules/core/dependency/python-ihm/test/test_dataset.py @@ -157,6 +157,13 @@ def test_y2h_dataset(self): d = ihm.dataset.YeastTwoHybridDataset(loc) self.assertEqual(d.data_type, 'Yeast two-hybrid screening data') + def test_genetic_dataset(self): + """Test GeneticInteractionsDataset""" + loc = ihm.location.FileLocation(repo='mydoi', path='a') + d = ihm.dataset.GeneticInteractionsDataset(loc) + self.assertEqual(d.data_type, + 'Quantitative measurements of genetic interactions') + def test_duplicate_datasets_details(self): """Datasets with differing details should be considered duplicates""" with utils.temporary_directory() as tmpdir: diff --git a/modules/core/dependency/python-ihm/test/test_dumper.py b/modules/core/dependency/python-ihm/test/test_dumper.py index 97dab52301..38e8fb50ec 100644 --- a/modules/core/dependency/python-ihm/test/test_dumper.py +++ b/modules/core/dependency/python-ihm/test/test_dumper.py @@ -792,8 +792,8 @@ def test_assembly_subset_modeled(self): a1 = ihm.AsymUnit(e1) system.entities.extend((e1, e2)) system.asym_units.append(a1) - # Note that no asym unit uses entity e2, so the assembly - # should omit the chain ID ('.') + # Note that no asym unit uses entity e2, so it won't be included + # in the assembly # Assign entity and asym IDs ihm.dumper._EntityDumper().finalize(system) @@ -814,7 +814,6 @@ def test_assembly_subset_modeled(self): _ihm_entity_poly_segment.comp_id_begin _ihm_entity_poly_segment.comp_id_end 1 1 1 3 ALA GLY -2 2 1 2 GLU TRP # """) @@ -838,14 +837,13 @@ def test_assembly_subset_modeled(self): _ihm_struct_assembly_details.asym_id _ihm_struct_assembly_details.entity_poly_segment_id 1 1 1 foo 1 A 1 -2 1 1 bar 2 . 2 # """) def test_external_reference_dumper(self): """Test ExternalReferenceDumper""" system = ihm.System() - repo1 = ihm.location.Repository(doi="foo") + repo1 = ihm.location.Repository(doi="foo", details='test repo') repo2 = ihm.location.Repository(doi="10.5281/zenodo.46266", url='nup84-v1.0.zip', top_directory=os.path.join('foo', @@ -895,10 +893,11 @@ def test_external_reference_dumper(self): _ihm_external_reference_info.reference _ihm_external_reference_info.refers_to _ihm_external_reference_info.associated_url -1 . DOI foo Other . -2 Zenodo DOI 10.5281/zenodo.46266 Archive nup84-v1.0.zip -3 Zenodo DOI 10.5281/zenodo.58025 File foo.spd -4 . 'Supplementary Files' . Other . +_ihm_external_reference_info.details +1 . DOI foo Other . 'test repo' +2 Zenodo DOI 10.5281/zenodo.46266 Archive nup84-v1.0.zip . +3 Zenodo DOI 10.5281/zenodo.58025 File foo.spd . +4 . 'Supplementary Files' . Other . . # # loop_ @@ -1123,7 +1122,8 @@ def test_model_representation_dump(self): rigid=False, primitive='gaussian') s4 = ihm.representation.FeatureSegment( asym(3,4), starting_model=None, - rigid=True, primitive='other', count=3) + rigid=True, primitive='other', count=3, + description='test segment') r1 = ihm.representation.Representation((s1, s2), name='foo', details='foo details') r2 = ihm.representation.Representation((s3, s4), name='bar') @@ -1159,10 +1159,11 @@ def test_model_representation_dump(self): _ihm_model_representation_details.model_mode _ihm_model_representation_details.model_granularity _ihm_model_representation_details.model_object_count -1 1 42 bar X 1 atomistic . rigid by-atom . -2 1 42 bar X 2 sphere . flexible by-residue . -3 2 42 bar X 1 gaussian . flexible multi-residue . -4 2 42 bar X 2 other . rigid by-feature 3 +_ihm_model_representation_details.description +1 1 42 bar X 1 atomistic . rigid by-atom . . +2 1 42 bar X 2 sphere . flexible by-residue . . +3 2 42 bar X 1 gaussian . flexible multi-residue . . +4 2 42 bar X 2 other . rigid by-feature 3 'test segment' # """) @@ -1210,7 +1211,8 @@ def get_seq_dif(self): script_file=script, software=software) system.orphan_starting_models.append(sm) - sm = TestStartingModel(asym(1,15), dstarget, 'A', []) + sm = TestStartingModel(asym(1,15), dstarget, 'A', [], + description="test desc") system.orphan_starting_models.append(sm) e1._id = 42 @@ -1251,8 +1253,9 @@ def get_seq_dif(self): _ihm_starting_model_details.starting_model_auth_asym_id _ihm_starting_model_details.starting_model_sequence_offset _ihm_starting_model_details.dataset_list_id -1 42 foo 99 2 'experimental model' A 10 102 -2 42 foo 99 1 'experimental model' A 0 102 +_ihm_starting_model_details.description +1 42 foo 99 2 'experimental model' A 10 102 . +2 42 foo 99 1 'experimental model' A 0 102 'test desc' # # loop_ @@ -1344,7 +1347,8 @@ class MockObject(object): p2.steps.append(ihm.protocol.Step(assembly=assembly, dataset_group=dsg2, method='Replica exchange', num_models_begin=2000, num_models_end=1000, multi_scale=True, - software=software, script_file=script)) + software=software, script_file=script, + description='test step')) system.orphan_protocols.append(p2) dumper = ihm.dumper._ProtocolDumper() @@ -1375,9 +1379,10 @@ class MockObject(object): _ihm_modeling_protocol_details.ordered_flag _ihm_modeling_protocol_details.software_id _ihm_modeling_protocol_details.script_file_id -1 1 1 42 99 foo s1 'Monte Carlo' 0 500 YES NO NO . . -2 1 2 42 99 foo . 'Replica exchange' 500 2000 YES NO NO . . -3 2 1 42 101 foo . 'Replica exchange' 2000 1000 YES NO NO 80 90 +_ihm_modeling_protocol_details.description +1 1 1 42 99 foo s1 'Monte Carlo' 0 500 YES NO NO . . . +2 1 2 42 99 foo . 'Replica exchange' 500 2000 YES NO NO . . . +3 2 1 42 101 foo . 'Replica exchange' 2000 1000 YES NO NO 80 90 'test step' # """) @@ -1410,7 +1415,8 @@ class MockObject(object): feature='energy/score', num_models_begin=42, num_models_end=42, assembly=asmb1, dataset_group=dg1, - software=software, script_file=script)) + software=software, script_file=script, + details='test step')) p1.analyses.extend((a1, a2)) dumper = ihm.dumper._ProtocolDumper() @@ -1434,10 +1440,11 @@ class MockObject(object): _ihm_modeling_post_process.dataset_group_id _ihm_modeling_post_process.software_id _ihm_modeling_post_process.script_file_id -1 1 1 1 none none . . . . . . -2 1 2 1 filter energy/score 1000 200 . . . . -3 1 2 2 cluster RMSD 200 42 . . . . -4 1 2 3 validation energy/score 42 42 101 301 401 501 +_ihm_modeling_post_process.details +1 1 1 1 none none . . . . . . . +2 1 2 1 filter energy/score 1000 200 . . . . . +3 1 2 2 cluster RMSD 200 42 . . . . . +4 1 2 3 validation energy/score 42 42 101 301 401 501 'test step' # """) @@ -1858,7 +1865,7 @@ def test_model_dumper_atoms(self): het=True), ihm.model.Atom(asym_unit=asym, seq_id=2, atom_id='N', type_symbol='N', x=4.0, y=5.0, z=6.0, - biso=42.0)] + biso=42.0, occupancy=0.2)] dumper = ihm.dumper._ModelDumper() dumper.finalize(system) # assign model/group IDs @@ -1899,14 +1906,15 @@ def test_model_dumper_atoms(self): _atom_site.Cartn_x _atom_site.Cartn_y _atom_site.Cartn_z +_atom_site.occupancy _atom_site.label_entity_id _atom_site.auth_asym_id _atom_site.B_iso_or_equiv _atom_site.pdbx_PDB_model_num _atom_site.ihm_model_id -ATOM 1 C C . ALA 1 X 1.000 2.000 3.000 9 X . 1 1 -HETATM 2 C CA . ALA 1 X 10.000 20.000 30.000 9 X . 1 1 -ATOM 3 N N . CYS 2 X 4.000 5.000 6.000 9 X 42.000 1 1 +ATOM 1 C C . ALA 1 X 1.000 2.000 3.000 . 9 X . 1 1 +HETATM 2 C CA . ALA 1 X 10.000 20.000 30.000 . 9 X . 1 1 +ATOM 3 N N . CYS 2 X 4.000 5.000 6.000 0.200 9 X 42.000 1 1 # # loop_ @@ -1936,7 +1944,7 @@ class MockObject(object): loc = ihm.location.OutputFileLocation(repo='foo', path='bar') loc._id = 3 e2 = ihm.model.Ensemble(model_group=group, num_models=10, - file=loc) + file=loc, details='test details') system.ensembles.extend((e1, e2)) dumper = ihm.dumper._EnsembleDumper() @@ -1955,8 +1963,9 @@ class MockObject(object): _ihm_ensemble_info.num_ensemble_models_deposited _ihm_ensemble_info.ensemble_precision_value _ihm_ensemble_info.ensemble_file_id -1 cluster1 99 42 Hierarchical RMSD 10 2 4.200 . -2 . . 42 . . 10 2 . 3 +_ihm_ensemble_info.details +1 cluster1 99 42 Hierarchical RMSD 10 2 4.200 . . +2 . . 42 . . 10 2 . 3 'test details' # """) @@ -2011,6 +2020,11 @@ def test_entity_poly_segment_dumper(self): a1._id = 'X' system.entities.extend((e1, e2, e3)) system.asym_units.append(a1) + res1 = e2.residue(1) + res2 = e2.residue(2) + system.orphan_features.append(ihm.restraint.ResidueFeature([e2])) + system.orphan_features.append(ihm.restraint.ResidueFeature([res2])) + system.orphan_features.append(ihm.restraint.NonPolyFeature([e3])) system._make_complete_assembly() @@ -2019,13 +2033,17 @@ def test_entity_poly_segment_dumper(self): dumper = ihm.dumper._EntityPolySegmentDumper() dumper.finalize(system) # assign IDs - # e1 isn't directly used in the assembly (a1 is used instead) so - # should have no range ID + # e1 isn't directly used in anything (a1 is used instead, in the + # assembly) so should have no range ID self.assertFalse(hasattr(e1, '_range_id')) self.assertEqual(a1._range_id, 1) + # e2 is use, in a ResidueFeature, so should have a range ID self.assertEqual(e2._range_id, 2) # non-polymers don't have ranges self.assertEqual(e3._range_id, None) + # res2 should have been assigned a range, but not res1 + self.assertFalse(hasattr(res1, '_range_id')) + self.assertEqual(res2._range_id, 3) out = _get_dumper_output(dumper, system) self.assertEqual(out, """# @@ -2038,6 +2056,7 @@ def test_entity_poly_segment_dumper(self): _ihm_entity_poly_segment.comp_id_end 1 1 1 4 ALA ASP 2 2 1 3 ALA GLY +3 2 2 2 CYS CYS # """) @@ -2366,7 +2385,8 @@ class MockObject(object): # duplicate crosslink, should be combined with the original (xxl2) xxl4 = ihm.restraint.ExperimentalCrossLink(e1.residue(2), e2.residue(3)) # should end up in own group, not with xxl4 (since xxl4==xxl2) - xxl5 = ihm.restraint.ExperimentalCrossLink(e1.residue(1), e2.residue(1)) + xxl5 = ihm.restraint.ExperimentalCrossLink(e1.residue(1), e2.residue(1), + details='test xl') r.experimental_cross_links.extend(([xxl1], [xxl2, xxl3], [xxl4, xxl5])) system.restraints.extend((r, MockObject())) @@ -2409,10 +2429,11 @@ class MockObject(object): _ihm_cross_link_list.linker_chem_comp_descriptor_id _ihm_cross_link_list.linker_type _ihm_cross_link_list.dataset_list_id -1 1 foo 1 2 THR foo 1 3 CYS 1 DSS 97 -2 2 foo 1 2 THR bar 2 3 PHE 1 DSS 97 -3 2 foo 1 2 THR bar 2 2 GLU 1 DSS 97 -4 3 foo 1 1 ALA bar 2 1 ASP 1 DSS 97 +_ihm_cross_link_list.details +1 1 foo 1 2 THR foo 1 3 CYS 1 DSS 97 . +2 2 foo 1 2 THR bar 2 3 PHE 1 DSS 97 . +3 2 foo 1 2 THR bar 2 2 GLU 1 DSS 97 . +4 3 foo 1 1 ALA bar 2 1 ASP 1 DSS 97 'test xl' # # loop_ @@ -2460,8 +2481,7 @@ def test_geometric_object_dumper(self): sphere = ihm.geometry.Sphere(center=center, transformation=trans, radius=2.2, name='my sphere', - description='a test sphere', - details='some details') + description='a test sphere') torus = ihm.geometry.Torus(center=center, transformation=trans, major_radius=5.6, minor_radius=1.2) half_torus = ihm.geometry.HalfTorus(center=center, transformation=trans, @@ -2516,12 +2536,11 @@ def test_geometric_object_dumper(self): _ihm_geometric_object_list.object_type _ihm_geometric_object_list.object_name _ihm_geometric_object_list.object_description -_ihm_geometric_object_list.other_details -1 sphere 'my sphere' 'a test sphere' 'some details' -2 torus . . . -3 half-torus . . . -4 axis . . . -5 plane . . . +1 sphere 'my sphere' 'a test sphere' +2 torus . . +3 half-torus . . +4 axis . . +5 plane . . # # loop_ @@ -2575,17 +2594,20 @@ def test_feature_dumper(self): a3 = ihm.AsymUnit(e2, 'heme') system.asym_units.extend((a1, a2, a3)) - f = ihm.restraint.ResidueFeature([a1, a2(2,3)]) + f = ihm.restraint.ResidueFeature([a1, a2(2,3), e1, e1(2,3)], + details='test feature') system.orphan_features.append(f) # Cannot make a ResidueFeature that includes a non-polymer 'residue' self.assertRaises(ValueError, ihm.restraint.ResidueFeature, [a1, a3]) # Polymeric atom feature f = ihm.restraint.AtomFeature([a1.residue(1).atom('CA'), - a2.residue(2).atom('N')]) + a2.residue(2).atom('N'), + e1.residue(1).atom('CB')]) system.orphan_features.append(f) # Nonpolymeric atom feature - f = ihm.restraint.AtomFeature([a3.residue(1).atom('FE')]) + f = ihm.restraint.AtomFeature([a3.residue(1).atom('FE'), + e2.residue(1).atom('FE')]) system.orphan_features.append(f) # Cannot make one feature that selects both polymer and nonpolymer self.assertRaises(ValueError, ihm.restraint.AtomFeature, @@ -2593,7 +2615,7 @@ def test_feature_dumper(self): a2.residue(2).atom('N'), a3.residue(1).atom('FE')]) # Nonpolymeric feature - f = ihm.restraint.NonPolyFeature([a3]) + f = ihm.restraint.NonPolyFeature([a3, e2]) system.orphan_features.append(f) # Cannot make a NonPolyFeature that includes a polymer 'residue' self.assertRaises(ValueError, ihm.restraint.NonPolyFeature, [a1, a3]) @@ -2617,11 +2639,12 @@ def test_feature_dumper(self): _ihm_feature_list.feature_id _ihm_feature_list.feature_type _ihm_feature_list.entity_type -1 'residue range' polymer -2 atom polymer -3 atom non-polymer -4 ligand non-polymer -5 'pseudo site' other +_ihm_feature_list.details +1 'residue range' polymer 'test feature' +2 atom polymer . +3 atom non-polymer . +4 ligand non-polymer . +5 'pseudo site' other . # # loop_ @@ -2635,6 +2658,8 @@ def test_feature_dumper(self): _ihm_poly_residue_feature.comp_id_end 1 1 1 A 1 ALA 4 THR 2 1 1 B 2 CYS 3 GLY +3 1 1 . 1 ALA 4 THR +4 1 1 . 2 CYS 3 GLY # # loop_ @@ -2647,6 +2672,7 @@ def test_feature_dumper(self): _ihm_poly_atom_feature.atom_id 1 2 1 A 1 ALA CA 2 2 1 B 2 CYS N +3 2 1 . 1 ALA CB # # loop_ @@ -2657,7 +2683,9 @@ def test_feature_dumper(self): _ihm_non_poly_feature.comp_id _ihm_non_poly_feature.atom_id 1 3 2 C HEM FE -2 4 2 C HEM . +2 3 2 . HEM FE +3 4 2 C HEM . +4 4 2 . HEM . # # loop_ @@ -2666,8 +2694,7 @@ def test_feature_dumper(self): _ihm_pseudo_site_feature.Cartn_y _ihm_pseudo_site_feature.Cartn_z _ihm_pseudo_site_feature.radius -_ihm_pseudo_site_feature.description -5 10.000 20.000 30.000 . . +5 10.000 20.000 30.000 . # """) @@ -2732,7 +2759,7 @@ class MockObject(object): probability=0.4) r3 = ihm.restraint.DerivedDistanceRestraint(dataset=dataset, feature1=feat1, feature2=feat2, distance=dist, - probability=0.6) + probability=0.6, mic_value=0.4) rg = ihm.restraint.RestraintGroup((r2, r3)) system.restraints.extend((r1, r2)) # r2 is in restraints and groups system.restraint_groups.append(rg) @@ -2751,11 +2778,12 @@ class MockObject(object): _ihm_derived_distance_restraint.distance_lower_limit _ihm_derived_distance_restraint.distance_upper_limit _ihm_derived_distance_restraint.probability +_ihm_derived_distance_restraint.mic_value _ihm_derived_distance_restraint.group_conditionality _ihm_derived_distance_restraint.dataset_list_id -1 . 44 84 'lower bound' 25.000 . 0.800 . 97 -2 1 44 84 'lower bound' 25.000 . 0.400 . 97 -3 1 44 84 'lower bound' 25.000 . 0.600 . 97 +1 . 44 84 'lower bound' 25.000 . 0.800 . . 97 +2 1 44 84 'lower bound' 25.000 . 0.400 . . 97 +3 1 44 84 'lower bound' 25.000 . 0.600 0.400 . 97 # """) @@ -2788,7 +2816,8 @@ class MockObject(object): s = ihm.System() dataset = MockObject() - assembly = MockObject() + dataset.parents = [] + assembly = ihm.Assembly() # Empty restraint groups are OK (even though they don't get IDs) rg = ihm.restraint.RestraintGroup([]) @@ -2913,8 +2942,10 @@ class MockObject(object): cur_entity_assembly.add_entity(entity=cur_entity_2, num_copies=2) cur_instrument = ihm.flr.Instrument(details='My_Instrument') - cur_exp_setting_1 = ihm.flr.ExpSetting(details='My_Exp_setting_1') - cur_exp_setting_2 = ihm.flr.ExpSetting(details='My_Exp_setting_2') + cur_inst_setting_1 = ihm.flr.InstSetting(details='My_Inst_setting_1') + cur_inst_setting_2 = ihm.flr.InstSetting(details='My_Inst_setting_2') + cur_exp_condition_1 = ihm.flr.ExpCondition(details='My_Exp_condition_1') + cur_exp_condition_2 = ihm.flr.ExpCondition(details='My_Exp_condition_2') cur_sample_condition_1 = ihm.flr.SampleCondition( details='My_Sample_condition_1') @@ -2933,13 +2964,27 @@ class MockObject(object): description='Sample_2', details='Details sample 2', solvent_phase='liquid') + ## Reference sample + cur_sample_3 = ihm.flr.Sample(entity_assembly=cur_entity_assembly, + num_of_probes=1, + condition=cur_sample_condition_1, + description='Reference Sample', + details='Details Reference Sample', + solvent_phase='liquid') + cur_experiment = ihm.flr.Experiment() cur_experiment.add_entry(instrument=cur_instrument, - exp_setting=cur_exp_setting_1, + inst_setting=cur_inst_setting_1, + exp_condition=cur_exp_condition_1, sample=cur_sample_1) cur_experiment.add_entry(instrument=cur_instrument, - exp_setting=cur_exp_setting_2, + inst_setting=cur_inst_setting_2, + exp_condition=cur_exp_condition_2, sample=cur_sample_2) + cur_experiment.add_entry(instrument=cur_instrument, + inst_setting=cur_inst_setting_1, + exp_condition=cur_exp_condition_1, + sample=cur_sample_3) ## Probes cur_probe_1 = ihm.flr.Probe() cur_probe_2 = ihm.flr.Probe() @@ -3026,6 +3071,11 @@ class MockObject(object): fluorophore_type='acceptor', poly_probe_position=cur_poly_probe_position_3, description='Acceptor in position2-position3') + cur_sample_probe_details_5 = ihm.flr.SampleProbeDetails(sample=cur_sample_3 , + probe=cur_probe_1, + fluorophore_type='donor', + poly_probe_position=cur_poly_probe_position_1, + description='Donor-only on reference sample') ## Poly_probe_conjugate ## Chem Descriptor ID 5 cur_poly_probe_conjugate_chem_descriptor = ihm.ChemDescriptor(auth_name='Conjugate', @@ -3043,9 +3093,13 @@ class MockObject(object): cur_poly_probe_conjugate_4 = ihm.flr.PolyProbeConjugate(sample_probe=cur_sample_probe_details_4, chem_descriptor=cur_poly_probe_conjugate_chem_descriptor, ambiguous_stoichiometry=False) + cur_poly_probe_conjugate_5 = ihm.flr.PolyProbeConjugate(sample_probe=cur_sample_probe_details_5, + chem_descriptor=cur_poly_probe_conjugate_chem_descriptor, + ambiguous_stoichiometry=False) cur_flr_data.poly_probe_conjugates.extend( (cur_poly_probe_conjugate_1, cur_poly_probe_conjugate_2, - cur_poly_probe_conjugate_3, cur_poly_probe_conjugate_4)) + cur_poly_probe_conjugate_3, cur_poly_probe_conjugate_4, + cur_poly_probe_conjugate_5)) ## Forster_radius cur_forster_radius = ihm.flr.FRETForsterRadius(donor_probe=cur_probe_1, @@ -3058,11 +3112,31 @@ class MockObject(object): phi_acceptor=0.35, alpha=2.4, gg_gr_ratio=0.4, a_b=0.8) cur_fret_calibration_parameters_2 = ihm.flr.FRETCalibrationParameters( phi_acceptor=0.35, alpha=2.4, gg_gr_ratio=0.38, a_b=0.8) - ## Fret_analysis + + ## LifetimeFitModel + cur_lifetime_fit_model = ihm.flr.LifetimeFitModel(name='Lifetime fit model 1', + description='Description of model') + + ## RefMeasurementLifetime + cur_lifetime_1 = ihm.flr.RefMeasurementLifetime(species_fraction=0.6, + lifetime=3.2) + cur_lifetime_2 = ihm.flr.RefMeasurementLifetime(species_fraction=0.4, + lifetime=1.4) + ## RefMeasurement + cur_ref_measurement_1 = ihm.flr.RefMeasurement(ref_sample_probe=cur_sample_probe_details_5, + details='Reference Measurement 1') + cur_ref_measurement_1.add_lifetime(cur_lifetime_1) + cur_ref_measurement_1.add_lifetime(cur_lifetime_2) + ## RefMeasurementGroup + cur_lifetime_ref_measurement_group = ihm.flr.RefMeasurementGroup(details='Reference measurement group 1') + cur_lifetime_ref_measurement_group.add_ref_measurement(cur_ref_measurement_1) + + ## FretAnalysis cur_fret_analysis_1 = ihm.flr.FRETAnalysis(experiment=cur_experiment, sample_probe_1=cur_sample_probe_details_1, sample_probe_2=cur_sample_probe_details_2, forster_radius=cur_forster_radius, + type='intensity-based', calibration_parameters=cur_fret_calibration_parameters_1, method_name='PDA', chi_square_reduced=1.5, @@ -3071,10 +3145,23 @@ class MockObject(object): sample_probe_1=cur_sample_probe_details_3, sample_probe_2=cur_sample_probe_details_4, forster_radius=cur_forster_radius, + type='intensity-based', calibration_parameters=cur_fret_calibration_parameters_2, method_name='PDA', chi_square_reduced=1.8, dataset=dataset_1) + ## lifetime-based FRETAnalysis + cur_fret_analysis_3 = ihm.flr.FRETAnalysis(experiment=cur_experiment, + sample_probe_1=cur_sample_probe_details_1, + sample_probe_2=cur_sample_probe_details_2, + forster_radius=cur_forster_radius, + type='lifetime-based', + lifetime_fit_model=cur_lifetime_fit_model, + ref_measurement_group=cur_lifetime_ref_measurement_group, + method_name='Lifetime fit', + chi_square_reduced=1.6, + dataset=dataset_1) + ## Peak_assignment cur_peak_assignment = ihm.flr.PeakAssignment(method_name='Population', details='Peaks were assigned by population fractions.') @@ -3099,10 +3186,21 @@ class MockObject(object): state=cur_state, population_fraction=0.80, peak_assignment=cur_peak_assignment) + cur_fret_distance_restraint_3 = ihm.flr.FRETDistanceRestraint(sample_probe_1=cur_sample_probe_details_1, + sample_probe_2=cur_sample_probe_details_2, + analysis=cur_fret_analysis_3, + distance=53.5, + distance_error_plus=2.5, + distance_error_minus=2.3, + distance_type='_E', + state=cur_state, + population_fraction=0.80, + peak_assignment=cur_peak_assignment) cur_fret_distance_restraint_group = ihm.flr.FRETDistanceRestraintGroup() cur_fret_distance_restraint_group.add_distance_restraint(cur_fret_distance_restraint_1) cur_fret_distance_restraint_group.add_distance_restraint(cur_fret_distance_restraint_2) + cur_fret_distance_restraint_group.add_distance_restraint(cur_fret_distance_restraint_3) cur_flr_data.distance_restraint_groups.append( cur_fret_distance_restraint_group) @@ -3203,8 +3301,11 @@ class MockObject(object): experiment_dumper = ihm.dumper._FLRExperimentDumper() experiment_dumper.finalize(system) - exp_setting_dumper = ihm.dumper._FLRExpSettingDumper() - exp_setting_dumper.finalize(system) + inst_setting_dumper = ihm.dumper._FLRInstSettingDumper() + inst_setting_dumper.finalize(system) + + exp_condition_dumper = ihm.dumper._FLR_ExpConditionDumper() + exp_condition_dumper.finalize(system) instrument_dumper = ihm.dumper._FLRInstrumentDumper() instrument_dumper.finalize(system) @@ -3236,6 +3337,12 @@ class MockObject(object): parameters_dumper = ihm.dumper._FLRCalibrationParametersDumper() parameters_dumper.finalize(system) + lifetime_fit_model_dumper = ihm.dumper._FLRLifetimeFitModelDumper() + lifetime_fit_model_dumper.finalize(system) + + ref_measurement_dumper = ihm.dumper._FLRRefMeasurementDumper() + ref_measurement_dumper.finalize(system) + analysis_dumper = ihm.dumper._FLRAnalysisDumper() analysis_dumper.finalize(system) @@ -3266,21 +3373,33 @@ class MockObject(object): _flr_experiment.ordinal_id _flr_experiment.id _flr_experiment.instrument_id -_flr_experiment.exp_setting_id +_flr_experiment.inst_setting_id +_flr_experiment.exp_condition_id _flr_experiment.sample_id _flr_experiment.details -1 1 1 1 1 . -2 1 1 2 2 . +1 1 1 1 1 1 . +2 1 1 2 2 2 . +3 1 1 1 1 3 . # """) - out = _get_dumper_output(exp_setting_dumper, system) + out = _get_dumper_output(inst_setting_dumper, system) self.assertEqual(out, """# loop_ -_flr_exp_setting.id -_flr_exp_setting.details -1 My_Exp_setting_1 -2 My_Exp_setting_2 +_flr_inst_setting.id +_flr_inst_setting.details +1 My_Inst_setting_1 +2 My_Inst_setting_2 +# +""") + + out = _get_dumper_output(exp_condition_dumper, system) + self.assertEqual(out, """# +loop_ +_flr_exp_condition.id +_flr_exp_condition.details +1 My_Exp_condition_1 +2 My_Exp_condition_2 # """) @@ -3328,6 +3447,7 @@ class MockObject(object): _flr_sample.solvent_phase 1 1 2 1 Sample_1 'Details sample 1' liquid 2 1 2 2 Sample_2 'Details sample 2' liquid +3 1 1 1 'Reference Sample' 'Details Reference Sample' liquid # """) @@ -3367,6 +3487,7 @@ class MockObject(object): 2 1 2 acceptor 'Acceptor in position1-position3' 2 3 2 1 donor 'Donor in position2-position3' 3 4 2 2 acceptor 'Acceptor in position2-position3' 2 +5 3 1 donor 'Donor-only on reference sample' 1 # """) @@ -3416,6 +3537,7 @@ class MockObject(object): 2 2 5 NO . 3 3 5 NO . 4 4 5 NO . +5 5 5 NO . # """) @@ -3446,6 +3568,52 @@ class MockObject(object): 1 0.350 2.400 . 0.400 . . . 0.800 2 0.350 2.400 . 0.380 . . . 0.800 # +""") + + out = _get_dumper_output(ref_measurement_dumper, system) + self.assertEqual(out, """# +loop_ +_flr_reference_measurement_group.id +_flr_reference_measurement_group.num_measurements +_flr_reference_measurement_group.details +1 1 'Reference measurement group 1' +# +# +loop_ +_flr_reference_measurement_group_link.group_id +_flr_reference_measurement_group_link.reference_measurement_id +1 1 +# +# +loop_ +_flr_reference_measurement.id +_flr_reference_measurement.reference_sample_probe_id +_flr_reference_measurement.num_species +_flr_reference_measurement.details +1 5 2 'Reference Measurement 1' +# +# +loop_ +_flr_reference_measurement_lifetime.ordinal_id +_flr_reference_measurement_lifetime.reference_measurement_id +_flr_reference_measurement_lifetime.species_name +_flr_reference_measurement_lifetime.species_fraction +_flr_reference_measurement_lifetime.lifetime +1 1 . 0.600 3.200 +2 1 . 0.400 1.400 +# +""") + + out = _get_dumper_output(lifetime_fit_model_dumper, system) + self.assertEqual(out, """# +loop_ +_flr_lifetime_fit_model.id +_flr_lifetime_fit_model.name +_flr_lifetime_fit_model.description +_flr_lifetime_fit_model.external_file_id +_flr_lifetime_fit_model.citation_id +1 'Lifetime fit model 1' 'Description of model' . . +# """) out = _get_dumper_output(analysis_dumper, system) @@ -3453,17 +3621,40 @@ class MockObject(object): loop_ _flr_fret_analysis.id _flr_fret_analysis.experiment_id +_flr_fret_analysis.type _flr_fret_analysis.sample_probe_id_1 _flr_fret_analysis.sample_probe_id_2 _flr_fret_analysis.forster_radius_id -_flr_fret_analysis.calibration_parameters_id -_flr_fret_analysis.method_name -_flr_fret_analysis.chi_square_reduced _flr_fret_analysis.dataset_list_id _flr_fret_analysis.external_file_id _flr_fret_analysis.software_id -1 1 1 2 1 1 PDA 1.500 1 . . -2 1 3 4 1 2 PDA 1.800 1 . . +1 1 intensity-based 1 2 1 1 . . +2 1 intensity-based 3 4 1 1 . . +3 1 lifetime-based 1 2 1 1 . . +# +# +loop_ +_flr_fret_analysis_intensity.ordinal_id +_flr_fret_analysis_intensity.analysis_id +_flr_fret_analysis_intensity.calibration_parameters_id +_flr_fret_analysis_intensity.donor_only_fraction +_flr_fret_analysis_intensity.chi_square_reduced +_flr_fret_analysis_intensity.method_name +_flr_fret_analysis_intensity.details +1 1 1 . 1.500 PDA . +2 2 2 . 1.800 PDA . +# +# +loop_ +_flr_fret_analysis_lifetime.ordinal_id +_flr_fret_analysis_lifetime.analysis_id +_flr_fret_analysis_lifetime.reference_measurement_group_id +_flr_fret_analysis_lifetime.lifetime_fit_model_id +_flr_fret_analysis_lifetime.donor_only_fraction +_flr_fret_analysis_lifetime.chi_square_reduced +_flr_fret_analysis_lifetime.method_name +_flr_fret_analysis_lifetime.details +1 3 1 1 . 1.600 'Lifetime fit' . # """) @@ -3495,6 +3686,7 @@ class MockObject(object): _flr_fret_distance_restraint.peak_assignment_id 1 1 1 1 2 1 1 53.500 2.500 2.300 _E 0.800 1 2 2 1 3 4 1 2 49.000 2.000 2.100 _E 0.800 1 +3 3 1 1 2 1 3 53.500 2.500 2.300 _E 0.800 1 # """) diff --git a/modules/core/dependency/python-ihm/test/test_examples.py b/modules/core/dependency/python-ihm/test/test_examples.py index c4f79a1b69..8055de0fe9 100644 --- a/modules/core/dependency/python-ihm/test/test_examples.py +++ b/modules/core/dependency/python-ihm/test/test_examples.py @@ -29,7 +29,7 @@ def test_simple_docking_example(self): # can read it with open(os.path.join(tmpdir, 'output.cif')) as fh: contents = fh.readlines() - self.assertEqual(len(contents), 309) + self.assertEqual(len(contents), 312) with open(os.path.join(tmpdir, 'output.cif')) as fh: s, = ihm.reader.read(fh) @@ -43,7 +43,7 @@ def test_locations_example(self): # can read it with open(out) as fh: contents = fh.readlines() - self.assertEqual(len(contents), 67) + self.assertEqual(len(contents), 68) with open(out) as fh: s, = ihm.reader.read(fh) os.unlink(out) @@ -80,7 +80,7 @@ def test_non_standard_residues_example(self): # can read it with open(out) as fh: contents = fh.readlines() - self.assertEqual(len(contents), 87) + self.assertEqual(len(contents), 64) with open(out) as fh: s, = ihm.reader.read(fh) os.unlink(out) diff --git a/modules/core/dependency/python-ihm/test/test_flr.py b/modules/core/dependency/python-ihm/test/test_flr.py index 4dc780d655..e48127ec0f 100644 --- a/modules/core/dependency/python-ihm/test/test_flr.py +++ b/modules/core/dependency/python-ihm/test/test_flr.py @@ -249,94 +249,101 @@ def test_Experiment_Init(self): ## Initialization with only one parameter given should not add an entry e1 = ihm.flr.Experiment(instrument = 'foo') self.assertEqual(len(e1.instrument_list), 0) - self.assertEqual(len(e1.exp_setting_list),0) + self.assertEqual(len(e1.inst_setting_list),0) + self.assertEqual(len(e1.exp_condition_list),0) self.assertEqual(len(e1.sample_list),0) ## Correct initialization should fill the lists e2 = ihm.flr.Experiment(instrument = 'foo', - exp_setting = 'bar', - sample = 'foo2', + inst_setting = 'bar', + exp_condition='foo2', + sample = 'foo3', details = 'bar2') self.assertEqual(len(e2.instrument_list), 1) self.assertEqual(e2.instrument_list[0], 'foo') - self.assertEqual(e2.exp_setting_list[0], 'bar') - self.assertEqual(e2.sample_list[0], 'foo2') + self.assertEqual(e2.inst_setting_list[0], 'bar') + self.assertEqual(e2.exp_condition_list[0],'foo2') + self.assertEqual(e2.sample_list[0], 'foo3') self.assertEqual(e2.details_list[0], 'bar2') ## Initialization without details given should still have an entry in the list e3 = ihm.flr.Experiment(instrument = 'foo', - exp_setting = 'bar', + inst_setting = 'bar', + exp_condition = 'bar2', sample = 'foo2') self.assertEqual(len(e3.details_list),1) self.assertIsNone(e3.details_list[0]) - def test_Experiment_Add_entry(self): """ Test addition of an entry to the experiment. """ ## Adding to an empty Experiment e1 = ihm.flr.Experiment() e1.add_entry(instrument = 'foo', - exp_setting = 'bar', - sample = 'foo2', + inst_setting = 'bar', + exp_condition = 'foo2', + sample = 'foo3', details = 'bar2') self.assertEqual(e1.instrument_list[0],'foo') - self.assertEqual(e1.exp_setting_list[0], 'bar') - self.assertEqual(e1.sample_list[0],'foo2') + self.assertEqual(e1.inst_setting_list[0], 'bar') + self.assertEqual(e1.exp_condition_list[0],'foo2') + self.assertEqual(e1.sample_list[0],'foo3') self.assertEqual(e1.details_list[0], 'bar2') ## adding to an existing Experiment e2 = ihm.flr.Experiment(instrument = 'foo', - exp_setting = 'foo2', - sample = 'foo3', - details = 'foo4') + inst_setting = 'foo2', + exp_condition='foo3', + sample = 'foo4', + details = 'foo5') e2.add_entry(instrument = 'bar', - exp_setting = 'bar2', - sample = 'bar3', - details = 'bar4') + inst_setting = 'bar2', + exp_condition= 'bar3', + sample = 'bar4', + details = 'bar5') self.assertEqual(e2.instrument_list, ['foo','bar']) - self.assertEqual(e2.exp_setting_list, ['foo2','bar2']) - self.assertEqual(e2.sample_list, ['foo3','bar3']) - self.assertEqual(e2.details_list, ['foo4','bar4']) + self.assertEqual(e2.inst_setting_list, ['foo2','bar2']) + self.assertEqual(e2.exp_condition_list, ['foo3', 'bar3']) + self.assertEqual(e2.sample_list, ['foo4','bar4']) + self.assertEqual(e2.details_list, ['foo5','bar5']) def test_Experiment_Get_entry_by_index(self): """ Test access to entries by index. """ e = ihm.flr.Experiment() - e.add_entry(instrument = 'foo', exp_setting = 'foo2', sample = 'foo3', details = 'foo4') - e.add_entry(instrument = 'bar', exp_setting = 'bar2', sample = 'bar3', details = 'bar4') - e.add_entry(instrument = 'foobar', exp_setting = 'foobar2', sample = 'foobar3', details = 'foobar4') + e.add_entry(instrument = 'foo', inst_setting = 'foo2', exp_condition='foo3', sample = 'foo4', details = 'foo5') + e.add_entry(instrument = 'bar', inst_setting = 'bar2', exp_condition='bar3', sample = 'bar4', details = 'bar5') + e.add_entry(instrument = 'foobar', inst_setting = 'foobar2', exp_condition='foobar3', sample = 'foobar4', details = 'foobar5') return_value_index0 = e.get_entry_by_index(0) return_value_index1 = e.get_entry_by_index(1) return_value_index2 = e.get_entry_by_index(2) - self.assertEqual(return_value_index0, ('foo','foo2','foo3','foo4')) - self.assertEqual(return_value_index1, ('bar','bar2','bar3','bar4')) - self.assertEqual(return_value_index2, ('foobar','foobar2','foobar3','foobar4')) + self.assertEqual(return_value_index0, ('foo','foo2','foo3','foo4','foo5')) + self.assertEqual(return_value_index1, ('bar','bar2','bar3','bar4','bar5')) + self.assertEqual(return_value_index2, ('foobar','foobar2','foobar3','foobar4','foobar5')) def test_Experiment_Contains(self): """ Test whether experiment contains a combination of instrument, exp_setting, and sample. """ ## An empty experiment should not contain anything e1 = ihm.flr.Experiment() - self.assertFalse(e1.contains('foo','foo2','foo3')) + self.assertFalse(e1.contains('foo','foo2','foo3','foo4')) ## After addition, the entry should be contained - e1.add_entry(instrument = 'foo', exp_setting='foo2',sample = 'foo3') - e1.add_entry(instrument = 'bar', exp_setting='bar2',sample = 'bar3') - self.assertTrue(e1.contains('foo','foo2','foo3')) + e1.add_entry(instrument = 'foo', inst_setting='foo2', exp_condition='foo3', sample = 'foo4') + e1.add_entry(instrument = 'bar', inst_setting='bar2', exp_condition='bar3', sample = 'bar4') + self.assertTrue(e1.contains('foo','foo2','foo3','foo4')) ## If one of the entries is not contained, then False - self.assertFalse(e1.contains('foo2','foo2','foo4')) - self.assertFalse(e1.contains('foobar','foobar2','foobar3')) + self.assertFalse(e1.contains('foo2','foo2','foo4','foo5')) + self.assertFalse(e1.contains('foobar','foobar2','foobar3','foobar4')) def test_Experiment_Eq(self): """ Test equality and inequality of Experiment objects. """ e_ref = ihm.flr.Experiment() - e_ref.add_entry(instrument = 'foo', exp_setting='foo2',sample = 'foo3') + e_ref.add_entry(instrument = 'foo', inst_setting='foo2', exp_condition = 'foo3', sample = 'foo4') e_equal = ihm.flr.Experiment() - e_equal.add_entry(instrument = 'foo', exp_setting='foo2',sample = 'foo3') + e_equal.add_entry(instrument = 'foo', inst_setting='foo2', exp_condition = 'foo3', sample = 'foo4') e_unequal = ihm.flr.Experiment() - e_unequal.add_entry(instrument = 'bar', exp_setting='bar2',sample = 'bar3') + e_unequal.add_entry(instrument = 'bar', inst_setting='bar2', exp_condition = 'bar3', sample = 'bar4') self.assertTrue(e_ref == e_equal) self.assertFalse(e_ref == e_unequal) self.assertTrue(e_ref != e_unequal) - def test_Instrument_Init(self): """ Test initialization of Instrument. """ i = ihm.flr.Instrument(details = 'foo') @@ -351,16 +358,30 @@ def test_Instrument_Eq(self): self.assertFalse(i_ref == i_unequal) self.assertTrue(i_ref != i_unequal) - def test_exp_setting_init(self): - """ Test initialization of ExpSetting.""" - e = ihm.flr.ExpSetting(details = 'foo') + def test_inst_setting_init(self): + """ Test initialization of InstSetting.""" + e = ihm.flr.InstSetting(details = 'foo') + self.assertEqual(e.details, 'foo') + + def test_inst_setting_eq(self): + """ Test equality and inequality of InstSetting objects.""" + e_ref = ihm.flr.InstSetting(details = 'foo') + e_equal = ihm.flr.InstSetting(details = 'foo') + e_unequal = ihm.flr.InstSetting(details = 'bar') + self.assertTrue(e_ref == e_equal) + self.assertFalse(e_ref == e_unequal) + self.assertTrue(e_ref != e_unequal) + + def test_exp_condition_init(self): + """ Test initialization of ExpCondition.""" + e = ihm.flr.ExpCondition(details = 'foo') self.assertEqual(e.details, 'foo') - def test_exp_setting_eq(self): - """ Test equality and inequality of ExpSetting objects.""" - e_ref = ihm.flr.ExpSetting(details = 'foo') - e_equal = ihm.flr.ExpSetting(details = 'foo') - e_unequal = ihm.flr.ExpSetting(details = 'bar') + def test_exp_condition_eq(self): + """ Test equality and inequality of ExpCondition objects.""" + e_ref = ihm.flr.ExpCondition(details = 'foo') + e_equal = ihm.flr.ExpCondition(details = 'foo') + e_unequal = ihm.flr.ExpCondition(details = 'bar') self.assertTrue(e_ref == e_equal) self.assertFalse(e_ref == e_unequal) self.assertTrue(e_ref != e_unequal) @@ -371,9 +392,13 @@ def test_fret_analysis_init(self): sample_probe_1 = 'this_sample_probe_1', sample_probe_2 = 'this_sample_probe_2', forster_radius = 'this_forster_radius', + type = 'intensity-based', calibration_parameters = 'this_calibration_parameters', + lifetime_fit_model = 'this_lifetime_fit_model', + ref_measurement_group = 'this_ref_measurement_group', method_name = 'this_method_name', chi_square_reduced = 'this_chi_square_reduced', + donor_only_fraction='this_donly_fraction', dataset='this_dataset_list_id', external_file = 'this_external_file', software = 'this_software') @@ -381,22 +406,34 @@ def test_fret_analysis_init(self): self.assertEqual(f.sample_probe_1, 'this_sample_probe_1') self.assertEqual(f.sample_probe_2, 'this_sample_probe_2') self.assertEqual(f.forster_radius, 'this_forster_radius') + self.assertEqual(f.type, 'intensity-based') self.assertEqual(f.calibration_parameters, 'this_calibration_parameters') + self.assertEqual(f.lifetime_fit_model, 'this_lifetime_fit_model') + self.assertEqual(f.ref_measurement_group,'this_ref_measurement_group') self.assertEqual(f.method_name, 'this_method_name') self.assertEqual(f.chi_square_reduced, 'this_chi_square_reduced') + self.assertEqual(f.donor_only_fraction, 'this_donly_fraction') self.assertEqual(f.dataset, 'this_dataset_list_id') self.assertEqual(f.external_file, 'this_external_file') self.assertEqual(f.software, 'this_software') + self.assertRaises(ValueError, + ihm.flr.FRETAnalysis, experiment='this_experiment', + sample_probe_1='this_sample_probe_1', + sample_probe_2='this_sample_probe_2', + forster_radius='this_forster_radius', type='garbage') + def test_fret_analysis_eq(self): """ Test equality and inequality of FRETAnalysis objects. """ f_ref = ihm.flr.FRETAnalysis(experiment = 'this_experiment', sample_probe_1 = 'this_sample_probe_1', sample_probe_2 = 'this_sample_probe_2', forster_radius = 'this_forster_radius', + type = 'intensity-based', calibration_parameters = 'this_calibration_parameters', method_name = 'this_method_name', chi_square_reduced = 'this_chi_square_reduced', + donor_only_fraction='this_donly_fraction', dataset='this_dataset_list_id', external_file = 'this_external_file', software = 'this_software') @@ -404,9 +441,11 @@ def test_fret_analysis_eq(self): sample_probe_1 = 'this_sample_probe_1', sample_probe_2 = 'this_sample_probe_2', forster_radius = 'this_forster_radius', + type='intensity-based', calibration_parameters = 'this_calibration_parameters', method_name = 'this_method_name', chi_square_reduced = 'this_chi_square_reduced', + donor_only_fraction='this_donly_fraction', dataset='this_dataset_list_id', external_file = 'this_external_file', software = 'this_software') @@ -414,12 +453,141 @@ def test_fret_analysis_eq(self): sample_probe_1='foo', sample_probe_2='this_sample_probe_2', forster_radius='this_forster_radius', + type='intensity-based', calibration_parameters='this_calibration_parameters', method_name='this_method_name', chi_square_reduced='this_chi_square_reduced', + donor_only_fraction='this_donly_fraction', dataset='this_dataset_list_id', external_file='this_external_file', software='this_software') + f_unequal_type = ihm.flr.FRETAnalysis(experiment = 'this_experiment', + sample_probe_1 = 'this_sample_probe_1', + sample_probe_2 = 'this_sample_probe_2', + forster_radius = 'this_forster_radius', + type = 'lifetime-based', + calibration_parameters = 'this_calibration_parameters', + method_name = 'this_method_name', + chi_square_reduced = 'this_chi_square_reduced', + donor_only_fraction='this_donly', + dataset='this_dataset_list_id', + external_file = 'this_external_file', + software = 'this_software') + self.assertTrue(f_ref == f_equal) + self.assertFalse(f_ref == f_unequal) + self.assertTrue(f_ref != f_unequal) + self.assertFalse(f_ref == f_unequal_type) + self.assertTrue(f_ref != f_unequal_type) + + def test_lifetime_fit_model_init(self): + """ Test initialization of LifetimeFitModel.""" + f = ihm.flr.LifetimeFitModel(name = 'this_name', + description = 'this_description', + external_file = 'this_ext_file', + citation = 'this_citation') + self.assertEqual(f.name, 'this_name') + self.assertEqual(f.description, 'this_description') + self.assertEqual(f.external_file, 'this_ext_file') + self.assertEqual(f.citation, 'this_citation') + + def test_lifetime_fit_model_eq(self): + """ Test equality and inequality of LifeTimeFitModel objects.""" + f_ref = ihm.flr.LifetimeFitModel(name = 'this_name', description = 'this_desc') + f_equal = ihm.flr.LifetimeFitModel(name = 'this_name', description = 'this_desc') + f_unequal = ihm.flr.LifetimeFitModel(name = 'other_name', description = 'this_desc') + self.assertTrue(f_ref == f_equal) + self.assertFalse(f_ref == f_unequal) + self.assertTrue(f_ref != f_unequal) + + def test_ref_measurement_group_init(self): + """ Test initialization of RefMeasurementGroup.""" + r = ihm.flr.RefMeasurementGroup() + self.assertEqual(r.ref_measurement_list, []) + + def test_ref_measurement_group_add_ref_measurement(self): + """ Test the addition of a RefMeasurement to the group.""" + r = ihm.flr.RefMeasurementGroup() + r.add_ref_measurement('foo') + r.add_ref_measurement('bar') + self.assertEqual(r.ref_measurement_list, ['foo','bar']) + + def test_ref_measurement_group_get_info(self): + """ Test the retrieval of the ref_measurement_list.""" + r = ihm.flr.RefMeasurementGroup() + r.add_ref_measurement('foo') + r.add_ref_measurement('bar') + return_value = r.get_info() + self.assertEqual(return_value, ['foo', 'bar']) + + def test_ref_measurement_group_eq(self): + """ Test equality and inequality of RefMeasurementGroup objects.""" + r_ref = ihm.flr.RefMeasurementGroup() + r_ref.add_ref_measurement('foo') + r_equal = ihm.flr.RefMeasurementGroup() + r_equal.add_ref_measurement('foo') + r_unequal = ihm.flr.RefMeasurementGroup() + r_unequal.add_ref_measurement('foo2') + self.assertTrue(r_ref == r_equal) + self.assertFalse(r_ref == r_unequal) + self.assertTrue(r_ref != r_unequal) + + def test_ref_measurement_init(self): + """ Test initialization of RefMeasurement.""" + r1 = ihm.flr.RefMeasurement(ref_sample_probe = 'this_ref_sample_probe', + details = 'this_details') + self.assertEqual(r1.ref_sample_probe, 'this_ref_sample_probe') + self.assertEqual(r1.details, 'this_details') + self.assertEqual(r1.list_of_lifetimes, []) + + r2 = ihm.flr.RefMeasurement(ref_sample_probe = 'this_ref_sample_probe_2', + details = 'this_details_2', + list_of_lifetimes = ['foo','bar']) + self.assertEqual(r2.ref_sample_probe, 'this_ref_sample_probe_2') + self.assertEqual(r2.details, 'this_details_2') + self.assertEqual(r2.list_of_lifetimes, ['foo','bar']) + + def test_ref_measurement_add_lifetime(self): + """ Test addition of to the list_of_lifetimes.""" + r = ihm.flr.RefMeasurement(ref_sample_probe = 'this_ref_sample_probe', + details = 'this_details') + r.add_lifetime('foo') + r.add_lifetime('bar') + self.assertEqual(r.list_of_lifetimes, ['foo','bar']) + + def test_ref_measurement_eq(self): + """ Test equality and inequality of RefMeasurement objects.""" + r_ref = ihm.flr.RefMeasurement(ref_sample_probe = 'this_ref_sample_probe_1', + details = 'this_details_1') + r_equal = ihm.flr.RefMeasurement(ref_sample_probe='this_ref_sample_probe_1', + details='this_details_1') + r_unequal = ihm.flr.RefMeasurement(ref_sample_probe='this_ref_sample_probe_2', + details='this_details_2') + r_unequal_list = ihm.flr.RefMeasurement(ref_sample_probe = 'this_ref_sample_probe_1', + details = 'this_details_1', + list_of_lifetimes = ['foo']) + self.assertTrue(r_ref == r_equal) + self.assertTrue(r_ref != r_unequal) + self.assertTrue(r_ref != r_unequal_list) + self.assertFalse(r_ref == r_unequal) + self.assertFalse(r_ref == r_unequal_list) + + def test_ref_measurement_lifetime_init(self): + """ Test initialization of RefMeasuremenLifetime objects.""" + f = ihm.flr.RefMeasurementLifetime(species_fraction='this_frac', + lifetime='this_lifetime', + species_name='foo') + self.assertEqual(f.species_fraction, 'this_frac') + self.assertEqual(f.lifetime, 'this_lifetime') + self.assertEqual(f.species_name, 'foo') + + def test_ref_measurement_lifetime_eq(self): + """ Test equality and inequality of RefMeasurementLifetime objects.""" + f_ref = ihm.flr.RefMeasurementLifetime(species_fraction = 'this_frac_1', + lifetime = 'this_lifetime_1') + f_equal = ihm.flr.RefMeasurementLifetime(species_fraction = 'this_frac_1', + lifetime = 'this_lifetime_1') + f_unequal = ihm.flr.RefMeasurementLifetime(species_fraction = 'this_frac_2', + lifetime = 'this_lifetime_1') self.assertTrue(f_ref == f_equal) self.assertFalse(f_ref == f_unequal) self.assertTrue(f_ref != f_unequal) diff --git a/modules/core/dependency/python-ihm/test/test_geometry.py b/modules/core/dependency/python-ihm/test/test_geometry.py index 9f549a3d44..08fd036059 100644 --- a/modules/core/dependency/python-ihm/test/test_geometry.py +++ b/modules/core/dependency/python-ihm/test/test_geometry.py @@ -34,7 +34,6 @@ def test_geometric_object(self): g = ihm.geometry.GeometricObject(name='foo', description='bar') self.assertEqual(g.name, 'foo') self.assertEqual(g.description, 'bar') - self.assertIsNone(g.details) self.assertEqual(g.type, 'other') def test_sphere(self): diff --git a/modules/core/dependency/python-ihm/test/test_main.py b/modules/core/dependency/python-ihm/test/test_main.py index 37a74b8b76..f6f60c8929 100644 --- a/modules/core/dependency/python-ihm/test/test_main.py +++ b/modules/core/dependency/python-ihm/test/test_main.py @@ -780,9 +780,9 @@ class MockObject(object): s = ihm.System() r1 = MockObject() r2 = MockObject() - r2.feature = None + r2._all_features = (None,) r3 = MockObject() - r3.feature = f1 + r3._all_features = (f1,) s.orphan_features.extend((f1, f2)) s.restraints.extend((r1, r2, r3)) diff --git a/modules/core/dependency/python-ihm/test/test_reader.py b/modules/core/dependency/python-ihm/test/test_reader.py index 1e28f44c2b..8be8fd6734 100644 --- a/modules/core/dependency/python-ihm/test/test_reader.py +++ b/modules/core/dependency/python-ihm/test/test_reader.py @@ -679,9 +679,11 @@ def test_external_file_handler(self): _ihm_external_reference_info.reference _ihm_external_reference_info.refers_to _ihm_external_reference_info.associated_url +_ihm_external_reference_info.details 1 Zenodo DOI 10.5281/zenodo.1218053 Archive https://example.com/foo.zip -2 . 'Supplementary Files' . Other . -3 Zenodo DOI 10.5281/zenodo.1218058 File https://example.com/foo.dcd +'test repo' +2 . 'Supplementary Files' . Other . . +3 Zenodo DOI 10.5281/zenodo.1218058 File https://example.com/foo.dcd . """ ext_file_cat = """ loop_ @@ -705,6 +707,7 @@ def test_external_file_handler(self): self.assertEqual(l1.path, 'scripts/test.py') self.assertEqual(l1.details, 'Test script') self.assertEqual(l1.repo.doi, '10.5281/zenodo.1218053') + self.assertEqual(l1.repo.details, 'test repo') self.assertEqual(l1.__class__, ihm.location.WorkflowFileLocation) @@ -895,10 +898,11 @@ def test_model_representation_details_handler(self): _ihm_model_representation_details.model_mode _ihm_model_representation_details.model_granularity _ihm_model_representation_details.model_object_count -1 1 1 Nup84 A 1 sphere . flexible by-feature 1 -2 1 1 Nup84 A 2 sphere 1 rigid by-residue . -3 2 1 Nup84 A . atomistic . flexible by-atom . -4 3 2 Nup85 B . sphere . . multi-residue . +_ihm_model_representation_details.description +1 1 1 Nup84 A 1 sphere . flexible by-feature 1 'test segment' +2 1 1 Nup84 A 2 sphere 1 rigid by-residue . . +3 2 1 Nup84 A . atomistic . flexible by-atom . . +4 3 2 Nup85 B . sphere . . multi-residue . . """ for fh in cif_file_handles(cif): s, = ihm.reader.read(fh) @@ -911,6 +915,7 @@ def test_model_representation_details_handler(self): self.assertEqual(s1.rigid, False) self.assertIsNone(s1.starting_model) self.assertEqual(s1.asym_unit.seq_id_range, (1,6)) + self.assertEqual(s1.description, 'test segment') self.assertEqual(s2.__class__, ihm.representation.ResidueSegment) self.assertEqual(s2.primitive, 'sphere') @@ -918,6 +923,7 @@ def test_model_representation_details_handler(self): self.assertEqual(s2.rigid, True) self.assertEqual(s2.starting_model._id, '1') self.assertEqual(s2.asym_unit.seq_id_range, (7,20)) + self.assertIsNone(s2.description) self.assertEqual(len(r2), 1) s1, = r2 @@ -948,8 +954,9 @@ def test_starting_model_details_handler(self): _ihm_starting_model_details.starting_model_auth_asym_id _ihm_starting_model_details.starting_model_sequence_offset _ihm_starting_model_details.dataset_list_id -1 1 Nup84 A 1 'comparative model' Q 8 4 -2 1 Nup84 A . 'comparative model' X . 6 +_ihm_starting_model_details.description +1 1 Nup84 A 1 'comparative model' Q 8 4 . +2 1 Nup84 A . 'comparative model' X . 6 'test desc' """ for fh in cif_file_handles(cif): s, = ihm.reader.read(fh) @@ -959,11 +966,13 @@ def test_starting_model_details_handler(self): self.assertEqual(m1.asym_id, 'Q') self.assertEqual(m1.offset, 8) self.assertEqual(m1.dataset._id, '4') + self.assertIsNone(m1.description) self.assertEqual(m2.asym_unit._id, 'A') self.assertEqual(m2.asym_id, 'X') self.assertEqual(m2.offset, 0) self.assertEqual(m2.dataset._id, '6') + self.assertEqual(m2.description, 'test desc') def test_starting_computational_models_handler(self): """Test StartingComputationModelsHandler""" @@ -1050,8 +1059,9 @@ def test_protocol_details_handler(self): _ihm_modeling_protocol_details.ordered_flag _ihm_modeling_protocol_details.software_id _ihm_modeling_protocol_details.script_file_id -1 1 1 1 1 . Sampling 'Monte Carlo' 0 500 YES NO NO . . -2 1 2 1 2 . Sampling 'Monte Carlo' 500 5000 YES . NO 401 501 +_ihm_modeling_protocol_details.description +1 1 1 1 1 . Sampling 'Monte Carlo' 0 500 YES NO NO . . . +2 1 2 1 2 . Sampling 'Monte Carlo' 500 5000 YES . NO 401 501 'test step' """ for fh in cif_file_handles(cif): s, = ihm.reader.read(fh) @@ -1069,12 +1079,14 @@ def test_protocol_details_handler(self): self.assertEqual(p1.steps[0].ordered, False) self.assertIsNone(p1.steps[0].software) self.assertIsNone(p1.steps[0].script_file) + self.assertIsNone(p1.steps[0].description) self.assertEqual(p1.steps[1]._id, '2') self.assertEqual(p1.steps[1].multi_scale, True) self.assertIsNone(p1.steps[1].multi_state) self.assertEqual(p1.steps[1].ordered, False) self.assertEqual(p1.steps[1].software._id, '401') self.assertEqual(p1.steps[1].script_file._id, '501') + self.assertEqual(p1.steps[1].description, 'test step') def test_post_process_handler(self): """Test PostProcessHandler""" @@ -1092,12 +1104,13 @@ def test_post_process_handler(self): _ihm_modeling_post_process.dataset_group_id _ihm_modeling_post_process.software_id _ihm_modeling_post_process.script_file_id -1 1 1 1 'filter' 'energy/score' 15000 6520 . . 401 501 -2 1 1 2 'cluster' 'dRMSD' 6520 6520 . . . . -3 1 2 1 'filter' 'energy/score' 15000 6520 . . . . -4 1 2 2 'filter' 'composition' 6520 6520 . . . . -5 1 2 3 'cluster' 'dRMSD' 6520 6520 . . . . -6 2 3 1 'none' . . . . . . . +_ihm_modeling_post_process.details +1 1 1 1 'filter' 'energy/score' 15000 6520 . . 401 501 . +2 1 1 2 'cluster' 'dRMSD' 6520 6520 . . . . . +3 1 2 1 'filter' 'energy/score' 15000 6520 . . . . . +4 1 2 2 'filter' 'composition' 6520 6520 . . . . . +5 1 2 3 'cluster' 'dRMSD' 6520 6520 . . . . . +6 2 3 1 'none' . . . . . . . 'empty step' """ for fh in cif_file_handles(cif): s, = ihm.reader.read(fh) @@ -1115,6 +1128,7 @@ def test_post_process_handler(self): self.assertEqual(a1.steps[1].__class__, ihm.analysis.ClusterStep) self.assertIsNone(a1.steps[1].software) self.assertIsNone(a1.steps[1].script_file) + self.assertIsNone(a1.steps[1].details) self.assertEqual(len(a2.steps), 3) a1, = p2.analyses @@ -1123,6 +1137,7 @@ def test_post_process_handler(self): self.assertEqual(a1.steps[0].feature, 'none') self.assertIsNone(a1.steps[0].num_models_begin) self.assertIsNone(a1.steps[0].num_models_end) + self.assertEqual(a1.steps[0].details, 'empty step') def test_model_list_handler(self): """Test ModelListHandler and ModelGroupHandler""" @@ -1241,19 +1256,24 @@ def test_ensemble_handler(self): _ihm_ensemble_info.num_ensemble_models_deposited _ihm_ensemble_info.ensemble_precision_value _ihm_ensemble_info.ensemble_file_id -1 'Cluster 1' 2 3 . dRMSD 1257 1 15.400 9 +_ihm_ensemble_info.details +1 'Cluster 1' 2 3 . dRMSD 1257 1 15.400 9 . +2 'Cluster 2' 2 . . dRMSD 1257 1 15.400 9 'cluster details' """ for fh in cif_file_handles(cif): s, = ihm.reader.read(fh) - e, = s.ensembles + e, e2 = s.ensembles self.assertEqual(e.model_group._id, '3') self.assertEqual(e.num_models, 1257) self.assertEqual(e.post_process._id, '2') self.assertIsNone(e.clustering_method) self.assertEqual(e.clustering_feature, 'dRMSD') self.assertEqual(e.name, 'Cluster 1') + self.assertIsNone(e.details) self.assertAlmostEqual(e.precision, 15.4, places=1) self.assertEqual(e.file._id, '9') + self.assertIsNone(e2.model_group) + self.assertEqual(e2.details, 'cluster details') def test_density_handler(self): """Test DensityHandler""" @@ -1581,13 +1601,14 @@ def test_atom_site_handler(self): _atom_site.Cartn_x _atom_site.Cartn_y _atom_site.Cartn_z +_atom_site.occupancy _atom_site.label_entity_id _atom_site.auth_asym_id _atom_site.B_iso_or_equiv _atom_site.pdbx_PDB_model_num _atom_site.ihm_model_id -ATOM 1 N N . SER 1 A 54.401 -49.984 -35.287 1 A . 1 1 -HETATM 2 C CA . SER . B 54.452 -48.492 -35.210 1 A 42.0 1 1 +ATOM 1 N N . SER 1 A 54.401 -49.984 -35.287 . 1 A . 1 1 +HETATM 2 C CA . SER . B 54.452 -48.492 -35.210 0.200 1 A 42.0 1 1 """) s, = ihm.reader.read(fh) m = s.state_groups[0][0][0][0] @@ -1601,6 +1622,7 @@ def test_atom_site_handler(self): self.assertAlmostEqual(a1.z, -35.287, places=2) self.assertEqual(a1.het, False) self.assertIsNone(a1.biso) + self.assertIsNone(a1.occupancy) self.assertEqual(a2.asym_unit._id, 'B') self.assertIsNone(a2.seq_id) @@ -1608,6 +1630,7 @@ def test_atom_site_handler(self): self.assertEqual(a2.type_symbol, 'C') self.assertEqual(a2.het, True) self.assertAlmostEqual(a2.biso, 42.0, places=0) + self.assertAlmostEqual(a2.occupancy, 0.2, places=1) def test_atom_site_handler_auth_seq_id(self): """Test AtomSiteHandler handling of auth_seq_id""" @@ -1669,6 +1692,7 @@ def test_derived_distance_restraint_handler(self): _ihm_poly_atom_feature.comp_id _ihm_poly_atom_feature.atom_id 1 1 1 A 1 ALA CA +2 1 1 . 1 ALA CB # loop_ _ihm_poly_residue_feature.ordinal_id @@ -1680,6 +1704,7 @@ def test_derived_distance_restraint_handler(self): _ihm_poly_residue_feature.seq_id_end _ihm_poly_residue_feature.comp_id_end 1 2 1 B 2 CYS 3 GLY +2 2 1 . 2 CYS 3 GLY # loop_ _ihm_non_poly_feature.ordinal_id @@ -1689,7 +1714,9 @@ def test_derived_distance_restraint_handler(self): _ihm_non_poly_feature.comp_id _ihm_non_poly_feature.atom_id 1 3 3 C HEM FE -2 4 3 C HEM . +2 3 3 . HEM FE +3 4 3 C HEM . +4 4 3 . HEM . # loop_ _ihm_pseudo_site_feature.feature_id @@ -1702,6 +1729,17 @@ def test_derived_distance_restraint_handler(self): """ rsr = """ loop_ +_ihm_feature_list.feature_id +_ihm_feature_list.feature_type +_ihm_feature_list.entity_type +_ihm_feature_list.details +1 atom polymer 'test feature' +2 'residue range' polymer . +3 atom non-polymer . +4 atom non-polymer . +5 'pseudo site' other . +# +loop_ _ihm_derived_distance_restraint.id _ihm_derived_distance_restraint.group_id _ihm_derived_distance_restraint.feature_id_1 @@ -1710,12 +1748,13 @@ def test_derived_distance_restraint_handler(self): _ihm_derived_distance_restraint.distance_lower_limit _ihm_derived_distance_restraint.distance_upper_limit _ihm_derived_distance_restraint.probability +_ihm_derived_distance_restraint.mic_value _ihm_derived_distance_restraint.group_conditionality _ihm_derived_distance_restraint.dataset_list_id -1 . 1 2 'lower bound' 25.000 . 0.800 . 97 -2 . 1 4 'upper bound' . 45.000 0.800 ALL 98 -3 1 1 2 'lower and upper bound' 22.000 45.000 0.800 ANY 99 -4 1 5 3 'harmonic' 35.000 35.000 0.800 ALL . +1 . 1 2 'lower bound' 25.000 . 0.800 0.400 . 97 +2 . 1 4 'upper bound' . 45.000 0.800 . ALL 98 +3 1 1 2 'lower and upper bound' 22.000 45.000 0.800 . ANY 99 +4 1 5 3 'harmonic' 35.000 35.000 0.800 . ALL . """ # Test both ways to make sure features still work if they are # referenced by ID before their type is known @@ -1729,32 +1768,46 @@ def test_derived_distance_restraint_handler(self): self.assertEqual(r1.dataset._id, '97') self.assertIsInstance(r1.feature1, ihm.restraint.AtomFeature) - self.assertEqual(len(r1.feature1.atoms), 1) + self.assertEqual(len(r1.feature1.atoms), 2) self.assertEqual(r1.feature1.atoms[0].id, 'CA') self.assertEqual(r1.feature1.atoms[0].residue.seq_id, 1) + self.assertIsNone(r1.feature1.atoms[0].residue.entity) + self.assertEqual(r1.feature1.atoms[1].id, 'CB') + self.assertEqual(r1.feature1.atoms[1].residue.seq_id, 1) + self.assertIsNone(r1.feature1.atoms[1].residue.asym) + self.assertEqual(r1.feature1.details, 'test feature') self.assertIsInstance(r1.feature2, ihm.restraint.ResidueFeature) - self.assertEqual(len(r1.feature2.ranges), 1) + self.assertEqual(len(r1.feature2.ranges), 2) self.assertEqual(r1.feature2.ranges[0].seq_id_range, (2,3)) + self.assertIsInstance(r1.feature2.ranges[0], ihm.AsymUnitRange) + self.assertEqual(r1.feature2.ranges[1].seq_id_range, (2,3)) + self.assertIsInstance(r1.feature2.ranges[1], ihm.EntityRange) self.assertIsInstance(r1.distance, ihm.restraint.LowerBoundDistanceRestraint) self.assertAlmostEqual(r1.distance.distance, 25.000, places=1) self.assertAlmostEqual(r1.probability, 0.8000, places=1) + self.assertAlmostEqual(r1.mic_value, 0.4000, places=1) self.assertIsNone(r1.restrain_all) self.assertEqual(r2.restrain_all, True) self.assertEqual(r3.restrain_all, False) self.assertIsInstance(r2.feature2, ihm.restraint.NonPolyFeature) - self.assertEqual(len(r2.feature2.asyms), 1) - self.assertEqual(r2.feature2.asyms[0]._id, 'C') + self.assertEqual(len(r2.feature2.objs), 2) + self.assertIsInstance(r2.feature2.objs[0], ihm.AsymUnit) + self.assertEqual(r2.feature2.objs[0]._id, 'C') + self.assertIsInstance(r2.feature2.objs[1], ihm.Entity) self.assertIsInstance(r2.distance, ihm.restraint.UpperBoundDistanceRestraint) + self.assertIsNone(r2.mic_value) self.assertIsInstance(r3.distance, ihm.restraint.LowerUpperBoundDistanceRestraint) self.assertIsInstance(r4.distance, ihm.restraint.HarmonicDistanceRestraint) self.assertIsInstance(r4.feature2, ihm.restraint.AtomFeature) + self.assertIsNone(r4.feature2.atoms[0].residue.entity) + self.assertIsNone(r4.feature2.atoms[1].residue.asym) self.assertIsInstance(r4.feature1, ihm.restraint.PseudoSiteFeature) self.assertAlmostEqual(r4.feature1.x, 10.0, places=1) @@ -1771,8 +1824,7 @@ def test_sphere_handler(self): _ihm_geometric_object_list.object_type _ihm_geometric_object_list.object_name _ihm_geometric_object_list.object_description -_ihm_geometric_object_list.other_details -1 sphere 'my sphere' 'a test sphere' 'some details' +1 sphere 'my sphere' 'a test sphere' """ spheres = """ loop_ @@ -1792,7 +1844,6 @@ def test_sphere_handler(self): self.assertIsInstance(s2, ihm.geometry.Sphere) self.assertEqual(s1.name, 'my sphere') self.assertEqual(s1.description, 'a test sphere') - self.assertEqual(s1.details, 'some details') self.assertAlmostEqual(s1.center.x, 1.000, places=1) self.assertAlmostEqual(s1.center.y, 2.000, places=1) self.assertAlmostEqual(s1.center.z, 3.000, places=1) @@ -1810,8 +1861,7 @@ def test_torus_handler(self): _ihm_geometric_object_list.object_type _ihm_geometric_object_list.object_name _ihm_geometric_object_list.object_description -_ihm_geometric_object_list.other_details -1 torus . . . +1 torus . . """ tori = """ loop_ @@ -1846,10 +1896,9 @@ def test_half_torus_handler(self): _ihm_geometric_object_list.object_type _ihm_geometric_object_list.object_name _ihm_geometric_object_list.object_description -_ihm_geometric_object_list.other_details -1 half-torus . . . -2 half-torus . . . -3 half-torus . . . +1 half-torus . . +2 half-torus . . +3 half-torus . . """ tori = """ loop_ @@ -1901,9 +1950,8 @@ def test_axis_handler(self): _ihm_geometric_object_list.object_type _ihm_geometric_object_list.object_name _ihm_geometric_object_list.object_description -_ihm_geometric_object_list.other_details -1 axis . . . -2 axis . . . +1 axis . . +2 axis . . """ axes = """ loop_ @@ -1932,9 +1980,8 @@ def test_plane_handler(self): _ihm_geometric_object_list.object_type _ihm_geometric_object_list.object_name _ihm_geometric_object_list.object_description -_ihm_geometric_object_list.other_details -1 plane . . . -2 plane . . . +1 plane . . +2 plane . . """ planes = """ loop_ @@ -2160,12 +2207,13 @@ def test_cross_link_list_handler(self): _ihm_cross_link_list.comp_id_2 _ihm_cross_link_list.linker_chem_comp_descriptor_id _ihm_cross_link_list.dataset_list_id -1 1 foo 1 2 THR foo 1 3 CYS 44 97 -2 2 foo 1 2 THR bar 2 3 PHE 44 97 -3 2 foo 1 2 THR bar 2 2 GLU 44 97 -4 3 foo 1 1 ALA bar 2 1 ASP 44 97 -5 4 foo 1 1 ALA bar 2 1 ASP 88 97 -6 5 foo 1 1 ALA bar 2 1 ASP 44 98 +_ihm_cross_link_list.details +1 1 foo 1 2 THR foo 1 3 CYS 44 97 . +2 2 foo 1 2 THR bar 2 3 PHE 44 97 'test xl' +3 2 foo 1 2 THR bar 2 2 GLU 44 97 . +4 3 foo 1 1 ALA bar 2 1 ASP 44 97 . +5 4 foo 1 1 ALA bar 2 1 ASP 88 97 . +6 5 foo 1 1 ALA bar 2 1 ASP 44 98 . """) s, = ihm.reader.read(fh) # Check grouping @@ -2179,6 +2227,7 @@ def test_cross_link_list_handler(self): self.assertEqual(xl.residue2.entity._id, '2') self.assertEqual(xl.residue1.seq_id, 2) self.assertEqual(xl.residue2.seq_id, 3) + self.assertEqual(xl.details, 'test xl') def test_cross_link_list_handler_linker_type(self): """Test CrossLinkListHandler with old-style linker_type""" @@ -2407,6 +2456,43 @@ def test_read_full_pdbx(self): s, = ihm.reader.read(f) f.close() + def test_old_file_read_default(self): + """Test default handling of old files""" + cif = """ +loop_ +_audit_conform.dict_name +_audit_conform.dict_version +mmcif_pdbx.dic 5.311 +ihm-extension.dic 0.14 +""" + s, = ihm.reader.read(StringIO(cif)) + + def test_old_file_read_fail(self): + """Test failure reading old files""" + cif = """ +loop_ +_audit_conform.dict_name +_audit_conform.dict_version +mmcif_pdbx.dic 5.311 +ihm-extension.dic 0.14 +""" + self.assertRaises(ihm.reader.OldFileError, + ihm.reader.read, StringIO(cif), reject_old_file=True) + + def test_new_file_read_ok(self): + """Test success reading not-old files""" + # File read is OK if version is new enough, or version cannot be parsed + # because it is non-int or has too many elements + for ver in ('1.0', '0.0.4', '0.0a'): + cif = """ +loop_ +_audit_conform.dict_name +_audit_conform.dict_version +mmcif_pdbx.dic 5.311 +ihm-extension.dic %s +""" % ver + s, = ihm.reader.read(StringIO(cif), reject_old_file=True) + def test_warn_unknown_category(self): """Test warnings for unknown categories""" cif = """ @@ -2604,47 +2690,72 @@ def test_flr_experiment_handler(self): _flr_experiment.ordinal_id _flr_experiment.id _flr_experiment.instrument_id -_flr_experiment.exp_setting_id +_flr_experiment.inst_setting_id +_flr_experiment.exp_condition_id _flr_experiment.sample_id _flr_experiment.details -1 1 1 22 42 "exp 1" -2 1 1 2 2 . +1 1 1 12 22 42 "exp 1" +2 1 1 2 2 2 . """) s, = ihm.reader.read(fh) flr, = s.flr_data experiment, = list(flr._collection_flr_experiment.values()) self.assertIsInstance(experiment, ihm.flr.Experiment) self.assertIsInstance(experiment.instrument_list[0], ihm.flr.Instrument) - self.assertIsInstance(experiment.exp_setting_list[0], - ihm.flr.ExpSetting) + self.assertIsInstance(experiment.inst_setting_list[0], + ihm.flr.InstSetting) + self.assertIsInstance(experiment.exp_condition_list[0], + ihm.flr.ExpCondition) self.assertIsInstance(experiment.sample_list[0], ihm.flr.Sample) self.assertEqual([i._id for i in experiment.instrument_list], ['1', '1']) - self.assertEqual([i._id for i in experiment.exp_setting_list], + self.assertEqual([i._id for i in experiment.inst_setting_list], + ['12', '2']) + self.assertEqual([i._id for i in experiment.exp_condition_list], ['22', '2']) self.assertEqual([i._id for i in experiment.sample_list], ['42', '2']) self.assertEqual(experiment.details_list, ["exp 1", None]) - def test_flr_exp_setting_handler(self): - """Test FLRExpSettingHandler""" + def test_flr_inst_setting_handler(self): + """Test FLRInstSettingHandler""" fh = StringIO(""" loop_ -_flr_exp_setting.id -_flr_exp_setting.details -1 My_Exp_setting_1 +_flr_inst_setting.id +_flr_inst_setting.details +1 My_Inst_setting_1 2 . """) s, = ihm.reader.read(fh) flr, = s.flr_data - self.assertEqual(sorted(flr._collection_flr_exp_setting.keys()), + self.assertEqual(sorted(flr._collection_flr_inst_setting.keys()), ['1', '2']) - es1 = flr._collection_flr_exp_setting['1'] - self.assertIsInstance(es1, ihm.flr.ExpSetting) - self.assertEqual(es1.details, 'My_Exp_setting_1') - es2 = flr._collection_flr_exp_setting['2'] - self.assertIsInstance(es2, ihm.flr.ExpSetting) - self.assertIsNone(es2.details) + is1 = flr._collection_flr_inst_setting['1'] + self.assertIsInstance(is1, ihm.flr.InstSetting) + self.assertEqual(is1.details, 'My_Inst_setting_1') + is2 = flr._collection_flr_inst_setting['2'] + self.assertIsInstance(is2, ihm.flr.InstSetting) + self.assertIsNone(is2.details) + + def test_flr_exp_condition_handler(self): + """Test FLRExpConditionHandler""" + fh = StringIO(""" +loop_ +_flr_exp_condition.id +_flr_exp_condition.details +1 My_Exp_condition_1 +2 . +""") + s, = ihm.reader.read(fh) + flr, = s.flr_data + self.assertEqual(sorted(flr._collection_flr_exp_condition.keys()), + ['1', '2']) + ec1 = flr._collection_flr_exp_condition['1'] + self.assertIsInstance(ec1, ihm.flr.ExpCondition) + self.assertEqual(ec1.details, 'My_Exp_condition_1') + ec2 = flr._collection_flr_exp_condition['2'] + self.assertIsInstance(ec2, ihm.flr.ExpCondition) + self.assertIsNone(ec2.details) def test_flr_instrument_handler(self): """Test FLRInstrumentHandler""" @@ -2991,7 +3102,7 @@ def test_flr_fret_calibration_parameters_handler(self): s, = ihm.reader.read(fh) flr, = s.flr_data p1 = flr._collection_flr_fret_calibration_parameters['1'] - self.assertAlmostEqual(p1.phi_acceptor, 0.350, places=1) + self.assertAlmostEqual(p1.phi_acceptor, 0.350, places=2) self.assertAlmostEqual(p1.alpha, 2.400, places=1) self.assertAlmostEqual(p1.alpha_sd, 0.1, places=1) self.assertAlmostEqual(p1.gg_gr_ratio, 0.4, places=1) @@ -3006,16 +3117,15 @@ def test_flr_fret_analysis_handler(self): loop_ _flr_fret_analysis.id _flr_fret_analysis.experiment_id +_flr_fret_analysis.type _flr_fret_analysis.sample_probe_id_1 _flr_fret_analysis.sample_probe_id_2 _flr_fret_analysis.forster_radius_id -_flr_fret_analysis.calibration_parameters_id -_flr_fret_analysis.method_name -_flr_fret_analysis.chi_square_reduced _flr_fret_analysis.dataset_list_id _flr_fret_analysis.external_file_id _flr_fret_analysis.software_id -1 8 9 2 11 12 PDA 1.500 18 42 99 +1 8 intensity-based 9 2 11 18 42 99 +2 13 lifetime-based 24 5 19 32 81 98 """) s, = ihm.reader.read(fh) flr, = s.flr_data @@ -3028,17 +3138,210 @@ def test_flr_fret_analysis_handler(self): self.assertEqual(a.sample_probe_2._id, '2') self.assertIsInstance(a.forster_radius, ihm.flr.FRETForsterRadius) self.assertEqual(a.forster_radius._id, '11') - self.assertIsInstance(a.calibration_parameters, - ihm.flr.FRETCalibrationParameters) - self.assertEqual(a.calibration_parameters._id, '12') - self.assertEqual(a.method_name, 'PDA') - self.assertAlmostEqual(a.chi_square_reduced, 1.500, places=1) + self.assertEqual(a.type, 'intensity-based') self.assertIsInstance(a.dataset, ihm.dataset.Dataset) self.assertEqual(a.dataset._id, '18') self.assertIsInstance(a.external_file, ihm.location.Location) self.assertEqual(a.external_file._id, '42') self.assertIsInstance(a.software, ihm.Software) self.assertEqual(a.software._id, '99') + b = flr._collection_flr_fret_analysis['2'] + self.assertIsInstance(b.experiment, ihm.flr.Experiment) + self.assertEqual(b.experiment._id, '13') + self.assertIsInstance(b.sample_probe_1, ihm.flr.SampleProbeDetails) + self.assertEqual(b.sample_probe_1._id, '24') + self.assertIsInstance(b.sample_probe_2, ihm.flr.SampleProbeDetails) + self.assertEqual(b.sample_probe_2._id, '5') + self.assertIsInstance(b.forster_radius, ihm.flr.FRETForsterRadius) + self.assertEqual(b.forster_radius._id, '19') + self.assertEqual(b.type, 'lifetime-based') + self.assertIsInstance(b.dataset, ihm.dataset.Dataset) + self.assertEqual(b.dataset._id, '32') + self.assertIsInstance(b.external_file, ihm.location.Location) + self.assertEqual(b.external_file._id, '81') + self.assertIsInstance(b.software, ihm.Software) + self.assertEqual(b.software._id, '98') + + def test_flr_fret_analysis_intensity_handler(self): + """Test FLRFretAnalysisIntensityHandler""" + fh = StringIO(""" +loop_ +_flr_fret_analysis_intensity.ordinal_id +_flr_fret_analysis_intensity.analysis_id +_flr_fret_analysis_intensity.calibration_parameters_id +_flr_fret_analysis_intensity.donor_only_fraction +_flr_fret_analysis_intensity.chi_square_reduced +_flr_fret_analysis_intensity.method_name +_flr_fret_analysis_intensity.details +2 5 3 0.200 1.400 PDA Details +""") + s, = ihm.reader.read(fh) + flr, = s.flr_data + a = flr._collection_flr_fret_analysis['5'] + self.assertEqual(a.type, 'intensity-based') + self.assertIsInstance(a.calibration_parameters, ihm.flr.FRETCalibrationParameters) + self.assertEqual(a.calibration_parameters._id, '3') + self.assertAlmostEqual(a.donor_only_fraction, 0.2, places=1) + self.assertAlmostEqual(a.chi_square_reduced, 1.4, places=1) + self.assertEqual(a.method_name, 'PDA') + self.assertEqual(a.details, 'Details') + + def test_flr_fret_analysis_lifetime_handler(self): + """Test FLRFretAnalysisLifetimeHandler""" + fh = StringIO(""" +loop_ +_flr_fret_analysis_lifetime.ordinal_id +_flr_fret_analysis_lifetime.analysis_id +_flr_fret_analysis_lifetime.reference_measurement_group_id +_flr_fret_analysis_lifetime.lifetime_fit_model_id +_flr_fret_analysis_lifetime.donor_only_fraction +_flr_fret_analysis_lifetime.chi_square_reduced +_flr_fret_analysis_lifetime.method_name +_flr_fret_analysis_lifetime.details +4 2 19 23 0.300 1.500 'Lifetime fit' 'Details on lifetime fit' +""") + s, = ihm.reader.read(fh) + flr, = s.flr_data + a = flr._collection_flr_fret_analysis['2'] + self.assertEqual(a.type, 'lifetime-based') + self.assertIsInstance(a.reference_measurement_group, ihm.flr.RefMeasurementGroup) + self.assertEqual(a.reference_measurement_group._id, '19') + self.assertIsInstance(a.lifetime_fit_model, ihm.flr.LifetimeFitModel) + self.assertEqual(a.lifetime_fit_model._id, '23') + self.assertAlmostEqual(a.donor_only_fraction, 0.3, places=1) + self.assertAlmostEqual(a.chi_square_reduced, 1.5, places=1) + self.assertEqual(a.method_name, 'Lifetime fit') + self.assertEqual(a.details, 'Details on lifetime fit') + + def test_flr_lifetime_fit_model_handler(self): + """Test FLRLifetimeFitModelHandler""" + fh = StringIO(""" +loop_ +_flr_lifetime_fit_model.id +_flr_lifetime_fit_model.name +_flr_lifetime_fit_model.description +_flr_lifetime_fit_model.external_file_id +_flr_lifetime_fit_model.citation_id +1 'FitModel 15' 'Description of the fit model' 3 8 +""") + s, = ihm.reader.read(fh) + flr, = s.flr_data + f = flr._collection_flr_lifetime_fit_model['1'] + self.assertEqual(f.name, 'FitModel 15') + self.assertEqual(f.description, 'Description of the fit model') + self.assertIsInstance(f.external_file, ihm.location.Location) + self.assertEqual(f.external_file._id, '3') + self.assertIsInstance(f.citation, ihm.Citation) + self.assertEqual(f.citation._id, '8') + + def test_flr_ref_measurement_handler(self): + """Test FLRRefMeasurementHandler""" + fh = StringIO(""" +loop_ +_flr_reference_measurement.id +_flr_reference_measurement.reference_sample_probe_id +_flr_reference_measurement.num_species +_flr_reference_measurement.details +4 9 2 Details1 +""") + s, = ihm.reader.read(fh) + flr, = s.flr_data + r = flr._collection_flr_ref_measurement['4'] + self.assertIsInstance(r.ref_sample_probe, ihm.flr.SampleProbeDetails) + self.assertEqual(r.ref_sample_probe._id, '9') + self.assertEqual(r.details, 'Details1') + ## num_species is set automatically when adding lifetimes to the object + self.assertEqual(r.num_species, 0) + r.add_lifetime('1') + r.add_lifetime('2') + self.assertEqual(r.num_species, 2) + + def test_flr_ref_measurement_group_handler(self): + """Test FLRRefMeasurementGroupHandler""" + fh = StringIO(""" +loop_ +_flr_reference_measurement_group.id +_flr_reference_measurement_group.num_measurements +_flr_reference_measurement_group.details +5 3 Details +""") + s, = ihm.reader.read(fh) + flr, = s.flr_data + r = flr._collection_flr_ref_measurement_group['5'] + self.assertEqual(r.details, 'Details') + ## num_measurements is set automatically when adding measurements to the object + self.assertEqual(r.num_measurements, 0) + r.add_ref_measurement('1') + self.assertEqual(r.num_measurements, 1) + r.add_ref_measurement('2') + self.assertEqual(r.num_measurements, 2) + + def test_flr_ref_measurement_group_link_handler(self): + """Test FLRRefMeasurementGroupLinkHandler""" + fh = StringIO(""" +loop_ +_flr_reference_measurement_group_link.group_id +_flr_reference_measurement_group_link.reference_measurement_id +3 12 +3 25 +5 19 +""") + s, = ihm.reader.read(fh) + flr, = s.flr_data + g1 = flr._collection_flr_ref_measurement_group['3'] + self.assertEqual(g1.num_measurements, 2) + self.assertIsInstance(g1.ref_measurement_list[0], ihm.flr.RefMeasurement) + self.assertEqual(g1.ref_measurement_list[0]._id, '12') + self.assertIsInstance(g1.ref_measurement_list[1], ihm.flr.RefMeasurement) + self.assertEqual(g1.ref_measurement_list[1]._id, '25') + g2 = flr._collection_flr_ref_measurement_group['5'] + self.assertEqual(g2.num_measurements, 1) + self.assertIsInstance(g2.ref_measurement_list[0], ihm.flr.RefMeasurement) + self.assertEqual(g2.ref_measurement_list[0]._id, '19') + + def test_flr_ref_measurement_lifetime_handler(self): + """Test FLRRefMeasurementLifetimeHandler""" + fh = StringIO(""" +loop_ +_flr_reference_measurement_lifetime.ordinal_id +_flr_reference_measurement_lifetime.reference_measurement_id +_flr_reference_measurement_lifetime.species_name +_flr_reference_measurement_lifetime.species_fraction +_flr_reference_measurement_lifetime.lifetime +1 15 species1 0.300 4.100 +2 15 species2 0.700 2.100 +3 12 species1 1.000 3.800 +""") + s, = ihm.reader.read(fh) + flr, = s.flr_data + ## Check the lifetime objects themselves + f1 = flr._collection_flr_ref_measurement_lifetime['1'] + self.assertEqual(f1.species_name, 'species1') + self.assertAlmostEqual(f1.species_fraction, 0.3, places=1) + self.assertAlmostEqual(f1.lifetime, 4.1, places=1) + f2 = flr._collection_flr_ref_measurement_lifetime['2'] + self.assertEqual(f2.species_name, 'species2') + self.assertAlmostEqual(f2.species_fraction, 0.7, places=1) + self.assertAlmostEqual(f2.lifetime, 2.1, places=1) + f3 = flr._collection_flr_ref_measurement_lifetime['3'] + self.assertEqual(f3.species_name, 'species1') + self.assertAlmostEqual(f3.species_fraction, 1.0, places=1) + self.assertAlmostEqual(f3.lifetime, 3.8, places=1) + ## And check the respective reference measurement objects + r1 = flr._collection_flr_ref_measurement['15'] + self.assertIsInstance(r1.list_of_lifetimes[0], ihm.flr.RefMeasurementLifetime) + self.assertEqual(r1.list_of_lifetimes[0].species_name, 'species1') + self.assertAlmostEqual(r1.list_of_lifetimes[0].species_fraction, 0.3, places=1) + self.assertAlmostEqual(r1.list_of_lifetimes[0].lifetime, 4.1, places=1) + self.assertIsInstance(r1.list_of_lifetimes[1], ihm.flr.RefMeasurementLifetime) + self.assertEqual(r1.list_of_lifetimes[1].species_name, 'species2') + self.assertAlmostEqual(r1.list_of_lifetimes[1].species_fraction, 0.7, places=1) + self.assertAlmostEqual(r1.list_of_lifetimes[1].lifetime, 2.1, places=1) + r2 = flr._collection_flr_ref_measurement['12'] + self.assertIsInstance(r2.list_of_lifetimes[0], ihm.flr.RefMeasurementLifetime) + self.assertEqual(r2.list_of_lifetimes[0].species_name, 'species1') + self.assertAlmostEqual(r2.list_of_lifetimes[0].species_fraction, 1.0, places=1) + self.assertAlmostEqual(r2.list_of_lifetimes[0].lifetime, 3.8, places=1) def test_flr_peak_assignment_handler(self): """Test FLRPeakAssignmentHandler""" diff --git a/modules/core/dependency/python-ihm/test/test_restraint.py b/modules/core/dependency/python-ihm/test/test_restraint.py index cde1451c3f..586fb67d54 100644 --- a/modules/core/dependency/python-ihm/test/test_restraint.py +++ b/modules/core/dependency/python-ihm/test/test_restraint.py @@ -138,6 +138,7 @@ def test_atom_cross_link(self): def test_feature(self): """Test Feature base class""" f = ihm.restraint.Feature() # does nothing + self.assertEqual(f._all_entities_or_asyms(), []) def test_residue_feature(self): """Test ResidueFeature class""" @@ -154,6 +155,10 @@ def test_residue_feature(self): f.ranges.append(a(2,2)) self.assertEqual(f.type, 'residue') + # Should work with actual Residue objects too + f.ranges.append(a.residue(3)) + self.assertEqual(f.type, 'residue') + # At least one range is a true range f.ranges.append(a(3,4)) self.assertEqual(f.type, 'residue range') @@ -194,6 +199,8 @@ def test_outer_surface_geometric_restraint(self): self.assertEqual(f.dataset, 'foo') self.assertEqual(f.object_characteristic, 'outer surface') self.assertIsNone(f.assembly) + self.assertEqual(f.feature, 'feat') + self.assertEqual(f._all_features, ('feat',)) def test_derived_distance_restraint(self): """Test DerivedDistanceRestraint class""" @@ -202,6 +209,9 @@ def test_derived_distance_restraint(self): distance='dist') self.assertEqual(f.dataset, 'foo') self.assertIsNone(f.assembly) + self.assertEqual(f.feature1, 'feat1') + self.assertEqual(f.feature2, 'feat2') + self.assertEqual(f._all_features, ('feat1', 'feat2')) if __name__ == '__main__': diff --git a/modules/core/include/TruncatedHarmonic.h b/modules/core/include/TruncatedHarmonic.h index 9b39bed8b3..876e355a61 100644 --- a/modules/core/include/TruncatedHarmonic.h +++ b/modules/core/include/TruncatedHarmonic.h @@ -31,8 +31,8 @@ enum BoundDirection { without notice unless someone tells us it is important that it does not. - \param[in] DIRECTION Whether to be an upper bound, lower bound, or - both directions. It should be one of the BoundDirection enum + \param[in] DIRECTION Whether the harmonic is of an upper bound, lower bound, or + both directions type. It should be one of the BoundDirection enum values. If it is LOWER, than the function is 0 for all values above the passed center. \see Harmonic @@ -42,8 +42,8 @@ enum BoundDirection { template class TruncatedHarmonic : public UnaryFunction { public: - /** \param[in] center The center point for the harmonic. - \param[in] k The spring constant for the harmonic. + /** \param[in] center The center point for the truncated harmonic. + \param[in] k The spring constant for the truncated harmonic. \param[in] threshold How far the harmonic term extends from the center. \param[in] limit The value to which the function converges above the threshold. @@ -54,7 +54,7 @@ class TruncatedHarmonic : public UnaryFunction { */ TruncatedHarmonic(Float center, Float k, Float threshold, Float limit) : d_(center, k, threshold, limit) {} - /** Set limit to a reasonable value. */ + /** Same as other constructor, but automatically set limit to a reasonable default value. */ TruncatedHarmonic(Float center, Float k, Float threshold) : d_(center, k, threshold, k * square(threshold)) {} virtual DerivativePair evaluate_with_derivative(double feature) const @@ -77,11 +77,16 @@ class TruncatedHarmonic : public UnaryFunction { internal::TruncatedHarmonicData d_; }; -//! A specialization for the upper bound +//! A specialization of TruncatedHarmonic that may be non-zero only +//! above the center value (always zero below it) typedef TruncatedHarmonic TruncatedHarmonicUpperBound; -//! A specialization for the lower bound + +//! A specialization of TruncatedHarmonic that may be non-zero only +//! below the center value (always zero above it) typedef TruncatedHarmonic TruncatedHarmonicLowerBound; -//! A specialization + +//! A specialization of TruncatedHarmonic that may be non-zero in both +//! directions, below or above the center value typedef TruncatedHarmonic TruncatedHarmonicBound; IMPCORE_END_NAMESPACE diff --git a/modules/core/include/direction.h b/modules/core/include/direction.h index 230cf6bc83..ad9e9a51ae 100644 --- a/modules/core/include/direction.h +++ b/modules/core/include/direction.h @@ -139,7 +139,7 @@ IMP_DECORATORS(DirectionAngle, DirectionAngles, Decorators); #if !defined(IMP_DOXYGEN) && !defined(SWIG) -//! Constrain a direction as unit and its derivative as tangent. +//! Constrain a direction as unit. class IMPCOREEXPORT DirectionUnitConstraint : public IMP::Constraint { private: ParticleIndex pi_; diff --git a/modules/core/include/rigid_bodies.h b/modules/core/include/rigid_bodies.h index aadca7af8c..987d44a3ae 100644 --- a/modules/core/include/rigid_bodies.h +++ b/modules/core/include/rigid_bodies.h @@ -123,6 +123,11 @@ class IMPCOREEXPORT RigidBody : public XYZ { public: RigidMembers get_rigid_members() const; + //! Get keys for rotation quaternion. + static FloatKeys get_rotation_keys() { + return internal::rigid_body_data().quaternion_; + } + //! Returns a list of all members that are not themselves decorated as //! rigid bodies, in the form of particle indexes. const ParticleIndexes &get_member_particle_indexes() const { @@ -302,6 +307,86 @@ class IMPCOREEXPORT RigidBody : public XYZ { */ void set_reference_frame_from_members(const ParticleIndexes &members); + //! Pull back global adjoints from members. + /** Adjoints (reverse-mode sensitivities) are partial derivatives of the + score with respect to intermediate values in the scoring function + computation, such as the global coordinates of a bead within a rigid + body or the global reference frame of a nested rigid body. + + This function pulls back (back-propagates) global adjoints and local + torque on all members to the global rotation, global coordinates, and + local torque on this rigid body and the internal coordinates and + rotation of any non-rigid members. + + This is called by an internal score state after scoring function + evaluation and is not meant to be called by the user. + */ + void pull_back_members_adjoints(DerivativeAccumulator &da); + + //! Pull back global adjoints from member that is a point. + /** + @param pi index of member particle + @param da accumulator for the adjoints + */ + void pull_back_member_adjoints(ParticleIndex pi, + DerivativeAccumulator &da); + +#ifndef SWIG + /** Same as above, but uses fewer allocations. + + @param pi index of member particle + @param T transformation from this body's local coordinates to global + @param x local coordinates of the member + @param Dy adjoint on the member's global coordinates + @param Dx adjoint on the member's local coordinates + @param DT adjoint on the transformation + @param xtorque torque contribution from Dy in local coordinates + @param da accumulator for the adjoints + */ + void pull_back_member_adjoints(ParticleIndex pi, + const algebra::Transformation3D &T, + algebra::Vector3D &x, + algebra::Vector3D &Dy, + algebra::Vector3D &Dx, + algebra::Transformation3DAdjoint &DT, + algebra::Vector3D &xtorque, + DerivativeAccumulator &da); +#endif + + //! Pull back global adjoints from member that is also a rigid body. + /** + @param pi index of member particle + @param da accumulator for the adjoints + */ + void pull_back_body_member_adjoints(ParticleIndex pi, + DerivativeAccumulator &da); + +#ifndef SWIG + /** Same as above, but uses fewer allocations. + + @param pi index of member particle + @param TA transformation from this body's local coordinates to global + @param TB transformation from member's local coordinates to this + body's local coordinates + @param DTC adjoint on composition of TA and TB, which is the + transformation from the member's local coordinates to + global + @param DTA adjoint on TA + @param DTB adjoint on TB + @param betatorque torque contribution from DTC in local coordinates at + beta, the position of the member in local coordinates. + @param da accumulator for the adjoints + */ + void pull_back_body_member_adjoints(ParticleIndex pi, + const algebra::Transformation3D &TA, + algebra::Transformation3D &TB, + algebra::Transformation3DAdjoint &DTC, + algebra::Transformation3DAdjoint &DTA, + algebra::Transformation3DAdjoint &DTB, + algebra::Vector3D &betatorque, + DerivativeAccumulator &da); +#endif + /** Update the translational and rotational derivatives on the rigid body center of mass, using the Cartesian derivative vector at a speicified location (the point where the force is @@ -315,7 +400,8 @@ class IMPCOREEXPORT RigidBody : public XYZ { @param da Accumulates the output derivative over the rigid body center of mass (translation and rotation torque, quaternion) */ - inline void add_to_derivatives(const algebra::Vector3D &local_derivative, + IMPCORE_DEPRECATED_METHOD_DECL(2.12) + void add_to_derivatives(const algebra::Vector3D &local_derivative, const algebra::Vector3D &local_location, DerivativeAccumulator &da); @@ -330,7 +416,8 @@ class IMPCOREEXPORT RigidBody : public XYZ { @param da Accumulates the output derivative over the rigid body center of mass (translation and rotation torque, quaternion) */ - inline void add_to_derivatives(const algebra::Vector3D &local_derivative, + IMPCORE_DEPRECATED_METHOD_DECL(2.12) + void add_to_derivatives(const algebra::Vector3D &local_derivative, const algebra::Vector3D &global_derivative, const algebra::Vector3D &local_location, const algebra::Rotation3D &rot_local_to_global, @@ -351,7 +438,8 @@ class IMPCOREEXPORT RigidBody : public XYZ { global coordinates. @param da Accumulates the output derivatives. */ - inline void add_to_rotational_derivatives(const algebra::Vector4D &other_qderiv, + IMPCORE_DEPRECATED_METHOD_DECL(2.12) + void add_to_rotational_derivatives(const algebra::Vector4D &other_qderiv, const algebra::Rotation3D &rot_other_to_local, const algebra::Rotation3D &rot_local_to_global, DerivativeAccumulator &da); @@ -515,53 +603,6 @@ void RigidBody::add_to_torque(const algebra::Vector3D &torque_local, } } - -// inline implementation -void RigidBody::add_to_derivatives(const algebra::Vector3D &deriv_local, - const algebra::Vector3D &deriv_global, - const algebra::Vector3D &local, - const algebra::Rotation3D &rot_local_to_global, - DerivativeAccumulator &da) { - // IMP_LOG_TERSE( "Accumulating rigid body derivatives" << std::endl); - XYZ::add_to_derivatives(deriv_global, da); - - Eigen::RowVector4d q = - Eigen::RowVector3d(deriv_global.get_data()) * - rot_local_to_global.get_gradient(Eigen::Vector3d(local.get_data())); - - for (unsigned int i = 0; i < 4; ++i) { - get_model()->add_to_derivative(internal::rigid_body_data().quaternion_[i], - get_particle_index(), q[i], da); - } - algebra::Vector3D torque = algebra::get_vector_product(local, deriv_local); - add_to_torque(torque, da); -} - -// inline implementation -void RigidBody::add_to_derivatives(const algebra::Vector3D &deriv_local, - const algebra::Vector3D &local, - DerivativeAccumulator &da) { - algebra::Rotation3D rot_local_to_global = - get_reference_frame().get_transformation_to().get_rotation(); - const algebra::Vector3D deriv_global = rot_local_to_global * deriv_local; - add_to_derivatives(deriv_local, deriv_global, local, - rot_local_to_global, da); -} - -// inline implementation -void RigidBody::add_to_rotational_derivatives(const algebra::Vector4D &other_qderiv, - const algebra::Rotation3D &rot_other_to_local, - const algebra::Rotation3D &rot_local_to_global, - DerivativeAccumulator &da) { - Eigen::MatrixXd derivs = - algebra::get_gradient_of_composed_with_respect_to_first( - rot_local_to_global, rot_other_to_local); - Eigen::RowVector4d qderiv = Eigen::RowVector4d(other_qderiv.get_data()) * derivs; - for (unsigned int i = 0; i < 4; ++i) { - get_model()->add_to_derivative(internal::rigid_body_data().quaternion_[i], - get_particle_index(), qderiv[i], da); - } -} #endif @@ -742,19 +783,12 @@ class IMPCOREEXPORT NonRigidMember : public RigidBodyMember { to global coordinates. @param da Accumulates the output derivatives. */ - void add_to_internal_rotational_derivatives(const algebra::Vector4D &local_qderiv, - const algebra::Rotation3D &rot_local_to_parent, - const algebra::Rotation3D &rot_parent_to_global, - DerivativeAccumulator &da) { - Eigen::MatrixXd derivs = - algebra::get_gradient_of_composed_with_respect_to_second( - rot_parent_to_global, rot_local_to_parent); - Eigen::RowVector4d qderiv = Eigen::RowVector4d(local_qderiv.get_data()) * derivs; - for (unsigned int i = 0; i < 4; ++i) { - get_model()->add_to_derivative(get_internal_rotation_keys()[i], - get_particle_index(), qderiv[i], da); - } - } + IMPCORE_DEPRECATED_METHOD_DECL(2.12) + void add_to_internal_rotational_derivatives( + const algebra::Vector4D &local_qderiv, + const algebra::Rotation3D &rot_local_to_parent, + const algebra::Rotation3D &rot_parent_to_global, + DerivativeAccumulator &da); /** Add to internal quaternion derivatives of this non-rigid body diff --git a/modules/core/src/direction.cpp b/modules/core/src/direction.cpp index e31ac4ab47..a34e182bbb 100644 --- a/modules/core/src/direction.cpp +++ b/modules/core/src/direction.cpp @@ -188,15 +188,7 @@ void DirectionUnitConstraint::do_update_attributes() { d.set_direction(d.get_direction()); } -void DirectionUnitConstraint::do_update_derivatives(DerivativeAccumulator *da) { - // project derivative onto tangent plane - Direction d(get_model(), pi_); - algebra::Vector3D derv, rad, correction; - derv = d.get_direction_derivatives(); - rad = d.get_direction(); - correction = -(derv * rad) * rad; - d.add_to_direction_derivatives(correction, *da); -} +void DirectionUnitConstraint::do_update_derivatives(DerivativeAccumulator *) {} ModelObjectsTemp DirectionUnitConstraint::do_get_inputs() const { return ModelObjectsTemp(1, get_model()->get_particle(pi_)); diff --git a/modules/core/src/rigid_bodies.cpp b/modules/core/src/rigid_bodies.cpp index 0a71791fc8..ec37462445 100644 --- a/modules/core/src/rigid_bodies.cpp +++ b/modules/core/src/rigid_bodies.cpp @@ -180,50 +180,8 @@ void AccumulateRigidBodyDerivatives::apply_index( internal::rigid_body_data().quaternion_[j]); } #endif - algebra::Rotation3D rot = //! from global to internal - rb.get_reference_frame().get_transformation_from().get_rotation(); - algebra::Rotation3D roti = //! from internal to global - rb.get_reference_frame().get_transformation_to().get_rotation(); - const ParticleIndexes &rbis = rb.get_member_particle_indexes(); - for (unsigned int i = 0; i < rbis.size(); ++i) { - RigidBodyMember d(rb.get_model(), rbis[i]); - const algebra::Vector3D &deriv = d.get_derivatives(); - if (deriv.get_squared_magnitude() > 0) { - algebra::Vector3D dv = rot * deriv; - rb.add_to_derivatives(dv, deriv, d.get_internal_coordinates(), roti, da); - if (NonRigidMember::get_is_setup(d)) { - NonRigidMember(d).add_to_internal_derivatives(dv, da); - } - } - } - const ParticleIndexes &rbbis = rb.get_body_member_particle_indexes(); - for (unsigned int i = 0; i < rbbis.size(); ++i) { - RigidBodyMember d(rb.get_model(), rbbis[i]); - bool is_nonrigid = NonRigidMember::get_is_setup(d); - const algebra::Vector3D &deriv = d.get_derivatives(); - if (deriv.get_squared_magnitude() > 0) { - algebra::Vector3D dv = rot * deriv; - rb.add_to_derivatives(dv, deriv, d.get_internal_coordinates(), roti, da); - if (is_nonrigid) { - NonRigidMember(d).add_to_internal_derivatives(dv, da); - } - } - - algebra::Rotation3D rot_memloc_to_loc = d.get_internal_transformation().get_rotation(); - algebra::Vector3D mtorque = RigidBody(d).get_torque(); - if (mtorque.get_squared_magnitude() > 0) { - rb.add_to_torque(rot_memloc_to_loc * mtorque, da); - } - algebra::Vector4D mderiv = RigidBody(d).get_rotational_derivatives(); - if (mderiv.get_squared_magnitude() > 0) { - rb.add_to_rotational_derivatives(mderiv, rot_memloc_to_loc, roti, da); - if (is_nonrigid) { - NonRigidMember(d).add_to_internal_rotational_derivatives( - mderiv, rot_memloc_to_loc, roti, da); - } - } - } + rb.pull_back_members_adjoints(da); IMP_LOG_TERSE("Rigid body derivative is " << m->get_particle(pi)->get_derivative( @@ -249,15 +207,15 @@ void AccumulateRigidBodyDerivatives::apply_index( algebra::Vector3D dv = d.get_derivatives(); v += dv; // IMP_LOG_TERSE( "Adding " << dv << " to derivative" << std::endl); - q += rot.get_gradient(Eigen::Vector3d( - d.get_internal_coordinates().get_data())).transpose() * + q += rot.get_jacobian_of_rotated(Eigen::Vector3d( + d.get_internal_coordinates().get_data()), false).transpose() * Eigen::Vector3d(dv.get_data()); if (RigidBody::get_is_setup(d)) { algebra::Rotation3D mrot = RigidBodyMember(d).get_internal_transformation().get_rotation(); Eigen::Vector4d mq(RigidBody(d).get_rotational_derivatives().get_data()); Eigen::MatrixXd dq = - algebra::get_gradient_of_composed_with_respect_to_first(rot, mrot).transpose(); + algebra::get_jacobian_of_composed_wrt_first(rot, mrot, false).transpose(); q += dq * mq; } } @@ -277,10 +235,10 @@ void AccumulateRigidBodyDerivatives::apply_index( internal::rigid_body_data().quaternion_[1]) << " " << rb.get_particle()->get_derivative( - internal::rigid_body_data().quaternion_[1]) + internal::rigid_body_data().quaternion_[2]) << " " << rb.get_particle()->get_derivative( - internal::rigid_body_data().quaternion_[2]) + internal::rigid_body_data().quaternion_[3]) << ": " << q); } #if IMP_HAS_CHECKS >= IMP_INTERNAL @@ -658,6 +616,124 @@ void RigidBody::update_members() { } } +void RigidBody::pull_back_members_adjoints(DerivativeAccumulator &da) { + algebra::Transformation3D TA = get_reference_frame().get_transformation_to(); + algebra::Transformation3D TB; + algebra::Transformation3DAdjoint DTA, DTB, DTC; + algebra::Vector3D beta, betatorque; + + const ParticleIndexes &mis = get_member_particle_indexes(); + for (unsigned int i = 0; i < mis.size(); ++i) { + // y = T(A, alpha) * beta = A * beta + alpha + pull_back_member_adjoints(mis[i], TA, beta, + DTC.second, DTB.second, DTA, betatorque, da); + } + + const ParticleIndexes &bmis = get_body_member_particle_indexes(); + for (unsigned int i = 0; i < bmis.size(); ++i) { + // TC = T(A, alpha) * T(B, beta) = T(A B, A * beta + alpha) + pull_back_body_member_adjoints( + bmis[i], TA, TB, DTC, DTA, DTB, betatorque, da + ); + } +} + +void RigidBody::pull_back_member_adjoints(ParticleIndex pi, + const algebra::Transformation3D &T, + algebra::Vector3D &x, + algebra::Vector3D &Dy, + algebra::Vector3D &Dx, + algebra::Transformation3DAdjoint &DT, + algebra::Vector3D &xtorque, + DerivativeAccumulator &da) { + IMP_INTERNAL_CHECK(RigidBodyMember::get_is_setup(get_model(), pi), + "Particle must be a rigid body member."); + RigidBodyMember rbm(get_model(), pi); + IMP_INTERNAL_CHECK( + rbm.get_rigid_body().get_particle_index() == get_particle_index(), + "Rigid body member must be a member of this rigid body." + ); + + x = rbm.get_internal_coordinates(); + Dy = rbm.get_derivatives(); + + bool is_nonrigid = get_model()->get_attribute( + internal::rigid_body_data().is_rigid_key_, pi) != 1; + + T.get_transformed_adjoint(x, Dy, &Dx, &DT); + + if (is_nonrigid) + NonRigidMember(get_model(), pi).add_to_internal_derivatives(Dx, da); + + add_to_rotational_derivatives(DT.first, da); + XYZ::add_to_derivatives(DT.second, da); + + xtorque = algebra::get_vector_product(x, Dx); + add_to_torque(xtorque, da); +} + +void RigidBody::pull_back_member_adjoints(ParticleIndex pi, + DerivativeAccumulator &da) { + algebra::Transformation3D T = get_reference_frame().get_transformation_to(); + algebra::Vector3D x, Dx, Dy, xtorque; + algebra::Transformation3DAdjoint DT; + + pull_back_member_adjoints(pi, T, x, Dy, Dx, DT, xtorque, da); +} + +void RigidBody::pull_back_body_member_adjoints(ParticleIndex pi, + const algebra::Transformation3D &TA, + algebra::Transformation3D &TB, + algebra::Transformation3DAdjoint &DTC, + algebra::Transformation3DAdjoint &DTA, + algebra::Transformation3DAdjoint &DTB, + algebra::Vector3D &betatorque, + DerivativeAccumulator &da) { + IMP_INTERNAL_CHECK(RigidBodyMember::get_is_setup(get_model(), pi), + "Particle must be a rigid body member."); + IMP_INTERNAL_CHECK(RigidBody::get_is_setup(get_model(), pi), + "Particle must be a rigid body."); + RigidBodyMember rbm(get_model(), pi); + IMP_INTERNAL_CHECK( + rbm.get_rigid_body().get_particle_index() == get_particle_index(), + "Rigid body member must be a member of this rigid body." + ); + RigidBody rb(get_model(), pi); + + TB = rbm.get_internal_transformation(); + DTC.first = rb.get_rotational_derivatives(); + DTC.second = rb.get_derivatives(); + + bool is_nonrigid = get_model()->get_attribute( + internal::rigid_body_data().is_rigid_key_, pi) != 1; + + algebra::compose_adjoint(TA, TB, DTC, &DTA, &DTB); + + if (is_nonrigid) { + algebra::compose_adjoint(TA, TB, DTC, &DTA, &DTB); + NonRigidMember rm(get_model(), pi); + rm.add_to_internal_derivatives(DTB.second, da); + rm.add_to_internal_rotational_derivatives(DTB.first, da); + } + + add_to_rotational_derivatives(DTA.first, da); + XYZ::add_to_derivatives(DTA.second, da); + + betatorque = algebra::get_vector_product(TB.get_translation(), DTB.second) + + TB.get_rotation() * rb.get_torque(); + add_to_torque(betatorque, da); +} + +void RigidBody::pull_back_body_member_adjoints(ParticleIndex pi, + DerivativeAccumulator &da) { + algebra::Transformation3D TA = get_reference_frame().get_transformation_to(); + algebra::Transformation3D TB; + algebra::Transformation3DAdjoint DTA, DTB, DTC; + algebra::Vector3D betatorque; + + pull_back_body_member_adjoints(pi, TA, TB, DTC, DTA, DTB, betatorque, da); +} + RigidMembers RigidBody::get_rigid_members() const { RigidMembers ret; { @@ -1013,4 +1089,68 @@ ParticleIndex get_root_rigid_body(RigidMember m) { return body.get_particle_index(); } +void RigidBody::add_to_derivatives(const algebra::Vector3D &deriv_local, + const algebra::Vector3D &deriv_global, + const algebra::Vector3D &local, + const algebra::Rotation3D &rot_local_to_global, + DerivativeAccumulator &da) { + IMPCORE_DEPRECATED_FUNCTION_DEF(2.12, "Updating of derivatives is now handled after evaluation by RigidBody::pull_back_members_adjoints."); + // IMP_LOG_TERSE( "Accumulating rigid body derivatives" << std::endl); + XYZ::add_to_derivatives(deriv_global, da); + + Eigen::RowVector4d q = + Eigen::RowVector3d(deriv_global.get_data()) * + rot_local_to_global.get_jacobian_of_rotated(Eigen::Vector3d(local.get_data()), + false); + + for (unsigned int i = 0; i < 4; ++i) { + get_model()->add_to_derivative(internal::rigid_body_data().quaternion_[i], + get_particle_index(), q[i], da); + } + algebra::Vector3D torque = algebra::get_vector_product(local, deriv_local); + add_to_torque(torque, da); +} + +void RigidBody::add_to_derivatives(const algebra::Vector3D &deriv_local, + const algebra::Vector3D &local, + DerivativeAccumulator &da) { + IMPCORE_DEPRECATED_FUNCTION_DEF(2.12, "Updating of derivatives is now handled after evaluation by RigidBody::pull_back_members_adjoints."); + algebra::Rotation3D rot_local_to_global = + get_reference_frame().get_transformation_to().get_rotation(); + const algebra::Vector3D deriv_global = rot_local_to_global * deriv_local; + add_to_derivatives(deriv_local, deriv_global, local, + rot_local_to_global, da); +} + +void RigidBody::add_to_rotational_derivatives(const algebra::Vector4D &other_qderiv, + const algebra::Rotation3D &rot_other_to_local, + const algebra::Rotation3D &rot_local_to_global, + DerivativeAccumulator &da) { + IMPCORE_DEPRECATED_FUNCTION_DEF(2.12, "Updating of derivatives is now handled after evaluation by RigidBody::pull_back_members_adjoints."); + Eigen::MatrixXd derivs = + algebra::get_jacobian_of_composed_wrt_first( + rot_local_to_global, rot_other_to_local, false); + Eigen::RowVector4d qderiv = Eigen::RowVector4d(other_qderiv.get_data()) * derivs; + for (unsigned int i = 0; i < 4; ++i) { + get_model()->add_to_derivative(internal::rigid_body_data().quaternion_[i], + get_particle_index(), qderiv[i], da); + } +} + +void NonRigidMember::add_to_internal_rotational_derivatives( + const algebra::Vector4D &local_qderiv, + const algebra::Rotation3D &rot_local_to_parent, + const algebra::Rotation3D &rot_parent_to_global, + DerivativeAccumulator &da) { + IMPCORE_DEPRECATED_FUNCTION_DEF(2.12, "Updating of derivatives is now handled after evaluation by RigidBody::pull_back_members_adjoints."); + Eigen::MatrixXd derivs = + algebra::get_jacobian_of_composed_wrt_second( + rot_parent_to_global, rot_local_to_parent, false); + Eigen::RowVector4d qderiv = Eigen::RowVector4d(local_qderiv.get_data()) * derivs; + for (unsigned int i = 0; i < 4; ++i) { + get_model()->add_to_derivative(get_internal_rotation_keys()[i], + get_particle_index(), qderiv[i], da); + } +} + IMPCORE_END_NAMESPACE diff --git a/modules/core/test/standards_exceptions b/modules/core/test/standards_exceptions index 110d5327b4..69dca31c9a 100644 --- a/modules/core/test/standards_exceptions +++ b/modules/core/test/standards_exceptions @@ -1,10 +1,16 @@ value_object_exceptions=['HierarchyVisitor', 'ModifierVisitor'] function_name_exceptions=['RigidBody.normalize_rotation', + 'RigidBody.pull_back_body_member_adjoints', + 'RigidBody.pull_back_member_adjoints', + 'RigidBody.pull_back_members_adjoints', 'transform', 'ExperimentalTree.connect', 'ExperimentalTree.finalize', 'Gaussian.normalize_rotation', + 'Gaussian.pull_back_body_member_adjoints', + 'Gaussian.pull_back_member_adjoints', + 'Gaussian.pull_back_members_adjoints', 'assign_blame', 'Surface.reflect', 'Direction.reflect', diff --git a/modules/core/test/test_direction.py b/modules/core/test/test_direction.py index 388e0d2d9e..69f534a335 100644 --- a/modules/core/test/test_direction.py +++ b/modules/core/test/test_direction.py @@ -21,11 +21,11 @@ def do_add_score_and_derivatives(self, sa): angle = math.acos(d.get_direction() * self.u) self.uf.evaluate(angle) if sa.get_derivative_accumulator(): - score, derv = self.uf.evaluate_with_derivative(angle) + score, aderv = self.uf.evaluate_with_derivative(angle) if angle == 0: derv = IMP.algebra.Vector3D(0, 0, 0) else: - derv = derv * -self.u / math.sin(angle) + derv = aderv * -self.u / math.sin(angle) d.add_to_direction_derivatives(derv, sa.get_derivative_accumulator()) else: @@ -88,16 +88,17 @@ def test_direction_unit_length_enforced(self): self.assertAlmostEqual((IMP.algebra.Vector3D(0, 1, 0) - d.get_direction()).get_magnitude(), 0.) - def test_direction_tangent_enforced(self): - """Test derivative is tangent after score evaluate.""" + def test_direction_derivative(self): m = IMP.Model() p = IMP.Particle(m) d = IMP.core.Direction.setup_particle( - p, IMP.algebra.Vector3D(0, 0, 1)) - r = DirectionRestraint(m, p.get_index(), - IMP.algebra.Vector3D(1, 1, 1), 10.) + p, IMP.algebra.Vector3D(0, 0, 1) + ) + k = 10 + v = IMP.algebra.Vector3D(1, 1, 1).get_unit_vector() + r = DirectionRestraint(m, p.get_index(), v, k) r.evaluate(True) - exp_derv = -10 * IMP.algebra.Vector3D(1, 1, 0) / 3**.5 + exp_derv = -k * v self.assertAlmostEqual( (exp_derv - d.get_direction_derivatives()).get_magnitude(), 0., delta=1e-6) @@ -120,7 +121,9 @@ def test_optimization(self): self.assertGreater(sf.evaluate(False), 1.) opt = IMP.core.ConjugateGradients(m) opt.set_scoring_function(sf) - self.assertLess(opt.optimize(50), 1e-6) + for i in range(50): + opt.optimize(1) + self.assertLess(opt.optimize(1), 1e-6) self.assertAlmostEqual(math.acos(d.get_direction() * v), 0., delta=1e-6) diff --git a/modules/core/test/test_direction_angle.py b/modules/core/test/test_direction_angle.py index 017f6aea80..76e1e515e5 100644 --- a/modules/core/test/test_direction_angle.py +++ b/modules/core/test/test_direction_angle.py @@ -89,7 +89,6 @@ def test_derivatives_updated(self): ds[1].set_direction(rot.get_rotated(ds[0].get_direction())) r.evaluate(True) exp_derv = 2**.5 * math.pi / 4. * ds[1].get_direction() - exp_derv -= (exp_derv * ds[0].get_direction()) * ds[0].get_direction() self.assertAlmostEqual( (exp_derv - ds[0].get_direction_derivatives()).get_magnitude(), 0., delta=1e-6) diff --git a/modules/core/test/test_nested_rigid_bodies.py b/modules/core/test/test_nested_rigid_bodies.py index 3774092c60..203360880f 100644 --- a/modules/core/test/test_nested_rigid_bodies.py +++ b/modules/core/test/test_nested_rigid_bodies.py @@ -6,6 +6,40 @@ import numpy as np +class GradientCalculator: + + def __init__(self, sf, pis, fks): + self.sf = sf + self.m = sf.get_model() + self.pis = pis + self.fks = fks + + def get_approximate_gradient(self, eps=1e-6): + """Approximate gradient with central differences""" + grad = [] + for pi, fk in zip(self.pis, self.fks): + if not self.m.get_particle(pi).get_is_optimized(fk): + grad.append(0.0) + continue + v0 = self.m.get_attribute(fk, pi) + self.m.set_attribute(fk, pi, v0 - eps / 2) + Sminus = self.sf.evaluate(True) + self.m.set_attribute(fk, pi, v0 + eps / 2) + Splus = self.sf.evaluate(True) + self.m.set_attribute(fk, pi, v0) + self.m.update() + grad.append((Splus - Sminus) / eps) + return np.array(grad) + + def get_exact_gradient(self): + """Get IMP's computed gradient""" + self.sf.evaluate(True) + return np.array( + [self.m.get_particle(pi).get_derivative(fk) + for pi, fk in zip(self.pis, self.fks)] + ) + + class DummyRestraint(IMP.Restraint): """Adds random derivatives to particles.""" @@ -140,6 +174,97 @@ def test_torque_from_nested_equal_to_unnested(self): rb_nested.get_torque(), rb_unnested.get_torque(), delta=1e-6) + def _make_beads(self, m, nbeads, center=[0, 0, 0], radius=10): + sphere = IMP.algebra.Sphere3D(center, radius) + xyzs = [] + for n in range(nbeads): + v = IMP.algebra.get_random_vector_in(sphere) + xyz = IMP.core.XYZ.setup_particle(IMP.Particle(m), v) + xyzs.append(xyz) + return xyzs + + def test_all_derivatives_correct(self): + m = IMP.Model() + + nlevels = 4 + nbeads = 10 + prob_nonrigid = 0.5 + tf = IMP.algebra.get_random_local_transformation((0, 0, 0), 10) + rf = IMP.algebra.ReferenceFrame3D(tf) + rb = IMP.core.RigidBody.setup_particle(IMP.Particle(m), rf) + rb.set_coordinates_are_optimized(True) + all_beads = [] + all_rbs = [rb] + for level in range(nlevels): + center = tf.get_translation() + ps = self._make_beads(m, nbeads, center=center, radius=10) + all_beads += ps + for p in ps: + if np.random.uniform() < prob_nonrigid: + rb.add_non_rigid_member(p) + nrm = IMP.core.NonRigidMember(p) + for k in nrm.get_internal_coordinate_keys(): + nrm.get_particle().set_is_optimized(k, True) + else: + rb.add_member(p) + + tf = IMP.algebra.get_random_local_transformation((0, 0, 0), 10) + rf = IMP.algebra.ReferenceFrame3D(tf) + rb_nested = IMP.core.RigidBody.setup_particle(IMP.Particle(m), rf) + if np.random.uniform() < prob_nonrigid: + rb.add_non_rigid_member(rb_nested) + nrm = IMP.core.NonRigidMember(rb_nested) + for k in nrm.get_internal_coordinate_keys() + nrm.get_internal_rotation_keys(): + nrm.get_particle().set_is_optimized(k, True) + else: + rb.add_member(rb_nested) + rb = rb_nested + all_rbs.append(rb) + + m.update() + ss = IMP.core.DistanceToSingletonScore( + IMP.core.Harmonic(0, 1), (0, 0, 0) + ) + rs = [IMP.core.SingletonRestraint(m, ss, p.get_particle_index()) + for p in all_beads] + sf = IMP.core.RestraintsScoringFunction(rs) + + pis, fks = list( + zip(*[(p.get_particle_index(), fk) + for p in all_beads + all_rbs[1:] + for fk in IMP.core.RigidBodyMember(p).get_internal_coordinate_keys() + ] + ) + ) + + gc = GradientCalculator(sf, pis, fks) + grad_approx = gc.get_approximate_gradient(eps=1e-6) + grad_exact = gc.get_exact_gradient() + self.assertSequenceAlmostEqual( + list(grad_exact), list(grad_approx), delta=1e-3 + ) + + for rb in all_rbs[1:]: + pi = rb.get_particle_index() + nrm = IMP.core.RigidBodyMember(rb) + fks = nrm.get_internal_rotation_keys() + gc = GradientCalculator(sf, [pi] * 4, fks) + grad_approx = gc.get_approximate_gradient(eps=1e-6) + grad_exact = gc.get_exact_gradient() + self.assertSequenceAlmostEqual( + list(grad_exact), list(grad_approx), delta=1e-3 + ) + + rb = all_rbs[0] + pi = rb.get_particle_index() + fks = rb.get_rotation_keys() + gc = GradientCalculator(sf, [pi] * 4, fks) + grad_approx = gc.get_approximate_gradient(eps=1e-6) + grad_exact = gc.get_exact_gradient() + self.assertSequenceAlmostEqual( + list(grad_exact), list(grad_approx), delta=1e-3 + ) + def test_quaternion_derivatives_from_nested_equal_to_unnested(self): """Test nested rigid bodies have same quaternion derivatives as unnested.""" sphere1 = IMP.algebra.Sphere3D([0, 0, 0], 5) @@ -229,7 +354,10 @@ def test_non_rigid_member_local_derivatives_are_correct(self): exp_lqderv = IMP.algebra.Vector4D() for i in range(4): exp_lqderv[i] = ( - lderv * rot_local_to_parent.get_derivative(lcoord, i)) + lderv * rot_local_to_parent.get_gradient_of_rotated( + lcoord, i, False + ) + ) lqderv = IMP.core.NonRigidMember( nrb).get_internal_rotational_derivatives() @@ -273,9 +401,7 @@ def test_identity_internal_rotation_propagates_projected_global_derivative(self) delta=1e-6 ) - q = rb.get_rotation().get_quaternion() - projection_matrix = np.eye(4) - np.outer(q, q) - exp_deriv = np.dot(projection_matrix, -kappa * mu) + exp_deriv = -kappa * mu # ensure that derivative is propagated correctly self.assertSequenceAlmostEqual( diff --git a/modules/foxs/src/internal/Gnuplot.cpp b/modules/foxs/src/internal/Gnuplot.cpp index df8882c548..4898036192 100644 --- a/modules/foxs/src/internal/Gnuplot.cpp +++ b/modules/foxs/src/internal/Gnuplot.cpp @@ -68,7 +68,8 @@ void Gnuplot::print_canvas_script(const std::vector& pdbs, plt_file << "set terminal canvas solid butt size 400,350 fsize 10 lw 1.5 " << "fontscale 1 name \"jsoutput_1\" jsdir \".\"" << std::endl; plt_file << "set output 'jsoutput.1.js'" << std::endl; - plt_file << "set xlabel 'q';set ylabel 'log intensity';set format y '';" + plt_file << "set xlabel 'q [Å^{-1}]';set ylabel 'log intensity'; " + << "set format y '10^{%L}'; set logscale y\n" << "set xtics nomirror;set ytics nomirror; set border 3\n" << "set style line 11 lc rgb '#808080' lt 1;unset key;" << "set border 3 back ls 11;" << std::endl; @@ -77,7 +78,7 @@ void Gnuplot::print_canvas_script(const std::vector& pdbs, for (int i = 0; i < (int)pdbs.size() && i < max_num; i++) { ColorCoder::html_hex_color(hex_color, i); std::string profile_file_name = pdbs[i] + ".dat"; - plt_file << "'" << profile_file_name << "' u 1:(log($2)) " + plt_file << "'" << profile_file_name << "' u 1:2 " << "w lines lw 2.5 lc rgb '#" << hex_color << "'"; if (i == static_cast(pdbs.size()) - 1 || i == max_num - 1) plt_file << std::endl; @@ -226,8 +227,9 @@ void Gnuplot::print_canvas_script( plt_file << "set output 'jsoutput.1.js'" << std::endl; plt_file << "set multiplot\n"; + plt_file << "set lmargin 7\n"; plt_file << "set origin 0,0;set size 1,0.3; set tmargin 0;" - << "set xlabel 'q';set ylabel ' ';set format y '';" + << "set xlabel 'q [Å^{-1}]';set ylabel 'Residual';set format y '';" << "set xtics nomirror;set ytics nomirror;unset key;" << "set border 3; set style line 11 lc rgb '#808080' lt 1;" << "set border 3 back ls 11" << std::endl; @@ -245,8 +247,9 @@ void Gnuplot::print_canvas_script( } plt_file << std::endl; // actual plots - plt_file << "set origin 0,0.3;set size 1,0.69; set bmargin 0;" - << "set xlabel ''; set format x ''; set ylabel 'log intensity';\n"; + plt_file << "set origin 0,0.3;set size 1,0.69; set bmargin 0; set tmargin 1; " + << "set xlabel ''; set format x ''; set ylabel 'log intensity'; " + << "set format y '10^{%L}'; set logscale y\n"; for (int i = 0; i < (int)fps.size() && i < max_num; i++) { ColorCoder::html_hex_color(hex_color, i); std::string pdb_name = saxs::trim_extension(fps[i].get_pdb_file_name()); @@ -255,9 +258,9 @@ void Gnuplot::print_canvas_script( std::string fit_file_name = pdb_name + "_" + profile_name + ".fit"; if (i == 0) { plt_file << "plot '" << fit_file_name - << "' u 1:(log($2)) lc rgb '#333333' pt 6 ps 0.8 "; + << "' u 1:2 lc rgb '#333333' pt 6 ps 0.8 "; } - plt_file << ", '" << fit_file_name << "' u 1:(log($4)) " + plt_file << ", '" << fit_file_name << "' u 1:4 " << "w lines lw 2.5 lc rgb '#" << hex_color << "'"; } plt_file << std::endl; diff --git a/modules/integrative_docking/bin/cross_links_score.cpp b/modules/integrative_docking/bin/cross_links_score.cpp index 5c7f123f1f..f67006cb72 100644 --- a/modules/integrative_docking/bin/cross_links_score.cpp +++ b/modules/integrative_docking/bin/cross_links_score.cpp @@ -70,7 +70,7 @@ Each docked complex will be compared against cross links in cross_links_file.")( cross_links_file = files[3]; // read pdb files, prepare particles - IMP::Model *model = new IMP::Model(); + IMP_NEW(IMP::Model, model, ()); IMP::atom::Hierarchy mhd = IMP::atom::read_pdb( receptor_pdb, model, new IMP::atom::NonWaterNonHydrogenPDBSelector(), true, true); diff --git a/modules/integrative_docking/bin/cross_links_single_score.cpp b/modules/integrative_docking/bin/cross_links_single_score.cpp index 20bbf4bd0d..ed1c8e9207 100644 --- a/modules/integrative_docking/bin/cross_links_single_score.cpp +++ b/modules/integrative_docking/bin/cross_links_single_score.cpp @@ -67,7 +67,7 @@ Each docked complex will be compared against cross links in cross_links_file.")( cross_links_file = files[1]; // read pdb files, prepare particles - IMP::Model *model = new IMP::Model(); + IMP_NEW(IMP::Model, model, ()); IMP::atom::Hierarchy mhd = IMP::atom::read_pdb( pdb, model, new IMP::atom::NonWaterNonHydrogenPDBSelector(), true, true); IMP::Particles residue_particles = diff --git a/modules/integrative_docking/bin/interface_cross_links.cpp b/modules/integrative_docking/bin/interface_cross_links.cpp index aafff8d234..50e520f936 100644 --- a/modules/integrative_docking/bin/interface_cross_links.cpp +++ b/modules/integrative_docking/bin/interface_cross_links.cpp @@ -100,7 +100,7 @@ int main(int argc, char** argv) { float distance_threshold = atof(files[2].c_str()); // read pdb files, prepare particles - IMP::Model* model = new IMP::Model(); + IMP_NEW(IMP::Model, model, ()); IMP::atom::Hierarchy mhd = IMP::atom::read_pdb( receptor_pdb, model, new IMP::atom::NonWaterNonHydrogenPDBSelector(), true, true); diff --git a/modules/integrative_docking/bin/interface_rtc.cpp b/modules/integrative_docking/bin/interface_rtc.cpp index 13e1bded1f..d34ecb0377 100644 --- a/modules/integrative_docking/bin/interface_rtc.cpp +++ b/modules/integrative_docking/bin/interface_rtc.cpp @@ -117,7 +117,7 @@ defined by two molecules.")( ligand_pdb = files[1]; // read pdb files, prepare particles - IMP::Model* model = new IMP::Model(); + IMP_NEW(IMP::Model, model, ()); IMP::atom::Hierarchy mhd = IMP::atom::read_pdb( receptor_pdb, model, new IMP::atom::NonWaterNonHydrogenPDBSelector(), true, true); diff --git a/modules/integrative_docking/bin/nmr_rtc_score.cpp b/modules/integrative_docking/bin/nmr_rtc_score.cpp index 86f4eb5d50..4db892bc61 100644 --- a/modules/integrative_docking/bin/nmr_rtc_score.cpp +++ b/modules/integrative_docking/bin/nmr_rtc_score.cpp @@ -78,7 +78,7 @@ put '-' if there is no file for one of the molecules.")( } // read pdb files, prepare particles - IMP::Model *model = new IMP::Model(); + IMP_NEW(IMP::Model, model, ()); IMP::atom::Hierarchy mhd = IMP::atom::read_pdb( receptor_pdb, model, new IMP::atom::NonWaterNonHydrogenPDBSelector(), true, true); diff --git a/modules/integrative_docking/bin/soap_score.cpp b/modules/integrative_docking/bin/soap_score.cpp index d2c9c6ace1..4ea462e523 100644 --- a/modules/integrative_docking/bin/soap_score.cpp +++ b/modules/integrative_docking/bin/soap_score.cpp @@ -135,7 +135,7 @@ each pair of PDB file names in the input file filenames.txt.") } // init model - IMP::Model* model = new IMP::Model(); + IMP_NEW(IMP::Model, model, ()); IMP::ParticleIndexes pis1, pis2; std::vector results; // scored complexes std::ofstream out_file(out_file_name.c_str()); // open output file diff --git a/modules/integrative_docking/include/internal/CrossLink.h b/modules/integrative_docking/include/internal/CrossLink.h index b590dd977e..53b0b762c6 100644 --- a/modules/integrative_docking/include/internal/CrossLink.h +++ b/modules/integrative_docking/include/internal/CrossLink.h @@ -24,11 +24,13 @@ class CrossLink { residue_number2_(0), chain_id2_("-"), min_distance_(0.0), - max_distance_(0.0) {} + max_distance_(0.0), + weight_(1.0) {} CrossLink(int residue_number1, std::string chain_id1, int residue_number2, std::string chain_id2, float min_dist, float max_dist, - float actual_distance = 0.0, float actual_cb_distance = 0.0) + float actual_distance = 0.0, float actual_cb_distance = 0.0, + float weight = 1.0) : residue_number1_(residue_number1), chain_id1_(chain_id1), residue_number2_(residue_number2), @@ -36,7 +38,8 @@ class CrossLink { min_distance_(min_dist), max_distance_(max_dist), actual_distance_(actual_distance), - actual_cb_distance_(actual_cb_distance) {} + actual_cb_distance_(actual_cb_distance), + weight_(weight) {} int get_residue1() const { return residue_number1_; } int get_residue2() const { return residue_number2_; } @@ -49,6 +52,8 @@ class CrossLink { float get_actual_distance() const { return actual_distance_; } float get_actual_cb_distance() const { return actual_cb_distance_; } + float get_weight() const { return weight_; } + friend std::ostream& operator<<(std::ostream& q, const CrossLink& cl); friend std::istream& operator>>(std::istream& s, CrossLink& cl); @@ -61,6 +66,7 @@ class CrossLink { float max_distance_; float actual_distance_; // from PDB - for testing purposes float actual_cb_distance_; // from PDB - for testing purposes + float weight_; }; IMPINTEGRATIVEDOCKINGEXPORT diff --git a/modules/integrative_docking/src/internal/CrossLink.cpp b/modules/integrative_docking/src/internal/CrossLink.cpp index ce42b7ab8b..8039e72632 100644 --- a/modules/integrative_docking/src/internal/CrossLink.cpp +++ b/modules/integrative_docking/src/internal/CrossLink.cpp @@ -19,7 +19,11 @@ int read_cross_link_file(const std::string& file_name, IMP::IOException); } CrossLink cl; - while (s >> cl) cross_links.push_back(cl); + while(s) { + if( s >> cl) cross_links.push_back(cl); + } + std::cout << cross_links.size() << " cross links were read from file " + << file_name << std::endl; return cross_links.size(); } @@ -48,7 +52,7 @@ std::ostream& operator<<(std::ostream& s, const CrossLink& cl) { s << "-"; else s << cl.chain_id2_; - s << " " << cl.min_distance_ << " " << cl.max_distance_; + s << " " << cl.min_distance_ << " " << cl.max_distance_ << " " << cl.weight_; return s; } @@ -57,6 +61,14 @@ std::istream& operator>>(std::istream& s, CrossLink& cl) { cl.chain_id2_ >> cl.min_distance_ >> cl.max_distance_; if (cl.chain_id1_ == "-") cl.chain_id1_ = " "; if (cl.chain_id2_ == "-") cl.chain_id2_ = " "; + + int c = s.peek(); + if(c == ' ') { // read weight + s >> cl.weight_; + } else { + cl.weight_ = 1.0; + } + return s; } diff --git a/modules/integrative_docking/test/test_cross_links.py b/modules/integrative_docking/test/test_cross_links.py index 452c1491de..d7ea39e327 100644 --- a/modules/integrative_docking/test/test_cross_links.py +++ b/modules/integrative_docking/test/test_cross_links.py @@ -19,9 +19,8 @@ def test_simple_interface(self): self.assertApplicationExitedCleanly(p.returncode, err) # count the number of lines in output file - fin = open('cross_links.dat', 'r') - text = fin.read() - fin.close() + with open('cross_links.dat', 'r') as fin: + text = fin.read() number_of_lines = text.count('\n') self.assertEqual(number_of_lines, 4) os.unlink('cross_links.dat') @@ -29,13 +28,13 @@ def test_simple_interface(self): os.unlink('cxms_all.dat') def test_simple_single_structure_score(self): - """Simple test of interface cross links single structure score application""" - destination = open('complex.pdb', 'w') - file1 = open(self.get_input_file_name('static.pdb'), 'r') - file2 = open(self.get_input_file_name('transformed.pdb'), 'r') - destination.write(file1.read()) - destination.write(file2.read()) - destination.close() + """Simple test of interface cross links single structure score + application""" + with open('complex.pdb', 'w') as destination: + with open(self.get_input_file_name('static.pdb'), 'r') as fh: + destination.write(fh.read()) + with open(self.get_input_file_name('transformed.pdb'), 'r') as fh: + destination.write(fh.read()) p = self.run_application('cross_links_single_score', ['complex.pdb', self.get_input_file_name('cxms.dat')]) @@ -62,9 +61,8 @@ def test_simple_score(self): self.assertApplicationExitedCleanly(p.returncode, err) # count the number of lines in output file - fin = open('cxms_score.res', 'r') - text = fin.read() - fin.close() + with open('cxms_score.res', 'r') as fin: + text = fin.read() number_of_lines = text.count('\n') self.assertEqual(number_of_lines, 15) os.unlink('cxms_score.res') diff --git a/modules/isd/include/CrossLinkMSRestraint.h b/modules/isd/include/CrossLinkMSRestraint.h index 25ebd0ab22..b6f4741394 100644 --- a/modules/isd/include/CrossLinkMSRestraint.h +++ b/modules/isd/include/CrossLinkMSRestraint.h @@ -57,6 +57,9 @@ class IMPISDEXPORT CrossLinkMSRestraint : public Restraint { double get_probability() const; + //! Get the particle indexes from a contribution + ParticleIndexPair get_contribution_particle_indexes(int i) const { return ppis_[i]; } + unsigned int get_number_of_contributions() const { return ppis_.size(); } virtual double unprotected_evaluate( diff --git a/modules/isd/include/GammaPrior.h b/modules/isd/include/GammaPrior.h new file mode 100644 index 0000000000..07b2e1c2d6 --- /dev/null +++ b/modules/isd/include/GammaPrior.h @@ -0,0 +1,48 @@ +/** + * \file IMP/isd/GammaPrior.h + * \brief A restraint on a scale parameter. + * + * Copyright 2007-2019 IMP Inventors. All rights reserved. + * + */ + +#ifndef IMPISD_GAMMA_PRIOR_H +#define IMPISD_GAMMA_PRIOR_H + +#include "isd_config.h" +#include +#include + +IMPISD_BEGIN_NAMESPACE + +//! Uniform distribution with harmonic boundaries + +class IMPISDEXPORT GammaPrior : public Restraint +{ + Pointer p_; + Float theta_; + Float k_; + +public: + //! Create the restraint. + GammaPrior(IMP::Model* m, Particle *p, Float k, + Float theta, std::string name="GammaPrior%1%"); + + virtual double + unprotected_evaluate(IMP::DerivativeAccumulator *accum) + const IMP_OVERRIDE; + virtual double evaluate_at(Float val) const; + virtual IMP::ModelObjectsTemp do_get_inputs() const IMP_OVERRIDE; + IMP_OBJECT_METHODS(GammaPrior); + + /* call for probability */ + virtual double get_probability() const + { + return exp(-unprotected_evaluate(nullptr)); + } + +}; + +IMPISD_END_NAMESPACE + +#endif /* IMPISD_GAMMA_PRIOR_H */ diff --git a/modules/isd/include/Weight.h b/modules/isd/include/Weight.h index 3e7c4b0579..d206f90ef1 100755 --- a/modules/isd/include/Weight.h +++ b/modules/isd/include/Weight.h @@ -1,6 +1,6 @@ /** * \file IMP/isd/Weight.h - * \brief Add weights for a set of states to a particle. + * \brief Add weights constrained to the unit simplex to a particle. * * Copyright 2007-2019 IMP Inventors. All rights reserved. * @@ -10,56 +10,162 @@ #define IMPISD_WEIGHT_H #include "isd_config.h" - +#include #include #include #include #include -#include -#include IMPISD_BEGIN_NAMESPACE -//! Add weights for a set of states to a particle. +static const int IMPISD_MAX_WEIGHTS = 1000; + +//! Add weights to a particle. +/** Weights are constrained to the unit simplex. + + \see algebra::UnitSimplexD + \ingroup decorators +*/ class IMPISDEXPORT Weight : public Decorator { - static const int nstates_max = 20; static void do_setup_particle(Model *m, ParticleIndex pi); + static void do_setup_particle(Model *m, ParticleIndex pi, Int nweights); + + static void do_setup_particle(Model *m, ParticleIndex pi, + const algebra::VectorKD& w); + + //! Add unit simplex constraint. + static void add_constraint(Model *m, ParticleIndex pi); + + static ObjectKey get_constraint_key(); + public: IMP_DECORATOR_METHODS(Weight, Decorator); + + //! Set up an empty Weight. + /** Weights must be added with add_weight() before use. */ IMP_DECORATOR_SETUP_0(Weight); - //! Get number of states key + //! Set up Weight with a fixed number of weights. + /** All weights are initialized with the same value. */ + IMP_DECORATOR_SETUP_1(Weight, Int, nweights); + + //! Set up Weight from the provided weight vector. + IMP_DECORATOR_SETUP_1(Weight, const algebra::VectorKD&, w); + + IMPISD_DEPRECATED_METHOD_DECL(2.12) static IntKey get_nstates_key(); - //! Get i-th weight key + //! Get number of weights key + static IntKey get_number_of_weights_key(); + + //! Get ith weight key static FloatKey get_weight_key(int i); - //! Set all the weights - void set_weights(algebra::VectorKD w); + //! Get all weight keys + FloatKeys get_weight_keys() const; + + //! Get the ith weight + Float get_weight(int i) const; + + //! Get all weights + algebra::VectorKD get_weights() const; - //! Add one weight - void add_weight(); + //! Set the ith weight lazily. + /** Delay enforcing the simplex constraint until Model::update(). */ + void set_weight_lazy(int i, Float wi); - //! Get the i-th weight - Float get_weight(int i); + //! Set all the weights + /** Delay enforcing the simplex constraint until Model::update(). */ + void set_weights_lazy(const algebra::VectorKD& w); + + //! Set all weights, enforcing the simplex constraint + /** \see algebra::get_projected */ + void set_weights(const algebra::VectorKD& w); - //! Get all the weights - algebra::VectorKD get_weights(); + //! Get weights are optimized + bool get_weights_are_optimized() const; //! Set weights are optimized void set_weights_are_optimized(bool tf); - //! Get number of states - Int get_number_of_states(); - - static bool get_is_setup(Model *m, ParticleIndex pi) { - return m->get_has_attribute(get_nstates_key(), pi); - } + //! Get derivative wrt ith weight. + Float get_weight_derivative(int i) const; + + //! Get derivatives wrt all weights. + algebra::VectorKD get_weights_derivatives() const; + + //! Add to derivative wrt ith weight. + void add_to_weight_derivative(int i, Float dwi, + const DerivativeAccumulator &da); + + //! Add to derivatives wrt all weights. + void add_to_weights_derivatives(const algebra::VectorKD& dw, + const DerivativeAccumulator &da); + + //! Extend the weight vector by one element lazily. + /** This should only be called during set-up and cannot be called + for an optimized Weight. + This version delays enforcing the simplex constraint until + Model::update(). + */ + void add_weight_lazy(Float wi = 0); + + //! Extend the weight vector by one element. + /** This should only be called during set-up and cannot be called + for an optimized Weight. + */ + void add_weight(Float wi = 0); + + IMPISD_DEPRECATED_METHOD_DECL(2.12) + Int get_number_of_states() const; + + //! Get number of weights. + Int get_number_of_weights() const; + + //! Get unit simplex on which weight vector lies. + algebra::UnitSimplexKD get_unit_simplex() const; + + //! Set number of weights lazily + /** This should only be called during set-up and cannot be called + for an optimized Weight. New weights are initialized to 0. + This version delays enforcing the simplex constraint until + Model::update(). + */ + void set_number_of_weights_lazy(Int nweights); + + //! Set number of weights. + /** This should only be called during set-up and cannot be called + for an optimized Weight. New weights are initialized to 0. + */ + void set_number_of_weights(Int nweights); + + static bool get_is_setup(Model *m, ParticleIndex pi); }; -IMP_VALUES(Weight, Weights); +IMP_DECORATORS(Weight, Weights, Decorators); + + +#if !defined(IMP_DOXYGEN) && !defined(SWIG) +class IMPISDEXPORT WeightSimplexConstraint : public IMP::Constraint { + private: + ParticleIndex pi_; + + private: + WeightSimplexConstraint(Particle *p) + : IMP::Constraint(p->get_model(), "WeightSimplexConstraint%1%") + , pi_(p->get_index()) {} + + public: + friend class Weight; + virtual void do_update_attributes() IMP_OVERRIDE; + virtual void do_update_derivatives(DerivativeAccumulator *da) IMP_OVERRIDE; + virtual ModelObjectsTemp do_get_inputs() const IMP_OVERRIDE; + virtual ModelObjectsTemp do_get_outputs() const IMP_OVERRIDE; + IMP_OBJECT_METHODS(WeightSimplexConstraint); +}; +#endif IMPISD_END_NAMESPACE diff --git a/modules/isd/pyext/swig.i-in b/modules/isd/pyext/swig.i-in index df8bc99a1f..a128105e1a 100644 --- a/modules/isd/pyext/swig.i-in +++ b/modules/isd/pyext/swig.i-in @@ -57,6 +57,7 @@ IMP_SWIG_OBJECT(IMP::isd, WeightRestraint, WeightRestraints); IMP_SWIG_OBJECT(IMP::isd, AtomicCrossLinkMSRestraint, AtomicCrossLinkMSRestraints); IMP_SWIG_OBJECT(IMP::isd, GaussianEMRestraint, GaussianEMRestraints); IMP_SWIG_OBJECT(IMP::isd, GaussianAnchorEMRestraint, GaussianAnchorEMRestraints); +IMP_SWIG_OBJECT(IMP::isd, GammaPrior, GammaPriors); /* One can add python methods to your module by putting code in %pythoncode blocks This function can be called as IMP.isds.say_hello(). */ @@ -122,4 +123,5 @@ def create_model_and_particles(): %include "IMP/isd/AtomicCrossLinkMSRestraint.h" %include "IMP/isd/GaussianEMRestraint.h" %include "IMP/isd/GaussianAnchorEMRestraint.h" +%include "IMP/isd/GammaPrior.h" %include "IMP/isd/em_utilities.h" diff --git a/modules/isd/src/CysteineCrossLinkRestraint.cpp b/modules/isd/src/CysteineCrossLinkRestraint.cpp index 5a3069cd11..4f0751c130 100644 --- a/modules/isd/src/CysteineCrossLinkRestraint.cpp +++ b/modules/isd/src/CysteineCrossLinkRestraint.cpp @@ -66,9 +66,11 @@ void CysteineCrossLinkRestraint::add_contribution(ParticleIndexAdaptor p1, Model *m = get_model(); ps1_.push_back(p1); ps2_.push_back(p2); - if (Weight(m, weight_).get_number_of_states() < + Weight w(m, weight_); + if (w.get_number_of_weights() < static_cast(get_number_of_contributions())) { - Weight(m, weight_).add_weight(); + w.add_weight(); + w.set_weights_lazy(w.get_unit_simplex().get_barycenter()); } } @@ -85,9 +87,11 @@ void CysteineCrossLinkRestraint::add_contribution(ParticleIndexes p1, pslist1_.push_back(p1); pslist2_.push_back(p2); Model *m = get_model(); - if (Weight(m, weight_).get_number_of_states() < + Weight w(m, weight_); + if (w.get_number_of_weights() < static_cast(get_number_of_contributions())) { - Weight(m, weight_).add_weight(); + w.add_weight(); + w.set_weights_lazy(w.get_unit_simplex().get_barycenter()); } } @@ -187,9 +191,14 @@ Floats CysteineCrossLinkRestraint::get_frequencies() const { Floats frequencies; + Weight w(m, weight_); + IMP_INTERNAL_CHECK( + get_number_of_contributions() == w.get_number_of_weights(), + "Number of contributions does not equal weights dimension." + ); for (unsigned i = 0; i < get_number_of_contributions(); ++i) { - double ww = Weight(m, weight_).get_weight(i); + double ww = w.get_weight(i); double fi = (1.0 - pow(epsilon, nus[i] / numax)) * ww; frequencies.push_back(fi); diff --git a/modules/isd/src/GammaPrior.cpp b/modules/isd/src/GammaPrior.cpp new file mode 100644 index 0000000000..be604a292b --- /dev/null +++ b/modules/isd/src/GammaPrior.cpp @@ -0,0 +1,57 @@ +/** + * \file IMP/isd/GammaPrior.cpp + * \brief Restrain a scale particle with a gamma distribution + * + * Copyright 2007-2019 IMP Inventors. All rights reserved. + * + */ + +#include +#include +#include +#include + +IMPISD_BEGIN_NAMESPACE + +GammaPrior::GammaPrior(IMP::Model *m, Particle *p, Float k, + Float theta, std::string name): + Restraint(m, name), p_(p), theta_(theta), k_(k) {} + + +/* Apply the score if it's a scale decorator. + */ +double +GammaPrior::unprotected_evaluate(DerivativeAccumulator *accum) const +{ + IMP::isd::Scale sig(p_); + double prob=0.0; + double s=sig.get_scale()/10; + double gam = boost::math::tgamma(k_) * std::pow(theta_, k_); + prob = std::pow(s, k_-1) * std::exp(-s/theta_)/gam; + + if (accum) { + } + + double score = -1*log (prob); + return score; +} + +double +GammaPrior::evaluate_at(Float val) const +{ + double prob=0.0; + double gam = boost::math::tgamma(k_) * std::pow(theta_, k_); + prob = std::pow(val, k_-1) * std::exp(-val/theta_)/gam; + + double score = -1*log (prob); + return score; +} + +/* Return all particles whose attributes are read by the restraints. To + do this, ask the pair score what particles it uses.*/ +ModelObjectsTemp GammaPrior::do_get_inputs() const +{ + return ParticlesTemp(1,p_); +} + +IMPISD_END_NAMESPACE diff --git a/modules/isd/src/Weight.cpp b/modules/isd/src/Weight.cpp index 60143ffa30..b269563911 100644 --- a/modules/isd/src/Weight.cpp +++ b/modules/isd/src/Weight.cpp @@ -1,31 +1,71 @@ /** * \file isd/Weight.cpp - * \brief Add a name to a particle. + * \brief Add weights constrained to the unit simplex to a particle. * * Copyright 2007-2019 IMP Inventors. All rights reserved. * */ #include +#include +#include +#include IMPISD_BEGIN_NAMESPACE void Weight::do_setup_particle(Model *m, ParticleIndex pi) { - m->add_attribute(get_nstates_key(), pi, 0); - for (int i = 0; i < nstates_max; ++i) { - m->add_attribute(get_weight_key(i), pi, 0.0); - } + m->add_attribute(get_number_of_weights_key(), pi, 0); + add_constraint(m, pi); +} + +void Weight::do_setup_particle(Model *m, ParticleIndex pi, Int nweights) { + Weight pw = setup_particle(m, pi); + pw.set_number_of_weights_lazy(nweights); + pw.set_weights_lazy(pw.get_unit_simplex().get_barycenter()); +} + +void Weight::do_setup_particle(Model *m, ParticleIndex pi, + const algebra::VectorKD &w) { + Weight pw = setup_particle(m, pi); + pw.set_number_of_weights_lazy(w.get_dimension()); + pw.set_weights(w); +} + +void Weight::add_constraint(Model *m, ParticleIndex pi) { + ObjectKey k(get_constraint_key()); + Pointer c(new WeightSimplexConstraint( + m->get_particle(pi))); + c->set_was_used(true); + m->get_particle(pi)->add_attribute(k, c); + m->add_score_state(c); +} + +bool Weight::get_is_setup(Model *m, ParticleIndex pi) { + if (!m->get_has_attribute(get_number_of_weights_key(), pi)) return false; + if (!m->get_has_attribute(get_constraint_key(), pi)) return false; + Int nweights = m->get_attribute(get_number_of_weights_key(), pi); + for (int i = 0; i < nweights; ++i) + if (!m->get_has_attribute(get_weight_key(i), pi)) return false; + return true; } IntKey Weight::get_nstates_key() { - static IntKey k("nstates"); + IMPISD_DEPRECATED_METHOD_DEF( + 2.12, + "Use get_number_of_weights_key() instead." + ); + return get_number_of_weights_key(); +} + +IntKey Weight::get_number_of_weights_key() { + static IntKey k("nweights"); return k; } FloatKey Weight::get_weight_key(int j) { static FloatKeys kk; if (kk.empty()) { - for (int i = 0; i < nstates_max; ++i) { + for (int i = 0; i < IMPISD_MAX_WEIGHTS; ++i) { std::stringstream out; out << i; kk.push_back(FloatKey("weight" + out.str())); @@ -34,53 +74,172 @@ FloatKey Weight::get_weight_key(int j) { return kk[j]; } -//! Set all the weights -void Weight::set_weights(algebra::VectorKD w) { - IMP_USAGE_CHECK(static_cast(w.get_dimension()) == get_number_of_states(), - "Out of range"); - for (int i = 0; i < get_number_of_states(); ++i) { - get_particle()->set_value(get_weight_key(i), w[i]); - } +FloatKeys Weight::get_weight_keys() const { + FloatKeys fks; + for (int i = 0; i < get_number_of_weights(); ++i) + fks.push_back(get_weight_key(i)); + return fks; } -//! Add one weight -void Weight::add_weight() { - int i = get_particle()->get_value(get_nstates_key()); - IMP_USAGE_CHECK(i < nstates_max, "Out of range"); - get_particle()->set_value(get_nstates_key(), i + 1); - Float w = 1.0 / static_cast(get_number_of_states()); - for (int i = 0; i < get_number_of_states(); ++i) { - get_particle()->set_value(get_weight_key(i), w); - } +ObjectKey Weight::get_constraint_key() { + static ObjectKey k("weight_const"); + return k; } -//! Get the i-th weight -Float Weight::get_weight(int i) { - IMP_USAGE_CHECK(i < get_number_of_states(), "Out of range"); +Float Weight::get_weight(int i) const { + IMP_USAGE_CHECK(i < get_number_of_weights(), "Out of range"); return get_particle()->get_value(get_weight_key(i)); } -//! Get all weights -algebra::VectorKD Weight::get_weights() { - algebra::VectorKD ww = algebra::get_zero_vector_kd(get_number_of_states()); - for (int i = 0; i < get_number_of_states(); ++i) { - ww[i] = get_weight(i); +algebra::VectorKD Weight::get_weights() const { + Int nweights = get_number_of_weights(); + algebra::VectorKD w = algebra::get_zero_vector_kd(nweights); + for (int i = 0; i < nweights; ++i) + w[i] = get_particle()->get_value(get_weight_key(i)); + return w; +} + +void Weight::set_weight_lazy(int i, Float wi) { + IMP_USAGE_CHECK(i < get_number_of_weights(), + "Out of range"); + get_particle()->set_value(get_weight_key(i), wi); +} + +void Weight::set_weights_lazy(const algebra::VectorKD& w) { + Int nweights = w.get_dimension(); + IMP_USAGE_CHECK(nweights == get_number_of_weights(), + "Out of range"); + for (int i = 0; i < nweights; ++i) + get_particle()->set_value(get_weight_key(i), w[i]); +} + +void Weight::set_weights(const algebra::VectorKD& w) { + set_weights_lazy(algebra::get_projected(get_unit_simplex(), w)); +} + +bool Weight::get_weights_are_optimized() const { + Int nweights = get_number_of_weights(); + if (nweights == 0) return false; + for (int i = 0; i < get_number_of_weights(); ++i){ + if (!get_particle()->get_is_optimized(get_weight_key(i))) + return false; } - return ww; + return true; } -//! Set weights are optimized void Weight::set_weights_are_optimized(bool tf) { - for (int i = 0; i < nstates_max; ++i) { + for (int i = 0; i < get_number_of_weights(); ++i) get_particle()->set_is_optimized(get_weight_key(i), tf); +} + +Float Weight::get_weight_derivative(int i) const { + int nweights = get_number_of_weights(); + IMP_USAGE_CHECK(i < nweights, "Out of bounds."); + return get_particle()->get_derivative(get_weight_key(i)); +} + +algebra::VectorKD Weight::get_weights_derivatives() const { + int nweights = get_number_of_weights(); + algebra::VectorKD dw = algebra::get_zero_vector_kd(nweights); + for (int i = 0; i < nweights; ++i) + dw[i] = get_particle()->get_derivative(get_weight_key(i)); + return dw; +} + +void Weight::add_to_weight_derivative(int i, Float dwi, + const DerivativeAccumulator &da) { + int nweights = get_number_of_weights(); + IMP_USAGE_CHECK(i < nweights, "Out of bounds."); + get_particle()->add_to_derivative(get_weight_key(i), dwi, da); +} + +void Weight::add_to_weights_derivatives(const algebra::VectorKD& dw, + const DerivativeAccumulator &da) { + int nweights = dw.get_dimension(); + IMP_USAGE_CHECK(nweights == get_number_of_weights(), "Out of range"); + for (int i = 0; i < nweights; ++i) + get_particle()->add_to_derivative(get_weight_key(i), dw[i], da); +} + +void Weight::set_number_of_weights_lazy(Int nweights) { + IMP_USAGE_CHECK(nweights > 0, "Number of weights must be greater than zero."); + IMP_USAGE_CHECK(nweights <= IMPISD_MAX_WEIGHTS, + "Number of weights exceeds the maximum allowed number of " + << IMPISD_MAX_WEIGHTS << "."); + + Int old_nweights = get_number_of_weights(); + if (nweights > old_nweights) { + IMP_USAGE_CHECK(!get_weights_are_optimized(), + "Number of weights cannot be changed for optimized Weight"); + for (int i = old_nweights; i < nweights; ++i) { + if (!get_model()->get_has_attribute(get_weight_key(i), + get_particle_index())) { + get_model()->add_attribute(get_weight_key(i), + get_particle_index(), 0); + } else { + get_particle()->set_value(get_weight_key(i), 0); + } + } + get_particle()->set_value(get_number_of_weights_key(), nweights); + } else if (nweights < old_nweights) { + IMP_USAGE_CHECK(!get_weights_are_optimized(), + "Number of weights cannot be changed for optimized Weight"); + get_particle()->set_value(get_number_of_weights_key(), nweights); } } -//! Get number of states -Int Weight::get_number_of_states() { - return get_particle()->get_value(get_nstates_key()); +void Weight::set_number_of_weights(Int nweights) { + set_number_of_weights_lazy(nweights); + set_weights(get_weights()); +} + +void Weight::add_weight_lazy(Float wi) { + Int old_nweights = get_number_of_weights(); + set_number_of_weights_lazy(old_nweights + 1); + get_particle()->set_value(get_weight_key(old_nweights), wi); +} + +void Weight::add_weight(Float wi) { + add_weight_lazy(wi); + set_weights(get_weights()); +} + +Int Weight::get_number_of_states() const { + IMPISD_DEPRECATED_METHOD_DEF( + 2.12, + "Use get_number_of_weights() instead." + ); + return get_number_of_weights(); } -void Weight::show(std::ostream &out) const { out << "Weight "; } +Int Weight::get_number_of_weights() const { + return get_particle()->get_value(get_number_of_weights_key()); +} + +algebra::UnitSimplexKD Weight::get_unit_simplex() const { + return algebra::UnitSimplexKD(get_number_of_weights()); +} + +void Weight::show(std::ostream &out) const { + out << "Weight: " << get_weights(); +} + + + +void WeightSimplexConstraint::do_update_attributes() { + Weight w(get_model(), pi_); + w.set_weights(w.get_weights()); +} + +void WeightSimplexConstraint::do_update_derivatives( + DerivativeAccumulator *) {} + +ModelObjectsTemp WeightSimplexConstraint::do_get_inputs() const { + return ModelObjectsTemp(1, get_model()->get_particle(pi_)); +} + +ModelObjectsTemp WeightSimplexConstraint::do_get_outputs() const { + return ModelObjectsTemp(1, get_model()->get_particle(pi_)); +} IMPISD_END_NAMESPACE diff --git a/modules/isd/src/WeightRestraint.cpp b/modules/isd/src/WeightRestraint.cpp index c8c6362620..d2b9ce46b2 100644 --- a/modules/isd/src/WeightRestraint.cpp +++ b/modules/isd/src/WeightRestraint.cpp @@ -29,18 +29,34 @@ WeightRestraint::WeightRestraint(Particle *w, Float wmin, Float wmax, double WeightRestraint::unprotected_evaluate(DerivativeAccumulator *accum) const { // retrieve weights - algebra::VectorKD weight = Weight(w_).get_weights(); + Weight w(w_); + algebra::VectorKD weight = w.get_weights(); + Int dim = w.get_number_of_weights(); Float dw = 0.; - for (unsigned i = 0; i < weight.get_dimension(); ++i) { - if (weight[i] > wmax_) - dw += (weight[i] - wmax_) * (weight[i] - wmax_); - else if (weight[i] < wmin_) - dw += (wmin_ - weight[i]) * (wmin_ - weight[i]); - } - if (accum) { + Float deltaw; + algebra::VectorKD wderiv = algebra::get_zero_vector_kd(dim); + for (int i = 0; i < dim; ++i) { + if (weight[i] > wmax_) { + deltaw = weight[i] - wmax_; + dw += deltaw * deltaw; + wderiv[i] = kappa_ * deltaw; + } else if (weight[i] < wmin_) { + deltaw = wmin_ - weight[i]; + dw += deltaw * deltaw; + wderiv[i] = -kappa_ * deltaw; + } + } + w.add_to_weights_derivatives(wderiv, *accum); + } else { + for (int i = 0; i < dim; ++i) { + if (weight[i] > wmax_) + dw += (weight[i] - wmax_) * (weight[i] - wmax_); + else if (weight[i] < wmin_) + dw += (wmin_ - weight[i]) * (wmin_ - weight[i]); + } } return 0.5 * kappa_ * dw; diff --git a/modules/isd/test/expensive_test_CysteineCrossLinkRestraint.py b/modules/isd/test/expensive_test_CysteineCrossLinkRestraint.py index c441e95459..0c968b7ee1 100644 --- a/modules/isd/test/expensive_test_CysteineCrossLinkRestraint.py +++ b/modules/isd/test/expensive_test_CysteineCrossLinkRestraint.py @@ -63,9 +63,9 @@ def setup_nuisance( nuisance.set_is_optimized(nuisance.get_nuisance_key(), isoptimized) return nuisance - def setup_weight(self, m, isoptimized=True): + def setup_weight(self, m, dim, isoptimized=True): pw = IMP.Particle(m) - weight = IMP.isd.Weight.setup_particle(pw) + weight = IMP.isd.Weight.setup_particle(pw, dim) weight.set_weights_are_optimized(True) return weight @@ -135,7 +135,7 @@ def testSetupAndEvaluate(self): sigmatuple[1], True) # population particle - weight = self.setup_weight(m, True) + weight = self.setup_weight(m, 2, True) # epsilon epsilon = self.setup_nuisance( m, diff --git a/modules/isd/test/test_GammaPrior.py b/modules/isd/test/test_GammaPrior.py new file mode 100644 index 0000000000..9eb38b94e0 --- /dev/null +++ b/modules/isd/test/test_GammaPrior.py @@ -0,0 +1,55 @@ +#!/usr/bin/env python +import numpy +import random +import IMP +from IMP.isd import GammaPrior +from IMP.isd import Scale +import IMP.test + + +class Tests(IMP.test.TestCase): + + def initialize(self): + m = IMP.Model() + p = IMP.Particle(m) + + return m, p + + def test_evaluate_at(self): + try: + from scipy.stats import gamma + except ImportError: + self.skipTest("this test requires the scipy Python module") + theta = 2.0 + kappa = 1.4 + + m, p = self.initialize() + + gp = GammaPrior(m, p, kappa, theta) + + randnum = random.uniform(0.0,10.0) + + self.assertAlmostEqual(gp.evaluate_at(randnum), -1*numpy.log(gamma.pdf(randnum, kappa, 0.0, theta))) + + def test_unprotected_evaluate(self): + try: + from scipy.stats import gamma + except ImportError: + self.skipTest("this test requires the scipy Python module") + theta = 2.0 + kappa = 1.6 + + sigma_value = 15.0 + + m,p = self.initialize() + sigma = Scale.setup_particle(p, sigma_value) + + DA = IMP.DerivativeAccumulator() + + gp = GammaPrior(m, p, kappa, theta) + + self.assertAlmostEqual(gp.unprotected_evaluate(DA), -1*numpy.log(gamma.pdf(sigma_value/10, kappa, 0.0, theta))) + + +if __name__ == '__main__': + IMP.test.main() diff --git a/modules/isd/test/test_Weight.py b/modules/isd/test/test_Weight.py index 08e839b12f..fbeabd1e4e 100644 --- a/modules/isd/test/test_Weight.py +++ b/modules/isd/test/test_Weight.py @@ -10,6 +10,8 @@ # unit testing framework import IMP.test +import numpy as np + class TestWeightParam(IMP.test.TestCase): @@ -20,67 +22,216 @@ def setUp(self): # IMP.set_log_level(IMP.MEMORY) IMP.set_log_level(0) self.m = IMP.Model() - self.w = Weight.setup_particle(IMP.Particle(self.m)) - def test_Setup_iw(self): - "Test weight_initial_values" + def test_setup_empty_add_weight(self): + "Test setup weight as empty and add weights" w = Weight.setup_particle(IMP.Particle(self.m)) - w.add_weight() - for n in range(19): - for k in range(n + 1): - self.assertEqual(w.get_weight(k), 1.0 / (n + 1)) + for n in range(1, 20): w.add_weight() + self.assertEqual(w.get_number_of_weights(), n) + self.assertSequenceAlmostEqual( + list(w.get_weights()), [1] + [0] * (n - 1) + ) - def test_Setup_number(self): - "Test weights_initial_values" - w = Weight.setup_particle(IMP.Particle(self.m)) - w.add_weight() - for n in range(19): - ws = w.get_weights() - self.assertEqual(len(ws), n + 1) - for k in range(n + 1): - self.assertEqual(ws[k], 1.0 / (n + 1)) - w.add_weight() + def test_setup_number_of_weights(self): + "Test setup weight with number of weights" + for n in range(1, 20): + p = IMP.Particle(self.m) + w = Weight.setup_particle(p, n) + self.assertTrue(Weight.get_is_setup(p)) + self.assertEqual(w.get_number_of_weights(), n) + for k in range(n): + self.assertAlmostEqual(w.get_weight(k), 1.0 / n, delta=1e-6) - def test_Setup_size(self): - "Test weights_size" - w = Weight.setup_particle(IMP.Particle(self.m)) - w.add_weight() - for n in range(19): - ws = w.get_weights() - ns = w.get_number_of_states() - self.assertEqual(n + 1, ns) - self.assertEqual(len(ws), ns) - w.add_weight() + def test_setup_weights(self): + "Test setup weight with initial values" + for n in range(1, 20): + p = IMP.Particle(self.m) + ws = np.random.uniform(size=n) + w = Weight.setup_particle(p, ws) + self.assertTrue(Weight.get_is_setup(p)) + self.assertEqual(w.get_number_of_weights(), n) + self.assertSequenceAlmostEqual( + w.get_weights(), + IMP.algebra.get_projected( + w.get_unit_simplex(), IMP.algebra.VectorKD(ws) + ), + ) - def test_assign_values(self): - "Test weights_set" - w = Weight.setup_particle(IMP.Particle(self.m)) - w.add_weight() - ws = [1.0] - for n in range(19): + def test_set_weights(self): + for n in range(1, 20): + p = IMP.Particle(self.m) + w = Weight.setup_particle(p, n) + ws = np.random.uniform(size=n) + w.set_weights(ws) + self.assertSequenceAlmostEqual( + w.get_weights(), + IMP.algebra.get_projected( + w.get_unit_simplex(), IMP.algebra.VectorKD(ws) + ), + ) + + def test_set_weights_lazy(self): + for n in range(1, 20): + p = IMP.Particle(self.m) + w = Weight.setup_particle(p, n) + ws = np.random.uniform(size=n) + w.set_weights_lazy(ws) + for k in range(n): + self.assertAlmostEqual(w.get_weight(k), ws[k], delta=1e-6) + + self.m.update() + self.assertSequenceAlmostEqual( + w.get_weights(), + IMP.algebra.get_projected( + w.get_unit_simplex(), IMP.algebra.VectorKD(ws) + ), + ) + + def test_set_weight_lazy(self): + for n in range(1, 20): + p = IMP.Particle(self.m) + w = Weight.setup_particle(p, n) + ws = np.random.uniform(size=n) + for k in range(n): + w.set_weight_lazy(k, ws[k]) + self.assertAlmostEqual(w.get_weight(k), ws[k], delta=1e-6) + + self.m.update() + self.assertSequenceAlmostEqual( + w.get_weights(), + IMP.algebra.get_projected( + w.get_unit_simplex(), IMP.algebra.VectorKD(ws) + ), + ) + + def test_set_weights_zero(self): + p = IMP.Particle(self.m) + n = 5 + w = Weight.setup_particle(p, n) + ws = np.zeros(n) + w.set_weights(ws) + for k in range(n): + self.assertAlmostEqual(w.get_weight(k), 1.0 / n, delta=1e-6) + + def test_add_to_weight_derivative(self): + for n in range(1, 20): + w = Weight.setup_particle(IMP.Particle(self.m), n) + ws = np.random.uniform(size=n) + ws /= np.sum(ws) + w.set_weights(ws) + + for k in range(0, n): + dwk = np.random.normal() + w.add_to_weight_derivative(k, dwk, IMP.DerivativeAccumulator()) + self.assertAlmostEqual( + w.get_weight_derivative(k), dwk, delta=1e-6 + ) + + def test_add_to_weights_derivatives(self): + for n in range(1, 20): + w = Weight.setup_particle(IMP.Particle(self.m), n) + ws = np.random.uniform(size=n) + ws /= np.sum(ws) w.set_weights(ws) - ws2 = w.get_weights() - for k in range(n + 1): - self.assertEqual(ws[k], ws2[k]) - ws.append(1.0) - w.add_weight() + + dw = np.random.normal(size=n) + w.add_to_weights_derivatives(dw, IMP.DerivativeAccumulator()) + dw2 = w.get_weights_derivatives() + dw2 = [dw2[i] for i in range(n)] + self.assertSequenceAlmostEqual(list(dw), dw2, delta=1e-6) + + def test_add_weight(self): + "Test add_weight" + w = Weight.setup_particle(IMP.Particle(self.m), 1) + for n in range(2, 20): + ws = list(w.get_weights()) + wi = np.random.uniform() + w.add_weight(wi) + ws.append(wi) + self.assertEqual(w.get_number_of_weights(), n) + ws = list(IMP.algebra.get_projected(w.get_unit_simplex(), ws)) + self.assertSequenceAlmostEqual(list(w.get_weights()), ws) def test_set_optimized(self): "Test weights_optimized" - w = Weight.setup_particle(IMP.Particle(self.m)) - w.add_weight() - for n in range(19): + for n in range(1, 20): + w = Weight.setup_particle(IMP.Particle(self.m), n) + self.assertFalse(w.get_weights_are_optimized()) w.set_weights_are_optimized(True) - for k in range(n + 1): + for k in range(n): b = w.get_is_optimized(w.get_weight_key(k)) self.assertEqual(b, True) + self.assertTrue(w.get_weights_are_optimized()) w.set_weights_are_optimized(False) - for k in range(n + 1): + for k in range(n): b = w.get_is_optimized(w.get_weight_key(k)) self.assertEqual(b, False) - w.add_weight() + self.assertFalse(w.get_weights_are_optimized()) + + def test_change_nweights_for_optimized_raises_error(self): + for n in range(1, 20): + w = Weight.setup_particle(IMP.Particle(self.m), n) + self.assertFalse(w.get_weights_are_optimized()) + w.set_number_of_weights(n + 1) + w.set_number_of_weights(n) + w.set_weights_are_optimized(True) + self.assertRaisesUsageException(w.set_number_of_weights, n + 1) + + def test_set_nweights_lazy(self): + for n in range(1, 20): + w = Weight.setup_particle(IMP.Particle(self.m), n) + ws = IMP.algebra.get_random_vector_on(w.get_unit_simplex()) + w.set_weights(ws) + + w.set_number_of_weights_lazy(n + 1) + self.assertEqual(w.get_number_of_weights(), n + 1) + self.assertSequenceAlmostEqual( + list(w.get_weights()), list(ws) + [0] + ) + + w.set_number_of_weights_lazy(n) + self.assertEqual(w.get_number_of_weights(), n) + self.assertSequenceAlmostEqual(list(w.get_weights()), list(ws)) + + def test_set_nweights(self): + for n in range(2, 20): + w = Weight.setup_particle(IMP.Particle(self.m), n) + ws = IMP.algebra.get_random_vector_on(w.get_unit_simplex()) + w.set_weights(ws) + + w.set_number_of_weights(n + 1) + self.assertEqual(w.get_number_of_weights(), n + 1) + self.assertSequenceAlmostEqual( + list(w.get_weights()), list(ws) + [0] + ) + + w.set_number_of_weights(n - 1) + self.assertEqual(w.get_number_of_weights(), n - 1) + self.assertSequenceAlmostEqual( + list(w.get_weights()), + list( + IMP.algebra.get_projected( + w.get_unit_simplex(), list(ws)[:-1] + ) + ), + ) + + def test_get_weight_keys(self): + for n in range(1, 20): + w = Weight.setup_particle(IMP.Particle(self.m), n) + wks = [w.get_weight_key(i) for i in range(n)] + self.assertSequenceEqual(wks, list(w.get_weight_keys())) + + def test_get_unit_simplex(self): + for n in range(1, 20): + p = IMP.Particle(self.m) + ws = np.random.uniform(size=n) + w = Weight.setup_particle(p, ws) + s = w.get_unit_simplex() + self.assertIsInstance(s, IMP.algebra.UnitSimplexKD) + self.assertEqual(s.get_dimension(), n) -if __name__ == '__main__': +if __name__ == "__main__": IMP.test.main() diff --git a/modules/isd/test/test_mc_WeightMover.py b/modules/isd/test/test_mc_WeightMover.py index fef2f61c45..b29003684a 100644 --- a/modules/isd/test/test_mc_WeightMover.py +++ b/modules/isd/test/test_mc_WeightMover.py @@ -20,11 +20,12 @@ def setUp(self): IMP.test.TestCase.setUp(self) # IMP.set_log_level(IMP.MEMORY) IMP.set_log_level(0) + self.setup_system(2) + + def setup_system(self, nweights): self.m = IMP.Model() - self.w = Weight.setup_particle(IMP.Particle(self.m)) + self.w = Weight.setup_particle(IMP.Particle(self.m), nweights) self.w.set_weights_are_optimized(True) - self.w.add_weight() - self.w.add_weight() self.wm = WeightMover(self.w, 0.1) self.mc = IMP.core.MonteCarlo(self.m) self.mc.set_scoring_function([]) @@ -34,16 +35,13 @@ def setUp(self): def test_run(self): "Test weight mover mc run" - self.setUp() for n in range(5): + self.setup_system(n + 2) for j in range(10): self.mc.optimize(10) ws = self.w.get_weights() - sum = 0 - for k in range(self.w.get_number_of_states()): - sum += self.w.get_weight(k) - self.assertAlmostEqual(sum, 1.0, delta=0.0000001) - self.w.add_weight() + wsum = sum([ws[i] for i in range(len(ws))]) + self.assertAlmostEqual(wsum, 1.0, delta=0.0000001) if __name__ == '__main__': diff --git a/modules/kernel/Setup.cmake b/modules/kernel/Setup.cmake index 3ddfe62d01..65e9bb64ce 100644 --- a/modules/kernel/Setup.cmake +++ b/modules/kernel/Setup.cmake @@ -10,7 +10,8 @@ set(IMP_kernel_CONFIG IMP_BUILD=IMP_${build}:IMP_HAS_LOG=IMP_${IMP_MAX_LOG}:IMP_ imp_execute_process("generate paths.cpp" ${CMAKE_BINARY_DIR} - COMMAND ${CMAKE_SOURCE_DIR}/tools/build/setup_paths.py + COMMAND ${PYTHON_EXECUTABLE} + ${CMAKE_SOURCE_DIR}/tools/build/setup_paths.py "--datapath=${CMAKE_INSTALL_FULL_DATADIR}/IMP" "--examplepath=${CMAKE_INSTALL_FULL_DOCDIR}/examples" "--output=src/kernel/paths.cpp") diff --git a/modules/kernel/include/internal/swig_helpers_base.h b/modules/kernel/include/internal/swig_helpers_base.h index 3db05c1e2e..7df53e3214 100644 --- a/modules/kernel/include/internal/swig_helpers_base.h +++ b/modules/kernel/include/internal/swig_helpers_base.h @@ -558,7 +558,7 @@ struct ConvertFloatBase { return PyNumber_Check(o); } template - static PyObject* create_python_object(float f, SwigData, int) { + static PyObject* create_python_object(double f, SwigData, int) { // these may or may not have a refcount return PyFloat_FromDouble(f); } diff --git a/modules/kernel/pyext/include/IMP_kernel.exceptions.i b/modules/kernel/pyext/include/IMP_kernel.exceptions.i index a076cfff33..c7fca6360d 100644 --- a/modules/kernel/pyext/include/IMP_kernel.exceptions.i +++ b/modules/kernel/pyext/include/IMP_kernel.exceptions.i @@ -179,6 +179,17 @@ static PyObject *imp_exception, *imp_internal_exception, *imp_model_exception, } SWIG_fail; } +// If we're doing cleanup as a result of a previous Python exception +// (e.g. StopIteration), don't return a value (otherwise we'll get a +// SystemError "returned a result with an error set"). +// SWIG doesn't appear to allow us to do this only for delete_* wrappers, +// and the C preprocessor isn't up to the job, so #ifdelete is mapped to +// #if 0 or #if 1 by tools/build/make_swig_wrapper.py +%#ifdelete $symname + if (PyErr_Occurred()) { + SWIG_fail; + } +%#endif } // If Python exceptions are raised in a director method, temporarily reraise diff --git a/modules/kernel/test/test_refcount.py b/modules/kernel/test/test_refcount.py index 0ac12e2f1a..39728cc069 100644 --- a/modules/kernel/test/test_refcount.py +++ b/modules/kernel/test/test_refcount.py @@ -60,7 +60,6 @@ def test_delete_model_accessor_restraint(self): print("adding") rs.add_restraint(r) print(r) - print(r.__del__) # Now create new Python particle p from a C++ vector accessor # (front(), back(), [], etc.) # (not the Python IMP.Particle() constructor) @@ -90,7 +89,6 @@ def test_delete_model_accessor_restraint(self): print("deleting p") #del p # p.__del__ - print(p.__del__) del p # Python reference m, r, rp print("checking p") diff --git a/modules/mmcif/test/test_dumper.py b/modules/mmcif/test/test_dumper.py index 72b154f2ae..1c08c7727e 100644 --- a/modules/mmcif/test/test_dumper.py +++ b/modules/mmcif/test/test_dumper.py @@ -202,6 +202,21 @@ def test_assembly_subset_modeled(self): system.system._make_complete_assembly() self._assign_entity_ids(system) self._assign_asym_ids(system) + d = ihm.dumper._EntityDumper() + out = _get_dumper_output(d, system.system) + self.assertEqual(out, """# +loop_ +_entity.id +_entity.type +_entity.src_method +_entity.pdbx_description +_entity.formula_weight +_entity.pdbx_number_of_molecules +_entity.details +1 polymer man foo 267.282 1 . +2 polymer man bar 178.188 1 . +# +""") d = ihm.dumper._EntityPolySegmentDumper() d.finalize(system.system) out = _get_dumper_output(d, system.system) @@ -214,7 +229,6 @@ def test_assembly_subset_modeled(self): _ihm_entity_poly_segment.comp_id_begin _ihm_entity_poly_segment.comp_id_end 1 1 1 3 ALA ALA -2 2 1 2 ALA ALA # """) d = ihm.dumper._AssemblyDumper() @@ -225,8 +239,7 @@ def test_assembly_subset_modeled(self): _ihm_struct_assembly.id _ihm_struct_assembly.name _ihm_struct_assembly.description -1 'Complete assembly' 'All known components' -2 'Modeled assembly' 'All components modeled by IMP' +1 'Complete assembly' 'All known components & All components modeled by IMP' # # loop_ @@ -238,8 +251,6 @@ def test_assembly_subset_modeled(self): _ihm_struct_assembly_details.asym_id _ihm_struct_assembly_details.entity_poly_segment_id 1 1 1 foo 1 A 1 -2 1 1 bar 2 . 2 -3 2 2 foo 1 A 1 # """) @@ -298,8 +309,9 @@ def test_model_representation_dumper(self): _ihm_model_representation_details.model_mode _ihm_model_representation_details.model_granularity _ihm_model_representation_details.model_object_count -1 1 1 foo A 1 sphere 1 flexible by-residue . -2 1 1 foo A 2 sphere . flexible by-feature 1 +_ihm_model_representation_details.description +1 1 1 foo A 1 sphere 1 flexible by-residue . . +2 1 1 foo A 2 sphere . flexible by-feature 1 . # """) @@ -367,8 +379,9 @@ def test_starting_model_dumper(self): _ihm_starting_model_details.starting_model_auth_asym_id _ihm_starting_model_details.starting_model_sequence_offset _ihm_starting_model_details.dataset_list_id -1 1 Nup84 A 3 'comparative model' A 0 3 -2 2 Nup85 B 4 'comparative model' A 0 4 +_ihm_starting_model_details.description +1 1 Nup84 A 3 'comparative model' A 0 3 . +2 2 Nup85 B 4 'comparative model' A 0 4 . # # loop_ @@ -437,7 +450,8 @@ def test_workflow(self): _ihm_external_reference_info.reference _ihm_external_reference_info.refers_to _ihm_external_reference_info.associated_url -1 . DOI foo Other . +_ihm_external_reference_info.details +1 . DOI foo Other . . # # loop_ @@ -491,10 +505,11 @@ def test_modeling_protocol(self): _ihm_modeling_protocol_details.ordered_flag _ihm_modeling_protocol_details.software_id _ihm_modeling_protocol_details.script_file_id +_ihm_modeling_protocol_details.description 1 1 1 2 . 'All components modeled by IMP' Sampling 'Monte Carlo' 0 500 YES NO NO -. . +. . . 2 2 1 2 . 'All components modeled by IMP' Sampling -'Replica exchange Molecular Dynamics' 400 2000 YES NO NO . . +'Replica exchange Molecular Dynamics' 400 2000 YES NO NO . . . # """) @@ -526,8 +541,9 @@ def test_post_process(self): _ihm_modeling_post_process.dataset_group_id _ihm_modeling_post_process.software_id _ihm_modeling_post_process.script_file_id -1 1 1 1 filter energy/score 500 400 2 . . . -2 2 1 1 cluster RMSD 2000 2000 2 . . . +_ihm_modeling_post_process.details +1 1 1 1 filter energy/score 500 400 2 . . . . +2 2 1 1 cluster RMSD 2000 2000 2 . . . . # """) @@ -554,7 +570,8 @@ def test_ensemble_info(self): _ihm_ensemble_info.num_ensemble_models_deposited _ihm_ensemble_info.ensemble_precision_value _ihm_ensemble_info.ensemble_file_id -1 'cluster 1' . 1 . . 1 1 . . +_ihm_ensemble_info.details +1 'cluster 1' . 1 . . 1 1 . . . # """) diff --git a/modules/mpi/dependency/MPI.cmake b/modules/mpi/dependency/MPI.cmake index bb0930a2a4..3d2dcf19ab 100644 --- a/modules/mpi/dependency/MPI.cmake +++ b/modules/mpi/dependency/MPI.cmake @@ -2,8 +2,15 @@ find_package(MPI) if("${MPI_CXX_FOUND}") message(STATUS "MPI found") - set(MPI_CXX_FLAGS ${MPI_CXX_COMPILE_FLAGS} CACHE INTERNAL "" FORCE) - #set(CMAKE_CXX_LINK_FLAGS ${CMAKE_CXX_LINK_FLAGS} ${MPI_CXX_LINK_FLAGS}) + # Workaround for CMake bug #18349, RHEL8 bug #1749463 + # (MPI_CXX_COMPILE_FLAGS is supposed to be a string but actually comes out + # as a ;-separated list, which results in literal semicolons in the compiler + # command line) + set(MPI_CXX_FLAGS) + foreach(_MPI_FLAG ${MPI_CXX_COMPILE_FLAGS}) + set(MPI_CXX_FLAGS "${MPI_CXX_FLAGS} ${_MPI_FLAG}") + endforeach() + set(MPI_CXX_FLAGS ${MPI_CXX_FLAGS} CACHE INTERNAL "" FORCE) set(MPI_LIBRARIES ${MPI_CXX_LIBRARIES} CACHE INTERNAL "" FORCE) set(MPI_INCLUDE_PATH ${MPI_CXX_INCLUDE_PATH} CACHE INTERNAL "" FORCE) # Run tests of IMP.mpi module on 2 processors, if we found a working mpiexec diff --git a/modules/multi_state/include/MultiStateModelScore.h b/modules/multi_state/include/MultiStateModelScore.h index 9c18a5d083..339c2b9667 100644 --- a/modules/multi_state/include/MultiStateModelScore.h +++ b/modules/multi_state/include/MultiStateModelScore.h @@ -43,6 +43,8 @@ class MultiStateModelScore { // get data name virtual std::string get_dataset_name() const = 0; + + virtual ~MultiStateModelScore() {} }; IMPMULTISTATE_END_NAMESPACE diff --git a/modules/npc/pyext/src/npc_restraints.py b/modules/npc/pyext/src/npc_restraints.py index 5c09e7668f..ad2e9da12d 100644 --- a/modules/npc/pyext/src/npc_restraints.py +++ b/modules/npc/pyext/src/npc_restraints.py @@ -1,5 +1,6 @@ import IMP.npc -import IMP.pmi.representation +import IMP.pmi1.tools + class XYRadialPositionRestraint(object): """Create XYRadial Position Restraint @@ -30,8 +31,8 @@ def __init__(self, self.label = "None" xyr = IMP.npc.XYRadialPositionRestraint(self.m, lower_bound, upper_bound, consider_radius, sigma) - #terminal_residue = IMP.pmi.tools.get_terminal_residue(representation, representation.hier_dict[protein], terminus="C") - residues = IMP.pmi.tools.select_by_tuple(representation, protein, resolution=1) + #terminal_residue = IMP.pmi1.tools.get_terminal_residue(representation, representation.hier_dict[protein], terminus="C") + residues = IMP.pmi1.tools.select_by_tuple(representation, protein, resolution=1) if (term == 'C'): terminal = residues[-1] #print (terminal, type(terminal)) @@ -57,7 +58,7 @@ def set_label(self, label): self.label = label def add_to_model(self): - IMP.pmi.tools.add_restraint_to_model(self.m, self.rs) + IMP.pmi1.tools.add_restraint_to_model(self.m, self.rs) def get_restraint(self): return self.rs @@ -104,8 +105,8 @@ def __init__(self, self.label = "None" xyr = IMP.npc.XYRadialPositionLowerRestraint(self.m, lower_bound, consider_radius, sigma) - #terminal_residue = IMP.pmi.tools.get_terminal_residue(representation, representation.hier_dict[protein], terminus="C") - residues = IMP.pmi.tools.select_by_tuple(representation, protein, resolution=1) + #terminal_residue = IMP.pmi1.tools.get_terminal_residue(representation, representation.hier_dict[protein], terminus="C") + residues = IMP.pmi1.tools.select_by_tuple(representation, protein, resolution=1) cterminal = residues[-1] #nterminal = residues[0] #print (cterminal, type(cterminal)) @@ -116,7 +117,7 @@ def set_label(self, label): self.label = label def add_to_model(self): - IMP.pmi.tools.add_restraint_to_model(self.m, self.rs) + IMP.pmi1.tools.add_restraint_to_model(self.m, self.rs) def get_restraint(self): return self.rs @@ -163,8 +164,8 @@ def __init__(self, self.label = "None" xyr = IMP.npc.XYRadialPositionUpperRestraint(self.m, upper_bound, consider_radius, sigma) - #terminal_residue = IMP.pmi.tools.get_terminal_residue(representation, representation.hier_dict[protein], terminus="C") - residues = IMP.pmi.tools.select_by_tuple(representation, protein, resolution=1) + #terminal_residue = IMP.pmi1.tools.get_terminal_residue(representation, representation.hier_dict[protein], terminus="C") + residues = IMP.pmi1.tools.select_by_tuple(representation, protein, resolution=1) cterminal = residues[-1] #nterminal = residues[0] #print (cterminal, type(cterminal)) @@ -175,7 +176,7 @@ def set_label(self, label): self.label = label def add_to_model(self): - IMP.pmi.tools.add_restraint_to_model(self.m, self.rs) + IMP.pmi1.tools.add_restraint_to_model(self.m, self.rs) def get_restraint(self): return self.rs @@ -225,8 +226,8 @@ def __init__(self, self.label = "None" zax = IMP.npc.ZAxialPositionRestraint(self.m, lower_bound, upper_bound, consider_radius, sigma) - #terminal_residue = IMP.pmi.tools.get_terminal_residue(representation, representation.hier_dict[protein], terminus="C") - residues = IMP.pmi.tools.select_by_tuple(representation, protein, resolution=1) + #terminal_residue = IMP.pmi1.tools.get_terminal_residue(representation, representation.hier_dict[protein], terminus="C") + residues = IMP.pmi1.tools.select_by_tuple(representation, protein, resolution=1) if term == 'C': residues = residues[-1:] elif term == 'N': @@ -246,7 +247,7 @@ def set_label(self, label): self.label = label def add_to_model(self): - IMP.pmi.tools.add_restraint_to_model(self.m, self.rs) + IMP.pmi1.tools.add_restraint_to_model(self.m, self.rs) def get_restraint(self): return self.rs @@ -293,8 +294,8 @@ def __init__(self, self.label = "None" zax = IMP.npc.ZAxialPositionLowerRestraint(self.m, lower_bound, consider_radius, sigma) - #terminal_residue = IMP.pmi.tools.get_terminal_residue(representation, representation.hier_dict[protein], terminus="C") - residues = IMP.pmi.tools.select_by_tuple(representation, protein, resolution=1) + #terminal_residue = IMP.pmi1.tools.get_terminal_residue(representation, representation.hier_dict[protein], terminus="C") + residues = IMP.pmi1.tools.select_by_tuple(representation, protein, resolution=1) cterminal = residues[-1] #nterminal = residues[0] #print (cterminal, type(cterminal)) @@ -305,7 +306,7 @@ def set_label(self, label): self.label = label def add_to_model(self): - IMP.pmi.tools.add_restraint_to_model(self.m, self.rs) + IMP.pmi1.tools.add_restraint_to_model(self.m, self.rs) def get_restraint(self): return self.rs @@ -352,8 +353,8 @@ def __init__(self, self.label = "None" zax = IMP.npc.ZAxialPositionUpperRestraint(self.m, upper_bound, consider_radius, sigma) - #terminal_residue = IMP.pmi.tools.get_terminal_residue(representation, representation.hier_dict[protein], terminus="C") - residues = IMP.pmi.tools.select_by_tuple(representation, protein, resolution=1) + #terminal_residue = IMP.pmi1.tools.get_terminal_residue(representation, representation.hier_dict[protein], terminus="C") + residues = IMP.pmi1.tools.select_by_tuple(representation, protein, resolution=1) cterminal = residues[-1] #nterminal = residues[0] #print (cterminal, type(cterminal)) @@ -364,7 +365,7 @@ def set_label(self, label): self.label = label def add_to_model(self): - IMP.pmi.tools.add_restraint_to_model(self.m, self.rs) + IMP.pmi1.tools.add_restraint_to_model(self.m, self.rs) def get_restraint(self): return self.rs @@ -414,8 +415,8 @@ def __init__(self, self.label = "None" yax = IMP.npc.YAxialPositionRestraint(self.m, lower_bound, upper_bound, consider_radius, sigma) - #terminal_residue = IMP.pmi.tools.get_terminal_residue(representation, representation.hier_dict[protein], terminus="C") - residues = IMP.pmi.tools.select_by_tuple(representation, protein, resolution=1) + #terminal_residue = IMP.pmi1.tools.get_terminal_residue(representation, representation.hier_dict[protein], terminus="C") + residues = IMP.pmi1.tools.select_by_tuple(representation, protein, resolution=1) if (term == 'C'): terminal = residues[-1] #print (terminal, type(terminal)) @@ -441,7 +442,7 @@ def set_label(self, label): self.label = label def add_to_model(self): - IMP.pmi.tools.add_restraint_to_model(self.m, self.rs) + IMP.pmi1.tools.add_restraint_to_model(self.m, self.rs) def get_restraint(self): return self.rs @@ -488,8 +489,8 @@ def __init__(self, self.label = "None" yax = IMP.npc.YAxialPositionLowerRestraint(self.m, lower_bound, consider_radius, sigma) - #terminal_residue = IMP.pmi.tools.get_terminal_residue(representation, representation.hier_dict[protein], terminus="C") - residues = IMP.pmi.tools.select_by_tuple(representation, protein, resolution=1) + #terminal_residue = IMP.pmi1.tools.get_terminal_residue(representation, representation.hier_dict[protein], terminus="C") + residues = IMP.pmi1.tools.select_by_tuple(representation, protein, resolution=1) cterminal = residues[-1] #nterminal = residues[0] #print (cterminal, type(cterminal)) @@ -500,7 +501,7 @@ def set_label(self, label): self.label = label def add_to_model(self): - IMP.pmi.tools.add_restraint_to_model(self.m, self.rs) + IMP.pmi1.tools.add_restraint_to_model(self.m, self.rs) def get_restraint(self): return self.rs @@ -547,8 +548,8 @@ def __init__(self, self.label = "None" yax = IMP.npc.YAxialPositionUpperRestraint(self.m, upper_bound, consider_radius, sigma) - #terminal_residue = IMP.pmi.tools.get_terminal_residue(representation, representation.hier_dict[protein], terminus="C") - residues = IMP.pmi.tools.select_by_tuple(representation, protein, resolution=1) + #terminal_residue = IMP.pmi1.tools.get_terminal_residue(representation, representation.hier_dict[protein], terminus="C") + residues = IMP.pmi1.tools.select_by_tuple(representation, protein, resolution=1) cterminal = residues[-1] #nterminal = residues[0] #print (cterminal, type(cterminal)) @@ -559,7 +560,7 @@ def set_label(self, label): self.label = label def add_to_model(self): - IMP.pmi.tools.add_restraint_to_model(self.m, self.rs) + IMP.pmi1.tools.add_restraint_to_model(self.m, self.rs) def get_restraint(self): return self.rs @@ -609,7 +610,7 @@ def __init__(self, self.label = "None" msl = IMP.npc.MembraneSurfaceLocationRestraint(self.m, tor_R, tor_r, tor_th, sigma) - residues = IMP.pmi.tools.select_by_tuple(representation, protein, resolution=resolution) + residues = IMP.pmi1.tools.select_by_tuple(representation, protein, resolution=resolution) for residue in residues: #print (residue, type(residue)) msl.add_particle(residue) @@ -626,7 +627,7 @@ def set_label(self, label): self.label = label def add_to_model(self): - IMP.pmi.tools.add_restraint_to_model(self.m, self.rs) + IMP.pmi1.tools.add_restraint_to_model(self.m, self.rs) def get_restraint(self): return self.rs @@ -679,11 +680,11 @@ def __init__(self, self.label = "None" msl = IMP.npc.MembraneSurfaceLocationConditionalRestraint(self.m, tor_R, tor_r, tor_th, sigma) - residues1 = IMP.pmi.tools.select_by_tuple(representation, protein1, resolution=resolution) + residues1 = IMP.pmi1.tools.select_by_tuple(representation, protein1, resolution=resolution) for residue in residues1: #print (residue, type(residue)) msl.add_particle1(residue) - residues2 = IMP.pmi.tools.select_by_tuple(representation, protein2, resolution=resolution) + residues2 = IMP.pmi1.tools.select_by_tuple(representation, protein2, resolution=resolution) for residue in residues2: #print (residue, type(residue)) msl.add_particle2(residue) @@ -702,7 +703,7 @@ def set_label(self, label): self.label = label def add_to_model(self): - IMP.pmi.tools.add_restraint_to_model(self.m, self.rs) + IMP.pmi1.tools.add_restraint_to_model(self.m, self.rs) def get_restraint(self): return self.rs @@ -752,7 +753,7 @@ def __init__(self, self.label = "None" mex = IMP.npc.MembraneExclusionRestraint(self.m, tor_R, tor_r, tor_th, sigma) - residues = IMP.pmi.tools.select_by_tuple(representation, protein, resolution=resolution) + residues = IMP.pmi1.tools.select_by_tuple(representation, protein, resolution=resolution) for residue in residues: #print (residue, type(residue)) mex.add_particle(residue) @@ -769,7 +770,7 @@ def set_label(self, label): self.label = label def add_to_model(self): - IMP.pmi.tools.add_restraint_to_model(self.m, self.rs) + IMP.pmi1.tools.add_restraint_to_model(self.m, self.rs) def get_restraint(self): return self.rs diff --git a/modules/npctransport b/modules/npctransport index 47bbe027a5..f93fcb9401 160000 --- a/modules/npctransport +++ b/modules/npctransport @@ -1 +1 @@ -Subproject commit 47bbe027a54d3e9521d07ca904197200c75114e2 +Subproject commit f93fcb9401a0f5ccdd4a4e8f0deeec872de7e0fb diff --git a/modules/pmi/.travis.yml b/modules/pmi/.travis.yml index 1d20863125..ddbad830fe 100644 --- a/modules/pmi/.travis.yml +++ b/modules/pmi/.travis.yml @@ -14,8 +14,8 @@ install: script: - mkdir build - cd build - - PYTHON_INC=$(echo $(dirname $(which python))/../include/python*/) - - cmake .. -DIMP_DIR=${CONDA_PREFIX}/lib/cmake/IMP -DPYTHON_INCLUDE_DIR=$PYTHON_INC -DCMAKE_CXX_FLAGS="-fprofile-arcs -ftest-coverage" + - if test "${TRAVIS_PYTHON_VERSION}" = "2.7" ; then PY2=on; else PY2=off; fi + - cmake .. -DIMP_DIR=${CONDA_PREFIX}/lib/cmake/IMP -DUSE_PYTHON2=${PY2} -DCMAKE_CXX_FLAGS="-fprofile-arcs -ftest-coverage" - make - ./setup_environment.sh ../test/nosetests --with-coverage --cover-branches -e 'probabilistic' ../test/test_*.py ../test/medium_test_*.py test/*/*.py > /dev/null after_script: diff --git a/modules/pmi/CMakeLists.txt b/modules/pmi/CMakeLists.txt index ef70bf40f5..c84d77f587 100644 --- a/modules/pmi/CMakeLists.txt +++ b/modules/pmi/CMakeLists.txt @@ -1,6 +1,7 @@ # Are we running cmake from this directory (out of tree build) ? if(CMAKE_SOURCE_DIR STREQUAL CMAKE_CURRENT_SOURCE_DIR) cmake_minimum_required(VERSION 2.8.3) + project(imp_module) if(POLICY CMP0058) cmake_policy(SET CMP0058 NEW) diff --git a/modules/pmi/Setup.cmake b/modules/pmi/Setup.cmake deleted file mode 100644 index 33d28d5436..0000000000 --- a/modules/pmi/Setup.cmake +++ /dev/null @@ -1,9 +0,0 @@ -execute_process(COMMAND python setup_git.py - RESULT_VARIABLE setup - OUTPUT_VARIABLE toutput - ERROR_VARIABLE error - WORKING_DIRECTORY ${PROJECT_SOURCE_DIR} - OUTPUT_STRIP_TRAILING_WHITESPACE) -if(NOT ${setup} EQUAL 0) - message(FATAL_ERROR " Failed to run setup_git.py: ${setup}; ${error}") -endif() diff --git a/modules/pmi/benchmark/benchmark_loop_reconstruction.py b/modules/pmi/benchmark/benchmark_loop_reconstruction.py index 5673a63ff1..2c374adf4a 100644 --- a/modules/pmi/benchmark/benchmark_loop_reconstruction.py +++ b/modules/pmi/benchmark/benchmark_loop_reconstruction.py @@ -7,9 +7,14 @@ import time import sys import os +try: + from time import process_time # needs python 3.3 or later +except ImportError: + from time import clock as process_time import IMP.pmi.restraints.stereochemistry -import IMP.pmi.representation as representation +import IMP.pmi.topology +import IMP.pmi.dof import IMP.pmi.tools as tools import IMP.pmi.samplers as samplers import IMP.pmi.output as output @@ -22,6 +27,8 @@ old_stdout = sys.stdout class DummyFile(object): + def flush(self): + pass def write(self, txt): pass sys.stdout = DummyFile() @@ -30,15 +37,7 @@ def write(self, txt): pdbfile = IMP.pmi.get_data_path("benchmark_starting_structure.pdb") fastafile = IMP.pmi.get_data_path("benchmark_sequence.fasta") -fastids = tools.get_ids_from_fasta_file(fastafile) -missing_bead_size = 1 - - -# Component pdbfile chainid rgb color fastafile sequence id -# in fastafile -data = [("chainA", pdbfile, "A", 0.00000000, (fastafile, 0)), - ("chainB", pdbfile, "B", 0.50000000, (fastafile, 0))] - +sequences = IMP.pmi.topology.Sequences(fastafile) # create the representation log_objects = [] @@ -48,62 +47,48 @@ def write(self, txt): log_objects.append(sw) m = IMP.Model() -with IMP.allow_deprecated(): - r = representation.Representation(m) - -hierarchies = {} - -for d in data: - component_name = d[0] - pdb_file = d[1] - chain_id = d[2] - color_id = d[3] - fasta_file = d[4][0] - fasta_file_id = d[4][1] - # avoid to add a component with the same name - r.create_component(component_name, - color=color_id) - - r.add_component_sequence(component_name, - fasta_file, - id=fastids[fasta_file_id]) - - hierarchies = r.autobuild_model(component_name, - pdb_file, - chain_id, - resolutions=[1, 10], - missingbeadsize=missing_bead_size) - - r.show_component_table(component_name) - -rbAB = r.set_rigid_bodies(["chainA", "chainB"]) - -r.set_floppy_bodies() -r.fix_rigid_bodies([rbAB]) -r.setup_bonds() - - -log_objects.append(r) +s = IMP.pmi.topology.System(m) +st = s.create_state() + +cA = st.create_molecule("chainA", sequence=sequences[0]) +atomic = cA.add_structure(pdbfile, chain_id='A') +cA.add_representation(atomic, resolutions=[1, 10], color=0.) +cA.add_representation(cA.get_non_atomic_residues(), + resolutions=[1], color=0.) + +cB = st.create_molecule("chainB", sequence=sequences[0]) +atomic = cB.add_structure(pdbfile, chain_id='B') +cB.add_representation(atomic, resolutions=[1, 10], color=0.5) +cB.add_representation(cB.get_non_atomic_residues(), + resolutions=[1], color=0.) +root_hier = s.build() + +dof = IMP.pmi.dof.DegreesOfFreedom(m) +dof.create_rigid_body(cA) +dof.create_rigid_body(cB) + +cr = IMP.pmi.restraints.stereochemistry.ConnectivityRestraint(root_hier) +cr.add_to_model() +log_objects.append(cr) listofexcludedpairs = [] -lof = [(1, 12, "chainA"), (1, 12, "chainB"), - (294, 339, "chainA"), (294, 339, "chainB"), - (686, 701, "chainA"), (686, 701, "chainB"), - (454, 464, "chainA"), (454, 464, "chainB"), - (472, 486, "chainA"), (472, 486, "chainB"), - (814, 859, "chainA"), (814, 859, "chainB")] - +lof = [cA[:12], cB[:12], + cA[293:339], cB[293:339], + cA[685:701], cB[685:701], + cA[453:464], cB[453:464], + cA[471:486], cB[471:486], + cA[813:859], cB[813:859]] # add bonds and angles for l in lof: - rbr = IMP.pmi.restraints.stereochemistry.ResidueBondRestraint(r, l) + rbr = IMP.pmi.restraints.stereochemistry.ResidueBondRestraint(objects=l) rbr.add_to_model() listofexcludedpairs += rbr.get_excluded_pairs() log_objects.append(rbr) - rar = IMP.pmi.restraints.stereochemistry.ResidueAngleRestraint(r, l) + rar = IMP.pmi.restraints.stereochemistry.ResidueAngleRestraint(objects=l) rar.add_to_model() listofexcludedpairs += rar.get_excluded_pairs() log_objects.append(rar) @@ -111,24 +96,23 @@ def write(self, txt): # add excluded volume ev = IMP.pmi.restraints.stereochemistry.ExcludedVolumeSphere( - r, - resolution=10.0) + included_objects=root_hier, resolution=10.0) ev.add_excluded_particle_pairs(listofexcludedpairs) ev.add_to_model() log_objects.append(ev) -mc = samplers.MonteCarlo(m, [r], 1.0) +mc = samplers.MonteCarlo(m, dof.get_movers(), 1.0) log_objects.append(mc) -start_time = time.clock() +start_time = process_time() # In debug mode things are way too slow to actually run MC if IMP.get_check_level() < IMP.USAGE_AND_INTERNAL: o = output.Output() - rmf = o.init_rmf("conformations.rmf3", [r.prot]) + rmf = o.init_rmf("conformations.rmf3", [root_hier]) o.init_stat2("modeling.stat", log_objects) o.write_rmf("conformations.rmf3") - o.init_pdb("conformations.pdb", r.prot) + o.init_pdb("conformations.pdb", root_hier) for i in range(0, 10): # print("Running job, frame number ", i) @@ -140,7 +124,7 @@ def write(self, txt): o.close_rmf("conformations.rmf3") sys.stdout = old_stdout -IMP.benchmark.report("pmi loop", time.clock() - start_time, 3*10+5) +IMP.benchmark.report("pmi loop", process_time() - start_time, 3*10+5) if IMP.get_check_level() < IMP.USAGE_AND_INTERNAL: for output in ["conformations.pdb", "conformations.rmf3", "modeling.stat"]: diff --git a/modules/pmi/benchmark/benchmark_replica_exchange.py b/modules/pmi/benchmark/benchmark_replica_exchange.py index f5b912019f..c680b6c3c7 100644 --- a/modules/pmi/benchmark/benchmark_replica_exchange.py +++ b/modules/pmi/benchmark/benchmark_replica_exchange.py @@ -1,40 +1,55 @@ import IMP import IMP.benchmark import IMP.pmi.samplers -import IMP.pmi.representation +import IMP.pmi.topology +import IMP.pmi.dof import IMP.pmi.restraints.basic import IMP.pmi.macros import IMP.pmi.output import time import sys import shutil +try: + from time import process_time # needs python 3.3 or later +except ImportError: + from time import clock as process_time + IMP.setup_from_argv(sys.argv, "Replica exchange benchmark.") IMP.set_log_level(IMP.SILENT) old_stdout = sys.stdout class DummyFile(object): + def flush(self): + pass def write(self, txt): pass sys.stdout = DummyFile() """setting up the representation -PMI 1.0 representation. Creates two particles and +PMI 2 representation. Creates two particles and an harmonic distance restraints between them""" m=IMP.Model() -with IMP.allow_deprecated(): - r=IMP.pmi.representation.Representation(m) -r.create_component("A") -r.add_component_beads("A",[(1,1),(2,2)]) -ps=IMP.atom.get_leaves(r.prot) -dr=IMP.pmi.restraints.basic.DistanceRestraint(r,(1,1,"A"),(2,2,"A"),10,10) +s = IMP.pmi.topology.System(m) +st1 = s.create_state() +mol = st1.create_molecule("A", "GG", "A") +mol.add_representation(resolutions=[1]) +hier = s.build() + +dof = IMP.pmi.dof.DegreesOfFreedom(mol) +dof.create_flexible_beads(mol, max_trans=3.0, resolution=1) + +ps=IMP.atom.get_leaves(hier) +dr=IMP.pmi.restraints.basic.DistanceRestraint( + root_hier=hier, tuple_selection1=(1,1,"A"), + tuple_selection2=(2,2,"A"), distancemin=10, distancemax=10) dr.add_to_model() -start_time = time.clock() +start_time = process_time() rex=IMP.pmi.macros.ReplicaExchange0(m, - r, - monte_carlo_sample_objects=[r], - output_objects=[r,dr], + root_hier=hier, + monte_carlo_sample_objects=dof.get_movers(), + output_objects=[dr], monte_carlo_temperature=1.0, replica_exchange_minimum_temperature=1.0, replica_exchange_maximum_temperature=2.5, @@ -68,4 +83,4 @@ def write(self, txt): temperatures=rex.replica_exchange_object.get_my_parameter("temp") sys.stdout = old_stdout -IMP.benchmark.report("replica "+str(my_index), time.clock() - start_time, 8.5) +IMP.benchmark.report("replica "+str(my_index), process_time() - start_time, 8.5) diff --git a/modules/pmi/pyext/src/dof/__init__.py b/modules/pmi/pyext/src/dof/__init__.py index 78abe839fe..df7e28c0e9 100644 --- a/modules/pmi/pyext/src/dof/__init__.py +++ b/modules/pmi/pyext/src/dof/__init__.py @@ -300,6 +300,7 @@ def _setup_srb(self,hiers,max_trans,max_rot,axis): srbm = IMP.pmi.TransformMover(hiers[0][0].get_model(), max_trans, max_rot) else: srbm = IMP.pmi.TransformMover(hiers[0][0].get_model(),axis[0],axis[1],max_trans, max_rot) + srbm.set_was_used(True) super_rigid_rbs,super_rigid_xyzs = IMP.pmi.tools.get_rbs_and_beads(hiers) ct = 0 self.movers_particles_map[srbm]=[] diff --git a/modules/pmi/pyext/src/io/crosslink.py b/modules/pmi/pyext/src/io/crosslink.py index ca9d0906bd..54addf6f05 100644 --- a/modules/pmi/pyext/src/io/crosslink.py +++ b/modules/pmi/pyext/src/io/crosslink.py @@ -20,6 +20,7 @@ import ihm.location import ihm.dataset from collections import defaultdict +import numpy # json default serializations def set_json_default(obj): @@ -513,15 +514,15 @@ def __init__(self,format): def get_data(self,input_string): if self.format is "PROXL": - tockens=input_string.split("\t") + tokens=input_string.split("\t") xl={} - if tockens[0]=="SEARCH ID(S)": + if tokens[0]=="SEARCH ID(S)": return None else: - xl[self.protein1_key]=tockens[2] - xl[self.protein2_key]=tockens[4] - xl[self.residue1_key]=int(tockens[3]) - xl[self.residue2_key]=int(tockens[5]) + xl[self.protein1_key]=tokens[2] + xl[self.protein2_key]=tokens[4] + xl[self.residue1_key]=int(tokens[3]) + xl[self.residue2_key]=int(tokens[5]) return xl class CrossLinkDataBase(_CrossLinkDataBaseStandardKeys): @@ -911,6 +912,8 @@ def append_database(self,CrossLinkDataBase2): if name1 == name2: name1=id(self) name2=id(CrossLinkDataBase2) + self.set_name(name1) + CrossLinkDataBase2.set_name(name2) #rename first database: new_data_base={} @@ -996,6 +999,28 @@ def rename_proteins(self,old_to_new_names_dictionary, protein_to_rename="both"): fo2=FilterOperator(self.protein2_key,operator.eq,old_name) self.set_value(self.protein2_key,new_name,fo2) + def classify_crosslinks_by_score(self,number_of_classes): + ''' + This function creates as many classes as in the input (number_of_classes: integer) + and partition crosslinks according to their identification scores. Classes are defined in the psi key. + ''' + + if self.id_score_key is not None: + scores=self.get_values(self.id_score_key) + else: + raise ValueError('The crosslink database does not contain score values') + minscore=min(scores) + maxscore=max(scores) + scoreclasses=numpy.linspace(minscore, maxscore, number_of_classes+1) + if self.psi_key is None: + self.create_new_keyword(self.psi_key,values_from_keyword=None) + for xl in self: + score=xl[self.id_score_key] + for n,classmin in enumerate(scoreclasses[0:-1]): + if score>=classmin and score<=scoreclasses[n+1]: + xl[self.psi_key]=str("CLASS_"+str(n)) + self._update() + def clone_protein(self,protein_name,new_protein_name): new_xl_dict={} for id in self.data_base.keys(): @@ -1499,12 +1524,8 @@ class CrossLinkDataBaseFromStructure(object): ''' This class generates a CrossLinkDataBase from a given structure ''' - def __init__(self,representation=None, - system=None, - residue_types_1=["K"], - residue_types_2=["K"], - reactivity_range=[0,1], - kt=1.0): + def __init__(self, system, residue_types_1=["K"], + residue_types_2=["K"], reactivity_range=[0,1], kt=1.0): import numpy.random import math @@ -1514,19 +1535,8 @@ def __init__(self,representation=None, cldbkc.set_residue1_key("Residue1") cldbkc.set_residue2_key("Residue2") self.cldb=CrossLinkDataBase(cldbkc) - if representation is not None: - #PMI 1.0 mode - self.mode="pmi1" - self.representation=representation - self.model=self.representation.model - elif system is not None: - #PMI 2.0 mode - self.system=system - self.model=self.system.model - self.mode="pmi2" - else: - print("Argument error: please provide either a representation object or a IMP.Hierarchy") - raise + self.system=system + self.model=self.system.model self.residue_types_1=residue_types_1 self.residue_types_2=residue_types_2 self.kt=kt @@ -1538,68 +1548,37 @@ def __init__(self,representation=None, self.xwalk_interacting_pairs=None import random - if self.mode=="pmi1": - for protein in self.representation.sequence_dict.keys(): + for state in self.system.get_states(): + for moleculename,molecules in state.get_molecules().items(): + for molecule in molecules: # we are saving a dictionary with protein name, residue number and random reactivity of the residue - seq=self.representation.sequence_dict[protein] - residues=[i for i in range(1,len(seq)+1) if ((seq[i-1] in self.residue_types_1) or (seq[i-1] in self.residue_types_2))] - - for r in residues: - # uniform random reactivities - #self.reactivity_dictionary[(protein,r)]=random.uniform(reactivity_range[0],reactivity_range[1]) - # getting reactivities from the CDF of an exponential distribution - rexp=numpy.random.exponential(0.1) - prob=1.0-math.exp(-rexp) - self.reactivity_dictionary[(protein,r)]=prob - - - residues1=[i for i in range(1,len(seq)+1) if seq[i-1] in self.residue_types_1] - residues2=[i for i in range(1,len(seq)+1) if seq[i-1] in self.residue_types_2] - for r in residues1: - h=IMP.pmi.tools.select_by_tuple(self.representation,(r,r,protein),resolution=1)[0] - p=h.get_particle() - index=p.get_index() - self.indexes_dict1[index]=(protein,r) - self.protein_residue_dict[(protein,r)]=index - for r in residues2: - h=IMP.pmi.tools.select_by_tuple(self.representation,(r,r,protein),resolution=1)[0] - p=h.get_particle() - index=p.get_index() - self.indexes_dict2[index]=(protein,r) - self.protein_residue_dict[(protein,r)]=index - - if self.mode=="pmi2": - for state in self.system.get_states(): - for moleculename,molecules in state.get_molecules().items(): - for molecule in molecules: - # we are saving a dictionary with protein name, residue number and random reactivity of the residue - seq=molecule.sequence - residues=[i for i in range(1,len(seq)+1) if ((seq[i-1] in self.residue_types_1) or (seq[i-1] in self.residue_types_2))] - - for r in residues: - # uniform random reactivities - #self.reactivity_dictionary[(protein,r)]=random.uniform(reactivity_range[0],reactivity_range[1]) - # getting reactivities from the CDF of an exponential distribution - rexp=numpy.random.exponential(0.00000001) - prob=1.0-math.exp(-rexp) - self.reactivity_dictionary[(molecule,r)]=prob - - residues1=[i for i in range(1,len(seq)+1) if seq[i-1] in self.residue_types_1] - residues2=[i for i in range(1,len(seq)+1) if seq[i-1] in self.residue_types_2] - for r in residues1: - s=IMP.atom.Selection(molecule.hier,residue_index=r,resolution=1) - ps=s.get_selected_particles() - for p in ps: - index=p.get_index() - self.indexes_dict1[index]=(molecule,r) - self.protein_residue_dict[(molecule,r)]=index - for r in residues2: - s=IMP.atom.Selection(molecule.hier,residue_index=r,resolution=1) - ps=s.get_selected_particles() - for p in ps: - index=p.get_index() - self.indexes_dict2[index]=(molecule,r) - self.protein_residue_dict[(molecule,r)]=index + seq=molecule.sequence + residues=[i for i in range(1,len(seq)+1) if ((seq[i-1] in self.residue_types_1) or (seq[i-1] in self.residue_types_2))] + + for r in residues: + # uniform random reactivities + #self.reactivity_dictionary[(protein,r)]=random.uniform(reactivity_range[0],reactivity_range[1]) + # getting reactivities from the CDF of an exponential distribution + rexp=numpy.random.exponential(0.00000001) + prob=1.0-math.exp(-rexp) + self.reactivity_dictionary[(molecule,r)]=prob + + residues1=[i for i in range(1,len(seq)+1) if seq[i-1] in self.residue_types_1] + residues2=[i for i in range(1,len(seq)+1) if seq[i-1] in self.residue_types_2] + for r in residues1: + s=IMP.atom.Selection(molecule.hier,residue_index=r,resolution=1) + ps=s.get_selected_particles() + for p in ps: + index=p.get_index() + self.indexes_dict1[index]=(molecule,r) + self.protein_residue_dict[(molecule,r)]=index + for r in residues2: + s=IMP.atom.Selection(molecule.hier,residue_index=r,resolution=1) + ps=s.get_selected_particles() + for p in ps: + index=p.get_index() + self.indexes_dict2[index]=(molecule,r) + self.protein_residue_dict[(molecule,r)]=index def get_all_possible_pairs(self): @@ -1618,14 +1597,10 @@ def get_all_feasible_pairs(self,distance=21): particle_distance=IMP.core.get_distance(IMP.core.XYZ(IMP.get_particles(self.model,[index1])[0]),IMP.core.XYZ(IMP.get_particles(self.model,[index2])[0])) if particle_distance <= distance: particle_index_pairs.append((index1,index2)) - if self.mode=="pmi1": - new_xl[self.cldb.protein1_key]=a[0] - new_xl[self.cldb.protein2_key]=b[0] - elif self.mode=="pmi2": - new_xl[self.cldb.protein1_key]=a[0].get_name() - new_xl[self.cldb.protein2_key]=b[0].get_name() - new_xl["molecule_object1"]=a[0] - new_xl["molecule_object2"]=b[0] + new_xl[self.cldb.protein1_key]=a[0].get_name() + new_xl[self.cldb.protein2_key]=b[0].get_name() + new_xl["molecule_object1"]=a[0] + new_xl["molecule_object2"]=b[0] new_xl[self.cldb.residue1_key]=a[1] new_xl[self.cldb.residue2_key]=b[1] self.cldb.data_base[str(nxl)]=[new_xl] @@ -1669,14 +1644,10 @@ def get_data_base(self,total_number_of_spectra, noisy=True new_xl={} - if self.mode=="pmi1": - new_xl[self.cldb.protein1_key]=pra[0] - new_xl[self.cldb.protein2_key]=pra[1] - elif self.mode=="pmi2": - new_xl[self.cldb.protein1_key]=pra[0].get_name() - new_xl[self.cldb.protein2_key]=pra[1].get_name() - new_xl["molecule_object1"]=pra[0] - new_xl["molecule_object2"]=pra[1] + new_xl[self.cldb.protein1_key]=pra[0].get_name() + new_xl[self.cldb.protein2_key]=pra[1].get_name() + new_xl["molecule_object1"]=pra[0] + new_xl["molecule_object2"]=pra[1] new_xl[self.cldb.residue1_key]=pra[2] new_xl[self.cldb.residue2_key]=pra[3] new_xl["Noisy"]=noisy @@ -1725,26 +1696,13 @@ def get_random_residue_pair(self,distance=None,xwalk_bin_path=None,max_delta_dis if distance is None: # get a random pair while True: - if self.mode=="pmi1": - protein1=choice(self.representation.sequence_dict.keys()) - protein2=choice(self.representation.sequence_dict.keys()) - seq1=self.representation.sequence_dict[protein1] - seq2=self.representation.sequence_dict[protein2] - residue1=choice([i for i in range(1,len(seq1)+1) if seq1[i-1] in self.residue_types_1]) - residue2=choice([i for i in range(1,len(seq2)+1) if seq2[i-1] in self.residue_types_2]) - h1=IMP.pmi.tools.select_by_tuple(self.representation,(residue1,residue1,protein1),resolution=1)[0] - h2=IMP.pmi.tools.select_by_tuple(self.representation,(residue2,residue2,protein2),resolution=1)[0] - particle_distance=IMP.core.get_distance(IMP.core.XYZ(h1.get_particle()),IMP.core.XYZ(h2.get_particle())) - if (protein1,residue1) != (protein2,residue2): - break - elif self.mode=="pmi2": - (protein1,residue1)=choice(self.protein_residue_dict.keys()) - (protein2,residue2)=choice(self.protein_residue_dict.keys()) - index1=self.protein_residue_dict[(protein1,residue1)] - index2=self.protein_residue_dict[(protein2,residue2)] - particle_distance=IMP.core.get_distance(IMP.core.XYZ(IMP.get_particles(self.model,[index1])[0]),IMP.core.XYZ(IMP.get_particles(self.model,[index2])[0])) - if (protein1,residue1) != (protein2,residue2): - break + (protein1,residue1)=choice(self.protein_residue_dict.keys()) + (protein2,residue2)=choice(self.protein_residue_dict.keys()) + index1=self.protein_residue_dict[(protein1,residue1)] + index2=self.protein_residue_dict[(protein2,residue2)] + particle_distance=IMP.core.get_distance(IMP.core.XYZ(IMP.get_particles(self.model,[index1])[0]),IMP.core.XYZ(IMP.get_particles(self.model,[index2])[0])) + if (protein1,residue1) != (protein2,residue2): + break else: # get a pair of residues whose distance is below the threshold if not xwalk_bin_path: @@ -1862,10 +1820,10 @@ def get_xwalk_distances(self,xwalk_bin_path,distance): output_list_of_distance=[] for line in xwalkout.split("\n")[0:-2]: - tockens=line.split() - first=tockens[2] - second=tockens[3] - distance=float(tockens[6]) + tokens=line.split() + first=tokens[2] + second=tokens[3] + distance=float(tokens[6]) fs=first.split("-") ss=second.split("-") chainid1=fs[2] diff --git a/modules/pmi/pyext/src/macros.py b/modules/pmi/pyext/src/macros.py index 4cc64b5889..0373a1c895 100644 --- a/modules/pmi/pyext/src/macros.py +++ b/modules/pmi/pyext/src/macros.py @@ -4,7 +4,6 @@ from __future__ import print_function, division import IMP -import IMP.pmi.representation import IMP.pmi.tools import IMP.pmi.samplers import IMP.pmi.output @@ -2197,14 +2196,17 @@ def update_seldicts(self): def align(self): - print("alignment") tr = IMP.atom.get_transformation_aligning_first_to_second(self.sel1_alignment, self.sel0_alignment) for rb in self.rbs1: IMP.core.transform(rb, tr) for bead in self.beads1: - IMP.core.transform(IMP.core.XYZ(bead), tr) + try: + IMP.core.transform(IMP.core.XYZ(bead), tr) + except: + continue + self.model.update() diff --git a/modules/pmi/pyext/src/mmcif.py b/modules/pmi/pyext/src/mmcif.py index ed3d1c99c7..d8d2f24779 100644 --- a/modules/pmi/pyext/src/mmcif.py +++ b/modules/pmi/pyext/src/mmcif.py @@ -11,12 +11,12 @@ from __future__ import print_function import copy +import RMF import IMP.core import IMP.algebra import IMP.atom import IMP.em import IMP.isd -import IMP.pmi.representation import IMP.pmi.tools from IMP.pmi.tools import OrderedDict import IMP.pmi.output @@ -943,10 +943,12 @@ def load_all_models(self, simo, state): # Correct path rmf_file = os.path.join(os.path.dirname(stat_fname), "%d.rmf3" % model_num) - for c in state.all_modeled_components: - # todo: this only works with PMI 1 - state._pmi_object.set_coordinates_from_rmf(c, rmf_file, 0, - force_rigid_update=True) + # todo: test with real PMI2 systems + if os.path.exists(rmf_file): + rh = RMF.open_rmf_file_read_only(rmf_file) + system = state._pmi_object.system + IMP.rmf.link_hierarchies(rh, [system.hier]) + IMP.rmf.load_frame(fh, RMF.FrameID(0)) # todo: fill in other data from stat file, e.g. crosslink phi/psi yield stats model_num += 1 @@ -1157,8 +1159,6 @@ def __init__(self, fh): self.fh = fh self._state_ensemble_offset = 0 - self._each_metadata = [] # list of metadata for each representation - self._file_datasets = [] self._main_script = os.path.abspath(sys.argv[0]) # Point to the main modeling script @@ -1199,7 +1199,7 @@ def __init__(self, fh): def create_representation(self, name): """Create a new Representation and return it. This can be passed to add_model(), add_bead_element() or add_pdb_element().""" - r = ihm.representation.Representation() + r = ihm.representation.Representation(name=name) self.system.orphan_representations.append(r) return r @@ -1323,19 +1323,6 @@ def finalize(self): and/or modified using the ihm API.""" self._add_restraint_model_fits() - # Add metadata to ihm.System - self.system.software.extend(m for m in self._metadata - if isinstance(m, ihm.Software)) - self.system.citations.extend(m for m in self._metadata - if isinstance(m, ihm.Citation)) - self.system.locations.extend(m for m in self._metadata - if isinstance(m, ihm.location.FileLocation)) - - # Point all locations to repos, if applicable - all_repos = [m for m in self._metadata - if isinstance(m, ihm.location.Repository)] - self.system.update_locations_in_repositories(all_repos) - def add_pdb_element(self, state, name, start, end, offset, pdbname, chain, hier, representation=None): if self._is_excluded(name, start, end): @@ -1611,16 +1598,6 @@ def add_model(self, group, assembly=None, representation=None): group.append(m) return m - def _update_locations(self, filelocs): - """Update FileLocation to point to a parent repository, if any""" - all_repos = [m for m in self._metadata - if isinstance(m, ihm.location.Repository)] - for fileloc in filelocs: - ihm.location.Repository._update_in_repos(fileloc, all_repos) - - _metadata = property(lambda self: - itertools.chain.from_iterable(self._each_metadata)) - def read(fh, model_class=ihm.model.Model, format='mmCIF', handlers=[]): """Read data from the mmCIF file handle `fh`. diff --git a/modules/pmi/pyext/src/output.py b/modules/pmi/pyext/src/output.py index daa02aa05b..a0f9898068 100644 --- a/modules/pmi/pyext/src/output.py +++ b/modules/pmi/pyext/src/output.py @@ -42,9 +42,7 @@ class ProtocolOutput(object): Unlike simple output of model coordinates, a complete protocol includes the input data used, details on the restraints, sampling, and clustering, as well as output models. - Use via IMP.pmi.representation.Representation.add_protocol_output() - (for PMI 1) or - IMP.pmi.topology.System.add_protocol_output() (for PMI 2). + Use via IMP.pmi.topology.System.add_protocol_output(). @see IMP.pmi.mmcif.ProtocolOutput for a concrete subclass that outputs mmCIF files. @@ -364,7 +362,8 @@ def write_pdb_best_scoring(self, score): # update the score list if self.replica_exchange: # read the self.best_score_list from the file - exec(open(self.best_score_file_name).read()) + with open(self.best_score_file_name) as fh: + exec(fh.read()) if len(self.best_score_list) < self.nbestscoring: self.best_score_list.append(score) @@ -406,10 +405,10 @@ def write_pdb_best_scoring(self, score): def init_rmf(self, name, hierarchies, rs=None, geometries=None, listofobjects=None): """ - This function initialize an RMF file + Initialize an RMF file @param name the name of the RMF file - @param hierarchies the hiearchies to be included (it is a list) + @param hierarchies the hierarchies to be included (it is a list) @param rs optional, the restraint sets (it is a list) @param geometries optional, the geometries (it is a list) @param listofobjects optional, the list of objects for the stat (it is a list) @@ -630,6 +629,7 @@ def test(self, name, listofobjects, tolerance=1e-5): else: print("%s from old objects (file %s) not in new objects" % (str(k), str(name)), file=sys.stderr) + flstat.close() return passed def get_environment_variables(self): diff --git a/modules/pmi/pyext/src/representation.py b/modules/pmi/pyext/src/representation.py deleted file mode 100644 index 53afd0638d..0000000000 --- a/modules/pmi/pyext/src/representation.py +++ /dev/null @@ -1,2640 +0,0 @@ -#!/usr/bin/env python - -"""@namespace IMP.pmi.representation - Representation of the system. -""" - -from __future__ import print_function -import IMP -import IMP.core -import IMP.algebra -import IMP.atom -import IMP.display -import IMP.isd -import IMP.pmi -import IMP.pmi.tools -import IMP.pmi.output -import IMP.rmf -import IMP.pmi.topology -import RMF -from math import pi, sqrt -from operator import itemgetter -import os -import warnings -import weakref - -class _Repo(object): - def __init__(self, doi, root): - self.doi = doi - self._root = root - - def get_fname(self, fname): - """Return a path relative to the top of the repository""" - return os.path.relpath(fname, self._root) - -class _StateInfo(object): - """Score state-specific information about this representation.""" - short_name = None - long_name = None - -@IMP.pmi.deprecated_pmi1_object("2.10", - "use IMP.pmi.topology.System instead") -class Representation(object): - # Authors: Peter Cimermancic, Riccardo Pellarin, Charles Greenberg - - ''' - Set up the representation of all proteins and nucleic acid macromolecules. - - Create the molecular hierarchies, representation, - sequence connectivity for the various involved proteins and - nucleic acid macromolecules: - - Create a protein, DNA or RNA, represent it as a set of connected balls of appropriate - radii and number of residues, PDB at given resolution(s), or ideal helices. - - How to use the SimplifiedModel class (typical use): - - see test/test_hierarchy_contruction.py - - examples: - - 1) Create a chain of helices and flexible parts - - c_1_119 =self.add_component_necklace("prot1",1,119,20) - c_120_131 =self.add_component_ideal_helix("prot1",resolutions=[1,10],resrange=(120,131)) - c_132_138 =self.add_component_beads("prot1",[(132,138)]) - c_139_156 =self.add_component_ideal_helix("prot1",resolutions=[1,10],resrange=(139,156)) - c_157_174 =self.add_component_beads("prot1",[(157,174)]) - c_175_182 =self.add_component_ideal_helix("prot1",resolutions=[1,10],resrange=(175,182)) - c_183_194 =self.add_component_beads("prot1",[(183,194)]) - c_195_216 =self.add_component_ideal_helix("prot1",resolutions=[1,10],resrange=(195,216)) - c_217_250 =self.add_component_beads("prot1",[(217,250)]) - - - self.set_rigid_body_from_hierarchies(c_120_131) - self.set_rigid_body_from_hierarchies(c_139_156) - self.set_rigid_body_from_hierarchies(c_175_182) - self.set_rigid_body_from_hierarchies(c_195_216) - - clist=[c_1_119,c_120_131,c_132_138,c_139_156,c_157_174,c_175_182,c_183_194,c_195_216, - c_217_250] - - self.set_chain_of_super_rigid_bodies(clist,2,3) - - self.set_super_rigid_bodies(["prot1"]) - - ''' - - def __init__(self, model, upperharmonic=True, disorderedlength=True): - """Constructor. - @param model the model - @param upperharmonic This flag uses either harmonic (False) - or upperharmonic (True) in the intra-pair - connectivity restraint. - @param disorderedlength This flag uses either disordered length - calculated for random coil peptides (True) or zero - surface-to-surface distance between beads (False) - as optimal distance for the sequence connectivity - restraint. - """ - - self.state = _StateInfo() - self._metadata = [] - self._protocol_output = [] - self._non_modeled_components = {} - - # this flag uses either harmonic (False) or upperharmonic (True) - # in the intra-pair connectivity restraint. Harmonic is used whe you want to - # remove the intra-ev term from energy calculations, e.g.: - # upperharmonic=False - # ip=self.get_connected_intra_pairs() - # ev.add_excluded_particle_pairs(ip) - - self.upperharmonic = upperharmonic - self.disorderedlength = disorderedlength - self.rigid_bodies = [] - self.fixed_rigid_bodies = [] - self.fixed_floppy_bodies = [] - self.floppy_bodies = [] - # self.super_rigid_bodies is a list of tuples. - # each tuple, corresponding to a particular super rigid body - # the tuple is (super_rigid_xyzs,super_rigid_rbs) - # where super_rigid_xyzs are the flexible xyz particles - # and super_rigid_rbs is the list of rigid bodies. - self.super_rigid_bodies = [] - self.rigid_body_symmetries = [] - self.output_level = "low" - self.label = "None" - - self.maxtrans_rb = 2.0 - self.maxrot_rb = 0.04 - self.maxtrans_srb = 2.0 - self.maxrot_srb = 0.2 - self.rigidbodiesarefixed = False - self.floppybodiesarefixed = False - self.maxtrans_fb = 3.0 - self.resolution = 10.0 - self.bblenght = 100.0 - self.kappa = 100.0 - self.model = model - - self.representation_is_modified = False - self.unmodeledregions_cr_dict = {} - self.sortedsegments_cr_dict = {} - self.prot = IMP.atom.Hierarchy.setup_particle(IMP.Particle(self.model)) - self.connected_intra_pairs = [] - self.hier_dict = {} - self.color_dict = {} - self.sequence_dict = {} - self.hier_geometry_pairs = {} - self.hier_db = IMP.pmi.tools.HierarchyDatabase() - # this dictionary stores the hierarchies by component name and representation type - # self.hier_representation[name][representation_type] - # where representation type is Res:X, Beads, Densities, Representation, - # etc... - self.hier_representation = {} - self.hier_resolution = {} - # reference structures is a dictionary that contains the coordinates of - # structures that are used to calculate the rmsd - self.reference_structures = {} - self.elements = {} - self.linker_restraints = IMP.RestraintSet(self.model, "linker_restraints") - self.linker_restraints.set_was_used(True) - self.linker_restraints_dict = {} - self.threetoone = {'ALA': 'A', 'ARG': 'R', 'ASN': 'N', 'ASP': 'D', - 'CYS': 'C', 'GLU': 'E', 'GLN': 'Q', 'GLY': 'G', - 'HIS': 'H', 'ILE': 'I', 'LEU': 'L', 'LYS': 'K', - 'MET': 'M', 'PHE': 'F', 'PRO': 'P', 'SER': 'S', - 'THR': 'T', 'TRP': 'W', 'TYR': 'Y', 'VAL': 'V', 'UNK': 'X'} - - self.onetothree = dict((v, k) for k, v in self.threetoone.items()) - - self.residuenamekey = IMP.StringKey("ResidueName") - - def add_metadata(self, m): - """Associate some metadata with this modeling. - @param m an instance of an ihm metadata class, such as - ihm.Software, ihm.Citation, or ihm.location.Repository. - """ - self._metadata.append(m) - - def add_protocol_output(self, p): - """Capture details of the modeling protocol. - @param p an instance of IMP.pmi.output.ProtocolOutput or a subclass. - """ - state = p._add_state(self) - self._protocol_output.append((p, state)) - p._each_metadata.append(self._metadata) - state.model = self.model - state.prot = self.prot - protocol_output = property(lambda self: - [x[0] for x in self._protocol_output]) - - def set_label(self, label): - self.label = label - - def create_component(self, name, color=0.0): - # Note that we set up the component as *both* a Chain and a Molecule. - # This is because old PMI1 code expects the top-level particle to be - # a Molecule, but we also need it to be a Chain to set the sequence. - # This looks a little odd but is a valid IMP atom::Hierarchy. - protein_h = IMP.atom.Chain.setup_particle(IMP.Particle(self.model), 'X') - IMP.atom.Molecule.setup_particle(protein_h) - protein_h.set_name(name) - self.hier_dict[name] = protein_h - self.hier_representation[name] = {} - self.hier_db.add_name(name) - self.prot.add_child(protein_h) - self.color_dict[name] = color - self.elements[name] = [] - for p, state in self._protocol_output: - p.create_component(state, name, True) - - def create_transformed_component(self, name, original, transform): - """Create a transformed view of a component. - This does not copy the coordinates or representation of the - component, and the transformed component is not visible to IMP, - but when the model is written to a ProtocolOutput, the transformed - component is added. This can be used to easily add symmetry-related - 'copies' of a component without copying the underlying - IMP objects.""" - for p, state in self._protocol_output: - p.create_transformed_component(state, name, original, transform) - - def create_non_modeled_component(self, name): - """Create a component that isn't used in the modeling. - No coordinates or other structural data for this component will - be read or written, but a primary sequence can be assigned. This - is useful if the input experimental data is of a system larger - than that modeled. Any references to these non-modeled components - can then be correctly resolved later.""" - self._non_modeled_components[name] = None - self.elements[name] = [] - for p, state in self._protocol_output: - p.create_component(state, name, False) - - def get_component_names(self): - return list(self.hier_dict.keys()) - - def add_component_sequence(self, name, filename, id=None, offs=None, - format="FASTA"): - ''' - Add the primary sequence for a single component. - - @param name Human-readable name of the component - @param filename Name of the FASTA file - @param id Identifier of the sequence in the FASTA file header - (if not provided, use `name` instead) - ''' - record_dict = IMP.pmi.topology.Sequences(filename) - if id is None: - id = name - if id not in record_dict: - raise KeyError("id %s not found in fasta file" % id) - length = len(record_dict[id]) - self.sequence_dict[name] = str(record_dict[id]) - # No Hierarchy for this component if it is non-modeled - if name not in self._non_modeled_components: - protein_h = self.hier_dict[name] - protein_h.set_sequence(self.sequence_dict[name]) - if offs is not None: - offs_str="-"*offs - self.sequence_dict[name]=offs_str+self.sequence_dict[name] - - self.elements[name].append((length, length, " ", "end")) - for p, state in self._protocol_output: - p.add_component_sequence(state, name, self.sequence_dict[name]) - - def autobuild_model(self, name, pdbname, chain, - resolutions=None, resrange=None, - missingbeadsize=20, - color=None, pdbresrange=None, offset=0, - show=False, isnucleicacid=False, - attachbeads=False): - - self.representation_is_modified = True - outhiers = [] - - if color is None: - color = self.color_dict[name] - else: - self.color_dict[name] = color - - if resolutions is None: - resolutions = [1] - print("autobuild_model: constructing %s from pdb %s and chain %s" % (name, pdbname, str(chain))) - - # get the initial and end residues of the pdb - t = IMP.atom.read_pdb(pdbname, self.model, - IMP.atom.AndPDBSelector(IMP.atom.ChainPDBSelector(chain), IMP.atom.CAlphaPDBSelector())) - - # find start and end indexes - - start = IMP.atom.Residue( - t.get_children()[0].get_children()[0]).get_index() - end = IMP.atom.Residue( - t.get_children()[0].get_children()[-1]).get_index() - - # check if resrange was defined, otherwise - # use the sequence, or the pdb resrange - - if resrange is None: - if name in self.sequence_dict: - resrange = (1, len(self.sequence_dict[name])) - else: - resrange = (start + offset, end + offset) - else: - if resrange[1] in (-1, 'END'): - resrange = (resrange[0],end) - start = resrange[0] - offset - end = resrange[1] - offset - - gaps = IMP.pmi.tools.get_residue_gaps_in_hierarchy( - t, - resrange[0], - resrange[1]) - - xyznter = IMP.pmi.tools.get_closest_residue_position( - t, - resrange[0], - terminus="N") - xyzcter = IMP.pmi.tools.get_closest_residue_position( - t, - resrange[1], - terminus="C") - # Done with the PDB - IMP.atom.destroy(t) - - # construct pdb fragments and intervening beads - for n, g in enumerate(gaps): - first = g[0] - last = g[1] - if g[2] == "cont": - print("autobuild_model: constructing fragment %s from pdb" % (str((first, last)))) - outhiers += self.add_component_pdb(name, pdbname, - chain, resolutions=resolutions, - color=color, cacenters=True, - resrange=(first, last), - offset=offset, isnucleicacid=isnucleicacid) - elif g[2] == "gap" and n > 0: - print("autobuild_model: constructing fragment %s as a bead" % (str((first, last)))) - parts = self.hier_db.get_particles_at_closest_resolution(name, - first + offset - 1, - 1) - xyz = IMP.core.XYZ(parts[0]).get_coordinates() - outhiers += self.add_component_necklace(name, - first+offset, last+offset, missingbeadsize, incoord=xyz) - - elif g[2] == "gap" and n == 0: - # add pre-beads - print("autobuild_model: constructing fragment %s as a bead" % (str((first, last)))) - outhiers += self.add_component_necklace(name, - first+offset, last+offset, missingbeadsize, incoord=xyznter) - - return outhiers - - def add_component_pdb(self, name, pdbname, chain, resolutions, color=None, - resrange=None, offset=0, cacenters=True, show=False, - isnucleicacid=False, readnonwateratoms=False, - read_ca_cb_only=False): - ''' - Add a component that has an associated 3D structure in a PDB file. - - Reads the PDB, and constructs the fragments corresponding to contiguous - sequence stretches. - - @return a list of hierarchies. - - @param name (string) the name of the component - @param pdbname (string) the name of the PDB file - @param chain (string or integer) can be either a string (eg, "A") - or an integer (eg, 0 or 1) in case you want - to get the corresponding chain number in the PDB. - @param resolutions (integers) a list of integers that corresponds - to the resolutions that have to be generated - @param color (float from 0 to 1) the color applied to the - hierarchies generated - @param resrange (tuple of integers): the residue range to extract - from the PDB. It is a tuple (beg,end). If not specified, - all residues belonging to the specified chain are read. - @param offset (integer) specifies the residue index offset to be - applied when reading the PDB (the FASTA sequence is - assumed to start from residue 1, so use this parameter - if the PDB file does not start at residue 1) - @param cacenters (boolean) if True generates resolution=1 beads - centered on C-alpha atoms. - @param show (boolean) print out the molecular hierarchy at the end. - @param isnucleicacid (boolean) use True if you're reading a PDB - with nucleic acids. - @param readnonwateratoms (boolean) if True fixes some pathological PDB. - @param read_ca_cb_only (boolean) if True, only reads CA/CB - ''' - - self.representation_is_modified = True - if color is None: - # if the color is not passed, then get the stored color - color = self.color_dict[name] - protein_h = self.hier_dict[name] - outhiers = [] - - # determine selector - sel = IMP.atom.NonWaterNonHydrogenPDBSelector() - if read_ca_cb_only: - cacbsel = IMP.atom.OrPDBSelector( - IMP.atom.CAlphaPDBSelector(), - IMP.atom.CBetaPDBSelector()) - sel = IMP.atom.AndPDBSelector(cacbsel, sel) - if isinstance(chain, str): - sel = IMP.atom.AndPDBSelector( - IMP.atom.ChainPDBSelector(chain), - sel) - t = IMP.atom.read_pdb(pdbname, self.model, sel) - - # get the first and last residue - start = IMP.atom.Residue( - t.get_children()[0].get_children()[0]).get_index() - end = IMP.atom.Residue( - t.get_children()[0].get_children()[-1]).get_index() - c = IMP.atom.Chain(IMP.atom.get_by_type(t, IMP.atom.CHAIN_TYPE)[0]) - else: - t = IMP.atom.read_pdb(pdbname, self.model, sel) - c = IMP.atom.Chain( - IMP.atom.get_by_type(t, IMP.atom.CHAIN_TYPE)[chain]) - - # get the first and last residue - start = IMP.atom.Residue(c.get_children()[0]).get_index() - end = IMP.atom.Residue(c.get_children()[-1]).get_index() - chain = c.get_id() - - if not resrange is None: - if resrange[0] > start: - start = resrange[0] - if resrange[1] < end: - end = resrange[1] - - if not isnucleicacid: - # do what you have to do for proteins - sel = IMP.atom.Selection( - c, - residue_indexes=list(range( - start, - end + 1)), - atom_type=IMP.atom.AT_CA) - - else: - # do what you have to do for nucleic-acids - # to work, nucleic acids should not be indicated as HETATM in the pdb - sel = IMP.atom.Selection( - c, - residue_indexes=list(range( - start, - end + 1)), - atom_type=IMP.atom.AT_P) - - ps = sel.get_selected_particles() - if len(ps) == 0: - raise ValueError("%s no residue found in pdb %s chain %s that overlaps with the queried stretch %s-%s" \ - % (name, pdbname, str(chain), str(resrange[0]), str(resrange[1]))) - c0 = IMP.atom.Chain.setup_particle(IMP.Particle(self.model), "X") - - for p in ps: - par = IMP.atom.Atom(p).get_parent() - ri = IMP.atom.Residue(par).get_index() - # Move residue from original PDB hierarchy to new chain c0 - IMP.atom.Residue(par).set_index(ri + offset) - if par.get_parent() != c0: - par.get_parent().remove_child(par) - c0.add_child(par) - start = start + offset - end = end + offset - - self.elements[name].append( - (start, end, pdbname.split("/")[-1] + ":" + chain, "pdb")) - - hiers = self.coarse_hierarchy(name, start, end, - resolutions, isnucleicacid, c0, protein_h, "pdb", color) - self._copy_pdb_provenance(t, hiers[0], offset) - outhiers += hiers - for p, state in self._protocol_output: - p.add_pdb_element(state, name, start, end, offset, pdbname, chain, - hiers[0]) - - if show: - IMP.atom.show_molecular_hierarchy(protein_h) - - # We cannot simply destroy(c0) since it might not be a well-behaved - # hierarchy; in some cases it could contain a given residue more than - # once (this is surely a bug but we need to keep this behavior for - # backwards compatibility). - residues = {} - for p in ps: - par = IMP.atom.Atom(p).get_parent() - residues[par] = None - for r in residues.keys(): - IMP.atom.destroy(r) - self.model.remove_particle(c0) - - IMP.atom.destroy(t) - - return outhiers - - def _copy_pdb_provenance(self, pdb, h, offset): - """Copy the provenance information from the PDB to our hierarchy""" - c = IMP.atom.Chain(IMP.atom.get_by_type(pdb, IMP.atom.CHAIN_TYPE)[0]) - prov = IMP.core.Provenanced(c).get_provenance() - newprov = IMP.core.create_clone(prov) - IMP.core.Provenanced.setup_particle(h, newprov) - assert(IMP.core.StructureProvenance.get_is_setup(newprov)) - IMP.core.StructureProvenance(newprov).set_residue_offset(offset) - - def add_component_ideal_helix( - self, - name, - resolutions, - resrange, - color=None, - show=False): - - self.representation_is_modified = True - from math import pi, cos, sin - - protein_h = self.hier_dict[name] - outhiers = [] - if color is None: - color = self.color_dict[name] - - start = resrange[0] - end = resrange[1] - self.elements[name].append((start, end, " ", "helix")) - c0 = IMP.atom.Chain.setup_particle(IMP.Particle(self.model), "X") - for n, res in enumerate(range(start, end + 1)): - if name in self.sequence_dict: - try: - rtstr = self.onetothree[ - self.sequence_dict[name][res-1]] - except: - rtstr = "UNK" - rt = IMP.atom.ResidueType(rtstr) - else: - rt = IMP.atom.ResidueType("ALA") - - # get the residue volume - try: - vol = IMP.atom.get_volume_from_residue_type(rt) - # mass=IMP.atom.get_mass_from_residue_type(rt) - except IMP.ValueException: - vol = IMP.atom.get_volume_from_residue_type( - IMP.atom.ResidueType("ALA")) - # mass=IMP.atom.get_mass_from_residue_type(IMP.atom.ResidueType("ALA")) - radius = IMP.algebra.get_ball_radius_from_volume_3d(vol) - - r = IMP.atom.Residue.setup_particle(IMP.Particle(self.model), rt, res) - p = IMP.Particle(self.model) - d = IMP.core.XYZR.setup_particle(p) - x = 2.3 * cos(n * 2 * pi / 3.6) - y = 2.3 * sin(n * 2 * pi / 3.6) - z = 6.2 / 3.6 / 2 * n * 2 * pi / 3.6 - d.set_coordinates(IMP.algebra.Vector3D(x, y, z)) - d.set_radius(radius) - # print d - a = IMP.atom.Atom.setup_particle(p, IMP.atom.AT_CA) - r.add_child(a) - c0.add_child(r) - - outhiers += self.coarse_hierarchy(name, start, end, - resolutions, False, c0, protein_h, "helix", color) - - if show: - IMP.atom.show_molecular_hierarchy(protein_h) - - IMP.atom.destroy(c0) - return outhiers - - def add_component_beads(self, name, ds, colors=None, incoord=None): - """ add beads to the representation - @param name the component name - @param ds a list of tuples corresponding to the residue ranges - of the beads - @param colors a list of colors associated to the beads - @param incoord the coordinate tuple corresponding to the position - of the beads - """ - - from math import pi - self.representation_is_modified = True - - protein_h = self.hier_dict[name] - outhiers = [] - if colors is None: - colors = [self.color_dict[name]] - - - for n, dss in enumerate(ds): - ds_frag = (dss[0], dss[1]) - self.elements[name].append((dss[0], dss[1], " ", "bead")) - prt = IMP.Particle(self.model) - if ds_frag[0] == ds_frag[1]: - # if the bead represent a single residue - if name in self.sequence_dict: - try: - rtstr = self.onetothree[ - self.sequence_dict[name][ds_frag[0]-1]] - except: - rtstr = "UNK" - rt = IMP.atom.ResidueType(rtstr) - else: - rt = IMP.atom.ResidueType("ALA") - h = IMP.atom.Residue.setup_particle(prt, rt, ds_frag[0]) - h.set_name(name + '_%i_bead' % (ds_frag[0])) - prt.set_name(name + '_%i_bead' % (ds_frag[0])) - resolution = 1 - else: - h = IMP.atom.Fragment.setup_particle(prt) - h.set_name(name + '_%i-%i_bead' % (ds_frag[0], ds_frag[1])) - prt.set_name(name + '_%i-%i_bead' % (ds_frag[0], ds_frag[1])) - h.set_residue_indexes(list(range(ds_frag[0], ds_frag[1] + 1))) - resolution = len(h.get_residue_indexes()) - if "Beads" not in self.hier_representation[name]: - root = IMP.atom.Hierarchy.setup_particle(IMP.Particle(self.model)) - root.set_name("Beads") - self.hier_representation[name]["Beads"] = root - protein_h.add_child(root) - self.hier_representation[name]["Beads"].add_child(h) - - for kk in range(ds_frag[0], ds_frag[1] + 1): - self.hier_db.add_particles(name, kk, resolution, [h]) - - try: - clr = IMP.display.get_rgb_color(colors[n]) - except: - clr = IMP.display.get_rgb_color(colors[0]) - - IMP.display.Colored.setup_particle(prt, clr) - - # decorate particles according to their resolution - IMP.pmi.Resolution.setup_particle(prt, resolution) - - IMP.core.XYZR.setup_particle(prt) - ptem = IMP.core.XYZR(prt) - mass = IMP.atom.get_mass_from_number_of_residues(resolution) - if resolution == 1: - try: - vol = IMP.atom.get_volume_from_residue_type(rt) - except IMP.ValueException: - vol = IMP.atom.get_volume_from_residue_type( - IMP.atom.ResidueType("ALA")) - radius = IMP.algebra.get_ball_radius_from_volume_3d(vol) - IMP.atom.Mass.setup_particle(prt, mass) - ptem.set_radius(radius) - else: - volume = IMP.atom.get_volume_from_mass(mass) - radius = 0.8 * (3.0 / 4.0 / pi * volume) ** (1.0 / 3.0) - IMP.atom.Mass.setup_particle(prt, mass) - ptem.set_radius(radius) - try: - if not tuple(incoord) is None: - ptem.set_coordinates(incoord) - except TypeError: - pass - IMP.pmi.Uncertainty.setup_particle(prt, radius) - IMP.pmi.Symmetric.setup_particle(prt, 0) - self.floppy_bodies.append(prt) - IMP.core.XYZ(prt).set_coordinates_are_optimized(True) - outhiers += [h] - - for p, state in self._protocol_output: - p.add_bead_element(state, name, ds[0][0], ds[-1][1], len(ds), - outhiers[0]) - - return outhiers - - def add_component_necklace(self, name, begin, end, length, color=None, - incoord=None): - ''' - Generates a string of beads with given length. - ''' - self.representation_is_modified = True - outhiers = [] - if color is None: - colors=None - else: - colors=[color] - for chunk in IMP.pmi.tools.list_chunks_iterator(range(begin, end + 1), length): - outhiers += self.add_component_beads(name, - [(chunk[0], chunk[-1])], colors=colors,incoord=incoord) - return outhiers - - def add_component_density( - self, name, hierarchies=None, selection_tuples=None, - particles=None, - resolution=0.0, num_components=10, - inputfile=None, outputfile=None, - outputmap=None, - kernel_type=None, - covariance_type='full', voxel_size=1.0, - out_hier_name='', - sampled_points=1000000, num_iter=100, - simulation_res=1.0, - multiply_by_total_mass=True, - transform=None, - intermediate_map_fn=None, - density_ps_to_copy=None, - use_precomputed_gaussians=False): - ''' - Sets up a Gaussian Mixture Model for this component. - Can specify input GMM file or it will be computed. - @param name component name - @param hierarchies set up GMM for some hierarchies - @param selection_tuples (list of tuples) example (first_residue,last_residue,component_name) - @param particles set up GMM for particles directly - @param resolution usual PMI resolution for selecting particles from the hierarchies - @param inputfile read the GMM from this file - @param outputfile fit and write the GMM to this file (text) - @param outputmap after fitting, create GMM density file (mrc) - @param kernel_type for creating the intermediate density (points are sampled to make GMM). Options are IMP.em.GAUSSIAN, IMP.em.SPHERE, and IMP.em.BINARIZED_SPHERE - @param covariance_type for fitting the GMM. options are 'full', 'diagonal' and 'spherical' - @param voxel_size for creating the intermediate density map and output map. - lower number increases accuracy but also rasterizing time grows - @param out_hier_name name of the output density hierarchy - @param sampled_points number of points to sample. more will increase accuracy and fitting time - @param num_iter num GMM iterations. more will increase accuracy and fitting time - @param multiply_by_total_mass multiply the weights of the GMM by this value (only works on creation!) - @param transform for input file only, apply a transformation (eg for multiple copies same GMM) - @param intermediate_map_fn for debugging, this will write the intermediate (simulated) map - @param density_ps_to_copy in case you already created the appropriate GMM (eg, for beads) - @param use_precomputed_gaussians Set this flag and pass fragments - will use roughly spherical Gaussian setup - ''' - import numpy as np - import sys - import IMP.em - import IMP.isd.gmm_tools - - # prepare output - self.representation_is_modified = True - out_hier = [] - protein_h = self.hier_dict[name] - if "Densities" not in self.hier_representation[name]: - root = IMP.atom.Hierarchy.setup_particle(IMP.Particle(self.model)) - root.set_name("Densities") - self.hier_representation[name]["Densities"] = root - protein_h.add_child(root) - - # gather passed particles - fragment_particles = [] - if not particles is None: - fragment_particles += particles - if not hierarchies is None: - fragment_particles += IMP.pmi.tools.select( - self, resolution=resolution, - hierarchies=hierarchies) - if not selection_tuples is None: - for st in selection_tuples: - fragment_particles += IMP.pmi.tools.select_by_tuple( - self, tupleselection=st, - resolution=resolution, - name_is_ambiguous=False) - - # compute or read gaussians - density_particles = [] - if inputfile: - IMP.isd.gmm_tools.decorate_gmm_from_text( - inputfile, density_particles, - self.model, transform) - elif density_ps_to_copy: - for ip in density_ps_to_copy: - p = IMP.Particle(self.model) - shape = IMP.core.Gaussian(ip).get_gaussian() - mass = IMP.atom.Mass(ip).get_mass() - IMP.core.Gaussian.setup_particle(p, shape) - IMP.atom.Mass.setup_particle(p, mass) - density_particles.append(p) - elif use_precomputed_gaussians: - if len(fragment_particles) == 0: - print("add_component_density: no particle was selected") - return out_hier - for p in fragment_particles: - if not (IMP.atom.Fragment.get_is_setup(self.model,p.get_particle_index()) and - IMP.core.XYZ.get_is_setup(self.model,p.get_particle_index())): - raise Exception("The particles you selected must be Fragments and XYZs") - nres=len(IMP.atom.Fragment(self.model,p.get_particle_index()).get_residue_indexes()) - pos=IMP.core.XYZ(self.model,p.get_particle_index()).get_coordinates() - density_particles=[] - try: - IMP.isd.get_data_path("beads/bead_%i.txt"%nres) - except: - raise Exception("We haven't created a bead file for",nres,"residues") - transform = IMP.algebra.Transformation3D(pos) - IMP.isd.gmm_tools.decorate_gmm_from_text( - IMP.isd.get_data_path("beads/bead_%i.txt"%nres), density_particles, - self.model, transform) - else: - #compute the gaussians here - if len(fragment_particles) == 0: - print("add_component_density: no particle was selected") - return out_hier - - density_particles = IMP.isd.gmm_tools.sample_and_fit_to_particles( - self.model, - fragment_particles, - num_components, - sampled_points, - simulation_res, - voxel_size, - num_iter, - covariance_type, - multiply_by_total_mass, - outputmap, - outputfile) - - # prepare output hierarchy - s0 = IMP.atom.Fragment.setup_particle(IMP.Particle(self.model)) - s0.set_name(out_hier_name) - self.hier_representation[name]["Densities"].add_child(s0) - out_hier.append(s0) - for nps, p in enumerate(density_particles): - s0.add_child(p) - p.set_name(s0.get_name() + '_gaussian_%i' % nps) - return out_hier - - def get_component_density(self, name): - return self.hier_representation[name]["Densities"] - - def add_all_atom_densities(self, name, hierarchies=None, - selection_tuples=None, - particles=None, - resolution=0, - output_map=None, - voxel_size=1.0): - '''Decorates all specified particles as Gaussians directly. - @param name component name - @param hierarchies set up GMM for some hierarchies - @param selection_tuples (list of tuples) example (first_residue,last_residue,component_name) - @param particles set up GMM for particles directly - @param resolution usual PMI resolution for selecting particles from the hierarchies - ''' - - import IMP.em - import numpy as np - import sys - from math import sqrt - self.representation_is_modified = True - - if particles is None: - fragment_particles = [] - else: - fragment_particles = particles - - if not hierarchies is None: - fragment_particles += IMP.pmi.tools.select( - self, resolution=resolution, - hierarchies=hierarchies) - - if not selection_tuples is None: - for st in selection_tuples: - fragment_particles += IMP.pmi.tools.select_by_tuple( - self, tupleselection=st, - resolution=resolution, - name_is_ambiguous=False) - - if len(fragment_particles) == 0: - print("add all atom densities: no particle was selected") - return - - # create a spherical gaussian for each particle based on atom type - print('setting up all atom gaussians num_particles',len(fragment_particles)) - for n,p in enumerate(fragment_particles): - if IMP.core.Gaussian.get_is_setup(p): continue - center=IMP.core.XYZ(p).get_coordinates() - rad=IMP.core.XYZR(p).get_radius() - mass=IMP.atom.Mass(p).get_mass() - trans=IMP.algebra.Transformation3D(IMP.algebra.get_identity_rotation_3d(),center) - shape=IMP.algebra.Gaussian3D(IMP.algebra.ReferenceFrame3D(trans),[rad]*3) - IMP.core.Gaussian.setup_particle(p,shape) - print('setting up particle',p.get_name(), " as individual gaussian particle") - - if not output_map is None: - print('writing map to', output_map) - IMP.isd.gmm_tools.write_gmm_to_map( - fragment_particles, - output_map, - voxel_size) - - def add_component_hierarchy_clone(self, name, hierarchy): - ''' - Make a copy of a hierarchy and append it to a component. - ''' - outhier = [] - self.representation_is_modified = True - protein_h = self.hier_dict[name] - hierclone = IMP.atom.create_clone(hierarchy) - hierclone.set_name(hierclone.get_name() + "_clone") - protein_h.add_child(hierclone) - outhier.append(hierclone) - - psmain = IMP.atom.get_leaves(hierarchy) - psclone = IMP.atom.get_leaves(hierclone) - - # copying attributes - for n, pmain in enumerate(psmain): - pclone = psclone[n] - if IMP.pmi.Resolution.get_is_setup(pmain): - resolution = IMP.pmi.Resolution(pmain).get_resolution() - IMP.pmi.Resolution.setup_particle(pclone, resolution) - for kk in IMP.pmi.tools.get_residue_indexes(pclone): - self.hier_db.add_particles( - name, - kk, - IMP.pmi.Resolution(pclone).get_resolution(), - [pclone]) - - if IMP.pmi.Uncertainty.get_is_setup(pmain): - uncertainty = IMP.pmi.Uncertainty(pmain).get_uncertainty() - IMP.pmi.Uncertainty.setup_particle(pclone, uncertainty) - - if IMP.pmi.Symmetric.get_is_setup(pmain): - symmetric = IMP.pmi.Symmetric(pmain).get_symmetric() - IMP.pmi.Symmetric.setup_particle(pclone, symmetric) - - return outhier - - def dump_particle_descriptors(self): - import numpy - import pickle - import IMP.isd - import IMP.isd.gmm_tools - - particles_attributes={} - floppy_body_attributes={} - gaussians=[] - for h in IMP.atom.get_leaves(self.prot): - leaf=h - name=h.get_name() - hroot=self.prot - hparent=h.get_parent() - while hparent != hroot: - hparent=h.get_parent() - name+="|"+hparent.get_name() - h=hparent - particles_attributes[name]={"COORDINATES":numpy.array(IMP.core.XYZR(leaf.get_particle()).get_coordinates()), - "RADIUS":IMP.core.XYZR(leaf.get_particle()).get_radius(), - "MASS":IMP.atom.Mass(leaf.get_particle()).get_mass()} - if IMP.core.Gaussian.get_is_setup(leaf.get_particle()): - gaussians.append(IMP.core.Gaussian(leaf.get_particle())) - - rigid_body_attributes={} - for rb in self.rigid_bodies: - name=rb.get_name() - rf=rb.get_reference_frame() - t=rf.get_transformation_to() - trans=t.get_translation() - rot=t.get_rotation() - rigid_body_attributes[name]={"TRANSLATION":numpy.array(trans), - "ROTATION":numpy.array(rot.get_quaternion()), - "COORDINATES_NONRIGID_MEMBER":{}, - "COORDINATES_RIGID_MEMBER":{}} - for mi in rb.get_member_indexes(): - rm=self.model.get_particle(mi) - if IMP.core.NonRigidMember.get_is_setup(rm): - name_part=rm.get_name() - xyz=[self.model.get_attribute(fk, rm) for fk in [IMP.FloatKey(4), IMP.FloatKey(5), IMP.FloatKey(6)]] - rigid_body_attributes[name]["COORDINATES_NONRIGID_MEMBER"][name_part]=numpy.array(xyz) - else: - name_part=rm.get_name() - xyz=IMP.core.XYZ(rm).get_coordinates() - rigid_body_attributes[name]["COORDINATES_RIGID_MEMBER"][name_part]=numpy.array(xyz) - - - IMP.isd.gmm_tools.write_gmm_to_text(gaussians,"model_gmm.txt") - pickle.dump(particles_attributes, - open("particles_attributes.pkl", "wb")) - pickle.dump(rigid_body_attributes, - open("rigid_body_attributes.pkl", "wb")) - - - - def load_particle_descriptors(self): - import numpy - import pickle - import IMP.isd - import IMP.isd.gmm_tools - - particles_attributes = pickle.load(open("particles_attributes.pkl", - "rb")) - rigid_body_attributes = pickle.load(open("rigid_body_attributes.pkl", - "rb")) - - particles=[] - hierarchies=[] - gaussians=[] - for h in IMP.atom.get_leaves(self.prot): - leaf=h - name=h.get_name() - hroot=self.prot - hparent=h.get_parent() - while hparent != hroot: - hparent=h.get_parent() - name+="|"+hparent.get_name() - h=hparent - - xyzr=IMP.core.XYZR(leaf.get_particle()) - xyzr.set_coordinates(particles_attributes[name]["COORDINATES"]) - #xyzr.set_radius(particles_attributes[name]["RADIUS"]) - #IMP.atom.Mass(leaf.get_particle()).set_mass(particles_attributes[name]["MASS"]) - if IMP.core.Gaussian.get_is_setup(leaf.get_particle()): - gaussians.append(IMP.core.Gaussian(leaf.get_particle())) - - for rb in self.rigid_bodies: - name=rb.get_name() - trans=rigid_body_attributes[name]["TRANSLATION"] - rot=rigid_body_attributes[name]["ROTATION"] - t=IMP.algebra.Transformation3D(IMP.algebra.Rotation3D(rot),trans) - rf=IMP.algebra.ReferenceFrame3D(t) - rb.set_reference_frame(rf) - coor_nrm_ref=rigid_body_attributes[name]["COORDINATES_NONRIGID_MEMBER"] - coor_rm_ref_dict=rigid_body_attributes[name]["COORDINATES_RIGID_MEMBER"] - coor_rm_model=[] - coor_rm_ref=[] - for mi in rb.get_member_indexes(): - rm=self.model.get_particle(mi) - if IMP.core.NonRigidMember.get_is_setup(rm): - name_part=rm.get_name() - xyz=coor_nrm_ref[name_part] - for n,fk in enumerate([IMP.FloatKey(4), IMP.FloatKey(5), IMP.FloatKey(6)]): - self.model.set_attribute(fk, rm,xyz[n]) - else: - name_part=rm.get_name() - coor_rm_ref.append(IMP.algebra.Vector3D(coor_rm_ref_dict[name_part])) - coor_rm_model.append(IMP.core.XYZ(rm).get_coordinates()) - if len(coor_rm_model)==0: continue - t=IMP.algebra.get_transformation_aligning_first_to_second(coor_rm_model,coor_rm_ref) - IMP.core.transform(rb,t) - - IMP.isd.gmm_tools.decorate_gmm_from_text("model_gmm.txt",gaussians,self.model) - - def _compare_rmf_repr_names(self, rmfname, reprname, component_name): - """Print a warning if particle names in RMF and model don't match""" - def match_any_suffix(): - # Handle common mismatches like 743 != Nup85_743_pdb - suffixes = ["pdb", "bead_floppy_body_rigid_body_member_floppy_body", - "bead_floppy_body_rigid_body_member", - "bead_floppy_body"] - for s in suffixes: - if "%s_%s_%s" % (component_name, rmfname, s) == reprname: - return True - if rmfname != reprname and not match_any_suffix(): - warnings.warn( - "set_coordinates_from_rmf: rmf particle and " - "representation particle names don't match %s %s" - % (rmfname, reprname), IMP.pmi.StructureWarning) - - def set_coordinates_from_rmf(self, component_name, rmf_file_name, - rmf_frame_number, - rmf_component_name=None, - check_number_particles=True, - representation_name_to_rmf_name_map=None, - state_number=0, - skip_gaussian_in_rmf=False, - skip_gaussian_in_representation=False, - save_file=False, - force_rigid_update=False): - '''Read and replace coordinates from an RMF file. - Replace the coordinates of particles with the same name. - It assumes that the RMF and the representation have the particles - in the same order. - @param component_name Component name - @param rmf_component_name Name of the component in the RMF file - (if not specified, use `component_name`) - @param representation_name_to_rmf_name_map a dictionary that map - the original rmf particle name to the recipient particle component name - @param save_file: save a file with the names of particles of the component - @param force_rigid_update: update the coordinates of rigid bodies - (normally this should be called before rigid bodies are set up) - ''' - import IMP.pmi.analysis - - tempm = IMP.Model() - prots = IMP.pmi.analysis.get_hiers_from_rmf( - tempm, - rmf_frame_number, - rmf_file_name) - - if not prots: - raise ValueError("cannot read hierarchy from rmf") - - prot=prots[0] - # Make sure coordinates of rigid body members in the RMF are correct - if force_rigid_update: - tempm.update() - - # if len(self.rigid_bodies)!=0: - # print "set_coordinates_from_rmf: cannot proceed if rigid bodies were initialized. Use the function before defining the rigid bodies" - # exit() - - allpsrmf = IMP.atom.get_leaves(prot) - psrmf = [] - for p in allpsrmf: - (protname, is_a_bead) = IMP.pmi.tools.get_prot_name_from_particle( - p, self.hier_dict.keys()) - if (protname is None) and (rmf_component_name is not None): - (protname, is_a_bead) = IMP.pmi.tools.get_prot_name_from_particle( - p, rmf_component_name) - if (skip_gaussian_in_rmf): - if (IMP.core.Gaussian.get_is_setup(p)) and not (IMP.atom.Fragment.get_is_setup(p) or IMP.atom.Residue.get_is_setup(p)): - continue - if (rmf_component_name is not None) and (protname == rmf_component_name): - psrmf.append(p) - elif (rmf_component_name is None) and (protname == component_name): - psrmf.append(p) - - psrepr = IMP.atom.get_leaves(self.hier_dict[component_name]) - if (skip_gaussian_in_representation): - allpsrepr = psrepr - psrepr = [] - for p in allpsrepr: - #(protname, is_a_bead) = IMP.pmi.tools.get_prot_name_from_particle( - # p, self.hier_dict.keys()) - if (IMP.core.Gaussian.get_is_setup(p)) and not (IMP.atom.Fragment.get_is_setup(p) or IMP.atom.Residue.get_is_setup(p)): - continue - psrepr.append(p) - - import itertools - reprnames=[p.get_name() for p in psrepr] - rmfnames=[p.get_name() for p in psrmf] - - if save_file: - fl=open(component_name+".txt","w") - for i in itertools.izip_longest(reprnames,rmfnames): fl.write(str(i[0])+","+str(i[1])+"\n") - - - if check_number_particles and not representation_name_to_rmf_name_map: - if len(psrmf) != len(psrepr): - fl=open(component_name+".txt","w") - for i in itertools.izip_longest(reprnames,rmfnames): fl.write(str(i[0])+","+str(i[1])+"\n") - raise ValueError("%s cannot proceed the rmf and the representation don't have the same number of particles; " - "particles in rmf: %s particles in the representation: %s" % (str(component_name), str(len(psrmf)), str(len(psrepr)))) - - - if not representation_name_to_rmf_name_map: - for n, prmf in enumerate(psrmf): - - prmfname = prmf.get_name() - preprname = psrepr[n].get_name() - if force_rigid_update: - if IMP.core.RigidBody.get_is_setup(psrepr[n]) \ - and not IMP.core.RigidBodyMember.get_is_setup(psrepr[n]): - continue - else: - if IMP.core.RigidBodyMember.get_is_setup(psrepr[n]): - raise ValueError("component %s cannot proceed if rigid bodies were initialized. Use the function before defining the rigid bodies" % component_name) - - self._compare_rmf_repr_names(prmfname, preprname, - component_name) - if IMP.core.XYZ.get_is_setup(prmf) and IMP.core.XYZ.get_is_setup(psrepr[n]): - xyz = IMP.core.XYZ(prmf).get_coordinates() - IMP.core.XYZ(psrepr[n]).set_coordinates(xyz) - if IMP.core.RigidBodyMember.get_is_setup(psrepr[n]): - # Set rigid body so that coordinates are preserved - # on future model updates - rbm = IMP.core.RigidBodyMember(psrepr[n]) - rbm.set_internal_coordinates(xyz) - tr = IMP.algebra.ReferenceFrame3D() - rb = rbm.get_rigid_body() - if IMP.core.RigidBodyMember.get_is_setup(rb): - raise ValueError("Cannot handle nested " - "rigid bodies yet") - rb.set_reference_frame_lazy(tr) - else: - warnings.warn( - "set_coordinates_from_rmf: particles are not XYZ " - "decorated %s %s " - % (str(IMP.core.XYZ.get_is_setup(prmf)), - str(IMP.core.XYZ.get_is_setup(psrepr[n]))), - IMP.pmi.StructureWarning) - - if IMP.core.Gaussian.get_is_setup(prmf) and IMP.core.Gaussian.get_is_setup(psrepr[n]): - gprmf=IMP.core.Gaussian(prmf) - grepr=IMP.core.Gaussian(psrepr[n]) - g=gprmf.get_gaussian() - grepr.set_gaussian(g) - - else: - repr_name_particle_map={} - rmf_name_particle_map={} - for p in psrmf: - rmf_name_particle_map[p.get_name()]=p - #for p in psrepr: - # repr_name_particle_map[p.get_name()]=p - - for prepr in psrepr: - try: - prmf=rmf_name_particle_map[representation_name_to_rmf_name_map[prepr.get_name()]] - except KeyError: - warnings.warn( - "set_coordinates_from_rmf: missing particle name in " - "representation_name_to_rmf_name_map, skipping...", - IMP.pmi.StructureWarning) - continue - xyz = IMP.core.XYZ(prmf).get_coordinates() - IMP.core.XYZ(prepr).set_coordinates(xyz) - - - - def check_root(self, name, protein_h, resolution): - ''' - If the root hierarchy does not exist, construct it. - ''' - if "Res:" + str(int(resolution)) not in self.hier_representation[name]: - root = IMP.atom.Hierarchy.setup_particle(IMP.Particle(self.model)) - root.set_name(name + "_Res:" + str(int(resolution))) - self.hier_representation[name][ - "Res:" + str(int(resolution))] = root - protein_h.add_child(root) - - def coarse_hierarchy(self, name, start, end, resolutions, isnucleicacid, - input_hierarchy, protein_h, type, color): - ''' - Generate all needed coarse grained layers. - - @param name name of the protein - @param resolutions list of resolutions - @param protein_h root hierarchy - @param input_hierarchy hierarchy to coarse grain - @param type a string, typically "pdb" or "helix" - ''' - outhiers = [] - - if (1 in resolutions) or (0 in resolutions): - # in that case create residues and append atoms - - if 1 in resolutions: - self.check_root(name, protein_h, 1) - s1 = IMP.atom.Fragment.setup_particle(IMP.Particle(self.model)) - s1.set_name('%s_%i-%i_%s' % (name, start, end, type)) - # s1.set_residue_indexes(range(start,end+1)) - self.hier_representation[name]["Res:1"].add_child(s1) - outhiers += [s1] - if 0 in resolutions: - self.check_root(name, protein_h, 0) - s0 = IMP.atom.Fragment.setup_particle(IMP.Particle(self.model)) - s0.set_name('%s_%i-%i_%s' % (name, start, end, type)) - # s0.set_residue_indexes(range(start,end+1)) - self.hier_representation[name]["Res:0"].add_child(s0) - outhiers += [s0] - - if not isnucleicacid: - sel = IMP.atom.Selection( - input_hierarchy, - atom_type=IMP.atom.AT_CA) - else: - sel = IMP.atom.Selection( - input_hierarchy, - atom_type=IMP.atom.AT_P) - - for p in sel.get_selected_particles(): - resobject = IMP.atom.Residue(IMP.atom.Atom(p).get_parent()) - if 0 in resolutions: - # if you ask for atoms - resclone0 = IMP.atom.create_clone(resobject) - resindex = IMP.atom.Residue(resclone0).get_index() - s0.add_child(resclone0) - self.hier_db.add_particles( - name, - resindex, - 0, - resclone0.get_children()) - - chil = resclone0.get_children() - for ch in chil: - IMP.pmi.Resolution.setup_particle(ch, 0) - try: - clr = IMP.display.get_rgb_color(color) - except: - clr = IMP.display.get_rgb_color(1.0) - IMP.display.Colored.setup_particle(ch, clr) - - if 1 in resolutions: - # else clone the residue - resclone1 = IMP.atom.create_clone_one(resobject) - resindex = IMP.atom.Residue(resclone1).get_index() - s1.add_child(resclone1) - self.hier_db.add_particles(name, resindex, 1, [resclone1]) - - rt = IMP.atom.Residue(resclone1).get_residue_type() - xyz = IMP.core.XYZ(p).get_coordinates() - prt = resclone1.get_particle() - prt.set_name('%s_%i_%s' % (name, resindex, type)) - IMP.core.XYZ.setup_particle(prt).set_coordinates(xyz) - - try: - vol = IMP.atom.get_volume_from_residue_type(rt) - # mass=IMP.atom.get_mass_from_residue_type(rt) - except IMP.ValueException: - vol = IMP.atom.get_volume_from_residue_type( - IMP.atom.ResidueType("ALA")) - # mass=IMP.atom.get_mass_from_residue_type(IMP.atom.ResidueType("ALA")) - radius = IMP.algebra.get_ball_radius_from_volume_3d(vol) - IMP.core.XYZR.setup_particle(prt).set_radius(radius) - IMP.atom.Mass.setup_particle(prt, 100) - IMP.pmi.Uncertainty.setup_particle(prt, radius) - IMP.pmi.Symmetric.setup_particle(prt, 0) - IMP.pmi.Resolution.setup_particle(prt, 1) - try: - clr = IMP.display.get_rgb_color(color) - except: - clr = IMP.display.get_rgb_color(1.0) - IMP.display.Colored.setup_particle(prt, clr) - - for r in resolutions: - if r != 0 and r != 1: - self.check_root(name, protein_h, r) - s = IMP.atom.create_simplified_along_backbone( - input_hierarchy, - r) - - chil = s.get_children() - s0 = IMP.atom.Fragment.setup_particle(IMP.Particle(self.model)) - s0.set_name('%s_%i-%i_%s' % (name, start, end, type)) - # Move all children from s to s0 - for ch in chil: - s.remove_child(ch) - s0.add_child(ch) - self.hier_representation[name][ - "Res:" + str(int(r))].add_child(s0) - outhiers += [s0] - IMP.atom.destroy(s) - - for prt in IMP.atom.get_leaves(s0): - ri = IMP.atom.Fragment(prt).get_residue_indexes() - first = ri[0] - last = ri[-1] - if first == last: - prt.set_name('%s_%i_%s' % (name, first, type)) - else: - prt.set_name('%s_%i_%i_%s' % (name, first, last, type)) - for kk in ri: - self.hier_db.add_particles(name, kk, r, [prt]) - - radius = IMP.core.XYZR(prt).get_radius() - IMP.pmi.Uncertainty.setup_particle(prt, radius) - IMP.pmi.Symmetric.setup_particle(prt, 0) - IMP.pmi.Resolution.setup_particle(prt, r) - - # setting up color for each particle in the - # hierarchy, if colors missing in the colors list set it to - # red - try: - clr = IMP.display.get_rgb_color(color) - except: - colors.append(1.0) - clr = IMP.display.get_rgb_color(colors[pdb_part_count]) - IMP.display.Colored.setup_particle(prt, clr) - - return outhiers - - def get_hierarchies_at_given_resolution(self, resolution): - ''' - Get the hierarchies at the given resolution. - - The map between resolution and hierarchies is cached to accelerate - the selection algorithm. The cache is invalidated when the - representation was changed by any add_component_xxx. - ''' - - if self.representation_is_modified: - rhbr = self.hier_db.get_all_root_hierarchies_by_resolution( - resolution) - self.hier_resolution[resolution] = rhbr - self.representation_is_modified = False - return rhbr - else: - if resolution in self.hier_resolution: - return self.hier_resolution[resolution] - else: - rhbr = self.hier_db.get_all_root_hierarchies_by_resolution( - resolution) - self.hier_resolution[resolution] = rhbr - return rhbr - - def shuffle_configuration( - self, max_translation=300., max_rotation=2.0 * pi, - avoidcollision=True, cutoff=10.0, niterations=100, - bounding_box=None, - excluded_rigid_bodies=None, - ignore_initial_coordinates=False, - hierarchies_excluded_from_collision=None): - ''' - Shuffle configuration; used to restart the optimization. - - The configuration of the system is initialized by placing each - rigid body and each bead randomly in a box with a side of - `max_translation` angstroms, and far enough from each other to - prevent any steric clashes. The rigid bodies are also randomly rotated. - - @param avoidcollision check if the particle/rigid body was - placed close to another particle; uses the optional - arguments cutoff and niterations - @param bounding_box defined by ((x1,y1,z1),(x2,y2,z2)) - ''' - - if excluded_rigid_bodies is None: - excluded_rigid_bodies = [] - if hierarchies_excluded_from_collision is None: - hierarchies_excluded_from_collision = [] - - if len(self.rigid_bodies) == 0: - print("shuffle_configuration: rigid bodies were not intialized") - - gcpf = IMP.core.GridClosePairsFinder() - gcpf.set_distance(cutoff) - ps = [] - hierarchies_excluded_from_collision_indexes = [] - for p in IMP.atom.get_leaves(self.prot): - if IMP.core.XYZ.get_is_setup(p): - ps.append(p) - if IMP.core.Gaussian.get_is_setup(p): - # remove the densities particles out of the calculation - hierarchies_excluded_from_collision_indexes += IMP.get_indexes([p]) - allparticleindexes = IMP.get_indexes(ps) - - if bounding_box is not None: - ((x1, y1, z1), (x2, y2, z2)) = bounding_box - lb = IMP.algebra.Vector3D(x1, y1, z1) - ub = IMP.algebra.Vector3D(x2, y2, z2) - bb = IMP.algebra.BoundingBox3D(lb, ub) - - for h in hierarchies_excluded_from_collision: - hierarchies_excluded_from_collision_indexes += IMP.get_indexes(IMP.atom.get_leaves(h)) - - - allparticleindexes = list( - set(allparticleindexes) - set(hierarchies_excluded_from_collision_indexes)) - - print(hierarchies_excluded_from_collision) - print(len(allparticleindexes),len(hierarchies_excluded_from_collision_indexes)) - - print('shuffling', len(self.rigid_bodies) - len(excluded_rigid_bodies), 'rigid bodies') - for rb in self.rigid_bodies: - if rb not in excluded_rigid_bodies: - if avoidcollision: - rbindexes = rb.get_member_particle_indexes() - rbindexes = list( - set(rbindexes) - set(hierarchies_excluded_from_collision_indexes)) - otherparticleindexes = list( - set(allparticleindexes) - set(rbindexes)) - - if len(otherparticleindexes) is None: - continue - - niter = 0 - while niter < niterations: - if (ignore_initial_coordinates): - # Move the particle to the origin - transformation = IMP.algebra.Transformation3D(IMP.algebra.get_identity_rotation_3d(), -IMP.core.XYZ(rb).get_coordinates()) - IMP.core.transform(rb, transformation) - rbxyz = IMP.core.XYZ(rb).get_coordinates() - - if bounding_box is not None: - # overrides the perturbation - translation = IMP.algebra.get_random_vector_in(bb) - rotation = IMP.algebra.get_random_rotation_3d() - transformation = IMP.algebra.Transformation3D(rotation, translation-rbxyz) - else: - transformation = IMP.algebra.get_random_local_transformation( - rbxyz, - max_translation, - max_rotation) - - IMP.core.transform(rb, transformation) - - if avoidcollision: - self.model.update() - npairs = len( - gcpf.get_close_pairs( - self.model, - otherparticleindexes, - rbindexes)) - if npairs == 0: - niter = niterations - if (ignore_initial_coordinates): - print (rb.get_name(), IMP.core.XYZ(rb).get_coordinates()) - else: - niter += 1 - print("shuffle_configuration: rigid body placed close to other %d particles, trying again..." % npairs) - print("shuffle_configuration: rigid body name: " + rb.get_name()) - if niter == niterations: - raise ValueError("tried the maximum number of iterations to avoid collisions, increase the distance cutoff") - else: - break - - print('shuffling', len(self.floppy_bodies), 'floppy bodies') - for fb in self.floppy_bodies: - if (avoidcollision): - rm = not IMP.core.RigidMember.get_is_setup(fb) - nrm = not IMP.core.NonRigidMember.get_is_setup(fb) - if rm and nrm: - fbindexes = IMP.get_indexes([fb]) - otherparticleindexes = list( - set(allparticleindexes) - set(fbindexes)) - if len(otherparticleindexes) is None: - continue - else: - continue - else: - rm = IMP.core.RigidMember.get_is_setup(fb) - nrm = IMP.core.NonRigidMember.get_is_setup(fb) - if (rm or nrm): - continue - - if IMP.core.RigidBodyMember.get_is_setup(fb): - d=IMP.core.RigidBodyMember(fb).get_rigid_body() - elif IMP.core.RigidBody.get_is_setup(fb): - d=IMP.core.RigidBody(fb) - elif IMP.core.XYZ.get_is_setup(fb): - d=IMP.core.XYZ(fb) - - niter = 0 - while niter < niterations: - if (ignore_initial_coordinates): - # Move the particle to the origin - transformation = IMP.algebra.Transformation3D(IMP.algebra.get_identity_rotation_3d(), -IMP.core.XYZ(fb).get_coordinates()) - IMP.core.transform(d, transformation) - fbxyz = IMP.core.XYZ(fb).get_coordinates() - - if bounding_box is not None: - # overrides the perturbation - translation = IMP.algebra.get_random_vector_in(bb) - transformation = IMP.algebra.Transformation3D(translation-fbxyz) - else: - transformation = IMP.algebra.get_random_local_transformation( - fbxyz, - max_translation, - max_rotation) - - IMP.core.transform(d, transformation) - - if (avoidcollision): - self.model.update() - npairs = len( - gcpf.get_close_pairs( - self.model, - otherparticleindexes, - fbindexes)) - if npairs == 0: - niter = niterations - if (ignore_initial_coordinates): - print (fb.get_name(), IMP.core.XYZ(fb).get_coordinates()) - else: - niter += 1 - print("shuffle_configuration: floppy body placed close to other %d particles, trying again..." % npairs) - print("shuffle_configuration: floppy body name: " + fb.get_name()) - if niter == niterations: - raise ValueError("tried the maximum number of iterations to avoid collisions, increase the distance cutoff") - else: - break - - def set_current_coordinates_as_reference_for_rmsd(self, label="None"): - # getting only coordinates from pdb - ps = IMP.pmi.tools.select(self, resolution=1.0) - # storing the reference coordinates and the particles - self.reference_structures[label] = ( - [IMP.core.XYZ(p).get_coordinates() for p in ps], - ps) - - def get_all_rmsds(self): - rmsds = {} - - for label in self.reference_structures: - - current_coordinates = [IMP.core.XYZ(p).get_coordinates() - for p in self.reference_structures[label][1]] - reference_coordinates = self.reference_structures[label][0] - if len(reference_coordinates) != len(current_coordinates): - print("calculate_all_rmsds: reference and actual coordinates are not the same") - continue - transformation = IMP.algebra.get_transformation_aligning_first_to_second( - current_coordinates, - reference_coordinates) - rmsd_global = IMP.algebra.get_rmsd( - reference_coordinates, - current_coordinates) - # warning: temporary we are calculating the drms, and not the rmsd, - # for the relative distance - rmsd_relative = IMP.atom.get_drms( - reference_coordinates, - current_coordinates) - rmsds[label + "_GlobalRMSD"] = rmsd_global - rmsds[label + "_RelativeDRMS"] = rmsd_relative - return rmsds - - def setup_component_geometry(self, name, color=None, resolution=1.0): - if color is None: - color = self.color_dict[name] - # this function stores all particle pairs - # ordered by residue number, to be used - # to construct backbone traces - self.hier_geometry_pairs[name] = [] - protein_h = self.hier_dict[name] - pbr = IMP.pmi.tools.select(self, name=name, resolution=resolution) - pbr = IMP.pmi.tools.sort_by_residues(pbr) - - for n in range(len(pbr) - 1): - self.hier_geometry_pairs[name].append((pbr[n], pbr[n + 1], color)) - - def setup_component_sequence_connectivity( - self, name, resolution=10, scale=1.0): - ''' - Generate restraints between contiguous fragments in the hierarchy. - The linkers are generated at resolution 10 by default. - - ''' - - unmodeledregions_cr = IMP.RestraintSet(self.model, "unmodeledregions") - sortedsegments_cr = IMP.RestraintSet(self.model, "sortedsegments") - - protein_h = self.hier_dict[name] - SortedSegments = [] - frs = self.hier_db.get_preroot_fragments_by_resolution( - name, - resolution) - - for fr in frs: - try: - start = fr.get_children()[0] - except: - start = fr - - try: - end = fr.get_children()[-1] - except: - end = fr - - startres = IMP.pmi.tools.get_residue_indexes(start)[0] - endres = IMP.pmi.tools.get_residue_indexes(end)[-1] - SortedSegments.append((start, end, startres)) - SortedSegments = sorted(SortedSegments, key=itemgetter(2)) - - # connect the particles - for x in range(len(SortedSegments) - 1): - last = SortedSegments[x][1] - first = SortedSegments[x + 1][0] - - nreslast = len(IMP.pmi.tools.get_residue_indexes(last)) - lastresn = IMP.pmi.tools.get_residue_indexes(last)[-1] - nresfirst = len(IMP.pmi.tools.get_residue_indexes(first)) - firstresn = IMP.pmi.tools.get_residue_indexes(first)[0] - - residuegap = firstresn - lastresn - 1 - if self.disorderedlength and (nreslast / 2 + nresfirst / 2 + residuegap) > 20.0: - # calculate the distance between the sphere centers using Kohn - # PNAS 2004 - optdist = sqrt(5 / 3) * 1.93 * \ - (nreslast / 2 + nresfirst / 2 + residuegap) ** 0.6 - # optdist2=sqrt(5/3)*1.93*((nreslast)**0.6+(nresfirst)**0.6)/2 - if self.upperharmonic: - hu = IMP.core.HarmonicUpperBound(optdist, self.kappa) - else: - hu = IMP.core.Harmonic(optdist, self.kappa) - dps = IMP.core.DistancePairScore(hu) - else: # default - optdist = (0.0 + (float(residuegap) + 1.0) * 3.6) * scale - if self.upperharmonic: # default - hu = IMP.core.HarmonicUpperBound(optdist, self.kappa) - else: - hu = IMP.core.Harmonic(optdist, self.kappa) - dps = IMP.core.SphereDistancePairScore(hu) - - pt0 = last.get_particle() - pt1 = first.get_particle() - r = IMP.core.PairRestraint(self.model, dps, (pt0.get_index(), pt1.get_index())) - - print("Adding sequence connectivity restraint between", pt0.get_name(), " and ", pt1.get_name(), 'of distance', optdist) - sortedsegments_cr.add_restraint(r) - self.linker_restraints_dict[ - "LinkerRestraint-" + pt0.get_name() + "-" + pt1.get_name()] = r - self.connected_intra_pairs.append((pt0, pt1)) - self.connected_intra_pairs.append((pt1, pt0)) - - self.linker_restraints.add_restraint(sortedsegments_cr) - self.linker_restraints.add_restraint(unmodeledregions_cr) - IMP.pmi.tools.add_restraint_to_model(self.model, self.linker_restraints) - self.sortedsegments_cr_dict[name] = sortedsegments_cr - self.unmodeledregions_cr_dict[name] = unmodeledregions_cr - - def optimize_floppy_bodies(self, nsteps, temperature=1.0): - import IMP.pmi.samplers - pts = IMP.pmi.tools.ParticleToSampleList() - for n, fb in enumerate(self.floppy_bodies): - pts.add_particle(fb, "Floppy_Bodies", 1.0, "Floppy_Body_" + str(n)) - if len(pts.get_particles_to_sample()) > 0: - mc = IMP.pmi.samplers.MonteCarlo(self.model, [pts], temperature) - print("optimize_floppy_bodies: optimizing %i floppy bodies" % len(self.floppy_bodies)) - mc.optimize(nsteps) - else: - print("optimize_floppy_bodies: no particle to optimize") - - def create_rotational_symmetry(self, maincopy, copies, rotational_axis=IMP.algebra.Vector3D(0, 0, 1.0), - nSymmetry=None, skip_gaussian_in_clones=False): - ''' - The copies must not contain rigid bodies. - The symmetry restraints are applied at each leaf. - ''' - - from math import pi - self.representation_is_modified = True - ncopies = len(copies) + 1 - main_hiers = IMP.atom.get_leaves(self.hier_dict[maincopy]) - - for k in range(len(copies)): - if (nSymmetry is None): - rotation_angle = 2.0 * pi / float(ncopies) * float(k + 1) - else: - if ( k % 2 == 0 ): - rotation_angle = 2.0 * pi / float(nSymmetry) * float((k + 2) / 2) - else: - rotation_angle = -2.0 * pi / float(nSymmetry) * float((k + 1) / 2) - - rotation3D = IMP.algebra.get_rotation_about_axis(rotational_axis, rotation_angle) - - sm = IMP.core.TransformationSymmetry(rotation3D) - clone_hiers = IMP.atom.get_leaves(self.hier_dict[copies[k]]) - - lc = IMP.container.ListSingletonContainer(self.model) - for n, p in enumerate(main_hiers): - if (skip_gaussian_in_clones): - if (IMP.core.Gaussian.get_is_setup(p)) and not (IMP.atom.Fragment.get_is_setup(p) or IMP.atom.Residue.get_is_setup(p)): - continue - pc = clone_hiers[n] - #print("setting " + p.get_name() + " as reference for " + pc.get_name()) - IMP.core.Reference.setup_particle(pc.get_particle(), p.get_particle()) - lc.add(pc.get_particle().get_index()) - - c = IMP.container.SingletonsConstraint(sm, None, lc) - self.model.add_score_state(c) - print("Completed setting " + str(maincopy) + " as a reference for " - + str(copies[k]) + " by rotating it by " - + str(rotation_angle / 2.0 / pi * 360) - + " degrees around the " + str(rotational_axis) + " axis.") - self.model.update() - - def create_rigid_body_symmetry(self, particles_reference, particles_copy,label="None", - initial_transformation=IMP.algebra.get_identity_transformation_3d()): - from math import pi - self.representation_is_modified = True - - mainparticles = particles_reference - - t=initial_transformation - p=IMP.Particle(self.model) - p.set_name("RigidBody_Symmetry") - rb=IMP.core.RigidBody.setup_particle(p,IMP.algebra.ReferenceFrame3D(t)) - - sm = IMP.core.TransformationSymmetry(rb) - - - copyparticles = particles_copy - - mainpurged = [] - copypurged = [] - for n, p in enumerate(mainparticles): - print(p.get_name()) - pc = copyparticles[n] - - mainpurged.append(p) - if not IMP.pmi.Symmetric.get_is_setup(p): - IMP.pmi.Symmetric.setup_particle(p, 0) - else: - IMP.pmi.Symmetric(p).set_symmetric(0) - - copypurged.append(pc) - if not IMP.pmi.Symmetric.get_is_setup(pc): - IMP.pmi.Symmetric.setup_particle(pc, 1) - else: - IMP.pmi.Symmetric(pc).set_symmetric(1) - - lc = IMP.container.ListSingletonContainer(self.model) - for n, p in enumerate(mainpurged): - - pc = copypurged[n] - print("setting " + p.get_name() + " as reference for " + pc.get_name()) - - IMP.core.Reference.setup_particle(pc, p) - lc.add(pc.get_index()) - - c = IMP.container.SingletonsConstraint(sm, None, lc) - self.model.add_score_state(c) - - self.model.update() - self.rigid_bodies.append(rb) - self.rigid_body_symmetries.append(rb) - rb.set_name(label+".rigid_body_symmetry."+str(len(self.rigid_body_symmetries))) - - - def create_amyloid_fibril_symmetry(self, maincopy, axial_copies, - longitudinal_copies, axis=(0, 0, 1), translation_value=4.8): - - from math import pi - self.representation_is_modified = True - - outhiers = [] - protein_h = self.hier_dict[maincopy] - mainparts = IMP.atom.get_leaves(protein_h) - - for ilc in range(-longitudinal_copies, longitudinal_copies + 1): - for iac in range(axial_copies): - copyname = maincopy + "_a" + str(ilc) + "_l" + str(iac) - self.create_component(copyname, 0.0) - for hier in protein_h.get_children(): - self.add_component_hierarchy_clone(copyname, hier) - copyhier = self.hier_dict[copyname] - outhiers.append(copyhier) - copyparts = IMP.atom.get_leaves(copyhier) - rotation3D = IMP.algebra.get_rotation_about_axis( - IMP.algebra.Vector3D(axis), - 2 * pi / axial_copies * (float(iac))) - translation_vector = tuple( - [translation_value * float(ilc) * x for x in axis]) - print(translation_vector) - translation = IMP.algebra.Vector3D(translation_vector) - sm = IMP.core.TransformationSymmetry( - IMP.algebra.Transformation3D(rotation3D, translation)) - lc = IMP.container.ListSingletonContainer(self.model) - for n, p in enumerate(mainparts): - pc = copyparts[n] - if not IMP.pmi.Symmetric.get_is_setup(p): - IMP.pmi.Symmetric.setup_particle(p, 0) - if not IMP.pmi.Symmetric.get_is_setup(pc): - IMP.pmi.Symmetric.setup_particle(pc, 1) - IMP.core.Reference.setup_particle(pc, p) - lc.add(pc.get_index()) - c = IMP.container.SingletonsConstraint(sm, None, lc) - self.model.add_score_state(c) - self.model.update() - return outhiers - - def link_components_to_rmf(self, rmfname, frameindex): - ''' - Load coordinates in the current representation. - This should be done only if the hierarchy self.prot is identical - to the one as stored in the rmf i.e. all components were added. - ''' - import IMP.rmf - import RMF - - rh = RMF.open_rmf_file_read_only(rmfname) - IMP.rmf.link_hierarchies(rh, [self.prot]) - IMP.rmf.load_frame(rh, RMF.FrameID(frameindex)) - del rh - - def create_components_from_rmf(self, rmfname, frameindex): - ''' - still not working. - create the representation (i.e. hierarchies) from the rmf file. - it will be stored in self.prot, which will be overwritten. - load the coordinates from the rmf file at frameindex. - ''' - rh = RMF.open_rmf_file_read_only(rmfname) - self.prot = IMP.rmf.create_hierarchies(rh, self.model)[0] - IMP.atom.show_molecular_hierarchy(self.prot) - IMP.rmf.link_hierarchies(rh, [self.prot]) - IMP.rmf.load_frame(rh, RMF.FrameID(frameindex)) - del rh - for p in self.prot.get_children(): - self.create_component(p.get_name()) - self.hier_dict[p.get_name()] = p - ''' - still missing: save rigid bodies contained in the rmf in self.rigid_bodies - save floppy bodies in self.floppy_bodies - get the connectivity restraints - ''' - - def set_rigid_body_from_hierarchies(self, hiers, particles=None): - ''' - Construct a rigid body from hierarchies (and optionally particles). - - @param hiers list of hierarchies to make rigid - @param particles list of particles to add to the rigid body - ''' - - if particles is None: - rigid_parts = set() - else: - rigid_parts = set(particles) - - name = "" - print("set_rigid_body_from_hierarchies> setting up a new rigid body") - for hier in hiers: - ps = IMP.atom.get_leaves(hier) - - for p in ps: - if IMP.core.RigidMember.get_is_setup(p): - rb = IMP.core.RigidMember(p).get_rigid_body() - warnings.warn( - "set_rigid_body_from_hierarchies> particle %s " - "already belongs to rigid body %s" - % (p.get_name(), rb.get_name()), - IMP.pmi.StructureWarning) - else: - rigid_parts.add(p) - name += hier.get_name() + "-" - print("set_rigid_body_from_hierarchies> adding %s to the rigid body" % hier.get_name()) - - if len(list(rigid_parts)) != 0: - rb = IMP.atom.create_rigid_body(list(rigid_parts)) - rb.set_coordinates_are_optimized(True) - rb.set_name(name + "rigid_body") - self.rigid_bodies.append(rb) - return rb - - else: - print("set_rigid_body_from_hierarchies> rigid body could not be setup") - - def set_rigid_bodies(self, subunits): - ''' - Construct a rigid body from a list of subunits. - - Each subunit is a tuple that identifies the residue ranges and the - component name (as used in create_component()). - - subunits: [(name_1,(first_residue_1,last_residue_1)),(name_2,(first_residue_2,last_residue_2)),.....] - or - [name_1,name_2,(name_3,(first_residue_3,last_residue_3)),.....] - - example: ["prot1","prot2",("prot3",(1,10))] - - sometimes, we know about structure of an interaction - and here we make such PPIs rigid - ''' - - rigid_parts = set() - for s in subunits: - if isinstance(s, tuple) and len(s) == 2: - sel = IMP.atom.Selection( - self.prot, - molecule=s[0], - residue_indexes=list(range(s[1][0], - s[1][1] + 1))) - if len(sel.get_selected_particles()) == 0: - print("set_rigid_bodies: selected particle does not exist") - for p in sel.get_selected_particles(): - # if not p in self.floppy_bodies: - if IMP.core.RigidMember.get_is_setup(p): - rb = IMP.core.RigidMember(p).get_rigid_body() - warnings.warn( - "set_rigid_body_from_hierarchies> particle %s " - "already belongs to rigid body %s" - % (p.get_name(), rb.get_name()), - IMP.pmi.StructureWarning) - else: - rigid_parts.add(p) - - elif isinstance(s, str): - sel = IMP.atom.Selection(self.prot, molecule=s) - if len(sel.get_selected_particles()) == 0: - print("set_rigid_bodies: selected particle does not exist") - for p in sel.get_selected_particles(): - # if not p in self.floppy_bodies: - if IMP.core.RigidMember.get_is_setup(p): - rb = IMP.core.RigidMember(p).get_rigid_body() - warnings.warn( - "set_rigid_body_from_hierarchies> particle %s " - "already belongs to rigid body %s" - % (p.get_name(), rb.get_name()), - IMP.pmi.StructureWarning) - else: - rigid_parts.add(p) - - rb = IMP.atom.create_rigid_body(list(rigid_parts)) - rb.set_coordinates_are_optimized(True) - rb.set_name(''.join(str(subunits)) + "_rigid_body") - self.rigid_bodies.append(rb) - return rb - - def set_super_rigid_body_from_hierarchies( - self, - hiers, - axis=None, - min_size=1): - # axis is the rotation axis for 2D rotation - super_rigid_xyzs = set() - super_rigid_rbs = set() - name = "" - print("set_super_rigid_body_from_hierarchies> setting up a new SUPER rigid body") - - for hier in hiers: - ps = IMP.atom.get_leaves(hier) - for p in ps: - if IMP.core.RigidMember.get_is_setup(p): - rb = IMP.core.RigidMember(p).get_rigid_body() - super_rigid_rbs.add(rb) - else: - super_rigid_xyzs.add(p) - print("set_rigid_body_from_hierarchies> adding %s to the rigid body" % hier.get_name()) - if len(super_rigid_rbs|super_rigid_xyzs) < min_size: - return - if axis is None: - self.super_rigid_bodies.append((super_rigid_xyzs, super_rigid_rbs)) - else: - # these will be 2D rotation SRB or a bond rotamer (axis can be a IMP.algebra.Vector3D or particle Pair) - self.super_rigid_bodies.append( - (super_rigid_xyzs, super_rigid_rbs, axis)) - - def fix_rigid_bodies(self, rigid_bodies): - self.fixed_rigid_bodies += rigid_bodies - - - def set_chain_of_super_rigid_bodies( - self, hiers, min_length=None, max_length=None, axis=None): - ''' - Make a chain of super rigid bodies from a list of hierarchies. - - Takes a linear list of hierarchies (they are supposed to be - sequence-contiguous) and produces a chain of super rigid bodies - with given length range, specified by min_length and max_length. - ''' - try: - hiers = IMP.pmi.tools.flatten_list(hiers) - except: - pass - for hs in IMP.pmi.tools.sublist_iterator(hiers, min_length, max_length): - self.set_super_rigid_body_from_hierarchies(hs, axis, min_length) - - def set_super_rigid_bodies(self, subunits, coords=None): - super_rigid_xyzs = set() - super_rigid_rbs = set() - - for s in subunits: - if isinstance(s, tuple) and len(s) == 3: - sel = IMP.atom.Selection( - self.prot, - molecule=s[2], - residue_indexes=list(range(s[0], - s[1] + 1))) - if len(sel.get_selected_particles()) == 0: - print("set_rigid_bodies: selected particle does not exist") - for p in sel.get_selected_particles(): - if IMP.core.RigidMember.get_is_setup(p): - rb = IMP.core.RigidMember(p).get_rigid_body() - super_rigid_rbs.add(rb) - else: - super_rigid_xyzs.add(p) - elif isinstance(s, str): - sel = IMP.atom.Selection(self.prot, molecule=s) - if len(sel.get_selected_particles()) == 0: - print("set_rigid_bodies: selected particle does not exist") - for p in sel.get_selected_particles(): - # if not p in self.floppy_bodies: - if IMP.core.RigidMember.get_is_setup(p): - rb = IMP.core.RigidMember(p).get_rigid_body() - super_rigid_rbs.add(rb) - else: - super_rigid_xyzs.add(p) - self.super_rigid_bodies.append((super_rigid_xyzs, super_rigid_rbs)) - - def remove_floppy_bodies_from_component(self, component_name): - ''' - Remove leaves of hierarchies from the floppy bodies list based - on the component name - ''' - hs=IMP.atom.get_leaves(self.hier_dict[component_name]) - ps=[h.get_particle() for h in hs] - for p in self.floppy_bodies: - try: - if p in ps: self.floppy_bodies.remove(p) - if p in hs: self.floppy_bodies.remove(p) - except: - continue - - - def remove_floppy_bodies(self, hierarchies): - ''' - Remove leaves of hierarchies from the floppy bodies list. - - Given a list of hierarchies, get the leaves and remove the - corresponding particles from the floppy bodies list. We need this - function because sometimes - we want to constrain the floppy bodies in a rigid body - for instance - when you want to associate a bead with a density particle. - ''' - for h in hierarchies: - ps = IMP.atom.get_leaves(h) - for p in ps: - if p in self.floppy_bodies: - print("remove_floppy_bodies: removing %s from floppy body list" % p.get_name()) - self.floppy_bodies.remove(p) - - - def set_floppy_bodies(self): - for p in self.floppy_bodies: - name = p.get_name() - p.set_name(name + "_floppy_body") - if IMP.core.RigidMember.get_is_setup(p): - print("I'm trying to make this particle flexible although it was assigned to a rigid body", p.get_name()) - rb = IMP.core.RigidMember(p).get_rigid_body() - try: - rb.set_is_rigid_member(p.get_index(), False) - except: - # some IMP versions still work with that - rb.set_is_rigid_member(p.get_particle_index(), False) - p.set_name(p.get_name() + "_rigid_body_member") - - def set_floppy_bodies_from_hierarchies(self, hiers): - for hier in hiers: - ps = IMP.atom.get_leaves(hier) - for p in ps: - IMP.core.XYZ(p).set_coordinates_are_optimized(True) - self.floppy_bodies.append(p) - - def get_particles_from_selection_tuples( - self, - selection_tuples, - resolution=None): - ''' - selection tuples must be [(r1,r2,"name1"),(r1,r2,"name2"),....] - @return the particles - ''' - particles = [] - print(selection_tuples) - for s in selection_tuples: - ps = IMP.pmi.tools.select_by_tuple( - representation=self, tupleselection=tuple(s), - resolution=None, name_is_ambiguous=False) - particles += ps - return particles - - def get_connected_intra_pairs(self): - return self.connected_intra_pairs - - def set_rigid_bodies_max_trans(self, maxtrans): - self.maxtrans_rb = maxtrans - - def set_rigid_bodies_max_rot(self, maxrot): - self.maxrot_rb = maxrot - - def set_super_rigid_bodies_max_trans(self, maxtrans): - self.maxtrans_srb = maxtrans - - def set_super_rigid_bodies_max_rot(self, maxrot): - self.maxrot_srb = maxrot - - def set_floppy_bodies_max_trans(self, maxtrans): - self.maxtrans_fb = maxtrans - - def set_rigid_bodies_as_fixed(self, rigidbodiesarefixed=True): - ''' - Fix rigid bodies in their actual position. - The get_particles_to_sample() function will return - just the floppy bodies (if they are not fixed). - ''' - self.rigidbodiesarefixed = rigidbodiesarefixed - - def set_floppy_bodies_as_fixed(self, floppybodiesarefixed=True): - ''' - Fix floppy bodies in their actual position. - The get_particles_to_sample() function will return - just the rigid bodies (if they are not fixed). - ''' - self.floppybodiesarefixed=floppybodiesarefixed - - def draw_hierarchy_graph(self): - for c in IMP.atom.Hierarchy(self.prot).get_children(): - print("Drawing hierarchy graph for " + c.get_name()) - IMP.pmi.output.get_graph_from_hierarchy(c) - - def get_geometries(self): - # create segments at the lowest resolution - seggeos = [] - for name in self.hier_geometry_pairs: - for pt in self.hier_geometry_pairs[name]: - p1 = pt[0] - p2 = pt[1] - color = pt[2] - try: - clr = IMP.display.get_rgb_color(color) - except: - clr = IMP.display.get_rgb_color(1.0) - coor1 = IMP.core.XYZ(p1).get_coordinates() - coor2 = IMP.core.XYZ(p2).get_coordinates() - seg = IMP.algebra.Segment3D(coor1, coor2) - seggeos.append(IMP.display.SegmentGeometry(seg, clr)) - return seggeos - - def setup_bonds(self): - # create segments at the lowest resolution - seggeos = [] - for name in self.hier_geometry_pairs: - for pt in self.hier_geometry_pairs[name]: - - p1 = pt[0] - p2 = pt[1] - if not IMP.atom.Bonded.get_is_setup(p1): - IMP.atom.Bonded.setup_particle(p1) - if not IMP.atom.Bonded.get_is_setup(p2): - IMP.atom.Bonded.setup_particle(p2) - - if not IMP.atom.get_bond(IMP.atom.Bonded(p1),IMP.atom.Bonded(p2)): - IMP.atom.create_bond( - IMP.atom.Bonded(p1), - IMP.atom.Bonded(p2), - 1) - - def show_component_table(self, name): - if name in self.sequence_dict: - lastresn = len(self.sequence_dict[name]) - firstresn = 1 - else: - residues = self.hier_db.get_residue_numbers(name) - firstresn = min(residues) - lastresn = max(residues) - - for nres in range(firstresn, lastresn + 1): - try: - resolutions = self.hier_db.get_residue_resolutions(name, nres) - ps = [] - for r in resolutions: - ps += self.hier_db.get_particles(name, nres, r) - print("%20s %7s" % (name, nres), " ".join(["%20s %7s" % (str(p.get_name()), - str(IMP.pmi.Resolution(p).get_resolution())) for p in ps])) - except: - print("%20s %20s" % (name, nres), "**** not represented ****") - - def draw_hierarchy_composition(self): - - ks = list(self.elements.keys()) - ks.sort() - - max = 0 - for k in self.elements: - for l in self.elements[k]: - if l[1] > max: - max = l[1] - - for k in ks: - self.draw_component_composition(k, max) - - def draw_component_composition(self, name, max=1000, draw_pdb_names=False): - from matplotlib import pyplot - import matplotlib as mpl - k = name - tmplist = sorted(self.elements[k], key=itemgetter(0)) - try: - endres = tmplist[-1][1] - except: - print("draw_component_composition: missing information for component %s" % name) - return - fig = pyplot.figure(figsize=(26.0 * float(endres) / max + 2, 2)) - ax = fig.add_axes([0.05, 0.475, 0.9, 0.15]) - - # Set the colormap and norm to correspond to the data for which - # the colorbar will be used. - cmap = mpl.cm.cool - norm = mpl.colors.Normalize(vmin=5, vmax=10) - bounds = [1] - colors = [] - - for n, l in enumerate(tmplist): - firstres = l[0] - lastres = l[1] - if l[3] != "end": - if bounds[-1] != l[0]: - colors.append("white") - bounds.append(l[0]) - if l[3] == "pdb": - colors.append("#99CCFF") - if l[3] == "bead": - colors.append("#FFFF99") - if l[3] == "helix": - colors.append("#33CCCC") - if l[3] != "end": - bounds.append(l[1] + 1) - else: - if l[3] == "pdb": - colors.append("#99CCFF") - if l[3] == "bead": - colors.append("#FFFF99") - if l[3] == "helix": - colors.append("#33CCCC") - if l[3] != "end": - bounds.append(l[1] + 1) - else: - if bounds[-1] - 1 == l[0]: - bounds.pop() - bounds.append(l[0]) - else: - colors.append("white") - bounds.append(l[0]) - - bounds.append(bounds[-1]) - colors.append("white") - cmap = mpl.colors.ListedColormap(colors) - cmap.set_over('0.25') - cmap.set_under('0.75') - - norm = mpl.colors.BoundaryNorm(bounds, cmap.N) - cb2 = mpl.colorbar.ColorbarBase(ax, cmap=cmap, - norm=norm, - # to use 'extend', you must - # specify two extra boundaries: - boundaries=bounds, - ticks=bounds, # optional - spacing='proportional', - orientation='horizontal') - - extra_artists = [] - npdb = 0 - - if draw_pdb_names: - for l in tmplist: - if l[3] == "pdb": - npdb += 1 - mid = 1.0 / endres * float(l[0]) - # t =ax.text(mid, float(npdb-1)/2.0+1.5, l[2], ha="left", va="center", rotation=0, - # size=10) - # t=ax.annotate(l[0],2) - t = ax.annotate( - l[2], xy=(mid, 1), xycoords='axes fraction', - xytext=(mid + 0.025, float(npdb - 1) / 2.0 + 1.5), textcoords='axes fraction', - arrowprops=dict(arrowstyle="->", - connectionstyle="angle,angleA=0,angleB=90,rad=10"), - ) - extra_artists.append(t) - - # set the title of the bar - title = ax.text(-0.005, 0.5, k, ha="right", va="center", rotation=90, - size=20) - - extra_artists.append(title) - # changing the xticks labels - labels = len(bounds) * [" "] - ax.set_xticklabels(labels) - mid = 1.0 / endres * float(bounds[0]) - t = ax.annotate(bounds[0], xy=(mid, 0), xycoords='axes fraction', - xytext=(mid - 0.01, -0.5), textcoords='axes fraction',) - extra_artists.append(t) - offsets = [0, -0.5, -1.0] - nclashes = 0 - for n in range(1, len(bounds)): - if bounds[n] == bounds[n - 1]: - continue - mid = 1.0 / endres * float(bounds[n]) - if (float(bounds[n]) - float(bounds[n - 1])) / max <= 0.01: - nclashes += 1 - offset = offsets[nclashes % 3] - else: - nclashes = 0 - offset = offsets[0] - if offset > -0.75: - t = ax.annotate( - bounds[n], xy=(mid, 0), xycoords='axes fraction', - xytext=(mid, -0.5 + offset), textcoords='axes fraction') - else: - t = ax.annotate( - bounds[n], xy=(mid, 0), xycoords='axes fraction', - xytext=(mid, -0.5 + offset), textcoords='axes fraction', arrowprops=dict(arrowstyle="-")) - extra_artists.append(t) - - cb2.add_lines(bounds, ["black"] * len(bounds), [1] * len(bounds)) - # cb2.set_label(k) - - pyplot.savefig( - k + "structure.pdf", - dpi=150, - transparent="True", - bbox_extra_artists=(extra_artists), - bbox_inches='tight') - pyplot.show() - - def draw_coordinates_projection(self): - import matplotlib.pyplot as pp - xpairs = [] - ypairs = [] - for name in self.hier_geometry_pairs: - for pt in self.hier_geometry_pairs[name]: - p1 = pt[0] - p2 = pt[1] - color = pt[2] - coor1 = IMP.core.XYZ(p1).get_coordinates() - coor2 = IMP.core.XYZ(p2).get_coordinates() - x1 = coor1[0] - x2 = coor2[0] - y1 = coor1[1] - y2 = coor2[1] - xpairs.append([x1, x2]) - ypairs.append([y1, y2]) - xlist = [] - ylist = [] - for xends, yends in zip(xpairs, ypairs): - xlist.extend(xends) - xlist.append(None) - ylist.extend(yends) - ylist.append(None) - pp.plot(xlist, ylist, 'b-', alpha=0.1) - pp.show() - - def get_prot_name_from_particle(self, particle): - names = self.get_component_names() - particle0 = particle - name = None - while not name in names: - h = IMP.atom.Hierarchy(particle0).get_parent() - name = h.get_name() - particle0 = h.get_particle() - return name - - - def get_particles_to_sample(self): - # get the list of samplable particles with their type - # and the mover displacement. Everything wrapped in a dictionary, - # to be used by samplers modules - ps = {} - - # remove symmetric particles: they are not sampled - rbtmp = [] - fbtmp = [] - srbtmp = [] - if not self.rigidbodiesarefixed: - for rb in self.rigid_bodies: - if IMP.pmi.Symmetric.get_is_setup(rb): - if IMP.pmi.Symmetric(rb).get_symmetric() != 1: - rbtmp.append(rb) - else: - if rb not in self.fixed_rigid_bodies: - rbtmp.append(rb) - - if not self.floppybodiesarefixed: - for fb in self.floppy_bodies: - if IMP.pmi.Symmetric.get_is_setup(fb): - if IMP.pmi.Symmetric(fb).get_symmetric() != 1: - fbtmp.append(fb) - else: - fbtmp.append(fb) - - for srb in self.super_rigid_bodies: - # going to prune the fixed rigid bodies out - # of the super rigid body list - rigid_bodies = list(srb[1]) - filtered_rigid_bodies = [] - for rb in rigid_bodies: - if rb not in self.fixed_rigid_bodies: - filtered_rigid_bodies.append(rb) - srbtmp.append((srb[0], filtered_rigid_bodies)) - - self.rigid_bodies = rbtmp - self.floppy_bodies = fbtmp - self.super_rigid_bodies = srbtmp - - ps["Rigid_Bodies_SimplifiedModel"] = ( - self.rigid_bodies, - self.maxtrans_rb, - self.maxrot_rb) - ps["Floppy_Bodies_SimplifiedModel"] = ( - self.floppy_bodies, - self.maxtrans_fb) - ps["SR_Bodies_SimplifiedModel"] = ( - self.super_rigid_bodies, - self.maxtrans_srb, - self.maxrot_srb) - return ps - - def set_output_level(self, level): - self.output_level = level - - def _evaluate(self, deriv): - """Evaluate the total score of all added restraints""" - r = IMP.pmi.tools.get_restraint_set(self.model) - return r.evaluate(deriv) - - def get_output(self): - output = {} - score = 0.0 - - output["SimplifiedModel_Total_Score_" + - self.label] = str(self._evaluate(False)) - output["SimplifiedModel_Linker_Score_" + - self.label] = str(self.linker_restraints.unprotected_evaluate(None)) - for name in self.sortedsegments_cr_dict: - partialscore = self.sortedsegments_cr_dict[name].evaluate(False) - score += partialscore - output[ - "SimplifiedModel_Link_SortedSegments_" + - name + - "_" + - self.label] = str( - partialscore) - partialscore = self.unmodeledregions_cr_dict[name].evaluate(False) - score += partialscore - output[ - "SimplifiedModel_Link_UnmodeledRegions_" + - name + - "_" + - self.label] = str( - partialscore) - for rb in self.rigid_body_symmetries: - name=rb.get_name() - output[name +"_" +self.label]=str(rb.get_reference_frame().get_transformation_to()) - for name in self.linker_restraints_dict: - output[ - name + - "_" + - self.label] = str( - self.linker_restraints_dict[ - name].unprotected_evaluate( - None)) - - if len(self.reference_structures.keys()) != 0: - rmsds = self.get_all_rmsds() - for label in rmsds: - output[ - "SimplifiedModel_" + - label + - "_" + - self.label] = rmsds[ - label] - - if self.output_level == "high": - for p in IMP.atom.get_leaves(self.prot): - d = IMP.core.XYZR(p) - output["Coordinates_" + - p.get_name() + "_" + self.label] = str(d) - - output["_TotalScore"] = str(score) - return output - - def get_test_output(self): - # this method is called by test functions and return an enriched output - output = self.get_output() - for n, p in enumerate(self.get_particles_to_sample()): - output["Particle_to_sample_" + str(n)] = str(p) - output["Number_of_particles"] = len(IMP.atom.get_leaves(self.prot)) - output["Hierarchy_Dictionary"] = list(self.hier_dict.keys()) - output["Number_of_floppy_bodies"] = len(self.floppy_bodies) - output["Number_of_rigid_bodies"] = len(self.rigid_bodies) - output["Number_of_super_bodies"] = len(self.super_rigid_bodies) - output["Selection_resolution_1"] = len( - IMP.pmi.tools.select(self, resolution=1)) - output["Selection_resolution_5"] = len( - IMP.pmi.tools.select(self, resolution=5)) - output["Selection_resolution_7"] = len( - IMP.pmi.tools.select(self, resolution=5)) - output["Selection_resolution_10"] = len( - IMP.pmi.tools.select(self, resolution=10)) - output["Selection_resolution_100"] = len( - IMP.pmi.tools.select(self, resolution=100)) - output["Selection_All"] = len(IMP.pmi.tools.select(self)) - output["Selection_resolution=1"] = len( - IMP.pmi.tools.select(self, resolution=1)) - output["Selection_resolution=1,resid=10"] = len( - IMP.pmi.tools.select(self, resolution=1, residue=10)) - for resolution in self.hier_resolution: - output["Hier_resolution_dictionary" + - str(resolution)] = len(self.hier_resolution[resolution]) - for name in self.hier_dict: - output[ - "Selection_resolution=1,resid=10,name=" + - name] = len( - IMP.pmi.tools.select( - self, - resolution=1, - name=name, - residue=10)) - output[ - "Selection_resolution=1,resid=10,name=" + - name + - ",ambiguous"] = len( - IMP.pmi.tools.select( - self, - resolution=1, - name=name, - name_is_ambiguous=True, - residue=10)) - output[ - "Selection_resolution=1,resid=10,name=" + - name + - ",ambiguous"] = len( - IMP.pmi.tools.select( - self, - resolution=1, - name=name, - name_is_ambiguous=True, - residue=10)) - output[ - "Selection_resolution=1,resrange=(10,20),name=" + - name] = len( - IMP.pmi.tools.select( - self, - resolution=1, - name=name, - first_residue=10, - last_residue=20)) - output[ - "Selection_resolution=1,resrange=(10,20),name=" + - name + - ",ambiguous"] = len( - IMP.pmi.tools.select( - self, - resolution=1, - name=name, - name_is_ambiguous=True, - first_residue=10, - last_residue=20)) - output[ - "Selection_resolution=10,resrange=(10,20),name=" + - name] = len( - IMP.pmi.tools.select( - self, - resolution=10, - name=name, - first_residue=10, - last_residue=20)) - output[ - "Selection_resolution=10,resrange=(10,20),name=" + - name + - ",ambiguous"] = len( - IMP.pmi.tools.select( - self, - resolution=10, - name=name, - name_is_ambiguous=True, - first_residue=10, - last_residue=20)) - output[ - "Selection_resolution=100,resrange=(10,20),name=" + - name] = len( - IMP.pmi.tools.select( - self, - resolution=100, - name=name, - first_residue=10, - last_residue=20)) - output[ - "Selection_resolution=100,resrange=(10,20),name=" + - name + - ",ambiguous"] = len( - IMP.pmi.tools.select( - self, - resolution=100, - name=name, - name_is_ambiguous=True, - first_residue=10, - last_residue=20)) - - return output diff --git a/modules/pmi/pyext/src/restraints/crosslinking.py b/modules/pmi/pyext/src/restraints/crosslinking.py index a2a27d54c6..c253903ca7 100644 --- a/modules/pmi/pyext/src/restraints/crosslinking.py +++ b/modules/pmi/pyext/src/restraints/crosslinking.py @@ -28,8 +28,7 @@ class CrossLinkingMassSpectrometryRestraint(IMP.pmi.restraints.RestraintBase): is inferred using Bayes theory of probability @note Wraps an IMP::isd::CrossLinkMSRestraint """ - def __init__(self, representation=None, - root_hier=None, + def __init__(self, root_hier, CrossLinkDataBase=None, length=10.0, resolution=None, @@ -39,8 +38,6 @@ def __init__(self, representation=None, attributes_for_label=None, weight=1.): """Constructor. - @param representation DEPRECATED The IMP.pmi.representation.Representation - object that contain the molecular system @param root_hier The canonical hierarchy containing all the states @param CrossLinkDataBase The IMP.pmi.io.crosslink.CrossLinkDataBase object that contains the cross-link dataset @@ -58,19 +55,7 @@ def __init__(self, representation=None, @param weight Weight of restraint """ - use_pmi2 = True - if representation is not None: - use_pmi2 = False - if type(representation) != list: - representations = [representation] - else: - representations = representation - model = representations[0].prot.get_model() - elif root_hier is not None: - representations = [] - model = root_hier.get_model() - else: - raise Exception("You must pass either representation or root_hier") + model = root_hier.get_model() super(CrossLinkingMassSpectrometryRestraint, self).__init__( model, weight=weight, label=label) @@ -109,39 +94,38 @@ def __init__(self, representation=None, xl_groups = [p.get_cross_link_group(self) for p, state in IMP.pmi.tools._all_protocol_outputs( - representations, root_hier)] - - # if PMI2, first add all the molecule copies as clones to the database - if use_pmi2: - copies_to_add = defaultdict(int) - print('gathering copies') - for xlid in self.CrossLinkDataBase.xlid_iterator(): - for xl in self.CrossLinkDataBase[xlid]: - r1 = xl[self.CrossLinkDataBase.residue1_key] - c1 = xl[self.CrossLinkDataBase.protein1_key] - r2 = xl[self.CrossLinkDataBase.residue2_key] - c2 = xl[self.CrossLinkDataBase.protein2_key] - for c,r in ((c1,r1),(c2,r2)): - if c in copies_to_add: - continue - sel = IMP.atom.Selection(root_hier, - state_index=0, - molecule=c, - residue_index=r, - resolution=resolution).get_selected_particles() - if len(sel)>0: - copies_to_add[c] = len(sel)-1 - print(copies_to_add) - for molname in copies_to_add: - if copies_to_add[molname]==0: - continue - fo1 = IMP.pmi.io.crosslink.FilterOperator(self.CrossLinkDataBase.protein1_key,operator.eq,molname) - self.CrossLinkDataBase.set_value(self.CrossLinkDataBase.protein1_key,molname+'.0',fo1) - fo2 = IMP.pmi.io.crosslink.FilterOperator(self.CrossLinkDataBase.protein2_key,operator.eq,molname) - self.CrossLinkDataBase.set_value(self.CrossLinkDataBase.protein2_key,molname+'.0',fo2) - for ncopy in range(copies_to_add[molname]): - self.CrossLinkDataBase.clone_protein('%s.0'%molname,'%s.%i'%(molname,ncopy+1)) - print('done pmi2 prelims') + None, root_hier)] + + # first add all the molecule copies as clones to the database + copies_to_add = defaultdict(int) + print('gathering copies') + for xlid in self.CrossLinkDataBase.xlid_iterator(): + for xl in self.CrossLinkDataBase[xlid]: + r1 = xl[self.CrossLinkDataBase.residue1_key] + c1 = xl[self.CrossLinkDataBase.protein1_key] + r2 = xl[self.CrossLinkDataBase.residue2_key] + c2 = xl[self.CrossLinkDataBase.protein2_key] + for c,r in ((c1,r1),(c2,r2)): + if c in copies_to_add: + continue + sel = IMP.atom.Selection(root_hier, + state_index=0, + molecule=c, + residue_index=r, + resolution=resolution).get_selected_particles() + if len(sel)>0: + copies_to_add[c] = len(sel)-1 + print(copies_to_add) + for molname in copies_to_add: + if copies_to_add[molname]==0: + continue + fo1 = IMP.pmi.io.crosslink.FilterOperator(self.CrossLinkDataBase.protein1_key,operator.eq,molname) + self.CrossLinkDataBase.set_value(self.CrossLinkDataBase.protein1_key,molname+'.0',fo1) + fo2 = IMP.pmi.io.crosslink.FilterOperator(self.CrossLinkDataBase.protein2_key,operator.eq,molname) + self.CrossLinkDataBase.set_value(self.CrossLinkDataBase.protein2_key,molname+'.0',fo2) + for ncopy in range(copies_to_add[molname]): + self.CrossLinkDataBase.clone_protein('%s.0'%molname,'%s.%i'%(molname,ncopy+1)) + print('done pmi2 prelims') for xlid in self.CrossLinkDataBase.xlid_iterator(): new_contribution=True @@ -157,55 +141,39 @@ def __init__(self, representation=None, group), group) for p, group in zip(IMP.pmi.tools._all_protocol_outputs( - representations, root_hier), + None, root_hier), xl_groups)] - if use_pmi2: - iterlist = range(len(IMP.atom.get_by_type(root_hier,IMP.atom.STATE_TYPE))) - else: - iterlist = representations + iterlist = range(len(IMP.atom.get_by_type(root_hier, + IMP.atom.STATE_TYPE))) for nstate, r in enumerate(iterlist): # loop over every state xl[self.CrossLinkDataBase.state_key]=nstate xl[self.CrossLinkDataBase.data_set_name_key]=self.label - if use_pmi2: - name1 = c1 - name2 = c2 - copy1 = 0 - copy2 = 0 - if '.' in c1: - name1,copy1 = c1.split('.') - if '.' in c2: - name2,copy2 = c2.split('.') - ps1 = IMP.atom.Selection(root_hier, - state_index=nstate, - molecule=name1, - copy_index=int(copy1), - residue_index=r1, - resolution=resolution).get_selected_particles() - ps2 = IMP.atom.Selection(root_hier, - state_index=nstate, - molecule=name2, - copy_index=int(copy2), - residue_index=r2, - resolution=resolution).get_selected_particles() - - ps1 = [IMP.atom.Hierarchy(p) for p in ps1] - ps2 = [IMP.atom.Hierarchy(p) for p in ps2] - else: - ps1 = IMP.pmi.tools.select( - r, - resolution=resolution, - name=c1, - name_is_ambiguous=False, - residue=r1) - ps2 = IMP.pmi.tools.select( - r, - resolution=resolution, - name=c2, - name_is_ambiguous=False, - residue=r2) + name1 = c1 + name2 = c2 + copy1 = 0 + copy2 = 0 + if '.' in c1: + name1,copy1 = c1.split('.') + if '.' in c2: + name2,copy2 = c2.split('.') + ps1 = IMP.atom.Selection(root_hier, + state_index=nstate, + molecule=name1, + copy_index=int(copy1), + residue_index=r1, + resolution=resolution).get_selected_particles() + ps2 = IMP.atom.Selection(root_hier, + state_index=nstate, + molecule=name2, + copy_index=int(copy2), + residue_index=r2, + resolution=resolution).get_selected_particles() + + ps1 = [IMP.atom.Hierarchy(p) for p in ps1] + ps2 = [IMP.atom.Hierarchy(p) for p in ps2] if len(ps1) > 1: raise ValueError("residue %d of chain %s selects multiple particles %s" % (r1, c1, str(ps1))) @@ -291,7 +259,7 @@ def __init__(self, representation=None, print("CrossLinkingMassSpectrometryRestraint: between particles %s and %s" % (p1.get_name(), p2.get_name())) print("==========================================\n") for p, ex_xl in zip(IMP.pmi.tools._all_protocol_outputs( - representations, root_hier), + None, root_hier), ex_xls): p[0].add_cross_link(p[1], ex_xl[0], p1, p2, length, sigma1, sigma2, psi, ex_xl[1]) @@ -1047,523 +1015,8 @@ def get_output(self): return output -@IMP.deprecated_object("2.5", "Use IMP.pmi.restraints.crosslinking.CrossLinkingMassSpectrometryRestraint instead.") -class ISDCrossLinkMS(IMP.pmi.restraints._NuisancesBase): - def __init__(self, representation, - restraints_file, - length, - jackknifing=None, - # jackknifing (Float [0,1]) default=None; percentage of data to be - # removed randomly - resolution=None, - # resolution (Non-negative Integer) default=None; percentage of data - # to be removed randomly - slope=0.0, - inner_slope=0.0, - columnmapping=None, - rename_dict=None, - offset_dict=None, - csvfile=False, - ids_map=None, - radius_map=None, - filters=None, - label="None", - filelabel="None", - automatic_sigma_classification=False, - attributes_for_label=None): - - # columnindexes is a list of column indexes for protein1, protein2, residue1, residue2,idscore, XL unique id - # by default column 0 = protein1; column 1 = protein2; column 2 = residue1; column 3 = residue2; - # column 4 = idscores - # attributes_for_label: anything in the csv database that must be added to the label - # slope is the slope defined on the linear function - # inner_slope is the slope defined on the restraint directly - # suggestion: do not use both! - - if type(representation) != list: - representations = [representation] - else: - representations = representation - - if columnmapping is None: - columnmapping = {} - columnmapping["Protein1"] = 0 - columnmapping["Protein2"] = 1 - columnmapping["Residue1"] = 2 - columnmapping["Residue2"] = 3 - columnmapping["IDScore"] = 4 - columnmapping["XLUniqueID"] = 5 - - if csvfile: - # in case the file is a csv file - # columnmapping will contain the field names - # that compare in the first line of the csv file - db = IMP.pmi.tools.get_db_from_csv(restraints_file) - else: - db = IMP.pmi.tools.open_file_or_inline_text(restraints_file) - - indb = open("included." + filelabel + ".xl.db", "w") - exdb = open("excluded." + filelabel + ".xl.db", "w") - midb = open("missing." + filelabel + ".xl.db", "w") - - self.m = representations[0].prot.get_model() - self.rs = IMP.RestraintSet(self.m, 'data') - self.rspsi = IMP.RestraintSet(self.m, 'prior_psi') - self.rssig = IMP.RestraintSet(self.m, 'prior_sigmas') - self.rslin = IMP.RestraintSet(self.m, 'prior_linear') - self.rslen = IMP.RestraintSet(self.m, 'prior_length') - - self.weight = 1.0 - self.label = label - self.pairs = [] - self.sigma_dictionary = {} - self.psi_dictionary = {} - self.psi_is_sampled = True - self.sigma_is_sampled = True - - if os.path.exists(restraints_file): - l = ihm.location.InputFileLocation(restraints_file, - details="Crosslinks") - self.dataset = ihm.dataset.CXMSDataset(l) - else: - self.dataset = None - - xl_groups = [p.get_cross_link_group(self) - for p, state in representations[0]._protocol_output] - - # isd_map is a dictionary/map that is used to determine the psi - # parameter from identity scores (such as ID-Score, or FDR) - if ids_map is None: - self.ids_map = IMP.pmi.tools.map() - self.ids_map.set_map_element(20.0, 0.05) - self.ids_map.set_map_element(65.0, 0.01) - else: - self.ids_map = ids_map - - if radius_map is None: - self.radius_map = IMP.pmi.tools.map() - if automatic_sigma_classification: - self.radius_map.set_map_element(10, 10) - self.radius_map.set_map_element(1, 1) - else: - self.radius_map = radius_map - - self.outputlevel = "low" - - # small linear contribution for long range - self.linear = IMP.core.Linear(0, 0.0) - self.linear.set_slope(slope) - dps2 = IMP.core.DistancePairScore(self.linear) - - protein1 = columnmapping["Protein1"] - protein2 = columnmapping["Protein2"] - residue1 = columnmapping["Residue1"] - residue2 = columnmapping["Residue2"] - idscore = columnmapping["IDScore"] - try: - xluniqueid = columnmapping["XLUniqueID"] - except: - xluniqueid = None - - restraints = [] - - # we need this dictionary to create ambiguity (i.e., multistate) - # if one id is already present in the dictionary, add the term to the - # corresponding already generated restraint - - uniqueid_restraints_map = {} - - for nxl, entry in enumerate(db): - - if not jackknifing is None: - - # to be implemented - # the problem is that in the replica exchange - # you have to broadcast the same restraints to every - # replica - - raise NotImplementedError("jackknifing not yet implemented") - - if not csvfile: - tokens = entry.split() - if len(tokens)==0: - continue - - # skip character - if (tokens[0] == "#"): - continue - try: - r1 = int(tokens[residue1]) - c1 = tokens[protein1] - r2 = int(tokens[residue2]) - c2 = tokens[protein2] - - if offset_dict is not None: - if c1 in offset_dict: r1+=offset_dict[c1] - if c2 in offset_dict: r2+=offset_dict[c2] - - if rename_dict is not None: - if c1 in rename_dict: c1=rename_dict[c1] - if c2 in rename_dict: c2=rename_dict[c2] - - if idscore is None: - ids = 1.0 - else: - ids = float(tokens[idscore]) - if xluniqueid is None: - xlid = str(nxl) - else: - xlid = tokens[xluniqueid] - except: - print("this line was not accessible " + str(entry)) - if residue1 not in entry: print(str(residue1)+" keyword not in database") - if residue2 not in entry: print(str(residue2)+" keyword not in database") - if protein1 not in entry: print(str(protein1)+" keyword not in database") - if protein2 not in entry: print(str(protein2)+" keyword not in database") - if idscore not in entry: print(str(idscore)+" keyword not in database") - if xluniqueid not in entry: print(str(xluniqueid)+" keyword not in database") - continue - - else: - if filters is not None: - if eval(IMP.pmi.tools.cross_link_db_filter_parser(filters)) == False: - exdb.write(str(entry) + "\n") - continue - - try: - r1 = int(entry[residue1]) - c1 = entry[protein1] - r2 = int(entry[residue2]) - c2 = entry[protein2] - - if offset_dict is not None: - if c1 in offset_dict: r1+=offset_dict[c1] - if c2 in offset_dict: r2+=offset_dict[c2] - - if rename_dict is not None: - if c1 in rename_dict: c1=rename_dict[c1] - if c2 in rename_dict: c2=rename_dict[c2] - - if idscore is None: - ids = 1.0 - else: - try: - ids = float(entry[idscore]) - except ValueError: - ids = entry[idscore] - if xluniqueid is None: - xlid = str(nxl) - else: - xlid = entry[xluniqueid] - - except: - print("this line was not accessible " + str(entry)) - if residue1 not in entry: print(str(residue1)+" keyword not in database") - if residue2 not in entry: print(str(residue2)+" keyword not in database") - if protein1 not in entry: print(str(protein1)+" keyword not in database") - if protein2 not in entry: print(str(protein2)+" keyword not in database") - if idscore not in entry: print(str(idscore)+" keyword not in database") - if xluniqueid not in entry: print(str(xluniqueid)+" keyword not in database") - continue - - # todo: check that offset is handled correctly - ex_xls = [(p[0].add_experimental_cross_link(r1, c1, r2, c2, - group), group) - for p, group in zip(representations[0]._protocol_output, - xl_groups)] - - for nstate, r in enumerate(representations): - # loop over every state - - ps1 = IMP.pmi.tools.select( - r, - resolution=resolution, - name=c1, - name_is_ambiguous=False, - residue=r1) - ps2 = IMP.pmi.tools.select( - r, - resolution=resolution, - name=c2, - name_is_ambiguous=False, - residue=r2) - - if len(ps1) > 1: - raise ValueError("residue %d of chain %s selects multiple particles %s" % (r1, c1, str(ps1))) - elif len(ps1) == 0: - warnings.warn("ISDCrossLinkMS: residue %d of chain %s " - "is not there" % (r1, c1), - IMP.pmi.StructureWarning) - midb.write(str(entry) + "\n") - continue - - if len(ps2) > 1: - raise ValueError("residue %d of chain %s selects multiple particles %s" % (r2, c2, str(ps2))) - elif len(ps2) == 0: - warnings.warn("ISDCrossLinkMS: residue %d of chain %s " - "is not there" % (r2, c2), - IMP.pmi.StructureWarning) - midb.write(str(entry) + "\n") - continue - - p1 = ps1[0] - p2 = ps2[0] - - if (p1 == p2) and (r1 == r2) : - warnings.warn("ISDCrossLinkMS Restraint: on the identical " - "bead particles and the identical residues, " - "thus skipping this cross-link.", - IMP.pmi.StructureWarning) - continue - - if xlid in uniqueid_restraints_map: - print("Appending a crosslink restraint into the uniqueID %s" % str(xlid)) - dr = uniqueid_restraints_map[xlid] - else: - print("Generating a NEW crosslink restraint with a uniqueID %s" % str(xlid)) - dr = IMP.isd.CrossLinkMSRestraint( - self.m, - length, - inner_slope) - restraints.append(dr) - uniqueid_restraints_map[xlid] = dr - - mappedr1 = self.radius_map.get_map_element( - IMP.pmi.Uncertainty(p1).get_uncertainty()) - sigma1 = self.get_sigma(mappedr1)[0] - mappedr2 = self.radius_map.get_map_element( - IMP.pmi.Uncertainty(p2).get_uncertainty()) - sigma2 = self.get_sigma(mappedr2)[0] - - psival = self.ids_map.get_map_element(ids) - psi = self.get_psi(psival)[0] - - - p1i = p1.get_particle_index() - p2i = p2.get_particle_index() - s1i = sigma1.get_particle().get_index() - s2i = sigma2.get_particle().get_index() - - #print nstate, p1i, p2i, p1.get_name(), p2.get_name() - - psii = psi.get_particle().get_index() - - dr.add_contribution((p1i, p2i), (s1i, s2i), psii) - print("--------------") - print("ISDCrossLinkMS: generating cross-link restraint between") - print("ISDCrossLinkMS: residue %d of chain %s and residue %d of chain %s" % (r1, c1, r2, c2)) - print("ISDCrossLinkMS: with sigma1 %f sigma2 %f psi %s" % (mappedr1, mappedr2, psival)) - print("ISDCrossLinkMS: between particles %s and %s" % (p1.get_name(), p2.get_name())) - print("==========================================\n") - indb.write(str(entry) + "\n") - for p, ex_xl in zip(representations[0]._protocol_output, - ex_xls): - p[0].add_cross_link(p[1], ex_xl[0], p1, p2, length, - sigma1, sigma2, psi, ex_xl[1]) - - # check if the two residues belong to the same rigid body - if(IMP.core.RigidMember.get_is_setup(p1) and - IMP.core.RigidMember.get_is_setup(p2) and - IMP.core.RigidMember(p1).get_rigid_body() == - IMP.core.RigidMember(p2).get_rigid_body()): - xlattribute = "intrarb" - else: - xlattribute = "interrb" - - if csvfile: - if not attributes_for_label is None: - for a in attributes_for_label: - xlattribute = xlattribute + "_" + str(entry[a]) - - xlattribute = xlattribute + "-State:" + str(nstate) - - dr.set_name( - xlattribute + "-" + c1 + ":" + str(r1) + "-" + c2 + ":" + str(r2) + "_" + self.label) - - if p1i != p2i: - pr = IMP.core.PairRestraint(self.m, dps2, (p1i, p2i)) - pr.set_name( - xlattribute + "-" + c1 + ":" + str(r1) + "-" + c2 + ":" + str(r2) + "_" + self.label) - self.rslin.add_restraint(pr) - - - self.pairs.append( - (p1, - p2, - dr, - r1, - c1, - r2, - c2, - xlattribute, - mappedr1, - mappedr2, - psival, - xlid, - nstate, - ids)) - - lw = IMP.isd.LogWrapper(restraints, self.weight) - self.rs.add_restraint(lw) - - # Provide self.model for compatibility with newer code - model = property(lambda s: s.m) - - def set_weight(self, weight): - self.weight = weight - self.rs.set_weight(weight) - - def set_slope_linear_term(self, slope): - self.linear.set_slope(slope) - - def set_label(self, label): - self.label = label - - def add_to_model(self): - IMP.pmi.tools.add_restraint_to_model(self.m, self.rs) - IMP.pmi.tools.add_restraint_to_model(self.m, self.rspsi) - IMP.pmi.tools.add_restraint_to_model(self.m, self.rssig) - IMP.pmi.tools.add_restraint_to_model(self.m, self.rslen) - IMP.pmi.tools.add_restraint_to_model(self.m, self.rslin) - - def get_hierarchies(self): - return self.prot - - def get_restraint_sets(self): - return self.rs - - def get_restraint(self): - return self.rs - - def get_restraint_for_rmf(self): - return self.rslin - - def get_restraints(self): - rlist = [] - for r in self.rs.get_restraints(): - rlist.append(IMP.core.PairRestraint.get_from(r)) - return rlist - - def get_particle_pairs(self): - ppairs = [] - for i in range(len(self.pairs)): - p0 = self.pairs[i][0] - p1 = self.pairs[i][1] - ppairs.append((p0, p1)) - return ppairs - - def set_output_level(self, level="low"): - # this might be "low" or "high" - self.outputlevel = level - - def set_psi_is_sampled(self, is_sampled=True): - self.psi_is_sampled = is_sampled - - def set_sigma_is_sampled(self, is_sampled=True): - self.sigma_is_sampled = is_sampled - - def get_label(self,pairs_index): - resid1 = self.pairs[pairs_index][3] - chain1 = self.pairs[pairs_index][4] - resid2 = self.pairs[pairs_index][5] - chain2 = self.pairs[pairs_index][6] - attribute = self.pairs[pairs_index][7] - rad1 = self.pairs[pairs_index][8] - rad2 = self.pairs[pairs_index][9] - psi = self.pairs[pairs_index][10] - xlid= self.pairs[pairs_index][11] - label = attribute + "-" + \ - str(resid1) + ":" + chain1 + "_" + str(resid2) + ":" + \ - chain2 + "-" + str(rad1) + "-" + str(rad2) + "-" + str(psi) - return label - - def write_db(self,filename): - cldb=IMP.pmi.output.CrossLinkIdentifierDatabase() - - for pairs_index in range(len(self.pairs)): - - resid1 = self.pairs[pairs_index][3] - chain1 = self.pairs[pairs_index][4] - resid2 = self.pairs[pairs_index][5] - chain2 = self.pairs[pairs_index][6] - attribute = self.pairs[pairs_index][7] - rad1 = self.pairs[pairs_index][8] - rad2 = self.pairs[pairs_index][9] - psi = self.pairs[pairs_index][10] - xlid= self.pairs[pairs_index][11] - nstate=self.pairs[pairs_index][12] - ids=self.pairs[pairs_index][13] - - label=self.get_label(pairs_index) - cldb.set_unique_id(label,xlid) - cldb.set_protein1(label,chain1) - cldb.set_protein2(label,chain2) - cldb.set_residue1(label,resid1) - cldb.set_residue2(label,resid2) - cldb.set_idscore(label,ids) - cldb.set_state(label,nstate) - cldb.set_sigma1(label,rad1) - cldb.set_sigma2(label,rad2) - cldb.set_psi(label,psi) - cldb.write(filename) - - - def get_output(self): - # content of the crosslink database pairs - # self.pairs.append((p1,p2,dr,r1,c1,r2,c2)) - output = {} - score = self.weight * self.rs.unprotected_evaluate(None) - output["_TotalScore"] = str(score) - output["ISDCrossLinkMS_Data_Score_" + self.label] = str(score) - output["ISDCrossLinkMS_PriorSig_Score_" + - self.label] = self.rssig.unprotected_evaluate(None) - output["ISDCrossLinkMS_PriorPsi_Score_" + - self.label] = self.rspsi.unprotected_evaluate(None) - output["ISDCrossLinkMS_Linear_Score_" + - self.label] = self.rslin.unprotected_evaluate(None) - for i in range(len(self.pairs)): - - label=self.get_label(i) - ln = self.pairs[i][2] - p0 = self.pairs[i][0] - p1 = self.pairs[i][1] - output["ISDCrossLinkMS_Score_" + - label + "_" + self.label] = str(self.weight * -log(ln.unprotected_evaluate(None))) - - d0 = IMP.core.XYZ(p0) - d1 = IMP.core.XYZ(p1) - output["ISDCrossLinkMS_Distance_" + - label + "_" + self.label] = str(IMP.core.get_distance(d0, d1)) - - - for psiindex in self.psi_dictionary: - output["ISDCrossLinkMS_Psi_" + - str(psiindex) + "_" + self.label] = str(self.psi_dictionary[psiindex][0].get_scale()) - - for resolution in self.sigma_dictionary: - output["ISDCrossLinkMS_Sigma_" + - str(resolution) + "_" + self.label] = str(self.sigma_dictionary[resolution][0].get_scale()) - - - return output - - def get_particles_to_sample(self): - ps = {} - for resolution in self.sigma_dictionary: - if self.sigma_dictionary[resolution][2] and self.sigma_is_sampled: - ps["Nuisances_ISDCrossLinkMS_Sigma_" + str(resolution) + "_" + self.label] =\ - ([self.sigma_dictionary[resolution][0]], - self.sigma_dictionary[resolution][1]) - if self.psi_is_sampled: - for psiindex in self.psi_dictionary: - if self.psi_dictionary[psiindex][2]: - ps["Nuisances_ISDCrossLinkMS_Psi_" + - str(psiindex) + "_" + self.label] = ([self.psi_dictionary[psiindex][0]], self.psi_dictionary[psiindex][1]) - return ps - -# class CysteineCrossLinkRestraint(object): - def __init__(self, representations, filename, cbeta=False, + def __init__(self, root_hier, filename, cbeta=False, betatuple=(0.03, 0.1), disttuple=(0.0, 25.0, 1000), omegatuple=(1.0, 1000.0, 50), @@ -1578,8 +1031,7 @@ def __init__(self, representations, filename, cbeta=False, # residue pair, eg, "Epsilon-Intra-Solvent", or # "Epsilon-Solvent-Membrane", etc. - self.representations = representations - self.m = self.representations[0].prot.get_model() + self.m = root_hier.get_model() self.rs = IMP.RestraintSet(self.m, 'Cysteine_Crosslink') self.cbeta = cbeta self.epsilonmaxtrans = 0.01 @@ -1618,7 +1070,7 @@ def __init__(self, representations, filename, cbeta=False, # population particle self.weight = IMP.pmi.tools.SetupWeight( self.m, - weightissampled).get_particle( + isoptimized=False).get_particle( ) # read the file @@ -1687,78 +1139,84 @@ def __init__(self, representations, filename, cbeta=False, ccldata) failed = False - for i, representation in enumerate(self.representations): - - if not self.cbeta: - p1 = None - p2 = None - - p1 = IMP.pmi.tools.select(representation, - resolution=1, name=chain1, - name_is_ambiguous=False, residue=resid1)[0] + if not self.cbeta: + p1 = None + p2 = None + + p1 = IMP.atom.Selection(root_hier, resolution=1, + molecule=chain1, residue_index=resid1, + copy_index=0) + p1 = p1.get_selected_particles() + if len(p1) > 0: + p1 = p1[0] + else: + failed = True + + p2 = IMP.atom.Selection(root_hier, resolution=1, + molecule=chain2, residue_index=resid2, + copy_index=0) + p2 = p2.get_selected_particles() + if len(p2) > 0: + p2 = p2[0] + else: + failed = True - if p1 is None: + else: + # use cbetas + p1 = [] + p2 = [] + for t in range(-1, 2): + p = IMP.atom.Selection(root_hier, resolution=1, + molecule=chain1, copy_index=0, + residue_index=resid1 + t) + p = p.get_selected_particles() + if len(p) == 1: + p1 += p + else: + failed = True + print("\033[93m CysteineCrossLink: missing representation for residue %d of chain %s \033[0m" % (resid1 + t, chain1)) + + p = IMP.atom.Selection(root_hier, resolution=1, + molecule=chain2, copy_index=0, + residue_index=resid2 + t) + p = p.get_selected_particles() + if len(p) == 1: + p2 += p + else: failed = True + print("\033[93m CysteineCrossLink: missing representation for residue %d of chain %s \033[0m" % (resid2 + t, chain2)) - p2 = IMP.pmi.tools.select(representation, - resolution=1, name=chain2, - name_is_ambiguous=False, residue=resid2)[0] + if not self.cbeta: + if (p1 is not None and p2 is not None): + ccl.add_contribution(p1, p2) + d1 = IMP.core.XYZ(p1) + d2 = IMP.core.XYZ(p2) - if p2 is None: - failed = True + print("Distance_" + str(resid1) + "_" + chain1 + ":" + str(resid2) + "_" + chain2, IMP.core.get_distance(d1, d2)) - else: - # use cbetas - p1 = [] - p2 = [] - for t in range(-1, 2): - p = IMP.pmi.tools.select(representation, - resolution=1, name=chain1, - name_is_ambiguous=False, residue=resid1 + t) - - if len(p) == 1: - p1 += p - else: - failed = True - print("\033[93m CysteineCrossLink: missing representation for residue %d of chain %s \033[0m" % (resid1 + t, chain1)) - - p = IMP.pmi.tools.select(representation, - resolution=1, name=chain2, - name_is_ambiguous=False, residue=resid2 + t) - - if len(p) == 1: - p2 += p - else: - failed = True - print("\033[93m CysteineCrossLink: missing representation for residue %d of chain %s \033[0m" % (resid2 + t, chain2)) - - if not self.cbeta: - if (p1 is not None and p2 is not None): - ccl.add_contribution(p1, p2) - d1 = IMP.core.XYZ(p1) - d2 = IMP.core.XYZ(p2) - - print("Distance_" + str(resid1) + "_" + chain1 + ":" + str(resid2) + "_" + chain2, IMP.core.get_distance(d1, d2)) + else: + if (len(p1) == 3 and len(p2) == 3): + p11n = p1[0].get_name() + p12n = p1[1].get_name() + p13n = p1[2].get_name() + p21n = p2[0].get_name() + p22n = p2[1].get_name() + p23n = p2[2].get_name() - else: - if (len(p1) == 3 and len(p2) == 3): - p11n = p1[0].get_name() - p12n = p1[1].get_name() - p13n = p1[2].get_name() - p21n = p2[0].get_name() - p22n = p2[1].get_name() - p23n = p2[2].get_name() + print("CysteineCrossLink: generating CB cysteine cross-link restraint between") + print("CysteineCrossLink: residue %d of chain %s and residue %d of chain %s" % (resid1, chain1, resid2, chain2)) + print("CysteineCrossLink: between particles %s %s %s and %s %s %s" % (p11n, p12n, p13n, p21n, p22n, p23n)) - print("CysteineCrossLink: generating CB cysteine cross-link restraint between") - print("CysteineCrossLink: residue %d of chain %s and residue %d of chain %s" % (resid1, chain1, resid2, chain2)) - print("CysteineCrossLink: between particles %s %s %s and %s %s %s" % (p11n, p12n, p13n, p21n, p22n, p23n)) + ccl.add_contribution(p1, p2) - ccl.add_contribution(p1, p2) + if not failed: + self.rs.add_restraint(ccl) + ccl.set_name("CysteineCrossLink_" + str(resid1) + + "_" + chain1 + ":" + str(resid2) + "_" + chain2) - if not failed: - self.rs.add_restraint(ccl) - ccl.set_name("CysteineCrossLink_" + str(resid1) - + "_" + chain1 + ":" + str(resid2) + "_" + chain2) + IMP.isd.Weight( + self.weight.get_particle() + ).set_weights_are_optimized(weightissampled) def set_label(self, label): self.label = label @@ -1819,10 +1277,157 @@ def get_output(self): rst).get_name( ) + "_" + self.label] = IMP.isd.CysteineCrossLinkRestraint.get_from(rst).get_standard_error() - if len(self.representations) > 1: - for i in range(len(self.prots)): - output["CysteineCrossLinkRestraint_Frequency_Contribution_" + - IMP.isd.CysteineCrossLinkRestraint.get_from(rst).get_name() + - "_State_" + str(i) + "_" + self.label] = IMP.isd.CysteineCrossLinkRestraint.get_from(rst).get_frequencies()[i] + return output + + +class DisulfideCrossLinkRestraint(object): + def __init__(self, representation_or_hier, + selection_tuple1, + selection_tuple2, + length=6.5, + resolution=1, + slope=0.01, + label="None"): + + self.m = representation_or_hier.get_model() + ps1 = IMP.pmi.tools.select_by_tuple_2(representation_or_hier, + selection_tuple1, + resolution=resolution) + ps2 = IMP.pmi.tools.select_by_tuple_2(representation_or_hier, + selection_tuple2, + resolution=resolution) + + self.rs = IMP.RestraintSet(self.m, 'likelihood') + self.rslin = IMP.RestraintSet(self.m, 'linear_dummy_restraints') + + # dummy linear restraint used for Chimera display + self.linear = IMP.core.Linear(0, 0.0) + self.linear.set_slope(0.0) + dps2 = IMP.core.DistancePairScore(self.linear) + + self.label = label + self.psi_dictionary={} + self.sigma_dictionary={} + self.psi_is_sampled = False + self.sigma_is_sampled = False + self.xl={} + + if len(ps1) > 1 or len(ps1) == 0: + raise ValueError("DisulfideBondRestraint: ERROR> first selection pattern selects multiple particles or sero particles") + + if len(ps2) > 1 or len(ps2) == 0: + raise ValueError("DisulfideBondRestraint: ERROR> second selection pattern selects multiple particles or sero particles") + + p1 = ps1[0] + p2 = ps2[0] + + sigma=self.create_sigma("SIGMA_DISULFIDE_BOND") + psi=self.create_psi("PSI_DISULFIDE_BOND") + + p1i = p1.get_index() + p2i = p2.get_index() + si = sigma.get_particle().get_index() + psii = psi.get_particle().get_index() + + dr = IMP.isd.CrossLinkMSRestraint( + self.m, + length, + slope) + + dr.add_contribution((p1i, p2i), (si, si), psii) + + if p1i != p2i: + pr = IMP.core.PairRestraint(self.m, dps2, (p1i, p2i)) + pr.set_name("DISULFIDE_BOND_"+self.label) + self.rslin.add_restraint(pr) + + lw = IMP.isd.LogWrapper([dr],1.0) + self.rs.add_restraint(lw) + + self.xl["Particle1"]=p1 + self.xl["Particle2"]=p2 + self.xl["Sigma"]=sigma + self.xl["Psi"]=psi + + def add_to_model(self): + IMP.pmi.tools.add_restraint_to_model(self.m, self.rs) + IMP.pmi.tools.add_restraint_to_model(self.m, self.rslin) + + def get_hierarchies(self): + return self.prot + def get_restraint_sets(self): + return self.rs + + def get_restraint(self): + return self.rs + + def get_restraint_for_rmf(self): + return self.rslin + + def get_restraints(self): + rlist = [] + for r in self.rs.get_restraints(): + rlist.append(IMP.core.PairRestraint.get_from(r)) + return rlist + + def set_psi_is_sampled(self, is_sampled=True): + self.psi_is_sampled = is_sampled + + def set_sigma_is_sampled(self, is_sampled=True): + self.sigma_is_sampled = is_sampled + + + def create_sigma(self, name): + ''' a nuisance on the structural uncertainty ''' + if name in self.sigma_dictionary: + return self.sigma_dictionary[name][0] + + sigmainit = 1.0 + sigmaminnuis = 0.0000001 + sigmamaxnuis = 1000.0 + sigmamin = 0.01 + sigmamax = 100.0 + sigmatrans = 0.5 + sigma = IMP.pmi.tools.SetupNuisance(self.m, sigmainit, + sigmaminnuis, sigmamaxnuis, self.sigma_is_sampled).get_particle() + self.sigma_dictionary[name] = ( + sigma, + sigmatrans, + self.sigma_is_sampled) + + return sigma + + def create_psi(self, name): + ''' a nuisance on the inconsistency ''' + if name in self.psi_dictionary: + return self.psi_dictionary[name][0] + + psiinit=0.001 + psiminnuis = 0.0000001 + psimaxnuis = 0.4999999 + psimin = 0.01 + psimax = 0.49 + psitrans = 0.1 + psi = IMP.pmi.tools.SetupNuisance(self.m, psiinit, + psiminnuis, psimaxnuis, + self.psi_is_sampled).get_particle() + self.psi_dictionary[name] = ( + psi, + psitrans, + self.psi_is_sampled) + + return psi + + + def get_output(self): + output = {} + score = self.rs.unprotected_evaluate(None) + output["_TotalScore"] = str(score) + output["DisulfideBondRestraint_Data_Score_" + self.label] = str(score) + output["DisulfideBondRestraint_Linear_Score_" + + self.label] = self.rslin.unprotected_evaluate(None) return output + + def get_particles_to_sample(self): + raise NotImplementedError(" ") diff --git a/modules/pmi/pyext/src/restraints/crosslinking_new.py b/modules/pmi/pyext/src/restraints/crosslinking_new.py index e916d30df8..db285e8bbf 100644 --- a/modules/pmi/pyext/src/restraints/crosslinking_new.py +++ b/modules/pmi/pyext/src/restraints/crosslinking_new.py @@ -1,171 +1,11 @@ """@namespace IMP.pmi.restraints.crosslinking_new -Restraints for handling crosslinking data. This temporary module will be soon -deprecated and merged with IMP.pmi.restraints.crosslinking +Restraints for handling crosslinking data. This temporary module has been +merged with IMP.pmi.restraints.crosslinking and will be removed in the next +IMP release. """ -from __future__ import print_function import IMP -import IMP.core -import IMP.algebra -import IMP.atom -import IMP.container -import IMP.pmi.tools -import IMP.pmi.restraints.crosslinking -import pdb +from IMP.pmi.restraints.crosslinking import DisulfideCrossLinkRestraint -class DisulfideCrossLinkRestraint(object): - def __init__(self, representation, - selection_tuple1, - selection_tuple2, - length=6.5, - resolution=1, - slope=0.01, - label="None"): - - self.m = representation.prot.get_model() - self.rs = IMP.RestraintSet(self.m, 'likelihood') - self.rslin = IMP.RestraintSet(self.m, 'linear_dummy_restraints') - - # dummy linear restraint used for Chimera display - self.linear = IMP.core.Linear(0, 0.0) - self.linear.set_slope(0.0) - dps2 = IMP.core.DistancePairScore(self.linear) - - self.label = label - self.psi_dictionary={} - self.sigma_dictionary={} - self.psi_is_sampled = False - self.sigma_is_sampled = False - self.xl={} - - ps1 = IMP.pmi.tools.select_by_tuple( - representation, - selection_tuple1, - resolution=resolution) - - ps2 = IMP.pmi.tools.select_by_tuple( - representation, - selection_tuple2, - resolution=resolution) - - if len(ps1) > 1 or len(ps1) == 0: - raise ValueError("DisulfideBondRestraint: ERROR> first selection pattern selects multiple particles or sero particles") - - if len(ps2) > 1 or len(ps2) == 0: - raise ValueError("DisulfideBondRestraint: ERROR> second selection pattern selects multiple particles or sero particles") - - p1 = ps1[0] - p2 = ps2[0] - - - - sigma=self.create_sigma("SIGMA_DISULFIDE_BOND") - psi=self.create_psi("PSI_DISULFIDE_BOND") - - p1i = p1.get_particle_index() - p2i = p2.get_particle_index() - si = sigma.get_particle().get_index() - psii = psi.get_particle().get_index() - - dr = IMP.isd.CrossLinkMSRestraint( - self.m, - length, - slope) - - dr.add_contribution((p1i, p2i), (si, si), psii) - - if p1i != p2i: - pr = IMP.core.PairRestraint(self.m, dps2, (p1i, p2i)) - pr.set_name("DISULFIDE_BOND_"+self.label) - self.rslin.add_restraint(pr) - - lw = IMP.isd.LogWrapper([dr],1.0) - self.rs.add_restraint(lw) - - self.xl["Particle1"]=p1 - self.xl["Particle2"]=p2 - self.xl["Sigma"]=sigma - self.xl["Psi"]=psi - - def add_to_model(self): - IMP.pmi.tools.add_restraint_to_model(self.m, self.rs) - IMP.pmi.tools.add_restraint_to_model(self.m, self.rslin) - - def get_hierarchies(self): - return self.prot - - def get_restraint_sets(self): - return self.rs - - def get_restraint(self): - return self.rs - - def get_restraint_for_rmf(self): - return self.rslin - - def get_restraints(self): - rlist = [] - for r in self.rs.get_restraints(): - rlist.append(IMP.core.PairRestraint.get_from(r)) - return rlist - - def set_psi_is_sampled(self, is_sampled=True): - self.psi_is_sampled = is_sampled - - def set_sigma_is_sampled(self, is_sampled=True): - self.sigma_is_sampled = is_sampled - - - def create_sigma(self, name): - ''' a nuisance on the structural uncertainty ''' - if name in self.sigma_dictionary: - return self.sigma_dictionary[name][0] - - sigmainit = 1.0 - sigmaminnuis = 0.0000001 - sigmamaxnuis = 1000.0 - sigmamin = 0.01 - sigmamax = 100.0 - sigmatrans = 0.5 - sigma = IMP.pmi.tools.SetupNuisance(self.m, sigmainit, - sigmaminnuis, sigmamaxnuis, self.sigma_is_sampled).get_particle() - self.sigma_dictionary[name] = ( - sigma, - sigmatrans, - self.sigma_is_sampled) - - return sigma - - def create_psi(self, name): - ''' a nuisance on the inconsistency ''' - if name in self.psi_dictionary: - return self.psi_dictionary[name][0] - - psiinit=0.001 - psiminnuis = 0.0000001 - psimaxnuis = 0.4999999 - psimin = 0.01 - psimax = 0.49 - psitrans = 0.1 - psi = IMP.pmi.tools.SetupNuisance(self.m, psiinit, - psiminnuis, psimaxnuis, - self.psi_is_sampled).get_particle() - self.psi_dictionary[name] = ( - psi, - psitrans, - self.psi_is_sampled) - - return psi - - - def get_output(self): - output = {} - score = self.rs.unprotected_evaluate(None) - output["_TotalScore"] = str(score) - output["DisulfideBondRestraint_Data_Score_" + self.label] = str(score) - output["DisulfideBondRestraint_Linear_Score_" + - self.label] = self.rslin.unprotected_evaluate(None) - return output - - def get_particles_to_sample(self): - raise NotImplementedError(" ") +IMP.deprecated_module("2.12", __name__, + "Use IMP.pmi.restraints.crosslinking instead") diff --git a/modules/pmi/pyext/src/restraints/em.py b/modules/pmi/pyext/src/restraints/em.py index 46bb2bac90..a7d0208d30 100644 --- a/modules/pmi/pyext/src/restraints/em.py +++ b/modules/pmi/pyext/src/restraints/em.py @@ -245,12 +245,7 @@ def center_target_density_on_origin(self): # IMP.pmi.tools.translate_hierarchies(self.densities,v) def center_model_on_target_density(self, input_object): - if type(input_object) is IMP.pmi.representation.Representation: - hier = input_object.prot - elif type(input_object) is IMP.pmi.topology.State: - hier = input_object.get_hierarchy() - else: - raise Exception("Input must be a Representation or topology.State object") + hier = input_object.get_hierarchy() target_com = self.get_center_of_mass() print('target com', target_com) model_com = self.get_center_of_mass(target=False) diff --git a/modules/pmi/pyext/src/restraints/proteomics.py b/modules/pmi/pyext/src/restraints/proteomics.py index 7a2f79b765..daadf33e9a 100644 --- a/modules/pmi/pyext/src/restraints/proteomics.py +++ b/modules/pmi/pyext/src/restraints/proteomics.py @@ -24,42 +24,25 @@ class ConnectivityRestraint(object): generate a connectivity restraint between domains setting up the restraint example: - cr=restraints.ConnectivityRestraint(simo,["CCC",(1,100,"TTT"),(100,150,"AAA")]) + sel1 = IMP.atom.Selection(root_hier, molecule="Rpb3", + residue_indexes=range(1,100)) + sel2 = IMP.atom.Selection(root_hier, molecule="Rpb4", + residue_indexes=range(1,100)) + cr=restraints.ConnectivityRestraint((sel1, sel2), label='CR1') cr.add_to_model() - cr.set_label("CR1") Multistate support =No - Selection type=selection tuple Resolution=Yes ''' - def __init__( - self, - representation, - selection_tuples, - kappa=10.0, - resolution=None, - label="None"): - + def __init__(self, domains, kappa=10.0, resolution=None, label="None"): self.weight = 1.0 self.kappa = kappa self.label = label - if self.label == "None": - self.label = str(selection_tuples) - self.m = representation.prot.get_model() - self.rs = IMP.RestraintSet(self.m, label) - - sels = [] - - for s in selection_tuples: - particles = IMP.pmi.tools.select_by_tuple(representation, s, - resolution=resolution, name_is_ambiguous=True) - sel = IMP.atom.Selection(particles) - sels.append(sel) cr = IMP.atom.create_connectivity_restraint( - sels, - self.kappa, - self.label) + domains, self.kappa, self.label) + self.m = cr.get_model() + self.rs = IMP.RestraintSet(self.m, label) self.rs.add_restraint(cr) def set_label(self, label): @@ -97,49 +80,31 @@ def get_output(self): class CompositeRestraint(object): ''' - handleparticles is a selection tuple - compositeparticles is a list of selection tuples + handleparticles a list of particles + compositeparticles is a list of list of particles ''' - def __init__( - self, - representation, - handleparticles_tuples, - compositeparticles_tuple_list, - cut_off=5.0, - lam=1.0, - plateau=0.0, - resolution=None, - label="None"): - + def __init__(self, handle_particles, composite_particles, cut_off=5.0, + lam=1.0, plateau=0.0, resolution=None, label="None"): # composite particles: all particles beside the handle self.label = label - self.m = representation.prot.get_model() + + hs = IMP.pmi.tools.input_adaptor(handle_particles, resolution, + flatten=True) + self.handleparticles = [h.get_particle() for h in hs] + self.m = self.handleparticles[0].get_model() self.rs = IMP.RestraintSet(self.m, 'cr') - self.handleparticles = [] - for s in handleparticles_tuples: - self.handleparticles += IMP.pmi.tools.select_by_tuple( - representation, s, - resolution=resolution, name_is_ambiguous=True) self.compositeparticles = [] compositeparticle_list = [] - for list in compositeparticles_tuple_list: - tmplist = [] - for s in list: - tmplist += IMP.pmi.tools.select_by_tuple( - representation, s, - resolution=resolution, name_is_ambiguous=True) + for cp in composite_particles: + hs = IMP.pmi.tools.input_adaptor(cp, resolution, flatten=True) + tmplist = [h.get_particle() for h in hs] compositeparticle_list.append(tmplist) self.compositeparticles += tmplist - ln = IMP.pmi.CompositeRestraint( - self.m, - self.handleparticles, - cut_off, - lam, - True, - plateau) + ln = IMP.pmi.CompositeRestraint(self.m, self.handleparticles, cut_off, + lam, True, plateau) for ps in compositeparticle_list: # composite particles is a list of list of particles @@ -179,18 +144,10 @@ class AmbiguousCompositeRestraint(object): It allows name ambiguity ''' - def __init__( - self, - representation, - restraints_file, - cut_off=5.0, - lam=1.0, - plateau=0.01, - resolution=None, - label="None"): - + def __init__(self, root_hier, restraints_file, cut_off=5.0, lam=1.0, + plateau=0.01, resolution=None, label="None"): self.weight = 1.0 - self.m = representation.prot.get_model() + self.m = root_hier.get_model() self.rs = IMP.RestraintSet(self.m, 'data') self.label = "None" self.pairs = [] @@ -213,19 +170,15 @@ def __init__( r2 = int(tokens[3]) c2 = tokens[1] - ps1 = IMP.pmi.tools.select( - representation, - resolution=resolution, - name=c1, - name_is_ambiguous=True, - residue=r1) - hrc1 = [representation.hier_db.particle_to_name[p] for p in ps1] - ps1nosym = [ - p for p in ps1 if IMP.pmi.Symmetric( - p).get_symmetric( - ) == 0] - hrc1nosym = [representation.hier_db.particle_to_name[p] - for p in ps1nosym] + ps1 = IMP.atom.Selection(root_hier, resolution=resolution, + molecule=c1, residue_index=r1) + ps1 = ps1.get_selected_particles() + hrc1 = [p.get_name() for p in ps1] + def nosym_subset(ps): + return [p for p in ps if not IMP.pmi.Symmetric.get_is_setup(p) + or IMP.pmi.Symmetric(p).get_symmetric() == 0] + ps1nosym = nosym_subset(ps1) + hrc1nosym = [p.get_name() for p in ps1nosym] if len(ps1) == 0: warnings.warn( @@ -233,19 +186,12 @@ def __init__( "is not there" % (r1, c1), IMP.pmi.StructureWarning) continue - ps2 = IMP.pmi.tools.select( - representation, - resolution=resolution, - name=c2, - name_is_ambiguous=True, - residue=r2) - hrc2 = [representation.hier_db.particle_to_name[p] for p in ps2] - ps2nosym = [ - p for p in ps2 if IMP.pmi.Symmetric( - p).get_symmetric( - ) == 0] - hrc2nosym = [representation.hier_db.particle_to_name[p] - for p in ps2nosym] + ps2 = IMP.atom.Selection(root_hier, resolution=resolution, + molecule=c2, residue_index=r2) + ps2 = ps2.get_selected_particles() + hrc2 = [p.get_name() for p in ps2] + ps2nosym = nosym_subset(ps2) + hrc2nosym = [p.get_name() for p in ps2nosym] if len(ps2) == 0: warnings.warn( @@ -254,12 +200,7 @@ def __init__( continue cr = IMP.pmi.CompositeRestraint( - self.m, - ps1nosym, - self.cut_off, - self.lam, - True, - self.plateau) + self.m, ps1nosym, self.cut_off, self.lam, True, self.plateau) cr.add_composite_particle(ps2) self.rs.add_restraint(cr) @@ -275,12 +216,7 @@ def __init__( cr)) cr = IMP.pmi.CompositeRestraint( - self.m, - ps1, - self.cut_off, - self.lam, - True, - self.plateau) + self.m, ps1, self.cut_off, self.lam, True, self.plateau) cr.add_composite_particle(ps2nosym) self.rs.add_restraint(cr) @@ -386,15 +322,9 @@ def get_output(self): # class SimplifiedPEMAP(object): - def __init__( - self, - representation, - restraints_file, - expdistance, - strength, - resolution=None): - - self.m = representation.prot.get_model() + def __init__(self, root_hier, restraints_file, expdistance, strength, + resolution=None): + self.m = root_hier.get_model() self.rs = IMP.RestraintSet(self.m, 'data') self.label = "None" self.pairs = [] @@ -417,12 +347,10 @@ def __init__( c2 = tokens[1] pcc = float(tokens[4]) - ps1 = IMP.pmi.tools.select( - representation, - resolution=resolution, - name=c1, - name_is_ambiguous=False, - residue=r1) + ps1 = IMP.atom.Selection(root_hier, resolution=resolution, + molecule=c1, residue_index=r1, + copy_index=0) + ps1 = ps1.get_selected_particles() if len(ps1) == 0: warnings.warn( "SimplifiedPEMAP: residue %d of chain %s is not there " @@ -434,12 +362,10 @@ def __init__( "multiple particles" % (r1, c1), IMP.pmi.StructureWarning) continue - ps2 = IMP.pmi.tools.select( - representation, - resolution=resolution, - name=c2, - name_is_ambiguous=False, - residue=r2) + ps2 = IMP.atom.Selection(root_hier, resolution=resolution, + molecule=c2, residue_index=r2, + copy_index=0) + ps2 = ps2.get_selected_particles() if len(ps2) == 0: warnings.warn( "SimplifiedPEMAP: residue %d of chain %s is not there " @@ -456,13 +382,9 @@ def __init__( # This is harmonic potential for the pE-MAP data upperdist = self.get_upper_bond(pcc) - limit = self.strength * (upperdist + 15) ** 2 + 10.0 + limit = 0.5 * self.strength * 15.0 ** 2 + 10.0 hub = IMP.core.TruncatedHarmonicUpperBound( - upperdist, - self.strength, - upperdist + - 15, - limit) + upperdist, self.strength, 15, limit) # This is harmonic for the X-link #hub= IMP.core.TruncatedHarmonicBound(17.0,self.strength,upperdist+15,limit) @@ -474,19 +396,15 @@ def __init__( # Lower-bound restraint lowerdist = self.get_lower_bond(pcc) - limit = self.strength * (lowerdist - 15) ** 2 + 10.0 + limit = 0.5 * self.strength * 15.0 ** 2 + 10.0 hub2 = IMP.core.TruncatedHarmonicLowerBound( - lowerdist, - self.strength, - lowerdist + - 15, - limit) + lowerdist, self.strength, 15, limit) # This is harmonic for the X-link #hub2= IMP.core.TruncatedHarmonicBound(17.0,self.strength,upperdist+15,limit) df2 = IMP.core.SphereDistancePairScore(hub2) - dr2 = IMP.core.PairRestraint(df2, (p1, p2)) + dr2 = IMP.core.PairRestraint(self.m, df2, (p1, p2)) self.rs.add_restraint(dr2) self.pairs.append((p1, p2, dr2, r1, c1, r2, c2)) diff --git a/modules/pmi/pyext/src/restraints/saxs.py b/modules/pmi/pyext/src/restraints/saxs.py index 87253f05c0..c4c7013fd2 100644 --- a/modules/pmi/pyext/src/restraints/saxs.py +++ b/modules/pmi/pyext/src/restraints/saxs.py @@ -136,7 +136,7 @@ def __init__(self, representation, profile, resolution=0, weight=1, # weight, optimized self.w = IMP.pmi.tools.SetupWeight(self.model).get_particle() - IMP.isd.Weight(self.w).set_weights_are_optimized(True) + IMP.isd.Weight(self.w).set_weights_are_optimized(False) # take identity covariance matrix for the start self.cov = [[1 if i == j else 0 for j in range(self.prof.size())] @@ -162,6 +162,10 @@ def __init__(self, representation, profile, resolution=0, weight=1, j3 = IMP.isd.JeffreysRestraint(self.model, self.gamma) self.rs2.add_restraint(j3) + pw = IMP.isd.Weight(self.w) + pw.set_weights(pw.get_unit_simplex().get_barycenter()) + pw.set_weights_are_optimized(True) + def optimize_sigma(self): """Set sigma to the value that maximizes its conditional likelihood""" self.model.update() diff --git a/modules/pmi/pyext/src/restraints/stereochemistry.py b/modules/pmi/pyext/src/restraints/stereochemistry.py index 6b7e5399e8..704fac311d 100644 --- a/modules/pmi/pyext/src/restraints/stereochemistry.py +++ b/modules/pmi/pyext/src/restraints/stereochemistry.py @@ -11,7 +11,6 @@ import IMP.isd import itertools import IMP.pmi.tools -import IMP.pmi.representation from operator import itemgetter from math import pi,log,sqrt import sys @@ -217,19 +216,8 @@ def __init__(self, included_ps = [h.get_particle() for h in hierarchies] if bipartite: other_ps = [h.get_particle() for h in other_hierarchies] - elif isinstance(representation, IMP.pmi.representation.Representation): - self.mdl = representation.model - included_ps = IMP.pmi.tools.select( - representation, - resolution=resolution, - hierarchies=hierarchies) - if bipartite: - other_ps = IMP.pmi.tools.select( - representation, - resolution=resolution, - hierarchies=other_hierarchies) else: - raise Exception("Must pass Representation or included_objects") + raise Exception("Must pass included_objects") # setup score self.rs = IMP.RestraintSet(self.mdl, 'excluded_volume') @@ -402,7 +390,7 @@ def __init__(self, raise ValueError("wrong length of pair") for p in ps: if not IMP.atom.Residue.get_is_setup(p): - raise TypeError("not a residue") + raise TypeError("%s is not a residue" % p) else: pair.append(p) print("ResidueBondRestraint: adding a restraint between %s %s" % (pair[0].get_name(), pair[1].get_name())) @@ -477,7 +465,7 @@ def __init__(self, raise ValueError("wrong length of triplet") for p in ps: if not IMP.atom.Residue.get_is_setup(p): - raise TypeError("not a residue") + raise TypeError("%s is not a residue" % p) else: triplet.append(p) print("ResidueAngleRestraint: adding a restraint between %s %s %s" % (triplet[0].get_name(), triplet[1].get_name(), triplet[2].get_name())) @@ -556,7 +544,7 @@ def __init__( raise ValueError("wrong length of quadruplet") for p in ps: if not IMP.atom.Residue.get_is_setup(p): - raise TypeError("not a residue") + raise TypeError("%s is not a residue" % p) else: quadruplet.append(p) dihedraltype = stringsequence[n] @@ -875,17 +863,8 @@ def __init__(self,representation=None, copy_index=copy_index, atom_type=IMP.atom.AtomType("CA")) particles+=sel.get_selected_particles() - elif representation is not None and type(representation)==IMP.pmi.representation.Representation: - self.m = representation.model - for st in selection_tuples: - print('selecting with',st) - for p in IMP.pmi.tools.select_by_tuple(representation,st,resolution=resolution): - if (resolution==0 and ca_only and IMP.atom.Atom(p).get_atom_type()!=IMP.atom.AtomType("CA")): - continue - else: - particles.append(p.get_particle()) else: - raise Exception("must pass representation or hierarchy") + raise Exception("must pass hierarchy") self.weight = 1 self.label = "None" diff --git a/modules/pmi/pyext/src/samplers.py b/modules/pmi/pyext/src/samplers.py index 6339c13a06..299392bfa1 100644 --- a/modules/pmi/pyext/src/samplers.py +++ b/modules/pmi/pyext/src/samplers.py @@ -649,137 +649,3 @@ def get_percentile(self,name): ind=values.index(value) percentile=float(ind)/len(values) return percentile - - - -@IMP.deprecated_object(2.11, - "If you use this class please let the PMI developers know.") -class PyMCMover(object): - # only works if the sampled particles are rigid bodies - - def __init__(self, representation, mcchild, n_mc_steps): - - # mcchild must be pmi.samplers.MonteCarlo - # representation must be pmi.representation - - self.rbs = representation.get_rigid_bodies() - - self.mc = mcchild - self.n_mc_steps = n_mc_steps - - def store_move(self): - # get all xyz coordinates of all rigid bodies of all copies - self.oldcoords = [] - for copy in self.rbs: - crd = [] - for rb in copy: - crd.append(rb.get_reference_frame()) - self.oldcoords.append(crd) - - def propose_move(self, prob): - self.mc.run(self.n_mc_steps) - - def reset_move(self): - # reset each copy to crd - for copy, crd in zip(self.rbs, self.oldcoords): - for rb, ref in zip(copy, crd): - rb.set_reference_frame(ref) - - def get_number_of_steps(self): - return self.n_mc_steps - - def set_number_of_steps(self, nsteps): - self.n_mc_steps = nsteps - - -@IMP.deprecated_object(2.11, - "If you use this class please let the PMI developers know.") -class PyMC(object): - - def __init__(self, model): - from math import exp - import random - - self.model = model - self.restraints = None - self.first_call = True - self.nframe = -1 - - def add_mover(self, mv): - self.mv = mv - - def set_kt(self, kT): - self.kT = kT - - def set_return_best(self, thing): - pass - - def set_move_probability(self, thing): - pass - - def get_energy(self): - if self.restraints: - pot = sum([r.evaluate(False) for r in self.restraints]) - else: - pot = self.model.evaluate(False) - return pot - - def metropolis(self, old, new): - deltaE = new - old - print(": old %f new %f deltaE %f new_epot: %f" % (old, new, deltaE, - self.model.evaluate( - False)), end=' ') - kT = self.kT - if deltaE < 0: - return True - else: - return exp(-deltaE / kT) > random.uniform(0, 1) - - def optimize(self, nsteps): - self.naccept = 0 - self.nframe += 1 - print("=== new MC call") - # store initial coordinates - if self.first_call: - self.mv.store_move() - self.first_call = False - for i in range(nsteps): - print("MC step %d " % i, end=' ') - # get total energy - old = self.get_energy() - # make a MD move - self.mv.propose_move(1) - # get new total energy - new = self.get_energy() - if self.metropolis(old, new): - # move was accepted: store new conformation - self.mv.store_move() - self.naccept += 1 - print("accepted ") - else: - # move rejected: restore old conformation - self.mv.reset_move() - print(" ") - - def get_number_of_forward_steps(self): - return self.naccept - - def set_restraints(self, restraints): - self.restraints = restraints - - def set_scoring_function(self, objects): - # objects should be pmi.restraints - rs = IMP.RestraintSet(self.model, 1.0, 'sfo') - for ob in objects: - rs.add_restraint(ob.get_restraint()) - self.set_restraints([rs]) - - def get_output(self): - output = {} - acceptances = [] - output["PyMC_Temperature"] = str(self.kT) - output["PyMC_Nframe"] = str(self.nframe) - return output - - -# 3 diff --git a/modules/pmi/pyext/src/tools.py b/modules/pmi/pyext/src/tools.py index b62ae9ad2a..83274e4037 100644 --- a/modules/pmi/pyext/src/tools.py +++ b/modules/pmi/pyext/src/tools.py @@ -16,7 +16,10 @@ import sys,os import random import ast -import time +try: + from time import process_time # needs python 3.3 or later +except ImportError: + from time import clock as process_time import RMF import IMP.rmf from collections import defaultdict @@ -125,7 +128,7 @@ def __init__(self, isdelta=True): """Constructor. @param isdelta if True (the default) then report the time since the last use of this class; if False, report cumulative time.""" - self.starttime = time.clock() + self.starttime = process_time() self.label = "None" self.isdelta = isdelta @@ -135,13 +138,13 @@ def set_label(self, labelstr): def get_output(self): output = {} if self.isdelta: - newtime = time.clock() + newtime = process_time() output["Stopwatch_" + self.label + "_delta_seconds"] \ = str(newtime - self.starttime) self.starttime = newtime else: output["Stopwatch_" + self.label + "_elapsed_seconds"] \ - = str(time.clock() - self.starttime) + = str(process_time() - self.starttime) return output @@ -165,10 +168,21 @@ def get_particle(self): class SetupWeight(object): - def __init__(self, m, isoptimized=True): + def __init__(self, m, isoptimized=True, nweights_or_weights=None): pw = IMP.Particle(m) - self.weight = IMP.isd.Weight.setup_particle(pw) - self.weight.set_weights_are_optimized(True) + if isinstance(nweights_or_weights, int): + self.weight = IMP.isd.Weight.setup_particle( + pw, nweights_or_weights + ) + else: + try: + nweights_or_weights = list(nweights_or_weights) + self.weight = IMP.isd.Weight.setup_particle( + pw, nweights_or_weights + ) + except (TypeError, IMP.UsageException): + self.weight = IMP.isd.Weight.setup_particle(pw) + self.weight.set_weights_are_optimized(isoptimized) def get_particle(self): return self.weight @@ -245,35 +259,6 @@ def get_particles_to_sample(self): return ps -def get_random_cross_link_dataset(representation, - resolution=1.0, - number_of_cross_links=10, - ambiguity_probability=0.1, - confidence_score_range=[0,100], - avoid_same_particles=False): - '''Return a random cross-link dataset as a string. - Every line is a residue pair, together with UniqueIdentifier - and XL score.''' - - residue_pairs=get_random_residue_pairs(representation, resolution, number_of_cross_links, avoid_same_particles=avoid_same_particles) - - unique_identifier=0 - cmin=float(min(confidence_score_range)) - cmax=float(max(confidence_score_range)) - - dataset="#\n" - - for (name1, r1, name2, r2) in residue_pairs: - if random.random() > ambiguity_probability: - unique_identifier+=1 - score=random.random()*(cmax-cmin)+cmin - dataset+=str(name1)+" "+str(name2)+" "+str(r1)+" "+str(r2)+" "+str(score)+" "+str(unique_identifier)+"\n" - - return dataset - - - #------------------------------- - def get_cross_link_data(directory, filename, dist, omega, sigma, don=None, doff=None, prior=0, type_of_profile="gofr"): @@ -418,52 +403,6 @@ def get_closest_residue_position(hier, resindex, terminus="N"): else: raise ValueError("got multiple residues for hierarchy %s and residue %i; the list of particles is %s" % (hier, resindex, str([pp.get_name() for pp in p]))) -def get_terminal_residue(representation, hier, terminus="C", resolution=1): - ''' - Get the particle of the terminal residue at the GIVEN resolution - (NOTE: not the closest resolution!). - To get the terminal residue at the closest resolution use: - particles=IMP.pmi.tools.select_by_tuple(representation,molecule_name) - particles[0] and particles[-1] will be the first and last particles - corresponding to the two termini. - It is needed for instance to determine the last residue of a pdb. - @param hier hierarchy containing the terminal residue - @param terminus either 'N' or 'C' - @param resolution resolution to use. - ''' - termresidue = None - termparticle = None - - ps=select(representation, - resolution=resolution, - hierarchies=[hier]) - - for p in ps: - if IMP.pmi.Resolution(p).get_resolution() == resolution: - residues = IMP.pmi.tools.get_residue_indexes(p) - if terminus == "C": - if termresidue is None: - termresidue = max(residues) - termparticle = p - elif max(residues) >= termresidue: - termresidue = max(residues) - termparticle = p - elif terminus == "N": - if termresidue is None: - termresidue = min(residues) - termparticle = p - elif min(residues) <= termresidue: - termresidue = min(residues) - termparticle = p - else: - raise ValueError("terminus argument should be either N or C") - return termparticle - -def get_terminal_residue_position(representation, hier, terminus="C", - resolution=1): - """Get XYZ coordinates of the terminal residue at the GIVEN resolution""" - p = get_terminal_residue(representation, hier, terminus, resolution) - return IMP.core.XYZ(p).get_coordinates() def get_residue_gaps_in_hierarchy(hierarchy, start, end): ''' @@ -542,125 +481,6 @@ def get_map_element(self, invalue): raise TypeError("wrong type for map") -def select(representation, - resolution=None, - hierarchies=None, - selection_arguments=None, - name=None, - name_is_ambiguous=False, - first_residue=None, - last_residue=None, - residue=None, - representation_type=None): - ''' - this function uses representation=SimplifiedModel - it returns the corresponding selected particles - representation_type="Beads", "Res:X", "Densities", "Representation", "Molecule" - ''' - - if resolution is None: - allparticles = IMP.atom.get_leaves(representation.prot) - resolution_particles = None - hierarchies_particles = None - names_particles = None - residue_range_particles = None - residue_particles = None - representation_type_particles = None - - if not resolution is None: - resolution_particles = [] - hs = representation.get_hierarchies_at_given_resolution(resolution) - for h in hs: - resolution_particles += IMP.atom.get_leaves(h) - - if not hierarchies is None: - hierarchies_particles = [] - for h in hierarchies: - hierarchies_particles += IMP.atom.get_leaves(h) - - if not name is None: - names_particles = [] - if name_is_ambiguous: - for namekey in representation.hier_dict: - if name in namekey: - names_particles += IMP.atom.get_leaves( - representation.hier_dict[namekey]) - elif name in representation.hier_dict: - names_particles += IMP.atom.get_leaves(representation.hier_dict[name]) - else: - print("select: component %s is not there" % name) - - if not first_residue is None and not last_residue is None: - sel = IMP.atom.Selection(representation.prot, - residue_indexes=range(first_residue, last_residue + 1)) - residue_range_particles = [IMP.atom.Hierarchy(p) - for p in sel.get_selected_particles()] - - if not residue is None: - sel = IMP.atom.Selection(representation.prot, residue_index=residue) - residue_particles = [IMP.atom.Hierarchy(p) - for p in sel.get_selected_particles()] - - if not representation_type is None: - representation_type_particles = [] - if representation_type == "Molecule": - for name in representation.hier_representation: - for repr_type in representation.hier_representation[name]: - if repr_type == "Beads" or "Res:" in repr_type: - h = representation.hier_representation[name][repr_type] - representation_type_particles += IMP.atom.get_leaves(h) - - elif representation_type == "PDB": - for name in representation.hier_representation: - for repr_type in representation.hier_representation[name]: - if repr_type == "Res:" in repr_type: - h = representation.hier_representation[name][repr_type] - representation_type_particles += IMP.atom.get_leaves(h) - - else: - for name in representation.hier_representation: - h = representation.hier_representation[ - name][ - representation_type] - representation_type_particles += IMP.atom.get_leaves(h) - - selections = [hierarchies_particles, names_particles, - residue_range_particles, residue_particles, representation_type_particles] - - if resolution is None: - selected_particles = set(allparticles) - else: - selected_particles = set(resolution_particles) - - for s in selections: - if not s is None: - selected_particles = (set(s) & selected_particles) - - return list(selected_particles) - - -def select_by_tuple( - representation, - tupleselection, - resolution=None, - name_is_ambiguous=False): - if isinstance(tupleselection, tuple) and len(tupleselection) == 3: - particles = IMP.pmi.tools.select(representation, resolution=resolution, - name=tupleselection[2], - first_residue=tupleselection[0], - last_residue=tupleselection[1], - name_is_ambiguous=name_is_ambiguous) - elif isinstance(tupleselection, str): - particles = IMP.pmi.tools.select(representation, resolution=resolution, - name=tupleselection, - name_is_ambiguous=name_is_ambiguous) - else: - raise ValueError('you passed something bad to select_by_tuple()') - # now order the result by residue number - particles = IMP.pmi.tools.sort_by_residues(particles) - - return particles - def select_by_tuple_2(hier,tuple_selection,resolution): """New tuple format: molname OR (start,stop,molname,copynum,statenum) Copy and state are optional. Can also use 'None' for them which will get all. @@ -711,154 +531,6 @@ def get_db_from_csv(csvfilename): return outputlist -class HierarchyDatabase(object): - """Store the representations for a system.""" - - def __init__(self): - self.db = {} - # this dictionary map a particle to its root hierarchy - self.root_hierarchy_dict = {} - self.preroot_fragment_hierarchy_dict = {} - self.particle_to_name = {} - self.model = None - - def add_name(self, name): - if name not in self.db: - self.db[name] = {} - - def add_residue_number(self, name, resn): - resn = int(resn) - self.add_name(name) - if resn not in self.db[name]: - self.db[name][resn] = {} - - def add_resolution(self, name, resn, resolution): - resn = int(resn) - resolution = float(resolution) - self.add_name(name) - self.add_residue_number(name, resn) - if resolution not in self.db[name][resn]: - self.db[name][resn][resolution] = [] - - def add_particles(self, name, resn, resolution, particles): - resn = int(resn) - resolution = float(resolution) - self.add_name(name) - self.add_residue_number(name, resn) - self.add_resolution(name, resn, resolution) - self.db[name][resn][resolution] += particles - for p in particles: - (rh, prf) = self.get_root_hierarchy(p) - self.root_hierarchy_dict[p] = rh - self.preroot_fragment_hierarchy_dict[p] = prf - self.particle_to_name[p] = name - if self.model is None: - self.model = particles[0].get_model() - - def get_model(self): - return self.model - - def get_names(self): - names = list(self.db.keys()) - names.sort() - return names - - def get_particles(self, name, resn, resolution): - resn = int(resn) - resolution = float(resolution) - return self.db[name][resn][resolution] - - def get_particles_at_closest_resolution(self, name, resn, resolution): - resn = int(resn) - resolution = float(resolution) - closestres = min(self.get_residue_resolutions(name, resn), - key=lambda x: abs(float(x) - float(resolution))) - return self.get_particles(name, resn, closestres) - - def get_residue_resolutions(self, name, resn): - resn = int(resn) - resolutions = list(self.db[name][resn].keys()) - resolutions.sort() - return resolutions - - def get_molecule_resolutions(self, name): - resolutions = set() - for resn in self.db[name]: - resolutions.update(list(self.db[name][resn].keys())) - resolutions.sort() - return resolutions - - def get_residue_numbers(self, name): - residue_numbers = list(self.db[name].keys()) - residue_numbers.sort() - return residue_numbers - - def get_particles_by_resolution(self, name, resolution): - resolution = float(resolution) - particles = [] - for resn in self.get_residue_numbers(name): - result = self.get_particles_at_closest_resolution( - name, - resn, - resolution) - pstemp = [p for p in result if p not in particles] - particles += pstemp - return particles - - def get_all_particles_by_resolution(self, resolution): - resolution = float(resolution) - particles = [] - for name in self.get_names(): - particles += self.get_particles_by_resolution(name, resolution) - return particles - - def get_root_hierarchy(self, particle): - prerootfragment = particle - while IMP.atom.Atom.get_is_setup(particle) or \ - IMP.atom.Residue.get_is_setup(particle) or \ - IMP.atom.Fragment.get_is_setup(particle): - if IMP.atom.Atom.get_is_setup(particle): - p = IMP.atom.Atom(particle).get_parent() - elif IMP.atom.Residue.get_is_setup(particle): - p = IMP.atom.Residue(particle).get_parent() - elif IMP.atom.Fragment.get_is_setup(particle): - p = IMP.atom.Fragment(particle).get_parent() - prerootfragment = particle - particle = p - return ( - (IMP.atom.Hierarchy(particle), IMP.atom.Hierarchy(prerootfragment)) - ) - - def get_all_root_hierarchies_by_resolution(self, resolution): - hierarchies = [] - resolution = float(resolution) - particles = self.get_all_particles_by_resolution(resolution) - for p in particles: - rh = self.root_hierarchy_dict[p] - if rh not in hierarchies: - hierarchies.append(IMP.atom.Hierarchy(rh)) - return hierarchies - - def get_preroot_fragments_by_resolution(self, name, resolution): - fragments = [] - resolution = float(resolution) - particles = self.get_particles_by_resolution(name, resolution) - for p in particles: - fr = self.preroot_fragment_hierarchy_dict[p] - if fr not in fragments: - fragments.append(fr) - return fragments - - def show(self, name): - print(name) - for resn in self.get_residue_numbers(name): - print(resn) - for resolution in self.get_residue_resolutions(name, resn): - print("----", resolution) - for p in self.get_particles(name, resn, resolution): - print("--------", p.get_name()) - - def get_prot_name_from_particle(p, list_of_names): '''Return the component name provided a particle and a list of names''' root = p @@ -1100,35 +772,6 @@ def log_normal_density_function(expected_value, sigma, x): ) -def get_random_residue_pairs(representation, resolution, - number, - max_distance=None, - avoid_same_particles=False, - names=None): - - particles = [] - if names is None: - names=list(representation.hier_dict.keys()) - - for name in names: - prot = representation.hier_dict[name] - particles += select(representation,name=name,resolution=resolution) - random_residue_pairs = [] - while len(random_residue_pairs)<=number: - p1 = random.choice(particles) - p2 = random.choice(particles) - if max_distance is not None and \ - core.get_distance(core.XYZ(p1), core.XYZ(p2)) > max_distance: - continue - r1 = random.choice(IMP.pmi.tools.get_residue_indexes(p1)) - r2 = random.choice(IMP.pmi.tools.get_residue_indexes(p2)) - if r1==r2 and avoid_same_particles: continue - name1 = representation.get_prot_name_from_particle(p1) - name2 = representation.get_prot_name_from_particle(p2) - random_residue_pairs.append((name1, r1, name2, r2)) - - return random_residue_pairs - def print_multicolumn(list_of_strings, ncolumns=2, truncate=40): l = list_of_strings diff --git a/modules/pmi/pyext/src/topology/__init__.py b/modules/pmi/pyext/src/topology/__init__.py index 9da81a9a6c..fb1e77158d 100644 --- a/modules/pmi/pyext/src/topology/__init__.py +++ b/modules/pmi/pyext/src/topology/__init__.py @@ -724,9 +724,14 @@ def build(self): # build all the representations built_reps = [] + + rephandler = _RepresentationHandler(self._name_with_copy, + list(self._all_protocol_output()), + self._pdb_elements) + for rep in self.representations: built_reps += system_tools.build_representation( - self, rep, self.coord_finder) + self, rep, self.coord_finder, rephandler) # sort them before adding as children built_reps.sort(key=lambda r: IMP.atom.Fragment(r).get_residue_indexes()[0]) @@ -735,9 +740,6 @@ def build(self): br.update_parents() self.built = True - rephandler = _RepresentationHandler(self._name_with_copy, - list(self._all_protocol_output()), - self._pdb_elements) for res in self.residues: idx = res.get_index() diff --git a/modules/pmi/pyext/src/topology/system_tools.py b/modules/pmi/pyext/src/topology/system_tools.py index fda1d1a742..0405cc9862 100644 --- a/modules/pmi/pyext/src/topology/system_tools.py +++ b/modules/pmi/pyext/src/topology/system_tools.py @@ -211,7 +211,21 @@ def _get_color_for_representation(rep): raise TypeError("Color must be Chimera color name, a hex " "string, a float or (r,g,b) tuple") -def build_representation(parent, rep, coord_finder): + +def _add_fragment_provenance(fragment, first_residue, rephandler): + """Track the original source of a fragment's structure. + If the residues in the given fragment were extracted from a PDB + file, add suitable provenance information to the Model (the name + of that file, chain ID, and residue index offset).""" + pdb_element = rephandler.pdb_for_residue.get(first_residue.get_index()) + if pdb_element: + m = fragment.get_model() + sp = IMP.core.StructureProvenance.setup_particle(IMP.Particle(m), + pdb_element.filename, pdb_element.chain_id, pdb_element.offset) + IMP.core.add_provenance(m, fragment, sp) + + +def build_representation(parent, rep, coord_finder, rephandler): """Create requested representation. For beads, identifies continuous segments and sets up as Representation. If any volume-based representations (e.g.,densities) are requested, @@ -323,6 +337,8 @@ def build_representation(parent, rep, coord_finder): this_resolution = IMP.atom.Fragment.setup_particle(fp,res_nums) this_resolution.set_name("%s: Res %i"%(name,resolution)) if frag_res[0].get_has_structure(): + _add_fragment_provenance(this_resolution, frag_res[0], + rephandler) # if structured, merge particles as needed if resolution==atomic_res: for residue in frag_res: diff --git a/modules/pmi/setup_git.py b/modules/pmi/setup_git.py index df93fb40b1..ae9b9edb83 100755 --- a/modules/pmi/setup_git.py +++ b/modules/pmi/setup_git.py @@ -2,6 +2,10 @@ """Call the main setup_git.py. This should be copied to the main directory of your project and named setup_git.py.""" -import os +import subprocess import os.path -os.system(os.path.join("tools", "dev_tools", "git", "setup_git.py")) +import sys + + +subprocess.check_call( + [sys.executable, os.path.join("tools", "dev_tools", "git", "setup_git.py")]) diff --git a/modules/pmi/test/expensive_test_cross_link_ms_restraint.py b/modules/pmi/test/expensive_test_cross_link_ms_restraint.py deleted file mode 100644 index 3d07efc937..0000000000 --- a/modules/pmi/test/expensive_test_cross_link_ms_restraint.py +++ /dev/null @@ -1,356 +0,0 @@ -from __future__ import print_function -import IMP -import os -import IMP.test -import IMP.core -import IMP.container -import IMP.pmi -import IMP.pmi.representation -import IMP.pmi.restraints -import IMP.pmi.restraints.crosslinking -from math import * - -def sphere_cap(r1, r2, d): - sc = 0.0 - if d <= max(r1, r2) - min(r1, r2): - sc = min(4.0 / 3 * pi * r1 * r1 * r1, - 4.0 / 3 * pi * r2 * r2 * r2) - elif d >= r1 + r2 : - sc = 0 - else: - sc = (pi / 12 / d * (r1 + r2 - d) * (r1 + r2 - d)) * \ - (d * d + 2 * d * r1 - 3 * r1 * r1 + 2 * d * r2 + 6 * r1 * r2 - - 3 * r2 * r2) - return sc - -def get_probability(xyz1s,xyz2s,sigma1s,sigma2s,psis,length,slope): - onemprob = 1.0 - - for n in range(len(xyz1s)): - xyz1=xyz1s[n] - xyz2=xyz2s[n] - sigma1=sigma1s[n] - sigma2=sigma2s[n] - psi = psis[n] - psi = psi.get_scale() - dist=IMP.core.get_distance(xyz1, xyz2) - - sigmai = sigma1.get_scale() - sigmaj = sigma2.get_scale() - voli = 4.0 / 3.0 * pi * sigmai * sigmai * sigmai - volj = 4.0 / 3.0 * pi * sigmaj * sigmaj * sigmaj - fi = 0 - fj = 0 - if dist < sigmai + sigmaj : - xlvol = 4.0 / 3.0 * pi * (length / 2) * (length / 2) * \ - (length / 2) - fi = min(voli, xlvol) - fj = min(volj, xlvol) - else: - di = dist - sigmaj - length / 2 - dj = dist - sigmai - length / 2 - fi = sphere_cap(sigmai, length / 2, abs(di)) - fj = sphere_cap(sigmaj, length / 2, abs(dj)) - pofr = fi * fj / voli / volj - - factor = (1.0 - (psi * (1.0 - pofr) + pofr * (1 - psi))*exp(-slope*dist)) - onemprob = onemprob * factor - prob = 1.0 - onemprob - return prob - -def log_evaluate(restraints): - prob = 1.0 - score = 0.0 - - for r in restraints: - prob *= r.unprotected_evaluate(None) - if prob<=0.0000000001: - score=score-log(prob) - prob=1.0 - - score=score-log(prob) - return score - -def init_representation_beads(m): - with IMP.allow_deprecated(): - r = IMP.pmi.representation.Representation(m) - r.create_component("ProtA",color=1.0) - r.add_component_beads("ProtA", [(1,10)],incoord=(0,0,0)) - r.add_component_beads("ProtA", [(11,20)],incoord=(10,0,0)) - r.add_component_beads("ProtA", [(21,30)],incoord=(20,0,0)) - r.create_component("ProtB",color=1.0) - r.add_component_beads("ProtB", [(1,10)],incoord=(0,10,0)) - r.add_component_beads("ProtB", [(11,20)],incoord=(10,10,0)) - r.add_component_beads("ProtB", [(21,30)],incoord=(20,10,0)) - r.set_floppy_bodies() - return r - - -def setup_crosslinks_complex(representation,mode): - - if mode=="single_category": - columnmap={} - columnmap["Protein1"]="pep1.accession" - columnmap["Protein2"]="pep2.accession" - columnmap["Residue1"]="pep1.xlinked_aa" - columnmap["Residue2"]="pep2.xlinked_aa" - columnmap["IDScore"]=None - columnmap["XLUniqueID"]=None - - ids_map=IMP.pmi.tools.map() - ids_map.set_map_element(1.0,1.0) - - with IMP.allow_deprecated(): - xl = IMP.pmi.restraints.crosslinking.ISDCrossLinkMS(representation, - IMP.pmi.get_data_path("polii_xlinks.csv"), - length=21.0, - slope=0.01, - columnmapping=columnmap, - ids_map=ids_map, - resolution=1.0, - label="XL", - csvfile=True) - xl.add_to_model() - xl.set_label("XL") - psi=xl.get_psi(1.0)[0] - psi.set_scale(0.05) - sigma=xl.get_sigma(1)[0] - sigma.set_scale(10.0) - return xl - -def setup_crosslinks_beads(representation,mode): - - restraints_beads=IMP.pmi.tools.get_random_cross_link_dataset(representation, - number_of_cross_links=100, - resolution=1.0, - avoid_same_particles=True, - ambiguity_probability=0.3, - confidence_score_range=[0,100]) - - ids_map=IMP.pmi.tools.map() - ids_map.set_map_element(25.0,0.1) - ids_map.set_map_element(75,0.01) - - with IMP.allow_deprecated(): - xl = IMP.pmi.restraints.crosslinking.ISDCrossLinkMS( - representation, restraints_beads, 21, label="XL", ids_map=ids_map, - resolution=1, inner_slope=0.01) - - sig = xl.get_sigma(1.0)[0] - psi1 = xl.get_psi(25.0)[0] - psi2 = xl.get_psi(75.0)[0] - sig.set_scale(10.0) - psi1.set_scale(0.1) - psi2.set_scale(0.01) - - return xl,restraints_beads - - - - - -class Tests(IMP.test.TestCase): - - def init_representation_complex(self, m): - pdbfile = self.get_input_file_name("1WCM.pdb") - fastafile = self.get_input_file_name("1WCM.fasta.txt") - components = ["Rpb1","Rpb2","Rpb3","Rpb4"] - chains = "ABCD" - colors = [0.,0.1,0.5,1.0] - beadsize = 20 - fastids = IMP.pmi.tools.get_ids_from_fasta_file(fastafile) - - with IMP.allow_deprecated(): - r = IMP.pmi.representation.Representation(m) - hierarchies = {} - for n in range(len(components)): - r.create_component(components[n], color=colors[n]) - r.add_component_sequence(components[n], fastafile, - id="1WCM:"+chains[n]) - hierarchies[components[n]] = r.autobuild_model( - components[n], pdbfile, chains[n], - resolutions=[1, 10, 100], missingbeadsize=beadsize) - r.setup_component_sequence_connectivity(components[n], 1) - return r - - def test_restraint_probability_complex(self): - m = IMP.Model() - rcomplex=self.init_representation_complex(m) - xlc=setup_crosslinks_complex(rcomplex,"single_category") - - # check all internals didn't change since last time - o=IMP.pmi.output.Output() - o.write_test("expensive_test_cross_link_ms_restraint.dat", [xlc]) - - passed=o.test(self.get_input_file_name("expensive_test_cross_link_ms_restraint.dat"), [xlc]) - self.assertEqual(passed, True) - rs=xlc.get_restraint() - - # check the probability of cross-links - restraints=[] - for p in xlc.pairs: - p0 = p[0] - p1 = p[1] - prob = p[2].get_probability() - resid1 = p[3] - chain1 = p[4] - resid2 = p[5] - chain2 = p[6] - attribute = p[7] - d0 = IMP.core.XYZ(p0) - d1 = IMP.core.XYZ(p1) - dist=IMP.core.get_distance(d0, d1) - - sig1 = xlc.get_sigma(p[8])[0] - sig2 = xlc.get_sigma(p[9])[0] - psi = xlc.get_psi(p[10])[0] - test_prob=get_probability([d0],[d1],[sig1],[sig2],[psi],21.0,0.0) - restraints.append(p[2]) - - - # check that the probability is the same for - # each cross-link - self.assertAlmostEqual(prob, test_prob, delta=0.00001) - - # check the log_wrapper - log_wrapper_score=rs.unprotected_evaluate(None) - test_log_wrapper_score=log_evaluate(restraints) - self.assertAlmostEqual(log_wrapper_score, test_log_wrapper_score, delta=0.00001) - for output in ['excluded.None.xl.db', - 'expensive_test_cross_link_ms_restraint.dat', - 'included.None.xl.db', 'missing.None.xl.db']: - os.unlink(output) - - def test_restraint_ambiguity(self): - m=IMP.Model() - with IMP.allow_deprecated(): - r = IMP.pmi.representation.Representation(m) - r.create_component("ProtA",color=1.0) - r.add_component_beads("ProtA", [(1,10)],incoord=(0,0,0)) - r.create_component("ProtB",color=1.0) - r.add_component_beads("ProtB", [(1,10)],incoord=(0,10,0)) - r.add_component_beads("ProtB", [(11,20)],incoord=(10,10,0)) - r.add_component_beads("ProtB", [(21,30)],incoord=(20,10,0)) - r.set_floppy_bodies() - - restraints_beads='''# -ProtA ProtB 1 1 94.81973271 1 -ProtA ProtB 1 11 52.4259605298 1 -ProtA ProtB 1 21 87.4778223289 1''' - - - ids_map=IMP.pmi.tools.map() - ids_map.set_map_element(25.0,0.1) - ids_map.set_map_element(75,0.01) - - with IMP.allow_deprecated(): - xl = IMP.pmi.restraints.crosslinking.ISDCrossLinkMS( - r, restraints_beads, 21, label="XL", slope=0.0, - ids_map=ids_map, resolution=1, inner_slope=0.01) - - sig = xl.get_sigma(1.0)[0] - psi1 = xl.get_psi(25.0)[0] - psi2 = xl.get_psi(75.0)[0] - sig.set_scale(10.0) - psi1.set_scale(0.1) - psi2.set_scale(0.01) - - - for i in range(100): - r.shuffle_configuration(max_translation=10) - for p in xl.pairs: - p0 = p[0] - p1 = p[1] - ln = p[2] - resid1 = p[3] - chain1 = p[4] - resid2 = p[5] - chain2 = p[6] - attribute = p[7] - xlid=p[11] - d0 = IMP.core.XYZ(p0) - d1 = IMP.core.XYZ(p1) - sig1 = xl.get_sigma(p[8])[0] - sig2 = xl.get_sigma(p[9])[0] - psi = xl.get_psi(p[10])[0] - self.assertEqual(-log(ln.get_probability()),xl.rs.unprotected_evaluate(None)) - - @IMP.test.expectedFailure - def test_restraint_probability_beads(self): - m = IMP.Model() - rbeads=init_representation_beads(m) - xlb,restraints_beads=setup_crosslinks_beads(rbeads,"single_category") - - # check internal data structure - ds=xlb.pairs - nxl=0 - for l in restraints_beads.split("\n"): - if not l.strip(): continue - if l[0]=="#": continue - t=l.split() - - chain1 = t[0] - chain2 = t[1] - res1 = int(t[2]) - res2 = int(t[3]) - ids = float(t[4]) - xlid = int(t[5]) - dsres1 = ds[nxl][3] - dschain1 = ds[nxl][4] - dsres2 = ds[nxl][5] - dschain2 = ds[nxl][6] - dsxlid= int(ds[nxl][11]) - ''' - self.assertEqual(chain1,dschain1) - self.assertEqual(chain2,dschain2) - self.assertEqual(res1,dsres1) - self.assertEqual(res2,dsres2) - self.assertEqual(xlid,dsxlid) - ''' - nxl+=1 - - # randomize coordinates and check that the probability is OK - for i in range(100): - rbeads.shuffle_configuration(max_translation=10) - cross_link_dict={} - for p in xlb.pairs: - p0 = p[0] - p1 = p[1] - prob = p[2].get_probability() - resid1 = p[3] - chain1 = p[4] - resid2 = p[5] - chain2 = p[6] - attribute = p[7] - xlid=p[11] - d0 = IMP.core.XYZ(p0) - d1 = IMP.core.XYZ(p1) - sig1 = xlb.get_sigma(p[8])[0] - sig2 = xlb.get_sigma(p[9])[0] - psi = xlb.get_psi(p[10])[0] - - if xlid not in cross_link_dict: - cross_link_dict[xlid]=([d0],[d1],[sig1],[sig2],[psi],prob) - else: - cross_link_dict[xlid][0].append(d0) - cross_link_dict[xlid][1].append(d1) - cross_link_dict[xlid][2].append(sig1) - cross_link_dict[xlid][3].append(sig2) - cross_link_dict[xlid][4].append(psi) - - for xlid in cross_link_dict: - - test_prob=get_probability(cross_link_dict[xlid][0], - cross_link_dict[xlid][1], - cross_link_dict[xlid][2], - cross_link_dict[xlid][3], - cross_link_dict[xlid][4],21.0,0.01) - prob=cross_link_dict[xlid][5] - - self.assertAlmostEqual(prob/test_prob,1.0, delta=0.0001) - for output in ['excluded.None.xl.db', - 'included.None.xl.db', 'missing.None.xl.db']: - os.unlink(output) - -if __name__ == '__main__': - IMP.test.main() diff --git a/modules/pmi/test/expensive_test_disulfide_bond_restraint.py b/modules/pmi/test/expensive_test_disulfide_bond_restraint.py index cb73cf20ce..fa57717955 100644 --- a/modules/pmi/test/expensive_test_disulfide_bond_restraint.py +++ b/modules/pmi/test/expensive_test_disulfide_bond_restraint.py @@ -6,9 +6,8 @@ import IMP.pmi import IMP.pmi.io import IMP.pmi.io.crosslink -import IMP.pmi.representation import IMP.pmi.restraints -import IMP.pmi.restraints.crosslinking_new +import IMP.pmi.restraints.crosslinking from math import * def sphere_cap(r1, r2, d): @@ -77,16 +76,16 @@ class Tests(IMP.test.TestCase): def test_restraint_probability_beads(self): m = IMP.Model() - with IMP.allow_deprecated(): - r = IMP.pmi.representation.Representation(m) - r.create_component("ProtA",color=1.0) - r.add_component_beads("ProtA", [(1,10)],incoord=(0,0,0)) - r.create_component("ProtB",color=1.0) - r.add_component_beads("ProtB", [(1,10)],incoord=(0,10,0)) - r.set_floppy_bodies() - - xl = IMP.pmi.restraints.crosslinking_new.DisulfideCrossLinkRestraint( - r, + simo = IMP.pmi.topology.System(m) + st1 = simo.create_state() + for molname in ("ProtA", "ProtB"): + mol = st1.create_molecule(molname, "A" * 10) + mol.add_representation(mol.get_non_atomic_residues(), + resolutions=[10]) + root_hier = simo.build() + + xl = IMP.pmi.restraints.crosslinking.DisulfideCrossLinkRestraint( + root_hier, (1,1,"ProtA"), (1,1,"ProtB"), label="DisulfideBond1", diff --git a/modules/pmi/test/expensive_test_input.py b/modules/pmi/test/expensive_test_input.py index a5a8fc3fd2..bd0f28201d 100644 --- a/modules/pmi/test/expensive_test_input.py +++ b/modules/pmi/test/expensive_test_input.py @@ -25,7 +25,9 @@ def test_save_best_models(self): number_of_best_scoring_models=3, score_key=self.score_key, feature_keys=self.feature_keys) - po = IMP.pmi.output.ProcessOutput('top_3.out') + # This makes a (deprecated) v1 statfile + with IMP.allow_deprecated(): + po = IMP.pmi.output.ProcessOutput('top_3.out') fields = po.get_fields([self.score_key]) self.assertEqual(len(fields[self.score_key]),3) self.assertEqual(float(fields[self.score_key][0]),301.048975729) diff --git a/modules/pmi/test/expensive_test_loop_reconstruction.py b/modules/pmi/test/expensive_test_loop_reconstruction.py index 36298b1cc5..3715084973 100644 --- a/modules/pmi/test/expensive_test_loop_reconstruction.py +++ b/modules/pmi/test/expensive_test_loop_reconstruction.py @@ -4,7 +4,8 @@ import IMP.test import IMP.pmi.restraints.stereochemistry -import IMP.pmi.representation as representation +import IMP.pmi.topology +import IMP.pmi.dof import IMP.pmi.tools as tools import IMP.pmi.samplers as samplers import IMP.pmi.output as output @@ -12,14 +13,6 @@ IMP.set_log_level(IMP.SILENT) -def check_time(stop_watch_object, threshold): - elapsed_time = float(stop_watch_object.get_output() - ["Stopwatch_None_delta_seconds"]) - if (elapsed_time > threshold): - print("WARNING: the calculation time of " + str(elapsed_time) + " is above the " + str(threshold) + " threshold") - else: - print("the calculation time of " + str(elapsed_time) + " is below the threshold") - class Tests(IMP.test.TestCase): def test_loop_reconstruction(self): """Test loop reconstruction""" @@ -29,13 +22,7 @@ def test_loop_reconstruction(self): "loop_reconstruction/starting.structure.pdb") fastafile = self.get_input_file_name( "loop_reconstruction/sequence.fasta") - fastids = tools.get_ids_from_fasta_file(fastafile) - missing_bead_size = 1 - - # Component pdbfile chainid rgb color fastafile sequence id - # in fastafile - data = [("chainA", pdbfile, "A", 0.00000000, (fastafile, 0)), - ("chainB", pdbfile, "B", 0.50000000, (fastafile, 0))] + sequences = IMP.pmi.topology.Sequences(fastafile) # create the representation log_objects = [] @@ -45,58 +32,51 @@ def test_loop_reconstruction(self): log_objects.append(sw) m = IMP.Model() - with IMP.allow_deprecated(): - r = representation.Representation(m) - - hierarchies = {} - - for d in data: - component_name = d[0] - pdb_file = d[1] - chain_id = d[2] - color_id = d[3] - fasta_file = d[4][0] - fasta_file_id = d[4][1] - # avoid adding a component with the same name - r.create_component(component_name, color=color_id) - - r.add_component_sequence(component_name, - fasta_file, - id=fastids[fasta_file_id]) - - hierarchies = r.autobuild_model(component_name, pdb_file, - chain_id, resolutions=[1, 10], - missingbeadsize=missing_bead_size) - - r.show_component_table(component_name) - - rbAB = r.set_rigid_bodies(["chainA", "chainB"]) - - r.set_floppy_bodies() - r.fix_rigid_bodies([rbAB]) - r.setup_bonds() - - log_objects.append(r) + s = IMP.pmi.topology.System(m) + st = s.create_state() + + cA = st.create_molecule("chainA", sequence=sequences[0]) + atomic = cA.add_structure(pdbfile, chain_id='A') + cA.add_representation(atomic, resolutions=[1, 10], color=0.) + cA.add_representation(cA.get_non_atomic_residues(), + resolutions=[1], color=0.) + + cB = st.create_molecule("chainB", sequence=sequences[0]) + atomic = cB.add_structure(pdbfile, chain_id='B') + cB.add_representation(atomic, resolutions=[1, 10], color=0.5) + cB.add_representation(cB.get_non_atomic_residues(), + resolutions=[1], color=0.) + root_hier = s.build() + + dof = IMP.pmi.dof.DegreesOfFreedom(m) + dof.create_rigid_body(cA) + dof.create_rigid_body(cB) + + cr = IMP.pmi.restraints.stereochemistry.ConnectivityRestraint(root_hier) + cr.add_to_model() + log_objects.append(cr) listofexcludedpairs = [] - lof = [(1, 12, "chainA"), (1, 12, "chainB"), - (294, 339, "chainA"), (294, 339, "chainB"), - (686, 701, "chainA"), (686, 701, "chainB"), - (454, 464, "chainA"), (454, 464, "chainB"), - (472, 486, "chainA"), (472, 486, "chainB"), - (814, 859, "chainA"), (814, 859, "chainB")] - + lof = [cA[:12], cB[:12], + cA[293:339], cB[293:339], + cA[685:701], cB[685:701], + cA[453:464], cB[453:464], + cA[471:486], cB[471:486], + cA[813:859], cB[813:859]] # add bonds and angles for l in lof: - rbr = IMP.pmi.restraints.stereochemistry.ResidueBondRestraint(r, l) + print(l) + rbr = IMP.pmi.restraints.stereochemistry.ResidueBondRestraint( + objects=l) rbr.add_to_model() listofexcludedpairs += rbr.get_excluded_pairs() log_objects.append(rbr) - rar = IMP.pmi.restraints.stereochemistry.ResidueAngleRestraint(r, l) + rar = IMP.pmi.restraints.stereochemistry.ResidueAngleRestraint( + objects=l) rar.add_to_model() listofexcludedpairs += rar.get_excluded_pairs() log_objects.append(rar) @@ -104,12 +84,12 @@ def test_loop_reconstruction(self): # add excluded volume ev = IMP.pmi.restraints.stereochemistry.ExcludedVolumeSphere( - r, resolution=10.0) + included_objects=root_hier, resolution=10.0) ev.add_excluded_particle_pairs(listofexcludedpairs) ev.add_to_model() log_objects.append(ev) - mc = samplers.MonteCarlo(m, [r], 1.0) + mc = samplers.MonteCarlo(m, dof.get_movers(), 1.0) log_objects.append(mc) o = output.Output() diff --git a/modules/pmi/test/expensive_test_new_cross_link_ms_restraint.py b/modules/pmi/test/expensive_test_new_cross_link_ms_restraint.py index 88a58877ce..f6f362fa40 100644 --- a/modules/pmi/test/expensive_test_new_cross_link_ms_restraint.py +++ b/modules/pmi/test/expensive_test_new_cross_link_ms_restraint.py @@ -9,7 +9,6 @@ import IMP.pmi.dof import IMP.pmi.io import IMP.pmi.io.crosslink -import IMP.pmi.representation import IMP.pmi.restraints import IMP.pmi.restraints.crosslinking import IMP.pmi.macros @@ -151,28 +150,6 @@ def setup_crosslinks_beads(self,representation=None,mode=None,root_hier=None): return xl,cldb - def init_representation_complex(self, m): - pdbfile = self.get_input_file_name("1WCM.pdb") - fastafile = self.get_input_file_name("1WCM.fasta.txt") - components = ["Rpb1","Rpb2","Rpb3","Rpb4"] - chains = "ABCD" - colors = [0.,0.1,0.5,1.0] - beadsize = 20 - fastids = IMP.pmi.tools.get_ids_from_fasta_file(fastafile) - - with IMP.allow_deprecated(): - r = IMP.pmi.representation.Representation(m) - hierarchies = {} - for n in range(len(components)): - r.create_component(components[n], color=colors[n]) - r.add_component_sequence(components[n], fastafile, - id="1WCM:"+chains[n]) - hierarchies[components[n]] = r.autobuild_model( - components[n], pdbfile, chains[n], - resolutions=[1, 10, 100], missingbeadsize=beadsize) - r.setup_component_sequence_connectivity(components[n], 1) - return r - def init_representation_complex_pmi2(self,m): pdbfile = self.get_input_file_name("1WCM.pdb") fastafile = self.get_input_file_name("1WCM.fasta.txt") @@ -198,20 +175,6 @@ def init_representation_complex_pmi2(self,m): nonrigid_parts = molecule.get_non_atomic_residues()) return hier,dof - def init_representation_beads(self,m): - with IMP.allow_deprecated(): - r = IMP.pmi.representation.Representation(m) - r.create_component("ProtA",color=1.0) - r.add_component_beads("ProtA", [(1,10)],incoord=(0,0,0)) - r.add_component_beads("ProtA", [(11,20)],incoord=(10,0,0)) - r.add_component_beads("ProtA", [(21,30)],incoord=(20,0,0)) - r.create_component("ProtB",color=1.0) - r.add_component_beads("ProtB", [(1,10)],incoord=(0,10,0)) - r.add_component_beads("ProtB", [(11,20)],incoord=(10,10,0)) - r.add_component_beads("ProtB", [(21,30)],incoord=(20,10,0)) - r.set_floppy_bodies() - return r - def init_representation_beads_pmi2(self,m): s = IMP.pmi.topology.System(m) st = s.create_state() @@ -231,31 +194,85 @@ def init_representation_beads_pmi2(self,m): def test_restraint_probability_complex(self): """Test restraint gets correct probabilities""" - for i in range(2): - m = IMP.Model() - print("Testing PMI version",i+1) - if i==0: - rcomplex = self.init_representation_complex(m) - xlc,cldb = self.setup_crosslinks_complex(rcomplex,"single_category") - else: - rcomplex,dof=self.init_representation_complex_pmi2(m) - xlc,cldb = self.setup_crosslinks_complex(root_hier=rcomplex, - mode="single_category") - self.assertEqual(len(dof.get_movers()),42) - dof.get_nuisances_from_restraint(xlc) - self.assertEqual(len(dof.get_movers()),44) - # check all internals didn't change since last time - o=IMP.pmi.output.Output() - o.write_test("expensive_test_new_cross_link_ms_restraint.dat", [xlc]) - - passed=o.test(self.get_input_file_name("expensive_test_new_cross_link_ms_restraint.dat"), - [xlc]) - self.assertEqual(passed, True) - rs = xlc.get_restraint() - - # check the probability of cross-links - restraints=[] - for xl in xlc.xl_list: + m = IMP.Model() + rcomplex,dof=self.init_representation_complex_pmi2(m) + xlc,cldb = self.setup_crosslinks_complex(root_hier=rcomplex, + mode="single_category") + self.assertEqual(len(dof.get_movers()),42) + dof.get_nuisances_from_restraint(xlc) + self.assertEqual(len(dof.get_movers()),44) + # check all internals didn't change since last time + o=IMP.pmi.output.Output() + o.write_test("expensive_test_new_cross_link_ms_restraint.dat", [xlc]) + + passed=o.test(self.get_input_file_name("expensive_test_new_cross_link_ms_restraint.dat"), + [xlc]) + self.assertEqual(passed, True) + rs = xlc.get_restraint() + + # check the probability of cross-links + restraints=[] + for xl in xlc.xl_list: + p0 = xl["Particle1"] + p1 = xl["Particle2"] + prob = xl["Restraint"].get_probability() + resid1 = xl[cldb.residue1_key] + chain1 = xl[cldb.protein1_key] + resid2 = xl[cldb.residue2_key] + chain2 = xl[cldb.protein2_key] + d0 = IMP.core.XYZ(p0) + d1 = IMP.core.XYZ(p1) + sig1 = xl["Particle_sigma1"] + sig2 = xl["Particle_sigma2"] + psi = xl["Particle_psi"] + d0 = IMP.core.XYZ(p0) + d1 = IMP.core.XYZ(p1) + dist=IMP.core.get_distance(d0, d1) + + test_prob=get_probability([d0],[d1],[sig1],[sig2],[psi],21.0,0.0) + restraints.append(xl["Restraint"]) + + # check that the probability is the same for + # each cross-link + self.assertAlmostEqual(prob, test_prob, delta=0.00001) + + # check the log_wrapper + log_wrapper_score=rs.unprotected_evaluate(None) + test_log_wrapper_score=log_evaluate(restraints) + self.assertAlmostEqual(log_wrapper_score, test_log_wrapper_score, delta=0.00001) + rex = IMP.pmi.macros.ReplicaExchange0(m, + root_hier=rcomplex, + monte_carlo_sample_objects=dof.get_movers(), + number_of_frames=2, + test_mode=True, + replica_exchange_object = rem) + rex.execute_macro() + for output in ['excluded.None.xl.db', + 'expensive_test_new_cross_link_ms_restraint.dat', + 'included.None.xl.db', 'missing.None.xl.db']: + os.unlink(output) + + def test_restraint_probability_beads(self): + """Test restraint works for all-bead systems""" + m = IMP.Model() + rbeads,dof=self.init_representation_beads_pmi2(m) + xlbeads,cldb=self.setup_crosslinks_beads(root_hier=rbeads,mode="single_category") + self.assertEqual(len(dof.get_movers()),60) + dof.get_nuisances_from_restraint(xlbeads) + self.assertEqual(len(dof.get_movers()),62) + for xl in xlbeads.xl_list: + + chain1 = xl[cldb.protein1_key] + chain2 = xl[cldb.protein2_key] + res1 = xl[cldb.residue1_key] + res2 = xl[cldb.residue2_key] + ids = xl[cldb.unique_id_key] + + # randomize coordinates and check that the probability is OK + for j in range(100): + IMP.pmi.tools.shuffle_configuration(rbeads,max_translation=10) + cross_link_dict={} + for xl in xlbeads.xl_list: p0 = xl["Particle1"] p1 = xl["Particle2"] prob = xl["Restraint"].get_probability() @@ -263,128 +280,41 @@ def test_restraint_probability_complex(self): chain1 = xl[cldb.protein1_key] resid2 = xl[cldb.residue2_key] chain2 = xl[cldb.protein2_key] + xlid=xl[cldb.unique_id_key] d0 = IMP.core.XYZ(p0) d1 = IMP.core.XYZ(p1) sig1 = xl["Particle_sigma1"] sig2 = xl["Particle_sigma2"] psi = xl["Particle_psi"] - d0 = IMP.core.XYZ(p0) - d1 = IMP.core.XYZ(p1) - dist=IMP.core.get_distance(d0, d1) - - test_prob=get_probability([d0],[d1],[sig1],[sig2],[psi],21.0,0.0) - restraints.append(xl["Restraint"]) - - # check that the probability is the same for - # each cross-link - self.assertAlmostEqual(prob, test_prob, delta=0.00001) - - # check the log_wrapper - log_wrapper_score=rs.unprotected_evaluate(None) - test_log_wrapper_score=log_evaluate(restraints) - self.assertAlmostEqual(log_wrapper_score, test_log_wrapper_score, delta=0.00001) - if i==0: - rex0 = IMP.pmi.macros.ReplicaExchange0(m, - rcomplex, - monte_carlo_sample_objects=[rcomplex], - number_of_frames=2, - test_mode=True, - replica_exchange_object = rem) - rex0.execute_macro() - else: - rex = IMP.pmi.macros.ReplicaExchange0(m, - root_hier=rcomplex, - monte_carlo_sample_objects=dof.get_movers(), - number_of_frames=2, - test_mode=True, - replica_exchange_object = rem) - rex.execute_macro() - for output in ['excluded.None.xl.db', - 'expensive_test_new_cross_link_ms_restraint.dat', - 'included.None.xl.db', 'missing.None.xl.db']: - os.unlink(output) - def test_restraint_probability_beads(self): - """Test restraint works for all-bead systems""" - for i in range(2): - m = IMP.Model() - if i==0: - rbeads=self.init_representation_beads(m) - xlbeads,cldb=self.setup_crosslinks_beads(rbeads,"single_category") - else: - rbeads,dof=self.init_representation_beads_pmi2(m) - xlbeads,cldb=self.setup_crosslinks_beads(root_hier=rbeads,mode="single_category") - self.assertEqual(len(dof.get_movers()),60) - dof.get_nuisances_from_restraint(xlbeads) - self.assertEqual(len(dof.get_movers()),62) - for xl in xlbeads.xl_list: - - chain1 = xl[cldb.protein1_key] - chain2 = xl[cldb.protein2_key] - res1 = xl[cldb.residue1_key] - res2 = xl[cldb.residue2_key] - ids = xl[cldb.unique_id_key] - - # randomize coordinates and check that the probability is OK - print("testing PMI version "+str(i+1)) - for j in range(100): - if i==0: - rbeads.shuffle_configuration(max_translation=10) + if xlid not in cross_link_dict: + cross_link_dict[xlid]=([d0],[d1],[sig1],[sig2],[psi],prob) else: - IMP.pmi.tools.shuffle_configuration(rbeads,max_translation=10) - cross_link_dict={} - for xl in xlbeads.xl_list: - p0 = xl["Particle1"] - p1 = xl["Particle2"] - prob = xl["Restraint"].get_probability() - resid1 = xl[cldb.residue1_key] - chain1 = xl[cldb.protein1_key] - resid2 = xl[cldb.residue2_key] - chain2 = xl[cldb.protein2_key] - xlid=xl[cldb.unique_id_key] - d0 = IMP.core.XYZ(p0) - d1 = IMP.core.XYZ(p1) - sig1 = xl["Particle_sigma1"] - sig2 = xl["Particle_sigma2"] - psi = xl["Particle_psi"] - - if xlid not in cross_link_dict: - cross_link_dict[xlid]=([d0],[d1],[sig1],[sig2],[psi],prob) - else: - cross_link_dict[xlid][0].append(d0) - cross_link_dict[xlid][1].append(d1) - cross_link_dict[xlid][2].append(sig1) - cross_link_dict[xlid][3].append(sig2) - cross_link_dict[xlid][4].append(psi) - - for xlid in cross_link_dict: - test_prob=get_probability(cross_link_dict[xlid][0], - cross_link_dict[xlid][1], - cross_link_dict[xlid][2], - cross_link_dict[xlid][3], - cross_link_dict[xlid][4],21.0,0.01) - prob=cross_link_dict[xlid][5] - - self.assertAlmostEqual(test_prob,prob, delta=0.0001) - if i==0: - rex0 = IMP.pmi.macros.ReplicaExchange0(m, - rbeads, - monte_carlo_sample_objects=[rbeads], - number_of_frames=2, - test_mode=True, - replica_exchange_object = rem) - rex0.execute_macro() - else: - rex = IMP.pmi.macros.ReplicaExchange0(m, - root_hier=rbeads, - monte_carlo_sample_objects=dof.get_movers(), - number_of_frames=2, - test_mode=True, - replica_exchange_object = rem) - rex.execute_macro() - for output in ['excluded.None.xl.db', - 'included.None.xl.db', 'missing.None.xl.db']: - os.unlink(output) + cross_link_dict[xlid][0].append(d0) + cross_link_dict[xlid][1].append(d1) + cross_link_dict[xlid][2].append(sig1) + cross_link_dict[xlid][3].append(sig2) + cross_link_dict[xlid][4].append(psi) + + for xlid in cross_link_dict: + test_prob=get_probability(cross_link_dict[xlid][0], + cross_link_dict[xlid][1], + cross_link_dict[xlid][2], + cross_link_dict[xlid][3], + cross_link_dict[xlid][4],21.0,0.01) + prob=cross_link_dict[xlid][5] + + self.assertAlmostEqual(test_prob,prob, delta=0.0001) + rex = IMP.pmi.macros.ReplicaExchange0(m, + root_hier=rbeads, + monte_carlo_sample_objects=dof.get_movers(), + number_of_frames=2, + test_mode=True, + replica_exchange_object = rem) + rex.execute_macro() + for output in ['excluded.None.xl.db', + 'included.None.xl.db', 'missing.None.xl.db']: + os.unlink(output) def test_restraint_copy_ambiguity(self): """Test restraint works for systems with configuration ambiguity in PMI2""" diff --git a/modules/pmi/test/expensive_test_replica_exchange.py b/modules/pmi/test/expensive_test_replica_exchange.py index a9891dae5d..8c417b5478 100644 --- a/modules/pmi/test/expensive_test_replica_exchange.py +++ b/modules/pmi/test/expensive_test_replica_exchange.py @@ -1,10 +1,11 @@ import IMP import IMP.test import IMP.pmi.samplers -import IMP.pmi.representation import IMP.pmi.restraints.basic import IMP.pmi.macros import IMP.pmi.output +import IMP.pmi.topology +import IMP.pmi.dof import RMF import glob import time @@ -14,22 +15,29 @@ class Tests(IMP.test.TestCase): def test_macro(self): """setting up the representation - PMI 1.0 representation. Creates two particles and - an harmonic distance restraints between them""" + PMI 2 representation. Creates two particles and + a harmonic distance restraint between them""" import shutil import itertools m=IMP.Model() - with IMP.allow_deprecated(): - r=IMP.pmi.representation.Representation(m) - r.create_component("A") - r.add_component_beads("A",[(1,1),(2,2)]) - ps=IMP.atom.get_leaves(r.prot) - dr=IMP.pmi.restraints.basic.DistanceRestraint(r,(1,1,"A"),(2,2,"A"),10,10) + s = IMP.pmi.topology.System(m) + st1 = s.create_state() + mol = st1.create_molecule("A", "GG", "A") + mol.add_representation(resolutions=[1]) + hier = s.build() + + dof = IMP.pmi.dof.DegreesOfFreedom(mol) + dof.create_flexible_beads(mol, max_trans=3.0, resolution=1) + + ps=IMP.atom.get_leaves(hier) + dr=IMP.pmi.restraints.basic.DistanceRestraint( + root_hier=hier, tuple_selection1=(1,1,"A"), + tuple_selection2=(2,2,"A"), distancemin=10, distancemax=10) dr.add_to_model() rex=IMP.pmi.macros.ReplicaExchange0(m, - r, - monte_carlo_sample_objects=[r], - output_objects=[r,dr], + root_hier=hier, + monte_carlo_sample_objects=dof.get_movers(), + output_objects=[dr], monte_carlo_temperature=1.0, replica_exchange_minimum_temperature=1.0, replica_exchange_maximum_temperature=2.5, @@ -116,7 +124,7 @@ def test_macro(self): rex_max_temp_key="ReplicaExchange_MaxTempFrequency" rex_min_temp_key="ReplicaExchange_MinTempFrequency" rex_swap_key="ReplicaExchange_SwapSuccessRatio" - rex_score_key="SimplifiedModel_Total_Score_None" + rex_score_key="Total_Score" rmf_file_key="rmf_file" rmf_file_index="rmf_frame_index" o=IMP.pmi.output.ProcessOutput(rex_out_file) @@ -152,23 +160,31 @@ def check_rmf_file(self, fname): def test_macro_rmf_stat(self): """setting up the representation - PMI 1.0 representation. Creates two particles and - an harmonic distance restraints between them""" + PMI 2.0 representation. Creates two particles and + a harmonic distance restraint between them""" import shutil import itertools m=IMP.Model() - with IMP.allow_deprecated(): - r=IMP.pmi.representation.Representation(m) - r.create_component("A") - r.add_component_beads("A",[(1,1),(2,2)]) - ps=IMP.atom.get_leaves(r.prot) - dr=IMP.pmi.restraints.basic.DistanceRestraint(r,(1,1,"A"),(2,2,"A"),10,10) + s = IMP.pmi.topology.System(m) + st1 = s.create_state() + mol = st1.create_molecule("A", "GG", "A") + mol.add_representation(resolutions=[1]) + hier = s.build() + + dof = IMP.pmi.dof.DegreesOfFreedom(mol) + dof.create_flexible_beads(mol, max_trans=3.0, resolution=1) + + ps=IMP.atom.get_leaves(hier) + dr=IMP.pmi.restraints.basic.DistanceRestraint( + root_hier=hier, tuple_selection1=(1,1,"A"), + tuple_selection2=(2,2,"A"), distancemin=10, distancemax=10) + dr.add_to_model() rex=IMP.pmi.macros.ReplicaExchange0(m, - r, - monte_carlo_sample_objects=[r], + root_hier=hier, + monte_carlo_sample_objects=dof.get_movers(), output_objects=None, - rmf_output_objects=[r,dr], + rmf_output_objects=[dr], monte_carlo_temperature=1.0, replica_exchange_minimum_temperature=1.0, replica_exchange_maximum_temperature=2.5, @@ -255,7 +271,7 @@ def test_macro_rmf_stat(self): rex_max_temp_key="ReplicaExchange_MaxTempFrequency" rex_min_temp_key="ReplicaExchange_MinTempFrequency" rex_swap_key="ReplicaExchange_SwapSuccessRatio" - rex_score_key="SimplifiedModel_Total_Score_None" + rex_score_key="Total_Score" rmf_file_key="rmf_file" rmf_file_index="rmf_frame_index" o=IMP.pmi.output.ProcessOutput(rex_out_file) diff --git a/modules/pmi/test/expensive_test_restraints.py b/modules/pmi/test/expensive_test_restraints.py index 39b12272c9..6ef36e292e 100644 --- a/modules/pmi/test/expensive_test_restraints.py +++ b/modules/pmi/test/expensive_test_restraints.py @@ -6,7 +6,7 @@ import IMP.pmi.restraints.basic import IMP.pmi.restraints.proteomics import IMP.pmi.restraints.crosslinking -import IMP.pmi.representation +import IMP.pmi.topology import IMP.pmi.tools try: import IMP.isd_emxl @@ -15,60 +15,57 @@ no_isd_emxl = True class Tests(IMP.test.TestCase): - @IMP.test.expectedFailure def test_restraints(self): """Test PMI restraints""" # input parameter pdbfile = self.get_input_file_name("1WCM.pdb") - fastafile = self.get_input_file_name("1WCM.fasta.txt") + seqs = IMP.pmi.topology.Sequences(self.get_input_file_name( + '1WCM.fasta.txt')) components = ["Rpb3", "Rpb3.copy", "Rpb4"] - chains = "CCD" - colors = [0., 0.5, 1.0] - beadsize = 20 - fastids = IMP.pmi.tools.get_ids_from_fasta_file(fastafile) - m = IMP.Model() - with IMP.allow_deprecated(): - simo = IMP.pmi.representation.Representation(m) - - hierarchies = {} + simo = IMP.pmi.topology.System(m) + st1 = simo.create_state() + hier_dict = {} for n in range(len(components)): - simo.create_component(components[n], color=colors[n]) - simo.add_component_sequence(components[n], fastafile, id=fastids[n + 2]) - hierarchies[components[n]] = simo.autobuild_model( - components[n], pdbfile, chains[n], - resolutions=[1, 10, 100], missingbeadsize=beadsize) - simo.setup_component_sequence_connectivity(components[n], 1) - - ev1 = IMP.pmi.restraints.stereochemistry.ExcludedVolumeSphere(simo) + mol = st1.create_molecule(components[n], + sequence=seqs["1WCM:" + chains[n]]) + hier_dict[components[n]] = mol + atomic = mol.add_structure(pdbfile, chain_id=chains[n]) + mol.add_representation(mol.get_atomic_residues(), + resolutions=[1, 10, 100], color=colors[n]) + mol.add_representation(mol.get_non_atomic_residues(), + resolutions=[10], color=colors[n]) + root_hier = simo.build() + + ev1 = IMP.pmi.restraints.stereochemistry.ExcludedVolumeSphere( + included_objects=root_hier) ev1.add_to_model() print(ev1.get_output()) ev2 = IMP.pmi.restraints.stereochemistry.ExcludedVolumeSphere( - simo, - [simo.hier_dict["Rpb3"]], - [simo.hier_dict["Rpb4"]]) + included_objects=[hier_dict["Rpb3"]], + other_objects=[hier_dict["Rpb4"]]) ev2.add_to_model() print(ev2.get_output()) rb = IMP.pmi.restraints.stereochemistry.ResidueBondRestraint( - simo, (30, 40, "Rpb3")) + objects=hier_dict["Rpb3"][29:40]) rb.add_to_model() print(rb.get_output()) ra = IMP.pmi.restraints.stereochemistry.ResidueAngleRestraint( - simo, (30, 40, "Rpb3")) + objects=hier_dict["Rpb3"][29:40]) ra.add_to_model() print(ra.get_output()) rd = IMP.pmi.restraints.stereochemistry.ResidueDihedralRestraint( - simo, (30, 40, "Rpb3")) + objects=hier_dict["Rpb3"][29:40]) rd.add_to_model() print(ra.get_output()) @@ -78,18 +75,24 @@ def test_restraints(self): ss.add_to_model() print(ss.get_output()) - eb1 = IMP.pmi.restraints.basic.ExternalBarrier(simo, 50) + eb1 = IMP.pmi.restraints.basic.ExternalBarrier( + hierarchies=root_hier, radius=50) eb1.add_to_model() print(eb1.get_output()) + sel1 = IMP.atom.Selection(root_hier, molecule="Rpb3", + residue_indexes=range(1,100)) + sel2 = IMP.atom.Selection(root_hier, molecule="Rpb4", + residue_indexes=range(1,100)) cr1 = IMP.pmi.restraints.proteomics.ConnectivityRestraint( - simo, [(1, 100, "Rpb3"), (1, 100, "Rpb4")], resolution=100) + (sel1, sel2), resolution=100) cr1.add_to_model() print(cr1.get_output()) cr2 = IMP.pmi.restraints.proteomics.CompositeRestraint( - simo, [(1, 100, "Rpb3"), (200, 300, "Rpb3")], - [[(1, 100, "Rpb4")], [(200, 500, "Rpb4")]], resolution=100) + [hier_dict["Rpb3"][:102], hier_dict["Rpb3"][298:308]], + [[hier_dict["Rpb4"][:106]], [hier_dict["Rpb4"][117:500]]], + resolution=100) cr2.add_to_model() print(cr2.get_output()) @@ -98,9 +101,7 @@ def test_restraints(self): Rpb4 Rpb4 50 150''' cr2 = IMP.pmi.restraints.proteomics.AmbiguousCompositeRestraint( - simo, - restraints, - resolution=1) + root_hier, restraints, resolution=1) cr2.add_to_model() print(cr2.get_output()) @@ -110,33 +111,16 @@ def test_restraints(self): Rpb4 Rpb4 50 150 0.7''' pm = IMP.pmi.restraints.proteomics.SimplifiedPEMAP( - simo, - restraints, - 20, - 1, - resolution=1) + root_hier, restraints, 20, 1, resolution=1) pm.add_to_model() print(pm.get_output()) - restraints = '''# - Rpb3 Rpb4 100 150 1 - Rpb4 Rpb4 50 150 1''' - - xl4 = IMP.pmi.restraints.crosslinking.ISDCrossLinkMS( - simo, - restraints, - 25, - resolution=1) - xl4.add_to_model() - print(xl4.get_output()) - restraints = '''# 100 Rpb3 150 Rpb4 0.5 epsilon1 50 Rpb4 150 Rpb4 0.7 epsilon2''' xl5 = IMP.pmi.restraints.crosslinking.CysteineCrossLinkRestraint( - [simo], - restraints) + root_hier, restraints) xl5.add_to_model() xl5.set_output_level("high") print(xl5.get_output()) diff --git a/modules/pmi/test/input/multistate.csv b/modules/pmi/test/input/multistate.csv new file mode 100644 index 0000000000..ff00cdb378 --- /dev/null +++ b/modules/pmi/test/input/multistate.csv @@ -0,0 +1,3 @@ +id,prot1,prot2,res1,res2 +1,particle2,particle3,5,5 +2,particle1,particle3,5,5 diff --git a/modules/pmi/test/medium_test_crosslink.py b/modules/pmi/test/medium_test_crosslink.py index 37a60fb2f4..e5ca8e16e6 100644 --- a/modules/pmi/test/medium_test_crosslink.py +++ b/modules/pmi/test/medium_test_crosslink.py @@ -1,33 +1,27 @@ import IMP import IMP.test -import IMP.pmi.representation +import IMP.pmi.topology import IMP.pmi.restraints.crosslinking import os class Tests(IMP.test.TestCase): def test_cysteine_cross_link(self): """Test creation of CysteineCrossLinkRestraint""" - with open('seq.fasta', 'w') as fh: - fh.write('>chainA\nA\n>chainB\nE\n') with open('expdata.txt', 'w') as fh: - fh.write('962 alpha 691 beta 1 Epsilon-Intra-Solvent\n') + fh.write('2 alpha 2 beta 1 Epsilon-Intra-Solvent\n') m = IMP.Model() - with IMP.allow_deprecated(): - r = IMP.pmi.representation.Representation(m) - r.create_component("alpha",color=0.25) - r.add_component_sequence("alpha","seq.fasta", id="chainA", offs=962) - r.add_component_beads("alpha",[(962,962)]) + simo = IMP.pmi.topology.System(m) + st1 = simo.create_state() + alpha = st1.create_molecule("alpha", sequence="AA") + alpha.add_representation(alpha, resolutions=[1], color=0.25) + beta = st1.create_molecule("beta", sequence="EE") + beta.add_representation(beta, resolutions=[1], color=0.5) + hier = simo.build() - r.create_component("beta",color=0.5) - r.add_component_sequence("beta","seq.fasta", id="chainB",offs=691) - r.add_component_beads("beta",[(691,691)]) - - - xl = IMP.pmi.restraints.crosslinking.CysteineCrossLinkRestraint([r], + xl = IMP.pmi.restraints.crosslinking.CysteineCrossLinkRestraint(hier, filename="expdata.txt", cbeta=True) xl.add_to_model() - os.unlink('seq.fasta') os.unlink('expdata.txt') if __name__ == '__main__': diff --git a/modules/pmi/test/medium_test_selection.py b/modules/pmi/test/medium_test_selection.py index a68f0d9cf4..ab118a2273 100644 --- a/modules/pmi/test/medium_test_selection.py +++ b/modules/pmi/test/medium_test_selection.py @@ -1,295 +1,124 @@ import IMP import IMP.test -import IMP.pmi.representation as representation import IMP.pmi.tools as tools class Tests(IMP.test.TestCase): def init_representation_complex(self, m): + seqs = IMP.pmi.topology.Sequences(self.get_input_file_name( + '1WCM.fasta.txt')) + pdbfile = self.get_input_file_name("1WCM.pdb") - fastafile = self.get_input_file_name("1WCM.fasta.txt") + components = ["Rpb3","Rpb4","Rpb5","Rpb6"] chains = "CDEF" colors = [0.,0.1,0.5,1.0] beadsize = 20 - fastids = IMP.pmi.tools.get_ids_from_fasta_file(fastafile) - with IMP.allow_deprecated(): - r = IMP.pmi.representation.Representation(m) - hierarchies = {} - for n in range(len(components)): - r.create_component(components[n], color=colors[n]) - r.add_component_sequence(components[n], fastafile, - id="1WCM:"+chains[n]) - hierarchies[components[n]] = r.autobuild_model( - components[n], pdbfile, chains[n], - resolutions=[1, 10, 100], missingbeadsize=beadsize) - r.setup_component_sequence_connectivity(components[n], 1) + m = IMP.Model() + simo = IMP.pmi.topology.System(m) + st1 = simo.create_state() - return r + for n in range(len(components)): + mol = st1.create_molecule(components[n], + sequence=seqs["1WCM:" + chains[n]]) + atomic = mol.add_structure(pdbfile, chain_id=chains[n]) + mol.add_representation(mol.get_atomic_residues(), + resolutions=[1, 10, 100], color=colors[n]) + mol.add_representation(mol.get_non_atomic_residues(), + resolutions=[10], color=colors[n]) + return simo.build() def test_get_terminal_residue_two_methods(self): m = IMP.Model() - rcomplex=self.init_representation_complex(m) - - - results={("Rpb4","cter"):221, - ("Rpb4","nter"):4, - ("Rpb5","cter"):215, - ("Rpb5","nter"):1, - ("Rpb6","cter"):155, - ("Rpb6","nter"):72, - ("Rpb3","cter"):268, - ("Rpb3","nter"):3} - - for name in ["Rpb3","Rpb4","Rpb5","Rpb6"]: - self.assertRaises(ValueError, IMP.pmi.tools.get_terminal_residue, - rcomplex, rcomplex.hier_dict[name], - terminus="notCotN", resolution=1) - cter=IMP.pmi.tools.get_terminal_residue(rcomplex, rcomplex.hier_dict[name], terminus="C", resolution=1) - self.assertEqual(results[(name,"cter")],IMP.atom.Residue(cter).get_index()) - nter=IMP.pmi.tools.get_terminal_residue(rcomplex, rcomplex.hier_dict[name], terminus="N", resolution=1) - self.assertEqual(results[(name,"nter")],IMP.atom.Residue(nter).get_index()) - pos = IMP.pmi.tools.get_terminal_residue_position( - rcomplex, rcomplex.hier_dict[name], terminus="N", - resolution=1) - self.assertIsInstance(pos, IMP.algebra.Vector3D) - - results={("Rpb4","cter"):(IMP.atom.Residue,221), - ("Rpb4","nter"):(IMP.atom.Fragment,list(range(1,4))), - ("Rpb5","cter"):(IMP.atom.Residue,215), - ("Rpb5","nter"):(IMP.atom.Residue,1), - ("Rpb6","cter"):(IMP.atom.Residue,155), - ("Rpb6","nter"):(IMP.atom.Fragment,list(range(1,21))), - ("Rpb3","cter"):(IMP.atom.Fragment,list(range(309, 319))), - ("Rpb3","nter"):(IMP.atom.Fragment,list(range(1, 3)))} - - for name in ["Rpb3","Rpb4","Rpb5","Rpb6"]: - all=IMP.pmi.tools.select_by_tuple(rcomplex,name,resolution=1) - nter=all[0] - cter=all[-1] - result=results[(name,"nter")] - if result[0] is IMP.atom.Residue: - self.assertEqual(result[1],IMP.atom.Residue(nter).get_index()) - if result[0] is IMP.atom.Fragment: - self.assertEqual(result[1],IMP.atom.Fragment(nter).get_residue_indexes()) - result=results[(name,"cter")] - if result[0] is IMP.atom.Residue: - self.assertEqual(result[1],IMP.atom.Residue(cter).get_index()) - if result[0] is IMP.atom.Fragment: - self.assertEqual(result[1],IMP.atom.Fragment(cter).get_residue_indexes()) + root_hier = self.init_representation_complex(m) + + def test(molecule, terminus, residue_index=None, fragment_range=None): + s = IMP.atom.Selection(root_hier, molecule=molecule, resolution=1, + terminus=terminus) + r = s.get_selected_particles(with_representation=False) + self.assertEqual(len(r), 1) + if residue_index is not None: + self.assertTrue(IMP.atom.Residue.get_is_setup(r[0])) + r = IMP.atom.Residue(r[0]) + self.assertEqual(r.get_index(), residue_index) + if fragment_range is not None: + self.assertTrue(IMP.atom.Fragment.get_is_setup(r[0])) + f = IMP.atom.Fragment(r[0]) + inds = f.get_residue_indexes() + self.assertEqual((inds[0], inds[-1]), fragment_range) + + test("Rpb4", IMP.atom.Selection.C, residue_index=221) + test("Rpb4", IMP.atom.Selection.N, fragment_range=(1,3)) + test("Rpb5", IMP.atom.Selection.C, residue_index=215) + test("Rpb5", IMP.atom.Selection.N, residue_index=1) + test("Rpb6", IMP.atom.Selection.C, residue_index=155) + test("Rpb6", IMP.atom.Selection.N, fragment_range=(1,10)) + test("Rpb3", IMP.atom.Selection.C, fragment_range=(309,318)) + test("Rpb3", IMP.atom.Selection.N, fragment_range=(1,2)) def test_selection(self): """Test selection""" - # input parameter + seqs = IMP.pmi.topology.Sequences(self.get_input_file_name( + '1WCM.fasta.txt')) + pdbfile = self.get_input_file_name("1WCM.pdb") - fastafile = self.get_input_file_name("1WCM.fasta.txt") - components = ["Rpb3", "Rpb3.copy", "Rpb4"] - chains = "CCD" - colors = [0., 0.5, 1.0] + components = ["Rpb3", "Rpb4"] + chains = "CD" + colors = [0., 1.0] beadsize = 20 - fastids = ['1WCM:C','1WCM:C','1WCM:D'] + fastids = ['1WCM:C', '1WCM:D'] m = IMP.Model() - with IMP.allow_deprecated(): - simo = representation.Representation(m) - - hierarchies = {} + simo = IMP.pmi.topology.System(m) + st1 = simo.create_state() for n in range(len(components)): - simo.create_component(components[n], color=colors[n]) - simo.add_component_sequence(components[n], fastafile, - id=fastids[n]) - hierarchies[components[n]] \ - = simo.autobuild_model( - components[n], pdbfile, chains[n], - resolutions=[1, 10, 100], missingbeadsize=beadsize) - simo.setup_component_sequence_connectivity(components[n], 1) - - def test(a, b): - assert a==b, "%d != %d" % (a,b) - - result_dict = { - "All": 803, - "resolution=1": 721, - "resolution=1,resid=10": 3, - "resolution=1,resid=10,name=Rpb3": 1, - "resolution=1,resid=10,name=Rpb3,ambiguous": 2, - "resolution=1,resid=10,name=Rpb4,ambiguous": 1, - "resolution=1,resrange=(10,20),name=Rpb3": 11, - "resolution=1,resrange=(10,20),name=Rpb3,ambiguous": 22, - "resolution=10,resrange=(10,20),name=Rpb3": 2, - "resolution=10,resrange=(10,20),name=Rpb3,ambiguous": 4, - "resolution=100,resrange=(10,20),name=Rpb3": 1, - "resolution=100,resrange=(10,20),name=Rpb3,ambiguous": 2, - "Rpb4_1-3_bead resolution=1": 1, - "Rpb4_1-3_bead resolution=10": 1, - "Rpb4_1-3_bead resolution=100": 1, - "Rpb4_4-76_pdb resolution=1": 73, - "Rpb4_4-76_pdb resolution=10": 0, - "Rpb4_4-76_pdb resolution=100": 0, - "Rpb4_4-76_pdb#2 resolution=1": 0, - "Rpb4_4-76_pdb#2 resolution=10": 8, - "Rpb4_4-76_pdb#2 resolution=100": 0, - "Rpb4_4-76_pdb#3 resolution=1": 0, - "Rpb4_4-76_pdb#3 resolution=10": 0, - "Rpb4_4-76_pdb#3 resolution=100": 1, - "Rpb4_77-96_bead resolution=1": 1, - "Rpb4_77-96_bead resolution=10": 1, - "Rpb4_77-96_bead resolution=100": 1, - "Rpb4_97-116_bead resolution=1": 1, - "Rpb4_97-116_bead resolution=10": 1, - "Rpb4_97-116_bead resolution=100": 1, - "Rpb4_117_bead resolution=1": 1, - "Rpb4_117_bead resolution=10": 1, - "Rpb4_117_bead resolution=100": 1, - "Rpb4_118-221_pdb resolution=1": 104, - "Rpb4_118-221_pdb resolution=10": 0, - "Rpb4_118-221_pdb resolution=100": 0, - "Rpb4_118-221_pdb#2 resolution=1": 0, - "Rpb4_118-221_pdb#2 resolution=10": 11, - "Rpb4_118-221_pdb#2 resolution=100": 0, - "Rpb4_118-221_pdb#3 resolution=1": 0, - "Rpb4_118-221_pdb#3 resolution=10": 0, - "Rpb4_118-221_pdb#3 resolution=100": 2, - "Rpb3.copy_1-2_bead resolution=1": 1, - "Rpb3.copy_1-2_bead resolution=1": 1, - "Rpb3.copy_1-2_bead resolution=10": 1, - "Rpb3.copy_1-2_bead resolution=100": 1, - "Rpb3.copy_3-268_pdb resolution=1": 266, - "Rpb3.copy_3-268_pdb resolution=10": 0, - "Rpb3.copy_3-268_pdb resolution=100": 0, - "Rpb3.copy_3-268_pdb#2 resolution=1": 0, - "Rpb3.copy_3-268_pdb#2 resolution=10": 27, - "Rpb3.copy_3-268_pdb#2 resolution=100": 0, - "Rpb3.copy_3-268_pdb#3 resolution=1": 0, - "Rpb3.copy_3-268_pdb#3 resolution=10": 0, - "Rpb3.copy_3-268_pdb#3 resolution=100": 3, - "Rpb3.copy_269-288_bead resolution=1": 1, - "Rpb3.copy_269-288_bead resolution=10": 1, - "Rpb3.copy_269-288_bead resolution=100": 1, - "Rpb3.copy_289-308_bead resolution=1": 1, - "Rpb3.copy_289-308_bead resolution=10": 1, - "Rpb3.copy_289-308_bead resolution=100": 1, - "Rpb3.copy_309-318_bead resolution=1": 1, - "Rpb3.copy_309-318_bead resolution=10": 1, - "Rpb3.copy_309-318_bead resolution=100": 1, - "Rpb3_1-2_bead resolution=1": 1, - "Rpb3_1-2_bead resolution=10": 1, - "Rpb3_1-2_bead resolution=100": 1, - "Rpb3_3-268_pdb resolution=1": 266, - "Rpb3_3-268_pdb resolution=10": 0, - "Rpb3_3-268_pdb resolution=100": 0, - "Rpb3_3-268_pdb#2 resolution=1": 0, - "Rpb3_3-268_pdb#2 resolution=10": 27, - "Rpb3_3-268_pdb#2 resolution=100": 0, - "Rpb3_3-268_pdb#3 resolution=1": 0, - "Rpb3_3-268_pdb#3 resolution=10": 0, - "Rpb3_3-268_pdb#3 resolution=100": 3, - "Rpb3_269-288_bead resolution=1": 1, - "Rpb3_269-288_bead resolution=10": 1, - "Rpb3_269-288_bead resolution=100": 1, - "Rpb3_289-308_bead resolution=1": 1, - "Rpb3_289-308_bead resolution=10": 1, - "Rpb3_289-308_bead resolution=100": 1, - "Rpb3_309-318_bead resolution=1": 1, - "Rpb3_309-318_bead resolution=10": 1, - "Rpb3_309-318_bead resolution=100": 1, - "Beads": 12, - "Molecule": 803, - "resolution=1,Molecule": 721, - "resolution=10,Molecule": 85, - "resolution=100,Molecule": 21, - "resolution=1,Beads": 12, - "resolution=10,Beads": 12, - "resolution=100,Beads": 12, - "resolution=2": 721, - "resolution=7": 85, - "resolution=10": 85, - "resolution=100": 21} - - test(result_dict["All"], len(tools.select(simo))) - test(result_dict["resolution=1"], len(tools.select(simo, resolution=1))) - test(result_dict["resolution=1,resid=10"], - len(tools.select(simo, resolution=1, residue=10))) - test(result_dict["resolution=1,resid=10,name=Rpb3"], - len(tools.select(simo, resolution=1, name="Rpb3", residue=10))) - test(result_dict["resolution=1,resid=10,name=Rpb3,ambiguous"], - len(tools.select(simo, resolution=1, name="Rpb3", - name_is_ambiguous=True, residue=10))) - test(result_dict["resolution=1,resid=10,name=Rpb4,ambiguous"], - len(tools.select(simo, resolution=1, name="Rpb4", - name_is_ambiguous=True, residue=10))) - test(result_dict["resolution=1,resrange=(10,20),name=Rpb3"], - len(tools.select(simo, resolution=1, name="Rpb3", - first_residue=10, last_residue=20))) - test(result_dict["resolution=1,resrange=(10,20),name=Rpb3,ambiguous"], - len(tools.select(simo, resolution=1, name="Rpb3", - name_is_ambiguous=True, first_residue=10, - last_residue=20))) - test(result_dict["resolution=10,resrange=(10,20),name=Rpb3"], - len(tools.select(simo, resolution=10, name="Rpb3", - first_residue=10, last_residue=20))) - test(result_dict["resolution=10,resrange=(10,20),name=Rpb3,ambiguous"], - len(tools.select(simo, resolution=10, name="Rpb3", - name_is_ambiguous=True, first_residue=10, - last_residue=20))) - test(result_dict["resolution=100,resrange=(10,20),name=Rpb3"], - len(tools.select(simo, resolution=100, name="Rpb3", - first_residue=10, last_residue=20))) - test(result_dict["resolution=100,resrange=(10,20),name=Rpb3,ambiguous"], - len(tools.select(simo, resolution=100, name="Rpb3", - name_is_ambiguous=True, first_residue=10, - last_residue=20))) - - for key in hierarchies: - seen = {} - for h in hierarchies[key]: - # Handle duplicate names - if h.get_name() in seen: - name = h.get_name() + "#%d" % seen[h.get_name()] - seen[h.get_name()] += 1 - else: - name = h.get_name() - seen[h.get_name()] = 2 - - test(result_dict[name + " resolution=1"], - len(tools.select(simo, resolution=1, hierarchies=[h]))) - test(result_dict[name + " resolution=10"], - len(tools.select(simo, resolution=10, hierarchies=[h]))) - test(result_dict[name + " resolution=100"], - len(tools.select(simo, resolution=100, hierarchies=[h]))) - - test(result_dict["Beads"], - len(tools.select(simo, representation_type="Beads"))) - test(result_dict["Molecule"], - len(tools.select(simo, representation_type="Molecule"))) - test(result_dict["resolution=1,Molecule"], - len(tools.select(simo, resolution=1, - representation_type="Molecule"))) - test(result_dict["resolution=10,Molecule"], - len(tools.select(simo, resolution=10, - representation_type="Molecule"))) - test(result_dict["resolution=100,Molecule"], - len(tools.select(simo, resolution=100, - representation_type="Molecule"))) - test(result_dict["resolution=1,Beads"], - len(tools.select(simo, resolution=1, representation_type="Beads"))) - test(result_dict["resolution=10,Beads"], - len(tools.select(simo, resolution=10, - representation_type="Beads"))) - test(result_dict["resolution=100,Beads"], - len(tools.select(simo, resolution=100, - representation_type="Beads"))) - test(result_dict["resolution=2"], len(tools.select(simo, resolution=2))) - - test(result_dict["resolution=7"], - len(tools.select(simo, resolution=7))) - test(result_dict["resolution=10"], - len(tools.select(simo, resolution=10))) - test(result_dict["resolution=100"], - len(tools.select(simo, resolution=100))) + mol = st1.create_molecule(components[n], sequence=seqs[fastids[n]]) + atomic = mol.add_structure(pdbfile, chain_id=chains[n]) + mol.add_representation(mol.get_atomic_residues(), + resolutions=[1, 10, 100], color=colors[n]) + mol.add_representation(mol.get_non_atomic_residues(), + resolutions=[10], color=colors[n]) + if components[n] == 'Rpb3': + mol.create_clone(chains[n]) + root_hier = simo.build() + + def test(k, selection): + self.assertEqual(k, len(selection.get_selected_particle_indexes())) + + test(727, IMP.atom.Selection(root_hier)) + test(727, IMP.atom.Selection(root_hier, resolution=1)) + test(727, IMP.atom.Selection(root_hier, resolution=2)) + test(91, IMP.atom.Selection(root_hier, resolution=7)) + test(91, IMP.atom.Selection(root_hier, resolution=10)) + test(27, IMP.atom.Selection(root_hier, resolution=100)) + test(3, IMP.atom.Selection(root_hier, resolution=1, residue_index=10)) + test(1, IMP.atom.Selection(root_hier, resolution=1, residue_index=10, + molecule="Rpb3", copy_index=0)) + test(2, IMP.atom.Selection(root_hier, resolution=1, residue_index=10, + molecule="Rpb3")) + test(1, IMP.atom.Selection(root_hier, resolution=1, residue_index=10, + molecule="Rpb4")) + test(11, IMP.atom.Selection(root_hier, resolution=1, + residue_indexes=range(10,21), + molecule="Rpb3", copy_index=0)) + test(22, IMP.atom.Selection(root_hier, resolution=1, + residue_indexes=range(10,21), + molecule="Rpb3")) + test(2, IMP.atom.Selection(root_hier, resolution=10, + residue_indexes=range(10,21), + molecule="Rpb3", copy_index=0)) + test(4, IMP.atom.Selection(root_hier, resolution=10, + residue_indexes=range(10,21), + molecule="Rpb3")) + test(1, IMP.atom.Selection(root_hier, resolution=100, + residue_indexes=range(10,21), + molecule="Rpb3", copy_index=0)) + test(2, IMP.atom.Selection(root_hier, resolution=100, + residue_indexes=range(10,21), + molecule="Rpb3")) if __name__ == '__main__': diff --git a/modules/pmi/test/medium_test_stereochemistry.py b/modules/pmi/test/medium_test_stereochemistry.py index fd5fa8f087..a860135521 100644 --- a/modules/pmi/test/medium_test_stereochemistry.py +++ b/modules/pmi/test/medium_test_stereochemistry.py @@ -130,56 +130,6 @@ def test_elastic_network(self): self.assertEqual(hr.get_number_of_dihedrals(),lhelix-2) self.assertEqual(hr.get_number_of_bonds(),lhelix-4) - - def test_excluded_volume_sphere(self): - """Test excluded volume in PMI1""" - pdbfile = self.get_input_file_name("mini.pdb") - fastafile = self.get_input_file_name("mini.fasta") - - comps = ["P1", "P2", "P3"] - chains = "ABB" - colors = [0., 0.5, 1.0] - beadsize = 20 - fastids = IMP.pmi.tools.get_ids_from_fasta_file(fastafile) - - m = IMP.Model() - with IMP.allow_deprecated(): - simo = IMP.pmi.representation.Representation(m) - - simo.create_component(comps[0], color=colors[0]) - simo.add_component_sequence(comps[0], fastafile, id=fastids[0]) - simo.autobuild_model(comps[0], pdbfile, chains[0], - resolutions=[1], missingbeadsize=beadsize) - - simo.create_component(comps[1], color=colors[1]) - simo.add_component_sequence(comps[1], fastafile, id=fastids[1]) - simo.autobuild_model(comps[1], pdbfile, chains[1], - resolutions=[10], missingbeadsize=beadsize) - - simo.create_component(comps[2], color=colors[2]) - simo.add_component_sequence(comps[2], fastafile, id=fastids[1]) - simo.autobuild_model(comps[2], pdbfile, chains[2], - resolutions=[1,10], missingbeadsize=2) - - ev1 = IMP.pmi.restraints.stereochemistry.ExcludedVolumeSphere(simo) - ev1.add_to_model() - print(ev1.get_output()) - sf = IMP.core.RestraintsScoringFunction( - IMP.pmi.tools.get_restraint_set(m)) - print(sf.evaluate(False)) - self.assertEqual(len(ev1.cpc.get_all_possible_indexes()),16) - - ev2 = IMP.pmi.restraints.stereochemistry.ExcludedVolumeSphere( - simo, - [simo.hier_dict["P2"]], - [simo.hier_dict["P3"]]) - ev2.add_to_model() - print(ev2.get_output()) - sf = IMP.core.RestraintsScoringFunction( - IMP.pmi.tools.get_restraint_set(m)) - print(sf.evaluate(False)) - self.assertEqual(len(ev2.cpc.get_all_possible_indexes()),8) - def test_excluded_volume_sphere_pmi2(self): """ Tests excluded volume restraint in PMI2 """ diff --git a/modules/pmi/test/test_metadata.py b/modules/pmi/test/test_metadata.py deleted file mode 100644 index 051b374f45..0000000000 --- a/modules/pmi/test/test_metadata.py +++ /dev/null @@ -1,19 +0,0 @@ -from __future__ import print_function -import IMP.test -import IMP.pmi.representation -import ihm.location -import os - -class Tests(IMP.test.TestCase): - - def test_repr_add(self): - """Test Representation.add_metadata()""" - m = IMP.Model() - with IMP.allow_deprecated(): - r = IMP.pmi.representation.Representation(m) - r.add_metadata(ihm.location.Repository( - doi='10.5281/zenodo.46266', root='..')) - self.assertEqual(r._metadata[0]._root, os.path.abspath('..')) - -if __name__ == '__main__': - IMP.test.main() diff --git a/modules/pmi/test/test_mmcif.py b/modules/pmi/test/test_mmcif.py index 45c46aedcd..93285f87ec 100644 --- a/modules/pmi/test/test_mmcif.py +++ b/modules/pmi/test/test_mmcif.py @@ -1,6 +1,5 @@ from __future__ import print_function import IMP.test -import IMP.pmi.representation import IMP.pmi.restraints.em import IMP.pmi.mmcif import IMP.pmi.dof @@ -301,39 +300,33 @@ def test_all_datasets_all_group(self): def test_model_dumper_sphere(self): """Test ModelDumper sphere_obj output""" m = IMP.Model() - with IMP.allow_deprecated(): - simo = IMP.pmi.representation.Representation(m) po = DummyPO(None) - simo.add_protocol_output(po) - state = simo._protocol_output[0][1] - po.exclude_coordinates('Nup84', (3,4)) - simo.create_component("Nup84", True) - simo.add_component_sequence("Nup84", - self.get_input_file_name("test.fasta")) - nup84 = simo.autobuild_model("Nup84", - self.get_input_file_name("test.nup84.pdb"), - "A") - simo.create_transformed_component("Nup84.2", "Nup84", - IMP.algebra.Transformation3D(IMP.algebra.Vector3D(1,2,3))) + s = IMP.pmi.topology.System(m) + s.add_protocol_output(po) + st1 = s.create_state() + nup84 = st1.create_molecule("Nup84", "MELS", "X") + nup84.add_structure(self.get_input_file_name('test.nup84.pdb'), 'A') + nup84.add_representation(resolutions=[1]) + hier = s.build() + + state = po._last_state d = ihm.dumper._ModelDumper() - asym1 = po.asym_units['Nup84'] - asym2 = po.asym_units['Nup84.2'] - assembly = ihm.Assembly([asym1, asym2]) + asym1 = po.asym_units['Nup84.0'] + assembly = ihm.Assembly([asym1]) assembly._id = 42 s1 = ihm.representation.ResidueSegment(asym1, True, 'sphere') - s2 = ihm.representation.ResidueSegment(asym2, True, 'sphere') - representation = ihm.representation.Representation([s1, s2]) + representation = ihm.representation.Representation([s1]) representation._id = 99 protocol = ihm.protocol.Protocol() protocol._id = 93 group = ihm.model.ModelGroup(name="all models") state.append(group) - model = IMP.pmi.mmcif._Model(simo.prot, po, protocol, assembly, + model = IMP.pmi.mmcif._Model(hier, po, protocol, assembly, representation) group.append(model) - self.assertEqual(model.get_rmsf('Nup84', (1,)), None) + self.assertEqual(model.get_rmsf('Nup84.0', (1,)), None) fh = StringIO() self.assign_entity_asym_ids(po.system) w = ihm.format.CifWriter(fh) @@ -377,8 +370,8 @@ def test_model_dumper_sphere(self): _ihm_sphere_obj_site.model_id 1 1 1 1 A -8.986 11.688 -5.817 3.068 . 1 2 1 2 2 A -8.986 11.688 -5.817 2.997 . 1 -3 1 1 1 B -7.986 13.688 -2.817 3.068 . 1 -4 1 2 2 B -7.986 13.688 -2.817 2.997 . 1 +3 1 3 3 A -8.986 11.688 -5.817 3.052 . 1 +4 1 4 4 A -8.986 11.688 -5.817 2.609 . 1 # """) @@ -455,13 +448,14 @@ def test_model_dumper_atom(self): _atom_site.Cartn_x _atom_site.Cartn_y _atom_site.Cartn_z +_atom_site.occupancy _atom_site.label_entity_id _atom_site.auth_asym_id _atom_site.B_iso_or_equiv _atom_site.pdbx_PDB_model_num _atom_site.ihm_model_id -ATOM 1 . CA . MET 1 A -8.986 11.688 -5.817 1 A . 1 1 -ATOM 2 . CA . GLU 2 A -8.986 11.688 -5.817 1 A . 1 1 +ATOM 1 . CA . MET 1 A -8.986 11.688 -5.817 . 1 A . 1 1 +ATOM 2 . CA . GLU 2 A -8.986 11.688 -5.817 . 1 A . 1 1 # # loop_ @@ -629,8 +623,9 @@ def test_starting_model_dumper(self): _ihm_starting_model_details.starting_model_auth_asym_id _ihm_starting_model_details.starting_model_sequence_offset _ihm_starting_model_details.dataset_list_id -1 1 Nup84 A 7 'comparative model' A 0 3 -2 2 Nup85 B 8 'comparative model' A -7 4 +_ihm_starting_model_details.description +1 1 Nup84 A 7 'comparative model' A 0 3 . +2 2 Nup85 B 8 'comparative model' A -7 4 . # # loop_ @@ -682,20 +677,6 @@ def test_starting_model_dumper(self): # """) - def get_dumper_sources(self, pdbname): - m = IMP.Model() - with IMP.allow_deprecated(): - simo = IMP.pmi.representation.Representation(m) - po = DummyPO(None) - simo.add_protocol_output(po) - state = simo._protocol_output[0][1] - chain = 'A' - fragment = IMP.pmi.mmcif._PDBFragment(state, "mypdb", 1, 10, 0, - pdbname, chain, None) - model = IMP.pmi.mmcif._StartingModel(fragment) - sources = po.starting_model_dump.get_sources(model, pdbname, chain) - return m, model, sources - def test_protocol_dumper(self): """Test ModelProtocolDumper output""" m = IMP.Model() @@ -754,10 +735,11 @@ def test_protocol_dumper(self): _ihm_modeling_protocol_details.ordered_flag _ihm_modeling_protocol_details.software_id _ihm_modeling_protocol_details.script_file_id +_ihm_modeling_protocol_details.description 1 1 1 2 1 'All components modeled by IMP in state State_0' Sampling -'Replica exchange monte carlo' 0 1000 YES NO NO 1 . +'Replica exchange monte carlo' 0 1000 YES NO NO 1 . . 2 1 2 2 1 'All components modeled by IMP in state State_0' Sampling -'Replica exchange monte carlo' 1000 1000 YES NO NO 1 . +'Replica exchange monte carlo' 1000 1000 YES NO NO 1 . . # """) @@ -856,10 +838,11 @@ class DummyProtocolStep(object): _ihm_modeling_post_process.dataset_group_id _ihm_modeling_post_process.software_id _ihm_modeling_post_process.script_file_id -1 1 1 1 cluster RMSD 10 90 . . . . -2 1 1 2 cluster RMSD 12 90 . . . . -3 2 1 1 cluster RMSD 34 56 . . . . -4 3 1 1 cluster RMSD 20 80 . . . . +_ihm_modeling_post_process.details +1 1 1 1 cluster RMSD 10 90 . . . . . +2 1 1 2 cluster RMSD 12 90 . . . . . +3 2 1 1 cluster RMSD 34 56 . . . . . +4 3 1 1 cluster RMSD 20 80 . . . . . # """) @@ -900,7 +883,8 @@ class DummyProtocolStep(object): _ihm_modeling_post_process.dataset_group_id _ihm_modeling_post_process.software_id _ihm_modeling_post_process.script_file_id -1 1 1 1 none none 10 10 . . . . +_ihm_modeling_post_process.details +1 1 1 1 none none 10 10 . . . . . # """) @@ -1033,10 +1017,15 @@ class DummyProtocolStep(object): class DummyRex(object): _number_of_clusters = 1 m = IMP.Model() - with IMP.allow_deprecated(): - simo = IMP.pmi.representation.Representation(m) + s = IMP.pmi.topology.System(m) + st = s.create_state() + nup84 = st.create_molecule("Nup84", "MELS", "X") + nup84.add_representation(resolutions=[1]) po = DummyPO(None) - simo.add_protocol_output(po) + s.add_protocol_output(po) + po_state = st._protocol_output[0][1] + hier = s.build() + with IMP.test.temporary_directory() as tmpdir: rex = DummyRex() rex._outputdir = tmpdir @@ -1048,9 +1037,8 @@ class DummyRex(object): fh.write("{'modelnum': 1}\n") prot = DummyProtocolStep() prot.num_models_end = 10 - po.all_protocols.add_step(prot, po._last_state) - po.add_replica_exchange_analysis(simo._protocol_output[0][1], - rex, {}) + po.all_protocols.add_step(prot, po_state) + po.add_replica_exchange_analysis(po_state, rex, {}) def test_ensemble_dumper(self): """Test dumping of simple ensembles""" @@ -1090,8 +1078,9 @@ class DummyPostProcess(object): _ihm_ensemble_info.num_ensemble_models_deposited _ihm_ensemble_info.ensemble_precision_value _ihm_ensemble_info.ensemble_file_id -1 'Ensemble 1 in state State_0' 99 1 . dRMSD 5 1 0.100 . -2 'Ensemble 2 in state State_0' 99 2 . dRMSD 5 1 0.100 42 +_ihm_ensemble_info.details +1 'Ensemble 1 in state State_0' 99 1 . dRMSD 5 1 0.100 . . +2 'Ensemble 2 in state State_0' 99 2 . dRMSD 5 1 0.100 42 . # """) @@ -1145,17 +1134,18 @@ class DummyDataset(object): class DummyRestraint(object): label = 'foo' m = IMP.Model() - with IMP.allow_deprecated(): - simo = IMP.pmi.representation.Representation(m) + s = IMP.pmi.topology.System(m) po = DummyPO(None) - simo.add_protocol_output(po) - state = simo._protocol_output[0][1] - simo.create_component("Nup84", True) - simo.add_component_sequence("Nup84", - self.get_input_file_name("test.fasta")) - nup84 = simo.autobuild_model("Nup84", - self.get_input_file_name("test.nup84.pdb"), - "A") + s.add_protocol_output(po) + state = s.create_state() + po_state = state._protocol_output[0][1] + nup84 = state.create_molecule("Nup84", "MELS", "A") + nup84.add_structure(self.get_input_file_name('test.nup84.pdb'), 'A') + nup84.add_representation(nup84.get_atomic_residues(), resolutions=[1]) + nup84.add_representation(nup84.get_non_atomic_residues(), + resolutions=[10]) + hier = s.build() + r = DummyRestraint() r.dataset = DummyDataset() r.dataset._id = 42 @@ -1170,14 +1160,16 @@ class DummyRestraint(object): nm_ex_xl = po.add_experimental_cross_link(1, 'Nup85', 2, 'Nup84', xl_group) self.assertEqual(nm_ex_xl, None) - rs = nup84[0].get_children() + sel = IMP.atom.Selection(hier, molecule='Nup84', + resolution=1) + rs = sel.get_selected_particles() sigma1 = IMP.isd.Scale.setup_particle(IMP.Particle(m), 1.0) sigma2 = IMP.isd.Scale.setup_particle(IMP.Particle(m), 0.5) psi = IMP.isd.Scale.setup_particle(IMP.Particle(m), 0.8) - po.add_cross_link(state, ex_xl, rs[0], rs[1], 42.0, sigma1, sigma2, + po.add_cross_link(po_state, ex_xl, rs[0], rs[1], 42.0, sigma1, sigma2, psi, xl_group) # Duplicates should be ignored - po.add_cross_link(state, ex_xl, rs[0], rs[1], 42.0, sigma1, sigma2, + po.add_cross_link(po_state, ex_xl, rs[0], rs[1], 42.0, sigma1, sigma2, psi, xl_group) fh = StringIO() @@ -1204,8 +1196,9 @@ class DummyRestraint(object): _ihm_cross_link_list.linker_chem_comp_descriptor_id _ihm_cross_link_list.linker_type _ihm_cross_link_list.dataset_list_id -1 1 Nup84 1 1 MET Nup84 1 2 GLU 1 foo 42 -2 2 Nup84 1 1 MET Nup84 1 3 LEU 1 foo 42 +_ihm_cross_link_list.details +1 1 Nup84 1 1 MET Nup84 1 2 GLU 1 foo 42 . +2 2 Nup84 1 1 MET Nup84 1 3 LEU 1 foo 42 . # # loop_ @@ -1333,8 +1326,7 @@ def test_add_zaxial_restraint(self): _ihm_geometric_object_list.object_type _ihm_geometric_object_list.object_name _ihm_geometric_object_list.object_description -_ihm_geometric_object_list.other_details -1 plane . . . +1 plane . . # # loop_ @@ -1353,8 +1345,7 @@ def test_add_yaxial_restraint(self): _ihm_geometric_object_list.object_type _ihm_geometric_object_list.object_name _ihm_geometric_object_list.object_description -_ihm_geometric_object_list.other_details -1 plane . . . +1 plane . . # # loop_ @@ -1373,8 +1364,7 @@ def test_add_xyradial_restraint(self): _ihm_geometric_object_list.object_type _ihm_geometric_object_list.object_name _ihm_geometric_object_list.object_description -_ihm_geometric_object_list.other_details -1 axis . . . +1 axis . . # # loop_ @@ -1389,29 +1379,28 @@ def _check_geom_restraint(self, method_name, expected_output): class MockObject(object): pass m = IMP.Model() - with IMP.allow_deprecated(): - simo = IMP.pmi.representation.Representation(m) + s = IMP.pmi.topology.System(m) po = DummyPO(None) - simo.add_protocol_output(po) - state = simo._protocol_output[0][1] - - simo.create_component("Nup84", True) - simo.add_component_sequence("Nup84", - self.get_input_file_name("test.fasta")) - nup84 = simo.autobuild_model("Nup84", - self.get_input_file_name("test.nup84.pdb"), - "A") - residues = IMP.pmi.tools.select_by_tuple(simo, "Nup84", resolution=1) + s.add_protocol_output(po) + state = s.create_state() + nup84 = state.create_molecule("Nup84", "MELS", "A") + nup84.add_structure(self.get_input_file_name('test.nup84.pdb'), 'A') + nup84.add_representation(nup84, resolutions=[1]) + hier = s.build() + po_state = po._last_state + + residues = IMP.atom.Selection(hier, molecule='Nup84', + resolution=1).get_selected_particles() pmi_r = MockObject() pmi_r.dataset = None method = getattr(po, method_name) - method(state, residues, lower_bound=4.0, + method(po_state, residues, lower_bound=4.0, upper_bound=8.0, sigma=2.0, pmi_restraint=pmi_r) # duplicate restraint should use the same feature - method(state, residues, lower_bound=4.0, + method(po_state, residues, lower_bound=4.0, upper_bound=8.0, sigma=2.0, pmi_restraint=pmi_r) self.assertRaises(ValueError, method, - state, [simo.hier_dict['Nup84']], lower_bound=4.0, + po_state, [hier], lower_bound=4.0, upper_bound=8.0, sigma=2.0, pmi_restraint=pmi_r) self.assign_entity_asym_ids(po.system) d = ihm.dumper._GeometricObjectDumper() @@ -1432,7 +1421,8 @@ class MockObject(object): _ihm_feature_list.feature_id _ihm_feature_list.feature_type _ihm_feature_list.entity_type -1 'residue range' polymer +_ihm_feature_list.details +1 'residue range' polymer . # # loop_ @@ -1497,24 +1487,23 @@ def _check_membrane_restraint(self, method_name, expected_dist_rsr): class MockObject(object): pass m = IMP.Model() - with IMP.allow_deprecated(): - simo = IMP.pmi.representation.Representation(m) + s = IMP.pmi.topology.System(m) po = DummyPO(None) - simo.add_protocol_output(po) - state = simo._protocol_output[0][1] - - simo.create_component("Nup84", True) - simo.add_component_sequence("Nup84", - self.get_input_file_name("test.fasta")) - nup84 = simo.autobuild_model("Nup84", - self.get_input_file_name("test.nup84.pdb"), - "A") - residues = IMP.pmi.tools.select_by_tuple(simo, "Nup84", resolution=1) + s.add_protocol_output(po) + state = s.create_state() + nup84 = state.create_molecule("Nup84", "MELS", "A") + nup84.add_structure(self.get_input_file_name('test.nup84.pdb'), 'A') + nup84.add_representation(nup84, resolutions=[1]) + hier = s.build() + po_state = po._last_state + + residues = IMP.atom.Selection(hier, molecule='Nup84', + resolution=1).get_selected_particles() nterm = residues[:1] pmi_r = MockObject() pmi_r.dataset = None method = getattr(po, method_name) - method(state, nterm, tor_R=4.0, tor_r=1.0, tor_th=0.1, sigma=2.0, + method(po_state, nterm, tor_R=4.0, tor_r=1.0, tor_th=0.1, sigma=2.0, pmi_restraint=pmi_r) self.assign_entity_asym_ids(po.system) d = ihm.dumper._GeometricObjectDumper() @@ -1555,8 +1544,7 @@ class MockObject(object): _ihm_geometric_object_list.object_type _ihm_geometric_object_list.object_name _ihm_geometric_object_list.object_description -_ihm_geometric_object_list.other_details -1 half-torus Membrane . . +1 half-torus Membrane . # # loop_ @@ -1587,7 +1575,8 @@ class MockObject(object): _ihm_feature_list.feature_id _ihm_feature_list.feature_type _ihm_feature_list.entity_type -1 residue polymer +_ihm_feature_list.details +1 residue polymer . # # loop_ @@ -1732,85 +1721,18 @@ class DummyProtocolStep(object): # """) - def test_metadata(self): - """Test adding metadata to ihm.System""" - # todo: remove this test; modern usage would add Software to - # po.system.software and Citations to po.system.citations instead - m = IMP.Model() - po = DummyPO(None) - with IMP.allow_deprecated(): - simo = IMP.pmi.representation.Representation(m) - simo.add_protocol_output(po) - - simo.add_metadata(ihm.Software(name='t', classification='c', - description='d', version='2.0.16', location='u')) - self.assertEqual(len(po.system.software), 2) # IMP & PMI - - simo.add_metadata(ihm.Citation(pmid='p', title='t', - journal="j", volume=13, page_range=(2927,2943), year=2014, - authors=['A B'], doi='10.1074/mcp.M114.041673')) - self.assertEqual(len(po.system.citations), 0) - - with IMP.test.temporary_directory() as tmpdir: - bar = os.path.join(tmpdir, 'bar') - with open(bar, 'w') as f: - f.write("") - local = ihm.location.WorkflowFileLocation(bar) - simo.add_metadata(local) - self.assertEqual(len(po.system.locations), 1) # This script - - po.finalize() - self.assertEqual(len(po.system.software), 3) - self.assertEqual(len(po.system.citations), 1) - self.assertEqual(len(po.system.locations), 2) - - def test_update_locations(self): - """Test update_locations() method""" - # todo: remove this test; modern usage would call - # po.system.update_locations_in_repositories() instead - m = IMP.Model() - with IMP.allow_deprecated(): - simo = IMP.pmi.representation.Representation(m) - po = DummyPO(None) - simo.add_protocol_output(po) - - with IMP.test.temporary_directory() as tmpdir: - bar = os.path.join(tmpdir, 'bar') - with open(bar, 'w') as f: - f.write("") - local = ihm.location.InputFileLocation(bar) - # No Repository set, so cannot map local to repository location - po._update_locations([local]) - self.assertEqual(local.repo, None) - - simo.add_metadata(ihm.Software( - name='test', classification='test code', - description='Some test program', - version=1, location='http://salilab.org')) - simo.add_metadata(ihm.location.Repository(doi='foo', - root=tmpdir)) - loc = ihm.location.InputFileLocation(bar) - po._update_locations([loc]) - self.assertEqual(loc.repo.doi, 'foo') - self.assertEqual(loc.path, 'bar') - # Further calls shouldn't change things - po._update_locations([loc]) - self.assertEqual(loc.repo.doi, 'foo') - self.assertEqual(loc.path, 'bar') - def test_dump_atoms_restype_mismatch(self): """Test StartingModelDumper.dump_atoms with residue type mismatch""" m = IMP.Model() po = DummyPO(None) - with IMP.allow_deprecated(): - simo = IMP.pmi.representation.Representation(m) - simo.add_protocol_output(po) - simo.create_component("Nup84", True) - simo.add_component_sequence("Nup84", - self.get_input_file_name("test.fasta")) - nup84 = simo.autobuild_model("Nup84", - self.get_input_file_name("test.nup84.pdb"), - "A") + s = IMP.pmi.topology.System(m) + s.add_protocol_output(po) + st1 = s.create_state() + nup84 = st1.create_molecule("Nup84", "MELS", "X") + nup84.add_structure(self.get_input_file_name('test.nup84.pdb'), 'A') + nup84.add_representation(resolutions=[1]) + hier = s.build() + # Create sequence mismatch po.system.entities[0].sequence = po.system.entities[0].sequence[1:] self.assign_entity_asym_ids(po.system) @@ -1946,19 +1868,15 @@ def test_model_repr_dump_add_frag(self): def test_model_repr_dump(self): """Test ModelRepresentationDumper""" m = IMP.Model() - with IMP.allow_deprecated(): - simo = IMP.pmi.representation.Representation(m) + s = IMP.pmi.topology.System(m) po = DummyPO(None) - simo.add_protocol_output(po) - po.exclude_coordinates('Nup84', (3,4)) - simo.create_component("Nup84", True) - simo.add_component_sequence("Nup84", - self.get_input_file_name("test.fasta")) - nup84 = simo.autobuild_model("Nup84", - self.get_input_file_name("test.nup84.pdb"), - "A") - simo.create_transformed_component("Nup84.2", "Nup84", - IMP.algebra.Transformation3D(IMP.algebra.Vector3D(1,2,3))) + s.add_protocol_output(po) + state = s.create_state() + nup84 = state.create_molecule("Nup84", "ME", "A") + nup84.add_structure(self.get_input_file_name('test.nup84.pdb'), 'A') + nup84.add_representation(nup84, resolutions=[1]) + hier = s.build() + fh = StringIO() w = ihm.format.CifWriter(fh) self.assign_entity_asym_ids(po.system) @@ -1969,14 +1887,14 @@ def test_model_repr_dump(self): d.finalize(po.system) d.dump(po.system, w) r, = po.system.orphan_representations - self.assertEqual([f.asym_unit.seq_id_range for f in r], [(1,2), (1,2)]) + self.assertEqual([f.asym_unit.seq_id_range for f in r], [(1,2)]) out = fh.getvalue() self.assertEqual(out, """# loop_ _ihm_model_representation.id _ihm_model_representation.name _ihm_model_representation.details -1 . . +1 'Default representation' . # # loop_ @@ -1991,8 +1909,8 @@ def test_model_repr_dump(self): _ihm_model_representation_details.model_mode _ihm_model_representation_details.model_granularity _ihm_model_representation_details.model_object_count -1 1 1 Nup84 A 1 sphere 1 flexible by-residue . -2 1 1 Nup84 B 1 sphere 1 flexible by-residue . +_ihm_model_representation_details.description +1 1 1 Nup84 A 1 sphere 1 flexible by-residue . . # """) @@ -2028,7 +1946,7 @@ def test_model_repr_dump_rigid(self): _ihm_model_representation.id _ihm_model_representation.name _ihm_model_representation.details -1 . . +1 'Default representation' . # # loop_ @@ -2043,8 +1961,9 @@ def test_model_repr_dump_rigid(self): _ihm_model_representation_details.model_mode _ihm_model_representation_details.model_granularity _ihm_model_representation_details.model_object_count -1 1 1 Nup84 A 1 sphere 1 rigid by-residue . -2 1 1 Nup84 A 2 sphere . flexible by-feature 1 +_ihm_model_representation_details.description +1 1 1 Nup84 A 1 sphere 1 rigid by-residue . . +2 1 1 Nup84 A 2 sphere . flexible by-feature 1 . # """) @@ -2054,15 +1973,11 @@ class MockSystem(ihm.System): def __init__(self): super(MockSystem, self).__init__() self.actions = [] - def update_locations_in_repositories(self, all_repos): - super(MockSystem, self).update_locations_in_repositories( - all_repos) - self.actions.append('ul') fh = StringIO() po = IMP.pmi.mmcif.ProtocolOutput(fh) po.system = MockSystem() po.flush() - self.assertEqual(po.system.actions, ['ul']) + self.assertEqual(po.system.actions, []) def test_state_prefix(self): """Test _State.get_prefixed_name()""" diff --git a/modules/pmi/test/test_mmcif_pmi2.py b/modules/pmi/test/test_mmcif_pmi2.py index 5afedf5dc5..3e02af3a72 100644 --- a/modules/pmi/test/test_mmcif_pmi2.py +++ b/modules/pmi/test/test_mmcif_pmi2.py @@ -168,7 +168,7 @@ def test_model_representation(self): _ihm_model_representation.id _ihm_model_representation.name _ihm_model_representation.details -1 . . +1 'Default representation' . # # loop_ @@ -183,8 +183,9 @@ def test_model_representation(self): _ihm_model_representation_details.model_mode _ihm_model_representation_details.model_granularity _ihm_model_representation_details.model_object_count -1 1 1 Nup84 A 1 sphere 1 flexible by-residue . -2 1 1 Nup84 A 2 sphere . flexible by-feature 2 +_ihm_model_representation_details.description +1 1 1 Nup84 A 1 sphere 1 flexible by-residue . . +2 1 1 Nup84 A 2 sphere . flexible by-feature 2 . # """) diff --git a/modules/pmi/test/test_multistate.py b/modules/pmi/test/test_multistate.py index 0f79e92502..1f66682864 100644 --- a/modules/pmi/test/test_multistate.py +++ b/modules/pmi/test/test_multistate.py @@ -2,7 +2,8 @@ import IMP import IMP.test import os -import IMP.pmi.representation +import IMP.pmi.topology +import IMP.pmi.dof import IMP.pmi.restraints.basic import IMP.core import IMP.pmi.restraints.crosslinking @@ -17,94 +18,78 @@ def test_multistate(self): length=21 inputx=70 - representations = [] - - # define the particles in state 1 - - m = IMP.Model() - with IMP.allow_deprecated(): - r = IMP.pmi.representation.Representation(m) - - r.create_component("particle1", color=0.1) - p11 = r.add_component_beads("particle1", [(1, 10)]) - r.create_component("particle2", color=0.5) - p21 = r.add_component_beads("particle2", [(1, 10)]) - r.create_component("particle3", color=1.0) - p31 = r.add_component_beads("particle3", [(1, 10)]) + s = IMP.pmi.topology.System(m) - representations.append(r) + # define the particles in state 1 + st1 = s.create_state() + mol1 = [] + for mn in range(1, 4): + mol = st1.create_molecule("particle%d" % mn, "G" * 10, "ABC"[mn-1]) + mol.add_representation(resolutions=[10]) + mol1.append(mol) # define the particles in state 2 + st2 = s.create_state() + mol2 = [] + for mn in range(1, 4): + mol = st2.create_molecule("particle%d" % mn, "G" * 10, "ABC"[mn-1]) + mol.add_representation(resolutions=[10]) + mol2.append(mol) - with IMP.allow_deprecated(): - r = IMP.pmi.representation.Representation(m) - - r.create_component("particle1", color=0.1) - p12 = r.add_component_beads("particle1", [(1, 10)]) - r.create_component("particle2", color=0.5) - p22 = r.add_component_beads("particle2", [(1, 10)]) - r.create_component("particle3", color=0.3) - p32 = r.add_component_beads("particle3", [(1, 10)]) - - representations.append(r) + hier = s.build() - # remove paricles 1 and 2 from the otpmized articles list + dof = IMP.pmi.dof.DegreesOfFreedom(m) - representations[0].floppy_bodies.pop(0) - representations[0].floppy_bodies.pop(0) - representations[1].floppy_bodies.pop(0) - representations[1].floppy_bodies.pop(0) + # only allow particle3 in each state to move + dof.create_flexible_beads(mol1[2]) + dof.create_flexible_beads(mol2[2]) - print(representations[0].floppy_bodies) - print(representations[1].floppy_bodies) + pp1, pp2, pp3 = IMP.atom.get_leaves(st1.hier) - pp1 = IMP.atom.get_leaves(p11[0])[0] - pp2 = IMP.atom.get_leaves(p21[0])[0] - pp3 = IMP.atom.get_leaves(p31[0])[0] - xyz11 = IMP.core.XYZ(pp1.get_particle()) - xyz21 = IMP.core.XYZ(pp2.get_particle()) - xyz31 = IMP.core.XYZ(pp3.get_particle()) + xyz11 = IMP.core.XYZ(pp1) + xyz21 = IMP.core.XYZ(pp2) + xyz31 = IMP.core.XYZ(pp3) xyz11.set_coordinates((0, 0, 0)) print(xyz11.get_coordinates()) xyz21.set_coordinates((inputx, 0, 0)) xyz31.set_coordinates((0, 0, 0)) - pp1 = IMP.atom.get_leaves(p12[0])[0] - pp2 = IMP.atom.get_leaves(p22[0])[0] - pp3 = IMP.atom.get_leaves(p32[0])[0] - xyz12 = IMP.core.XYZ(pp1.get_particle()) - xyz22 = IMP.core.XYZ(pp2.get_particle()) - xyz32 = IMP.core.XYZ(pp3.get_particle()) + pp1, pp2, pp3 = IMP.atom.get_leaves(st2.hier) + xyz12 = IMP.core.XYZ(pp1) + xyz22 = IMP.core.XYZ(pp2) + xyz32 = IMP.core.XYZ(pp3) xyz12.set_coordinates((0, 0, 0)) xyz22.set_coordinates((inputx, 0, 0)) xyz32.set_coordinates((inputx, 0, 0)) - eb = IMP.pmi.restraints.basic.ExternalBarrier(representations[0], 1000) + eb = IMP.pmi.restraints.basic.ExternalBarrier(hierarchies=st1.hier, + radius=1000) eb.add_to_model() - eb = IMP.pmi.restraints.basic.ExternalBarrier(representations[1], 1000) + eb = IMP.pmi.restraints.basic.ExternalBarrier(hierarchies=st2.hier, + radius=1000) eb.add_to_model() - restraints = '''# - particle2 particle3 5 5 1 1 - particle1 particle3 5 5 1 2 ''' + xldbkwc = IMP.pmi.io.crosslink.CrossLinkDataBaseKeywordsConverter() + xldbkwc.set_protein1_key("prot1") + xldbkwc.set_protein2_key("prot2") + xldbkwc.set_residue1_key("res1") + xldbkwc.set_residue2_key("res2") + xldbkwc.set_unique_id_key("id") - with IMP.allow_deprecated(): - xl = IMP.pmi.restraints.crosslinking.ISDCrossLinkMS(representations, - restraints, - length=length, - slope=0.0, - inner_slope=slope, - resolution=1.0) + cldb = IMP.pmi.io.crosslink.CrossLinkDataBase(xldbkwc) + cldb.create_set_from_file(self.get_input_file_name('multistate.csv')) - psi = xl.get_psi(0.05) + xl = IMP.pmi.restraints.crosslinking.CrossLinkingMassSpectrometryRestraint( + root_hier=hier, CrossLinkDataBase=cldb, length=length, slope=slope, + resolution=1.0) - psi[0].set_scale(psiv) + psi = xl.psi_dictionary['PSI'][0] + psi.set_scale(psiv) - sigma = xl.get_sigma(1.0) - - sigma[0].set_scale(sigmav) + sigma = xl.sigma_dictionary['SIGMA'][0] + sigma.set_scale(sigmav) xl.set_psi_is_sampled(False) xl.set_sigma_is_sampled(False) @@ -126,18 +111,17 @@ def test_multistate(self): o = IMP.pmi.output.Output() o.init_rmf( "trajectory.rmf3", - [representations[0].prot, - representations[1].prot]) + [st1.hier, st2.hier]) print(o.dictionary_rmfs) - mc = IMP.pmi.samplers.MonteCarlo(m, representations, 1.0) + mc = IMP.pmi.samplers.MonteCarlo(m, dof.get_movers(), 1.0) mc.set_simulated_annealing(min_temp=1.0, max_temp=2.0, min_temp_time=200, max_temp_time=50) - o.init_stat2("modeling.stat", [mc, xl] + representations) + o.init_stat2("modeling.stat", [mc, xl]) for i in range(1,20): @@ -149,19 +133,16 @@ def test_multistate(self): po = IMP.pmi.output.ProcessOutput("modeling.stat") + print(po.get_keys()) - self.assertEqual(len(po.get_keys()), 20) + self.assertEqual(len(po.get_keys()), 14) fs = po.get_fields( - ['ISDCrossLinkMS_Distance_interrb-State:0-5:particle1_5:particle3-1-1-0.05_None', - 'ISDCrossLinkMS_Distance_interrb-State:0-5:particle2_5:particle3-1-1-0.05_None', - 'ISDCrossLinkMS_Distance_interrb-State:1-5:particle1_5:particle3-1-1-0.05_None', - 'ISDCrossLinkMS_Distance_interrb-State:1-5:particle2_5:particle3-1-1-0.05_None', - 'SimplifiedModel_Total_Score_None', - 'ISDCrossLinkMS_Data_Score_None', - 'ISDCrossLinkMS_Linear_Score_None', - 'ISDCrossLinkMS_Psi_0.05_None']) - + ['CrossLinkingMassSpectrometryRestraint_Distance_||2.1|particle1|5|particle3|5|1|PSI|', + 'CrossLinkingMassSpectrometryRestraint_Distance_||1.1|particle2|5|particle3|5|1|PSI|', + 'CrossLinkingMassSpectrometryRestraint_Data_Score', + 'CrossLinkingMassSpectrometryRestraint_Linear_Score', + 'CrossLinkingMassSpectrometryRestraint_Psi_PSI']) print(fs.keys()) o.close_rmf("trajectory.rmf3") diff --git a/modules/pmi/test/test_provenance.py b/modules/pmi/test/test_provenance.py index b0ed61aac0..a7dc80c739 100644 --- a/modules/pmi/test/test_provenance.py +++ b/modules/pmi/test/test_provenance.py @@ -1,37 +1,45 @@ import IMP import IMP.test import IMP.pmi.macros +import IMP.pmi.topology import IMP.pmi.tools class Tests(IMP.test.TestCase): def make_representation(self): pdbfile = self.get_input_file_name("nonbond.pdb") - fastafile = self.get_input_file_name("nonbond.fasta") - fastids = IMP.pmi.tools.get_ids_from_fasta_file(fastafile) m = IMP.Model() - with IMP.allow_deprecated(): - r = IMP.pmi.representation.Representation(m) + s = IMP.pmi.topology.System(m) + state = s.create_state() - r.create_component("A", color=0.) - r.add_component_sequence("A", fastafile, id=fastids[0]) - r.autobuild_model("A", pdbfile, "A", offset=1, - resolutions=[1, 10], missingbeadsize=1) - return m, r + c = state.create_molecule("A", sequence='KF') + struc = c.add_structure(pdbfile, chain_id="A", offset=-10) + c.add_representation(struc, resolutions=[1, 10]) + root_hier = s.build() + return m, root_hier def test_pdb_provenance(self): """Make sure that provenance info is added for each input PDB file""" - m, r = self.make_representation() - pdb_frags = [f for f in IMP.pmi.tools.select(r, resolution=1) - if 'pdb' in f.get_name()] - parent = pdb_frags[0].get_parent() - prov = list(IMP.core.get_all_provenance(parent)) - self.assertEqual(len(prov), 1) - self.assertEqual(prov[0].get_filename(), - self.get_input_file_name("nonbond.pdb")) - self.assertEqual(prov[0].get_chain_id(), 'A') - self.assertEqual(prov[0].get_residue_offset(), 1) + def _check_provenance(prov): + self.assertEqual(len(prov), 1) + self.assertEqual(prov[0].get_filename(), + self.get_input_file_name("nonbond.pdb")) + self.assertEqual(prov[0].get_chain_id(), 'A') + self.assertEqual(prov[0].get_residue_offset(), -10) + m, root_hier = self.make_representation() + + # Check both resolution=1 and resolution=10 + sel = IMP.atom.Selection(root_hier, residue_index=1, resolution=1) + p = IMP.atom.Hierarchy(sel.get_selected_particles()[0]).get_parent() + prov = list(IMP.core.get_all_provenance(p)) + _check_provenance(prov) + + sel = IMP.atom.Selection(root_hier, residue_index=1, resolution=10) + p = IMP.atom.Hierarchy(sel.get_selected_particles()[0]).get_parent() + prov = list(IMP.core.get_all_provenance(p)) + _check_provenance(prov) + if __name__ == '__main__': IMP.test.main() diff --git a/modules/pmi/test/test_stopwatch.py b/modules/pmi/test/test_stopwatch.py index 4fed55a2c0..2e34c5c269 100644 --- a/modules/pmi/test/test_stopwatch.py +++ b/modules/pmi/test/test_stopwatch.py @@ -15,14 +15,14 @@ def mocked_object(parent, objname, replacement): def get_times(outkey, *args, **kwargs): """Get a sequence of times by calling Stopwatch repeatedly. Ensure that - times are reliable by overridding time.clock() to always return whole + times are reliable by overriding time.clock() to always return whole seconds.""" class MockClock(object): count = 0 def __call__(self): self.count += 1 return self.count - with mocked_object(time, 'clock', MockClock()): + with mocked_object(IMP.pmi.tools, 'process_time', MockClock()): s = IMP.pmi.tools.Stopwatch(*args, **kwargs) return [s.get_output()[outkey] for _ in range(4)] diff --git a/modules/pmi/tools/dev_tools/.travis.yml b/modules/pmi/tools/dev_tools/.travis.yml index b4f0a90316..26e96d8c67 100644 --- a/modules/pmi/tools/dev_tools/.travis.yml +++ b/modules/pmi/tools/dev_tools/.travis.yml @@ -5,6 +5,7 @@ python: - 2.7 - 3.6 - 3.7 + - 3.8 matrix: include: - dist: trusty diff --git a/modules/pmi/tools/dev_tools/git/hooks/post-checkout b/modules/pmi/tools/dev_tools/git/hooks/post-checkout index c84a009439..6c66ceaf07 100755 --- a/modules/pmi/tools/dev_tools/git/hooks/post-checkout +++ b/modules/pmi/tools/dev_tools/git/hooks/post-checkout @@ -2,8 +2,8 @@ git submodule update --recursive --init --rebase -"$(git rev-parse --show-toplevel)/tools/dev_tools/cleanup_pycs.py" -"$(git rev-parse --show-toplevel)/tools/dev_tools/setup_cmake.py" > /dev/null +"@PYTHON@" "$(git rev-parse --show-toplevel)/tools/dev_tools/cleanup_pycs.py" +"@PYTHON@" "$(git rev-parse --show-toplevel)/tools/dev_tools/setup_cmake.py" > /dev/null # Delete top-level VERSION file if untracked if git status --porcelain |grep -q '?? VERSION'; then diff --git a/modules/pmi/tools/dev_tools/git/hooks/pre-commit b/modules/pmi/tools/dev_tools/git/hooks/pre-commit index 3ea2837c62..d1ca648ddd 100755 --- a/modules/pmi/tools/dev_tools/git/hooks/pre-commit +++ b/modules/pmi/tools/dev_tools/git/hooks/pre-commit @@ -25,7 +25,7 @@ files=$(git diff --cached --name-only --diff-filter=ACM) if test -n "$files" then echo "checking standards on $files" - if python "$(git rev-parse --git-dir)/hooks/check_standards.py" $files + if @PYTHON@ "$(git rev-parse --git-dir)/hooks/check_standards.py" $files then echo "Standards checks OK." else diff --git a/modules/pmi/tools/dev_tools/git/setup_git.py b/modules/pmi/tools/dev_tools/git/setup_git.py index 59a574e3ce..be6accfc7c 100755 --- a/modules/pmi/tools/dev_tools/git/setup_git.py +++ b/modules/pmi/tools/dev_tools/git/setup_git.py @@ -39,7 +39,11 @@ out = os.path.join(hdir, os.path.split(f)[1]) if os.path.exists(out): os.unlink(out) - shutil.copy2(f, out) + with open(f) as fin: + with open(out, 'w') as fout: + for line in fin: + fout.write(line.replace('@PYTHON@', sys.executable)) + os.chmod(out, 0o755) shutil.copy2(os.path.join(dev_tools_path, "check_standards.py"), hdir) out_tools = os.path.join(hdir, "python_tools") if os.path.exists(out_tools): diff --git a/modules/pmi1 b/modules/pmi1 index be907cb357..1275076595 160000 --- a/modules/pmi1 +++ b/modules/pmi1 @@ -1 +1 @@ -Subproject commit be907cb357e7f3c63e4b00d1ad55cbfc5498a009 +Subproject commit 1275076595ef82d90fd40713e78e3e17eabf7584 diff --git a/modules/rmf/dependency/RMF/cmake_modules/IMPFindPython.cmake b/modules/rmf/dependency/RMF/cmake_modules/IMPFindPython.cmake index f926bc8278..ed8d87c14a 100644 --- a/modules/rmf/dependency/RMF/cmake_modules/IMPFindPython.cmake +++ b/modules/rmf/dependency/RMF/cmake_modules/IMPFindPython.cmake @@ -1,13 +1,32 @@ # Like cmake's FindPython but allows the user to override; should also # work (to some degree) with older cmake function(imp_find_python) + set(USE_PYTHON2 off CACHE BOOL + "Force use of Python2 (by default Python3 is used if available)") + if (${CMAKE_VERSION} VERSION_LESS "3.14.0") message(WARNING "Using old Python detection logic. Recommended to upgrade to cmake 3.14.0 or later") if(NOT DEFINED PYTHON_INCLUDE_DIRS) - execute_process(COMMAND python -c "import sys; print(sys.executable)" - WORKING_DIRECTORY ${PROJECT_BINARY_DIR} - OUTPUT_VARIABLE python_full_path - OUTPUT_STRIP_TRAILING_WHITESPACE) + if (USE_PYTHON2) + set(_SEARCH_PYTHON_BINARIES python2 python) + else() + set(_SEARCH_PYTHON_BINARIES python3 python2 python) + endif() + + foreach(pybinary ${_SEARCH_PYTHON_BINARIES}) + execute_process(COMMAND ${pybinary} -c "import sys; print(sys.executable)" + RESULT_VARIABLE retval + WORKING_DIRECTORY ${PROJECT_BINARY_DIR} + OUTPUT_VARIABLE python_full_path + OUTPUT_STRIP_TRAILING_WHITESPACE) + if(${retval} EQUAL 0) + break() + endif() + endforeach() + + if(NOT ${retval} EQUAL 0) + message(FATAL_ERROR "Could not find a suitable Python binary - looked for ${_SEARCH_PYTHON_BINARIES}") + endif() set(PYTHON_EXECUTABLE ${python_full_path} CACHE INTERNAL "" FORCE) set(PYTHON_TEST_EXECUTABLE ${python_full_path} CACHE STRING "") execute_process(COMMAND ${PYTHON_EXECUTABLE} -c "import sys; print('%d.%d.%d' % sys.version_info[:3])" @@ -50,9 +69,6 @@ function(imp_find_python) endif() else() - set(USE_PYTHON2 off CACHE BOOL - "Force use of Python2 (by default Python3 is used if available)") - if (NOT USE_PYTHON2) find_package(Python3 COMPONENTS Interpreter Development NumPy) endif() diff --git a/modules/rmf/dependency/RMF/include/CMakeLists.txt b/modules/rmf/dependency/RMF/include/CMakeLists.txt index 709a9621f6..d63b66d70b 100644 --- a/modules/rmf/dependency/RMF/include/CMakeLists.txt +++ b/modules/rmf/dependency/RMF/include/CMakeLists.txt @@ -8,7 +8,7 @@ set(decorators "${PROJECT_BINARY_DIR}/include/RMF/decorator/physics.h" "${PROJECT_BINARY_DIR}/include/RMF/decorator/publication.h") add_custom_command(OUTPUT ${decorators} - COMMAND "python" "${PROJECT_SOURCE_DIR}/tools/build/make_decorators.py" + COMMAND ${PYTHON_EXECUTABLE} "${PROJECT_SOURCE_DIR}/tools/build/make_decorators.py" DEPENDS "${PROJECT_SOURCE_DIR}/tools/build/make_decorators.py" "${PROJECT_SOURCE_DIR}/tools/build/_decorators.py" WORKING_DIRECTORY "${PROJECT_BINARY_DIR}" COMMENT "Making decorator headers") diff --git a/modules/rmf/dependency/RMF/src/CMakeLists.txt b/modules/rmf/dependency/RMF/src/CMakeLists.txt index 42f7f83412..c129ac408f 100644 --- a/modules/rmf/dependency/RMF/src/CMakeLists.txt +++ b/modules/rmf/dependency/RMF/src/CMakeLists.txt @@ -25,7 +25,8 @@ endforeach() endif() add_custom_command(OUTPUT "${PROJECT_BINARY_DIR}/src/generated/embed_jsons.h" "${PROJECT_BINARY_DIR}/src/generated/embed_jsons.cpp" - COMMAND python "${PROJECT_SOURCE_DIR}/tools/build/make-embed.py" + COMMAND ${PYTHON_EXECUTABLE} + "${PROJECT_SOURCE_DIR}/tools/build/make-embed.py" "${PROJECT_BINARY_DIR}/src/generated/" ${jsonfiles_expanded} DEPENDS ${jsonfiles} "${PROJECT_SOURCE_DIR}/tools/build/make-embed.py" COMMENT "Embedding json files" VERBATIM ) diff --git a/modules/rmf/dependency/RMF/tools/dev_tools/git/hooks/post-checkout b/modules/rmf/dependency/RMF/tools/dev_tools/git/hooks/post-checkout index c84a009439..6c66ceaf07 100755 --- a/modules/rmf/dependency/RMF/tools/dev_tools/git/hooks/post-checkout +++ b/modules/rmf/dependency/RMF/tools/dev_tools/git/hooks/post-checkout @@ -2,8 +2,8 @@ git submodule update --recursive --init --rebase -"$(git rev-parse --show-toplevel)/tools/dev_tools/cleanup_pycs.py" -"$(git rev-parse --show-toplevel)/tools/dev_tools/setup_cmake.py" > /dev/null +"@PYTHON@" "$(git rev-parse --show-toplevel)/tools/dev_tools/cleanup_pycs.py" +"@PYTHON@" "$(git rev-parse --show-toplevel)/tools/dev_tools/setup_cmake.py" > /dev/null # Delete top-level VERSION file if untracked if git status --porcelain |grep -q '?? VERSION'; then diff --git a/modules/rmf/dependency/RMF/tools/dev_tools/git/hooks/pre-commit b/modules/rmf/dependency/RMF/tools/dev_tools/git/hooks/pre-commit index 3ea2837c62..d1ca648ddd 100755 --- a/modules/rmf/dependency/RMF/tools/dev_tools/git/hooks/pre-commit +++ b/modules/rmf/dependency/RMF/tools/dev_tools/git/hooks/pre-commit @@ -25,7 +25,7 @@ files=$(git diff --cached --name-only --diff-filter=ACM) if test -n "$files" then echo "checking standards on $files" - if python "$(git rev-parse --git-dir)/hooks/check_standards.py" $files + if @PYTHON@ "$(git rev-parse --git-dir)/hooks/check_standards.py" $files then echo "Standards checks OK." else diff --git a/modules/rmf/dependency/RMF/tools/dev_tools/git/setup_git.py b/modules/rmf/dependency/RMF/tools/dev_tools/git/setup_git.py index 59a574e3ce..be6accfc7c 100755 --- a/modules/rmf/dependency/RMF/tools/dev_tools/git/setup_git.py +++ b/modules/rmf/dependency/RMF/tools/dev_tools/git/setup_git.py @@ -39,7 +39,11 @@ out = os.path.join(hdir, os.path.split(f)[1]) if os.path.exists(out): os.unlink(out) - shutil.copy2(f, out) + with open(f) as fin: + with open(out, 'w') as fout: + for line in fin: + fout.write(line.replace('@PYTHON@', sys.executable)) + os.chmod(out, 0o755) shutil.copy2(os.path.join(dev_tools_path, "check_standards.py"), hdir) out_tools = os.path.join(hdir, "python_tools") if os.path.exists(out_tools): diff --git a/modules/saxs/README.md b/modules/saxs/README.md index 3c52d4f8ed..9c6b66c99b 100644 --- a/modules/saxs/README.md +++ b/modules/saxs/README.md @@ -23,7 +23,7 @@ Calculate chi value for SAXS profiles. # compute_vr {#compute_vr_bin} Calculate volatility ratio for SAXS profiles. -# rg {#rg_bin} +# compute_rg {#compute_rg_bin} Compute radius of gyration from SAXS profiles. # validate_profile {#validate_profile_bin} diff --git a/modules/saxs/bin/cluster_profiles.cpp b/modules/saxs/bin/cluster_profiles.cpp index a41945bab3..9b8c5b30d6 100644 --- a/modules/saxs/bin/cluster_profiles.cpp +++ b/modules/saxs/bin/cluster_profiles.cpp @@ -31,9 +31,9 @@ Provide a text file with profile filenames:\n\ exp_profile_file\nfit_file1\nfit_file2\n...\nfit_filen\n"); desc.add_options()("help", "Clusters input profiles that were previously fitted \ -to exp_profile. Please profile the exp_profile and at least two fit files.\n")( +to exp_profile. Please provide the exp_profile and at least two fit files.\n")( "input-files", po::value >(), - "input profile files")( + "input profile-list file")( "threshold,t", po::value(&threshold)->default_value(1.0), "chi value for profile similarity (default = 1.0)")( "reference_profile,r", po::value(&reference_profile_file), diff --git a/modules/saxs/bin/compute_chi.cpp b/modules/saxs/bin/compute_chi.cpp index 13724a70e5..2a4dd587ee 100644 --- a/modules/saxs/bin/compute_chi.cpp +++ b/modules/saxs/bin/compute_chi.cpp @@ -46,7 +46,7 @@ The chi value is computed relative to the first profile using its error column") } if (vm.count("offset")) use_offset = true; - IMP::Vector exp_profiles; + IMP::saxs::Profiles exp_profiles; for (unsigned int i = 0; i < files.size(); i++) { // check if file exists std::ifstream in_file(files[i].c_str()); @@ -55,7 +55,7 @@ The chi value is computed relative to the first profile using its error column") exit(1); } - IMP::saxs::Profile *profile = new IMP::saxs::Profile(files[i]); + IMP_NEW(IMP::saxs::Profile, profile, (files[i])); if (profile->size() == 0) { std::cerr << "can't parse input file " << files[i] << std::endl; return 1; diff --git a/modules/saxs/bin/rg.cpp b/modules/saxs/bin/compute_rg.cpp similarity index 100% rename from modules/saxs/bin/rg.cpp rename to modules/saxs/bin/compute_rg.cpp diff --git a/modules/saxs/bin/compute_vr.cpp b/modules/saxs/bin/compute_vr.cpp index 325f7e6341..567e797dbc 100644 --- a/modules/saxs/bin/compute_vr.cpp +++ b/modules/saxs/bin/compute_vr.cpp @@ -46,7 +46,7 @@ The chi value is computed relative to the first profile using its error column") } if (vm.count("offset")) use_offset = true; - IMP::Vector exp_profiles; + IMP::saxs::Profiles exp_profiles; for (unsigned int i = 0; i < files.size(); i++) { // check if file exists std::ifstream in_file(files[i].c_str()); @@ -55,7 +55,7 @@ The chi value is computed relative to the first profile using its error column") exit(1); } - IMP::saxs::Profile *profile = new IMP::saxs::Profile(files[i]); + IMP_NEW(IMP::saxs::Profile, profile, (files[i])); if (profile->size() == 0) { std::cerr << "can't parse input file " << files[i] << std::endl; return 1; diff --git a/modules/saxs/bin/validate_profile.cpp b/modules/saxs/bin/validate_profile.cpp index 887995bcb0..058b85da36 100644 --- a/modules/saxs/bin/validate_profile.cpp +++ b/modules/saxs/bin/validate_profile.cpp @@ -58,7 +58,7 @@ Each profile is read and written back, with simulated error added if necessary") return 0; } - IMP::Vector exp_profiles; + IMP::saxs::Profiles exp_profiles; for (unsigned int i = 0; i < files.size(); i++) { // check if file exists std::ifstream in_file(files[i].c_str()); @@ -67,7 +67,7 @@ Each profile is read and written back, with simulated error added if necessary") exit(1); } - IMP::saxs::Profile *profile = new IMP::saxs::Profile(files[i]); + IMP_NEW(IMP::saxs::Profile, profile, (files[i])); if (profile->size() == 0) { std::cerr << "can't parse input file " << files[i] << std::endl; return 1; diff --git a/modules/saxs/src/FormFactorTable.cpp b/modules/saxs/src/FormFactorTable.cpp index e650fa5c92..9799e245bf 100644 --- a/modules/saxs/src/FormFactorTable.cpp +++ b/modules/saxs/src/FormFactorTable.cpp @@ -210,6 +210,7 @@ void FormFactorTable::init_residue_type_form_factor_map() { residue_type_form_factor_map_[atom::TYR] = FormFactor(14.156, 85.986, 71.83); residue_type_form_factor_map_[atom::TRP] = FormFactor(14.945, 98.979, 84.034); residue_type_form_factor_map_[atom::VAL] = FormFactor(7.173, 53.9896, 46.817); + residue_type_form_factor_map_[atom::POP] = FormFactor(45.616, 365.99, 320.41); residue_type_form_factor_map_[atom::UNK] = FormFactor(9.037, 37.991, 28.954); } diff --git a/modules/saxs/test/test_saxs_tools.py b/modules/saxs/test/test_saxs_tools.py index d6367ea9df..48f370bdc9 100644 --- a/modules/saxs/test/test_saxs_tools.py +++ b/modules/saxs/test/test_saxs_tools.py @@ -3,15 +3,16 @@ import sys import os import re +import shutil class SAXSToolsTest(IMP.test.ApplicationTestCase): - def test_rg(self): + def test_compute_rg(self): """Simple test of Rg calculation""" print(self.get_input_file_name('6lyz.pdb')) print(self.get_input_file_name('lyzexp.dat')) - p = self.run_application('rg', + p = self.run_application('compute_rg', [self.get_input_file_name('6lyz.pdb'), self.get_input_file_name('lyzexp.dat')]) out, err = p.communicate() @@ -56,5 +57,33 @@ def test_compute_vr(self): msg="vr value output not found in " + str(out)) self.assertAlmostEqual(float(m.group(1)), 5.78, delta=0.1) + def test_validate_profile(self): + """Simple test of validate_profile tool""" + with IMP.test.temporary_directory() as tmpdir: + shutil.copy(self.get_input_file_name('weighted.dat'), tmpdir) + p = self.run_application('validate_profile', ['weighted.dat'], + cwd=tmpdir) + out, err = p.communicate() + sys.stderr.write(err) + self.assertApplicationExitedCleanly(p.returncode, err) + with open(os.path.join(tmpdir, 'weighted_v.dat')) as fh: + wc = len(fh.readlines()) + self.assertEqual(wc, 503) + + def test_validate_profile_max_q(self): + """Simple test of validate_profile tool with maxq set""" + with IMP.test.temporary_directory() as tmpdir: + shutil.copy(self.get_input_file_name('weighted.dat'), tmpdir) + p = self.run_application('validate_profile', + ['-q', '0.4', 'weighted.dat'], + cwd=tmpdir) + out, err = p.communicate() + sys.stderr.write(err) + self.assertApplicationExitedCleanly(p.returncode, err) + with open(os.path.join(tmpdir, 'weighted_v.dat')) as fh: + wc = len(fh.readlines()) + self.assertEqual(wc, 402) + + if __name__ == '__main__': IMP.test.main() diff --git a/modules/statistics/src/internal/centrality_clustering.cpp b/modules/statistics/src/internal/centrality_clustering.cpp index 41d89c826d..746ea37535 100644 --- a/modules/statistics/src/internal/centrality_clustering.cpp +++ b/modules/statistics/src/internal/centrality_clustering.cpp @@ -30,7 +30,15 @@ using graph::has_no_edges; } #endif -#include +// Add workaround for https://github.com/boostorg/graph/issues/175 +// (Note that this may not be the right #if condition; we have only seen +// the issue with Boost 1.71 on Macs with Homebrew, but other systems may +// be affected.) +#if defined(__clang__) && BOOST_VERSION == 107100 +# include "our_bc_clustering.hpp" +#else +# include +#endif IMPSTATISTICS_BEGIN_INTERNAL_NAMESPACE diff --git a/modules/statistics/src/internal/our_bc_clustering.hpp b/modules/statistics/src/internal/our_bc_clustering.hpp new file mode 100644 index 0000000000..ac9e605256 --- /dev/null +++ b/modules/statistics/src/internal/our_bc_clustering.hpp @@ -0,0 +1,172 @@ +// Note that this is a slightly modified version of +// boost/graph/bc_clustering.hpp from boost 1.71, to work around +// issue https://github.com/boostorg/graph/issues/175 (compilation fails +// complaining that graph edge iterators aren't ForwardIterators) as per +// https://github.com/boostorg/graph/issues/175#issuecomment-539235011 + + +// Copyright 2004 The Trustees of Indiana University. + +// Distributed under the Boost Software License, Version 1.0. +// (See accompanying file LICENSE_1_0.txt or copy at +// http://www.boost.org/LICENSE_1_0.txt) + +// Authors: Douglas Gregor +// Andrew Lumsdaine +#ifndef BOOST_GRAPH_BETWEENNESS_CENTRALITY_CLUSTERING_HPP +#define BOOST_GRAPH_BETWEENNESS_CENTRALITY_CLUSTERING_HPP + +#include +#include +#include +#include +#include +#include +#include + +namespace boost { + +/** Threshold termination function for the betweenness centrality + * clustering algorithm. + */ +template +struct bc_clustering_threshold +{ + typedef T centrality_type; + + /// Terminate clustering when maximum absolute edge centrality is + /// below the given threshold. + explicit bc_clustering_threshold(T threshold) + : threshold(threshold), dividend(1.0) {} + + /** + * Terminate clustering when the maximum edge centrality is below + * the given threshold. + * + * @param threshold the threshold value + * + * @param g the graph on which the threshold will be calculated + * + * @param normalize when true, the threshold is compared against the + * normalized edge centrality based on the input graph; otherwise, + * the threshold is compared against the absolute edge centrality. + */ + template + bc_clustering_threshold(T threshold, const Graph& g, bool normalize = true) + : threshold(threshold), dividend(1.0) + { + if (normalize) { + typename graph_traits::vertices_size_type n = num_vertices(g); + dividend = T((n - 1) * (n - 2)) / T(2); + } + } + + /** Returns true when the given maximum edge centrality (potentially + * normalized) falls below the threshold. + */ + template + bool operator()(T max_centrality, Edge, const Graph&) + { + return (max_centrality / dividend) < threshold; + } + + protected: + T threshold; + T dividend; +}; + +/** Graph clustering based on edge betweenness centrality. + * + * This algorithm implements graph clustering based on edge + * betweenness centrality. It is an iterative algorithm, where in each + * step it compute the edge betweenness centrality (via @ref + * brandes_betweenness_centrality) and removes the edge with the + * maximum betweenness centrality. The @p done function object + * determines when the algorithm terminates (the edge found when the + * algorithm terminates will not be removed). + * + * @param g The graph on which clustering will be performed. The type + * of this parameter (@c MutableGraph) must be a model of the + * VertexListGraph, IncidenceGraph, EdgeListGraph, and Mutable Graph + * concepts. + * + * @param done The function object that indicates termination of the + * algorithm. It must be a ternary function object thats accepts the + * maximum centrality, the descriptor of the edge that will be + * removed, and the graph @p g. + * + * @param edge_centrality (UTIL/OUT) The property map that will store + * the betweenness centrality for each edge. When the algorithm + * terminates, it will contain the edge centralities for the + * graph. The type of this property map must model the + * ReadWritePropertyMap concept. Defaults to an @c + * iterator_property_map whose value type is + * @c Done::centrality_type and using @c get(edge_index, g) for the + * index map. + * + * @param vertex_index (IN) The property map that maps vertices to + * indices in the range @c [0, num_vertices(g)). This type of this + * property map must model the ReadablePropertyMap concept and its + * value type must be an integral type. Defaults to + * @c get(vertex_index, g). + */ +template +void +betweenness_centrality_clustering(MutableGraph& g, Done done, + EdgeCentralityMap edge_centrality, + VertexIndexMap vertex_index) +{ + typedef typename property_traits::value_type + centrality_type; + typedef typename graph_traits::edge_iterator edge_iterator; + typedef typename graph_traits::edge_descriptor edge_descriptor; + + if (has_no_edges(g)) return; + + // Function object that compares the centrality of edges + indirect_cmp > + cmp(edge_centrality); + + bool is_done; + do { + brandes_betweenness_centrality(g, + edge_centrality_map(edge_centrality) + .vertex_index_map(vertex_index)); + std::pair edges_iters = edges(g); + edge_descriptor e = *boost::first_max_element(edges_iters.first, + edges_iters.second, cmp); + is_done = done(get(edge_centrality, e), e, g); + if (!is_done) remove_edge(e, g); + } while (!is_done && !has_no_edges(g)); +} + +/** + * \overload + */ +template +void +betweenness_centrality_clustering(MutableGraph& g, Done done, + EdgeCentralityMap edge_centrality) +{ + betweenness_centrality_clustering(g, done, edge_centrality, + get(vertex_index, g)); +} + +/** + * \overload + */ +template +void +betweenness_centrality_clustering(MutableGraph& g, Done done) +{ + typedef typename Done::centrality_type centrality_type; + std::vector edge_centrality(num_edges(g)); + betweenness_centrality_clustering(g, done, + make_iterator_property_map(edge_centrality.begin(), get(edge_index, g)), + get(vertex_index, g)); +} + +} // end namespace boost + +#endif // BOOST_GRAPH_BETWEENNESS_CENTRALITY_CLUSTERING_HPP diff --git a/modules/test/pyext/IMP_test.init.i b/modules/test/pyext/IMP_test.init.i index bbc95ecc4a..f3b8942795 100644 --- a/modules/test/pyext/IMP_test.init.i +++ b/modules/test/pyext/IMP_test.init.i @@ -847,7 +847,7 @@ def main(*args, **keys): import subprocess class _SubprocessWrapper(subprocess.Popen): - def __init__(self, app, args): + def __init__(self, app, args, cwd=None): # For (non-Python) applications to work on Windows, the # PATH must include the directory containing built DLLs if sys.platform == 'win32' and app != sys.executable: @@ -860,7 +860,7 @@ class _SubprocessWrapper(subprocess.Popen): subprocess.Popen.__init__(self, [app]+list(args), stdin=subprocess.PIPE, stdout=subprocess.PIPE, - stderr=subprocess.PIPE, env=env, + stderr=subprocess.PIPE, env=env, cwd=cwd, universal_newlines=True) @@ -876,7 +876,7 @@ class ApplicationTestCase(TestCase): # return os.path.join(testdir, "build", "bin", filename) return filename - def run_application(self, app, args): + def run_application(self, app, args, cwd=None): """Run an application with the given list of arguments. @return a subprocess.Popen-like object containing the child stdin, stdout and stderr. @@ -885,9 +885,9 @@ class ApplicationTestCase(TestCase): if sys.platform == 'win32': # Cannot rely on PATH on wine builds, so use full pathname return _SubprocessWrapper(os.path.join(os.environ['IMP_BIN_DIR'], - filename), args) + filename), args, cwd=cwd) else: - return _SubprocessWrapper(filename, args) + return _SubprocessWrapper(filename, args, cwd=cwd) def run_python_application(self, app, args): """Run a Python application with the given list of arguments. diff --git a/setup_git.py b/setup_git.py index c655312821..15be3f0ddb 100755 --- a/setup_git.py +++ b/setup_git.py @@ -2,7 +2,13 @@ """Call the main setup_git.py. This should be copied to the main directory of your project and named setup_git.py.""" -import os +import subprocess import os.path -os.system(os.path.join("tools", "dev_tools", "git", "setup_git.py")) -os.system("git config commit.template tools/git/commit_message.txt") +import sys + + +subprocess.check_call( + [sys.executable, os.path.join("tools", "dev_tools", "git", "setup_git.py")]) + +subprocess.check_call(["git", "config", + "commit.template", "tools/git/commit_message.txt"]) diff --git a/tools/build/cmake_templates/Module.cmake b/tools/build/cmake_templates/Module.cmake index c3aeba9e7a..39d0f4cc57 100644 --- a/tools/build/cmake_templates/Module.cmake +++ b/tools/build/cmake_templates/Module.cmake @@ -13,7 +13,7 @@ imp_get_process_exit_code("Setting up module %(name)s" status ${CMAKE_BINARY_DIR if(${status} EQUAL 0) imp_execute_process("setup_swig_wrappers %(name)s" ${CMAKE_BINARY_DIR} - COMMAND %(tools_dir)sbuild/setup_swig_wrappers.py + COMMAND ${PYTHON_EXECUTABLE} %(tools_dir)sbuild/setup_swig_wrappers.py %(build_dir)s --module=%(name)s --datapath=${IMP_DATAPATH} --source=${CMAKE_SOURCE_DIR}) @@ -51,9 +51,9 @@ if(${status} EQUAL 0) COMMAND ln -s -f ../../doc/examples COMMAND ln -s -f ../../lib COMMAND ${IMP_DOXYGEN_EXECUTABLE} ../../doxygen/%(name)s/Doxyfile 2>&1 /dev/null - COMMAND %(tools_dir)sbuild/doxygen_patch_tags.py --module=%(name)s --file=../../doxygen/%(name)s/tags - COMMAND %(tools_dir)sbuild/doxygen_show_warnings.py --warn=../../doxygen/%(name)s/warnings.txt - COMMAND %(tools_dir)sbuild/doxygen_spell_check.py xml ${CMAKE_SOURCE_DIR}/%(module_dir)stest/standards_exceptions + COMMAND ${PYTHON_EXECUTABLE} %(tools_dir)sbuild/doxygen_patch_tags.py --module=%(name)s --file=../../doxygen/%(name)s/tags + COMMAND ${PYTHON_EXECUTABLE} %(tools_dir)sbuild/doxygen_show_warnings.py --warn=../../doxygen/%(name)s/warnings.txt + COMMAND ${PYTHON_EXECUTABLE} %(tools_dir)sbuild/doxygen_spell_check.py xml ${CMAKE_SOURCE_DIR}/%(module_dir)stest/standards_exceptions DEPENDS %(tags)s ${headers} ${docs} ${examples} ${CMAKE_SOURCE_DIR}/%(module_dir)sREADME.md ${IMP_%(name)s_TAG_DEPENDS} WORKING_DIRECTORY ${CMAKE_BINARY_DIR}/doxygen/%(name)s/ COMMENT "Running doxygen on %(name)s") @@ -77,7 +77,7 @@ if(${status} EQUAL 0) add_custom_command(OUTPUT ${CMAKE_BINARY_DIR}/lib/%(subdir)s/_version_check.py ${CMAKE_BINARY_DIR}/src/%(name)s_config.cpp - COMMAND %(tools_dir)sbuild/make_module_version.py --name=%(name)s --datapath=${IMP_DATAPATH} --source=${CMAKE_SOURCE_DIR} + COMMAND ${PYTHON_EXECUTABLE} %(tools_dir)sbuild/make_module_version.py --name=%(name)s --datapath=${IMP_DATAPATH} --source=${CMAKE_SOURCE_DIR} DEPENDS IMP-version WORKING_DIRECTORY ${CMAKE_BINARY_DIR}/ COMMENT "Building module version info") diff --git a/tools/build/make_containers.py b/tools/build/make_containers.py index 1ec944cbc8..296c88cb9a 100755 --- a/tools/build/make_containers.py +++ b/tools/build/make_containers.py @@ -143,7 +143,8 @@ def get_files(module, suffix, prefix, allh): out = """ add_custom_command(OUTPUT %s - COMMAND "python" "${PROJECT_SOURCE_DIR}/tools/build/make_containers.py" + COMMAND "${PYTHON_EXECUTABLE}" + "${PROJECT_SOURCE_DIR}/tools/build/make_containers.py" DEPENDS "${PROJECT_SOURCE_DIR}/tools/build/make_containers.py" %s WORKING_DIRECTORY "${PROJECT_BINARY_DIR}" COMMENT "Making decorator headers") diff --git a/tools/build/make_swig_wrapper.py b/tools/build/make_swig_wrapper.py index 7307aed643..19a8c06f8d 100755 --- a/tools/build/make_swig_wrapper.py +++ b/tools/build/make_swig_wrapper.py @@ -102,6 +102,8 @@ def patch_py_wrapper(infile, outfile, module): # as of 1.3.36 (Python imports incorrectly come out as 'import foo' # rather than 'import IMP.foo'). See also IMP bug #41 at # https://salilab.org/imp/bugs/show_bug.cgi?id=41 +# 3. Handle our custom %ifdelete directive +# (see modules/kernel/pyext/include/IMP_kernel.exceptions.i) def patch_file(infile, out, options): lines = open(infile, 'r').readlines() preproc = "IMP_%s" % options.module.upper() @@ -121,6 +123,12 @@ def patch_file(infile, out, options): line = line.replace("(VersionInfo ", "(IMP::VersionInfo ") line = line.replace("> ${TOP_DIR}/debian/imp.install rm debian/make-package.sh || exit 1 perl -pi -e "s/\@VERSION\@/$VERSION/; s/\@DATE\@/$DATE/; s/\@CODENAME\@/$CODENAME/;" debian/changelog || exit 1 if [ "${CODENAME}" = "precise" ]; then diff --git a/tools/debian/rules b/tools/debian/rules index 44065c23b6..411af33a2a 100755 --- a/tools/debian/rules +++ b/tools/debian/rules @@ -35,7 +35,7 @@ override_dh_auto_configure: override_dh_auto_build: mkdir build/logs - cd build && ../tools/nightly-tests/build_all.py --run-tests=fast --outdir=logs --summary=logs/summary.pck "make -k -j2" + cd build && ../tools/nightly-tests/build_all.py --run-tests=fast --outdir=logs --summary=logs/summary.pck "make -k -j4" override_dh_install: $(MAKE) -C build DESTDIR=$(CURDIR)/debian/tmp install @@ -52,7 +52,7 @@ override_dh_install: -DPYTHON_INCLUDE_DIRS=$${py3_inc} \ -DPYTHON_INCLUDE_PATH=$${py3_inc} -DPYTHON_LIBRARIES=$${py3_lib} \ -DUSE_PYTHON2=off \ - && $(MAKE) DESTDIR=$(CURDIR)/debian/tmp install + && $(MAKE) -j4 DESTDIR=$(CURDIR)/debian/tmp install # Bundle libTAU so users don't have to get it separately cp build/libTAU-1.0.1/lib/debian/libTAU.so.1 debian/tmp/usr/lib/*linux*/ (cd debian/tmp/usr/lib/*linux*/ && ln -sf libTAU.so.1 libTAU.so) diff --git a/tools/git/commit_message.txt b/tools/git/commit_message.txt index 239acb5132..19a0cb8b49 100644 --- a/tools/git/commit_message.txt +++ b/tools/git/commit_message.txt @@ -1,4 +1,4 @@ -One line subject (50 chars or less). +One line subject (50 chars or less) More detailed message describing the commit (if needed). Wrap lines at about 72 characters. diff --git a/tools/mac/make-package.sh b/tools/mac/make-package.sh index fef2dfb976..cccb7db82e 100755 --- a/tools/mac/make-package.sh +++ b/tools/mac/make-package.sh @@ -152,7 +152,7 @@ if [ "${TARGET_OSX_VER}" = "10.6" ]; then /usr/local/lib/libfftw3.3.dylib \ /usr/local/lib/libgsl.0.dylib \ /usr/local/lib/libgslcblas.0.dylib \ - /usr/local/lib/libhdf5.10.dylib \ + /usr/local/lib/libhdf5.103.dylib \ /usr/local/lib/libopencv_highgui.2.4.2.dylib \ /usr/local/lib/libopencv_highgui.2.4.dylib \ /usr/local/lib/libopencv_core.2.4.2.dylib \ diff --git a/tools/nightly-tests/auto-build.sh b/tools/nightly-tests/auto-build.sh index 6618bd813d..43b126dd6f 100755 --- a/tools/nightly-tests/auto-build.sh +++ b/tools/nightly-tests/auto-build.sh @@ -47,7 +47,6 @@ git clone -b ${BRANCH} -q ${GIT_TOP}/imp.git # Update any submodules, etc. if necessary (cd imp && git submodule --quiet update --init --recursive) > /dev/null -(cd imp && ./setup_git.py) > /dev/null || exit 1 if [ -d imp/modules/rmf/dependency/RMF_source ]; then # Get submodule revision @@ -66,7 +65,7 @@ if [ -e imp/VERSION ]; then IMPVERSION="`cat imp/VERSION | sed -e 's/[ /-]/./g'`" else # Make sure VERSION file is reasonable - (cd imp && tools/build/make_version.py --source=.) + (cd imp && python3 tools/build/make_version.py --source=.) if [ ${BRANCH} = "develop" ]; then # For nightly builds, prepend the date so the packages are upgradeable IMPVERSION="${SORTDATE}.develop.${shortrev}" @@ -109,7 +108,7 @@ fi # Write out list of all components compfile="${IMPINSTALL}/build/imp-components" -python < $TARGET" % (mpi, python)) diff --git a/tools/nightly-tests/test-install/test_mock.py b/tools/nightly-tests/test-install/test_mock.py index d458052660..7e6969124b 100644 --- a/tools/nightly-tests/test-install/test_mock.py +++ b/tools/nightly-tests/test-install/test_mock.py @@ -22,10 +22,12 @@ def test_modules_installed(self): # protobuf stuff also works import IMP.npctransport # Ubuntu only supports protobuf with Python 3 in 18.04 or later; - # our Windows protobuf install also is Python 2 only - py2_protobuf = frozenset(('ubuntu-trusty', 'ubuntu-xenial', - 'w32', 'w64')) - if sys.version_info[0] == 2 or mock_config not in py2_protobuf: + # our Windows protobuf install is Python 2 only; + # for RHEL8 we only have Python 3 protobuf wrappers. + py2only_pb = frozenset(('ubuntu-trusty', 'ubuntu-xenial', 'w32', 'w64')) + py3only_pb = frozenset(('epel-8-x86_64',)) + if ((sys.version_info[0] == 3 and mock_config not in py2only_pb) + or (sys.version_info[0] == 2 and mock_config not in py3only_pb)): x = IMP.npctransport.Configuration # Check that most other modules (particularly those with many # dependencies) are present @@ -45,6 +47,7 @@ def test_modules_installed(self): import IMP.symmetry import IMP.test import IMP.pmi1 + import IMP.bayesianem def test_applications_installed(self): """Check install of a fairly comprehensive list of applications""" @@ -61,7 +64,7 @@ def test_applications_installed(self): 'estimate_threshold_from_molecular_mass', 'foxs', 'imp_example_app', 'ligand_score', 'map2pca', 'mol2pca', 'multifit', 'pdb_check', 'pdb_rmf', 'resample_density', - 'rg', 'rmf3_dump', 'rmf_cat', 'rmf_display', + 'compute_rg', 'rmf3_dump', 'rmf_cat', 'rmf_display', 'rmf_frames', 'rmf_info', 'rmf_interpolate', 'rmf_pdb', 'rmf_show', 'rmf_signature', 'rmf_simplify', 'rmf_slice', 'rmf_transform', 'rmf_update', diff --git a/tools/nightly-tests/test-install/test_mpi.py b/tools/nightly-tests/test-install/test_mpi.py index 75585d994e..1861ac70dc 100644 --- a/tools/nightly-tests/test-install/test_mpi.py +++ b/tools/nightly-tests/test-install/test_mpi.py @@ -1,14 +1,23 @@ import unittest import sys +import os import subprocess + +mock_config = os.environ['MOCK_CONFIG'] + + class Tests(unittest.TestCase): def test_modules_installed(self): """Check MPI-dependent Python modules""" - # No IMP-python3-mpich package currently - if sys.version_info[0] >= 3: - self.skipTest("No Python3 support for MPI yet") + # Fedora and RHEL 8 only have a Python 3 MPI module; other systems + # only have Python 2 + if 'fedora' in mock_config or 'epel-8' in mock_config: + if sys.version_info[0] == 2: + self.skipTest("No Python2 support for MPI") + elif sys.version_info[0] >= 3: + self.skipTest("No Python3 support for MPI") import IMP.mpi import IMP.spb diff --git a/tools/rpm/IMP.spec.in b/tools/rpm/IMP.spec.in index e4d988b966..198908dddf 100644 --- a/tools/rpm/IMP.spec.in +++ b/tools/rpm/IMP.spec.in @@ -1,7 +1,20 @@ -%if 0%{?fedora} > 12 +# On modern Fedora/RHEL, use Python 3 by default (and provide an IMP-python2 +# subpackage). On older systems, the IMP package uses Python 2 only. +%if 0%{?fedora} > 12 || 0%{?rhel} >= 8 %define with_python3 1 +%define cmake_use_python2 off +%define default_python python3 %else %define with_python3 0 +%define cmake_use_python2 on +%define default_python python2 +%endif + +# Old RHEL only has a python-devel package, but RHEL8 only has python2-devel +%if 0%{?fedora} || 0%{?rhel} >= 8 +%define PYTHON2 python2 +%else +%define PYTHON2 python %endif %define with_mpich 1 @@ -24,16 +37,14 @@ Source0: imp-%{version}.tar.gz %if 0%{?with_python3} BuildRequires: python3-devel, symlinks %endif -BuildRequires: python-devel >= 2.6 -# /usr/bin/python is provided by a separate package in recent Fedora -%if 0%{?fedora} >= 29 -BuildRequires: python-unversioned-command -%endif +BuildRequires: %{PYTHON2}-devel >= 2.6 BuildRequires: hdf5-devel >= 1.8 BuildRequires: swig >= 1.3.40 BuildRequires: gsl-devel, fftw-devel -BuildRequires: zlib-devel, perl -BuildRequires: ann-devel, eigen3-devel +BuildRequires: zlib-devel, perl, eigen3-devel +%if 0%{?fedora} || 0%{?rhel} <= 7 +BuildRequires: ann-devel +%endif # Needed to build the cnmultifit module; obtain # from https://integrativemodeling.org/libTAU.html # (Or, if you don't need cnmultifit, you can comment out this line) @@ -42,8 +53,10 @@ BuildRequires: libTAU-devel # Need for npctransport module BuildRequires: protobuf-devel %if 0%{?fedora} -Requires: python2-protobuf -%else +Requires: python3-protobuf +%endif +# No Python protobuf package in RHEL8 +%if 0%{?rhel} >= 6 && 0%{?rhel} <= 7 Requires: protobuf-python %endif @@ -72,18 +85,24 @@ BuildRequires: opencv-nogui-devel %endif BuildRequires: CGAL-nogui-devel %if 0%{?rhel} < 7 -%define bundled 'libcv\\.so\\|libcxcore\\.so\\|libhighgui.so\\|libTAU\\.so\\|libCGAL\\.so' +%define bundled libcv\\.so\\|libcxcore\\.so\\|libhighgui.so\\|libTAU\\.so\\|libCGAL\\.so %else -%define bundled 'libTAU\\.so\\|libCGAL\\.so' +%define bundled libTAU\\.so\\|libCGAL\\.so %endif %else -%define bundled 'libTAU\\.so' +%define bundled libTAU\\.so %endif -# Many modules use Python's numpy and scipy modules, so pull those in +# The IMP kernel (both with Python 2 and Python3) links with numpy, so +# pull that in. Many modules also use Python's scipy module, so pull that in # (at build time for tests; at install time for using the modules). +%if 0%{?with_python3} +BuildRequires: python3-numpy, python3-scipy, python2-numpy +Requires: python3-numpy, python3-scipy +%else BuildRequires: numpy, scipy Requires: numpy, scipy +%endif # RHEL 6 ships with Python 2.6, which doesn't have the argparse module, # so pull in the extra package that provides it @@ -94,6 +113,11 @@ Requires: python-argparse %define cmake_opts -DCMAKE_INSTALL_RPATH=%{_libdir}/IMP +# Use user-visible (not "platform") Python on RHEL8 +%if 0%{?rhel} >= 8 +%define __python3 /usr/bin/python3 +%endif + %if 0%{?rhel} == 6 %define mpiprefix %{nil} %else @@ -102,7 +126,7 @@ Requires: python-argparse # Don't build debug source packages; they cause the build to fail with # error: Empty %files file [...]/debugsourcefiles.list -%if 0%{?fedora} > 26 +%if 0%{?fedora} > 26 || 0%{?rhel} >= 8 %undefine _debugsource_packages %endif @@ -136,9 +160,16 @@ as well as easy addition of new functionality. Group: Applications/Engineering Summary: Development package for IMP developers. Requires: %{name} = %{version}-%{release} -Requires: gsl-devel, fftw-devel, zlib-devel -Requires: ann-devel, eigen3-devel -Requires: boost-devel, python-devel, hdf5-devel, protobuf-devel +Requires: gsl-devel, fftw-devel, zlib-devel, eigen3-devel +Requires: boost-devel, hdf5-devel, protobuf-devel +%if 0%{?fedora} || 0%{?rhel} <= 7 +Requires: ann-devel +%endif +%if 0%{?with_python3} +Requires: python3-devel +%else +Requires: %{PYTHON2}-devel +%endif %if 0%{?fedora} Requires: CGAL-devel, opencv-devel %endif @@ -151,18 +182,19 @@ This package contains the include files for building applications that link against IMP. %if 0%{?with_python3} -%package python3 +%package python2 Group: Applications/Engineering -Summary: Python wrappers for Python 3 +Summary: Python wrappers for Python 2 Requires: %{name} = %{version}-%{release} -Requires: python3 +Requires: python2 +# No Python protobuf package in RHEL8 %if 0%{?fedora} -Requires: python3-protobuf +Requires: python2-protobuf %endif -%description python3 -This package contains wrappers for Python 3 (the base package already -includes Python 2 wrappers). +%description python2 +This package contains wrappers for Python 2 (the base package already +includes Python 3 wrappers). %endif %define MPI_MODULES mpi spb @@ -173,8 +205,8 @@ Group: Applications/Engineering Summary: MPI module and dependents, for mpich Requires: %{name} = %{version}-%{release} Requires: mpich -%if 0%{?fedora} -Requires: python2-mpich +%if 0%{?fedora} || 0%{?rhel} >= 8 +Requires: python3-mpich %endif BuildRequires: mpich-devel @@ -199,8 +231,8 @@ Group: Applications/Engineering Summary: MPI module and dependents, for openmpi Requires: %{name} = %{version}-%{release} Requires: openmpi -%if 0%{?fedora} -Requires: python2-openmpi +%if 0%{?fedora} || 0%{?rhel} >= 8 +Requires: python3-openmpi %endif BuildRequires: openmpi-devel @@ -240,12 +272,16 @@ Development files for IMP.mpi module (and dependents) with openmpi. cat < %{reqprog} #!/bin/bash -%{__find_requires} $@ | grep -v %{bundled} +%if 0%{?rhel} >= 8 && (0%{?with_mpich} || 0%{?with_openmpi}) +%{__find_requires} $@ | grep -v '%{bundled}\\|libmpi\\|libimp_mpi\\|libimp_spb' +%else +%{__find_requires} $@ | grep -v '%{bundled}' +%endif exit $? EOF cat < %{proprog} #!/bin/bash -%{__find_provides} $@ | grep -v %{bundled} +%{__find_provides} $@ | grep -v '%{bundled}' exit $? EOF chmod a+x %{reqprog} @@ -270,7 +306,7 @@ export CGAL_DIR=%{_libdir}/cmake/CGAL cmake ../imp -DCMAKE_BUILD_TYPE=Release \ -DCMAKE_INSTALL_PREFIX=%{_prefix} \ -DCMAKE_INSTALL_DOCDIR=%{_prefix}/share/doc/%{name}-%{version} \ - -DIMP_TIMEOUT_FACTOR=4 -DUSE_PYTHON2=on \ + -DIMP_TIMEOUT_FACTOR=4 -DUSE_PYTHON2=%{cmake_use_python2} \ -DIMP_DISABLED_MODULES=scratch %{cmake_opts} # Normally make will stop at the first error. Since an RPM cannot be built @@ -280,7 +316,7 @@ cmake ../imp -DCMAKE_BUILD_TYPE=Release \ # not just the first one. %if 0%{?keep_going} mkdir logs -../imp/tools/nightly-tests/build_all.py --run-tests=fast --ctest="ctest --output-on-failure" --outdir=logs --summary=logs/summary.pck "make -k" +%{default_python} ../imp/tools/nightly-tests/build_all.py --run-tests=fast --ctest="ctest --output-on-failure" --outdir=logs --summary=logs/summary.pck "make -k" %else make %endif @@ -293,46 +329,46 @@ cd build make DESTDIR=${RPM_BUILD_ROOT} install %if 0%{?with_python3} -# Build Python 3 wrappers +# Build Python 2 wrappers py2_ver=`python2 -c "import sys; print('%d.%d' % sys.version_info[:2])"` py3_ver=`python3 -c "import sys; print('%d.%d' % sys.version_info[:2])"` -py3_lib=`echo %{_libdir}/libpython3.*.so` -py3_inc=`echo /usr/include/python3.*` +py2_lib=`echo %{_libdir}/libpython2.*.so` +py2_inc=`echo /usr/include/python2.*` %if 0%{?with_mpich} module load %{mpiprefix}mpich-%{_arch} %endif cmake ../imp \ -DCMAKE_INSTALL_PREFIX=%{_prefix} \ - -DCMAKE_INSTALL_PYTHONDIR=%{_libdir}/python${py3_ver}/site-packages \ - -DSWIG_PYTHON_LIBRARIES=${py3_lib} \ - -DPYTHON_INCLUDE_DIRS=${py3_inc} \ - -DPYTHON_INCLUDE_PATH=${py3_inc} \ - -DPYTHON_LIBRARIES=${py3_lib} -DUSE_PYTHON2=off + -DCMAKE_INSTALL_PYTHONDIR=%{_libdir}/python${py2_ver}/site-packages \ + -DSWIG_PYTHON_LIBRARIES=${py2_lib} \ + -DPYTHON_INCLUDE_DIRS=${py2_inc} \ + -DPYTHON_INCLUDE_PATH=${py2_inc} \ + -DPYTHON_LIBRARIES=${py2_lib} -DUSE_PYTHON2=on make DESTDIR=${RPM_BUILD_ROOT} install %if 0%{?with_mpich} module purge %endif -# Don't build Python 3 MPI for now +# Don't build Python 2 MPI for now for mod in %{MPI_MODULES}; do - rm -rf ${RPM_BUILD_ROOT}%{_libdir}/python${py3_ver}/site-packages/IMP/${mod} - rm -f ${RPM_BUILD_ROOT}%{_libdir}/python${py3_ver}/site-packages/_IMP_${mod}.so + rm -rf ${RPM_BUILD_ROOT}%{_libdir}/python${py2_ver}/site-packages/IMP/${mod} + rm -f ${RPM_BUILD_ROOT}%{_libdir}/python${py2_ver}/site-packages/_IMP_${mod}.so done -# Replace .py files with symlinks to Python 2 files (since they are the same) -(cd ${RPM_BUILD_ROOT}%{_libdir}/python${py3_ver} \ +# Replace .py files with symlinks to Python 3 files (since they are the same) +(cd ${RPM_BUILD_ROOT}%{_libdir}/python${py2_ver} \ && find site-packages -name '*.py' \ - -exec ln -sf ${RPM_BUILD_ROOT}%{_libdir}/python${py2_ver}/\{\} \{\} \; \ + -exec ln -sf ${RPM_BUILD_ROOT}%{_libdir}/python${py3_ver}/\{\} \{\} \; \ && symlinks -rc .) %endif # Put MPI-dependent libraries and binaries in mpich directories %if 0%{?with_mpich} -(cd ${RPM_BUILD_ROOT}%{_libdir}/python2*/site-packages/ && mkdir mpich) +(cd ${RPM_BUILD_ROOT}%{_libdir}/%{default_python}*/site-packages/ && mkdir mpich) (cd ${RPM_BUILD_ROOT}%{_libdir} && mkdir -p mpich/bin mpich/lib) (cd ${RPM_BUILD_ROOT}%{_bindir} && mv spb* ${RPM_BUILD_ROOT}%{_libdir}/mpich/bin) for mod in %{MPI_MODULES}; do - (cd ${RPM_BUILD_ROOT}%{_libdir}/python2*/site-packages/ \ + (cd ${RPM_BUILD_ROOT}%{_libdir}/%{default_python}*/site-packages/ \ && mv _IMP_${mod}.so mpich/) (cd ${RPM_BUILD_ROOT}%{_libdir} && mv libimp_${mod}.so.* mpich/lib/) (cd ${RPM_BUILD_ROOT}%{_libdir} && rm libimp_${mod}.so \ @@ -341,9 +377,9 @@ done # Add MPI-dependent directory to Python search path %if 0%{?rhel} == 6 -(cd ${RPM_BUILD_ROOT}%{_libdir}/python2*/site-packages && cat > imp-mpi.pth < imp-mpi.pth < 2.12.0-1 +- 2.12.0 release. + * Thu Jul 18 2019 Ben Webb 2.11.1-1 - 2.11.1 release. diff --git a/tools/w32/add_search_path.py b/tools/w32/add_search_path.py index 4cca589484..dc9879a70a 100755 --- a/tools/w32/add_search_path.py +++ b/tools/w32/add_search_path.py @@ -1,4 +1,4 @@ -#!/usr/bin/env python +#!/usr/bin/env python3 """Patch IMP, ihm and RMF SWIG wrappers to search for Python extensions and DLLs in Python version-specific directories. These directories are created by @@ -39,6 +39,9 @@ def _add_pyext_to_path(): # Add DLL directory to PATH so Windows can find them if dlldir not in os.environ['PATH']: os.environ['PATH'] = dlldir + ';' + os.environ['PATH'] + # Python 3.8 or later don't look in PATH for DLLs + if hasattr(os, 'add_dll_directory'): + __dll_directory = os.add_dll_directory(dlldir) _add_pyext_to_path() """ @@ -60,6 +63,9 @@ def _add_pyext_to_path(): # Add DLL directory to PATH so Windows can find them if dlldir not in os.environ['PATH']: os.environ['PATH'] = dlldir + ';' + os.environ['PATH'] + # Python 3.8 or later don't look in PATH for DLLs + if hasattr(os, 'add_dll_directory'): + __dll_directory = os.add_dll_directory(dlldir) _add_pyext_to_path() """ diff --git a/tools/w32/make-package.sh b/tools/w32/make-package.sh index f6aefe071c..9299067e9a 100755 --- a/tools/w32/make-package.sh +++ b/tools/w32/make-package.sh @@ -11,7 +11,7 @@ # make DESTDIR=`pwd`/w32-inst install # # Where $w32py is the path containing Python headers and libraries. -# Repeat for all desired Python versions (2.7, 3.4, 3.5, 3.6, and 3.7 for us) +# Repeat for all desired Python versions (2.7, 3.5, 3.6, 3.7, and 3.8 for us) # # Then run (still in the binary directory) # /tools/w32/make-package.sh @@ -79,7 +79,7 @@ for app in ${ROOT}/bin/*; do done # Make Python version-specific directories for extensions (.pyd) -PYVERS="2.7 3.4 3.5 3.6 3.7" +PYVERS="2.7 3.5 3.6 3.7 3.8" for PYVER in ${PYVERS}; do mkdir ${ROOT}/python/python${PYVER} || exit 1 mkdir ${ROOT}/python/python${PYVER}/_ihm_pyd || exit 1 @@ -110,7 +110,7 @@ rm -rf ${ROOT}/bin/example \ rm -rf `find ${ROOT} -name .svn` if [ "${BITS}" = "32" ]; then - PYVERS="27 34 35 36 37" + PYVERS="27 35 36 37 38" MAKENSIS="makensis" # Add redist MSVC runtime DLLs DLLSRC=/usr/lib/w32comp/windows/system @@ -137,7 +137,7 @@ if [ "${BITS}" = "32" ]; then ${DLLSRC}/opencv_ffmpeg220.dll \ ${DLLSRC}/opencv_imgproc220.dll ${ROOT}/bin || exit 1 else - PYVERS="27 34 35 36 37" + PYVERS="27 35 36 37 38" MAKENSIS="makensis -DIMP_64BIT" # Add redist MSVC runtime DLLs DLLSRC=/usr/lib/w64comp/windows/system32 diff --git a/tools/w32/pkg-README.txt b/tools/w32/pkg-README.txt index fa9b230cb5..0e56df4fbd 100644 --- a/tools/w32/pkg-README.txt +++ b/tools/w32/pkg-README.txt @@ -4,7 +4,7 @@ so can be run simply by opening a Command Prompt, changing to this bin directory (or adding it to your PATH), and typing 'foxs'. The IMP Python libraries are automatically set up to work with Python 2.7, -3.4, 3.5, 3.6, or 3.7. (You need to download Python separately from +3.5, 3.6, 3.7, or 3.8. (You need to download Python separately from www.python.org and install it; be sure to get the 64-bit version of Python if you installed the 64-bit version of IMP, and likewise for the 32-bit version - you can't mix and match.) diff --git a/tools/w32/w32-install.nsi b/tools/w32/w32-install.nsi index 9d4cb7d640..f225dccf92 100644 --- a/tools/w32/w32-install.nsi +++ b/tools/w32/w32-install.nsi @@ -82,10 +82,10 @@ Section "" WriteRegDWORD HKLM "${UNINST_KEY}" "NoRepair" 1 WriteRegStr HKLM "Software\Python\PythonCore\2.7\PythonPath\${PRODVER}" "" "$INSTDIR\python" - WriteRegStr HKLM "Software\Python\PythonCore\3.4\PythonPath\${PRODVER}" "" "$INSTDIR\python" WriteRegStr HKLM "Software\Python\PythonCore\3.5${PYTHON_ARCH_SUFFIX}\PythonPath\${PRODVER}" "" "$INSTDIR\python" WriteRegStr HKLM "Software\Python\PythonCore\3.6${PYTHON_ARCH_SUFFIX}\PythonPath\${PRODVER}" "" "$INSTDIR\python" WriteRegStr HKLM "Software\Python\PythonCore\3.7${PYTHON_ARCH_SUFFIX}\PythonPath\${PRODVER}" "" "$INSTDIR\python" + WriteRegStr HKLM "Software\Python\PythonCore\3.8${PYTHON_ARCH_SUFFIX}\PythonPath\${PRODVER}" "" "$INSTDIR\python" !insertmacro MUI_STARTMENU_WRITE_BEGIN ${PRODVER} CreateDirectory "$SMPROGRAMS\$STARTMENU_FOLDER" @@ -115,10 +115,10 @@ Section "Uninstall" DeleteRegKey /ifempty HKLM "Software\${PRODVER}" DeleteRegKey HKLM "${UNINST_KEY}" DeleteRegKey HKLM "Software\Python\PythonCore\2.7\PythonPath\${PRODVER}" - DeleteRegKey HKLM "Software\Python\PythonCore\3.4\PythonPath\${PRODVER}" DeleteRegKey HKLM "Software\Python\PythonCore\3.5${PYTHON_ARCH_SUFFIX}\PythonPath\${PRODVER}" DeleteRegKey HKLM "Software\Python\PythonCore\3.6${PYTHON_ARCH_SUFFIX}\PythonPath\${PRODVER}" DeleteRegKey HKLM "Software\Python\PythonCore\3.7${PYTHON_ARCH_SUFFIX}\PythonPath\${PRODVER}" + DeleteRegKey HKLM "Software\Python\PythonCore\3.8${PYTHON_ARCH_SUFFIX}\PythonPath\${PRODVER}" SectionEnd