Merge pull request #1239 from aarchiba/clock-search

Improve clock file handling

CHANGELOG.md

@@ -7,9 +7,19 @@ and this project, at least loosely, adheres to [Semantic Versioning](https://sem
 ## Unreleased
 ### Added
 - Warning when A1DOT parameter used with DDK model
+- Added the limits="warn" or limits="error" to get_TOAs to select handling of uncorrected TOAs
+- Added functions in pint.observatory to request the status of PINT's available clock corrections
+- Added the ability to query clock correction files or observatories for their last corrected MJD
+- Added an example showing how to check the status of your clock corrections
 ### Fixed
 - WLS fitters no longer ignore EFAC/EQUAD (bug #1226)
 ### Changed
+- Clock correction files that are entirely missing are handled the same way as TOAs past the end of a clock correction file
+- Observatory objects can now note that their clock correction files include a bogus "99999" (or similar) entry at the end
+- Clock correction files are now checked for being in order (necessary for PINT's interpolation to function)
+- Observatories using TEMPO-format clock files now refer to the clock file (for example time_ao.dat) rather than via time.dat
+- Observatories that don't use clock corrections (for example CHIME uses GPS time directly) just use an empty list of clock correction files rather than files containing only zeros
+- Updated Jodrell clock corrections to include post-1997
 - DDK model will now use ICRS or ECL coordinates depending on what the input model is
 
 ## [0.8.6 == 0.8.7] 2022-05-10

docs/examples/check_clock_corrections.py

@@ -0,0 +1,41 @@
+# ---
+# jupyter:
+#   jupytext:
+#     formats: ipynb,py:percent
+#     text_representation:
+#       extension: .py
+#       format_name: percent
+#       format_version: '1.3'
+#       jupytext_version: 1.13.8
+#   kernelspec:
+#     display_name: Python 3 (ipykernel)
+#     language: python
+#     name: python3
+# ---
+
+# %% [markdown]
+# # Check the state of PINT's clock corrections
+#
+# In order to do precision pulsar timing, it is necessary to know how the observatory clocks differ from a global time standard so that TOAs can be corrected. This requires PINT to have access to a record of measured differences. This record needs to be updated when new data is available. This notebook demonstrates how you can check the status of the clock corrections in your version of PINT. The version in the documentation also records the state of the PINT distribution at the moment the documentation was generated (which is when the code was last changed).
+
+# %%
+import pint.observatory
+
+# %%
+# hide annoying INFO messages
+from loguru import logger as log
+import sys
+import pint.logging
+
+logfilter = pint.logging.LogFilter()
+log.remove()
+log.add(sys.stderr, level="WARNING", filter=logfilter, colorize=True)
+
+
+# %%
+pint.observatory.list_last_correction_mjds()
+
+# %%
+pint.observatory.check_for_new_clock_files_in_tempo12_repos()
+
+# %%

docs/explanation.rst

@@ -207,21 +207,54 @@ accuracy.
 Clock corrections
 '''''''''''''''''
 
-Not all the data that PINT uses is easily accessible for programs to
-download. Observatory clock corrections, for example, may need to be
-obtained from the observatory through various means (often talking to a
-support scientist). We intend that PINT should notify you when this is
-necessary, but be aware that you may obtain reduced accuracy if you
-have old clock correction files.
-
-The PINT distribution includes some clock files, but these are not necessarily
-up-to-date enough. They normally live in the ``src/datafiles/`` directory with
-names like ``time_gbt.dat``. There is a ``README.md`` in there describing some
-ways to update your clock files. PINT is also capable of using clock files from
-a TEMPO or TEMPO2 installation, if you have the ``TEMPO`` or ``TEMPO2``
-environment variables set. You can see what clock file your corrections are
-coming from with a command like
-``pint.observatory.get_observatory("GBT").clock_fullpath``.
+Not all the data that PINT uses is easily accessible for programs to download.
+Observatory clock corrections, for example, may need to be obtained from the
+observatory through various means (often talking to a support scientist). PINT
+tries to include all clock correction files from the TEMPO and TEMPO2
+repositories, and we intend to update the PINT source as these are updated.
+PINT should notify you when your clock files are out of date for the data you
+are using; be aware that you may obtain reduced accuracy if you have old clock
+correction files.
+
+Normally, if you try to do some operation that requires unavailable clock
+corrections, PINT will emit a warning but continue. If you want to be stricter,
+you can specify ``limit="error"`` to various functions like
+:func:`pint.toa.get_TOAs`.
+
+If you need to check how up to date your clock corrections are, you can use
+something like ``get_observatory("gbt").last_clock_correction_mjd()``: the
+function :func:`pint.observatory.Observatory.last_clock_correction_mjd` checks
+when clock corrections are valid for. For most telescopes, this combines the
+per-telescope clock correction with PINT's global GPS and BIPM clock
+corrections (both of which cannot be reliably extrapolated too far into the
+future). PINT provides two convenience functions,
+:func:`pint.observatory.list_last_correction_mjds` and
+:func:`pint.observatory.check_for_new_clock_files_in_tempo12_repos`, that will
+help you check the state of your clock corrections.
+
+If you need clock files that are not in PINT, either more recent versions or
+clock files for telescopes not included in PINT, you will have to modify PINT's
+source code. (This is not an ideal situation!) You have two options:
+
+#. Simply replace a clock correction file in the PINT version you are using.
+   The file PINT is using may be deep in a virtualenv directory somewhere, but
+   you should be able to locate it with
+   ``get_observatory("gbt").clock_fullpath``. If you replace this file, you
+   should see updated clock corrections.
+
+#. Modify ``src/pint/observatory/observatories.py`` so that the observatory you
+   are interested in points to the correct clock file. (You may have to redo
+   ``pip install`` for PINT to make this take effect.) If you set
+   ``clock_dir="TEMPO"`` or ``clock_dir="TEMPO2"`` then PINT will look in the
+   clock directory referenced by your environment variables ``$TEMPO`` or
+   ``$TEMPO2`` (and nowhere else; it will no longer find clock corrections for
+   this observatory that are included with PINT). You can also specify a
+   specific directory as ``clock_dir="/home/burnell/clock-files/"``. Editing
+   this file also allows you to choose between TEMPO- and TEMPO2-format clock
+   corrections with the ``clock_fmt`` argument.
+
+The PINT developers recognize that the situation regarding clock corrections is
+not ideal.  Possible more flexible and powerful options are under discussion.
 
 Structure of Pulsar Timing Data Formats
 ---------------------------------------

docs/tutorials.rst

@@ -34,7 +34,7 @@ For example, the file ``NGC6440E.par`` from the
 
 Examples
 --------
-   
+
 We don't really have any proper tutorials yet. But for the moment, we
 have a few examples that may be useful.  These tutorials examples are
 in the form of Jupyter notebooks, downloadable from a link at the top
@@ -51,6 +51,7 @@ notebooks you can download from the `PINT Wiki
    examples/time_a_pulsar.ipynb
    examples/PINT_walkthrough.ipynb
    examples/fit_NGC6440E.ipynb
+   examples/check_clock_corrections.ipynb
    examples/understanding_timing_models.ipynb
    examples/understanding_parameters.ipynb
    examples/build_model_from_scratch.ipynb

src/pint/data/runtime/time_ao.dat

@@ -969,10 +969,12 @@
  49683.40      88.861      88.704 3 f  26-NOV-94
  49684.38      89.092      88.856 3 f  27-NOV-94
  49685.39      89.309      89.039 3 f  28-NOV-94
- 49528.1       73.492      71.487 3 f  25-JUN-94  raw readouts
- 49544.9       72.0        73.0   3 f  11-JUL-94  no data, rough interp.-->
- 49545.9       72.0        73.1   3 f  12-JUL-94  --> need more precision
- 49557.79      74.213      74.287 3 f  24-JUL-94  your time.dat
+# Zapped these as they are out of order and the comments suggest they
+# are invalid. Anne Archibald, 2022 May 11
+# 49528.1       73.492      71.487 3 f  25-JUN-94  raw readouts
+# 49544.9       72.0        73.0   3 f  11-JUL-94  no data, rough interp.-->
+# 49545.9       72.0        73.1   3 f  12-JUL-94  --> need more precision
+# 49557.79      74.213      74.287 3 f  24-JUL-94  your time.dat
 50155.0        0.000       -0.007 3    13-Mar-1996  post-upgrade data from
 50156.0        0.000       -0.060 3    14-Mar-1996  aosun:~joe/tac/aoclk.1
 50157.0        0.000       -0.103 3    15-Mar-1996  12 October 1999

src/pint/data/runtime/time_gb140.dat

@@ -1486,8 +1486,9 @@
 51206.500       0.0        24.086 a    28-Jan-99
 51207.500       0.0        24.123 a    29-Jan-99
 51208.500       0.0        24.150 a    30-Jan-99
-51209.500       0.0        24.172 a    31-Jan-99
-51210.500       0.0        24.197 a     1-Feb-99
+# These seem to be repeated. Anne Archibald, 2022 May 11
+# 51209.500       0.0        24.172 a    31-Jan-99
+# 51210.500       0.0        24.197 a     1-Feb-99
 51209.500       0.0        24.190 a    31-Jan-99
 51210.500       0.0        24.218 a     1-Feb-99
 51211.500       0.0        24.272 a     2-Feb-99