Module v1.1.0 (#108)

* Major updates to find_sources.py, incl. adding ability to query multiple epochs * Query multiple epochs * Added EPOCH_INFO class * Added Query class * Added option to query all epochs * Added more logging * Added integrity check to existing files. (#104) Any existing files that fail the integrity check will be re-downloaded. Also changed README and option help text to be clear on this point. * PNG legend changes (#103) * PNG legend changes * Change selavy sources -> selavy components. * Only show selavy legend if selavy overlay is selected. * PEP8 * Dropbox: Added epoch filter and fixed field filter (#107) * Dropbox: Added epoch filter and fixed field filter * Added ability to filter by Epoch. * Fixed field filtering for + dec fields. * PEP8 * Removed not needed re.escape * Add lightcurves and capability to query multiple epochs * Add query.py for classes related to querying the survey data * Add utils.py for functions of general use (e.g. logging) * Add analysis.py for classes related to analysing the survey data * Add option to specify nice level of scripts * Cleaned up scripts, removing unnecessary imports * Version bump * Fix minor issues (#115) * Fixed noise map units * Fixed noise map units in analysis.py * Fixed catalog ordering for manual coordinate input * Updated missing RACS pointings where possible * Added fields query script (#116) * Added fields query script The field query script allows the user to get information on a specific VAST Pilot survey field(s) quickly. It will: * Print out observation information of the requested field (SBID, start date, end date, epochs etc.) * Can select to calculate the common psf of a field per epoch, or the largest beam. * Output can be saved. Addresses issue #86. * PEP8 and docstrings * More PEP8 * Added example to readme * Clarified saving in the readmes * Query islands (#117) * Initial island commit for testing * Fix self.args in Source class * Fix island columns typo * Updated build_lightcurves to handle islands. * The rms column in the islands is 'background_noise'. * It checks for 'component_id' in the find_sources.py output to judge whether it was run in island mode. * Fixed island detection in build_lightcurves. * Added yaxis-start flag to build_lightcurve * Fixed args and added 10% to y start 0 top * When yaxis-start 0 is selected the top is the max value + 10%. * Fixed args call to yaxis_start * Fixed df call in plot_lightcurves * Added linestyle="none" to non loop plot method In the previous commit I removed the for loop to do the plotting and instead used two filters on the dataframe. So now as a series is plotted the linestyle needs to be defined as none. * Fixed yaxis-start 0 upper plot range * Added new usage to readme, edited help statement * PEP8 fixes. * Updated checksums file To support epochs 5x, 6x, 7x, 10x and 11x. * Added logger output for failed Dropbox connection To give the user a bit more information on what could possibly be going wrong. * Dropbox README update Warning about a wrong password potentially locking a link. * Minor readme update. * Fixed common beam units --common-psf option was not printing out values in arcsec and degrees. Co-authored-by: Dougal Dobie <[email protected]>
askap-vast · Feb 28, 2020 · 808e301 · 808e301
1 parent 79adbb2
commit 808e301
Show file tree

Hide file tree

Showing 26 changed files with 17,071 additions and 776 deletions.
diff --git a/BUILDLIGHTCURVES.md b/BUILDLIGHTCURVES.md
@@ -0,0 +1,45 @@
+# build\_lightcurves.py
+
+This script allows you to quickly build lightcurves of sources you have extracted using `find_sources.py`.
+
+The script will output a csv file containing:
+* The start and end datetimes of the observation
+* Integrated flux density and associated uncertainty
+* The image noise in the local region
+* A flag stating whether the measurement is a detection or an upper limit.
+
+By default the script will also plot the lightcurve, although this can be disabled using the `--no-plotting` flag
+
+# Running the script
+Prior to running this script you should query the survey data using `find_sources.py` with the `--vast-pilot` flag set to `all`. Then run `build_lightcurves.py FOLDER` where `FOLDER` is the output folder of the previous query.
+
+## Usage
+
+Most options should be self explanatory. The lightcurve plots and csv files are saved in the same directory as the input
+
+```
+usage: build_lightcurves.py [-h] [--no-plotting] [--quiet] [--debug] [--min-points MIN_POINTS]
+                            [--min-detections MIN_DETECTIONS] [--mjd] [--grid]
+                            [--yaxis-start {auto,0}] [--nice NICE]
+                            folder
+
+positional arguments:
+  folder
+
+optional arguments:
+  -h, --help            show this help message and exit
+  --no-plotting         Write lightcurves to file without plotting (default: False)
+  --quiet               Turn off non-essential terminal output. (default: False)
+  --debug               Turn on debug output. (default: False)
+  --min-points MIN_POINTS
+                        Minimum number of epochs a source must be covered by (default: 2)
+  --min-detections MIN_DETECTIONS
+                        Minimum number of times a source must be detected (default: 1)
+  --mjd                 Plot lightcurve in MJD rather than datetime. (default: False)
+  --grid                Turn on the 'grid' in the lightcurve plot. (default: False)
+  --yaxis-start {auto,0}
+                        Define where the y axis on the lightcurve plot starts from. 'auto' will let
+                        matplotlib decide the best range and '0' will start from 0.
+                        (default: auto)
+  --nice NICE           Set nice level. (default: 5)
+```
diff --git a/FINDSOURCES.md b/FINDSOURCES.md
@@ -55,7 +55,7 @@ usage: find_sources.py [-h] [--vast-pilot {0,1,2,3x,4x}] [--imsize IMSIZE] [--ma
                        [--png-zscale-contrast PNG_ZSCALE_CONTRAST] [--png-hide-beam] [--png-no-island-labels]
                        [--png-no-colorbar] [--ann] [--reg] [--stokesv] [--quiet] [--crossmatch-only]
                        [--selavy-simple] [--process-matches] [--debug] [--no-background-rms] [--find-fields]
-                       [--clobber]
+                       [--clobber] [--nice NICE]
                        "HH:MM:SS [+/-]DD:MM:SS" OR input.csv
 
 positional arguments:
@@ -115,6 +115,7 @@ optional arguments:
   --no-background-rms   Do not estimate the background RMS around each source. (default: False)
   --find-fields         Only return the associated field for each source. (default: False)
   --clobber             Overwrite the output directory if it already exists. (default: False)
+  --nice NICE           Set nice level. (default: 5)
 ```
 
 ## Inputs

diff --git a/PILOTFIELDSINFO.md b/PILOTFIELDSINFO.md
@@ -0,0 +1,94 @@
+# pilot\_fields\_info.py
+
+This script allows you to obtain basic information on fields in the VAST Pilot Survey.
+
+The script will print out a table containing the following information for a searched field(s):
+* EPOCH
+* FIELD_NAME
+* SBID 
+* RA_HMS (Beam 0)
+* DEC_DMS (Beam 0)
+* DATEOBS
+* DATEEND
+
+In addition you can request the script return the largest or calculated common psf of the field per epoch, as well as printing all the information of the individual ASKAP beams. 
+
+# Running the script
+
+There are two methods in which to define the fields you wish to query:
+
+1. Command line: enter your field(s) as so:
+```
+pilot_fields_info.py VAST_0532-50A VAST_1212+00A VAST_2257-06A
+```
+
+2. Or use an input csv file, which must have the column `field_name`:
+```
+ra,dec,name,sbid,field_name
+321.749583333333,-44.2686111111111,Q 2123-4429B,9673,VAST_2112-43A
+348.945,-59.0544444444444,ESO 148-IG02,9673,VAST_2256-56A
+```
+```
+pilot_fields_info.py my_fields.csv
+```
+
+If you wish to save the output then using the `--save` flag will write the results for each field to their own individual file in the current working directory.
+
+See the section below for details on the other options available when running.
+
+## Usage
+
+Most options should be self explanatory. The lightcurve plots and csv files are saved in the same directory as the input
+
+```
+usage: pilot_fields_info.py [-h] [--psf] [--common-psf] [--all-psf] [--save] [--quiet] [--debug]
+                            [--nice NICE]
+                            fields [fields ...]
+
+positional arguments:
+  fields        Fields to query (or csv file containing fields).
+
+optional arguments:
+  -h, --help    show this help message and exit
+  --psf         Include the largest PSF of the 36 beams that make up the field. (default: False)
+  --common-psf  Include the common PSF of the 36 beams that make up the field. (default: False)
+  --all-psf     Include all the PSF information for the field. (default: False)
+  --save        Save the resulting information. Files will be saved to the current working directory
+                in the form of 'VAST_XXXX+/-XXA_field_info.csv'." (default: False)
+  --quiet       Turn off non-essential terminal output. (default: False)
+  --debug       Turn on debug output. (default: False)
+  --nice NICE   Set nice level. (default: 5)
+```
+
+## Example
+
+```
+❯ pilot_fields_info.py VAST_0532-50A VAST_1212+00A
+[2020-02-27 16:00:18] - INFO - Will find information for the following fields:
+[2020-02-27 16:00:18] - INFO - VAST_0532-50A
+[2020-02-27 16:00:18] - INFO - VAST_1212+00A
+[2020-02-27 16:00:18] - INFO - VAST_0532-50A information:
+EPOCH    FIELD_NAME       SBID  RA_HMS (Beam 0)    DEC_DMS (Beam 0)    DATEOBS                  DATEEND
+-------  -------------  ------  -----------------  ------------------  -----------------------  -----------------------
+01       VAST_0532-50A    9668  05:29:03.722       -49:34:04.84        2019-08-27 21:16:19.910  2019-08-27 21:28:26.499
+02       VAST_0532-50A   10343  05:29:03.722       -49:34:04.84        2019-10-30 15:26:18.379  2019-10-30 15:39:14.734
+03x      VAST_0532-50A   10336  05:29:03.722       -49:34:04.84        2019-10-29 16:33:44.940  2019-10-29 16:46:11.436
+04x      VAST_0532-50A   10901  05:29:03.722       -49:34:04.84        2019-12-19 15:49:51.855  2019-12-19 16:01:48.491
+05x      VAST_0532-50A   11120  05:29:03.722       -49:34:04.84        2020-01-10 14:26:48.248  2020-01-10 14:38:44.884
+07x      VAST_0532-50A   11341  05:29:03.722       -49:34:04.84        2020-01-16 14:00:36.072  2020-01-16 14:12:22.755
+08       VAST_0532-50A   11544  05:29:03.722       -49:34:04.84        2020-01-24 12:40:29.080  2020-01-24 12:52:25.716
+09       VAST_0532-50A   11568  05:29:03.722       -49:34:04.84        2020-01-25 12:36:34.625  2020-01-25 12:48:31.261
+10x      VAST_0532-50A   11391  05:29:03.722       -49:34:04.84        2020-01-17 13:45:35.521  2020-01-17 13:57:32.157
+11x      VAST_0532-50A   11463  05:29:03.722       -49:34:04.84        2020-01-18 13:46:39.665  2020-01-18 13:58:36.301
+[2020-02-27 16:00:18] - INFO - VAST_1212+00A information:
+EPOCH    FIELD_NAME       SBID  RA_HMS (Beam 0)    DEC_DMS (Beam 0)    DATEOBS                  DATEEND
+-------  -------------  ------  -----------------  ------------------  -----------------------  -----------------------
+01       VAST_1212+00A    9669  12:10:19.001       +00:31:29.93        2019-08-28 02:26:12.637  2019-08-28 02:38:19.226
+02       VAST_1212+00A   10380  12:10:19.001       +00:31:29.93        2019-10-31 22:14:59.611  2019-10-31 22:27:06.200
+05x      VAST_1212+00A   11140  12:10:19.001       +00:31:29.93        2020-01-10 20:16:49.668  2020-01-10 20:28:36.351
+07x      VAST_1212+00A   11357  12:10:19.001       +00:31:29.93        2020-01-16 19:45:28.941  2020-01-16 19:57:25.577
+08       VAST_1212+00A   11408  12:10:19.001       +00:31:29.93        2020-01-17 19:45:44.092  2020-01-17 19:57:40.728
+09       VAST_1212+00A   11480  12:10:19.001       +00:31:29.93        2020-01-18 19:41:49.637  2020-01-18 19:53:36.320
+[2020-02-27 16:00:18] - INFO - Processing took 0.0 minutes.
+
+```
diff --git a/README.md b/README.md
@@ -24,15 +24,20 @@ pip install -r requirements.txt
 ## Current Scripts
 As part of the installation the following scripts are made available in your path:
 
-* **find\_sources.py** - A tool to swiftly search VAST Pilot data at chosen coordinates (also supports RACS if available).
-    - See [FINDSOURCES.md](FINDSOURCES.md) for full instructions.
 * **get\_vast\_pilot\_dbx.py** - A script to allow simpler downloading of the VAST Pilot survey from Dropbox.
     - See [VASTDROPBOX.md](VASTDROPBOX.md) for full instructions.
+* **pilot\_fields\_info.py** - A script to get basic information of VAST Pilot survey fields.
+    - See [PILOTFIELDSINFO.md](PILOTFIELDSINFO.md) for full instructions.
+* **find\_sources.py** - A tool to swiftly search VAST Pilot data at chosen coordinates (also supports RACS if available).
+    - See [FINDSOURCES.md](FINDSOURCES.md) for full instructions.
+* **build\_lightcurves.py** - A script to allow easy creation of source lightcurves.
+    - See [BUILDLIGHTCURVES.md](BUILDLIGHTCURVES.md) for full instructions.
     
 ## Workflow Example
 
 This is an example of a basic workflow to query three sources of interest. The steps are:
 
 1. Determine which VAST Pilot fields contain the sources of interest. Do this using the `--find-fields ` mode in `find_sources.py` (see [FINDSOURCES.md](FINDSOURCES.md)).
 2. Download the required fields by passing the above output to `get_vast_pilot_dbx.py` (see [VASTDROPBOX.md](VASTDROPBOX.md)).
-3. Run `find_sources.py` for a second time, making sure to point to the newly downloaded data above, to gather information and cut outs of the sources and their crossmatches (see [FINDSOURCES.md](FINDSOURCES.md)).
+3. Run `find_sources.py` for a second time, making sure to point to the newly downloaded data above, to gather information and cut outs of the sources and their crossmatches (see [FINDSOURCES.md](FINDSOURCES.md)).
+4. Run `build_lightcurves.py` on the output directory to create lightcurves of the sources (see [BUILDLIGHTCURVES.md](BUILDLIGHTCURVES.md)).
diff --git a/VASTDROPBOX.md b/VASTDROPBOX.md
@@ -50,8 +50,9 @@ optional arguments:
                         get the list of files from Dropbox. (default: None)
   --files-list FILES_LIST
                         Input of files to fetch. (default: None)
-  --overwrite           Overwrite any files that already exist in the output directory. (default:
-                        False)
+  --overwrite           Overwrite any files that already exist in the output directory. If overwrite is 
+                        not selected, integrity checking will still be performed on the existing files 
+                        and if the check fails, the file will be re-downloaded. (default: False)
   --debug               Set logging level to debug. (default: False)
   --dropbox-config DROPBOX_CONFIG
                         Dropbox config file to be read in containing the shared url, password and
@@ -83,6 +84,8 @@ shared_url = ENTER_URL
 password = ENTER_PASSWORD
 access_token = ENTER_ACCESS_TOKEN
 ```
+**Double check that the password is correct before running!** Because of the many calls, a wrong password can lead to the link being locked for a period of time.
+
 There is no need to put quotes around the strings. A template can be generated by using:
 ```
 get_vast_pilot_dbx.py --write-template-dropbox-config

diff --git a/bin/build_lightcurves.py b/bin/build_lightcurves.py
@@ -0,0 +1,99 @@
+#!/usr/bin/env python
+
+# Example command:
+
+# ./build_lightcurves.py outputfolder
+
+from vasttools.analysis import BuildLightcurves
+from vasttools.utils import get_logger
+
+import argparse
+import os
+
+import datetime
+
+runstart = datetime.datetime.now()
+
+
+def parse_args():
+    '''
+    Parse arguments
+
+    :returns: Argument namespace
+    :rtype: `argparse.Namespace`
+    '''
+
+    parser = argparse.ArgumentParser(
+        formatter_class=argparse.ArgumentDefaultsHelpFormatter)
+
+    parser.add_argument(
+        'folder',
+        type=str,
+        help='')
+    parser.add_argument(
+        '--no-plotting',
+        action="store_true",
+        help='Write lightcurves to file without plotting')
+    parser.add_argument(
+        '--quiet',
+        action="store_true",
+        help='Turn off non-essential terminal output.')
+    parser.add_argument(
+        '--debug',
+        action="store_true",
+        help='Turn on debug output.')
+    parser.add_argument(
+        '--min-points',
+        type=int,
+        help='Minimum number of epochs a source must be covered by',
+        default=2)
+    parser.add_argument(
+        '--min-detections',
+        type=int,
+        help='Minimum number of times a source must be detected',
+        default=1)
+    parser.add_argument(
+        '--mjd',
+        action="store_true",
+        help='Plot lightcurve in MJD rather than datetime.')
+    parser.add_argument(
+        '--grid',
+        action="store_true",
+        help="Turn on the 'grid' in the lightcurve plot.")
+    parser.add_argument(
+        '--yaxis-start',
+        type=str,
+        choices=["auto", "0"],
+        default="auto",
+        help=(
+            "Define where the y axis on the lightcurve plot starts from."
+            " 'auto' will let matplotlib decide the best range and '0' "
+            " will start from 0."
+        ))
+    parser.add_argument(
+        '--nice',
+        type=int,
+        help='Set nice level.',
+        default=5)
+
+    args = parser.parse_args()
+
+    return args
+
+
+if __name__ == '__main__':
+    args = parse_args()
+    os.nice(args.nice)
+
+    logfile = "build_lightcurves_{}.log".format(
+        runstart.strftime("%Y%m%d_%H:%M:%S"))
+    logger = get_logger(args.debug, args.quiet, logfile=logfile)
+
+    query = BuildLightcurves(args)
+    query.run_query()
+
+    runend = datetime.datetime.now()
+    runtime = runend - runstart
+    logger.info(
+        "Processing took {:.1f} minutes.".format(
+            runtime.seconds / 60.))