marshallEngine ∞
the engine behind the marshall webapp.
Documentation for marshallEngine is hosted by Read the Docs (development version and master version). The code lives on github. Please report any issues you find here.
Features ∞
Installation ∞
The easiest way to install marshallEngine is to use pip (here we show the install inside of a conda environment):
conda create -n marshallEngine python=3.7 pip
conda activate marshallEngine
pip install marshallEngine
Or you can clone the github repo and install from a local version of the code:
git clone git@github.com:thespacedoctor/marshallEngine.git
cd marshallEngine
python setup.py install
To upgrade to the latest version of marshallEngine use the command:
pip install marshallEngine --upgrade
To check installation was successful run marshallEngine -v. This should return the version number of the install.
Development ∞
If you want to tinker with the code, then install in development mode. This means you can modify the code from your cloned repo:
git clone git@github.com:thespacedoctor/marshallEngine.git
cd marshallEngine
python setup.py develop
Pull requests are welcomed!
Initialisation ∞
Before using marshallEngine you need to use the init command to generate a user settings file. Running the following creates a yaml settings file in your home folder under ~/.config/marshallEngine/marshallEngine.yaml:
marshallEngine init
The file is initially populated with marshallEngine’s default settings which can be adjusted to your preference.
If at any point the user settings file becomes corrupted or you just want to start afresh, simply trash the marshallEngine.yaml file and rerun marshallEngine init.
Modifying the Settings ∞
Once created, open the settings file in any text editor and make any modifications needed.
Basic Python Setup ∞
If you plan to use marshallEngine in your own scripts you will first need to parse your settings file and set up logging etc. One quick way to do this is to use the fundamentals package to give you a logger, a settings dictionary and a database connection (if connection details given in settings file):
## SOME BASIC SETUP FOR LOGGING, SETTINGS ETC
from fundamentals import tools
from os.path import expanduser
home = expanduser("~")
settingsFile = home + "/.config/marshallEngine/marshallEngine.yaml"
su = tools(
arguments={"settingsFile": settingsFile},
docString=__doc__,
)
arguments, settings, log, dbConn = su.setup()
Todo List ∞
Todo
add usage info
create a sublime snippet for usage
write a command-line tool for this method
update package tutorial with command-line tool info if needed
(The original entry is located in /home/docs/checkouts/readthedocs.org/user_builds/marshallengine/envs/develop/lib/python3.7/site-packages/marshallEngine-1.0.9-py3.7.egg/marshallEngine/feeders/atels/data.py:docstring of marshallEngine.feeders.data.data.clean_up, line 28.)
Todo
add usage info
create a sublime snippet for usage
write a command-line tool for this method
update package tutorial with command-line tool info if needed
(The original entry is located in /home/docs/checkouts/readthedocs.org/user_builds/marshallengine/envs/develop/lib/python3.7/site-packages/marshallEngine-1.0.9-py3.7.egg/marshallEngine/feeders/atlas/data.py:docstring of marshallEngine.feeders.data.data.clean_up, line 28.)
Todo
create a frankenstein template for importer
(The original entry is located in /home/docs/checkouts/readthedocs.org/user_builds/marshallengine/envs/develop/lib/python3.7/site-packages/marshallEngine-1.0.9-py3.7.egg/marshallEngine/feeders/data.py:docstring of marshallEngine.feeders.data.data, line 6.)
Todo
add usage info
create a sublime snippet for usage
write a command-line tool for this method
update package tutorial with command-line tool info if needed
(The original entry is located in /home/docs/checkouts/readthedocs.org/user_builds/marshallengine/envs/develop/lib/python3.7/site-packages/marshallEngine-1.0.9-py3.7.egg/marshallEngine/feeders/data.py:docstring of marshallEngine.feeders.data.data.clean_up, line 28.)
Todo
add usage info
create a sublime snippet for usage
write a command-line tool for this method
update package tutorial with command-line tool info if needed
(The original entry is located in /home/docs/checkouts/readthedocs.org/user_builds/marshallengine/envs/develop/lib/python3.7/site-packages/marshallEngine-1.0.9-py3.7.egg/marshallEngine/feeders/panstarrs/data.py:docstring of marshallEngine.feeders.data.data.clean_up, line 28.)
Todo
add usage info
create a sublime snippet for usage
write a command-line tool for this method
update package tutorial with command-line tool info if needed
(The original entry is located in /home/docs/checkouts/readthedocs.org/user_builds/marshallengine/envs/develop/lib/python3.7/site-packages/marshallEngine-1.0.9-py3.7.egg/marshallEngine/feeders/tns/data.py:docstring of marshallEngine.feeders.data.data.clean_up, line 28.)
Todo
add usage info
create a sublime snippet for usage
write a command-line tool for this method
update package tutorial with command-line tool info if needed
(The original entry is located in /home/docs/checkouts/readthedocs.org/user_builds/marshallengine/envs/develop/lib/python3.7/site-packages/marshallEngine-1.0.9-py3.7.egg/marshallEngine/feeders/useradded/data.py:docstring of marshallEngine.feeders.data.data.clean_up, line 28.)
Todo
add usage info
create a sublime snippet for usage
write a command-line tool for this method
update package tutorial with command-line tool info if needed
(The original entry is located in /home/docs/checkouts/readthedocs.org/user_builds/marshallengine/envs/develop/lib/python3.7/site-packages/marshallEngine-1.0.9-py3.7.egg/marshallEngine/feeders/ztf/data.py:docstring of marshallEngine.feeders.data.data.clean_up, line 28.)
Todo
add a tutorial about
update_transient_summariesto documentation
(The original entry is located in /home/docs/checkouts/readthedocs.org/user_builds/marshallengine/envs/develop/lib/python3.7/site-packages/marshallEngine-1.0.9-py3.7.egg/marshallEngine/housekeeping/update_transient_summaries.py:docstring of marshallEngine.housekeeping.update_transient_summaries.update_transient_summaries, line 18.)
Todo
add usage info
create a sublime snippet for usage
write a command-line tool for this method
update package tutorial with command-line tool info if needed
(The original entry is located in /home/docs/checkouts/readthedocs.org/user_builds/marshallengine/envs/develop/lib/python3.7/site-packages/marshallEngine-1.0.9-py3.7.egg/marshallEngine/feeders/atels/data.py:docstring of marshallEngine.feeders.data.data.clean_up, line 28.)
Todo
add usage info
create a sublime snippet for usage
write a command-line tool for this method
update package tutorial with command-line tool info if needed
(The original entry is located in /home/docs/checkouts/readthedocs.org/user_builds/marshallengine/envs/develop/lib/python3.7/site-packages/marshallEngine-1.0.9-py3.7.egg/marshallEngine/feeders/atlas/data.py:docstring of marshallEngine.feeders.data.data.clean_up, line 28.)
Todo
create a frankenstein template for importer
(The original entry is located in /home/docs/checkouts/readthedocs.org/user_builds/marshallengine/envs/develop/lib/python3.7/site-packages/marshallEngine-1.0.9-py3.7.egg/marshallEngine/feeders/data.py:docstring of marshallEngine.feeders.data.data, line 6.)
Todo
add usage info
create a sublime snippet for usage
write a command-line tool for this method
update package tutorial with command-line tool info if needed
(The original entry is located in /home/docs/checkouts/readthedocs.org/user_builds/marshallengine/envs/develop/lib/python3.7/site-packages/marshallEngine-1.0.9-py3.7.egg/marshallEngine/feeders/data.py:docstring of marshallEngine.feeders.data.data.clean_up, line 28.)
Todo
add usage info
create a sublime snippet for usage
write a command-line tool for this method
update package tutorial with command-line tool info if needed
(The original entry is located in /home/docs/checkouts/readthedocs.org/user_builds/marshallengine/envs/develop/lib/python3.7/site-packages/marshallEngine-1.0.9-py3.7.egg/marshallEngine/feeders/panstarrs/data.py:docstring of marshallEngine.feeders.data.data.clean_up, line 28.)
Todo
add usage info
create a sublime snippet for usage
write a command-line tool for this method
update package tutorial with command-line tool info if needed
(The original entry is located in /home/docs/checkouts/readthedocs.org/user_builds/marshallengine/envs/develop/lib/python3.7/site-packages/marshallEngine-1.0.9-py3.7.egg/marshallEngine/feeders/tns/data.py:docstring of marshallEngine.feeders.data.data.clean_up, line 28.)
Todo
add usage info
create a sublime snippet for usage
write a command-line tool for this method
update package tutorial with command-line tool info if needed
(The original entry is located in /home/docs/checkouts/readthedocs.org/user_builds/marshallengine/envs/develop/lib/python3.7/site-packages/marshallEngine-1.0.9-py3.7.egg/marshallEngine/feeders/useradded/data.py:docstring of marshallEngine.feeders.data.data.clean_up, line 28.)
Todo
add usage info
create a sublime snippet for usage
write a command-line tool for this method
update package tutorial with command-line tool info if needed
(The original entry is located in /home/docs/checkouts/readthedocs.org/user_builds/marshallengine/envs/develop/lib/python3.7/site-packages/marshallEngine-1.0.9-py3.7.egg/marshallEngine/feeders/ztf/data.py:docstring of marshallEngine.feeders.data.data.clean_up, line 28.)
Todo
add a tutorial about
update_transient_summariesto documentation
(The original entry is located in /home/docs/checkouts/readthedocs.org/user_builds/marshallengine/envs/develop/lib/python3.7/site-packages/marshallEngine-1.0.9-py3.7.egg/marshallEngine/housekeeping/update_transient_summaries.py:docstring of marshallEngine.housekeeping.update_transient_summaries.update_transient_summaries, line 18.)
Todo
nice!
(The original entry is located in /home/docs/checkouts/readthedocs.org/user_builds/marshallengine/checkouts/develop/docs/source/_template_.md, line 1.)
Marshall Engine Release Notes ∞
adding tns-marker into the TNS search requests fixed atlas and panstarrs image stamp to download from new URLs
v1.0.9 - May 7, 2021
REFACTOR: resurrection code optimised to resurface a few more objects
REFACTOR: some MySQL queries rewritten and table indexes added to optimise common queries
REFACTOR: ATLAS forced photometry plotting code refactored to bring plots in-line with those produced by ATLAS forced photometry server. Biggest improvement is the more robust clipping of outlying data points.
v1.0.8 - March 17, 2021
REFACTOR: Occasional check to make sure all akas are set (i.e. more than just for transients discovered in the last 3 weeks).
FIXED: Links to the ASASSN Sky Patrol now added to all TNS reported ASASSN transient names (credentials discoverable in hover-over tool-tip)
FIXED: ATel comments where getting added correctly to associated object but ticket “ATel” drop-up menu was missing some ATel links occasionally.
v1.0.7 - February 20, 2021
ENHANCEMENT: Added cleanup function at end of ingests so objects appear in inbox quicker and akas are updated more frequently
REFACTORING: Reduced the crossmatch radius from 7 to 4 arcsec (it is easier to merge than split transients later on)
v1.0.6 - January 29, 2021
FEATURE: match transients to astronotes (not yet visualised in the webapp)
ENHANCEMENT: HTM indexing added the transientBucketSummaries table so we can spatially crossmatch
v1.0.5 - January 13, 2021
REFACTORING: atel parsing and matching within the marshall database now upgraded
v1.0.4 - January 11, 2021
REFACTORING: some database schema changes for latest version of Sherlock to run.
FIXED: added a function to recalculate sherlock original radii for merged sources. Fixes webapp visualisation.
FIXED: reduced number of ATLAS LC to be generated in a single batch.
v1.0.3 - December 10, 2020
REFACTORING: reading settings from marshall config folder instead of marshallEngine
FIXED: the save location of lightcurve files was resulting in files not being found in webapp
v1.0.2 - November 14, 2020
feature PS2 survey added as a new import (discoveries, lightcurves and stamps)
v1.0.1 - October 20, 2020
REFACTORING: added time filtering on ATLAS summary CSV file (thanks Ken)
FIXED: small fix to panstamp location map downloader
v1.0.0 - May 28, 2020
Now compatible with Python 3.* *
API Reference ∞
Modules ∞
common tools used throughout package |
|
Import codes for the various transients surveys that ‘feed’ the marshall inbox |
|
import code for the panstarrs survey |
|
import code for the panstarrs survey |
|
import code for the panstarrs survey |
|
import code for the panstarrs survey |
|
import code for the panstarrs survey |
|
import code for the panstarrs survey |
|
Housekeeping and bookkeeping actions for the marshall database |
|
Lightcurve plotting for the Marshall |
|
small services and tools for the marhall |
|
Generate the force-photometry lightcurves for ATLAS sources found in the marshall database |
|
Unit testing tools |
Classes ∞
Import the atels transient data into the marshall database |
|
cacher for the atels image stamps |
|
Import the ATLAS transient data into the marshall database |
|
cacher for the ATLAS image stamps |
|
This baseclass for the feeder survey data imports |
|
The base class for the feeder image cachers |
|
Import the PanSTARRS transient data into the marshall database |
|
cacher for the panstarrs image stamps |
|
Import the tns transient data into the marshall database |
|
cacher for the tns image stamps |
|
Import the useradded transient data into the marshall database |
|
cacher for the useradded image stamps |
|
Import the ZTF transient data into the marshall database |
|
cacher for the ZTF image stamps |
|
Update the transient summaries table in the marshall database |
|
The worker class for the marshall_lightcurves module |
|
The worker class for the panstarrs_location_stamps module |
Functions ∞
Get the root path for this python package |
|
|
generate all atlas FP lightcurves (clipped and stacked) |
plot single result |
|
clean up rouge data from the files by performing some basic clipping |
|
stack the photometry for the given temporal range |
A-Z Index ∞
Modules
common tools used throughout package |
|
Import codes for the various transients surveys that ‘feed’ the marshall inbox |
|
import code for the panstarrs survey |
|
import code for the panstarrs survey |
|
import code for the panstarrs survey |
|
import code for the panstarrs survey |
|
import code for the panstarrs survey |
|
import code for the panstarrs survey |
|
Housekeeping and bookkeeping actions for the marshall database |
|
Lightcurve plotting for the Marshall |
|
small services and tools for the marhall |
|
Generate the force-photometry lightcurves for ATLAS sources found in the marshall database |
|
Unit testing tools |
Classes
Import the atels transient data into the marshall database |
|
cacher for the atels image stamps |
|
Import the ATLAS transient data into the marshall database |
|
cacher for the ATLAS image stamps |
|
This baseclass for the feeder survey data imports |
|
The base class for the feeder image cachers |
|
Import the PanSTARRS transient data into the marshall database |
|
cacher for the panstarrs image stamps |
|
Import the tns transient data into the marshall database |
|
cacher for the tns image stamps |
|
Import the useradded transient data into the marshall database |
|
cacher for the useradded image stamps |
|
Import the ZTF transient data into the marshall database |
|
cacher for the ZTF image stamps |
|
Update the transient summaries table in the marshall database |
|
The worker class for the marshall_lightcurves module |
|
The worker class for the panstarrs_location_stamps module |
Functions
Get the root path for this python package |
|
|
generate all atlas FP lightcurves (clipped and stacked) |
plot single result |
|
clean up rouge data from the files by performing some basic clipping |
|
stack the photometry for the given temporal range |
Marshall Engine Release Notes ∞
adding tns-marker into the TNS search requests fixed atlas and panstarrs image stamp to download from new URLs
v1.0.9 - May 7, 2021
REFACTOR: resurrection code optimised to resurface a few more objects
REFACTOR: some MySQL queries rewritten and table indexes added to optimise common queries
REFACTOR: ATLAS forced photometry plotting code refactored to bring plots in-line with those produced by ATLAS forced photometry server. Biggest improvement is the more robust clipping of outlying data points.
v1.0.8 - March 17, 2021
REFACTOR: Occasional check to make sure all akas are set (i.e. more than just for transients discovered in the last 3 weeks).
FIXED: Links to the ASASSN Sky Patrol now added to all TNS reported ASASSN transient names (credentials discoverable in hover-over tool-tip)
FIXED: ATel comments where getting added correctly to associated object but ticket “ATel” drop-up menu was missing some ATel links occasionally.
v1.0.7 - February 20, 2021
ENHANCEMENT: Added cleanup function at end of ingests so objects appear in inbox quicker and akas are updated more frequently
REFACTORING: Reduced the crossmatch radius from 7 to 4 arcsec (it is easier to merge than split transients later on)
v1.0.6 - January 29, 2021
FEATURE: match transients to astronotes (not yet visualised in the webapp)
ENHANCEMENT: HTM indexing added the transientBucketSummaries table so we can spatially crossmatch
v1.0.5 - January 13, 2021
REFACTORING: atel parsing and matching within the marshall database now upgraded
v1.0.4 - January 11, 2021
REFACTORING: some database schema changes for latest version of Sherlock to run.
FIXED: added a function to recalculate sherlock original radii for merged sources. Fixes webapp visualisation.
FIXED: reduced number of ATLAS LC to be generated in a single batch.
v1.0.3 - December 10, 2020
REFACTORING: reading settings from marshall config folder instead of marshallEngine
FIXED: the save location of lightcurve files was resulting in files not being found in webapp
v1.0.2 - November 14, 2020
feature PS2 survey added as a new import (discoveries, lightcurves and stamps)
v1.0.1 - October 20, 2020
REFACTORING: added time filtering on ATLAS summary CSV file (thanks Ken)
FIXED: small fix to panstamp location map downloader
v1.0.0 - May 28, 2020
Now compatible with Python 3.* *