marshallEngine.feeders.data module ∞

Baseclass for survey data ingesters

Author

David Young

class data[source] ∞

Bases: object

This baseclass for the feeder survey data imports

Usage

Todo

  • create a frankenstein template for importer

To create a new survey data ingester create a new class using this class as the baseclass:

from ..data import data as basedata
class data(basedata):
    ....
get_csv_data(url, user=False, pwd=False)[source] ∞

collect the CSV data from a URL with option to supply basic auth credentials

Key Arguments

  • url – the url to the csv file

  • user – basic auth username

  • pwd – basic auth password

Return

  • csvData – a list of dictionaries from the csv file

Usage

To get the CSV data for a suvery from a given URL in the marshall settings file run something similar to:

from marshallEngine.feeders.panstarrs.data import data
ingester = data(
    log=log,
    settings=settings,
    dbConn=dbConn
)
csvDicts = ingester.get_csv_data(
    url=settings["panstarrs urls"]["3pi"]["summary csv"],
    user=settings["credentials"]["ps1-3pi"]["username"],
    pwd=settings["credentials"]["ps1-3pi"]["password"]
)

Note you will also be able to access the data via ingester.csvDicts

_import_to_feeder_survey_table()[source] ∞

import the list of dictionaries (self.dictList) into the marshall feeder survey table

Return

  • None

Usage

self._import_to_feeder_survey_table()
insert_into_transientBucket(importUnmatched=True, updateTransientSummaries=True)[source] ∞

insert objects/detections from the feeder survey table into the transientbucket

Key Arguments

  • importUnmatched – import unmatched (new) transients into the marshall (not wanted in some circumstances)

  • updateTransientSummaries – update the transient summaries and lightcurves? Can be True or False, or alternatively a specific transientBucketId

This method aims to reduce crossmatching and load on the database by:

  1. automatically assign the transientbucket id to feeder survey detections where the object name is found in the transientbukcet (no spatial crossmatch required). Copy matched feeder survey rows to the transientbucket.

  2. crossmatch remaining unique, unmatched sources in feeder survey with sources in the transientbucket. Add associated transientBucketIds to matched feeder survey sources. Copy matched feeder survey rows to the transientbucket.

  3. assign a new transientbucketid to any feeder survey source not matched in steps 1 & 2. Copy these unmatched feeder survey rows to the transientbucket as new transient detections.

Return

  • None

Usage

ingester.insert_into_transientBucket()
_feeder_survey_transientbucket_name_match_and_import()[source] ∞

automatically assign the transientbucket id to feeder survey detections where the object name is found in the transientbukcet (no spatial crossmatch required). Copy feeder survey rows to the transientbucket.

Return

  • None

Usage

self._feeder_survey_transientbucket_name_match_and_import()
_feeder_survey_transientbucket_crossmatch()[source] ∞

crossmatch remaining unique, unmatched sources in feeder survey with sources in the transientbucket & copy matched feeder survey rows to the transientbucket

Return

  • unmatched – a list of the unmatched (i.e. new to the marshall) feeder survey surveys

_import_unmatched_feeder_survey_sources_to_transientbucket(unmatched)[source] ∞

assign a new transientbucketid to any feeder survey source not yet matched in steps. Copy these unmatched feeder survey rows to the transientbucket as new transient detections.

Key Arguments

  • unmatched – the remaining unmatched feeder survey object names.

clean_up()[source] ∞

A few tasks to finish off the ingest

Key Arguments:

# -

Return:

- None

Usage:

usage code

Todo

  • add usage info

  • create a sublime snippet for usage

  • write a command-line tool for this method

  • update package tutorial with command-line tool info if needed