FeedManager

class sitchlib.FeedManager(config)[source]

Manage downloading the feed DB, and merging it into the sqlite DB.

apply_mode(mode)[source]

Adjust FeedManager behavior according to operating mode.

classmethod create_and_populate_db(db_schema, db_translate_schema, feed_files, db_file, target_radios)[source]

Create DB, then merge all records from file.

Parameters:
  • db_schema (list) – One K:V set from the top level of feed_db_schema.yaml
  • feed_files (list) – List of feed files to be merged.
  • db_file (str) – Full path of CGI DB file.
Returns:

Most recent timestamp from merge.

Return type:

str

classmethod create_db(db_file, db_schema)[source]

Create a DB.

This creates either the CGI or ARFCN database.

Parameters:
  • db_file (str) – Path to DB file.
  • db_schema (dict) – One top-level k:v from feed_db_schema.yaml
classmethod create_db_init_string(db_schema)[source]

Create DB initialization string based on db_schema input.

Expects a dictionary like this:
{“table_name”:
{“fields”: [“field_1”,
“field_2” “field_3”],
“unique”: [“field_1”,
“field_2”]}}
Parameters:db_schema (dict) – Dictionary describing the DB schema
classmethod dump_csv_to_db(db_schema, db_translate_schema, feed_file, db_file, target_radios, last_upd=0)[source]

Merge CSV into DB, taking into account the record update time.

Parameters:
  • db_schema (dict) – Dictionary produced from feed_db_schema.yaml. Only one key, cgi or arfcn.
  • feed_file (str) – Path to feed CSV file.
  • db_file (str) – Path to sqlite DB file.
  • last_upd (int, optional) – Epoch time. Records updated before this date will not be inserted into the DB.
get_newest_record_time(db_type)[source]

Get the newest record time from file in feed dir.

classmethod get_source_url(url_base, mcc)[source]

Create source URL for MCC file.

Parameters:
  • url_base (str) – Base URL for MCC file.
  • mcc (str) – MCC for feed file.
classmethod mass_insert(table, fields, rows, db_file)[source]

Mass-insert records into the DB.

Parameters:
  • schema (list) – List of DB fields.
  • rows (list) – List of tuples, each tuple contains values corresponding to the keys in schema.
  • db_file (str) – Path to sqlite file.
classmethod merge_feed_files_into_db(db_schema, db_translate_schema, feed_files, db_file, target_radios, last_upd)[source]

Wrapper for merging feed file data into CGI DB.

Parameters:
  • schema (list) – List of fields in DB
  • feed_file (str) – Path to feed file to be merged into CGI DB.
  • db_file (str) – Path to CGI DB file.
  • last_upd (str) – Epoch time stamp, will not attempt to merge any records with timestamps before this time.
Returns:

Most recent timestamp from merged feed file.

Return type:

str

classmethod place_feed_file(feed_dir, url_base, item_id)[source]

Retrieve and places feed files for use by the Enricher modules.

Parameters:
  • feed_dir (str) – Destination directory for feed files
  • url_base (str) – Base URL for hosted feed files
  • item_id (str) –

    For FCC, this is the two-letter (“CA” or “TN”, for example), which is used in the retrieval of the feed file

    as well as the construction of the local feed file name. For MCC this is the MCC, but in string form. Not integer.
classmethod reconcile_db(db_schema, db_translate_schema, feed_files, db_file, target_radios, last_update)[source]

Reconcile feed files against the target DB.

Parameters:
  • feed_files (list) – List of paths to feed files.
  • db_file (str) – Full path to CGI DB file.
  • last_update (str) – Epoch time of most recent record in DB
Returns:

Epoch timestamp of most recently updated DB record.

Return type:

str

set_newest_record_time(db_type, timestamp)[source]

Set the newest record time.

Parameters:timestamp (str) – Epoch time to be written to file If not string, will be coerced to string.
classmethod should_update_record_epoch(anchor_time, update_time)[source]

Compare timestamps to determine if a record should be updated.

classmethod should_update_record_iso(anchor_time, update_time)[source]

Compare timestamps to determine if a record should be updated.

classmethod tup_from_row(schema, row)[source]

Convert a row into a tuple, for insertion into DB.

Parameters:
  • schema (list) – Field list for DB.
  • row (dict) – Row of data. Keys align with items in schema.
Returns:

Tuple representing values to be inserted into DB, ordered

by fields in schema.

Return type:

tuple

update_feed_db()[source]

Wrapper for feed file reconciliation against DBs.

update_feed_files()[source]

Wrapper for feed file retrieval routines.