em_map_utils.farm_function¶
farm_function.py
- Author:
Ardan Patwardhan
- Affiliation:
EMBL-EBI, Wellcome Genome Campus, CB10 1SD, UK
- Date:
25/08/2025
- Description:
Run multiple processes of the same function, changing one parameter and collecting any results in a JSON file.
Attributes¶
Classes¶
Run multiple processes of a function f(v, args, kwargs), mapping |
Module Contents¶
- em_map_utils.farm_function.logger¶
- class em_map_utils.farm_function.FarmFunction(f, values, args=[], kwargs={}, num_workers=2, max_tries=2, monitoring_interval=5, timeout=15 * 60, file_root='dummy', resume=False, retry=False)¶
Run multiple processes of a function f(v, args, kwargs), mapping v over a list of values and collecting the results in a JSON file.
In contrast with the Pool function of multiprocessing, this class can attempt multiple tries of the function before giving up.
- max_cpu¶
- f¶
- values¶
- args = []¶
- kwargs¶
- num_workers¶
- max_tries = 2¶
- monitoring_interval = 5¶
- timeout = 900¶
- file_root = 'dummy'¶
- db_name = 'dummy.db'¶
- output_file = 'dummy.json'¶
- resume = False¶
- retry = False¶
- process_queue¶
- success_queue¶
- fail_queue¶
- num_processed¶
- num_error¶
- num_success¶
- old_values = []¶
- lock¶
- static dict_factory(cursor, row)¶
Format rows as a dictionary for SQLite output.
If the column result is present, it is assumed to be a json dump and loaded back into a dictionary.
- Parameters:
cursor – Database cursor.
row – Database row tuple.
- Returns:
Formatted row dictionary.
- write_to_files(con)¶
Write success, failed, and processing tables to a file.
- Parameters:
con – Database connection.
- Returns:
No return value.