qbiocode.utils.combine_evals_results module#

Utilities for tracking progress and combining results from interrupted jobs.

This module provides functions to help manage and combine results when computational jobs are interrupted and need to be restarted. These are generic utilities that can be used with any pipeline that produces CSV output files in subdirectories.

Summary#

Functions:

combine_results

Combine results from interrupted and resumed computational jobs.

track_progress

Track progress of a computational job by checking for completed datasets.

Reference#

track_progress(input_dataset_dir, current_results_dir, completion_marker='RawDataEvaluation.csv', prefix_length=8, input_extension='csv', verbose=True)[source]#

Track progress of a computational job by checking for completed datasets.

This function scans the results directory for completed datasets (identified by the presence of a specific marker file) and compares against the total number of input datasets to determine how many remain to be processed.

Parameters:

  • input_dataset_dir (str) – Path to the directory containing input datasets.

  • current_results_dir (str) – Path to the directory containing outputs of the current job.

  • completion_marker (str, optional) – Name of the file that indicates a dataset has been fully processed. Default is ‘RawDataEvaluation.csv’.

  • prefix_length (int, optional) – Number of characters to skip from the beginning of directory names when extracting dataset identifiers. Default is 8 (e.g., skips dataset_ prefix).

  • input_extension (str, optional) – File extension of input datasets (without dot). Default is ‘csv’.

  • verbose (bool, optional) – If True, prints progress information. Default is True.

Return type:

Tuple[List[str], int, int]

Returns:

  • completed_datasets (List[str]) – List of dataset identifiers that have been completed.

  • num_completed (int) – Number of completed datasets.

  • num_remaining (int) – Number of datasets remaining to be processed.

Examples

>>> from qbiocode.utils import track_progress
>>> completed, done, remaining = track_progress(
...     input_dataset_dir='data/inputs',
...     current_results_dir='results/run1'
... )
The completed datasets are: ['dataset1', 'dataset2']
You have finished running program on 2 out of a total of 10 input datasets.
You have 8 input datasets left before program finishes.

>>> # Custom completion marker
>>> completed, done, remaining = track_progress(
...     input_dataset_dir='data/inputs',
...     current_results_dir='results/run1',
...     completion_marker='final_output.csv',
...     prefix_length=0  # No prefix to skip
... )
combine_results(prev_results_dir, recent_results_dir, eval_file_prefix='Raw', results_file_prefix='Model', output_eval_file='RawDataEvaluation_Combined.csv', output_results_file='ModelResults_Combined.csv', save_intermediate=True, verbose=True)[source]#

Combine results from interrupted and resumed computational jobs.

This function merges CSV files from a previous (interrupted) job run with files from a recent (resumed) job run. It’s useful when a long-running computational job needs to be restarted and you want to combine all results.

Parameters:

  • prev_results_dir (str) – Path to the directory where the previous job stopped prematurely. Should contain subdirectories with individual result files.

  • recent_results_dir (str) – Path to the directory where the job was resumed and ran to completion. Should contain combined result files.

  • eval_file_prefix (str, optional) – Prefix of evaluation/assessment files to combine. Default is ‘Raw’.

  • results_file_prefix (str, optional) – Prefix of model results files to combine. Default is ‘Model’.

  • output_eval_file (str, optional) – Name of the combined evaluation output file. Default is ‘RawDataEvaluation_Combined.csv’.

  • output_results_file (str, optional) – Name of the combined results output file. Default is ‘ModelResults_Combined.csv’.

  • save_intermediate (bool, optional) – If True, saves intermediate combined files from previous run. Default is True.

  • verbose (bool, optional) – If True, prints shape information during processing. Default is True.

Return type:

Tuple[DataFrame, DataFrame]

Returns:

  • combined_eval_df (pd.DataFrame) – Combined dataframe of all evaluation/assessment data.

  • combined_results_df (pd.DataFrame) – Combined dataframe of all model results.

Examples

>>> from qbiocode.utils import combine_results
>>> eval_df, results_df = combine_results(
...     prev_results_dir='results/run1_interrupted',
...     recent_results_dir='results/run2_resumed'
... )
>>> print(f"Combined {len(eval_df)} evaluation records")
>>> print(f"Combined {len(results_df)} result records")

>>> # Custom file prefixes and output names
>>> eval_df, results_df = combine_results(
...     prev_results_dir='results/old',
...     recent_results_dir='results/new',
...     eval_file_prefix='Evaluation',
...     results_file_prefix='Results',
...     output_eval_file='AllEvaluations.csv',
...     output_results_file='AllResults.csv'
... )

Notes

The function expects: - prev_results_dir to contain subdirectories, each with individual CSV files - recent_results_dir to contain combined CSV files at the top level - Files are identified by their prefix (eval_file_prefix, results_file_prefix)