qbiocode.evaluation.model_evaluation module#

Summary#

Functions:

evaluation_metrics

Calculate evaluation metrics for classification predictions.

modeleval

Evaluates the model performance using accuracy, F1 score, and AUC.

Reference#

modeleval(y_test, y_predicted, beg_time, params, args, model, verbose=True, average='weighted')[source]#

Evaluates the model performance using accuracy, F1 score, and AUC.

Parameters:

  • y_test (array-like) – True labels for the test set.

  • y_predicted (array-like) – Predicted labels by the model.

  • beg_time (float) – Start time for measuring execution time.

  • params (dict) – Model parameters used during training.

  • args (dict) – Additional arguments, including grid search flag.

  • model (str) – Name of the model being evaluated.

  • verbose (bool) – If True, prints the evaluation results.

  • average (str) – Type of averaging to use for F1 score calculation. Default is ‘weighted’.

Returns:

DataFrame containing the evaluation results, including accuracy, F1 score, AUC, and model parameters.

Return type:

pd.DataFrame

evaluation_metrics(predictions, y_test, metrics=['accuracy', 'brier'], save=False)[source]#

Calculate evaluation metrics for classification predictions.

Computes specified metrics for model predictions. Supports accuracy, Brier score, F1 score, precision, recall, and AUC-ROC. The Brier score measures the mean squared difference between predicted probabilities and actual outcomes, providing a measure of calibration quality.

Parameters:

  • predictions (np.ndarray) – Predicted probabilities, shape (n_samples, n_classes)

  • y_test (np.ndarray) – True labels, shape (n_samples,)

  • metrics (list of str, optional) – List of metrics to compute. Options: ‘accuracy’, ‘brier’, ‘f1’, ‘precision’, ‘recall’, ‘auc’ (default: [‘accuracy’, ‘brier’])

  • save (bool, optional) – Whether to save results (reserved for future use, default: False)

Returns:

If metrics=[‘accuracy’, ‘brier’] (default): returns (accuracy, brier_score) Otherwise: returns dict with requested metrics as keys

Return type:

tuple or dict

Examples

>>> import numpy as np
>>> from qbiocode.evaluation import evaluation_metrics
>>>
>>> # Binary classification example - default metrics
>>> predictions = np.array([[0.8, 0.2], [0.3, 0.7], [0.9, 0.1]])
>>> y_test = np.array([0, 1, 0])
>>> accuracy, brier = evaluation_metrics(predictions, y_test)
>>> print(f"Accuracy: {accuracy:.2f}, Brier Score: {brier:.3f}")
Accuracy: 1.00, Brier Score: 0.060

>>> # Multiple metrics
>>> results = evaluation_metrics(predictions, y_test,
...                              metrics=['accuracy', 'brier', 'f1', 'auc'])
>>> print(results)
{'accuracy': 1.0, 'brier': 0.06, 'f1': 1.0, 'auc': 1.0}

Notes

  • For binary classification, Brier score is computed using the probability of the positive class

  • For multi-class classification, the average Brier score across all classes is returned

  • F1, precision, and recall use weighted averaging for multi-class

  • AUC uses one-vs-rest for multi-class

  • Lower Brier scores indicate better calibrated probability predictions

References

Brier, G. W. (1950). “Verification of forecasts expressed in terms of probability”. Monthly Weather Review, 78(1), 1-3.