Evaluating perfromance on a dataset¶
Generally a user of MedCAT needs to know what kind of performance the model can achieve on their dataset. Usually, the user will need to annotate a dataset on MedCATtrainer (though another dataset converted into the correct format will also work).
This part of the tutorial will introduce ways to evaluate the performance of a model on datasets and show in some more detail on how to inspect the output.
import os
import json
from medcat.cat import CAT
from medcat.stats import get_stats
Loading the data and model¶
# NOTE: using the same data as we did for supervised training
# but this time for a model that hasn't been trained on it yet
mct_export_path = os.path.join("in_data", "MCT_export_abscess.json")
with open(mct_export_path) as f:
mct_export = json.load(f)
model_path = "models/unsup_trained_model"
cat = CAT.load_model_pack(model_path)
Loading a model with incorrectly saved tokenizer internals path. Was saved as 'models/base_model/en_core_web_md' whereas should have had just 'en_core_web_md'. This is an automated fix - no further action is needed
Running metrics¶
The method has the following signature:¶
get_stats(cat: CAT, data: MedCATTrainerExport, epoch: int = 0, use_project_filters: bool = False, use_overlaps: bool = False, extra_cui_filter: Optional[set[str]] = None, do_print: bool = True) -> tuple[dict[str, int], dict[str, int], dict[str, int], dict[str, float], dict[str, float], dict[str, float], dict[str, int], dict]
And the following parameters:¶
cat (CAT) – (CAT): The model pack.
data (dict) – The json object that we get from MedCATtrainer on export.
epoch (int, default: 0 ) – Used during training, so we know what epoch is it.
use_project_filters (bool, default: False ) – Each project in MedCATtrainer can have filters, do we want to respect those filters when calculating metrics.
use_overlaps (bool, default: False ) – Allow overlapping entities, nearly always False as it is very difficult to annotate overlapping entities.
use_cui_doc_limit (bool) – If True the metrics for a CUI will be only calculated if that CUI appears in a document, in other words if the document was annotated for that CUI. Useful in very specific situations when during the annotation process the set of CUIs changed.
do_print (bool, default: True ) – Whether to print stats out. Defaults to True.
And returns a tuple with the follwoing information:¶
fps ( dict ) – False positives for each CUI.
fns ( dict ) – False negatives for each CUI.
tps ( dict ) – True positives for each CUI.
cui_prec ( dict ) – Precision for each CUI.
cui_rec ( dict ) – Recall for each CUI.
cui_f1 ( dict ) – F1 for each CUI.
cui_counts ( dict ) – Number of occurrence for each CUI.
examples ( dict ) – Examples for each of the fp, fn, tp. Format will be examples['fp']['cui'][].
fps, fns, tps, cui_prec, cui_rec, cui_f1, cui_counts, examples = get_stats(cat, mct_export, do_print=True)
Epoch: 0, Prec: 0.0, Rec: 0.0, F1: 0.0 Docs with false positives: Docs with false negatives: D0; D1 False Positives False Negatives 44132006 - 44132006 - 2 128477000 - 128477000 - 2 True Positives **************************************************************************************************************
As we can see, without training, the performance is 0. Now we will train and evaluate on the same dataset (NB! This is not standard practice - it's just to show the change in performance).
# train
cat.trainer.train_supervised_raw(mct_export)
# stats again
fps, fns, tps, cui_prec, cui_rec, cui_f1, cui_counts, examples = get_stats(cat, mct_export, do_print=True)
Epoch: 0%| | 0/1 [00:00<?, ?it/s]
Epoch: 0, Prec: 1.0, Rec: 1.0, F1: 1.0 Docs with false positives: Docs with false negatives: False Positives False Negatives True Positives abscess - 44132006 - 2 abscess - 128477000 - 2 **************************************************************************************************************
Now we can see the performance is 1. But if it wasn't, we could go further to look for examples. Right now, we'll just look at eamples for true positives, but normally you'd look at false negatives or false positives.
examples["tp"]
{'44132006': [{'text': '\r\nAn abscess, as a morphologic abnormality, refers to a localized collec',
'cui': '44132006',
'start': 5,
'end': 5,
'source value': 'abscess',
'acc': 0.8999993893310174,
'project name': 'MR-4-mct_v2-tut',
'document name': 'D0',
'project id': 281,
'document id': 332929},
{'text': "dy's response to contain the infection. The presence of an\r\nabscess can alter tissue architecture and may be identified histolo",
'cui': '44132006',
'start': 394,
'end': 394,
'source value': 'abscess',
'acc': 0.9864342483370128,
'project name': 'MR-4-mct_v2-tut',
'document name': 'D0',
'project id': 281,
'document id': 332929}],
'128477000': [{'text': '\r\nAn abscess, as a disorder, is a clinical condition characterized by th',
'cui': '128477000',
'start': 5,
'end': 5,
'source value': 'abscess',
'acc': 0.46294583776646375,
'project name': 'MR-4-mct_v2-tut',
'document name': 'D1',
'project id': 281,
'document id': 332930},
{'text': 'systemic symptoms such as fever may occur in severe cases.\r\nAbscesses can develop in various body regions, including the skin, in',
'cui': '128477000',
'start': 303,
'end': 303,
'source value': 'Abscesses',
'acc': 1.0,
'project name': 'MR-4-mct_v2-tut',
'document name': 'D1',
'project id': 281,
'document id': 332930}]}
If you've now seen performance that's too low, you will likely have to fine-tune the model (though config changes can also be useful) with annotated datasets from MedCATtrainer. Though it is also important to note that if you've got MetaCAT's attached to your model (see the relevant tutorials) you can (and often should) use the output of these to postprocess the output to only keep annotations that are relevant to you (e.g recent (not past / future), patient-related (not related to others) and true (not hypothetical or negated) ones).