Running model on large datasets¶

When running the model on large datasets there are multiple aspects to consider in terms of speeding up the process:

• The concept filter to use. Use cases often only care about a small subset of concepts a model is trained on. By pre-applying a narrow filter you can make the model's inference run a lot faster because it then does not need to spend time on trying to disambiguate spans that you do not care about.
• The available RAM on the device. If there's ample RAM available on the device, multiprocessing might be the right thing to focus on. However, if RAM is limited, processing single threaded might be preferred. That's because multi-processing works on different processes, each of which will effectively need to load the model to perform the inference.
• The amount of data in question. If the amount of data in question is larger than what would fit in memory, you may want to use multi-processing. The multiprocessing method is designed for large amounts of data: it processes the incoming data in batches and returns an iterable that needs to be yielded over in order for the method to do its work. This has explicitly been designed in a way that avoids loading all the data into memory at once. It does batching even if only one process is used. The method also provides options to save the output on disk if that's preferred at this stage. PS: You can also do batching of data yourself and use regular inference.

In [1]:

import os
import time
from medcat.cat import CAT
from medcat.utils.config_utils import temp_changed_config

import os import time from medcat.cat import CAT from medcat.utils.config_utils import temp_changed_config

Data preparations¶

NOTE: In this tutorial we do not have a very large amount of data, so we will artificially inflate it by repeating the same data over and over again. You would normally not do that (somewhat obviously).

In [2]:

DATA_PATH = os.path.join("in_data", "dummy_bulk_data.lines")
INFLATION_COEF = 100
with open(DATA_PATH) as f:
    raw_data = list(map(str.strip, f.readlines()))

# this will create a generator for the data
# that repeats INFLATION_COEF (100) times
# NOTE: it is best to have each piece of data
#       along with its identifier so you can stitch things
#       together later, but it's not strictly necessary
def data_iter():
    for dupe_num in range(INFLATION_COEF):
        for line_num, line in enumerate(raw_data):
            yield (f"{line_num:05d}_d{dupe_num:04d}", line)

DATA_PATH = os.path.join("in_data", "dummy_bulk_data.lines") INFLATION_COEF = 100 with open(DATA_PATH) as f: raw_data = list(map(str.strip, f.readlines())) # this will create a generator for the data # that repeats INFLATION_COEF (100) times # NOTE: it is best to have each piece of data # along with its identifier so you can stitch things # together later, but it's not strictly necessary def data_iter(): for dupe_num in range(INFLATION_COEF): for line_num, line in enumerate(raw_data): yield (f"{line_num:05d}_d{dupe_num:04d}", line)

Loading model pack¶

We will download the MedMentions model here. The models we have created earlier in the tutorials are a little too simplistic for the purposes relevant here.

In [3]:

# Download the MedMentions modelpack
! wget -N https://cogstack-medcat-example-models.s3.eu-west-2.amazonaws.com/medcat-example-models/medmen_wstatus_2021_oct.zip -P models/
model_path = os.path.join("models", "medmen_wstatus_2021_oct.zip")

cat = CAT.load_model_pack(model_path)

# Download the MedMentions modelpack ! wget -N https://cogstack-medcat-example-models.s3.eu-west-2.amazonaws.com/medcat-example-models/medmen_wstatus_2021_oct.zip -P models/ model_path = os.path.join("models", "medmen_wstatus_2021_oct.zip") cat = CAT.load_model_pack(model_path)

--2026-07-28 11:55:30--  https://cogstack-medcat-example-models.s3.eu-west-2.amazonaws.com/medcat-example-models/medmen_wstatus_2021_oct.zip
Resolving cogstack-medcat-example-models.s3.eu-west-2.amazonaws.com (cogstack-medcat-example-models.s3.eu-west-2.amazonaws.com)... 52.95.142.18, 3.5.246.104, 3.5.244.8, ...
Connecting to cogstack-medcat-example-models.s3.eu-west-2.amazonaws.com (cogstack-medcat-example-models.s3.eu-west-2.amazonaws.com)|52.95.142.18|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 561947681 (536M) [application/zip]
Saving to: ‘models/medmen_wstatus_2021_oct.zip’

medmen_wstatus_2021 100%[===================>] 535.92M  71.3MB/s    in 7.8s    

2026-07-28 11:55:39 (68.7 MB/s) - ‘models/medmen_wstatus_2021_oct.zip’ saved [561947681/561947681]

Doing legacy conversion on CAT (at 'models/medmen_wstatus_2021_oct'). Set the environment variable MEDCAT_AVOID_LECACY_CONVERSION to `True` to avoid this.
Missing class medcat.config.weighted_average, replacing with LegacyClassNotFound.

Conditions False False IN {'T019', 'T067', 'T028', 'T062', 'T101', 'T092', 'T109', 'T014', 'T097', 'T120', 'T203', 'T102', 'unk', 'T100', 'T201', 'T096', 'T041', 'T204', 'T060', 'T048', 'T077', 'T031', 'T033', 'T070', 'T045', 'T122', 'T024', 'T080', 'T069', 'T034', 'T013', 'T191', 'T039', 'T168', 'T049', 'T059', 'T192', 'T056', 'T054', 'T050', 'T197', 'T044', 'T052', 'T025', 'T065', 'T114', 'T087', 'T194', 'T002', 'T011', 'T125', 'T091', 'T030', 'T185', 'T017', 'T190', 'T082', 'T094', 'T042', 'T116', 'T090', 'T051', 'T123', 'T022', 'T032', 'T086', 'T047', 'T008', 'T010', 'T043', 'T055', 'T021', 'T064', 'T129', 'T075', 'T171', 'T046', 'T071', 'T001', 'T057', 'T200', 'T085', 'T103', 'T169', 'T130', 'T007', 'T020', 'T068', 'T058', 'T074', 'T099', 'T015', 'T121', 'T005', 'T063', 'T016', 'T038', 'T079', 'T012', 'T061', 'T196', 'T098', 'T126', 'T127', 'T073', 'T072', 'T053', 'T078', 'T095', 'T081', 'T093', 'T184', 'T029', 'T083', 'T195', 'T026', 'T131', 'T167', 'T004', 'T040', 'T023', 'T104', 'T037', 'T170', 'T018', 'T089', 'T066'}
Got TypeID2name for 127 TypeIDs out of 127

Trying to set 'cdb_source_name' for 'General' but no such attribute
Trying to set 'weighted_average_function' for 'Linking' but no such attribute
Optional path 'version.description' not found in old config. Ignoring
Optional path 'version.id' not found in old config. Ignoring
Optional path 'version.ontology' not found in old config. Ignoring
/Users/martratas/Documents/CogStack/.MedCAT.nosync/monorepo-nlp/medcat-v2-tutorials/.venv312/lib/python3.12/site-packages/spacy/util.py:969: UserWarning: [W095] Model 'en_core_web_md' (3.1.0) was trained with spaCy v3.1.0 and may not be 100% compatible with the current version (3.8.11). If you see errors or degraded performance, download a newer compatible model or retrain your custom model with the current spaCy version. For more details and available updates, run: python -m spacy validate
  warnings.warn(warn_msg)
/Users/martratas/Documents/CogStack/.MedCAT.nosync/monorepo-nlp/medcat-v2-tutorials/.venv312/lib/python3.12/site-packages/tqdm/auto.py:21: TqdmWarning: IProgress not found. Please update jupyter and ipywidgets. See https://ipywidgets.readthedocs.io/en/stable/user_install.html
  from .autonotebook import tqdm as notebook_tqdm

Demonstrating power of filtering in sequential inference¶

Runtime speed will differ from model to model and your model will (almost certainly) be bigger than the one we're using here.

However, just to demonstrate the power of filters, we will run the same data with and without them.

In [10]:

def time_entire_run():
    start_time = time.perf_counter()
    ent_counter = 0
    for cntr, (tid, text) in enumerate(data_iter()):
        # NOTE: we're currently not doing anything with the output
        #       but in a real use case you would want to
        ents = cat.get_entities(text)["entities"]
        ent_counter += len(ents)
    end_time = time.perf_counter()
    print("Took", end_time - start_time, "seconds for", cntr + 1, "texts")
    print("We found", ent_counter, "entities")

def time_entire_run(): start_time = time.perf_counter() ent_counter = 0 for cntr, (tid, text) in enumerate(data_iter()): # NOTE: we're currently not doing anything with the output # but in a real use case you would want to ents = cat.get_entities(text)["entities"] ent_counter += len(ents) end_time = time.perf_counter() print("Took", end_time - start_time, "seconds for", cntr + 1, "texts") print("We found", ent_counter, "entities")

In [11]:

# without any filtering
time_entire_run()

# without any filtering time_entire_run()

Took 32.59635237487964 seconds for 6700 texts
We found 17200 entities

In [12]:

# NOTE: You would likely need to come up with your own filter.
#       This model is UMLS based, but in Snomed the concept
#       IDs are fully numeric, but need to be sent to medcat
#       as strings
CUI_FILTER = {
    "C0039231", # tachycardia
    "C0428977", # bradycardia
    "C0018799", # heart disease
}
# NOTE: we're running a temporarily changed filter here so that
#       we don't have to change the model state, but you would
#       probably simply make the change permanent in the model:
# cat.config.components.linking.filters.cuis = CUI_FILTER
with temp_changed_config(cat.config.components.linking.filters, 'cuis', CUI_FILTER):
    time_entire_run()

# NOTE: You would likely need to come up with your own filter. # This model is UMLS based, but in Snomed the concept # IDs are fully numeric, but need to be sent to medcat # as strings CUI_FILTER = { "C0039231", # tachycardia "C0428977", # bradycardia "C0018799", # heart disease } # NOTE: we're running a temporarily changed filter here so that # we don't have to change the model state, but you would # probably simply make the change permanent in the model: # cat.config.components.linking.filters.cuis = CUI_FILTER with temp_changed_config(cat.config.components.linking.filters, 'cuis', CUI_FILTER): time_entire_run()

Took 31.438669624971226 seconds for 6700 texts
We found 200 entities

NOTE: In the real world, with bigger models, the time difference is likely to be a fair bit bigger. That's because the small model we're using here has less training and fewer ambiguous concepts, so it doesn't need to do as much work on the disamibguation.

Showing multiprocessing¶

We can also run the same data through the multiprocessing pipeline. The API reference of this method is available (here)[https://docs.cogstack.org/projects/nlp/en/latest/reference/medcat/cat/#medcat.cat.CAT.get_entities_multi_texts]. However, we'll give a quick overview of some of the key parameters of this (get_entities_multi_texts) method:

• texts - the input texts. This can be a list or a generator. Each item can be either a single string (the text) or a tuple of two strings (the text index along with the text). The latter helps with mapping the annotations to the documents if/when required.
• n_process - the number of processes to use. Each parallel process will consume more memory / RAM. But running the text in parallel should help speed up the process when running through a lot of data. The default is 1 (single-threaded, but still batched).
• batch_size - the size of a batch. That is, the number of texts / documents to batch at a time. For large datasets it's important not to load all the data into memory at once. And you can control that by changing this value. This defaults to -1 and in such a case character-based batching is used instead.
• batch_size_chars - the size of a batch in number of characters. In different use cases, the documents are different lengths. But the memory footprint is normally dictated by the number of characters. So this option allows one to specify the number of characters rather than the number of documents for batching. This is usually the more sensible soluton. This defaults to 1 million. Set to -1 to disable.
• save_dir_path - the folder to write the output to. If you wish the output to be written on disk instead of staying in memory, set this to the path you wish to use. The outputs (usually multiple) will be saved in part_<num>.pickle format in this folder.

NB! This method returns an iterator. That means that no work will be done unless the output is iterated over. This is a deliberate design approach to avoid loading all the input data (along with the annotated output) in memory at once.

In [17]:

def time_run_w_mp():
    start_time = time.perf_counter()
    ent_counter = 0
    for text_cntr, (text_id, ents) in enumerate(cat.get_entities_multi_texts(data_iter(), n_process=2)):
        ent_counter += len(ents["entities"])
    ent_counter += len(ents)
    end_time = time.perf_counter()
    print("Took", end_time - start_time, "seconds for", text_cntr + 1, "texts")
    print("We found", ent_counter, "entities")

def time_run_w_mp(): start_time = time.perf_counter() ent_counter = 0 for text_cntr, (text_id, ents) in enumerate(cat.get_entities_multi_texts(data_iter(), n_process=2)): ent_counter += len(ents["entities"]) ent_counter += len(ents) end_time = time.perf_counter() print("Took", end_time - start_time, "seconds for", text_cntr + 1, "texts") print("We found", ent_counter, "entities")

In [18]:

# regular
time_run_w_mp()
# with filter
with temp_changed_config(cat.config.components.linking.filters, 'cuis', CUI_FILTER):
    time_entire_run()

# regular time_run_w_mp() # with filter with temp_changed_config(cat.config.components.linking.filters, 'cuis', CUI_FILTER): time_entire_run()

Took 51.71125883399509 seconds for 6700 texts
We found 17202 entities
Took 32.595435000024736 seconds for 6700 texts
We found 200 entities

NOTE: In order to actually reap the benefit (in terms of time) from multi-processing, you will need to run through a lot more data than we did in this example. In this example, a lot of the time was probably spent setting up the orchestration for these separate processes. When running an example workflow with 1000 MIMIC-IV documents, the multiprocessing (with n_process=2) yielded about a 10% overall speedup.