SECTION I: Testing
|1||Has the R & D result been tested?|
|1a||In what mode has the result been tested?
• Pilot Application
• Alpha/BETA testing
|A number of tools and testing methods have been used to validate different components of the R&D result.
Integration of data sources
ECG Data sources used for testing of the integration capabilities are:
Dynamic Terminology Enhancement Method evaluation
Step 1.Concept suitability investigation evaluation.
Dataset: The dataset constitutes a combined set of medical terms stemming from unmapped table records related to medication, medication categories and free text entries of patient medical history, all originating from the SCP-ECG ontology. The testing set contains 85 medical terms retrieved from UMLS Metathesaurus. 76 were positive (suitable classes) and 9 were negative (inapt classes) with each medical term being represented by 3 features. The training set consists of 28 medical terms 9 of which were labelled as negative and 19 as positive ones.
Step 2.Hierarchy suitability investigation evaluation
Dataset: The datasets, engaged in the evaluation of hierarchy suitability investigation, were generated by UMLS Metathesaurus upon queried for the ancestry of a series of medical terms. The terms originated from patient diagnosis and medication domains and were extracted by SCP-ECG ontology and Interlife database. The training set comprises 55 instances, 49 of which being negative (−1) and 6 being positive (1). The testing set contains 61 instances partitioned into 51 negative and 10 positive ones.
|1b.||Please describe and discuss the testing results|
|The proposed application was tested in terms of integrating the data and metadata from the three different types of resources mentioned above. The main parts had to do with the correct understanding and mapping of each type’s data and metadata.
For the source transformation of the EDF and SCP-ECG files, relevant software programs were implemented to facilitate the SCP-ECG to OWL and EDF to OWL conversion. EDF and SCP-ECG viewers were also available.
Source schema instantiation: Three mapping ontologies were instantiated to express the correspondences among then Global Ontology and SCP-ECG, EDF and Interlife database. Schema instantiation Component(SCIC) registered 244 instances subsumed under ClassNames and 34 instances under Class Attributes for GO. 585,140 and 40 were the instances of the records, attributes and tables corresponding to Interlife database. SCP- ECG ontology resulted in 213 and 131 instances for classes and attributes respectively. Finally, EDF ontology contributed 54 class and 17 attribute instances in the related mapping ontology.
The Mapping instantiation component discovered a satisfying number of mappings among SCP-ECG and GO, since most of the former concepts were adopted by the latter. EDF to GO mapping procedure generated the lowest number of mappings as initially expected, considering that EDF ontology contains the least number of classes (standardized texts and textual information) compared to the other sources.
With respect to handling terms that were found in the data resources and were not initially mapped to the terms of the Global Ontology, two more steps were tested, related to the decision if and where the new terms belong.
Concept suitability investigation evaluation decided on whether the new candidate terms should be added to the global ontology terminology. Different classifiers were tested and the SVM-RBF classifier succeeded much higher sensitivity, specificity and accuracy (0.986842, 0.777778, 0.964706 (82/85), than the linear SVM classifier (0.973684, 0,666667, 0,941176 (80/85)) applied to the same training and test datasets.
Hierarchy suitability investigation evaluation was applied for the decision of the most appropriate placement of the new term. The values of sensitivity, specificity and accuracy, concerning Alternating Decision Tree algorithm’s performance, were 0.8, 0.727 and 91.803% (56/61) respectively. The relevant scores for both C4.5 and Naïve Bayes algorithm were 1, 0.8333 and 96.721 (59/61).
Finally, ROISES Interrogator was tested with queries that have been discussed with medical professionals.
The following queries illustrate Interrogator’s functionalities:
(a) “Show ECGs with Diagnosis: Myocardial Infarction, Medical History: Endocrine_disease, Weight ≥ 60 and Sex: Male”.
(b) “Show ECGs with number of Leads ≥ 2, Including Lead Names: V4, V2 and Sampling Frequency = 250 Hz”.
(c) “Show ECGs with Low Pass Filter = 30 Hz, average RRInterval (ms) > 800”.
SECTION 2: Current Stage of Development
|2a||To what extent does the development team have technical resources for supporting the production of a new product? (Researchers, human resources, hardware, etc. )|
|The Lab of Medical Informatics of AUTH is one of the major research and development centers in Medical Informatics and Biomedical Engineering. It has been very active and has great experience in the field of biomedical processing of brain and cardiovascular signals due to its participation in numerous research programs and the direct links that it has established with University Clinics. The current research team covers the following disciplines
— semantic technology
— biosignals management
— user interfaces
— medical experts in the field of cardiology. If expanded to other domains, relevant medical experts would be required to cooperate in the filtering and disambiguation of concepts, terms, etc
Furthermore, communications, computational and data management resources are available to some extent.
|2b||What are the technical issues that need to be tackled for full deployment, if needed?|
|More types of signals and data formats need to be supported, for example EEG.
This will require some alterations in the ontology schema, and potentially some support tools for the conversion.
More specifically, if for example EEG signals were encompassed in ROISES, the global ontology would be the element to be affected the most. Specifically, some upper classes containing the word ECG would be renamed e.g. ECGRecordContent while some others, like patientDiagnosis, annotations, Disease, waveforms, patientMedication would be leveraged with EEG domain specific terminology. Although the transformations on GO are not so dramatic, they have to be accomplished by the EEG domain expert, who would select the appropriate concepts to be used as query parameters by the researcher. The alterations specified in GO do not induce any modifications in the mapping procedure and the querying mechanism, thus the newly introduced signal needs no further adjustments to be encapsulated in ROISES. It has to be stated though, that the global ontology is ECG oriented since its structure is based on ecgML. Nevertheless the previously referenced alterations are feasible, revealing that a robust and interoperable system can be easily extended horizontally or vertically.
For the wider use of the application, it would be nice to work towards the integration and automated invocation of mechanisms generating signal annotations and interpretation content, as well as a generic standardized structure to express events (e.g. PhysioNet’s annotation scheme.
|2c||What additional technical resources are needed for the production of this new product?|
|Access to UMLS services (https://uts.nlm.nih.gov/license.html) is required, which at the moment offers free license for registered users.
The software employed is mainly open-source.
In case of extension with processing components, matlab license would be required.
|2d||Overall assessment of the current stage of technical development.|
|The R&D result has reached a high level of technical development. However, in order to be transformed into a commercially exploitable product it has to be improved in terms of the following:
SECTION 3: Deployment
|3a||Define the demands for large scale production in terms of|
|For a wider use, a server dedicated to ROISES would be required, accompanied by a backup or mirror server.|
|A person responsible for the maintenance of ROISES system and, temporarily, a second person during domain extension
One person part-tme for education and instructions
SECTION 4: Overall Assessment
|1||What is you overall assessment of the technical feasibility of the research result?|
KEYWORDS QUANTITATIVE ASSESSMENT (0-5).
|Please put X as appropriate.||1||2||3||4||5|
|Adequacy of testing activity undertaken so far||X|
|Adequacy and availability of technical resources of the development team||X|
|Current development stage||X|
|Overall technical feasibility||X|