Open access
1. Introduction
Electronic medical recording systems [1-4] have become widespread due to the improvement in hardware performance and user interfaces. Some recent systems are designed to support doctor–patient communication using a tablet PC [5-6]. However, usability is still an issue and medical professionals need more such user-friendly interfaces. To make these systems accessible to inexperienced users and to reduce the overhead of data entry, we have been developing various pen-based electronic medical recording systems [7-8]. Pen-based computing is an active research area for both user interfaces and computer graphics. Our work is based on recent advances in this area, especially the freeform user interfaces proposed by Igarashi [9]. Using this approach, the user draws freehand lines on the screen assisted by the system, and the result is directly stored as a vector image. Our systems feature special purpose functions for pen input including three-dimensional (3D) sketching, user-identification, and handwritten character recognition and search [8]. They are designed to help medical professionals to think more freely when working on difficult problems without being constrained by cumbersome interfaces.
One problem with these freeform pen-based systems, however, is that their output does not easily fit into a structure that lends itself to further machine processing or interface with other more traditional recording systems. Our goal in the project was to bridge this gap between freeform diagramming and more structured recording.
One strength of pen-based systems is that they make it easy to draw and add diagrams to medical records. This is particularly useful in ophthalmology, otolaryngology, and dentistry in which diagrams play an important role in medical records. Indeed, the frequent use of diagrams makes it difficult to use traditional GUI-based medical recording systems in these areas [10]. Cardiac catheterization is one of these areas in which the diagram is an indispensable tool for medical recording. Existing electronic medical recording systems rely on structured templates, but it is difficult to create an appropriate report of findings or treatment plan using these predefined templates. Most existing diagram editors are implemented as bitmap paint tools, not vector graphics. This makes it difficult to edit the geometry afterward and requires that a large amount of data be transmitted and stored
We therefore developed a pen-based interface for graphical reporting of findings in cardiac catheterization (Figure 1) [11]. Figure 2 shows an illustration of the human heart. The target of our system is the coronary arteries. Figure 3 shows a screenshot of our system. The user can freely “sketch” coronary arteries and stenoses on the screen using a pen on a template of coronary features. The location and degree of each stenosis, and various treatments such as bypass and stents, are visually represented. We developed an algorithm that can extract semantic information from the graphical representation and store it in XML format. The system can also generate a table in the format specified in the AHA (American Heart Association) committee report [12]. This system is useful not only as a tool for efficiently generating reports of findings but also as an effective explanation tool for patients.
Figure 1.
A screenshot of our system in use.
Figure 2.
An illustration of the human heart. The target of our system is the coronary arteries (the red vessels shown in this figure).
Figure 3.
Recording an example of cardiac catheterization.
2. User Interface
The user can draw on the diagram template using a pen as if drawing on real paper. Our system shows the name of each coronary artery and segment (e.g., proximal, middle, or distal) at the upper right of the screen when the cursor is over any vessel. We use the naming scheme defined in the AHA committee report. The system can show border lines of coronary artery segments if required.
Our system is a Windows application that provides a familiar interface to permit new users to work with it without extensive training. For example, the system displays a pop-up menu when the user clicks on a window while pressing the barrel button on the pen (Figure 4). This section describes the user interaction steps one by one. The next section describes our algorithm.
Figure 4.
Barrel button on pen.
2.1. Insertion / editing of a vessel
Upon start-up, our system displays a default cardiac catheterization coronary schema. The user can then draw a finding report or a treatment plan on the schema. The system provides several functions for editing the geometry on screen, including adding, deleting, and deforming arteries.
The user can draw a new coronary artery with the pen after choosing the “draw coronary artery” mode from the tool palette on the right (Figure 5 (a)). The system automatically creates an appropriate junction where the new artery is connected to another, and tapers the free end. The user can delete a vessel by clicking on it while holding the barrel button down and choosing “delete” from a pop-up menu. The system automatically updates the display on the screen. The user can move an artery by dragging it with the pen after choosing the “move coronary artery” mode from the tool palette (Figure 5 (b)). The user can deform an artery by dragging it with the pen (Figure 5 (c)) after choosing the “pull coronary artery” mode from the tool palette. We use a pulling interface for a curve that was introduced in [13]. It deforms a curve while preserving the local geometry. The user can also set the line width to small, normal, or large, using a pop-up panel (Figure 6). This system also supports the `absent‘ display, as shown in Figure 7.
Figure 5.
Editing operations on a coronary artery. (a) Draw a new coronary artery; (b) Move a coronary artery; (c) Pull a coronary artery
Figure 6.
The user can set the line width to small, normal, or large, using a pop-up panel. (a) Small; (b) Normal; (c) Large
Figure 7.
The system also supports the `absent’ display
2.2. Recording of stenoses
Once the user has sketched the geometry of the coronary arteries, he or she can record stenoses. To do this, the user chooses the “draw stenosis” mode from the tool palette and draws the stenosis on the target artery with the pen (Figure 8 (a)). When the user completes drawing and lifts the pen from the screen, the system displays a dialog box to specify the type and severity of the stenosis (Figure 8 (b)). If the user wants to change the properties of an existing stenosis, he or she can open the properties window by clicking on the affected artery while holding the barrel button down. The display of each stenosis on the screen includes the severity specified by the user (Figure 8 (c)).
The user can also move an existing stenosis by dragging it along the coronary artery (Figure 9). The stenosis snaps to the border of the appropriate section of the artery as it moves.
If the severity of a stenosis is set to 100%, the portion of the artery beyond the stenosis is shown as a thin line (Figure 10 (a)) representing a complete blockage where no blood flows. The system automatically analyzes the tree structure of arteries and closes any downstream vessels as well.
Figure 8.
Recording a stenosis. When the user draws a stenosis (a), the system displays a dialog box to specify the type and severity of the stenosis (b, c).
Figure 9.
The user can move an existing stenosis by dragging it along the coronary artery.
2.3. Bypasses and collateral
The user can add a bypass to the schema by drawing a line connecting coronary arteries. If the bypass connects an open artery to a closed one, the system automatically opens the blockage to indicate that blood flow has been restored (Figure 10 (b)). The user can place a stenosis on a bypass just like on a coronary artery and can also delete, move, and pull a bypass.
The user can draw a collateral (new blood vessels that reroute blood flow around a stenosis) by drawing a line between coronary arteries. This appears as an arrow in the schema (Figure 11). Our current implementation does not modify the blood flow automatically in response to a new collateral.
Figure 10.
Drawing a bypass. If the bypass connects an open coronary artery to a closed one (a), the system automatically opens the closed coronary artery to indicate that blood flow has resumed (b, c).
Figure 11.
Example of recording a collateral.
2.4. Stent
The user can place a stent in a coronary artery (Figure 12). Recording of stents is very important for documenting the treatment of the stenosis. The procedure for editing a stent is identical to that for editing a stenosis. The user creates and moves a stent by dragging it along a coronary artery and deletes it using a pop-up menu. The stent also snaps to the borders of the artery segments.
Figure 12.
Example of placing a stent.
2.5. Other Functions
The user can annotate the schema as a record of miscellaneous medical diagnosis and treatment. Our current implementation supports text and arrow marks in annotations (Figure 13). Unconstrained annotation encourages the user to think freely, similar to handwriting notes on traditional paper medical records. It is also helpful to remind the user of miscellaneous details associated with specific treatments.
The system can save an edited coronary schema, and then load it again for review or further editing. The schema is stored as vector graphics to reduce file size and facilitate editing. The system can export a schema in PNG image format for import into another system.
The user can create a new schema starting from a default coronary schema template and can also specify any schema to be the default template.
When the user places the mouse cursor on a vessel, the system displays the name of the vessel at the top right corner of the screen, as shown in Figure 14. The system displays not only the name of the vessel, but also its position (‘PROXIMAL’, ‘MIDDLE’ or ‘DISTAL’) in each blood vessel. Moreover, segment border lines are displayed when the user enables this feature in the pop-up menu shown in Figure 15.
Figure 13.
Example of recording text and arrow marks as an annotation.
Figure 14.
The system displays the name of the vessel at the top right of the screen when the user places the mouse cursor over it.
Figure 15.
The system displays segment border lines when the user enables this feature in the pop-up menu.
3. Dataset structure and cooperation with other systems
Many doctors use coronary angiography (CAG) to represent coronary stenosis pathology. CAG compactly shows the location and severity of stenoses. Our system supports the conversion of the graphical record to a CAG-compliant table dataset. The table is represented in the format specified in the AHA committee report and stored as an XML file. Figure 16 shows the relationship between our system and CAG. The top screen in Figure 16 (a) presents an example of recording stenoses using our system; the middle screen of Figure 16 (a) shows the CAG dataset it produces. Any other system that supports this format can use the data file as shown in the bottom screen of Figure 16 (a).
The user can also edit the exported CAG table. When this happens, our system automatically updates the corresponding stenosis on the coronary diagram including the information on the severity and character of the stenosis (Figure 16 (b)).
When a stenosis is straddling two or more segments, it is considered to belong to two or more segments. The stenosis drawn on LAD7 and LAD8 amid the strangulation shown in Figure 17 is an example.
Figure 16.
Example of the automatic relationship between the coronary diagram and the CAG table. (a) The system automatically generates a CAG table from a graphical coronary schema by checking the existence of a stenosis in each segment of the coronary arteries, and stores the results in XML format. (b) The system can automatically update the stenoses on the coronary schema from the corresponding CAG table.
Figure 17.
Example of a stenosis over two divisions.
4. Implementation
We designed our system as a platform-independent JavaTM program using the Java2DTM graphics application programming interface. This section describes the implementation details of the current prototype.
4.1. On-screen displays
The system displays coronary arteries as two parallel lines and handles the branches appropriately (Figure 18 (a)). A vessel is a polyline composed of small line segments. The system first draws a wide red line and then a narrow white line inside (Figure 18 (b1), (b2)). The width of these lines decreases toward the non-connected end of a vessel to represent the taper (Figure 18 (c1), (c2)).
A stenosis is displayed in a similar manner. The system first draws a wide black line inside the vessel and then a narrow white line inside that. A stent is rendered by drawing a hatching pattern after setting a stencil inside the stent area.
Figure 18.
Vessel representation. The system displays coronary arteries as two parallel lines and handles the branches appropriately (a). The system first draws a wide red line (b1), and then a narrow white line inside (b2). The width of these lines decreases toward the non-connected end of a vessel to represent the taper (c1, c2).
4.2. Geometry editing
The pulling interface deforms the curve while maintaining its local details (Figure 5 (c)) [13]. The system first generates triangles by connecting sets of three neighboring points on a polyline. As the user pulls a point along the curve, the system determines the location of free vertices so as to minimize the distortion of the triangles. We also used the peeling interface introduced in [13] to adjust the size of the region to be deformed, so that a larger area is deformed as the user pulls more. As the user pulls the curve further away, the influence region grows (Figure 19, left to right).
Figure 19.
We use the pulling and peeling interface introduced in [
4.3. Generation of the CAG table
The CAG table stores the following information for each segment of a coronary artery: the presence or absence of a stenosis, the severity of the stenosis, and the type of stenosis (Figure 16 (a), bottom). The system automatically generates a CAG table from a graphical coronary schema by checking for the existence of a stenosis in each segment. It stores the result in XML format (Figure 16 (a), middle and bottom).
When the user edits the CAG table, the system first finds the corresponding stenosis in the XML file (Figure 16 (b), top and middle). It then obtains the information for that stenosis and changes it on the coronary schema. In this way, the system automatically updates the stenoses on the coronary schema from the corresponding CAG table (Figure 16 (b), bottom).
5. Case report using our system
We illustrate the effectiveness of our system, utilizing two cases of coronary artery bypass surgery as examples. These examples were only described (not illustrated) in the original papers.
The first example is Case 1 of [14]. The paper describes it as follows:
`A man, 45 years of age, had suffered attacks of angina pectoris during many years. He had had infarction of the myocardium. During the operation it was noted that the left coronary artery and the initial portions of its main branches were calcified. We also noted density of the right coronary artery. Anastomosis was applied between the inner thoracic artery and the circumflex branch of the left coronary artery.’
Figure 20.
Case 1 of the report [
Figure 20 shows how to illustrate this process using our system. First, the user inputs a CAG table, as shown in Figure 20 (a). With our system, the user can set the type of the corresponding stenosis, as shown in the CAG table. The system then automatically generates a graphical coronary schema, as shown in Figure 20 (b). Figure 20 (c) shows the result of setting the severity of a stenosis of the left coronary artery to 100%.
The bypass connects an open vessel to the closed coronary artery, and the system automatically opens the closed coronary artery to indicate that blood flow is recovered, as shown in Figure 20 (d, e).
The second example is Case 3 of the report [14]. The paper describes it as follows:
`A male patient (40 years of age), had suffered from generalized atherosclerosis. (…) During the operation calcification and complete occlusion of the initial portion of both branches of the left coronary artery were found. An end-to-end anastomosis between the inner thoracic and interventricular arteries was made.’
Figure 21.
Case 3 of the report [
Figure 21 shows how to illustrate this process using our system. First, the user inputs a CAG table, as shown in Figure 21 (a). The system then automatically generates a graphical coronary schema, as shown in Figure 21 (b). By default, the system automatically generates the stenosis in the middle portion of the corresponding vessel, as shown in Figure 21 (b). The user can move the stenosis to the initial portion, as shown in Figure 21 (c). Figure 21 (d) shows the result of setting the severity of the stenosis to 100%.
The bypass connects an open vessel to the closed coronary artery, and the system automatically opens the closed coronary artery to indicate that blood flow is recovered, as shown in Figure 21 (e, f) and (g, h).
6. Discussion
Our current implementation is a research prototype and is not yet being used in clinical practice. However, we have already demonstrated it to medical professionals and confirmed the following benefits:
The user can easily modify the geometry of coronary arteries for individual patients.
The system can store the data compactly using vectors instead of bitmaps, which significantly improves the network response when storing information on a remote server.
The system can export the CAG table based on the AHA committee report in XML format. Therefore, the system can easily exchange data with other existing systems.
The user can edit a coronary schema while viewing a reference image on the same display.
The user can draw diagrams and text freely in our system, which allows the recording of new anomalies that have never been previously observed.
In addition, we received the following comment from another heart surgeon: `This is a user-friendly system. It is particularly effective for inexperienced doctors. Diagnosis is performed by a heart physician. But, I think that it is useful also for a young surgeon's training. ‘
The correspondence between a diagnosis and a dissection, as well as comparison between the diagnosis and a CT scan image, are important to a surgeonpreparing for an operation. However, even though there is an AHA standard that defines how to verbalize diagnosis results, there is significant variation in the way surgeons describe diagnosis results, even among experts. Accordingly, one specialist commented that it is useful to have a link between CT scan images of the circumflex branches to the corresponding locations in the schema. The specialist also commented that two-dimensional (2D) representation is sufficient if the purpose of the target system is diagnosis, but 3D representation is desirable for training purposes.
An issue with the current implementation is that it is limited by the AHA standards. The manner of recording schemas for cardiac catheterization varies widely among users and facilities. As the AHA committee report was designed more than 30 years ago, it cannot handle many cases well. Therefore, a more powerful and flexible representation is needed.
7. Conclusion and future work
We developed an effective interface for reporting graphical findings in cardiac catheterization using hand-drawn diagrams. The user can easily record the position and degree of a stenosis on a coronary schema template, and can also record treatments such as bypasses and stents. Once a bypass is added, the system automatically displays the resumption of blood flow. This type of automatic adaptation is not possible with paper-based medical records. Our system can store the data as a CAG table in an XML file in the AHA format for exchanging data with other existing systems. Our system makes it easier to handle graphical schemas in medical recording systems, encouraging the spread of medical recording systems in general.
Our system operates independently and does not require any other special infrastructure. Therefore, it can be easily introduced at low cost. We hope to put our system into actual clinical practice to make improvements based on feedback from actual users. We also plan to experiment with 3D schemas because 3D images are becoming increasingly widespread. We are also currently working with methods pertaining to 3D images. For example, Nakao et al. proposed a 3D cardiovascular modeling system based on neonatal echocardiographic images [15]. Using this system, medical doctors can interactively construct patient-specific cardiovascular models, and share the complex topology and shape information. Bo et al. introduced a lightweight sketching system that enables interactive illustration of complex fluid systems [16]. Users can sketch on a 2.5D canvas to design the shapes and connections of a fluid circuit. Ijiri et al. developed an efficient and robust framework for simulating the cardiac beating motion [17]. The global cardiac motion is generated by the accumulation of local myocardial fiber contractions. They compute such local-to-global deformations using a kinematic approach, dividing a heart mesh model into overlapping local regions, contracting them independently according to fiber orientation, and computing a global shape that fits the contracted shapes of all local regions as well as possible.
The interactive graphical schemas introduced in this paper should be useful in not only cardiac catheterization but also other areas that use schemas, for example, ophthalmology, otolaryngology, and dentistry. Such interactive schemas are useful not only for efficiently generating finding reports but also as an effective explanation tool for patients.
References
- 1.
Takashi Matsumura. Electronic Medical Record and Hospital Information System . Comprehensive Management of Patient Information--. Japan Journal of Medical Informatics, 21(3), 2001, 211-222 (in Japanese). - 2.
Yasushi Matsumura. Current Status of Electronic Medical Record and its Tasks. The Medical Frontline, 2003; 58(8): 46-51 (in Japanese). - 3.
Dean F. Sitting, Gilad J. Kuperman, Julie Fiskio. Evaluating Physician Satisfaction Regarding User Interactions with an Electronic Medical Record System. Proceedings of AMIA Annual Symposium 99, 1999, 400-404. - 4.
Mark Ginsburg. Pediatric Electronic Health Record Interface Design: The PedOne System. Proceedings of the 40th Hawaii International Conference on System Sciences, 2007, 139. - 5.
Erik K. Fromme, Tawni Kenworthy-Heinige and Michelle Hribar. Developing an easy-to-use tablet computer application for assessing patient-reported outcomes in patients with cancer. Supportive Care in Cancer, Volume 19, Number 6 (2011), 815-822. - 6.
Ole Andreas Alsos, Anita Das, Dag Svanæs. Mobile health IT: The effect of user interface and form factor on doctor–patient communication. International Journal of Medical Informatics, 81(1), 12-28, 2012. - 7.
Kazuo Nakazawa, Takeo Igarashi. A Comprehensive Computer Interface for the Spreading of Electronic Medical Recording Systems Supporting Hand-writing Inputs by Pen Device. New Medicine in Japan, 2003; 345: 74-77 (in Japanese). - 8.
Kazuo Nakazawa, Ryo Haraguchi, Takenori Yao, Satoru Nagata, Yoshihisa Sugimoto, Takashi Ashihara, Masahiro Takada, Hirokazu Namikawa, Yasushi Okada, Kohei Yoshimoto, Naomi Furuya, Shuji Senda, Yoshiyuki Uematsu, Takeo Igarashi. Pen-based Interface for Electronic Medical Record System with Advanced Recognition and Search Engine for the Hand-written Characters. Japan Journal of Medical Informatics, 2005; 25(2), 81-86 (in Japanese). - 9.
Takeo Igarashi, "Freeform User Interfaces for Graphical Computing", Proceedings of 3rd International Symposium on Smart Graphics, Lecture Notes in Computer Science, Springer, 39-48, 2003. - 10.
Sakabe Nagamasa, Arai Kazuo, Abe Kazuya, Inoue Hideo, Sakai Shunichi, Watanabe Yukio. A Survey of demerits of Electronic Medical Record in the Ear-Nose and Throat Department. Japan Journal of Medical Informatics, 2001; 21(1): 131-136 (in Japanese). - 11.
Yuki Mori, Takeo Igarashi, Ryo Haraguchi, and Kazuo Nakazawa. A Pen-based Interface for Generating Graphical Reports of Findings in Cardiac Catheterization. Methods of Information in Medicine, 2007;46(6) pp.694-699. - 12.
Austen G. W, Eswards J, Frye R. L., et al. "A Reporting System on Patients Evaluated for Coronary Artery Disease", Circulation, 1975; 51(4 Suppl.): 5-40. - 13.
Igarashi T, Moscovich T, Hughes J F. As-Rigid-As-Possible Shape Manipulation. ACM Transactions on Computer Graphics 2005; 24(3): 1134-1141 - 14.
Kolessov, V. I. Mammary artery-coronary artery anastomosis as method of treatment for angina pectoris. The Journal of Thoracic and Cardiovascular Surgery. 1967; 54(4): 535-544. - 15.
Megumi Nakao, Kazuma Maeda, Ryo Haraguchi, Ken-ichi Kurosaki, Koji Kagisaki, Isao Shiraishi, Kazuo Nakazawa, Kotaro Minato. "Cardiovascular Modeling of Congenital Heart Disease Based on Neonatal Echocardiographic Images", IEEE Transactions on Information Technology in BioMedicine, 16(1), 70-79, 2012. - 16.
Bo Zhu, Michiaki Iwata, Ryo Haraguchi, Takashi Ashihara, Nobuyuki Umetani, Takeo Igarashi, Kazuo Nakazawa. Sketch-based Dynamic Illustration of Fluid Systems. ACM Transactions on Graphics (Proceedings of SIGGRAPH Asia 2011), 30(6), Article No. 134, 2011. - 17.
Ijiri T, Ashihara T, Umetani N, Igarashi T, Haraguchi R, et al. (2012) A Kinematic Approach for Efficient and Robust Simulation of the Cardiac Beating Motion. PLoS ONE 7(5): e36706.