Intuitive Surgical Navigation System for Dental Implantology by Using Retinal Imaging Display

1. Introduction

As dental implants have become an established treatment, their application to aggressive cases with insufficient quantity and quality of bone has increased. To perform safe and precise surgery, overcoming these difficulties, computer-assisted systems have been developed (Jabero et al., 2006). There are two major classes of these systems, that is, a computer-guided (static) system (Schneider et al., 2009) and a computer-navigated (dynamic) system (Jung et al., 2009). The computer-guided system such as SimPlant software and SurgiGuide (Materialize Dental Inc., USA), NobelGuide (Nobel Biocare Management AG, Sweden), and 10DR (10DR JAPAN Co., Ltd., Japan) are a surgical template based system which can simulate an optimal implant location, angle, direction, depth and size pre-operatively. Our system, BoneNavi (Bionic Inc., Japan), has also been developed to simulate implant placement and the surgical template fabrication for dental implant surgery (Ohtani et al., 2009) and applied in more than 200 clinical cases in Japan (Sohmura et al., 2009). On the other hand, the computer-navigated system such as IGI system (Image Navigation Ltd., USA) offers freehand implant navigation empowered by highly accurate motion tracking technology that tracks the positions of the dental drill and patient throughout the surgery (Casap et al., 2004). It has also already been applied in clinical cases (Casap et al., 2005& 2008). Advantages and disadvantages of the computer-guided system and the computer-navigated system are shown in Table 1.

Unlike an open chest surgery and an open abdominal surgery, looking away from the oral cavity (narrow space (Fig.1)) intra-operatively involve a risk of operation errors in case of a dental implant surgery. However, with conventional systems, surgeons have to manipulate instruments in the patient’s oral cavity while watching a surgical monitor as shown in Fig.2(a) and they feel anxious during operations.

To overcome this problem, we have been developed a novel surgical navigation system by combining the retinal imaging display (RID) and the augmented reality (AR) techniques

	Advantages	Disadvantages
Computer-guided system	General familiarity; High degree of precision	Difficulties in cases of poor edentulous ridge form or loose teeth
Computer-navigated system	Modifiable surgical plan intra-operatively	Complications and difficulties for mounting markers on patiens or hand-piece

Table 1.

Comparison between advantages and disadvantages of the computer-guided system and the computer-navigated system

(Azuma, 1997) that can directly overlay pre-operative simulation images onto the real view of the surgeon (Yamaguchi et al., 2009) as shown in Fig. 2 (b). In the last few decades, AR techniques have been applied in wide areas such as medical treatment (Peters et al., 2006) and education (Dede, 2009; Thomas, 2010). Moreover, various possibilities are still expected (Buxton et al., 2008). However, only few attempts have so far been made at an application in dentistry (Birkfellner et al., 1999) because of challenging field by means of image registration accuracy. It is because I believe that our system lead to progress in future dentistry that I have written this chapter.

Figure 1.

Scene of dental implant surgical operation in narrow space of patient’s oral cavity.

Figure 2.

Schematic illustrations of a surgical navigation system for dental implantology. (a) Conventional approach: A direction of eyes of a dentist is directed to a navigation monitor. (b) Our approach: The direction of eyes of the dentist is directed to an oral cavity of a patient.

2. Materials and methods

2.1. System configurations

As shown in Fig. 3 system configurations, our system consists of a retinal imaging display (Prototype of RID (RID), Brother Industries, Ltd., Japan), an optical tracker (Micron Tracker 2 Sx60 (MT2), Claron Technology Inc., USA), and a laptop computer (Thinkpad X32, IBM, Japan). The RID have an image projection technology that focuses light, of an intensity

Figure 3.

System configurations

harmless to the eyes, onto the retina and then moves the light at high speed to create after images. This technology differs from an optical see-through head mounted display and a half mirror in respect of projection principle for eyes and has many advantages for dentists. Dentists can intuitively confirm an ideal implant placement without an uncomfortable feeling. And, dentist’s view is not interrupt by lenses such as glasses. Moreover, dentist can view private images and personal data of patients away from being viewed by others intra-operatively. To track a position and a pose of marked patient’s oral cavity, computer vision technologies are implemented on MT2 instead of conventional optical tracker by using infrared light emission. Thus, MT2 is a cost-effective and high accurate (jitter for moving target is less than 70 micron) optical tracker. A laptop PC receive signals from RID and MT2 and calculate a translate matrix for real-time image overlay.

2.2. Pre-operative procedures

1. To measure the position and orientation of a marker 1, the marker 1 is placed onto a partially edentulous model.

2. A placement position of dental implants for the scanned images is simulated by CAD software (FreeFormModeling, SensAble Technologies, Inc., USA) and the three-dimensional simulation image is generated as a computer graphics (CG). At this time, a translation matrix $${}_{marker1}{}^{implant}D$$ from the marker 1 to implants is recorded.

3. A translation matrix $${}_{marker2}{}^{eye}A$$ to dentist’s eye from the RID with a marker 2 attached thereto is calculated. By using the marker 1 for calibration, the $${}_{marker2}{}^{eye}A$$ is calculated from three-dimensional positions on the RID coordinate system and its corresponding two-dimensional positions on the dentist’s eye coordinate system (Kato et al., 1999).

2.3. Intra-operative procedures

1. Translation matrixes $${}_{tracker}{}^{marker1}D$$ and $${}_{tracker}{}^{marker2}D$$ are measured by the MT2 in real-time intra-operatively.

2. A translation matrix $${}_{implant}{}^{eye}P$$ is calculated from simulated dental implant CG image to the dentist’s eye as per a following equation (1). Finally, the CG image is projected onto the dentist’s view,

$${}_{implant}{}^{eye}P={}_{implant}{}^{marker1}D{}_{marker1}{}^{tracker}D{}_{tracker}{}^{marker2}D{}_{marker2}{}^{eye}A$$ E1

where, $${}_{implant}{}^{marker1}D={}_{marker1}{}^{implant}D{}^{-1}$$ and $${}_{marker1}{}^{tracker}D={}_{tracker}{}^{marker1}D{}^{-1}$$ .

2.4. Design of markers and partially edentulous model

CAD models of the marker 1 and the marker 2 were designed by the CAD software (Fig. 4). A partially edentulous model was scanned by three-dimensional measurement system (Rexcan scanner, Solutionix Corp., Korea). The marker 1, the marker 2, and the partially edentulous model were fabricated by rapid-prototyping machines (Connex500 (two-color process) and EDEN260 (one-color process), Objet Geometries Ltd., Israel). In clinical case, the marker 1 will be placed inside oral cavity of the patient. As shown in Fig.4 and Fig.5, the markers consist of three faces. Sizes of all faces were set to 26 × 21 mm. Both two faces were tilted to a center marker. Its tilt angle was 30 degree. Unlike a white and a black color, a grey and a black color were adopted to improve a contrast ratio for MT2. Both the marker 1 and the marker 2 have different patterns. Fig. 6 shows a pre-operative simulation result including five implant models and simplified models of a mandibular canal which was extracted from a CT data by using a volume rendering software (VG Studio Max 2.0, Volume Graphics GmbH, Germany).

Figure 4.

The CAD model of the marker 1 and the scanned model of the partially edentulous model: (a) top view, (b) front view, (c) lateral view

Figure 5.

The CAD model of marker 2: (a) top view, (b) front view, (c) lateral view

Figure 6.

Pre-operative simulation result: (a) top view, (b) front view, (c) lateral view

2.5. Calculations of $${}_{implant}{}^{eye}P$$

2.5.1. DLT (Direct Linear Transformation)

Direct linear transformation is a gold standard calibration method (Hartley, R. & Zisserman, A., 2003). By using the marker 1 for calibration, A translation matrix $${}_{marker2}{}^{eye}A$$ was calculated from three-dimensional positions on the RID coordinate system $$p\left(x,y,z\right)$$ and its corresponding two-dimensional positions on the dentist’s eye coordinate system $$P\left(X_C,Y_C\right)$$ . A transform relationship between a RID coordinate value $$p$$ and a dentist’s eye coordinate value $$P$$ was represented as per a following equation (2).

$$\left[H_C{X}_C,H_C{Y}_C,H_C\right]=\left[x,y,z,1\right]\left[\begin{array}{ccc}{C}_{11}& C_{21}& C_{31}\\ C_{12}& C_{22}& C_{32}\\ C_{13}& C_{23}& C_{33}\\ C_{14}& C_{24}& C_{34}\end{array}\right]$$ E2

These twelve parameters represent constant values such as a relative position, an orientation, and a focal length. A translation matrix $$C$$ calls camera parmeters. The camera parameters were calculated by collecting some three-dimensional positions on the RID coordinate system $$p_i\left(x_i,y_i,z_i\right)$$ and its corresponding two-dimensional positions on the dentist’s eye coordinate system $$P_i\left(X_{Ci},Y_{Ci}\right)$$ . Once a coordinate value $$x_1,y_1,z_1$$ of a three-dimensional position $$p_1$$ on the RID coordinate system and its corresponding coordinate value $$X_{C1},Y_{C1}$$ of a two-dimensional position $$P_1$$ on the dentist’s eye coordinate system was determined, an equation (3) was derived from the equation (2).

$$\begin{array}{l}{C}_{11}{x}_1+C{}_{12}y{}_1+C_{13}{z}_1+C_{14}-C_{31}{x}_1{X}_{C1}-C_{32}{y}_1{X}_{C1}-C_{33}{z}_1{X}_{C1}=C_{34}{X}_{C1}\\ C_{21}{x}_1+C_{22}{y}_1+C_{23}{z}_1+C_{24}-C_{31}{x}_1{Y}_{C1}-C_{32}{y}_1{Y}_{C1}-C_{32}{z}_1{Y}_{C1}=C_{34}{Y}_{C1}\end{array}$$ E3

Six noncoplaner fiducial points were required for calculation of twelve parameters $$C_{11}\cdots C_{34}$$ because we have two equations once a fiducial point. Actually, a least square method was applied for accurate calculations by using over six fiducial points. In this chapter, we collected fifteen fiducial points for the calculation. Here, we just have to collect a relative (not absolute) coordinate value of $$p_i$$ (a distance between each point and an orientation) on the RID coordinate system. If a set of n > 6 fiducial points were given, an equation (4) was derived. There is no problem either $$C_{34}$$ or $$H_C$$ are set to a value of constant because a scaling was determined by both parameter. Therefore, we assume that the $$C_{34}$$ in the camera parameters equal to 1.

$$AC=R$$ E4

where

$$A=\left[\begin{array}{ccccccccccc}{x}_1& y_1& z_1& 1& 0& 0& 0& 0& -x_1{X}_{C1}& -y_1{X}_{C1}& -z_1{X}_{C1}\\ 0& 0& 0& 0& x_1& y_1& z_1& 1& -x_1{Y}_{C1}& -y_1{Y}_{C1}& -z_1{Y}_{C1}\\ ⋮& ⋮& ⋮& ⋮& ⋮& ⋮& ⋮& ⋮& ⋮& ⋮& ⋮\\ x_n& y_n& z_n& 1& 0& 0& 0& 0& -x_n{X}_{Cn}& -y_n{X}_{Cn}& -z_n{X}_{Cn}\\ 0& 0& 0& 0& x_n& y_n& z_n& 1& -x_n{Y}_{Cn}& -y_n{Y}_{Cn}& -z_n{Y}_{Cn}\end{array}\right],$$ E5

$$C=\left[\begin{array}{c}{C}_{11}\\ C_{12}\\ C_{13}\\ ⋮\\ C_{32}\\ C_{33}\end{array}\right],R=\left[\begin{array}{c}{X}_{C1}\\ Y_{C1}\\ ⋮\\ X_{Cn}\\ Y_{Cn}\end{array}\right].$$ E6

If $$n$$ fiducial points were given, translation matrixies $$A$$ , $$C$$ and $$R$$ are represent as 2 $$n$$ ×11, 11×1 and 2 $$n$$ ×1 respectively. In cases where n > 6 fiducial points including measurement errors were given, each value of $$C$$ could not be calculate because of mathematically impossible problem. Consequently, a task is to find the matrix $$C$$ that minimizes a norm $$\Vert AC-R\Vert$$ . In the case of a determination of the camera parameters, eleven unknown parameters $$C_{11}\cdots C_{33}$$ were given and a number of equations is 2 $$n$$ (a number of fiducial points = $$n$$ ), that is, the impossible problem $$AC=R$$ was solved by means of a least square solution. Its solution $$\left[C\right]$$ was derived as an equation (7).

$$A^{T}A\left[C\right]=A^{T}R$$ E7

2.5.2. LMedS (Least Median Square)

Our system was operated not only by researchers in engineering and engineers but also dentists without technical knowledge. Therefore, our system was expected that anyone could perform an image overlay easily in not only laboratories but also operating rooms. In conventional system, we adopted a least square method which is best used in statistical methods to calculate a projection matrix. The least square method which determine parameters by minimizing a sum of square of fitting errors is used as a general calibration method. However, an unchanged application of the least square method is not enough. In case of including exception values, the projection matrix which is estimated by the least square method could not be trusted. If the exception values could be automatically eliminated by fitting a model to a major part in data set, parameters in the major part will be able to estimated accurately. To solve this problem, we introduce a least median square (LMedS) method as a typical robust statistics. A method and an algorithm are as follows. A least square criterion is

$$LMS=\min {\displaystyle \sum _i{\epsilon }_i^2}.$$ E8

A LMedS criterion is

$$LMedS=\min med{\epsilon }_i^2$$ E9

where $$med$$ means median. In case that the least square criterion, a solution is affected by one big exception value. However, even if a half of all data set were exception values, the solution of the LMedS criterion is not affected. Thus, the LMedS criterion is suitable for this situation.

1. Step 1. n click of a mouse for an origin position of the marker 1 on the view through the RID, varing location of the marker 1. By this task, n set of three-dimensional points on the marker 1 coordinate system and its corresponding two-dimensional points on the dentist eye’s coordinate system are collected. By selection F points from n set of corresponding points, candidates of the projection matrix are calculated by the calibration using the least square method.

2. Step 2. Calculate error values of the candidate of solution as following equation.

$$er{r}^2=med\left({\left(X_{Ci}-{\overline{X}}_{Ci}\right)}^2+{\left(Y_{Ci}-{\overline{Y}}_{Ci}\right)}^2\right)$$ E10

where $$\left(X_{Ci},Y_{Ci}\right)$$ is two-dimensional coordinate value correspoinding to ith origin of the marker 1 on the dentist eye’s coordinate system and $$\left({\overline{X}}_{Ci},{\overline{Y}}_{Ci}\right)$$ is reprojected two-dimensional coordinate value which is calculated by the candidate of solution. $$med\left(f\left(i\right)\right)$$ represent a median of $$f\left(i\right)$$ in all i.

1. Step 3. Repeat Step 1 and Step 2 q times. F correspoinding points that minimizing $$er{r}^2$$ in equation (10) are calculated as following equation (11). Then, camera parameters which is calculated by F corresponding points are selected. A required repitation time in this algorithm can be determined considering a probability that exception values are not included in at least one sample in q random samplings. Let $$\epsilon$$ be a ratio of exception value in all data. This probability is represented as following equation (11).

$$P=1-{\left\{1-{\left(1-\epsilon \right)}^F\right\}}^q$$ E11

For instance, the number of random samplings in $$\epsilon =0.3,F=3$$ is $$q=11$$ . In other words, this means the number of random samplings is over eleven. In our system, $$F=10,n=15,\epsilon =0.3,P=0.99$$ are adopted emprically.

Next, we shall focus on a measurement range. Let A be a measurable range of both the marker 1 and the marker 2 on the MT2 coordinate system. Let B be a measurable range of the marker 2 on the RID corrdinate system. A measurable range of our system represent $$A\cap B$$ . Within this measurable range, the calibration is available.

3. Results and discussions

Fig. 7 (a) shows fabricated models of the marker 1 and the partially edentulous model. Fig. 7 (b) shows the marker 2 attached to the RID. Few systematic errors are expected because both markers were fabricated together. Moreover, a stable intra-operative navigation is achieved even if any one of a pattern of the makers which consists of multiple patterns is hide out in surgical instruments.

Figure 7.

Fabricated models: (a) The marker 1 and the partially edentulous model, (b) The marker 2 attached to the RID

We adjusted a contrast ratio of colours using the markers on the grounds that the contrast ratio of a grey and black (GB) colour is 20 times higher than that of a white and black (WB) colour. Fig. 8 shows an adjustment result of the contrast ratio. Fig. 8 (a) shows a condition of no spot light in case of a combination with the white and the black colour. Fig. 8 (b) shows the condition of a spotlight in case of a combination with WB. Fig. 8 (c) shows the condition of the spotlight in case of the combination with the grey and the black colour. In the condition of Fig. 8 (b), the WB marker was not measurable because of halation cause the spotlight. However, as shown in Fig. 8 (c), the GB marker was measurable under the condition of the spotlight. Accordingly, we adopt the GB marker around the marker 1 attached to patient’s oral cavity with the spotlight and the WB marker 2 around the RID with an environmental light.

As shown in Fig. 9, a measurable range of a new marker was about 2.6 times higher than that of a conventional marker. A tilt angle was set to opposite direction between the marker

Figure 8.

Adjustment result of contrast ratio. Left marker is combined with a white and black colour (WB). Right marker is combined with a grey and black colour (GB). (a) No spot light, (b) WB marker with spot light, (c) GB marker with spot light

1 and the marker 2 to prevent a false recognition. Fig. 10 shows reprojection errors between the DLT and the LMedS method. An average and a standard deviation of the reprojection errors in the LMedS method indicated decreasing trend in comparison with the DLT method. One of the reasons for the reprojection errors in the LMedS method is dentist’s hand movement while sampling. As we have mentioned before, the sampling is performed manually by clicking a mouse in our system. Therefore, some errors may be observed cause dentist’s hand movement. So, there is room for further investigatioin about the sampling. And, it must be notice that calibration will be required again in case that a movement of dentist’s own misalign relationship between the RID and the dentists.

Fig. 11 and Fig. 12 show real-time image overlay results on the conventional head mounted display and the RID respectively. As shown in Fig. 12 (a) and (b), an image overlay result through the RID was crucially clear view in comparison with that of a conventional head-mounted display (Fig. 11 (a) and (b)). To take Fig. 11 and Fig. 12, we used an off-the-shelf USB camera in place of dentist’s eye. Therefore, actual view was better because of direct overlay onto dentist’s retina in case of the RID.

4. Conclusion

We proposed the dental implant surgical navigation system by combining the AR technology and the RID and its image overlay procedure, and our procedure proved to be performed as a real-time image overlay. By implementation of the LMedS method, more robust calibration was achieved. Further development of the markers will be planned to decrease the reprojection errors by using the other markers including known sampling points fixed on patient’s oral cavity. To achieve clinical applications of this system, evaluation of a fiducial registration error (FRE) and a target registration error (TRE) (Fitzpatrick et al., 1998) and the procedure by real CT data is in progress.

Figure 9.

Comparison with conventional and new measurable range

Figure 10.

Comparison with reprojection errors of the DLT method (Yamaguchi et al., 2009) and the LMedS method

Figure 11.

Real-time image overlay result on the conventional head mounted display: (a) scene 1, (b) scene 2

Figure 12.

Real-time image overlay result on the RID: (a) scene 1, (b) scene 2

Acknowledgments

This study was supported by Grants-in-Aid for Scientific Research No. 21791937 from the JSPS. For the taking pictures, grateful thanks are due to Noriko Negi at School of Dentistry Osaka University. For the use of RID, grateful thanks are due to Hirokuni Kato and Hideki Tani at Brother Industries, Ltd in Japan. For the advice of calibration techniques, grateful thanks are due to Prof. Hirokazu Kato at Nara Institute of Science and Technology in Japan.