Production of a Tunnel Mosaic Image Using Fisheye Lens Camera

Methods of monitoring tunnel interior walls have mainly focused on laser scanners or line scan cameras. In this study, we propose a method of acquiring images of tunnel interior walls using a fisheye lens camera. First, we produce rectangular images that include the tunnel walls and floors by applying a cylindrical projection transformation to the circular fisheye lens images. Distortions are corrected to produce more realistic images, which are then connected to produce a tunnel mosaic image. The advantage of this method is that it can save working time and relatively reduce costs. The tunnel mosaic image produced by our method can be used for tunnel surface inspection to detect problems such as large cracks, efflorescence caused by water leakage, and exfoliation.


Introduction
Photogrammetry has often been used to monitor deformations and cracks in civil structures. However, in the case of linear structures, such as tunnels, it is difficult to apply a general type of photogrammetry owing to difficulties in ensuring an appropriate photographing distance. Thus, the method of surveying such structures generally involves using laser scanners or line scan cameras. (1,2) Jeong et al. proposed a monitoring method for tunnel interior cracks using a line scan camera. This method involves using an image acquisition system that acquires tunnel images in real time and an image-merging system that produces a long image by combining the acquired images. (2) Although it was possible to successfully diagnose cracks by this method, some improvements, such as correcting several unknown errors and increasing the speed of the image-taking vehicle to above 20 km/h, were suggested. Paar and Kontrus investigated a reconstruction method for tunnel surfaces by combining high-resolution images and laser scanning. (3) In their study, two image acquisition methods were introduced. One obtained images using a camera fixed to the scanner head, and the other obtained scanner and camera data separately, and then determined the approximate relative position between the two devices using an odometer. They then described data processing, visualization, and crack monitoring. Sun et al. proposed a method of fabricating tunnel models using monocular images. (4) They extracted linear parallel shapes along the longitudinal direction in monocular images and determined a line projection matrix using the linear information obtained. A 3D tunnel model was then produced by resampling textures using this matrix. This method had the disadvantages that it can only be applied to straight tunnels and the amount of distortion increases considerably with the image-taking distance. Moisan et al. proposed a method for dynamically surveying the vault and underwater parts of a tunnel. (5) They reconstructed the vault and side walls of the tunnel for estimating the trajectory of a boat, which was necessary to rearrange sonar profiles to form a 3D model of the canal.

Fisheye Images
We have studied the acquisition of image information from tunnel interiors using a fisheye lens. Such a lens has a superwide view angle of 180°. Although this lens distorts the images, it has the advantage of including images from all directions in a single image, as shown in Fig. 1. Thus, it can reduce site working time by decreasing the number of images required.
A number of studies on the geometrical characteristics of fisheye images and the correction of distortions have been conducted. Schwalbe (6) and Schneider et al. (7) described a geometric projection model for a fisheye lens and performed its calibration. For this, a calibration room was built and targets were installed in the room to determine their distribution in the images. The calibration was performed by spatial resectioning from a single room image. Hughes et al. introduced some calibration methods for radial distortions and fisheye lenses. (8) Kweon developed an algorithm for extracting rectilinear images from fisheye images and produced polygonal panoramic images. (9) Abraham and Förstner presented a method of generating epipolar images for fisheye stereo images. (10) However, it was not possible to find an application of fisheye lenses to the acquisition of interior images in a linear structure with no targets for known control points, as proposed in this study.
The processing of fisheye images proposed in this study is illustrated in Fig. 2. First, we transform tunnel wall images, which are extracted from circular fisheye tunnel images, by applying a cylindrical projection to produce general rectangular images. We then analyze the major types of distortion appearing on the projected images and develop a method of calibrating each distortion type. Finally, we produce a mosaic image by connecting these corrected tunnel images. In this study, a SIGMA 8 mm 3.5 EX DG Circular Fisheye lens is used.

Image Processing and Correction
A tunnel image taken with a fisheye lens has a circular shape, as shown in Fig. 3(a). This circular image is unfolded, as illustrated in Fig. 3(b), to produce a rectangular image [ Fig. 3(c)]. Because such an unfolded image represents a projection of the tunnel wall onto a cylinder (Fig. 4), we name this process a cylindrical projection transformation. In this study, the circular image presented in Fig. 3(a) is called a 'fisheye image' and the transformed image in Fig. 3(c) is called a 'projected image'.
The radius of the fisheye image used in this study is about 1150 pixels. The hatched area in Fig. 3(a), which is located within 500 pixels of the image center, is neglected in the transformation owing to its low resolution. The latitudinal coordinate of a projected image is determined by multiplying the angular distance (rad) from the reference line AB in Fig. 3(a) by 1000. Thus, the width of the projected image is 6280 pixels. The longitudinal coordinate of the projected image is the distance from the center of the fisheye image. The height of the projected image is 650 pixels, which can be determined by subtracting the hatched area (500 pixels) from the radius of the fisheye image (1150 pixels). Thus, the fisheye image can be transformed to the projected image using the following equations: where x and y represent the coordinates of the fisheye image, r is the distance from the fisheye image center, and d and θ are the polar coordinates of the projected image.

Types of distortion in a projected image
To analyze the distortions in the fisheye and projected images, we take an experimental image, as shown in Fig. 5(a). A surface with rectangular patterns of the same size is fabricated as a test field for modeling the distortions. Figure 5(b) is a magnification of the box in Fig. 5(c), where the following distortions are presented. The numbers refer to the marking numbers inserted in Fig. 5(b). If a mosaic image is produced using projected images that include these distortions, it cannot maintain linear continuity and represents an uneven change in scale. Floor line distortion: This is represented by line A in Fig. 5(a) and lies in the same direction as that from which the image is taken. Although such a floor boundary has to be represented by a vertical straight line in the projected image, it is a tilted curve in Fig. 5(b). Horizontal line distortion: This is actually a straight line but is distorted into a parabolic shape in the fisheye image [ Fig. 5(a)]. This parabolic shape is distorted again in the transformation process for the projected image. Longitudinal scale distortion: The lattice appears increasingly flat as it approaches the lower section of the image, although its size actually remains the same. That is, the scale of the longitudinal direction becomes smaller as it approaches the lower section of the projected image.

Floor line correction
If the longitudinal direction in a tunnel agrees precisely with the direction of taking images, the center of the fisheye image and the vanishing point will be the same. However, in a practical image, this is not the case owing to small errors in the image capture process. The distortion of the floor line presented in Fig. 5(b) is caused by such disagreement. We use a vanishing point in the image for floor line correction. the lines directed to points away from the center of the fisheye image (PB) represent changes in θ according to their distance from the center. Point A on line PB will give a decrease in θ as it approaches B from P, i.e., a decrease in distance to the center, d. Thus, as the left section, 0 ≤ θ < π, in the projected image is considered as a reference, we obtain a tilted curve shaped like " / ", which represents a decrease in θ as it approaches the lower section. To derive the correction equation, we use the sine rule on ∆OAB: If α is small enough, we can use sinα sinα α ≅ α. Our analysis shows that the difference between sin α and α results in errors less than 1 pixel in the resulting image when α ≤ 10°. Subsequently, e d can be expressed as By applying Eq. (4) to the projected image, we obtain the correction effect shown in Fig. 7.

Horizontal line correction
Section A in Fig. 8 shows a curving deformation in the horizontal lines. The lines straighten as they approach the center of the image. Figure 9 shows the coordinates of the curved horizontal lines in the image at specific intervals. Each axis represents the distance in the xand y-directions from the center of the image. Line (1) corresponds to the horizontal line at A in Fig. 8. As we move closer to the center of the image, the line number increases. As mentioned above, the horizontal lines distorted in the fisheye image are deformed again in the cylindrical projection process. In the projected image, in contrast to the fisheye image, the curvature of the horizontal lines decreases as they become more distant from the center of the image. This is because the distortions in the fisheye image and the deformations generated in the projection process partly offset each other.
Let us estimate a trace from point P on horizontal line, such as line A in Fig. 8. In Fig. 10, a dotted circle with the same center as the fisheye image is presented as a horizontal straight line in the projected image, because all the points on the circle have the same distance from the image center. In the case of P on a horizontal line, however, the distance from the image center increases with increasing angular distance from the initial position P 0 . The increments are e h1 and e h2 . As the latitudinal and longitudinal axes in a projected image represent the angular distance θ and the center distance d, respectively, the horizontal line is transformed to one that is curved in an upward direction, as shown in Fig. 10. Because e h1 and e h2 increase as the curvature of the horizontal line decreases, the curvature of the line becomes more pronounced after its projection. By calculating e h1 and e h2 , we can correct the horizontal lines to straight lines in the projected image.     It is assumed that the function of the horizontal lines of a fisheye image can be expressed by a quadratic polynomial. In an orthogonal coordinate system whose origin is the center of a fisheye image, the equation of the horizontal line is given by y = a i x 2 + r i , in which a i can be obtained using the method of least squares. In Fig. 11, i is the horizontal line number and j is the serial number of the points that move on i. As P j (x j , y j ) satisfies y j = a i x j 2 + r i , tanθ j = x j / y j and 2 2 , tan tan 0 tan Using the quadratic formula, we can express x j as Thus, the distance from the center to P j on a horizontal line can be calculated as The correction can be obtained as Figure 12 shows the result of horizontal line correction in the projected image. At the left and right edges of the projected image, the correction amount e h is 0. This becomes larger as it approaches the center, leading to the curved horizontal lines being corrected to almost straight lines.
Although tunnels generally have similar cross sections, consisting of a softly curved face and a flat floor, the cross section of each tunnel varies slightly. In Fig. 12, both corrected parts correspond to the tunnel floor, and the space between them corresponds to the tunnel wall. Because the correction of this space must be designed differently according to the precise cross section of the tunnel, it is not discussed in this study. However, if a tunnel has a regular cross section, its correction can be implemented similarly to the previously applied method, by deriving a proper a i that determines the curvature of the horizontal lines. Section A in Fig. 12 is the part lost in the correction process. In addition, section B requires additional information after correction. As projected images have been corrected independently, there is no information about section B. Thus, the values of the uppermost pixels in the corresponding columns before correction are used. Enlarged images before and after correction are presented in Fig. 13.

Longitudinal scale correction
Objects towards the center of a fisheye tunnel image are more distant, and their scale is also decreased. Figure 14 shows a tunnel model. Although the actual areas of cross sections A and B are the same, cross section B has a smaller area than cross section A in the image. Figure 15 illustrates this tunnel image taken using a fisheye lens and transformed to a projected image. The objects on a cross section, which is perpendicular to the longitudinal direction (the xy plane in Fig. 14), are unfolded on the basis of the edge of the fisheye image (the upper side in a projected image), which has the largest scale. Thus, the x-y scale is unified, even though its precision is decreased. That is, although curve B is shorter than curve A in Fig. 14, the lengths of curves A and B are the same after the projection transformation, as shown in Fig. 15. However, differences in longitudinal scale remain unchanged in the projection transformation,   and the projected image shows more compressed shapes in its lower part. Although W A and W B have the same real length, W B has a smaller length than W A in the projected image.
To investigate such changes in longitudinal scale, we can measure the intervals between lattices in the fisheye image by varying the height from which images are taken, as shown in Fig. 16. The results of measuring these intervals in the lower center of the image, which has the smallest distortion, are presented in Fig. 17.
Although there are minor errors, the rate at which the lattice scales decrease according to the distance from the center of the image is generally linear. In addition, it is determined that the height of the lattice measured at the edge of the image, which has a center distance of 1150 pixels (y 1150 ), is about three times larger than that with a center distance of 500 pixels (y 500 ). That is, the longitudinal scale of the projected image is three times larger at the upper side than at the lower side.
Let us assume that the height of a pixel in the projected image prior to correction is 1. Although the height of the upper image pixel is maintained at 1 after correction, the height of the lower image pixel must be increased by a factor of n. As mentioned above, the value of n can be regarded as 3. As the height of the pixel is integrated along the y-axis of the projected image, it becomes the height of the resulting image. Thus, the areas of rectangle EBCD and rectangle ABCD in Fig. 18 represent the heights of the projected image before and after correction, respectively. Therefore, the height of the image after correction can be calculated as d max (n + 1)/2. Now, let us derive an equation for AD. As the gradient of AD is −(n − 1)/d max , the equation of AD is as follows: Hence, the height of the pixel that has a y-coordinate of d after the longitudinal scale correction is determined using Eq. (11). Figure 19 shows a process that extends the height of the projected image by a factor of 2. If the entire height of the projected image is expanded like this, the boundaries at which the multiplier turns from one to two or from two to three are clearly identified, and these result in the present uneven scales in the longitudinal direction, as shown in Fig. 20.   To improve this, the height of the projected image is divided into seven different sections, as shown in Fig. 21, and the correction rates in each section are independently calculated. Clearly, the summed height after applying the correction for each section is a factor of 2 larger than the height before correction. The result of this method is presented in Fig. 22. This method exhibits a much higher performance than the above method.

Results
As mentioned above, some of the image information is lost during the horizontal line correction for the tunnel floor. The rate of loss will increase with the road width. The major reason for taking tunnel interior images is to obtain information on the tunnel walls. Thus, it is reasonable to omit the correction of floor faces, i.e., horizontal line correction, rather than accept the loss of information in order to correct distortions on the floor. Figure 23 shows projected images of tunnel A, and Fig. 24 shows a mosaic image produced by correcting (including the horizontal line correction) the projected images. The diameter of the tunnel image is about 2000 pixels and that of tunnel A is 6 m. The finest pixel size within the object is 3 mm. Figure  25 shows fisheye images of tunnel B, and Fig. 26 shows a mosaic image produced using the fisheye images without the horizontal line correction.

Conclusions
In this paper, we presented a method of obtaining tunnel interior information using fisheye images. Our approach transforms a fisheye image to a projected image while analyzing and correcting distortions that appear in the projected image according to their type. Although the method proposed in this paper has limitations in terms of completely avoiding distortions and achieving a high accuracy, it has some considerable advantages in terms of very short image acquisition time on site and a relatively inexpensive solution. Our mosaic images of tunnel can be used for surface inspection to detect problems such as large cracks, efflorescence caused by water leakage, and exfoliation. In particular, for structures with almost circular cross sections, the cylindrical projection transformation can be applied very efficiently. More realistic tunnel images will be obtained if studies on lighting, the optimal image-taking position, and the correction equation for horizontal lines according to tunnel cross sections are conducted. In addition, we expect that this study will aid related work using fisheye lens cameras.