Biomimetic Multiaperture Imaging Systems: A Review

1Department of Electrical and Computer Engineering, College of Engineering, Seoul National University, 1 Gwanak-ro, Gwanak-gu, Seoul, Republic of Korea, 151-742 2Department of Electronics Engineering, College of Engineering, Chungnam National University, Gung-dong, Yuseong-gu, Daejeon, Republic of Korea, 305-764 3Seoul National University Hospital Biomedical Research Institute, 101 Daehak-ro, Jongno-gu, Seoul, Republic of Korea, 110-744 4Department of Biomedical Engineering, College of Engineering, University of Ulsan, 93 Darehak-ro, Nam-gu, Ulsan, Republic of Korea, 680-749


Introduction
Biomimetics is about making and analyzing systems that emulate the structure of natural organisms (1)(2)(3)(4) and functions, or behaviors that occur in nature. (5)(6)(7) Currently, most of today's vision systems are inspired by vertebrate's vision system that has a single aperture and an array of photoreceptors. Lenses consisting of a single-aperture vision system accomplished various functionalities such as aberration reduction, wide angle, and telescopic capabilities through the adding of more lenses. However, a single-aperture vision system has some fundamental limitations, such as the limitation of miniaturization and low time resolution. (8) Owing to recent brilliant progress in microelectronics, the miniaturization technique has been improved, so that photodetector arrays, signal amplification, and low power consumption can be integrated in a few square millimeters of area.
Although the progress in microelectronics is fast enough to miniaturize the size of a camera in the millimeter scale, optics has not yet been developed to meet today's demands for the miniaturization of cameras and wide-angle lenses. (9) In optics, fieldof-view (FOV) and focal length are inversely related, which means that increasing the thickness of a wide-angle lens system is inevitable in a single-aperture vision system. To avoid the relationship between focal length and FOV, mimicking an insect eye vision system that has both a wide angle view and a very small size is needed. The insect eye vision system captures light with spherically aligned small microlenses. (10,11) These spherically aligned microlenses form a compound eye. The insect eye vision system is a practical vision system with a large FOV and a miniaturized volume as well as low energy consumption. (12) Considering the light pathway, the compound eyes of insects are generally divided into two types of structure: apposition and superposition compound eyes. In the apposition compound eye, each microlens focuses light from an object onto a single photoreceptor. The lens and photoreceptor one-to-one system is called an ommatidium. The apposition compound eye generally has hundreds to tens of thousands of ommatidia, (13) and each photoreceptor has an inverted image of the object. In contrast, the superposition compound eye forms a single erect image by superposing the light from multiple lenses on the surface of a photoreceptor layer. Nature provides two optical methods to superpose the images. One is a refractive method using gradient refractive index (GRIN) lenses. The other is a reflective method using a micromirror array. (15) Another interesting approach is based on the neural superposition eye of the housefly, which is actually a type of apposition eye. (16) A single microlens of the neural superposition eye focuses light on seven photoreceptors. Signals from the different adjacent ommatidia are connected to each other and superposed on the next neural layer. The superposition characteristics of the neural structure enables the fly to be more sensitive to light and detect motion more accurately. (14,17,18) In this study, we investigate miniaturized artificial imaging systems mimicking the apposition and superposition eyes of insects using three major focusing elements: microlens, micromirror, and GRIN lens arrays. Practical principles and fabrication methods for these systems are introduced with the reported examples.

Artificial Compound Eye Using Microlens Array
As mentioned in the introduction, compound eyes are generally divided into two types of structure: apposition and superposition compound eyes. Imaging systems using the microlens array can also be divided into apposition and superposition types. In this section, we review these two types of imaging system that use a microlens array.

Artificial apposition compound eye using microlens array
The artificial apposition compound eye model does not supplant a conventional imaging system. It has an advantage in perceiving motion compared with the conventional single-aperture vision system. To derive benefits from the apposition compound eye model, Davis et al. developed a simple vision system inspired by an artificial apposition compound eye. (42) This artificial apposition compound eye system consisted of seven photodiodes and analog and digital processing stages combined on a ground-based vehicle as shown in Fig. 1. The seven front photodiodes covered 60° of FOV. The sensor output was passed through a notch filter to eliminate 120 Hz noise and a low-pass filter to remove high-frequency noise in the analog processing stage. An additional ambient light sensor detected overall brightness to generate a corresponding offset voltage as bias for the filtering stage. Two different bias levels of the ambient light sensors were calculated to provide a wide dynamic range. After the sensor information processing stage, the sensor signals converted into a tristate (light-shaded obstacles, dark-shaded target, and ambient light) signal via a robust object classification algorithm. In the digital processing stage, the direction of vehicle steering is chosen according to what was in front of the vehicle. This test setup mimics an insect with an apposition compound eye avoiding static obstacles while tracking a mate or food that may be moving.
Floreano et al. developed a miniaturized curved artificial compound eye referred to as CurvACE. (43) It consisted of three compositionally and functionally different layers: an optical layer, a photodetector layer, and an interconnection layer. The optical layer was composed of an array of 42 × 15 highly transparent polymer microlenses formed by reflow of a photoresist on the glass substrate. It was stacked on the photodetector layer and focused the light onto the 42 × 15 photodetectors fabricated in a silicon wafer using CMOS technology. The stacked layer was diced into 42 columns to add flexibility. The interconnection layer formed using a flexible polyimide printed circuit board (PCB) transfered the output signals from the individual ommatidium to the processing units. The fabricated CurvACE prototype had a FOV of 180 × 60° and a signal acquisition rate of 300 Hz. Moreover, a modified neuromorphic circuit, which is originally proposed by Delbruck and Mead, (44) was used to handle a wide dynamic range. Testing its motion detection capability using a modified Lucas-Kanade method (45,46)  showed that ego-motion can be estimated from flow fields.
Another interesting system of the artificial apposition compound eye was developed by Song et al. (47) This system can be divided into two main subsystems. The first  (42) subsystem was a 16 × 16 polydimethylsiloxane (PDMS) microlens array. It provided an optical focusing capability on the second subsystem consisting of 16 × 16 photodiodes and readout electrodes. The aligned bonding of these two subsystems yielded a planar integrated imaging system. The bonded planar layout was transformed into a full hemispherical shape with a radius of 6.96 mm by a hydraulic actuation system. The bonded hemispherical shape was supported by bulk black silicone. A perforated sheet of black elastomer serving as the screening pigment was placed on the microlens array to prevent light crosstalk between the photodiodes. The complete apposition camera system had a FOV of 160 × 160° and a large depth of focus capability due to the very short focal length of each microlens. (48) Each ommatidium of this imaging system contributed to a different region of the resultant image as a single pixel. To improve the effective resolution, active motion scanning was carried out from -5.5 to 5.5° in the vertical and horizontal directions in 1.1° steps. This active motion scanning may have blurred the resultant image. However, the number of effective image points was increased by about one hundred fold.

Artificial neural superposition compound eye using microlens array
Compared with the apposition compound eye, the neural superposition compound eye is more sensitive to light. High sensitivities to light and motion along with the wide angle viewing capability of the natural compound eye are useful in robot vision. Riley et al. developed a biomimetic motion detector based on the eye of Musca domestica, the common housefly. (49) A single ommatidium of M. domestica contains a pack of eight photoreceptors. Signals from these eight photoreceptors show near-Gaussian shapes. (50) Adjacent ommatidia of the compound eye can be superposed in the next neural layer. This resulted in an overlapping Gaussian-shaped photoreceptor response that has benefits in detecting motion with hyperacuity. (51)(52)(53) Motion detection capabilities using a biomimetic motion detector and a conventional charge-coupled device (CCD) camera sensor were calculated and compared. A biomimetic motion detector can detect a small moving object better than the CCD camera sensor at high speeds and low contrasts. (54) The results of the comparison are shown in Fig. 3.
Another application mimicking a neural superposition compound eye was developed by Bruckner et al. (12) The device was an ultrathin imaging system consisting of a planar microlens array on top of a glass substrate and a sensor array placed in the focal plane of the microlenses. The pitch difference between microlenses and sensors enabled different viewing directions, such that a larger FOV (23 × 21) could be achieved. Each pixel of the resultant image was digitally averaged over nine pixels with a common viewing direction. Sensitivity was improved by increasing the signal-to-noise ratio (SNR) up to 6 dB. This experiment proves that the neural superposition type of system has a higher sensitivity to light than the apposition type. The integrated camera system is only 450 um thick, which makes it useful in a compact vision system.
The thin observation module by bound optics (TOMBO) developed by Tanida et al. is also a compact vision system that mimics a neural superposition compound eye. (55) This system was composed of multiple sets of elemental optics, which consist of microlenses and opaque walls to prevent crosstalk and a conventional CCD chip with 739 × 575 pixels resolution. The image captured by this system was a set of multiple images of the object. The resultant erect image was retrieved by sampling multiple images using postdigital processing. However, the FOV of this system was small at about 35°( 56) because of the limitations of the planar substrate. A curved image sensor is needed to overcome the small FOV of this system. (57) Fig. 3. Comparison between the conventional CCD sensor and the biomimetic motion detector. The biomimetic motion detector shows a higher performance than a CCD sensor (a) when detecting a small object and (b) at low contrasts. (54) (a) (b) Instead of developing a curved image sensor, Afshari et al. developed a panoptic camera using a hundred classical CMOS image sensors arranged hemispherically. (58) Although this bioinspired vision system with a 13 cm diameter was very large compared with the other artificial compound eye, it had superior functionalities, such as a larger FOV in comparison with the conventional camera and a higher resolution with 1024 × 256 pixels in comparison with the natural compound eye. Another application having superior functionalities instead of having a small system size was developed by Brady et al. (59) They introduced the AWARE-2 camera with 50 gigapixels, including 98 microcameras, each with a 14 megapixel sensor. This multiaperture system was less challenging to fabricate than a single-aperture system with a resolving power of 10 9 . (60)

Artificial optical superposition compound eye using microlens array
A major characteristic of the optical superposition compound eye is a clear zone between the crystalline cones and the retina. (61) The clear zone in the optical superposition compound eye is used as space for the fusion of light bundles, so that the optical superposition compound eye apparently has a higher sensitivity than the apposition compound eye.
A basic scheme of the artificial optical superposition compound eye was the Gabor superlens, which was described by Gabor in 1940. (62) The Gabor superlens consists of two planar microlens arrays and four aperture layers, two for the microlens arrays and two for the space between the microlens arrays to suppress the crosstalk. Recently, the fabricated Gabor superlens has been used to weld a very thin imaging device with a total track length of 2 mm, a FOV of 30°, and a resolution of 156 × 156 pixels. (63) It is thin enough to apply in a compact vision system, but its resolution is very low for practical use. For higher resolutions, an optical cluster eye with four microlens arrays has been fabricated by Meyer et al. (64) The resolution of this optical cluster eye is 640 × 480 pixels, and the total track length of this system is 1.86 mm, which is smaller than that of the Gabor superlens. Table 1 shows the artificial compound eyes using the microlens array discussed above.  Table 1 Applications of microlens array.

Fabrication of microlens array
After Hooke produced microscope objectives by studying the effect of melting the ends of Venetian glass rods in the seventeenth century, many methods have been developed for the production of microlens arrays. (19) The fabrication method most commonly used today is thermal reflow, suggested by Popovic et al. in 1988. (20) Some of the methods used to produce microlens arrays are discussed in more detail in the following section.

Thermal reflow method
The simplest and best-known fabrication method for microlens arrays is thermal reflow. In this method, a photoresist is placed where the microlenses will be located by mask lithography. Any photoresists that do not crosslink, such as AZ1512, can be used for reflow. (21) A high thermal energy can turn a photoresist into a liquid state and the surface of a liquid photoresist on the substrate is formed into a spherical shape to minimize the liquid surface. (22) An illustration of the thermal reflow process is shown in Fig. 4.
The desired focal length, radius, and thickness of the photoresist must be considered according to the lens-maker's formula. (24) During fabrication, outgassing the photoresist, impurities in the photoresist, and any surface energy change resulting from evaporation of the photoresist must be considered to fabricate perfectly spherical microlens arrays. (19,23) After the photoresist microlens array is fabricated, a transparent polymer microlens array is produced by replica molding. Some researchers have been actively seeking for an innovative replica molding process to fabricate an omnidirectionally aligned microlens array. (25,26) Cherng and Su made a curved mold for a concave microlens array by pressing a planar microlens array with a spherical object. (25) With this curved mold, microlens arrays on a curved shell were fabricated.

Wet etching method
The isotropic etching behavior in wet etching may be utilized to produce microlens arrays. The first step in the process involves forming ablation-induced craters with diameters of a few micrometers using 30 fs and 800 nm laser pulses at a repetition rate of 1 kHz. (27) After the craters are fabricated, the concave microlens array may be fabricated by treatment with hydrofluoric (HF) acid solution. From the concave microlens array, a transparent polymer microlens array can be molded. A thermo-mechanical process, such as pressing the microlens array with heated spherical glass, is utilized to fabricate the microlens array on a spherical surface. (28)

Other methods
Several other methods have been developed to fabricate a microlens array. Kim and Yun used water droplets to mold a PDMS concave microlens array. (29) They modified the surface energy of a substrate to change the contact angle of a water droplet so that the focusing power of the microlens could be changed. Lee et al. fabricated polarization-dependent and polarization-independent microlens arrays using the electrohydrodynamic instability of an organic layer. (30) A reactive mesogen (RM) solution was coated on a circular electrode. A nonpatterned electrode beneath the circular electrode formed a pillar array where an electric field was applied between the two electrodes. Removing the top patterned electrode resulted in a microlens array on the nonpatterned electrode. In addition, E-beam lithography, (31) ink-jet fabrication, (32,33) a photothermal technique using photosensitive glass, (34) optically induced volume changes in recording materials, (35)(36)(37) hot embossing, (38) laser lithographic fabrication, (39) soft lithography, (40) and a hybrid sol-gel method (41) have all been used to fabricate microlens arrays.

Artificial Compound Eye Using Micro-mirror Array
A reflecting superposition compound eye is commonly found in lobsters, crayfish, and other long-body decapod crustaceans in nature. (65) This vision system has a high sensitivity to light and a wide-angle FOV similarly to other superposition compound eye systems. The fabrication of artificial reflecting superposition compound eyes has been introduced by Huang et al. (66) They used a p-type SOI wafer with (100) orientation as a substrate. Silicon micro-square tubes were patterned on the substrate using photolithography, reactive ion etching (RIE), and inductively coupled plasma deep RIE. Aluminum was sputtered onto the patterned micro-square tubes to obtain reflective surfaces. The fabricated planar micro-mirror array was transferred to a flexible, transparent, and hemispherical PDMS membrane.
The major advantage of the reflecting superposition optical system is minimum chromatic aberration. This characteristic is very useful when dealing with UV light or a higher frequency light, which is difficult to refract with a lens. Therefore, the focusing capability of a mirror array is applicable to solar power systems, (67) X-ray imaging systems, (68)(69)(70) or neutron focusing lens systems. (71)

Artificial Compound Eye Using a GRIN-Lens Array
The microlens array structures in insects with apposition and superposition compound eyes actually have a minor contribution to focusing at the image plane. (72) On the other hand, the gradient index parts in insects take a major role in focusing at the image plane. The gradient index lens array forms an erect image. The erect image making property of the gradient index lens array is especially useful in forming optical superposition structures found in krill eyes. (73) Hirura et al. mimicked the optical superposition structure using a gradient index lens array. (74) They developed a krill-eye beacon to measure the relative rotation angle between a light source and a camera. The beacon consists of nine gradient index rod lenses with 20° rotation and a mask patterned with a bar-code array representing the relative position of the mask. The mask was observed by an outside camera. The estimated angular position of the camera based on the observed bar-code array was almost the same as the actual angular position of the camera. This result established the reliable scanning property of the beacon.

Conclusions
Throughout this paper, the fabrication of a microlens array and the applications of an artificial compound eye were introduced. The applications were arranged according to the operating principles of the optical components proposed by Exner in 1891. (72) However, because biomimetic vision systems have not been studied in great detail, practical applications are not yet available. The artificial apposition compound eye has offered a large FOV in thin imaging systems, but it has a low resolution. The artificial neural superposition compound eye has a high sensitivity to light; however, an image reconstructed from the neural superposed signals needs a lot of computational power. The optical superposition compound eye is most similar to the conventional optical system, except for the presence of a focusing element array. It generates an erect image without reconstruction processes; however, it is hard to accomplish a practical resolution with the optical superposition compound eye, because the resolution limit and the size of the optical superposition compound eye are inversely related theoretically. (75) Therefore, we think that it is a difficult task to develop an alternate vision system. However, developing an artificial compound eye for special uses, such as motion detection, is necessary for practical applications, such as a robot vision system, because of its low power consumption and wide angle view. Research on neural connections in insect vision systems can provide creative perspectives for applications to vision systems. We hope that this study assists the development of appropriately useful insect vision systems.