Barcode Recognition for Multistrip Lateral Flow Assay Reader

In this paper, we present a method of recognizing the barcodes of individual strips using a single camera in a quantitative fluorescence analysis system that can process multiple lateral-flow strips simultaneously. The proposed method eliminates the need for multiple scanners or a scanning mechanism. The presence or absence of strips in each slot was determined, and the position of the start bit was detected by template matching to find the barcode area. After the start bit was located, the rest of the bit area was found using the barcode dimensions and the estimated image spatial resolution obtained while calibrating the instrument. The value of each bit was determined using the average along the central line with 60% of the width of the bit area. Sixteen cartridge images were obtained under various illuminations to statistically evaluate the thresholds for the investigation of various parameters, such as the existence of the strip, the successful detection of the start bit position, and each bit value. The threshold values were determined using the two-class mixed Gaussian model to minimize errors. Experimental results show that the error rates for all parameters were negligible.


Introduction
Lateral-flow strips are commonly used in field inspection and widely favored by both users and regulatory authorities owing to their low cost and simplicity. (1)(2)(3)(4)(5) This test is applied not only in clinical diagnosis but also in various fields, such as environmental, nutritional, and veterinary medical studies. Fluorescence optical detection is preferred for the quantitative inspection of strips, and methods to detect multiple strips simultaneously have been actively studied recently to increase the processing speed. (1,(4)(5)(6)(7) The quantification device requires the ability to read information such as the analyte or calibration curve of individual strips or patient information, which are generally encoded as a barcode on the strip. (1,(4)(5)(6)8) Although the optical device that reads strips can be used for barcode scanning, a barcode scanner is usually installed separately owing to differences in optical characteristics or scanning resolution. (9,10) In any case, the determination of a barcode area is essential for the equipment to recognize the barcodes for analysis. (5,(11)(12)(13)(14) When a common laser-scanning method is employed as the barcode scanner, a barcode area can be located by attaching scanners to each slot into which strips are mounted. However, this requires multiple scanning devices, resulting in the increased size and cost of the overall equipment. To overcome such limitations, the barcode can be scanned by moving a single scanner fixed on the main liquid-handling header. In this case, the fluctuation of the barcode position is subject to device-to-device deviation and requires a quality assurance process for precise control. These limitations of scanners can be easily addressed by applying a high-resolution smartphone camera, which allows the recognition of multiple barcodes captured in one image. Owing to the recent development of smartphone cameras, it is easy to obtain high-performance, high-resolution cameras that provide various standard interfaces such as mobile industry processor interface (MIPI) and USB, allowing the implementation of smartphone cameras in various applications including biomedical applications. (15)(16)(17)(18)(19)(20)(21) However, although there are many examples of barcode detection using a camera in logistics systems or outdoor environments, (11)(12)(13)22,23) it is difficult to find an example where a smartphone camera is physically integrated in bio-instruments. (7,9,10,24) In this paper, we present a method to recognize multiple barcodes within a single image acquired using a low-cost smartphone camera. The proposed algorithm finds the location of the start bit of the barcode (BSB) using the template-matching method after determining the presence or absence of the strip. As the BSB position is determined from the peak position of the template-matching response, a template that can provide a more stable peak position is investigated. Deviations between the camera and the main processing head and between the processing head and the strip are inevitable. Therefore, the maximum deviation is controlled by a calibration process and a quality assurance process for the relative distance between the main processing head, strip, and camera. The proposed algorithm is designed to be insensitive to this device-to-device variation. The decision thresholds for the presence of a strip or the success of BSB recognition were statistically evaluated taking into account this maximum deviation. Once the position of the BSB in each strip is determined, the remaining bit positions are obtained from the barcode dimensions and the image spatial resolution obtained during the manufacturing calibration procedure.
The feasibility of the proposed algorithm was verified by analyzing a total of 16 images containing 40 strips, which were prepared under different external and internal lighting conditions. The experimental results showed the successful detection of the BSB and the bit values without failure or error. Figure 1 shows the process of recognizing barcodes before starting the strip analysis. This process starts from the first slot of the captured tray image. The existence of the strip is first verified, and upon verification, the BSB is searched for. If the BSB search fails even though a strip exists, an error report is provided immediately to the user and processing stops. Following the BSB search, it is determined whether each barcode bit value is white or black for the remaining bits, and the next slot is examined. Strip analysis begins after all slots have been examined. In this paper, we propose a robust method for the strip presence detection, BSB search, and bit value decision for the barcode recognition process.

Preparation of experimental images
To statistically evaluate the parameters for the proposed algorithm, images with strips in even and odd slots were prepared (Fig. 2). Images were acquired under various conditions to investigate whether the proposed algorithm is sensitive to external lighting or variations among experiments. To reflect the effect of external lighting variations on the strip image, even and odd image pairs were acquired with and without indoor fluorescent light. This was done twice a day, once in the bright afternoon and once at night. The daily shooting process was repeated for two days as there may be daily fluctuations in the outside or inside lighting. Strips were collected and reinstalled for each image to allow for variations in barcode position that could occur between images. A total of 16 images with 80 slots, among which 40 have strips, were acquired. Figure 3 shows an even image with strips in even slots, in which the BSB initial positions and strip existence detection regions are marked with green crosses and orange boxes, respectively. In this paper, the limit of the device-to-device deviation was assumed to be controlled within 1 mm, as given in the manufacturing quality assurance process. Therefore, the position of the strip could be located 1 mm up, down, left, or right from the predefined position. The location of the orange box, which determines the presence or absence of strips, was manually selected considering the device deviation. The error that might occur due to manual selection can be neglected given the large difference in the widths of the strip and the detection region (orange squares), which are 15.6 and 2 mm, respectively. Since the black anodized tray is clearly distinguishable from the white matte paint strip, the decision boundary of the presence or absence of the strip was also not critical in detecting the existence using the average brightness of the detection region.

Determination of strip existence
To determine the decision boundary, it is assumed that the average brightnesses over the decision box region when a strip is present or absent both have Gaussian distributions. The mean and variance of each distribution are (m 1 , σ 1 ) and (m 2 , σ 2 ), respectively, and the classification error is as follows: where P e and t denote the classification error and decision boundary, respectively. The notation G(m 1 , σ 1 ) is the Gaussian distribution of the average brightness when a strip is present and G(m 2 , σ 2 ) is that when a strip is absent. If the two classes are sufficiently separated and can be easily distinguished, the mean and variance of the Gaussian distribution can be easily calculated from the sample mean and variance, respectively. In this paper, the decision boundary t was searched for to minimize the classification error P e .   Figure 4 shows two templates that can match the BSB pattern. The black rectangle is the object area with the same size as the BSB, and the brightness value in the area is 0. The white background has the same size as the BSB area, and the brightness value is designated as 1. The left template T1 in Fig. 3 will give a peak response when the center is aligned with the BSB, and the right template T2 will show a peak at the upper center of the BSB.

Locating BSB with template matching
If cross-correlation is chosen as the matching measure, the template is moved pixel-bypixel, multiplied by the image, and then summed to obtain the template response. However, since the brightness values in the object and the background areas of the template are 0 and 1, respectively, the response can be calculated by subtracting the averages of the image intensities over the object and the entire area as shown in the following equation: where r(·), T(·), I(·), and mean(·) represent the template response, template, image, and mean functions, respectively. B(x, y) and O(x, y) are the entire template area and the object area centered at the image location (x, y), respectively. Since all the areas of interest are rectangular, the moving average algorithm can be used to calculate the area average rapidly. In practice, area averaging is achieved by a function called an average filter that is based on such an algorithm generally implemented in the image processing library provided by most programming languages.
Since the template is a black rectangle, which is the same shape as the black bit, dominant peaks occur at all of the black bit locations, and the peak at the highest position will represent the BSB. Note that when template T1 is used, peak responses occur in the center of the black bits, whereas template T2 shows the peak response at the top. From these two templates, a template capable of stably obtaining the BSB peak was selected through a qualitative analysis of the experimental results.
To determine the threshold of whether the BSB search was successful, the peak outside the barcode was statistically compared with the BSB peak. Similar to determining the presence of the strip, it was assumed that the search window from the initial BSB could be misplaced by the deviation limit in 8-neighbor directions. Figure 5 shows examples where the search window is (a) and is not (b-d) at the BSB. In the figure, the red cross and red box indicate the centers of the search window and the search area, respectively. The yellow cross represents the BSB location. By assuming that both the BSB and external peaks have a Gaussian distribution, a threshold value that minimizes the classification error was calculated, similarly to in the determination of the strip existence. In this paper, the limit of deviation between devices is assumed to be 1 mm, in accordance with the manufacturing quality assurance process. For individual strips of each image, nine template-matching response images were obtained by searching for the first BSB location and eight neighbors, 1 mm horizontally or vertically from that location. Since there are nine template responses for one strip and 40 strips in all 16 images, a total of 360 template-matching responses were obtained and statistically evaluated to obtain the decision threshold for a successful BSB search.

Determination of bit positions and values of whole barcode
After the position of the BSB was located, the positions of the remaining bits were determined using the barcode dimension data and the spatial resolution obtained in the calibration process of the device. The blue rectangles in Fig. 6 indicate the locations of bits, where the sky-blue circles denote the BSB position. The value of each bit was determined by thresholding the average value of the pixels along the red line (Fig. 6). The length of the red line is 60% of the bit width and is located at the vertical center of each rectangle. The line area for the average brightness is set as such to obtain a stable bit value even when the original bit image deviates from the calculated bit area. The threshold was calculated to minimize the classification error when the average values for the white and black bits both have Gaussian distributions as in the previous subsections. Figure 7 shows the distributions of brightness with and without strips in each slot. The x-axis denotes the average brightness over the investigated region, and the blue and green dots indicate the brightness with and without the strip, respectively. It can be seen that their brightness distributions are clearly distinguishable. Assuming that both distributions are Gaussian, the red triangle depicts the minimum classification error threshold. Owing to the negligible error, determining the presence or absence of the strip was trivial. This is from the high contrast between the black anodized tray and the matte white strip. Figure 8 shows the template responses and center profiles when applying templates T1 (a) and T2 (b). There are two response images for each template. The left images show the response when the first bit is white, and the right images show the response when it is black. Each response image includes the original image, template response, and the response profile along the vertical center. As can be seen from the response center profile, the peak spacing is the same as the bit height, and the peaks are relatively close to each other when the first bit is black (Fig. 8(a), right). Narrow peak spacing is not desirable when finding the desired peak. Template T1 also delivers relatively large peaks at the top edge of the strip. In the case of incorrect installation due to user error, the edge may enter the search area and be mistaken for the BSB peak. On the other hand, when using template T2, the peak due to incorrect loading can be neglected given that the response value is negative [ Fig. 8(b)]. Moreover, the distance between the peaks are wider, and the closest gap will remain twice the bit height even when the first bit is black [ Fig. 8(b), left]. Therefore, the remaining experiments were performed with template T2. Figure 9 shows examples of false peaks (red marks) that can be mistaken for the BSB peak (blue mark). Figure 10 illustrates the decision boundary obtained by assuming that the responses at    the BSB positions and at the false peaks have Gaussian distributions. The threshold was 63 and the error rate was 3.4 × 10 −6 , which is negligible. This shows that the proposed BSB search algorithm is insensitive to the external lighting, experimental variation, and device-to-device deviation. Figure 11 shows the location of each bit (blue boxes) calculated from the spatial resolution and barcode dimensions, and a 60%-wide middle line area (red lines) to determine the bit value. The numbers to the left of the barcodes indicate the decided bit values. The bit value was successfully determined for all 40 barcodes. Figure 12 shows the minimum error threshold (red triangle) when the average brightness of the black bits (blue dots) and white bits (green dots) both had Gaussian distributions. The error rate was 2.1 × 10 −13 , which is also negligible. This indicates that the barcode decision is accurate and insensitive to the various deviations.

Conclusions
In this paper, we propose a barcode recognition method for a multistrip lateral flow assay reader using a camera that does not require multiple barcode readers or reader scanning mechanisms. Threshold values for determining the presence or absence of a strip, the position of a start code, each bit position, and each bit value were statistically evaluated using a twoclass mixed Gaussian model. The proposed algorithm determines the presence or absence of the strip and the position of the start code considering the device-to-device deviation. The position of each bit is determined using the barcode dimension and the image spatial resolution obtained during the manufacturing calibration process. Experimental results show that all of the decision errors were negligible. The decision criterion in this study was decided by emulating the device-to-device deviation with only one device. Therefore, the decision criterion should be reinvestigated through the in-depth experimentation of several devices in order to employ this method in the field. In addition, further statistical analysis of lighting variations among devices is also required.