Portable Device for Ornamental Shrimp Counting Using Unsupervised Machine Learning

With the rapid development of emerging technologies, intelligent agriculture is incorporating techniques such as the Internet of Things, big data, cloud computing, artificial intelligence, blockchains, and fifth-generation mobile communication to improve work efficiency, prevent various disasters, and change the sales mode of agricultural products. Ornamental fishery is a part of agriculture and accounts for a significant proportion of commercial trade. This paper introduces image processing technology to help ornamental fisheries calculate the number of shrimps quickly. To solve the problem of overlapping live shrimps when counting, K-means unsupervised machine learning is adopted to determine the area of one shrimp. In addition, the proposed method using unsupervised machine learning is able to count different types of shrimp with high accuracy, such as crystal red shrimps, fire red shrimps, and Takashi Amano shrimps. We also analyze two background subtraction techniques, hue/saturation/value (HSV) histogram-based detection and Sobel edge detection, to compare the accuracy and calculation time of this application.


Introduction
The size of the trade in global ornamental fish has steadily increased in recent decades. There were only 28 countries exporting ornamental fish in 1976, compared with more than 125 countries at present. Moreover, the global export market for ornamental fish almost doubled from US$ 177.7 million in 2000 to 337.7 million in 2016. (1,2) The global ornamental fish industry is worth US$ 15 billion, and more than 2 billion live ornamental aquatic animals are traded per year, making it necessary to consider how to deal with large numbers of live ornamental aquatic animals. Until now, the calculation of numbers of live fish or shrimps has mainly been carried out manually, which has many weaknesses including subjectivity, slowness, low accuracy, worker fatigue, and even the injury and death of animals during the calculation. (3) The development of computer vision technology with a tiny camera for image capture for counting shrimps is a strong possibility. In this study, we propose a method of counting ornamental shrimps that solves the above shortcomings of manual counting. The proposed Table 1 Related literature on the application of computer vision for object identification or counting in aquaculture.

Authors
Applications Targets Results Newbury et al., 1995 (6) backpropagation network artificial fish 94% accuracy, overlapping Huang, 2002 (7) HSV background subtraction fluorescent lamp fry error ranging between 0.9 and 32%, non-overlapping Friedland et al., 2005 (8) erosion-dilation filter American shad eggs FN classification error of 1%, nonoverlapping Zion, 2012 (4) shape model fish fry 98% accuracy, overlapping Flores et al., 2008 (9) adaptive threshold and fast Fourier transform (FFT) Peruvian scallop larvae accuracy of 95% in 2 min, overlapping Khantuwan and Khiripet, 2012 (10) adaptive threshold and co-occurrence color of histogram shrimp larvae accuracy of 97% in 1 min, overlapping Coronel et al., 2018 (11) local normalization filter and iterative selection threshold fish fingerlings precision, recall, F measure of 99.80, 97.90, 98.83%, respectively, in < 1 s, non-overlapping method employs a portable smart device consisting of a Raspberry Pi computer attached to a camera sensor and has the expected advantages of avoiding the need to physically touch the shrimps, rapid calculation, low cost, and portability. The remainder of this paper is organized as follows. Section 2 describes related work on aquaculture using computer vision techniques. Section 3 shows the hardware and software elements of the proposed system. Section 4 explains the process flow used with our proposed method. Section 5 reports experimental results obtained with the implemented system. Finally, the article is concluded in Sect. 6.

Related Work
Zion summarized the applications of computer version technology in aquaculture into five categories. His review described the state of the art and focused on needs in all stages of the aquaculture process from hatcheries to harvesting. These applications included counting, size measurement and mass estimation, gender detection and quality inspection, species and stock identification, and the monitoring of welfare and behavior. Optical methods to count objects were the most common approach. The counted objects included eggs, larvae, fry, and fish at various stages of growth. (4) Silvério categorized these computer vision systems in aquaculture. These systems have the following common steps: (5) • Capturing images • Extracting features • Detecting blobs • Counting based on the blobs detected.
Most computer vision applications that perform image recognition require the above four steps. The image processing techniques are mostly used in steps 2 and 3, i.e., extracting features such as boundaries or a given color range and detecting blobs to segment the image. In this paper, we present a state-of-the-art application of computer vision for counting in aquaculture. A comprehensive summary of related studies for object identification or counting in aquaculture from 1995 to 2018 is included in Table 1.
Newbury et al. used artificial fish to simulate the process of fish calculation, especially considering the fact that fish often overlap each other. Then they applied a backpropagation algorithm and achieved 94% accuracy when estimating the number of fish. (6) Huang adopted the value of intensity (V) in the hue/saturation/value (HSV) color space to filter the target image, then performed thresholding for image segmentation, and calculated the area of fluorescent lamp fry based on the obtained foreground. The depth of each fry in an aquarium is different, resulting in fish of the same size having different areas in an image. In addition, the overlapping of fry was not considered, so errors of between 0.9 and 32% were obtained. (7) In 2005, Friedland et al. used the mathematical morphological dilation-erosion algorithm to eliminate touching eggs, and then performed seven measurements to identify American shad eggs and debris in the water, enabling them to count the number of American shad eggs. The false negative (FN) rate in the identification, i.e., eggs judged to be debris, was 1%. (8) Zion et al. developed image processing algorithms using shape models of various species and detected the number of overlapping fish fry with 98% accuracy. (4) Flores et al. applied image processing technology to calculate the number of Peruvian scallop seedlings. The processing flow was as follows.
Step 1, place the samples on a Sedgewick Rafter cell slide under a microscope; step 2, capture an image; step 3, find the region of interest; step 4, filter the image using Laplacian and median filters to enhance the image and as preprocessing for the next step; step 5, use adaptive thresholding for binarization; step 6, divide the targets into A with one seedling and B with two or more overlapping seedlings; step 7, find the area of a scallop seedling through the FFT of the histogram; step 8, calculate the total number of scallop seedlings. The experimental results showed that the calculation can be completed in 2 min per image with 95% accuracy. (9) Khantuwan and Khiripet proposed an image processing method that automatically calculates the number of shrimp larvae using a co-occurrence color histogram template comparison technique. The whole algorithm can be divided into three parts: image enhancement, separation of juvenile shrimps from the background, and classification. The accuracy of automatic counting was experimentally shown to be 97%. (10) Compared with manual counting, the error between automatic counting and manual counting was between −1.5 and 6%, and the average error was 3.25%. They used MATLAB 2010a run on an Intel Core™ i5 laptop, and it took an average of 1 min to analyze an image of 100 juvenile shrimps. Coronel et al. proposed a set of efficient image filtering and segmentation methods that used two image processing technologies, a local normalization filter, and an iterative selection threshold to perform background removal. In terms of counting accuracy, the combination of these two methods obtained an average precision, recall, and F measure of as high as 99.80, 97.90, and 98.83%, respectively. (11) Table 2 shows the related studies on object recognition and counting that apply computer vision technology. Some of the objects included are as small as microbes, colonies, blood cells, rotifers, and Aedes aegypti larvae, as well as field insects, pests, and mice. Barbedo and Fuad et al. found in their review that color space conversion is mainly used in the calculation of these targets. (12,13) Alver et al., Mohamed and Far, Barbedo, and Das et al. used the conversion between the color models of RGB, cyan/magenta/yellow/key, International Commission on Illumination lab, and HSV. Then, they applied filter technologies such as Laplacian and median filters to enhance the image, and they removed the background using an appropriate thresholding technology. (14)(15)(16)(17) Other researchers such as Pistori et al., Lu and Qin, and Qing et al. adopted emerging methods to count and identify target images. The technologies include artificial neural networks, machine learning, and sample comparison techniques. (18)(19)(20)(21)(22)

Materials and Methods
To achieve our goals of cost-effectiveness and portability, the entire proposed system is based on Raspberry Pi with a Raspberry Pi camera for calculation and image acquisition. (24) Raspberry Pi is a tiny computer developed by Raspberry Pi Foundation. The Raspberry Pi Zero W model is mainly dependent on the Broadcom BCM2835 SoC with a core architecture of a 32-bit ARM11 processor and a basic size of 65 × 30 mm 2 . It has an operating frequency of 1 GHz, a memory of 512 MB SDRAM, two USB-2.0 ports, Wi-Fi and Bluetooth support, a mini high-definition multimedia interface port, a camera serial interface connector, and a mobile industry processor with a micro secure digital memory card slot. The Raspberry Pi camera module has an OmniVision OV5647 five-megapixel sensor. The camera module can be used to take highdefinition video as well as still photographs. There are numerous third-party libraries built for it, including the Picamera Python library. Figure 1 shows the entire hardware device. The lower left is the Raspberry Pi camera, and the upper right is the Raspberry Pi Zero W. The two devices are installed above and below the cover,  (20) improve K-means via hidden Markov model insects in stored grain recognition accuracy of 98% with normal pattern and 87% with lateral pattern Mohamed and Far, 2012 (15) Otsu thresholding white blood cell nuclei 80.6% accuracy Barbedo, 2012 (12) median filter and thresholding micro-organisms and colonies correct identification of more than 90% of cases and overall deviation of 8%  (13) color models transformation, thresholding Aedes aegypti larvae review of techniques respectively. As shown in the upper left of Fig. 1, the back of the lid is used to capture the picture, and an LED is used to provide the light source. As shown in the upper middle of Fig. 1, a battery is installed on the front of the lid to power the operation of the whole system. The lower right of Fig. 1 shows the inside of the plastic bucket containing shrimps, and the lower middle of Fig. 1 shows the appearance of the bucket. Table 3 lists all the software and hardware elements. The application installed in the mobile devices provides the user interface and controls the Raspberry Pi to perform the process of shrimp calculation. Therefore, the ionic framework is used to deploy the application in various mobile phones, including Android and iPhone. Raspberry Pi Zero W is the main computing unit, which calls the OpenCV library for image processing and the scikit-learn library for machine learning calculation. The Apache HTTP server allows users to interact with the Raspberry Pi.   Figure 2 shows the flowchart of image processing for the portable automated shrimpcounting system. Each image is subjected to processing techniques such as resizing, grayscale conversion, feature extraction, and thresholding. The program in the Raspberry Pi computes the areas of the extracted contours and stores them in an array. Then, the program determines whether each contour contains noise or shrimps depending on its area. Considering the overlapping of shrimps, it divides whole contours into two clusters. One cluster is for a single shrimp, and the other cluster is for more than one shrimp.

Proposed Method
Considering practical use, we use two different methods to extract features. Sobel edge detection uses a pair of 3 × 3 convolution masks, one to estimate the gradient in the x-direction and the other to estimate the gradient in the y-direction. The Sobel algorithm effectively highlights the shrimps via edge detection. (25) The other method, HSV filtering, can extract a specific range as a feature. (26) Figure 3 shows the results of using Sobel edge detection and HSV filtering after thresholding. In terms of noise and area, the HSV filtering method is clearly better than the Sobel method. However, HSV filtering is impossible to implement for shrimps with multiple colors, such crystal red shrimps.
The K-means clustering algorithm is an unsupervised machine learning algorithm. The aim of this algorithm is to divide S points into K (≤ S) clusters as shown in Eq. (1). The within-cluster sum of squares, ‖x − μ i ‖ 2 , which is the Euclidean distance, is minimized. The cluster center μ i is the mean value of all points x in S i . The K-means clustering method simply divides all the blob contours into two clusters. The cluster with the larger number, oneShrimp, is identified as a shrimp, and the other cluster, moreShrimps, is identified as multiple shrimps. For the oneShrimp cluster, the area of a shrimp is determined by the median value in this cluster. The program calculates the rounding integer operation for each element of the moreShrimps cluster.

Experimental Results and Discussion
We apply the HSV filter to calculate the number of fire red shrimps, and perform Sobel edge detection to calculate the numbers of fire red shrimps, crystal shrimps, and Takashi Amano shrimps. The experimental results in Table 4 show that the system based on Raspberry Pi can count shrimps accurately in 1 s. The average accuracy rate of 100 runs was maintained at above 96%, except for fire red shrimps counted by the Sobel method, and the standard deviation was low. According to the results of analyzing the counting accuracy, there are two conditions of the calculation results: shrimps were undercounted or overcounted.   The undercounted conditions are as follows: • Situation 1 -Direct neglect: (1) The feature of the target shrimps is not obvious enough to be extracted. For example, the color of the shrimp is not within the specified color space, or the edge is unable to be detected clearly. (2) The filtered area is too small to be considered as noise, as shown in red circle 1 of Fig. 3(b). • Situation 2 -Indirect neglect: There is too much overlap, as shown in red circle 2 of Fig. 3(b).
Actually, there are three shrimps in that blob, but two shrimps completely overlap except for their tails. On the basis of the area calculation, we divide the blob area by the area of a shrimp. In the case of rounding, the result shows two shrimps instead of three shrimps. The overcounted condition: Too many shrimps are so close together that the area of the blob contour includes the space between the shrimps.
The results of using different feature extraction methods are shown in Table 4. This system exhibits satisfactory speed (< 700 ms) and high accuracy. The average accuracy rate of 100 runs was maintained at above 96% in our experiment. The number in parentheses in the first column of Table 4 indicates the number of the shrimps. The number in parentheses in the fourth column of Table 4 indicates the direction of accuracy rate. A positive value indicates overcounting, and a negative value indicates undercounting. The low standard deviation indicates the high quality of the programming in terms of performing stable calculations. Furthermore, the different extraction methods have different accuracies. The HSV filter is suitable for fire red shrimps, whereas Sobel edge detection has poor accuracy as shown in Table 4.

Conclusions
In this study, we applied image segmentation, edge detection, and unsupervised machine learning methods to the Raspberry Pi Zero W. The proposed cost-effective and portable device increases the practicality of a portable automated shrimp-counting system. This system exhibits satisfactory speed to complete the calculation in one second and high accuracy at 96%. The experimental results show that different extraction methods have different accuracies on different species of shrimps. Therefore, we suggested that different extraction methods should be used for different species of shrimps to keep the accuracy high.