Vision-Based Mobile Manipulator for Handling and Transportation of Supermarket Products

Robot manipulators are growing more widely employed in the retail market, mostly for warehousing, but automating them instore logistics processes is still a dicult task. Supermarkets and large retail stores face many challenges: shortages, handling, and placement of a single product on shelves. Various issues needed to be considered to develop a robot which can manipulate products of dierent sizes, shapes, and weight in limited spaces on shelves.e aim of this article is to design and develop a system to address the issues of shortage, identication, moving, and placements of products in supermarkets by properly incorporating database, camera vision, and line following mobile manipulator. A four-wheeled dierential drive mobile robot was designed and developed which has a 5 DOF robotic manipulator on it. e line following technique is used to move it around the warehouse. e barcode recognition technique for the localization of product sections and object detection using SIFT is successfully and eciently employed. e demonstration of the usefulness of the method was shown by carrying out experiments in a relevant environment which imitates a real supermarket.


Introduction
e customer experience and comfort have been improved by the implementation of automation techniques at retail shops. A major example is the Amazon Go shops. On the other hand, automated solutions are still limited in the logistics processes of retail stores of the supermarket industry speci cally, in Pakistan. e supermarket industry is widely expanding in Pakistan, while being the fact that it still has not achieved the top-notch advancement.
ere are many areas of advancement and improvement in it following the problems and complications exist in it. One of those problems is the tracking of products on the shelves which are short as well as to put them right in their place on shelves. As the size of supermarkets and number of products increases, it becomes di cult for humans to manage.
Various commercial mobile robots can be used to monitor shelves for inventory management automatically by choosing diverse technologies, such as radio-frequency identi cation antennas in the Stockbot or the cameras in the Bossa Nova 2020, Tally 3.0, and LoweBot [1]. Another depalletizing-based automated process uses recent arti cial intelligence solutions like the Photoneo Depal [2]. Recently, some control solutions based on arti cial intelligence approaches have also been presented [3]. A new depalletizer which was able to manage not only boxes but also varied pallets has been proposed in [4]. e inventory management and unloading solutions are already in the market, whereas other in-store automated processes, such as transporting items from the warehouse to the retail store and shelf restocking, can still be taken as emerging technologies. is is a great demand from retailers due to the high costs of in-store logistics, up to 60% of the entire functioning store costs [5]. Fierce market competition has led to a decline in profits and high labor costs make it difficult to ensure that there is enough staff to handle all the work in a store [1]. e shelf refill is the most time-taking task with 50% of such of the time is committed in finding the proper slot on the shelf. Very few research literature is available on this particular automation issue such as in [6][7][8]. Most of the research issues are relevant to the convenience store robotic challenge which was launched by the World Robotic Summit [9]. e solutions are mainly based on the usage of vacuum grippers; therefore, they are applicable only in those situations where pick-and-place and grasp poses are similar. Whereas, the shelf refill objective in an actual environment may require cultured skills of manipulation. e REFILLS (robotic enabling fully integrated logistics lines for supermarket) [10] presented a software architecture which implements the shelf replenishment with a comparable speed of human. It also incorporated the KnowRob [11] based taskplanning framework. e current article's contribution is focused on the lower hierarchical level. It comprises of a 7-DOF mobile manipulator prototype that is integrated with a supermarket database system. e proposed system also features a graphical user interface (GUI) for experimentation. A camera vision is also included with the mobile manipulator. As a result of the billing, the supermarket database will be updated. Whenever a product is billed, the system will infer from the database that the product is out of stock. is data will be passed on to the mobile manipulator. After receiving the information, the mobile manipulator will grab the product from the supermarket's warehouse and position it on the shelf where it belongs.
Wireless connection between the database and the mobile manipulator is not used at this time for the sake of simplification. For the purposed system, a laptop will be kept on the mobile manipulator. is laptop will include a database and a graphical user interface. After placing the product in its proper location, the mobile manipulator will also update the database. As a result, the database and the mobile manipulator will be able to communicate in both directions.

Literature Review.
e first steps of any pick-and-place task are object detection and localization. DenseFusion [12] and PoseCNN [13], two of the most current and advanced localization approaches, are nevertheless insufficient for the in-hand maneuvers detailed in this article. e integration of camera vision with a robotic manipulator is a key aspect of our research. Camera vision may be used to identify and regulate the motion of a robotic manipulator, and it can also be used to improve the performance of a robotic manipulator. e proper operation and use of camera vision is fully dependent on determining the point of application. Because determining the point of application, which functions as an input for the system, is required when controlling a robotic manipulator based on its inverse kinematics. However, it is just necessary to specify the settings and specifications for a robot to automatically classify and stow goods [14,15].
Other than determining the point of application, another component of camera vision is the scanning of barcodes.
is domain also has a substantial amount of work [16]. Numerous strategies for barcode detection using camera vision have been developed over the years [17,18], and many methods and techniques for improving the barcode detection process have been implemented [19].
A great deal of study has also been done in relation to warehouse and inventory management systems and their various components [20][21][22][23][24][25][26][27][28][29]. e design of the mobile robot and mobile manipulator is critical in this project because proper and error-free design leads to the project's error-free operation. Many recent and old publications [15,30,31] propose novel techniques to kinematic modelling and comprehensive design of robot manipulators. e kinematic analysis of robot manipulators is a topic of debate in a number of classic literature [32,33]. Peter Corke created a software programme and MATLAB library for kinematic and dynamic design and calculations, which has become the industry standard [34]. Koyuncu and Guzel provided a software package called MSG that is used to test the behavior of robot motion [35]. ey established a way for solving the kinematics of the Lynx 6d of Robot and proposed a method for solving the kinematics of the Lynx 6d of Robot. To address the kinematics of the AL5B Robot arm, Qassem et al. proposed a software package [36]. More analysis have been achieved for modeling a 6DOF robotic manipulators using the MATLAB software for the purpose of their design, simulation, and analysis by Iqbal et al., Kumar

Methodology
To solve abovementioned problems, we developed a system which incorporates the basic components of the system which are mobile manipulator, microcontroller, database, and GUI in a way to timely track down the shortage of products on shelves. Database carries the information about all the products, their barcodes, their optimum quantity, and their actual quantity in warehouse as well as in supermarkets. e database management system is connected to the billing system in practice. However, for the purpose of simplicity, we developed a GUI in place of billing system. e GUI will be used to reduce the number of products from the database. e reduction in the number of products from GUI, then from database will result in the prompt tracking of short products from shelves. It means the system will no sooner have information of shortage of a product than the billing of that product. However, this is a part of usual supermarket management nowadays.
After the information of shortage, the next step is to assess the availability of that product in the warehouse. It has been mentioned above that database also includes information of barcode all products and the quantity of that product in warehouse. It should be kept in mind that, the shortage here does not imply that the products are also unavailable in the warehouse. Database will check if the product is available in the warehouse. If its checking results are positive, the system will communicate to the mobile manipulator. Mobile manipulator, which is included by a microcontroller, will be informed by the information of short products and their barcodes. e locations of di erent products with di erent barcodes are known in the warehouse as well as in supermarket by microcontroller. Microcontroller will travel toward that product in the warehouse by tracing down its location. 5 DOF robotic manipulator will pick up that product. e mobile robot will move toward the location of that product in the supermarket. Robotic manipulator will then put that product right on its place on the shelf.
Here, it must be clear that the line following, barcode scanning, and camera vision techniques will be used for mobile robot path planning, section allocation, and product point determination, respectively. e mobile manipulator will also provide feedback to the microcontroller in the form of feedback sensors. Moreover, after the microcontroller has completed the task of picking and placing the product in its actual place, it will be required to give this information back to PC, i.e., database. In this way microcontroller and mobile manipulator will have mutual communication. Similarly, microcontroller and PC will have mutual communication. Below, Figure 1 represents the block diagram of proposed methodology, and Figure 2 shows the complete 3D model of robotic manipulator. (1) where, ω is the required angular velocity of motor After unit conversion to RPM: It means that both motors of mobile robot should have an angular velocity of 240 RPM or greater.

Torque Calculations for the Mobile Base.
In this section, torque requirement of motor will be calculated. For that, we will use the techniques of statics. e rst step is to draw a free body diagram.
It can be seen in the free body diagram Figure 3 that the weight of whole mobile manipulator, camera, and laptop is assumed at the center. Due to existence of two motors, these forces are multiplied by 2. Now, we will apply conditions of static systems to evaluate forces: From (16), we have the following: Putting it in (15) results in the below:  Mathematical Problems in Engineering 3 Which means that both normal forces are equal, i.e., We know the radius of wheel, so we can nd the torque of motor required as follows: It means that both motors must have a torque of at least 1.47 Nm.

Selection of Motors.
e choice of motor of the mobile robot is very important in design calculation due to the fact that incorrect choice can lead a great loss in term of cost and performance. We are using DC gear motors due to following reasons: (1) DC motors are available with required torque and speed parameters. (2) DC motors are preferred over encoded DC motors due to the fact that we do not require high accuracy because we are using line following technique. We are using chain mechanism due to following reasons: (1) Chain drive o ers zero slip unlike belt and gear drives. (2) Chain drives are more stable than belt and gear drives. (3) Chain drives are highly e cient in power transmission. (4) Chain drives are compact in size and easy to install.

Design of a Robotic Manipulator.
We are using a 5 DOF robotic arm in our project. All of its joints are revolute. erefore, it can be said that it is kind of an articulated manipulator.

Torque Calculations.
In the articulated manipulator, the torques of the joints having rotations about x-axis are important to nd. Due to the downward act of gravitational force (weight) of the links, the links with the rotation about x-axis will have some natural torque. In order to resist the downward force acting on the links plus to lift the payload, the required torques for the motors will be calculated.
To calculate the torques of the motors, the worst condition is considered, i.e., the robotic arm is considered to be placed horizontal. Now, the torque required to lift the corresponding link with load will be maximum as shown in Figure 4.
From the gure, we are only interested to nd the maximum torques about points S, E, and R, respectively. Where W 1 , W 2 , W 3 , W 4 , W 5 , W 6 ,and W 7 are the masses of the links at di erent points.
Maximum Payload A 0.15kg, Using second condition of equilibrium, required torque of motor about point 'S' is as follows: 2.1.9. Forward Kinematics. e labeled kinematic diagram of robotic manipulator is shown in Figure 5. e forward kinematics calculations start with the labeling of kinematic diagram according to DH method, basic rules of which are as follows: (i) Direction of z-axis is toward of the joint axis.
(ii) x-axis is parallel to the common normal.
As shown in Table 1, there are four DH-parameters a, d α and θ. ese parameters are designated according to following rules: (i) θ i is the angle from x i-1 to x i along z i− 1 ; For revolute joint θ i is variable. (ii) d i is the distance from the intersection of z i− 1 with x i to the origin of (i− 1) system of axes; For prismatic joint d i variable. (iii) a i is the shortcut between z i− 1 and z i . (iv) α i is the angle from z i− 1 to z i along x i . e general transformation matrix is as follows:

Mathematical Problems in Engineering
Gripper's position is de ned by the p 5 vector while orientation de ned by x 5 , y 5 , z 5 vectors provided by the forward kinematics: 2.1.10. Inverse Kinematics. Solution to the inverse kinematics problem is reduced to the calculation of the arguments θ 1 , θ 2 , θ 3 , θ 4 , θ 5 based on the gripper's position and orientation. ese six equations in ve unknowns may have no solution, though it is possible to consider it for some instances [40]. Equations (16)- (19) can be rewritten in the following forms: θ 1 can be evaluated using (29) and (30) straightforwardly: Table 1: DH table. Link    Mathematical Problems in Engineering To evaluate θ 2 and θ 3 , we used the geometric method: where values of ∅ 1 and ∅ 2 can be evaluated as follows: where From (28), 2.1.11. Work Flow Diagram. As mentioned in previous sections, all important components and aspects of the project were made functional. Line follower robot, 5 DOF robotic manipulator, camera vision, GUI, and various peripheral components of these components were signi cant components of the project. Important portions and systems of the project, such as barcode identi cation, object detection, and inverse kinematics, were combined with these components. e project was made to function according to the ow diagram illustrated in Figure 6, after all project components and systems were integrated. is ow diagram depicts the project's whole operation after it has been integrated.

Development Stages and Final
Stage. e development stages and nal stage of the proposed model are given in Figures 7 and 8, respectively.

Implementation of the Barcode Recognition Technique.
For section tracking of warehouse and supermarket, we used barcodes as a visual marker. ese markers guided the robot toward the location of items. Each section had its di erent barcode on its front. OpenCV library was used to access the camera. Barcode and QRcode detection and decoding were done by using the pyzbar library with OpenCV. It rst reads the image using cv2.imread(). After reading the image, it decodes the image using pyzbar.decode() and extracts the bounding box (x, y) coordinates from the barcode data, enabling us to localize and determine where the barcode in the input image is. It draws a rectangle around the barcode with OpenCV rect function and also puts a decoded text on the image or scene. Our mobile base robotic arm operates on Raspberry Pi B3. We implemented this technique on Raspberry Pi by using a Pi camera. e owchart in Figure 9 shows the methodology of a barcode recognition technique and results of barcode recognition are shown in Figure 10.  Figure 6: Work ow diagram.

Object Detection
Mathematical Problems in Engineering 7 incorporated with a robotic arm to pick objects from the warehouse and place them in their respective shelves in the supermarket. To detect and recognize di erent object detection algorithm like SIFT, SURF, and ORB can be used. Scale invariant feature transform (SIFT) is used in our project because it is robust and invariant to scaling and lightning conditions. e owchart diagram of object detection using SIFT is shown in Figure 11.
To implement the SIFT algorithm in our project, we used openCV version 3.4.2. It is a built in detector in openCV. First, we have to construct a SIFT object using cv.xfeatur-es2d.SIFT_create(). After this, SIFT will detect key points of target object. OpenCV provides cv2.compute() which computes the descriptors from the key points. We have key points and descriptors of the target image.
We repeat the same process for the scene image. We got the descriptor vectors of both scene image and target image. Now, we match these two vectors. e minimum Euclidean distance between these two 128 dimensional descriptor vectors will be the best match. For matching brute force and fast library approximation nearest neighbor (FLANN) can be used. e object detection using SIFT before and after center point calculations are given in Figures 12 and  13,respectively.

Conclusions
e xed manipulators on the industry oor have already been developed and established decades ago with great efciency. e proposed project provided a mobile manipulator which solve many problems faced by today's supermarket and inventory industry. In time placements of products in supermarkets were handled by properly incorporating database, camera vision, mobile manipulator, and Raspberry Pi (Controller). e four-wheeled di erential drive mobile robot was designed with line following      capability covered the factory floor area by moving accurately around the warehouse. A 5 DOF robotic manipulator could pick the object effectively with the help of SIFT feature extraction and center point calculation of the object. e barcode recognition technique enabled the identification of product to be placed at right sections in the warehouse. It was done using PYZBAR. In spite of its good results, many advanced features and desired components could not be introduced due to financial and time limitations. e future recommendations will likewise focus on addressing the issues. To eliminate interaction with humans on the ground, a line follower robot could be replaced by an overhead transit system. It will significantly increase the speed of the entire operation. ere are a variety of additional ways that can be used in this situation. Redevelopment of the product localization process can be done in the future. We urge that additional tests be conducted by replacing the line follower with a SLAM-based navigation system, as we feel that using SLAM-based navigation on a mobile robot can open up new possibilities for the project's functionality.
RFID tags have proven their reliability in the supermarket and product handling applications [41], and they can be used here instead of barcodes to greatly improve results. To improve results, the model of a mobile robot and robotic manipulator can be changed in terms of materials, shape, joints, and actuators. Picking objects in a cluttered environment will be easier with the learning of a robotic manipulator utilizing a neural network. It is the most essential of all future recommendations because it influences the project's main goal.
Aside from these design changes, a significant step forward would be to fully automate this semiautomated system. For example, we used a GUI to perform billing actions in this prototype, but the actual system should be connected to a billing server or database and automatically take inputs from the database. In the future, it is recommended that mobile robots be entirely automated.

Data Availability
ere are no explicit data used in the proposed research. All the data are available in the paper.

Conflicts of Interest
e authors declare that they have no conflicts of interest.