An Innovative Direct-Interaction-Enabled Augmented-Reality 3D System

Previous augmented-reality (AR) applications have required users to observe the integration of real and virtual images on a display. This study proposes a novel concept regarding AR applications. By integrating AR techniques with marker identification, virtual-image output, imaging, and image-interaction processes, this study rendered virtual images that can interact with predefined markers in a real three-dimensional (3D) environment.


Introduction
Recently, augmented-reality (AR) techniques have been widely applied in modern technology, allowing developers to incorporate computer-generated virtual images into actual footage that was recorded using a camera.Thus, AR is observed in numerous applications and fields [1][2][3][4][5][6][7] such as advertising, films, visual conferences, education, and smart phones.
Previously, integrated real-space and virtual images using AR techniques could only be viewed on a display [8].For example, Milgram proposed a see-through and monitorbased AR display method that required a display to enable interaction and defined markers and virtual image as indirect interactions.Recent AR products, such as Google Glass, are also see-through methods.The AR system developed in this study differs from those methods because this system enables direct-interaction functionality.This functionality is achieved by using a system-imaging device, which renders virtual images that overlap predefined real space.In addition, the proposed system is in compliance with the three characteristics proposed by Azuma, who stated that AR applications must combine real and virtual elements, be interactive in real time, and operate in three-dimensional (3D) environment [9].

The Direct-Interaction 3D System Design
In the application process of AR techniques, this study designed a direct-interaction system that enables direct interactions between real and virtual environments, as shown in Figure 1.The figure depicts a user holding a marker that is used to interact with the AR system.When the camera and computer identify the shape of the marker, the virtual image of the marker overlays the real space by applying image structuring.The user can then directly observe the interaction between the marker and virtual imagery.To achieve this direct interaction, optical principles must be carefully applied to the reimage structure.The design concepts of the proposed system are explained in the following section.

The Interactive Design.
The interactive structure designed in this study is based on a 3D display technique (Taiwan Patent Publication number I28830 [10]) invented by the research team.By integrating AR application concepts and the 3D display technique, the researchers could design a display system that features direct-interaction functionalities.As shown in Figure 2, the structure of the display system primarily comprises two function types, specifically, an AR identification and interaction function and a virtual-imaging function.In Figure 2, components 1 and 2 are reflective mirrors   and component 3 is a Fresnel lens.Integrating these three components enables the system to render virtual images into a real-space platform.Component 4 is a virtual-image display that not only presents virtual images but also presents viewable 3D images.Component 5 is a camera that is used to extract the marker of the real space.A computer (component 6) is used to process the images captured by the imaging system (components 1, 2, and 3) and create virtual images.These images are then projected in a predefined location in which the marker can be used to interact with the realtime AR image directly.The size of the proposed structure is primarily based on two factors, namely, the focal length of the Fresnel lens and the size of the image display (component 4).

The Optical Design.
In this study, component 4 was a flat display instead of a 3D device.This device produces flat images similar to those of conventional indirect-interaction AR systems.To project these images in a real space, the display position and the influence of the Fresnel lens on image quality must be considered.Figure 3  angled placement of the mirrors is equivalent to a 180 ∘ rotation effect.
The relative positions of O and I are expressed according to the following: When the image is magnified to  = 1, the optimal image quality is obtained.Because  = (1+2)/(1+2), 1+2 = 1 + 2 is observed in the system structure.When the design of the structure is symmetrical, then 2 = 1 and 1 = 2.In addition, the size of I is similar to that of O.In this instance, (1) can be rewritten as where 1 + 2 = 1 + 2 = 2.In this study, the focal length of the optical condenser was  = 20 cm; therefore, 1 = 2 = 1 = 2 = 20 cm.In Figure 3, L represents the optical condenser.Considering generation capacity and component availability, a Fresnel lens was used as a substitute for conventional optical condensers.However, based on diffraction theory, the quality of the final product generated by using a Fresnel lens is different compared to that of a conventional optical condenser.The reason for these differences is demonstrated by the data shown in Figure 4.In this figure, the diffraction potential of a convex lens was stratified and each layer had a thickness of  (the red rectangles in Figure 4).When the red rectangles were removed, the shape of the light wavefront was unaffected and did not affect the imaging results.Thickness  can be obtained by using the following reasoning: let  represent the thickness of the lens medium.The optical wavelength () traveling a distance of  in space corresponds to a phase shift of Φ1, whereas that traveling a distance of  in the lens medium corresponds to Φ2.Therefore, the difference between phase shifts is Φ.If the reflective index of the lens medium is , then When Φ = Φ2 − Φ1 = 2, then  = 1, 2, 3, . ... The wavefronts of the mentioned light waves are identical.After traveling a distance of , the wavefronts of the light waves remain identical.Thus, when  = 1, the minimum thickness of the reduced lens can be expressed as follows: However, the curved surfaces of common Fresnel lenses are imperfect unlike those presented in Figure 4, because the design and manufacturing of these lenses typically involve a second-order manufacturing process.This process sacrifices the microstructures within the lenses [11], thereby substantially reducing the thickness, weight, and size of the lens at the cost of quality degradation.This study adopted a Fresnel lens as the imaging component and used a convex lens for comparing the quality differences.

Testing
By integrating the mentioned structural designs and optical principles, this study developed an innovative directinteraction AR system.Based on the operational principles presented in Figure 5, the proposed system was used to render virtual images into the real space, and the predefined marker was used to interact spontaneously with the system.

Marker Identification.
To render virtual images in 3D environments and enable direct-interaction functionalities, the researchers adopted AR techniques.A camera and computer were used to complete the identification process of the marker cards (Figure 5).The marker cards used in this study were created in black and white, which was easier for the computer to identify.However, other objects could also be used in the identification process.

Interactive-Image
Output.The camera captured the shapes of the marker cards, and the AR design program (VIZARD) was used to identify and confirm the shapes.In the AR program design, a predefined marker representing a 3D bouquet was used.After confirming the shape of the marker, the system produced a predefined 3D bouquet.The shape and angle of the 3D bouquet varied according to the shape and size of the marker determined by the AR system, subsequently producing images at different angles and sizes.In the AR program design,  and  mirrored the symmetric setting of the 3D bouquet on the liquid crystal display (LCD) and did not require adjustment because of the corresponding positions of the two reflective mirrors (Figure 2) and the two reflecting and single-diffracting processes.

3.3.
Imaging.The image displayed on the LCD was rendered onto a marker by using the Fresnel lens in the imaging system.As previously explained by (1), when  = 1, the imaging of M2 is rendered at 2, that is, the initial position of the marker.Because component 4 is not a 3D display (Figure 2), the objects were considered flat, and thus a flat image was produced.This restricted the marker identification and display position; that is, the 3D bouquet in this test could only be displayed as a flat image.

Image Interaction.
The 3D bouquet can be illustrated using the Autodesk 3ds Max software.When the image is rendered at the marker, the AR program specifies that the camera captures a specific size and determines the vertical variations of the marker, subsequently displaying the size of the marker.Based on Figure 6(a), when the distance between the marker and camera becomes great, the bouquet becomes small; conversely, at short distances, a large bouquet is displayed.
As established in the AR program, when a specific point on the marker presents a vertical line of 1 at the central point of the camera, the 3D bouquet becomes perpendicular on the marker card.Thus, when the surface of the marker card changes line angles 2 and 1, the bouquet rotates because the surface of the marker card is rotating, as shown in Figure 6(b).These results indicated that the marker card directly interacts with the predefined image.

Conclusions
This study successfully rendered virtual images into the real space and enabled the virtual images to interact with realtime objects.This achievement is a novel application in AR techniques.By integrating the proposed handheld 3D imageinteraction structure with an image display and marker identification, AR techniques can successfully be applied to the real space, thereby enabling actual objects to be involved in real-time and direct interaction with virtual images.

Figure 2 :
Figure 2: The structure of the direct-interaction 3D system.

Figure 3 :Figure 4 :
Figure 3: A conceptual diagram of the imaging process in the interactive structure.

Figure 5 :
Figure 5: The operational principles of the innovative AR direct-interaction system.

Figure 6 :
Figure 6: The interaction between the marker and the camera.