Tangible interfaces for interactive multimedia presentations

This article is devoted to tangible interfaces for steering and control of interactive multimedia presentations. Various methods for digital encoding of physical objects are considered and their applicability in surface encoding for tangible interface components is discussed. Experiments with presentation controls, based on direct interaction with digitally encoded printed handouts are reported. An innovative approach for transferring presentation controls from printed handouts to surfaces of real physical objects is introduced. Consequently labels, digitally enhanced with CLUSPI codes are created and presentation control trials involving real products with digitally encoded surfaces are conducted. USB and wireless cameras are employed as CLUSPI readers for implementing surface based interactions and a portable communication device with an embedded camera is considered as a possible truly mobile solution.


Introduction
Interactive presentations are commonly used for communicating ideas, for introducing new designs, and for marketing of consumer products.Slide-based presentations are traditionally employed for reporting research results at scientific workshops and conferences.Specialized presentations of architectural designs for example, in addition to slides would most often also use physical models.When new commercial products are launched, even more elaborate presentations including high impact video clips, demonstrations and live interactions with the products would be employed.
All above examples are about interactive presentations in which the final presentation content, timing and order of delivery, etc. are not pre-determined.Indeed, in any live presentation, adjustments to accommodate audience feedback, to stimulate attention and to raise interest need to be carried out on a continuous basis.In contrast to non-interactive presentations such as video clips and simple slideshows, an interactive presentation requires appropriate delivery control interface.When a dedicated presenter is in charge of the presentation, interface complexity is not of big concern, since ample time could be provided for the presenter to learn and train in advance.When a non-experienced person, such as a casual user or a customer is put in direct contact with the interface however, usability becomes an important issue.
In this work we discuss physical interfaces based on objects that are easy to handle in a natural, selfexplanatory way.We explore possibilities for integration of such interfaces into interactive multimedia presentations where presentation targets could ultimately become components of the physical interface.

Presentation interfaces and components
Traditional presentation facilities such as classic 35mm slide projectors, overhead projectors for transparencies, and document cameras for large screen projections of paper documents use tangible presentation media that can be handled in a natural way.People are accustomed to touching and flipping pages while reading books or while browsing through stacks of printed documents.May be that is why we feel so comfortable with presentations, based on direct projection of paper documents, transparencies and film slides.Even for computer-based Impress, PowerPoint and Keynote presentations, where only digital data content is needed, some presenters would prepare paper printouts, hold them and refer to them during the presentation.This is an indication that tangible presentation media and interfaces have not become obsolete and, when possible, should be supported in the scope of the new and emerging presentation technologies.
Interactive presentation technologies have been steadily developing over the last decade resulting in proliferation of interactive whiteboards, large displays with touch screen support and various interaction support hardware and software kits [2].While according to [2] the largest market sector may still remain primary and secondary education, expansion to higher education and corporate and public sectors is moving ahead, thus creating opportunities for devising new interaction concepts and for experimenting with new ideas.
A typical interactive whiteboard system nowadays consists of a flat surface presentation medium that can be used both, for writing and drawing, and as a projection screen [7,10].True advantages of the interactive whiteboard are best revealed when paired with a computer and used for simultaneous projection of presentation content and for annotating and control.The interactive whiteboard is shaped after the well-known blackboard, embodying its classic tangible presentation medium and interface, which enhances the perception of convenience and user friendliness.
Currently two major approaches are used for content presentation.One is based on projection devices placed in front of opaque, or behind translucent screen surfaces, and the other is based on large size TFT or plasma displays.In both cases interactive software intensive functionality is supported through built-in touch panel facilities or through additional technical means such as attachable ultrasound and infrared pen trackers and proximity sensors [7].More flexible control, including from locations that are not in the vicinity of the screen, could be ensured by providing support for laser pointing device tracking [1].Interaction support could be further extended to various mobile devices as reported in [11], where a novel software intensive multi-screen pick-and-drop approach is discussed.Similarly in [6] the position of a specialized PDA, held in the vicinity of the presentation surface is tracked and used for retrieving annotation histories and for corresponding data visualization.Truly mobile interactions become possible with the approach presented in [13], where a smart phone device is used as a mobile wireless camera for retrieving partial screen views and for detecting locations.Based on that, software intensive support for drawing, for placing of data objects, and for more general pick&drop actions has been reported.
A more general approach would be to have one or more fixed cameras providing live video input of all presentation related activities.Presentation control then could be based on scene analysis and recognition of user actions although, since this is a rather computationally expensive procedure, for practical reasons certain limitations need to be imposed.The system [3], for example, uses an infrared camera and reflective markers or infrared LEDs for simplification and image processing acceleration.In contrast, the system [15] attempts to optimize image processing and recognition by using tilted camera views that appear to be particularly well suited for surface touch detection.
Video input is also often used in Virtual Reality (VR) presentations.For delivering realistic and convincing presentations mixed reality techniques are applied, where the input video stream is dynamically blended with a VR model [14].The mixed reality method allows images of real artifacts to be transferred to the virtual world and used as a reference base for augmented presentations.

Digital encoding of physical objects
Barcode based product management is a mature technology and continuous attempts have been made to extend its use to widespread consumer products and recently to automated web content access through camera enabled mobile devices [5,12].Whatever barcode-based application is considered however, incorporating the barcodes into the product content seems to pose some problems.Main drawbacks appear to be the rigid requirements about the barcode size, color, shape, markers, margins and its general appearance that interfere with other information content [9].Recent breakthroughs in digital watermarking and color-based data encoding are addressing these problems and more visually acceptable solutions are becoming available [4].By using digital watermarking instead of barcoding, product information could be dispersed over the entire product surface and thus almost completely hidden.This way when used as an interface component, the product can be identified from any camera angle and view.
For effective presentation control however, identifying the product is only one of the required steps.User interactions with the product should also be recognized, for example based on the product position and orientation in respect to the image input device.Such kind of software intensive functionality is obviously not achievable through digital watermarking or any type of barcodes because they do not carry local position payload.
We address this problem by employing the Custer Pattern Interface (CLUSPI encoding scheme co-invented by the author [8], which has been especially designed to blend seamlessly with existing content and to provide reliable product identification with additional position and orientation feedback.Similarly to digital watermarking the CLUSPI code can be spread over the product surface while effectively covering less than 1% of it and thus staying practically invisible.The CLUSPI method supports cluster patterns of graphical objects with different shapes, sizes and colors and therefore allows building custom patterns that would blend well with any background.

Presentation control methods
Computer-based software intensive presentations are usually controlled through standard computer input devices namely a mouse, a touch pad or a keyboard.Specialized remote control units like wireless mice, laser pointers with augmented functionality, and others are also available.The device shown in Fig. 1 for example is a projector remote control that can also serve as a presentation control interface.This is achieved through the two additional buttons named "Previous Page" and "Next Page" that are converted to "PgDn" and "PgUp" keystrokes by the projector and then sent to the presentation computer over an USB link.
As previously mentioned many presenters prefer to have a tangible printed copy of the slides in their hands while conducting a computer based Impress, PowerPoint or Keynote presentation.That makes us believe that users, if offered such an option, might choose to use their slide printouts for direct control of the presentation flow instead of any other interface.A natural way to show a slide would be just to point and click on its printout with a handheld scanner.In contrast, when a keyboard, a mouse, or even a dedicated remote control device is used, presenters would most often just go forward or backward slide by slide.This way, even when a slide has to be skipped, it would be momentarily shown on the screen.We all know how inconvenient it feels when one needs to jump back and forth between distant slides in a presentation.Suggested direct point-and-click functionality for the printed slides naturally solves the above-mentioned problems.This functionality could be implemented with different technical means such as barcodes, OCR, CLUSPI codes, digital watermarking, and others.Despite of the selected technical means the following two steps are essential part of the implementation: -adding slide identifiers in the form of barcodes, text strings or other codes, and -mapping of slide identifiers to presentation control actions.
We are currently conducting experiments with various barcodes, OCR and CLUSPI codes, and we are planning to continue in the near future with digital watermarking and other methods.

OCR-based presentation control
Initially we thought that a C-Pen reader could directly use printed slide numbers as slide identifiers for OCR.We were surprised however by the relatively low recognition rate of single digits e.g. for slides numbered from one to nine.Changes of fonts and character sizes did not improve the recognition rate significantly, so we had to resort to adding leading zeroes in front of the slide numbers.With this modification in place, the recognition rate increased and approached the one for sequences of symbols such as words and phrases.We believe that the noted problem might have been caused by a glitch in the C-Pen driver software, which seems to have been tuned for scanning numbers and words rather than individual digits and letters.
Employment of the C-Pen as a presentation control device is illustrated in Fig. 2 where the user holds printed slide handouts with six slides per page.The button on the top of the C-Pen under the user's index finger can be programmed to emulate different key presses and we have opted to assign to it the CR functionality.By pressing it, the user can progress to the next action in the presentation sequence, which might be displaying additional content in the current slide or moving to the next slide, etc.There is another button on the tip of the C-Pen that activates the image acquisition and analysis process when the pen touches the document surface.Scanning of a slide number is conducted by a sweeping gesture with the C-Pen over it.There are limitations on the C-Pen tilting, velocity and acceleration that need to be observed in order to ensure reliable recognition.
Since during a presentation, visual feedback of the scanning recognition results could be perceived as disturbance, we suggest using audio feedback instead.With this functionality enabled, a discreet "click"  sound could indicate successful reading of a slide number while two clicks would indicate scanning failure.In our setup, scanning operations are confirmed by pressing the top button of the C-Pen.This way users can rescan as many times as it takes to obtain a successful reading and only afterwards press the top button to initiate a jump to the successfully read slide number.

Barcode based presentation control
Using barcodes for slide number encoding appears to be a viable alternative to OCR that ensures more robust recognition and control.A sample slide with a Code 39 barcode, printed next to the slide number is shown in Fig. 3.Most dedicated barcode scanners support Code 39 barcodes and provide fast and reliable recognition, by taking advantage of the error recovery mechanisms embedded into the barcodes.The C-Pen device, which is not a dedicated barcode scanner, has also performed well; in most cases demonstrating a recognition rate, better than that of slide number OCR.However, as shown in Fig. 3, a relatively large area of the slide had to be dedicated to the barcode and the required white margins around it in order to guarantee reliable recognition.

Surface encoding for presentation control
In the discussion so far we have explained about direct selection of slides by OCR of printed slide numbers or by scanning of printed barcodes.We would also like to provide support for additional actions and thus to allow using multiplicity of identifiers across different slides.We are developing for this purpose a new software intensive system component, dedicated to identifier-to-action mapping and consecutive presentation control.Through it, the original one-to-one mapping is extended to multiplicity mapping where modalities could also be taken into account.One and the same identifier, for example could lead to invoking different actions and action sequences, depending on the context.We illustrate advantages of this approach while discussing our CLUSPI based experimental implementation.
First of all, please recall that CLUSPI codes do not use any dedicated space and are actually overlaid with the existing slide content.Therefore the entire surface of all slides in the presentation could be easily covered by CLUSPI codes in an unobtrusive way.This will effectively make each and every point on any of the printed slides directly addressable.In result, a point and click with the CLUSPI reader on any printed slide would return a unique identifier.
Let us consider whether CLUSPI could provide functionality equivalent to the earlier explained slide number OCR and barcode scanning.Our objective is to enable slide selection by pointing with the CLUSPI reader to the printed slide numbers or barcodes.In this simple case, it comes down to a direct mapping of the unique identifier returned by the CLUSPI reader, to some corresponding slide selection actions.Obviously, pointing anywhere in the vicinity of a printed slide number or on its corresponding barcode should also produce the same result.To achieve this we define areas of interest around all printed slide numbers and barcodes and then match the position and orientation data returned by CLUSPI to them.
With CLUSPI different actions could easily be associated with different areas on the printed slide.The sample slide in Fig. 4 exposes several click-sensitive areas, enclosed in dashed rectangles for clarity.The slide number and barcode areas at the bottom implement the standard slide selection functionality as explained earlier, while the areas with the pictures implement presentation of different multimedia content.This way printed content elements of any color, size and shape could become active, clickable interface components without altering their visual appearance.

Commercial presentations and live demonstrations
So far we have considered using printed slides for direct control of a presentation, arguing that users might like it better than a traditional keyboard, mouse or a remote control.Commercial presentations however, in addition to slides and multimedia content would often include live demonstrations of real products.Our idea is to have some of the demonstrated physical objects or their replicas integrated with the presentation content and shown simultaneously with it.

Handheld readers and devices
To illustrate our approach, we will consider an imaginary presentation and live demo of the sample stapler shown in Fig. 5.
The slide shown in Fig. 4 contains a few drawings explaining the stapler refilling process where pointing and clicking on any of the drawings invokes additional multimedia content, linked to it.This kind of presentation control however will interfere with a simultaneously conducted live demonstration when the presenter is operating a physical stapler.Therefore we would like to eliminate the need of clickable printouts or separate presentation control interfaces and use instead the physical stapler as a presentation control.In other words we would like to transfer the presentation control functionality from the printed slides to the physical stapler.
In practice this could be done by simply changing the printing medium of the presentation slides.In our experiments, instead of paper, we have used adhesive labels for printing some of the slide content.Then slide numbers and corresponding barcodes, cut from the adhesive labels have been pasted on the stapler surface and consecutively used for presentation control.The method however does not work well for devices with mostly curved surfaces and when sizes of flat surface areas are below certain limit.
We address this problem by enhancing surfaces of physical objects and replicas with direct pointand-click functionality similar to the one used for slide printouts.In the stapler case for example, we prepare labels with CLUSPI codes that are much less sensitive to bending and distortion and we attach them to the stapler.This way we create click-sensitive areas on the stapler surface that are functionally equivalent to the links embedded in the slide in Fig. 4. Pointing and clicking with the CLUSPI reader on the sensitive areas of the stapler as shown in Fig. 5 for example, invokes video clips showing the staple  refilling process.Note how the presenter is conducting a live demonstration of the refilling process with his left hand, while controlling the presentation with the CLUSPI reader in his right hand.
The CLUSPI reader shown in Fig. 5 (see the enlarged view fragment between the user hands) is based on an inexpensive web camera with an USB interface physically connected to the presentation control computer.In Fig. 6 we show another CLUSPI reader based on a compact wireless camera which does not require any cable connections to the presentation control computer.Such wireless reader certainly provides more freedom and flexibility during the presentations.
A truly mobile solution, however, would need to employ a dedicated compact device with CLUSPI input capabilities, integrated multimedia and communication functionality and enough computing power.
For our experiments we have acquired the Windows Mobile based WS011SH communication device shown in Fig. 7, which has an embedded camera that seems to meet most of the CLUSPI related requirements.For a conclusive decision, however, we still need to adjust the CLUSPI related hardware, to optimize the developed software components and to conduct more detailed experiments.

Fixed-position readers and devices
Handheld readers and devices, while acceptable for commercial presenters, might not be very convenient for non-professionals.Potential buyers, for example, are often invited to attend and participate in commercial presentations by touching and experiencing concerned products.In such cases, an ideal solution would be to make the reader part of the product environment, eliminating the need to operate the reader by hand.This would lead to a presentation system preloaded with product-related content, one or more samples or replicas of the product placed next to the presentation system, and an appropriate presentation control interface that should deduct customer intentions and exercise appropriate controls based on user interactions with the products.
To address this, instead of a handheld CLUSPI reader we suggest using one or more fixed-position readers.Those readers should be able to extract digital information embedded in the product package and labels while the customer is examining the product.A preliminary design and layout of a presentation system based on this principle is shown in Fig. 8.The CLUSPI readers should be embedded in the environment in a way that ensures at least one clear view of the bottle label for any orientation of the bottle.For example, the readers could be integrated in the artificial stones and the vase placed around the bottle in Fig. 8.
In addition to CLUSPI readers, the system should be equipped with motion sensors that allow detection of present and approaching customers.With no customers within range of the motion sensors, the system stays in a stand-by mode.Once a motion sensor is activated, the system switches to an appeal mode, where a video is shown, inviting the customer to come closer and investigate the product.In our case, users are encouraged to turn the bottle on the stand, examine it from all directions, and read the inscriptions on the bottle labels.Once a change of the bottle position or orientation is detected, the system switches to presentation mode.In this mode the CLUSPI readers placed around the stand would provide feedback regarding current bottle position and orientation, and based on that, tracking the bottle label area currently in front of the user eyes would be done.Since the system identifies the area that is being watched by the user, presentation slides linked to that area could be shown on the large display behind the stand.This kind of system response is functionally equivalent to a direct point and click to the watched area with a handheld CLUSPI reader, as explained earlier.
Obviously users could change bottles on the stand, in which case the system should automatically switch to new content, corresponding to the product currently placed on the stand.This functionality could be easily implemented through the product identifiers returned by the CLUSPI reader.We are also considering extensions of this system that would be able to handle more than one product at a time.Then, for example, if two bottles with different drink brands are placed on the stand, the system could show a side-by-side comparison and give recommendations.

Software intensive systems and related interfacing issues
The discussion in this work has revolved around interactive presentations and methods for their control.
Throughout the text we have used illustration data from Impress, PowerPoint and Keynote presentations.Proposed approach, however, is quite general and could be successfully employed in a much broader range of interactive presentations and software intensive applications.
Interactive software intensive presentations take place, for example, in scientific visualization, where experimental data and results from numerical simulations need to be disclosed to users in an appropriate, easy to analyze and understand way.Virtual and mixed reality applications make another example of interactive software intensive presentations, where virtual and physical objects are integrated and simultaneously exposed to users.
Both, scientific visualization and virtual and mixed reality applications could obviously benefit from the discussed interactive presentation control methods.In scientific visualization related interactive presentations, for example, data-driven tangible interface controls could be produced ad-hoc, simply by printing out surface encoded data sheets with clickable functional buttons.In virtual and mixed reality related interactive presentations on the other hand, real physical objects and tangible models with digitally encoded surfaces could be used for direct interactions with their virtual counterparts.
In general, any software systems with an appropriate data visualization interface and interactive user controls could be enhanced through the proposed method.We believe, however, that employed tangible, self-explanatory interfaces are particularly suitable for software intensive systems and could help isolate users from the growing software, hardware and communications complexity.

Conclusions and future work
In this work we have introduced physical presentation control interfaces, based on tangible objects that are easy to handle in a natural, self-explanatory way.We have shown examples of digitally enhanced slide printouts that can be used for direct presentation control.We have also explained how direct presentation control functionality can be transferred from presentation slides to real objects used in live interactions.
In the context of software intensive systems, our approach allows for advanced interfaces that use real physical objects as interface components and actively engage the audience.When users handle and explore different tangible objects, a dedicated CLUSPI based software intensive system tracks the corresponding user actions and provides appropriate feedback and assistance.
We plan to continue this research by carrying out more extensive usability tests and conduct detailed user experience and satisfaction surveys that will be analyzed and reported in successive publications.

Fig. 2 .
Fig. 2. Using a C-Pen device for slide number scanning and presentation control.

Fig. 3 .
Fig. 3. Sample presentation slide with an embedded Code 39 barcode used for presentation control.

Fig. 5 .
Fig. 5. Using a real product for direct presentation control.

Fig. 7 .
Fig. 7.An integrated Windows Mobile based communication device with an embedded camera.

Fig. 8 .
Fig. 8. Preliminary design and layout of an experimental presentation system with fixed CLUSPI readers.