Design Space of '(Dis)Appearables'

With the idea of (Dis)Appearables to create a sense of appearing and disappearing for Tangible UIs, there are a variety of opportunities for expression and perceptual interactions. 

Beyond simple Appear and Disappear effects, this modality can be combined with graphical information (e.g. projection, display, etc) for representing the transition between Graphical Information and Tangible Information. By combining appear and disappear effects in separate locations, the effect of Teleportation can be presented. As such, we are exploring the possibility of new expressions and perceptual tricks with actuated TUIs in project (Dis)Appearables for novel interaction design.

Transition Portal and Prototypes

We developed a variety of transition portals, which are doorways for robotic TUIs to transit between the front stage and backstage. Different types of portals can present different expressibility and quality of appearing / disappearing, and they become the basic primitives to be used for a variety of applications.

Stage Design Pipeline

To explore the concept and design space of (Dis)Appearables, we have implemented a system based on two-wheeled robots, toio, used in HERMITS. By building on top of the previously implemented system, this project focuses on developing a design and fabrication pipeline for Stage. We describe the four-phase design and control pipeline; 1) Design of Stage, 2) Fabrication of Stage, 3) Control of self-propelled TUI on Stage, and 4) Control Execution.

Stage Design UI and Fabrication

First, to allow researchers and interaction designers to plan and design Stage as a platform for self-propelled TUIs, we have developed a GUI-based design tool for Stage using Processing. In this GUI-based software, users can freely design the size of the stage in a rectangular shape, the existence of an underground floor beneath the main stage, positioning of walls, and positioning of portals on the wall and on the floor. The maximum size of the stage was 1260 mm x 1188 mm based on available toio Mat. For the parameters of the portals, it is possible to adjust the types of portals (e.g. ramp or lift for floor portals) and other properties (e.g. direction of ramp).

With this software, users can plan the stage design and preview what the stage would look like with a real-time 3D visualization. After the design is completed, a design file can be exported to be used for the fabrication of Stage, as well as the control UI for the robotic hardware to navigate on the designed Stage. 

Robots Control and Multi-Agent Path Planning Algorithms

Once the stage is fully fabricated, the robots can be placed on the Stages for control. To control the robots on the stage, we have developed two software tools, a Control UI and a Path Planning Tool. The Control UI is an updated tool from the HERMITS' tool to handle the I/O data with robotic hardware via Raspberry Pi Bluetooth Modules. The tool was developed in Processing as in the previous version. We have updated this software with added 3D visualizations which render the detected robot positions in real-time on top of the model of Stage based on the exported design file from Design UI. With this control tool, users can control the behavior of robots using joysticks, keyboards, and other primitive control interfaces. Additionally, the dynamic portals activated by servo motors were controlled with Arduino-based controller which is connected to the control computer with a wire.

Furthermore, we have designed the control UI to communicate with a multi-agent path planning tool for the system to automatically generate paths for multiple robots to navigate on \textit{Stage} from designated origin points to target points.

To manipulate multiple robots on Stages, C++-based path planning tool was developed based on a basic version of Conflict-Based Search (CBS), which is a popular optimal algorithm from Multi-agent Pathfinding (MAPF) literature. [This part of the implementation was the focus contribution by the University of Southern California.]

Applications

Interactive Mobility Simulation

The interactive mobility simulation demonstrates the use of (Dis)Appearables for interaction with digital data. For the stage, the two walls placed on the stage hide the robots / actuated TUIs from the users' view, while the urban mobility simulation is projected on the front-stage (Figure below's a). Supplemental simulation information is projected on these walls for the users. As the user demands, the robots would appear from the backstage to snap to the graphical vehicle data point, so that users can directly grab the vehicle snapped to the data point (c). By reconfiguring the modules in the back-stage, the robots can gain different shapes of module that provides different functionality in the mobility simulation (c, d).

This application well-utilizes the capability of (Dis)Appeaerables by simulating a massive number of moving vehicles in a big city, but in a focused area, by circulating actuated TUIs with limited hardware resources in the background. 

Reconfigurable Desktop

(Dis)Appearable embedded desk can make distracting devices disappear. In the figure, b shows a smartphone which, after being placed on a charger (a), is pulled out of users' reach behind a wall with robots. While the user is focused on an important task in the front-stage without the distraction, the phone can be charged in the back-stage. While researchers and industries work on features to disable notifications to help users' focus, physically disappearing the UI itself from users' reach and view may be more effective. Accordingly, on the physical work desk, appropriate UIs can (dis)appear according to what the users are supposed to focus on (e.g. a multi-functional mouse appears to support CAD operation as in (c) By utilizing pixel to physical transition effect, the CAD model would appear from behind the monitor as if the digital data was physicalized instantly (d). 

Remote Table Hockey

The remote hockey application demonstrates the teleportation and physical-digital transition effects of (Dis)Appearables. As shown in the figure below, the hockey pucks, which are self-propelled robots wearing shells, can transition in between one person's field to another person's field remotely. Following a hit by a person with a paddle, the puck can disappear from one person's field via a wide portal on the wall with a curtain where the real-time image of the other person is projected (a , b). The puck, then, can teleport to another person's side of the field that is visible from the first person through the projected real-time image (c). Even if they are physically separated, they can have such a feeling of remote space with continuity through the virtually connected hockey field and teleporting pucks, which conveys the force and speed of the opponents' hit.

Furthermore, by interchanging the shells in the back-stage, other special effects to make the game more dynamic and challenging can be achieved, such as splitting a puck into multiple pucks (d, e) The paddle can also be composed with the Mechanical Shell with the embedded robot, to dynamically shift the size of the paddle according to the game dynamics (f).

Gaming and Storytelling

 As shown in the figure below,  just like the classic Pac-man, the characters can teleport from one end of the stage to another to represent the continuous connection of the field (a1). The character can also transition to a larger body behind a wall, to chase and beat the ghosts (a2). As for interaction, this could be a physical representation of the original gaming controlled by other interfaces, or direct tangible gaming with rich shape representations that are, for example, studied to train and support motor skills for the elderly.

Another gaming application is a card game. Many card games have employed the concept of ‘summoning', to make monsters or characters appear on the field using their cards. While researchers have developed combining augmented reality to create such summoning expression from cards with graphic overlays \cite{bedoya2014augmented}, the technique of \textit{(Dis)Appearables} would enable the cards to summon physical props. As shown in Figure \ref{fig:pacman}b, as the cards are placed on the floor portal by players (b1), the actuated TUI with an accompanying shell can appear from the floor surface, under the card, as if the physical characters emerged from the card (b2). The dragon shell prototype, as shown in (b3), is designed to transform its shape from the compact box to a larger dragon-shape. While portal sizes in (Dis)Appearables restrict certain sizes and shapes of shells from moving between the front- and back-stages, such expansion transformation capabilities for shells could help to bypass such limitations.

Next Steps and Future Work

We are currently seeking fo potential applications including tangible CAD applications, remote communication, storytelling, and gaming. (We are open to ideas and collaboration opportunities to explore this new research space).

Furthermore, we intend to investigate further through an evaluation to study the illusion and interpretation of the interaction with (Dis)Appearables, and advanced implementation for stage design and control UIs/algorithms.

Bonus Clip from Our Early Experiments ;)