Pier 39 is a popular spot for sea lion watching in San Francisco, the city I currently live in. However, when I brought my relatives who were visiting, we were disappointed to see that there were not that many sea lions at the pier that day. My five year old niece was especially upset by this turn of events. While consoling her, I wondered if there was an alternative way to seeing the sea lions in real life, in case they did not want to hang out near the pier on a particular day.
Empty dock :(
Because there are posters about sea lions at Pier 39, I decided to create an AR picture dictionary that would show a 3D model of a sea lion when the user interacts with the word "sea lion" on the posters. Ideally, the device for the application would be AR glasses, but since I had an iPad available, I chose it as my device of choice.
I had some previous programming experience, but this was my first time using RealityKit. The reason why I chose RealityKit and Swift over Unity is because the AR devices I own are an iPad and iPhone, and I knew that an AR experience on a smaller phone is unpleasant. To show the sea lion model on the screen, I used video-passthrough, where the camera feed is displayed on the iPad and features are augmented onto the screen. For this use case, RealityKit detects a horizontal plane and places an anchor (an entity that attaches virtual entities to the real world scene) on that horizontal plane. Then, virtual objects are attached to that anchor entity. I was able to utilize the ample resources online to learn about RealityKit and Swift, but this was the stage that took the longest for me. Nevertheless, I am proud to have learned Swift and built a working AR application from scratch as a side project.
Many nights were spent working on this
Because I wanted to have a model of a sea lion in my application, I found a model of the animal in USDZ file format online after lots of searching. Assuming that AR continues to grow with the same pace as now or even faster, I would expect to be able to find more high fidelity models in the future.
What the model looks like
My application had three stages where gesture interactions were needed: the sea lion model appearing, rotating, and disappearing based on user input. To designate the most user-friendly gesture interactions, I read various articles and publications that discuss this topic. Finally, I was able to narrow down to four gestures: tapping and pinching for the appearing and disappearing stages and swiveling and panning for the rotating stage.
Tap to appear
Un-pinch to appear
Swivel to rotate
Pan to rotate
In order to explore which gestures were the most appropriate, I recruited a friend to participate in a user study (with ice cream as bribery). I brought her to Pier 39 and asked her to interact with the AR application using the four narrowed down gestures: tap, pinch, swivel, and pan. After going through all the steps, the participant was asked which gestures were easiest and made the most sense. Here are some of the notes from the interview:
• Tapping was the most intuitive for the appearing and disappearing of the model.
• Swiveling makes sense for the rotating of the model. However, participant has carpal tunnel and thought swiveling was a difficult motion.
• Panning is preferred over swiveling because it is easier and still intuitive.
Thus, I found that tapping and panning were the easiest and most intuitive gestures for the AR Picture Dictionary. Below is a video of how the application works with those two gestures.
Overall, I was very pleased to have designed and developed an AR application on my own. I was also pleasantly surprised to find that RealityKit and Swift have ample resources online that can guide anyone to build an AR application. I still have a lot to learn in both designing and developing mixed reality applications, but this project reaffirmed my interest in mixed reality and the potential of this technology. If I had more time to work on this project, I would consider including other ways to interact with the 3D model, such as moving, scaling, or animating the model. Of course, I would consider which gestures to use for each of those interactions as well.
Testing the design is always crucial