Understanding Augmented Reality and its applications beyond Gaming
This is the second post in my series about Virtual Reality/Augmented Reality. You can read my first post about VR here.
It’s exciting that the five tech giants Apple, Amazon, Facebook, Google and Microsoft, as well as many promising startups, are exerting a lot of effort and making great progress in augmented reality. The recent release of Apple’s ARKit and Google’s ARCore will play a critical role in taking the technology mainstream. This is the perfect time to learn about AR, beginning with the fundamental concepts, to understand how this technology can help you solve your personal and/or business problems.
Similar to my previous post about VR, let’s discuss AR through these three categories:
- Definition of AR
- Key concepts in AR
- Problem sets that can benefit from AR
AR is often assumed to be synonymous with the overlay of visual content onto a person’s view of the real world, however, other sensory modalities including audio and haptic also can play an important role. This post will focus on the visual AR domain.
What is Augmented Reality?
Augmented Reality (AR) is a technology that overlays computer-generated contents onto a user’s view of the real world. Whereas virtual reality (VR) places a user inside a completely artificial environment, AR presents information that is directly registered to the physical world. This means that virtual information and the physical scene objects are aligned to each other in a coordinate system (in a way it was intended by the creator of the AR application), and the virtual content appears to become part of the real world.
AR is interactive in real time. Schmalstieg & Hollerer (2016) explained that “the user continuously navigates the AR scene and controls the AR experience. The system, in turn, picks up the user’s input by tracking the user’s viewpoint or pose. It registers the pose in the real world with the virtual content, and then presents to the user a situated visualization (a visualization that is registered to objects in the real world).”
(“pose” = position and orientation of the user relatively to the real world)
There are different interaction techniques in AR. Certain AR head-mounted displays allow users to control the virtual object with gaze, gestures, voice commands, and even the ability to touch and grab it with their hands.
Key AR concepts
Tracking
Tracking is a term used to describe dynamic sensing and measuring of AR systems.
To display virtual objects in the real world scene the way it was intended by the creator of the AR application, the AR device needs to have real-time information about the pose of a user, the surrounding environment or any object populating the AR scene. This requires the continuous tracking of such entities by the AR device, particularly through its sensors and cameras.
Tracking is the foundation of any AR experience. This page from Google ARCore is a good resource with simple explanations to help you understand more about tracking and how a well-done AR application “can make virtual contents appear to rest on real surfaces or be attached to real world locations.”
AR displays
Visual AR displays can be categorized according to the distance from eye to display:
- Head-mounted displays
HMD is a display device worn on the head that overlays virtual contents over the user’s view of the real world. The most common display mounting options currently in the market are optical see-though, in the form of eyewear glasses, and can be either monocular or binocular. AR headsets can also come in the form of a helmet, such as the Livemap motorcycle smart helmet.
- Handheld displays
Handheld displays employ small computing devices with a display that the user can hold in their hands, such as smartphones and tablets.
- Stationary displays
Stationary displays are displays that, as their name implies, don’t move during their typical use. Common AR systems in this category typically use computers that have a built-in or tethered camera or webcam to track target objects. Examples includes desktop displays, kiosk displays (e.g., Lego retail augmented reality kiosks), and mirror displays.
- Projected displays
In this type of applications, projectors are used to create spatial AR without any explicit displays. The projection is directly rendered on the surfaces of real objects. The projection cannot change the shape of the object, but adds surface details and dynamic visual effects, and can automatically fine-tune alignment when the object moves.
Collaboration
Collaborative AR enables multiple users to simultaneously experience an augmented environment. There are two types of collaborative AR situations: co-located collaboration and remote collaboration.
- Co-located collaboration
In co-located situations, where multiple users are simultaneously present at the same location, the users can perceive and interact with virtual objects together.
- Remote collaboration
The most popular remote collaboration concept in AR is application that allows remote users to connect via live video and be able to draw annotations directly in the synchronized video stream.
In the above example, a worker (left) with a mobile AR device streams video to the remote expert. The remote expert (right) can give instructions by drawing annotations directly into the mobile worker’s view. This kind of setup is suitable for various kinds of use case involving remote expert consultation.
AR also allows remote users in two different locations to interact with each other as if they are actually present in the same physical place.
Microsoft’s “holoportation” is a new type of 3D capture technology that allows a 3D model of the remote user to be reconstructed and transmitted to the local user’s real world in real time. When combined with HoloLens, the technology allows the local user to see, hear and interact with the life-sized holographic representation of the remote user in their real world, as if they are co-present. This helps the remote collaboration become as natural as face-to-face communication.
Types of AR application based on working approach
- Marker-based AR
Markers are known patterns placed on the surfaces of target objects, or are known trackable shapes attached to the target objects. In other words, markers are images encoded with information that’s translated by complex software to produce a desired visual effect. When an AR device scans such marker, it recognizes the pattern and overlay virtual objects into the scene.
- Markerless AR
In contrast to marker-based AR, a markerless AR application does not need any pre-knowledge of a user’s environment to accurately overlay virtual content into a scene. The AR device performs active tracking and recognition of target objects and the surrounding environment without needing to read a prepared marker. Such ability allows complex AR applications. Examples include any application that utilizes 3D object recognition to overlay visual effects such as Snapchat, and application that overlays virtual objects in a user’s space such as IKEA Place.
Types of problem that can benefit from AR
Because of the variety of key AR technology elements, as you can see from the previous sections, and the ability to combine AR and other advance technologies, such as Artificial Intelligence and Internet of Things (IoT), AR enables companies and individuals to solve many types of problem in a new and more effective way. Many times that are also problems that cannot be tacked in the physical world. Below are selected use cases.
1. Hands-free and Interactive information displays
Hands-free information displays
AR can present information and instructions directly superimposed in the field of view of a user. This allows a dynamic hands-free work style, which help reduce errors and optimize processes in many professions. It also helps provide more effective training, but more importantly, allows personnel with less training to correctly perform the work.
The hands-free information display application in AR can also help revolutionize the healthcare industry, as you can see in the following example:
Interactive information displays
Users can use AR to superimpose virtual content where it matters most to them and be able to interact with the virtual content in real time. This allows the users to present and learn a new subject more effectively.
Different brands and organizations also have been utilizing AR to create engaging marketing experiences for a long time. Below is the collection of some very cool AR advertising campaigns:
2. Design and Planning
Architects and designers can use AR devices to prototype in 3D and understand their creations in relation to the real world. This allows them to push the boundaries of their creativity, make better decisions and work faster.
For large construction planning, AR offers the opportunity to perform on-site inspection with holograms, bringing the CAD model to the facility in real time. This allows architects and designers to immerse their partners into the vision of what they are building and can significantly reduce cost and optimize the construction design and planning process.
3. Remote experts
As mentioned previously, AR allows remote users to connect with each other via live video and be able to draw annotations directly on the shared live video stream. This is a critical advantage in collaborative tasks that involve the physical world, such as equipment maintenance and repair, healthcare emergency, with no preparation of the environment required.
The video below demonstrates a scenario using remote AR technology to troubleshoot issues in an IT environment.
4. Navigation
AR is also used for navigation applications. The difference between regular GPS devices and AR navigation devices is that the AR device project navigation directions right on the user’s field of view. For drivers, this means they can fully focus on the vehicle windscreen and not have to look at other devices for directions, which enhances road safety.
5. Customer buying experience
AR allows customers to virtually try on products, such as cosmetics and clothes, to see how the products would look on them. It also can help customers see how furnishings would fit into their space before placing orders. This makes the shopping experience, particularly online, much more convenient for the customers. It also can help reduce merchandise returns for retailers, which, according to the National Retail Federation, represented a $260B problem during 2015 — in the US alone.
And AR does not just enhance the retail industry. Any industry that can utilize AR to help improve customer experience and assist customers with making decisions easier during the buying cycle will greatly benefit both buyers and sellers. In the above use case, designers and architects can utilize AR to help clients visualize project concepts on-site at the planning step.
6. Virtual workspace
AR headsets with powerful computing hardware, such as Microsoft HoloLens and Meta, allow you to work on office applications just like your desktop computer, but in 3D holograms. This means that a lot of work you have been doing in 2D at your office now can be done virtually in holograms and you can work with the virtual contents where it is most effective and/or most convenient for you (think about presenting complex data and concept in 3D at your client’s workplace). And San Francisco startup Meta wants to replace your office computer monitors, keyboards, and eventually even cubicles, with augmented reality.
Nothing beats a great live product demo, so check out the video below, which is a presentation about Meta’s AR Workspace by Meron Gribetz (CEO and Founder, Meta).
This picture is from a presentation by Edward Tang (Co-Founder and CTO, Avegant) at Augmented World Expo USA 2017. Tang compared the current state of the AR industry to what happened in the mobile phone space. I think it is an excellent analogy and wanted to share it with you.
The AR industry is still in its infancy, but plenty of exciting developments and innovations are coming soon. I hope this post helped you understand more about the technology and its potentials, and hope you will also feel excited about the future of AR and the transformation that AR is certain to bring to our lives. I will continue to share my research about VR/AR on this blog, so please stay tuned!
