What Is Augmented Reality? Your Guide to Immersive Technologies

What Is Augmented Reality?

With all the technology that goes into an AR experience, the definition of augmented reality is pretty straightforward.

Augmented reality (AR) is a computer-enhanced version of the physical world, with digital content used to amplify the user experience of reality.

Digital content in AR can range from simple graphics and animations to videos and GPS overlays. The rendered content can “feel” and respond to the user’s environment and the user can also manipulate the content (via gestures or movement, for example).

Historical Overview of Extended Reality and AR

1838: Stereoscope

In 1838, Charles Wheatstone invented the stereoscope, the first attempt to render three-dimensional objects.

Fun-fact: Google Cardboard’s design is based on the stereoscope.

1957: Sensorama

In 1957, Morton Heiling invented Sensorama. Although very crude compared to today’s AR technology (and not controlled by a computer), Sensorama offered viewers a blend of sounds, visuals, motion stimuli, and smells.

Heiling also invented and patented the Telesphere Mask in 1960, which is hailed as the first head-mounted display.

1968: “The Sword of Damocles”

Ivan Sutherland invented another early version of a head-mounted display. The device, which looked more like a sophisticated torture device than tech innovation, paved the way for other cross-reality solutions.

1974: Myron Kruger

Myron Kruger built Videoplace — a great example of projection-based augmented reality, where users didn’t need to wear a device to experience augmented reality.

1986: AR in IBM

In 1986, Ron Feigenblatt described the most commonly experienced version of augmented reality — mobile AR. Feigenblatt called the device through which users could display digital content the“Absolute Display Window Mouse/Mice.”

1990: The Term Augmented Reality Takes Shape

Over 150 years after the first “AR” goggles, Thomas P. Caudell of Boeing coined the phrase “augmented reality.”

1992: AR Telerobotics

Louis Rosenberg developed Virtual Fixtures — the first immersive AR system built at the United States Air Force Research Laboratory. Virtual Fixtures injected digital content into the real world, but the experience was unlike what we associate with AR today.

The image the user saw was an optical illusion, not a 3D rendering.

Virtual Fixtures was also an example of telerobotics — the user was connected to a robot via an exoskeleton that controlled the robot’s movements.

1999: Battlefield Augmented Reality System (BARS)

The US Naval Research Laboratory launched a decade-long research project to prototype wearable systems for soldiers to provide better situational awareness in urban environments.

2013: Google Glass

While Google Glass didn’t revolutionize the world of AR, it was a big step forward into decreasing the size of immersive tech devices.

2016: Pokemon Go and Microsoft HoloLens

Even if you hate Pokemons, this mobile augmented reality game has a solid place in the history of AR.

You can consider this year as the full-fledged introduction of mobile AR into the mainstream.

Now: Explosion of Capabilities

The AR technology is pushing the boundaries of what’s possible to achieve. Multiple tech giants are working on their AR devices.

Many companies across industries begin to recognize the potential in AR to improve workflows and processes.

Beyond: Who Knows?

With Elon Musk’s Neuralink eagerly exploring the brain-computer interface, in a few years we might see implantable chips that project augmented reality content directly into our brain.

The direct stimulation of the visual cortex paired with external sensors and computing units would allow us to “see” more of the world around — the digital and physical content would blend in a mind-bending immersion.

How Does Augmented Reality Work?

All right, so you know that augmented reality adds or expands the real world with digital content.

To correctly place digital objects in your environment, AR needs to employ several hardware and software solutions — there’s some serious computing taking place in augmented reality.

But depending on the type of augmented reality and the device used for AR rendering, different tech is involved.

What Devices Use Augmented Reality?

Here are some examples of AR devices:

• Head-mounted displays (or headsets) — HoloLens2, Magic Leap One
• Projection systems
• EyeTap
• Spatial Augmented Reality (SAR) — projector-based devices
• Eyeglasses — Facebook Ray Ban, Vuzix Blade
• Contact lenses — Mojo Lens
• Mobile phones
• Tablets — iPad (with Apple’s take on LiDAR Scanner)
• Virtual retina displays — Smartglasses Light Drive

As you can see, that’s somewhat of a crowd.

Now check out what needs to be used in augmented reality programming to render AR experiences. The below are just some of the tech available for AR development:

What Hardware Is Used for Augmented Reality?

• Processor
• Display
• Cameras
• Sensors — accelerometer, GPS, compass, eye tracking, gyroscope, microphone

What Software Is Used for Augmented Reality?

• Software development kits (SDK) — environments for developing AR applications. In mobile augmented reality development, these include:

• ARCore (Google)
• ARKit (Apple)

• Various algorithms that help different AR types understand the environment and render computer-generated virtual objects. For example, visual odometry, semantic image segmentation, object detection, or 3D reconstruction
• 3D engines such as Unity
• Cross-platform SDKs for mobile AR apps: Vuforia

How Does Augmented Reality Work on Mobile?

Let’s say you want to see an AR alligator in your room. You can view it via Google Chrome on your Android phone. Just type “alligator” in the search bar, and hit View in 3D.

Step 1. The cameras and sensors in your phone will probe your room. Using tech solutions such as depth-tracking and SLAM (simultaneous localization and mapping), the phone maps your environment.

Step 2. Processors and computing units analyze the data and calculate how the alligator is to be placed respective to your environment.

Processing units are similar to those found in computers, i.e., they include a graphics processor (GPU), central processor (CPU), RAM, and flash memory. That hardware calculates angles, orientation, direction, and speed.

Step 3. The display — in this case your phone screen — renders the image of the alligator.

You can view the alligator from all angles — processes from all three steps repeat with every frame to render an accurate 3D model.

The processing happens in milliseconds, which our brains are incapable of discerning — hence why we don’t see any delays in the rendering. On average, phones view 30 frames per second. Yup, that’s how quickly AR computes.

Which Phones Support Augmented Reality?

List of phones that support ARCore (Android AR).

List of phones that support ARKit (iOS AR) scroll down to the bottom of the page.

Types of Augmented Reality

For the sake of simplification, we can divide augmented reality into two types: marker-based and markerless. Both can be further subdivided.

Marker-based augmented reality apps need a trigger to render a 3D model.

Markerless augmented reality apps don’t need any visual cues to render AR experiences.

The alligator AR experience from above is an example of markerless AR.

With these two briefly explained, we can now branch off into other subtypes of augmented reality.

Location-Based AR

Probably the best illustrative example for location-based AR is the Pokemon Go mobile app game.

Location-based augmented reality uses GPS to match the digital content to the user’s real-world environment. As the user walks around the real world, the content appears based on predefined GPS markers and is rendered to fit into the real world.

Projection-Based Augmented Reality

Projection-based augmented reality is a large-scale solution, where the rendered images are projected on a surface, without the need for the user to wear any special gear.

Just like other augmented reality types, projection-based AR can also be interactive, provided sensors and cameras recognize gestures.

Superimposition AR

As the name suggests, superimposition-based AR replaces the real-world view of an object with digital content.

Image recognition algorithms help this AR type recognize objects to properly match the digital content.

Augmented Reality on the Spectrum of Extended Reality

In recent years, we’ve been pelted by multiple terms used for cross-reality technologies.

We have augmented reality, virtual reality, mixed reality, and many immersive realities in between.

All these subsets fall under extended reality (XR) — an umbrella term for all solutions where the real world blends with virtual elements.

You can think of extended reality as a spectrum.

On the one end of the spectrum, you have the real world.

On the other end, there’s virtual reality — i.e., where the real world is entirely replaced with a virtual rendering.

In between the two, you have different flavors of augmented reality — i.e., technologies where the real world is enhanced with digital content.

What Is Virtual Reality (VR)?

Virtual reality plunges the user into an entirely projected environment — in other words, the view of the real world is replaced with computer-generated content.

Learn more about virtual reality in our in-depth comparison article: Augmented reality vs. virtual reality.

Augmented Reality Apps

With all the theory checked off, let’s check out some of the most exciting augmented reality apps out there.

Mondly

Mondly helps users learn 33 languages via adaptive learning, which customizes the curriculum based on user progress. The app offers AR-supported conversations, pronunciation advice, and advanced statistics to improve language learning and knowledge retention.

Mondly also features augmented reality lessons where you can view animals and objects in your space for deeper engagement and better experience.

An augmented reality language lesson. Source: Mondly

80's AR Portal

Now this is a hugely entertaining AR mobile app that rubs my imagination in the right way.

Anyone a fan of Stranger Things, here? The 80’s AR Portal gives you a chance to immerse yourself in a fun experience and feel the crazy retro-cosmic spirit of the 80s. It’s definitely not a perfect app, but the concept is alluring.

ARLOOPA

ARLOOPA is an educational app with a flair for entertainment. With the app, you can view multiple AR models in your space and engage with them.

Interestingly, the app has three different types of AR available for you to experience: marker-based, markerless, and geo-location.

Human Anatomy Atlas 2021

The Human Anatomy Atlas 2021 is one of the best augmented reality examples out there. Excellent for medical students and health enthusiasts, the app features some of the most detailed human anatomy models, with tissues, muscles, bones, and nerves. All supported by interactive lectures.

And you can view all those fancy body parts in AR.

Bringing immersive technologies into the healthcare industry can help doctors explain complex procedures to patients and serve as an excellent reference during reconstructive surgeries.

We talked in detail about augmented reality in healthcare in one of our articles.

IKEA Place

No respected AR app list can go without the IKEA home decor app.

IKEA Place lets users render augmented reality furniture in their living spaces. The app views real-size furniture, letting homeowners better visualize the items from IKEA’s catalog before deciding on a purchase.

For home decor, there’s also the amazing Houzz app, which got the Editor's Choice award on the app store.

Shapr3D

Shapr3D is an award-winning app for 3D modeling workflows. It’s a professional CAD (computer-aided design) tool with robust capabilities. A work-horse for designers.

With the expansion of the app to macOS, the Shapr3D iPad users can now collaborate on prototypes across devices.

In 2020, Shapr3D got Apple’s Design Award (the first CAD application to get it).

Canvas

Canvas is a house scanning AR application that uses Apple’s raw lidar data to create interior CAD models and floor plans.

The improved depth-mapping properties of LiDAR technology result in more accurate models (the developer claims a 5x increase in accuracy compared to the non-lidar app version).

Where Is Augmented Reality Used?

Wondering what is AR used for aside from games, lifestyle applications, and entertainment?

Companies can use AR as a catalyst for process and workflow improvement across a variety of industries.

• Manufacturing
• Retail
• Healthcare
• Education
• Travel
• Automotive
• Construction

We wrote extensively about different use cases for augmented reality in our article.

Augmented reality is an innovative and effective tool to enhance several areas in businesses of all kinds.

AR can help:

• Boost brand engagement
• Improve customer experience
• Streamline workflows
• Increase employee retention

Future-thinking companies are eagerly implementing augmented reality in their projects and business operations.

For example, Lockheed Martin uses AR technology in manufacturing, assembly, maintenance, and sustainment. With AR, the company decreases information overhead present in the manufacturing process and optimizes processes.

Lockheed Martin also used AR when building the Orion spacecraft for NASA’s Artemis II — Orion’s first crewed mission.

Privacy and Security Considerations for Augmented Reality

While the idea of augmented reality and the opportunities it brings are alluring, this technology doesn’t come without issues. Specifically, privacy and security issues.

As AR devices and software evolve and become increasingly mainstream, topics such as ethics and legality begin to pop up during conversations regarding augmented reality.

The most important questions are:

• Who stores the data?
• What do the privacy policies state?

The thing is, AR relies on a number of sensors and cameras. Many devices, such as HoloLens 2 or Magic Leap, additionally track eye movement — in essence, they know what you’re looking at.

Now, with this hugely intimate knowledge — I mean knowing what you’re looking at is just a step away from knowing what you’re thinking — insight into our motivations and behavior becomes invaluable for companies.

From a purely marketing standpoint, who wouldn’t want to know all the minuscule details that make our decisions?

So there’s that.

However, not all is lost, and companies digging into the field of AR are trying to first figure out all the implications of this nature before actually releasing their devices into the market.

For example, Facebook has created Project Aria — a whole department strictly for analyzing privacy and ethical implications of the future AR glasses made by the company.

Since AR devices are in essence connected devices, just like IoT, they rely on the same connectivity protocols. As a result, this also generates the more “traditional” security concerns:

• Poor (or lack of) encryption
• Firmware vulnerabilities
• Third-party threats

Any AR-enabled assets have to be included in risk assessments and audited regularly to ensure access to sensitive data is protected.

The Future of Augmented Reality

Even though the adoption of AR applications hasn’t yet gained significant traction — still, 26% of US adults have used AR — the sophistication of devices and the drop in price for hardware can soon increase the pace of adoption.

Various sources predict the global AR market to reach billions of dollars in upcoming years. As use cases of AR expand, spreading across industries and sectors, AR-enabled devices emerge as must-have tools for experiencing new modes of entertainment and exploring new ways to collaborate and work.

Tech giants are in an arms race toward developing increasingly advanced AR devices that bring this technology closer to users, at an affordable price. 2021 and beyond will be pivoting in the realm of augmented reality, with Facebook, Apple, Microsoft, Google, Nvidia, Vuzix, and many more pioneering the AR technology.

The key to conquer the market early will be affordability, accessibility, convenience, and immersiveness. With that ticked off, augmented reality applications on mobile devices will become an invaluable extension of reality, paving the way for other AR devices.