Remote Assist App
Overview
We’ve built a prototype of a mobile app to show how technologies such as AR and WebRTC for mobile and web can be used to solve problems over distances.
Augmented Reality Devices for Remote Assistance
The global number of smartphone users in 2021 was 3.8 billion. Out of that number, there’s an estimated 2 billion devices with AR capability (1.25 billion running iOS and 851 million with Android), which makes smartphones an excellent and highly affordable vehicle for augmented reality.
With so many AR-enabled smartphones, access to mobile remote assist becomes significantly greater, giving everyday users a convenient tool that lets them easily connect with specialists (or their friends) to get advice.
Remote Assist App brings together two people: one in need of help, the other willing to provide it. When a person in need of help gets connected with an assistant, the assistant immediately sees the user’s environment and can point at specific elements in that environment.
Help centers could use Remote Assist App to instruct customers how to troubleshoot problems with a router or other appliance. This could potentially decrease the time to solve the problem and eliminate many friction points that occur when the assistant can’t see the device in question.
Building Remote Assist App
Building Remote Assist App was a highly demanding task and an inspiring challenge. During development, we encountered many issues:
- Overcoming WebRTC's entry barrier. We had to invest some time to understand what’s going on under the hood. We wanted to acquire an in-depth knowledge of WebRTC to be able to use it according to our requirements. That's why we resigned from ready-made solutions such as Twilio.
- Ensuring compatibility and stability between iOS, Android, and web. WebRTC is a flexible technology with many codecs — we learned how to configure WebRTC to make the system efficient and compatible across platforms.
- Connecting AR with WebRTC. An optimal image transfer for a smooth frame to frame transition and stability of the app.
- Making the app more UX friendly. We learned how to design the conversation screen and take advantage of the whole user screen.
Detailed application breakdown
We built the mobile apps for different platforms in Flutter. We wanted to rely on Flutter entirely but none of the existing modules met our expectations with AR quality. This led us to writing our own native modules responsible for the connection between users and AR drawing.
The web client is written in Next.js — it’s perfect for building efficient prototypes quickly.
All that is supported by our backend. The backend was written in Ruby on Rails, and it’s responsible for packet handling between users.
Remote Assist App use cases
Remote training
Remote Assist App helps teachers provide better learning opportunities for students by holding lessons everywhere, without the need for physical contact. By using the app, teachers simply walk the students through the subject in question, pinpointing elements that need more explanation.
Machinery repairs
Remote Assist App fits into contexts where the help of skilled engineers is needed on-site, but they are hundreds of miles away. With the assist app, engingeers can instruct on-site workers and guide them through the fixing process of highly specialized machinery.
Car service
Cars increasingly rely on electronics and computer-operated maintenance. Car shops may often lack the specialized workforce to perform repairs according to the manufacturer's requirements. Remote Assist App is an easy way through which mechanics can concult with engineers and service a car.
Call centers and technical support
Call center operators have a carefully designed guidelines on how to troubleshoot customer problems. However, they lack a visual on the problem that could let them pinpoint the exact problem faster and improve issue resolution times. Remote Assist App lets customers show the problematic hardware and apply the consultants advice faster.
What Is Remote Assist App?
The app is simple and intuitive — it doesn’t require login, email confirmation, or phone number. Everything is anonymous: user pairing happens when the user shares a code with a consultant.
After launching the app, the user can choose the mode:
- “I need a consultant’s help” — this mode generates a code that the user has to share with the consultant. The user also shares his back camera view with the consultant.
- Consultant mode — the consultant assists and helps the user solve a problem. The consultant enters the code the user shared with him and shares the front camera.
After giving the code, the user starts the connection by clicking “start a call” and waits for the consultant.
After the connection is established, the consultant and the user can hear each other. Both can see the user’s back camera view. They can draw in their assigned color and place 3D objects on detected surfaces.
After the call, the user can rate the service and give feedback on the app.
View prototype in Figma
