To @AndrewHartAR: “Our app is shit and we know it’s shit”.
My life changed forever when a senior exec from one of the world’s largest retailers messaged me on Twitter. This was back in 2017. My outdoor AR navigation demos had just gone viral, and my new open-source project for Apple had elevated me to be the top trending iOS developer on GitHub. Every developer meet-up I went to, everyone now seemed to know who I was. I was on top of the world.
I agreed to meet the exec for coffee in London, and she told me a story that would change the course of my life. One which I’d hear thousands of times over the next 8 years, from other businesses facing the same challenge. And one which would lead to a breakthrough that could improve the lives of a billion people.
“We’re building this new version of our app — this time it won’t be shit. But there’s one big thing we want to add, and we haven’t figured out a way.”
They wanted to bring indoor maps and navigation to their retail stores, so customers could find what they’re looking for, and they could pop up relevant promotions along the way. It turns out that this doesn’t just apply to retail. Every office, university campus, events venue, hotel, airport, warehouse, factory — basically everywhere indoors have some need to navigate people around, provide relevant information, and improve efficiency.
I assumed this was a solved problem. No. They do have maps on their app, and on their informational kiosks, but those often take months to update so are always out of date. They also aren’t able to navigate you, like GPS in a car, because GPS doesn’t work indoors. They can’t provide the blue dot. They can ask the user to manually update the route as they go, but maps are complicated and the experience is confusing.
I did know something about maps and location already — the first startup I worked at built an early version of Pokemon Go. I’d been tasked with generating the gamified maps, and populating the monsters and rewards. So I knew a bit about maps, coordinates and GPS — and monster training. But indoor navigation was new to me.
“We’ve tried everything out there. Nothing works.”
So I left that meeting and started digging in. Why has this indoor navigation problem not been solved? Over the years, I’ve slowly become an expert in this, so let me explain. It can essentially be broken down into three problems that all intertwine: indoor maps, indoor navigation, indoor location.
Indoor maps and navigation are relatively easy problems. With good mapping tools, it’s very possible to build a digital map and keep it up to date. Indoor navigation is a mix of classic Computer Science algorithms - like A* pathfinding, and “Travelling Salesperson”.
Indoor location turned out to be by far the hardest problem to solve — one of those foundational problems with decades of research behind it. The requirements are clear:
- Needs to be 2x as accurate as a strong GPS signal. An aisle in a store is usually about 2 meters wide, so anything less accurate than 2 meters would be fixing you in the wrong aisle.
- Needs to be scalable. Retailers have thousands of stores, so it needs to be easy to setup and maintain.
There are many research studies you can read on different approaches to this, and Apple and Google have made acquisitions to help them in this area over the years. There were also a few startups who claimed to have solutions. And yet! When I dived in and spoke to their customers, I discovered that they weren’t happy with anything they’d tried:
- Bluetooth beacons. The most common approach. Install thousands of these small sensors, which are a bit like AirTags, and use them for triangulation. But the bluetooth signals are noisy, making the location about 5 meters accurate, so it would jump you between multiple aisles in a store. Plus, this is very non-scalable, with a lot of infrastructure to maintain.
- WiFi. More promising than beacons, because every business has WiFi installed already. But the same radio signals problem means the location isn’t accurate enough.
- Magnetomers, which use the earth’s magnetic field. This one sounded more promising. But it takes several minutes of walking around in order to filter the noise, until it will give you a “blue dot”. So this was a bad user experience.
- Computer Vision, which works like Google Street View. The user holds up their phone to scan the environment, it recognises their surroundings and locks them in. But this is clunky for the user, and they need to do this repeatedly every time they want a location update. Once again, bad user experience.
An example of Apple’s wifi-based indoor location, across a speed run.
I could see for myself that none of these things had ever been able to scale — I’ve never personally turned up to an airport, or a store, and had something like this actually work. It felt like everyone had given up.
It reminds me of how I felt when I first started working with Augmented Reality. There were some clear fundamental technology reasons that AR navigation didn’t work. I remember a clear crossroads moment where I could have either given up, or asked “ok great, so how do I make it work”. Which then led to my viral open-source work, which led to that meeting with the retail executive, which in turn eventually led me to this new crossroads: Indoor location doesn’t work, for some clear fundamental technology reasons. “Ok great, so how do I make it work.”
This time, it would require more resources than my open-source project — a larger team than just me. That’s when I decided to start Hyper, a startup with the mission to build accurate indoor navigation, and scale it to a billion people.
I’m going to skip us forward — past the years of fundraising, hiring, R&D, failed pilots, successful pilots — and tell you how we built a 1-meter accurate indoor GPS.
Indoor Location 101
We started by defining the experience we wanted to achieve: the user should not have to think about blue-dot accuracy, or be confused by the experience. The technology should “just work”. We reviewed the existing solutions, and evaluated which directions seemed promising, and what seemed like a compromise.
- The solution should scale, with no new hardware install.
- The user’s location should appear instantly, without them having to scan the environment or walk around.
- It’s fine to start with a lower accuracy blue dot, such as 3 meters, as long as it quickly hones in to the desired 1 meter accuracy within seconds.
- It should stay accurate and reliable throughout the journey
- And it should provide updated navigational routes, turn-by-turn directions, and AR navigation.
This sounds like a tall order — and it is. We persevere.
We decided to use WiFi as a foundation to give us that initial blue dot.
WiFi location
We start by building a map of signal strength, and how it flows across a venue. This is just like the bars of signal strength you see on your phone, except it’s measuring it for each individual access point.
To collect this wifi data, we would send a member of our team to do an on-site survey. This survey involves walking around a venue, while the phone collects wifi data at each position. The large black dots you see on this visual are where we plotted our position. We’re using our algorithms to track precise motion between each truth point, so we can collect as much WiFi data as possible.
Once we had started to work with large global retailers, although our team loved all the travel, it became obvious that we’d need a better way to scale this process. Ideally, the staff at the store could do this themselves, and we could lower the cost and timelines. So we built a self-serve version of our survey app, with a tutorial mode designed for beginners. Over time, we collected millions of data points, and so we were able to develop an algorithm which would auto-correct mistakes. In other words, if the surveyor accidentally placed their ground-truth location in the wrong place on the map, we could use our algorithms to detect it, and correct it.
So now we have WiFi, and with and our efforts on producing a high quality survey, we have the best WiFi positioning available. With WiFi on its own, it’s achieving 3 meter accuracy. This is a great foundation to build on.
WiFi + Motion data
To refine this down to 1-meter accuracy, we realised that we could combine WiFi with the same technology behind self-driving cars and robotics: a motion system called SLAM (Simultaneous Localization and Mapping). SLAM uses the accelerometer, gyroscope and camera system to understand precise device motion. Imagine a car driving through a tunnel, using the motion since its last GPS ping to keep location accurate until it comes out the other side. On a phone, this technology is very reliable, and measures device motion with high precision.
WiFi gives us the initial blue dot, SLAM gives us motion, and as the user starts walking and we receive more data, our algorithms can refine location accuracy down to 1-meter.
SLAM challenges
Things are never quite as easy as they sound though. SLAM suffers from an issue called drift — where the motion sensors accumulate tiny errors over time, which can shift the user’s location several meters after a minute of walking. We developed a strategy to run our algorithms regularly, anchoring towards wifi data, to correct these errors and keep things stable.
Another challenge is understanding which direction the user is moving. SLAM tracks the user’s relative motion, like “moved forward 2 meters, then turned left”, but its internal coordinate system isn’t mapped to the real world. Does “forward” mean “north”, or some other direction? It simply doesn’t know. We can’t rely on the compass to help us out with this, because phone compasses are notoriously incorrect — everyone knows the frustration of being sent the wrong way down a street. So we designed an algorithm that could simulate every possibility, filter the unlikely scenarios, and hone in your location. As an extra win, we’re able to refine that compass accuracy down to be super precise, so you know exactly which way you’re going.
We’ve spent years developing these algorithms, and testing them on hundreds of hours of ground-truth data. Here’s the final result:
As the icing on top, precise location feeds into our other area of research: AR navigation. This is where I started my journey, with a far more primitive interface, and today we’ve iterated and refined this experience many times to be best-in-class, driven by Hyper-accurate location.
Our Next Act
Most of Hyper’s story still hasn’t been told. One day I’ll explain how we came to put this live in some of the most iconic stores in the world, working secretively for years with the world’s largest retailers. Many thousands of people have experienced our technology firsthand in these stores, navigating with that precise blue dot we dreamed up.
I love that we’ve invested years in R&D, and built breakthroughs that only could have been achieved by a small, visionary startup with high conviction. We’ve achieved the first half of our mission — to build accurate indoor navigation. Now for the second half: how do we scale this to a billion people?
This is a very different challenge from anything we’ve done so far, and perhaps the most exciting. This next chapter will require a new set of skills in enterprise-scale distribution.
Over the years I’ve watched other startups go on this journey — building their technology independently, in a way that only they could have done, then joining a much larger organisation with the commercial muscle and resource to sell to enterprise customers, built for operating in that world. And that’s the journey we intend to go on next.
In the near future, I’m excited to be meeting a variety of businesses to figure out how they can help us take everything we’ve built, and scale it to a billion people, for our next act.