Google's New Street View Cameras Will Help Algorithms Index The Real World

A big jump in resolution will help software understand street signs and business names.

Steve Silverman helped build cameras for two NASA rovers that went to Mars. In the less exotic landscape of a Google parking lot, he looks up fondly at his latest creation, bolted onto the roof of a Hyundai hatchback. The gawky assemblage almost doubles the car’s height: four white legs holding up a vertical black stalk sporting eight cameras. “We thought about covering it up, but we’re kind of nerds,” Silverman says. “We’re proud of it.”

Silverman and his team build the hardware that captures imagery for Google Street View, the project that since 2007 has put panoramas of more than 10 million miles of roads, buildings, and the occasional act of public urination online for all to see. The new camera design, the first major upgrade in eight years, started regularly patrolling the streets last month. The data that’s just starting to come back will strengthen Google’s digital grip on the world.

As you might expect if you think back to the camera in your 2009 cell phone, Street View imagery is about to get a lot clearer. Look forward to sliding through the world from your couch in higher resolution and punchier colors. But Google’s new hardware wasn’t designed with just human eyes in mind. The car-top rig includes two cameras that capture still HD images looking out to either side of the vehicle. They're there to feed clearer, closer shots of buildings and street signs into Google’s image recognition algorithms.

Those algorithms can pore over millions of signs and storefronts without getting tired. By hoovering up vast amounts of information visible on the world’s streets—signs, business names, perhaps even opening hours posted in the window of your corner deli—Google hopes to improve its already formidable digital mapping database. The company, built on the back of algorithms that indexed the web, is using the same strategy on the real world.

Global surveillance

The idea behind Street View is nearly as old as Google itself. In 2001, three years after the company’s founding, CEO Larry Page took a videotape he’d shot driving around the Bay Area into Stanford’s graphics lab. He asked researchers there to figure out a way to summarize it in images, and they began a project dubbed “crawling the physical web.” Its technology was absorbed by Google in 2006, when the company’s cars first hit the roads ahead of Street View’s public launch the following year.

A decade later, Street View cars have snapped more than 80 billion photos in thousands of cities and 85 countries. The company’s conventional mapping data is even more extensive. But Google still hungers for a better index of the world. Jen Fitzpatrick, the vice president who heads the company’s maps division, blames that on us. “People are coming to us every day with harder and deeper questions,” she says.

The first time you searched Google Maps or Street View you probably typed in a street address—perhaps your own. Fitzpatrick says the company now gets tougher queries that require a fresher, more detailed digital model of the world, like “What’s a Thai place open now that does delivery to my address?”

She wants her service to handle queries that assume knowledge of what the world looks like: “What’s the name of the pink store next to the church on the corner?” Google’s push to get us talking with its Siri-style virtual assistant encourages us to be more conversational in our demands. “These are questions we can only answer if we have richer and deeper information,” Fitzpatrick says.

Google’s huge investment in machine learning and AI provides a natural way to get that information. Thanks to recent research inside the maps division, when a Street View car captures photos of a stretch of road, algorithms can now automatically create new addresses in the company’s maps database by locating and transcribing any street names and numbers. Street View was the first of Google's product groups to use the company's powerful custom AI chips, dubbed TPUs.

The team's system has learned to figure out abbreviations, such as “AV.” for avenida, by taking hints from other signs in the country where they’re spelled out in full, and other clues in Google's maps data. Software has also been trained to recognize business names, and is smart enough to ignore visual trip hazards like the giant Bridgestone logo that might dwarf the name of a tire shop.

Higher quality images coming from the new hardware now atop Google’s Street View vehicles will allow those systems to extract information like that more reliably. “From a machine learning perspective, everything gets better,” says Andrew Lookingbill, an engineer working on the technology. It will also help his team’s efforts to build new software even better at understanding the world. They’re thinking about trying to automatically recognize different types of business from their appearance and reading finer-grained information like opening hours signs.

Google
New territory

Decoding Street View imagery with algorithms can be especially useful in places where roads, cities, and businesses are changing fastest—the less-developed economies where Google and its competitors hope to find their next few billion users. The government of India reported this year that it has recently laid an average of 14 miles of new road every day. Street View went live this summer in Nigerian megacity Lagos—population 21 million. Fitzpatrick says that Google’s image-scouring algorithms could help translate the new imagery into a significant bump in map quality. Google sells ads inside maps, so new coverage and accuracy can translate into more revenue if they draw new users and usage to the service.

Google wants you to help feed its image-hungry algorithms. The tech industry’s recent interest in virtual reality has made 360 degree cameras relatively cheap. This summer, Google began certifying some cameras as “Street View ready,” meaning you can upload your own panoramas through the Street View mobile app to live on the company’s service. That footage will be processed by Google’s image recognition algorithms for fresh map data just like its own imagery.

Google is counting on crowdsourcing to make Street View data fresher than it is now. “The expectation is that Google has the world indexed,” says Charles Armstrong, a product manager for Street View. “But it never lives up to expectations.” Google’s Street View mobile app rewards individuals who contribute photos with virtual trophies, and it will even suggest local spots to take your camera. More significantly, Armstrong predicts companies, tourist boards, and even governments will soon be driving their own camera-toting cars to make sure the world gets an up-to-date view of their streets and cities.

All the upgrades to Street View could help Google maintain its prime position in digital maps. The company is the most prominent among the handful of leading global mapping projects. The other heavy hitters are HERE, owned by a coalition of German auto companies; TomTom, known for stand-alone GPS units and watches; and collaborative project Open Street Map. “Each one measures themselves against the others,” says Alyssa Wright, president of the US chapter of Open Street Map. (Apple’s relatively young mapping operation licenses data from TomTom.) In a world where most of us carry GPS-equipped smartphones, maps data is important for much more than just directions. “Mapping is fundamental to how we build our digital future, from autonomous vehicles to dating apps,” says Wright.

Street View’s new cameras and Google’s push for crowdsourced imagery could also lead the company into new privacy controversies. Concern about Google making ephemeral public scenes into permanent internet fixtures has rumbled, and occasionally flared, since Street View began. Germany and Austria are largely invisible on Google Street View, and have been for years, after the company got in trouble for logging Wi-Fi data with Street View vehicles. Google's fleet only recently returned to both countries. In 2012, Switzerland’s highest court ordered Google to cut down its cameras to prevent them from peeping over walls and to blur certain places such as women’s shelters.

Fitzpatrick flicks away the suggestion that higher quality imagery could lead to more privacy concerns. “We haven’t seen or heard of places where there are additional sensitivities,” she says. Google will continue to automatically blur faces and license plates on its own Street View images. But it won’t do that by default on crowdsourced footage, instead leaving it up to users to choose whether to use Google’s blurring technology when they upload new 360 photos.

How much more could Google extract from Street View using image processing algorithms? A lot.

Earlier this year Stanford researchers, including professor Fei-Fei Li, now chief scientist at Google’s cloud division, showed they could predict income, race, and voting patterns for US cities with software that logs the make, model, and year of cars in Street View photos. When asked if anything like that was planned at Google, a spokesperson would say only that the company is always looking for ways to use Street View data to improve the company's platforms, including beyond maps.

Processing Street View images from Google and its users might also help the self-driving cars of fellow Alphabet subsidiary Waymo understand the world. “The team collaborates on things from time to time,” is all Fitzpatrick will say about that. But her team has as much to gain in return from Waymo.

Back in that Google parking lot, camera-wizard Silverman confesses that trolling highways in a Street View car sporting one of his team’s devices isn’t much fun. “After a day you’re ready to not be a bus driver and go back to engineering,” he says. Just as self-driving vehicles would shift the economics of on-demand ride services, not having to pay people to be bored behind the wheel would be a boon to Street View. The company’s algorithmic index of the physical world may just be getting started.