Mobile Search for a New Era: Voice, Location and Sight

Monday, December 7, 2009 | 11:30 AM

Editor's note: today Google held a launch event at the Computer History Museum in Mountain View, CA. Fresh off the stage, we've invited Vic to highlight the mobile team's announcements, and the unique set of technologies that make them possible. (All [video] links point to event footage that will be viewable later today.)

A New Era of Computing

Mobile devices straddle the intersection of three significant industry trends: computing (or Moore's Law), connectivity, and the cloud. Simply put:
  • Phones get more powerful and less expensive all the time
  • They're connected to the Internet more often, from more places; and
  • They tap into computational power that's available in datacenters around the world
These "Cs" aren't new: we've discussed them in isolation for over 40 years. But today's smartphones -- for the first time -- combine all three into a personal, handheld experience. We've only begun to appreciate the impact of these converged devices, but we're pretty sure about one thing: we've moved past the PC-only era, into a world where search is forever changed.

Just think: with a sensor-rich phone that's connected to the cloud, users can now search by voice (using the microphone), by location (using GPS and the compass), and by sight (using the camera). And we're excited to share Google's early contributions to this new era of computing.

Search by Voice

We first launched search by voice about a year ago, enabling millions of users to speak to Google. And we're constantly reminded that the combination of a powerful device, an Internet connection, and datacenters in the cloud is what makes it work. After all:
  • We first stream sound files to Google's datacenters in real-time
  • We then convert utterances into phonemes, into words, into phrases; and
  • We then compare phrases against Google's billions of daily queries to assign probability scores to all possible transcriptions; and
  • We do all of this in the time it takes to speak a few words
Over the past 12 months we've introduced the product on many more devices, in more languages, with vastly improved accuracy rates. And today we're announcing that search by voice understands Japanese, joining English and Mandarin.

Looking ahead, we dream of combining voice recognition with our language translation infrastructure to provide in-conversation translation [video]-- a UN interpreter for everyone! And we're just getting started.

Search by Location

Your phone's location is usually your location: it's in your pocket, in your purse, or on your nightstand, and as a result it's more personal than any PC before it. This intimacy is what makes location-based services possible, and for its part, Google continues to invest in things like My Location, real-time traffic, and turn-by-turn navigation. Today we're tackling a question that's simple to ask, but surprisingly difficult to answer: "What's around here, anyway?"

Suppose you're early to pickup your child from school, or your drive to dinner was quicker than expected, or you've just checked into a new hotel. Chances are you've got time to kill, but you don't want to spend it entering addresses, sifting through POI categories, or even typing a search. Instead you just want stuff nearby, whatever that might be. Your location is your query, and we hear you loud and clear.

Today we're announcing "What's Nearby" for Google Maps on Android 1.6+ devices, available as an update from Android Market. To use the feature just long press anywhere on the map, and we'll return a list of the 10 closest places, including restaurants, shops and other points of interest. It's a simple answer to a simple question, finally. (And if you visit from your iPhone or Android device in a few weeks, clicking "Near me now" will deliver the same experience [video].)

Of course our future plans include more than just nearby places. In the new year we'll begin showing local product inventory in search results [video]; and Google Suggest will even include location-specific search terms [video]. All thanks to powerful, Internet-enabled mobile devices.

Search by Sight

When you connect your phone's camera to datacenters in the cloud, it becomes an eye to see and search with. It sees the world like you do, but it simultaneously taps the world's info in ways that you can't. And this makes it a perfect answering machine for your visual questions.

Perhaps you're vacationing in a foreign country, and you want to learn more about the monument in your field of view. Maybe you're visiting a modern art museum, and you want to know who painted the work in front of you. Or maybe you want wine tasting notes for the Cabernet sitting on the dinner table. In every example, the query you care about isn't a text string, or a location -- it's whatever you're looking at. And today we're announcing a Labs product for Android 1.6+ devices that lets users search by sight: Google Goggles.

In a nutshell, Goggles lets users search for objects using images rather than words. Simply take a picture with your phone's camera, and if we recognize the item, Goggles returns relevant search results. Right now Goggles identifies landmarks, works of art, and products (among other things), and in all cases its ability to "see further" is rooted in powerful computing, pervasive connectivity, and the cloud:
  • We first send the user's image to Google's datacenters
  • We then create signatures of objects in the image using computer vision algorithms
  • We then compare signatures against all other known items in our image recognition databases; and
  • We then figure out how many matches exist; and
  • We then return one or more search results, based on available meta data and ranking signals; and
  • We do all of this in just a few seconds
Now, with all this talk of algorithms, image corpora and meta data, you may be wondering, "Why is Goggles in Labs?" The answer -- as you might guess -- lies in both the nascence of the technology, and the scope of our ambitions.

Computer vision, like all of Google's extra-sensory efforts, is still in its infancy. Today Goggles recognizes certain images in certain categories, but our goal is to return high quality results for any image. Today you frame and snap a photo to get results, but one day visual search will be as natural as pointing a finger -- like a mouse for the real world. Either way we've got plenty of work to do, so please download Goggles from Android Market and help us get started.

The Beginning of the Beginning

All of today's mobile announcements -- from Japanese Voice Search to a new version of Maps to Google Goggles -- are just early examples of what's possible when you pair sensor-rich devices with resources in the cloud. After all: we've only recently entered this new era, and we'll have more questions than answers for the foreseeable future. But something has changed. Computing has changed. And the possibilities inspire us.


hsjinsbsjvbwnhn said...'s nice...
i'll try this one, the voice feature it's kool...keep rockin :)

Gabh said...

What are the limits on this? Will one be able to plug one's Picasa account in, add more personal landmarks? This could be very interesting. Sci fi.

Unknown said...

Will Google Goggles be exposed as a service others can use?

Bertzijngedacht said...

Would love this for the iPhone

Nicholas Tollervey said...

Hi, I'm an Android developer... is there an Android based API I can use for my own applications..? An intent perhaps or remote API I can call..?

Greg said...

Keep changing the world Vic. Google does that much better now than MSFT ever did!

David H. said...

This comment has been removed by the author.

David H. said...

Wow, Goggles is awesome. Products, Contact Cards, etc. all worked great when I tried it.

I even tried taking a photo of a "funny picture" (bycicle rider crashing while somebody is getting himself photographed in front of the State Opera building in Vienna) off my computer's screen, worked as well.

Google, once more you really impressed me. Now only searching for faces is missing although I'm not sure yet whether I should really wish for that.

Sherry Tingley said...

Has anyone realized the potential this can have to the blind population? Maybe not right now, but I can picture that it could act almost like a seeing eye dog could except much better.

Unknown said...

I searched for "Google Goggles" and just plain "Goggles" in the android market but both returned no matches. What gives?

Armin L. said...

how come there is no starring in maps for android? That is a bummer!

Andrew said...

For someone living in Japan I want to offer my deepest gratitude to Google for offering such amazing services for free on your phones, even across the ocean. Looking forward to more surprises in the coming months and years.

Remi van Beekum said...

Good work! This seems to be a very cool new feature!

But I cannot play with it because it doesn;t come up in the Android Market when I search for "Google Goggles" of "Goggles". Is this only available at certain datacenters or something? Or is there some other reason for ths?

Jiří Doubravský said...

this app currently doesnt work on htc hero because it has only old version of android operating system (damn you HTC) ... its only Android 1.5 (not 1.6 or even 2.0) ... thats bad.

Jan said...

Any news on when the voice search will be released for other languages? Cant wait! ;)

Anonymous said...

Dear Google, I love you. Please buy a country somewhere so I can move to a place where everything works!

hungfai said...

hi, i am planning to get the htc tattoo which runs android 1.6 so that i can try to use google goggles. just like to confirm if the software supports the lower resolution on the htc tattoo.

Unknown said...

Just read Google Goggles : Visual Search through Mobile Phone and so curious about it. When this feature is gonna available in Indian market ?

Kevin said...

This comment has been removed by the author.

Nam said...

Absolutely impressive. Leading the way on how to find, collect and display information.

Unknown said...

It seems great. Need to download the Google goggles and see how it works. Nice presentation!

Baerana said...

Sounds like this would replace a project I've been working on - visors that close caption in real time, so deaf/hard of hearing persons can understand speaking people

that would be incredible!!

troy said...

object recognition is the obvious next step, searching from OCR and GPS functions are nice, but the app is really compelling when it can recognize objects through foreground/background segmentation! I have ideas to share.