Facebook CEO Mark Zuckerberg has built himself quite an impressive home-automation system, called Jarvis (of Iron Man fame), using Crestron as the connectivity hub, but adding a wealth of services from Facebook, along with a learning mechanism for artificial-intelligence (AI).

On the most recent update of his Jarvis smart-home project, Zuckerberg hints that he or Facebook might turn Jarvis into a product or platform for the mass market:

Finally, over time it would be interesting to find ways to make this available to the world. I considered open sourcing my code, but it's currently too tightly tied to my own home, appliances and network configuration. If I ever build a layer that abstracts more home automation functionality, I may release that. Or, of course, that could be a great foundation to build a new product.

Here are a few nuggets from Zuckerberg’s project.

Facebook Platforms

Facebook has amassed quite a few services that make useful tools for AI and home automation.

For example, the increasingly popular Facebook Messenger service is used in Jarvis to send and receive commands from the smart-home system:

I started off building a Messenger bot to communicate with Jarvis because it was so much easier than building a separate app. Messenger has a simple framework for building bots, and it automatically handles many things for you — working across both iOS and Android, supporting text, image and audio content, reliably delivering push notifications, managing identity and permissions for different people, and more. You can learn about the bot framework at messenger.com/platform.

Facebook’s facial-recognition engine also proved useful to Zuckerberg, who installed multiple cameras at the front door to capture visitors at different angles.

Facebook has gotten very good at face recognition for identifying when your friends are in your photos. That expertise is also useful when your friends are at your door and your AI needs to determine whether to let them in.

Zuckerberg notes that visual recognition can be used for other purposes, such as determining whether or not the baby is awake (in which case, play music or a Mandarin lesson).

At this point, Facebook may not have the in-home tools to compete technologically with the likes of Google, Amazon, Apple and other would-be giants in home automation. But it certainly has the reach (active users) and a very enthusiastic CEO if the company ever decides to pursue the business.

Text vs. Speech for Home Control

Zuckerberg makes some interesting observations about controlling a home via text vs. voice:

One thing that surprised me about my communication with Jarvis is that when I have the choice of either speaking or texting, I text much more than I would have expected. This is for a number of reasons, but mostly it feels less disturbing to people around me. If I'm doing something that relates to them, like playing music for all of us, then speaking feels fine, but most of the time text feels more appropriate. Similarly, when Jarvis communicates with me, I'd much rather receive that over text message than voice. That's because voice can be disruptive and text gives you more control of when you want to look at it. Even when I speak to Jarvis, if I'm using my phone, I often prefer it to text or display its response.

Voice User Interface (VUI)

We have discussed in the past the nuances of the VUI – how we talk to things, and how they respond – but Zuckerberg makes some fresh observations.

Namely, how we communicate differs according to context – if we’re texting, talking face-to-face, calling across a room, chatting on Facebook, speaking at close range into the phone ….

A good AI-enabled speech-recognition system should take these different patterns into consideration:

Another interesting limitation of speech recognition systems — and machine learning systems more generally — is that they are more optimized for specific problems than most people realize. For example, understanding a person talking to a computer is subtly different problem from understanding a person talking to another person. If you train a machine learning system on data from Google of people speaking to a search engine, it will perform relatively worse on Facebook at understanding people talking to real people. In the case of Jarvis, training an AI that you'll talk to at close range is also different from training a system you'll talk to from all the way across the room, like Echo. These systems are more specialized than it appears, and that implies we are further off from having general systems than it might seem.

Connected Devices

Product manufacturers need to do a better job of anticipating the automated home.

Consider cloud-based services like Google Docs. Initially you had to be connected to the Internet to access your documents, but now there are tools for offline access.

The same applies to many traditional devices that can’t work “offline.”

Zuckerberg notes:

Further, most appliances aren't even connected to the internet yet. It's possible to control some of these using internet-connected power switches that let you turn the power on and off remotely. But often that isn't enough. For example, one thing I learned is it's hard to find a toaster that will let you push the bread down while it's powered off so you can automatically start toasting when the power goes on.

UPDATE: A Final Note: Yes, there are products that can accomplish most of what Zuckerberg has. His interesting blog post was not about him reinventing the wheel, but understanding what goes into developing a home automation system with AI capabilities. He shares some nice insights about the process. – JJ

• Page 1