Mark Zuckerberg on Jarvis — his home-built AI platform

If you're the wealthy CEO of one of the biggest tech companies out there, you probably have a staff of talented people who can do just about anything you ask of them. But if you're Mark Zuckerberg — and the thing you want is something ultra-cool like a complete AI server that you built from scratch — you roll up your sleeves and fire up your favorite text editor and get back to your roots.

Seeing Zuckerberg face the same challenges as the average Joe is satisfying because he's a guy who can fix them.

In his latest Facebook post, Zuck details the story of Jarvis, his home-brewed automation system and AI server. It's a must-read for anyone interested in things like making smart devices talk to each other as well as trying to make them even smarter with their own AI. But it also has some deep insights into what one of the people who will be building the future of all this tech thinks about where we are heading and how we'll get there.

Along the way, Zuckerberg ran into the same challenges everyone else trying to patch together a custom home automation system. For starters, most of the things we want to be automagic aren't even connected to the internet yet. Sure, I can buy a really cool automated coffee maker but there are still all sorts of other things on the kitchen counter that play a role in making me breakfast.

Once you sort out equipment that does what you need it to do and creative wiring with smart outlets to do the rest, you find that none of these gadgets wants to talk to each other. Sonos and Crestron and Samsung make cool connected gadgets that work really well, but you still need to find a way to do everything from one place with one interface. Then you need to find a way to build controls that work when you're trying to get the right piece of gear to do the right thing at the right time.

Seeing how the CEO of a huge tech company thinks lets us know that they can still be nerdy college kids at heart.

It's sort of satisfying to see someone like Zuckerberg face these situations because he's a man that might be able to fix it for the rest of us. Zuckerberg-built smart gadgets would get a million likes.

The post is especially insightful once he dives into making Jarvis smarter. Facebook has a pretty expansive code library that can show you an animated cat picture or drive vision processors to recognize a person by their face. And most everything in between. But that's only a starting point and once you get an AI to do something, you need to program when and how to do it, as well as what happens after it's done.

Understanding context is important for any AI. For example, when I tell it to turn the AC up in "my office", that means something completely different from when Priscilla tells it the exact same thing. That one caused some issues! Or, for example, when you ask it to make the lights dimmer or to play a song without specifying a room, it needs to know where you are or it might end up blasting music in Max's room when we really need her to take a nap. Whoops.

Even the best have those whoops moments.

Zuckerberg's takeaway from all this is fascinating. Not because I agree with everything he says or all of his predictions, but because it gives us a glimpse of how someone with the means to shape the future thinks it's going to play out.

Zuck might not be right about how the future's going to play out, but he'll be a big part of it all.

While the Zuckerberg house gets more automated — he mentions that he wants his Big Green Egg to be next and I'm on board and have the wire strippers ready — he'll be working on ways to get a computer to mimic or expand our human senses. The big breakthrough that allows machines to learn without programming has yet to come and Zuckerberg has as much chance to make it as a fresh Google intern does. And when it does come, it's great knowing that the shapers of the information age are just as interested as they were 10 years ago.

Do yourself a favor and give this one a good read.

Read: Building Jarvis on Facebook

Jerry Hildenbrand
Senior Editor — Google Ecosystem

Jerry is an amateur woodworker and struggling shade tree mechanic. There's nothing he can't take apart, but many things he can't reassemble. You'll find him writing and speaking his loud opinion on Android Central and occasionally on Twitter.

24 Comments
  • Love that he calls it Jarvis. Hopefully they can get some Marvel licensing if they bring it mainstream!
  • -1? Someone's not a fan of Iron Man I take it. Well, you can suck an egg...
  • Maybe someone who finds it irritating that Zuckerberg is trying to compare himself to Tony Stark? That was my first thought when I read that he calls it Jarvis...
  • Or he's just a fan? Your argument is invalid, since Zuckerberg has no Goatee.
  • Your argument is invalid, since he's incapable of growing one.
  • Exactly my thought. He's a rich geek... But Tony Stark he is not!
  • Elon Musk is closer to Stark than Zuck ever will be
  • Musk yes. Suck...hell no
  • Jarvis today .... Ultron tomorrow
  • You have to admit, it's a pretty compelling Vision. (I'll see myself out)
  • So creative.
  • Bill Gates Home was completely automated and had AI back in 2000. When you entered rooms of his house the temp would automatically adjust and empty rooms would turn off everything including temperature for power saving. He could also tell it to play music back then with voice controls in any room he was in. The rich have always had access to incredible AI. It was a good 30 min show about his home he completely designed with tech built in.
  • He also used a sort of RFID badge so that it would know who was entering or leaving a room or issuing a command. Guests would see artwork based on their preferences, hear music they liked, etc.
  • There's a fundamental difference between the home automation you're describing and Zuckerberg's gate mounted camera that scans a person's face, ID's them, checks the address book to see if they're expected, then lets them into the house like he describes in his post. I really meant it when I said it's worth reading :)
  • meh. When I get up in the morning and hit the switch to my bathroom exhaust fan, my "AI" checks to see if it is a weekday, before 9 am and if the temperature outside is below 45 degrees. If so, it starts my car so it will be warm when I get in to commute to work. Then 30 minutes after I'm done showering it turns the fan off. When my smartphone leaves the GPS coordinates around my house, it turns off all the lights and all the power "vampire" plugs. None of what he's done is all that earth shattering, and like mwara244 says above, others have done it before.
  • When this happens > do this isn't the same as if this happens > decide what to do next And it's hard (for me anyway) to explain it. What you're both describing could be done with a light switch. Setting a trigger and running an action or a string of actions has been around forever. The trigger itself (your voice, your RFID or NFC card, your GPS coords, EZPass at the toll booth, etc.) depends on what's best to use for the situation. When it happens, a predefined and preprogrammed set of commands are run. If it doesn't happen, nothing is run. Zuck's front gate is different. I don;t know exactly how he's programmed it, but I know some stuff I worked on (and what got me excited for all this 25 years ago) Look at an apple. That apple arrived at your grocery in a box where it was put with other apples the same weight that all had a pleasant shape and color. Either a person put it in that box or some machinery did. There are hundreds of ways machinery can be used to sort that apple into a box with other apples that resemble it. The less complex the sorting system, the more machinery you need (you can drop it through metal rings to get the "perfect" shape, for example, weeding out the ones that aren't "perfect".) Or you can run the apple on a conveyor across a scale and under a high-speed camera at the same time. The scale reads the weight and tells the computer to push the apple to the next conveyor based on how the operator programmed it. If you have 10 conveyors that put apples in boxes, you can assign apples that weigh that much to one of 10. All apples the same weight will go to the same conveyor. But only if — At the beginning of your shift, you took the perfect apple based on the color and shape and set it under the high-speed camera and pressed a button to let the computer know this is the perfect apple. You tell the computer " now look at all the apples as they pass under this camera, and decide which ones are close to perfect based on their shape and color" If the computer thinks the apple is close to what you defined when you showed them the perfect apple, and it weighs (for example) 200 grams, stick it on the conveyor labeled number 1. If the apple isn't close enough to perfect and weighs 200grams, but still looks good enough to sell, stick it on conveyor 2. If the apple does not look good enough to sell, ignore the weight and send it to conveyor 10 so we can make apple sauce. The scale is automated. The camera that judges the shape and color is using artificial intelligence. The scale will be right every time as long as the scale itself is calibrated well. The camera will only be right if you can program it with a process to make the correct decisions. I hope this makes sense.
  • you're basically saying that if the conditions are more complex, it is "AI". But it's all hype. There are example of actual "machine learning" but none of this is that.
  • Nope. I'm not going to be able to explain the differences to you. That doesn't change the fact that they exist and are pretty major, though.
  • As long as the programmer is the one supplying the intelligence to the system, it's automation, a big programming trick. When it begins to draw it's own conclusion, it's AI. Jerry did put it best with if this > do this compared to if this happens > decide what to do next.
  • Rather disappointing he couldn't come up with a unique name. Let's call it Jarvis, since it's already a famous fictional AI.
  • All I caught out of this was big green egg... a 10 hour brisket with pecan smoking chips....
  • Neat. And I love that Morgan Freeman is the voice of Jarvis.
  • Respect for Zuckerberg! And I'm anti-Facebook
  • I love that he calls it Jarvis.