Apple describes its mobile devices as designed in California and assembled in China. You could also say they were made by the App Store, launched a decade ago next month, a year after the first iPhone.
Inviting outsiders to craft useful, entertaining, or even puerile extensions to the iPhone’s capabilities transformed the device into the era-defining franchise that enabled Uber and Snapchat. Craig Federighi, Apple’s head of software, is tasked with keeping that wellspring of new ideas flowing. One of his main strategies is to get more app developers to use artificial intelligence tools such as recognizing objects in front of an iPhone’s camera. The hope is that will spawn a new generation of ideas from Apple’s ecosystem of outsourced innovation.
“We have such a vibrant community of developers,” Federighi says. “We saw that if we could give them a big leg up toward incorporating machine learning into their apps they would do some really interesting things.”
He illustrates the point with a demo of an iPad app for basketball coaches called HomeCourt. You don’t have to be a pro; using the app is as easy as pointing an iPad’s camera at action on the court. Then the tricky stuff happens automatically. HomeCourt uses the support for machine learning added to Apple’s mobile operating system last year to analyze the video. The app tracks each time a player shoots, scores, or misses, and logs the shooter’s location on the court. Each event is indexed so a particular play can later be viewed with a single tap.
HomeCourt is built on tools announced by Federighi last summer, when he launched Apple’s bid to become a preferred playground for AI-curious developers. Known as Core ML, those tools help developers who’ve trained machine learning algorithms deploy them on Apple’s mobile devices and PCs.
At Apple’s Worldwide Developer Conference on Monday, Federighi revealed the next phase of his plan to enliven the app store with AI. It’s a tool called Create ML that’s something like a set of training wheels for building machine learning models in the first place. In a demo, training an image-recognition algorithm to distinguish different flavors of ice cream was as easy as dragging and dropping a folder containing a few dozen images and waiting a few seconds. In a session for developers, Apple engineers suggested Create ML could teach software to detect whether online comments are happy or angry, or predict the quality of wine from characteristics such as acidity and sugar content. Developers can use Create ML now but can’t ship apps using the technology until Apple’s latest operating systems arrive later this year.
Apple is far from the first tech company to release software to help developers build machine learning models. Facebook, Amazon, Microsoft, and Google have all done so, with Google’s TensorFlow most popular. Federighi claims none easily fit into an app developer’s regular workflow, limiting machine learning’s potential. “We’re really unleashing this capability for this vast developer community,” he says. Create ML is built on top of Apple’s Swift programming language, introduced in 2014 and popular in some developer circles for its ease of use.
Simplifying can bring limitations. Create ML looks useful, but creating complex or unique uses of machine learning requires building something from scratch, says Chris Nicholson, CEO of Skymind, which helps companies with machine learning projects. Predicting events over time, like what a customer will buy next, typically requires something bespoke, he says. “What will make apps stand out is a fully custom, proprietary model,” says Nicholson.
Create ML is also limited to Apple devices. WWDC attendee Wolfram Kerl, CTO of startup Smartpatient, would like to make his company’s medication-tracking app capable of reading the labels on medicines. Apple doesn’t yet offer specific support for reading text from images, and Kerl is hopeful that may change. But he’s also watching Google’s recently launched machine-learning tools for mobile developers, ML Kit. It supports text recognition, and Kerl’s app also has to work on Android. “Google tends to make things work on both platforms,” he says.
Apple says its tools are restricted to its own devices to get the best performance out of its carefully integrated software and hardware. Last year, the company added a “neural engine” to the iPhone’s processor to power machine learning software.
Federighi says Create ML has already proved that it’s ready to help companies improve their apps with machine learning. He points to Memrise, a startup with a popular language-learning app. With the help of Create ML the company added a feature that lets users point their phone at an object to learn its name in different languages. Running Create ML on a MacBook Pro to train the model with 20,000 images, instead of renting a cloud server with conventional software, shortened the process from a day to under an hour, Federighi says.
That speed boost comes from the way Create ML trains new models by adapting ones already built into Apple’s operating systems to power image recognition and other features in the company’s own apps. Re-training an existing algorithm is a standard trick in machine learning known as transfer learning, and can generate good results with less data. Create ML models can also be much smaller, something important for mobile developers, because they build on pre-existing models already on a device. Memrise’s conventional model was 90 megabytes in size; the one made with Create ML was just 3 megabytes.
Many developers at WWDC liked Federighi’s pitch. Nitish Mehta, a software engineer at Symantec, was planning to attend an in-depth session on Create ML on Tuesday afternoon. It ultimately attracted thousands, some of whom whooped while an Apple engineer coded a fruit detector live on stage.
Mehta has some experience using machine learning, but thinks Create ML could help him and many other developers make broader use of the technology. “If you make it easier, more people will do it,” he says.
Federighi believes that would inevitably change what Apple devices can offer their owners, although he won’t be drawn into predicting exactly how. “So much of the experience on our devices is what third parties end up creating as apps,” he says.