A highlight of WWDC 2019 – at least for developers – was Apple’s announcement that version 3 of the CoreML machine learning framework is running on the tvOS, macOS, watchOS, iOS and iPadOS system platforms and now machine learning directly on the device, for personalization purposes, for example (“Model Personalization”).
Judging by that, the news and improvements in CoreML turned out to be much more modest this year. It even looks like Apple dropped the version number: If last year’s update was titled CoreML 3, it is now simply called Core ML – even if the CoreMLTools show version 4.
Speaking of CoreML Tools: Creating your own models with the ML framework Create ML is fun for simple projects. In practice, however, models from TensorFlow or PyTorch are more common. To use such a model with Core ML, app developers must first convert it to the mlmodel file format. The CoreMLTools mentioned above are intended for this.
With the new CoreML version there is an interesting possibility to exchange the models (mlmodel file) instead of publishing them with an app update. In fairness, it must be said that this is not a new idea, but several third-party SDKs are already offering it. The big advantage of the in-house solution is that the models are hosted in the Apple cloud. Because app developers may have more than one model in their app, the new concept of model collections allows them to combine multiple models using the CloudKit dashboard. In order to prepare an ML core model for deployment via the cloud, there is now a “Create model archive” button in Xcode. This creates a .mlarchive file. If you take a look at this mlarchive, you will notice that it appears to be “only” a zip file with the mlmodelc folders. Developers can upload these to the CloudKit dashboard and include them in a model collection.
Another nice new feature is that you can provide different collections of models for different app users. For example, the camera on iPhone is different from the camera on iPad, so you might want to create two versions of a model and send one to iPhone users of the app and the other to iPad users. App developers can define targeting rules for the device class (iPhone / iPad / TV / Watch), the operating system and the version, the regional code, the language code and the version of the app. A new model version is not always provided immediately. Rather, at some point the app “recognizes” independently that a new model is available, automatically downloads the model and stores it within its sandbox. However, app developers seem to have no control over when and how this happens.
While this is a convenient solution and app developers don’t have to worry about hosting the models themselves, keep in mind that the models are in CloudKit. They thus burden the storage quota of the app publisher and the download of the models is included in the network traffic.
So far it has been really easy to steal your core ML model and use it in your own app. This changes with iOS 14 / macOS 11.0: The new CoreML can automatically encrypt and decrypt models so that strangers can no longer look into your mlmodelc folder. You can use encryption with or without the new CloudKit deployment. The Xcode development environment encrypts the compiled model, mlmodelc, and not the original mlmodel file. The model always remains in encrypted form on the user’s device. Only when the app instantiates the model does CoreML decrypt it automatically. This decrypted version only exists in memory, it is not saved anywhere as a file. In order for this to work, an app developer first needs an encryption key. Xcode offers a button for this. If a developer selects this button, Xcode generates a new encryption key, which it links to the Apple development team account.
In concrete terms, this process creates an .mlmodelkey file, from which each app developer receives a local copy for working. App developers do not have to embed this encryption key in their app – and they shouldn’t! The reason for this: The key is also stored on Apple’s servers. To decrypt the model when it is instantiated by the app, CoreML has to get the encryption key over the network from Apple’s servers. If the network fails before the encryption key has been downloaded, the application cannot instantiate the CoreML model accordingly. For this reason, app developers should use the new function YourModel.load (). This has a completion handler with which an app can react to loading errors. The error code modelKeyFetch indicates that Core ML was unable to load the decryption key from Apple’s servers.
- Facebook faces
Computers can learn to distinguish human faces. Facebook uses this for automatic face recognition.
- Machine learning
Contrary to what the picture suggests, machine learning is a sub-area of artificial intelligence – but a very important one.
Machine beats human: In 2016, Google’s machine learning system AlphaGo defeated the world champion in the game Go.
- GPUs GPU Nvidia
The leading companies in machine learning use graphics processors (GPUs) for parallel processing of data – for example from Nvidia.
- Deep learning
Deep learning methods first learn low-level elements such as brightness values, then elements on the middle level and finally high-level elements such as entire faces.
- IBM Watson
IBM Watson integrates several artificial intelligence methods: In addition to machine learning, these are algorithms for natural language processing and information retrieval, knowledge representation and automatic inference.
There are several high-level frameworks in the iOS / iPadOS SDKs that perform machine-related tasks. These are as old as CoreML itself. On the one hand there is the vision framework, which also received a number of new functions. If vision already had recognition models for faces, facial markings and human bodies, the following features are added in the new version:
In particular, the recording of the detection of multi-person whole body poses is an interesting feature. There are different open source models on the market, but they are not really good or slow. Commercial solutions, on the other hand, are expensive. In addition to static images,
the Apple framework can also analyze videos, whether as a file or from the camera in real time.
VNDetectContoursRequest is also new to detect the outlines of objects in an image. These are then returned as vector paths. VNGeometryUtils in turn has auxiliary functions for post-processing the recognized contours, such as simplifying them to basic geometric shapes. It can be assumed that PencilKit also uses this geometric basic shape function when entering a pen. App developers can use Apple’s Natural Language Framework for language processing. There are also some new features here:
NLTagger and NLModel can now find multiple tags and predict their relationship.
Embedding sentences: After it was already possible to use word embedding, NLEmbedding now also supports sentences. A built-in neural network is used to encode the entire set in a 512-dimensional vector. This helps to grasp the context.
In the area of the Speech & SoundAnalysis Frameworks, however, there seems to be no news. (mb / fm)