One of the highlights of ARKit 4 is the new depth API. It enables the developer to obtain very precise depth information about the current video signal. In combination with the scene geometry API that has been available since ARKit 3.5, ARKit 4 enables a 3D matrix of measured values from the immediate surroundings. Each point is given a value that evaluates how safely ARKit can sort it.
All of these measurements together provide detailed depth information, which should improve the understanding of a scene and the overlay of real objects by virtual ones. Developers can use these to better recognize a room, evaluate body measurements and much more. However, this requires a device with a LiDAR sensor such as the iPad Pro 2020 – so it is not entirely unlikely that the next iPhone will also be equipped with it.
With ARKit 4, Apple is also introducing a new type of anchor. These are marker points to connect the virtual with the real world. With ARKit Location Anchors, app developers can anchor virtual content in the real world, for example at points of interest in a city. In this way, the augmented reality experience is brought into the great outdoors and made available to a large number of users at the same time. For this purpose, AR experiences are created based on the geodata (latitude and longitude and altitude) and data from Apple Maps, which are linked to a specific location in the world.
ARKit 4 is ?? pic.twitter.com/UDRIPWit86
– Joel Bernstein (@CastIrony) June 23, 2020
Apple calls this process, in which an iPhone or iPad is located in relation to its (optical) environment with a higher degree of accuracy, “visual localization”. Unfortunately, this feature is not (yet) available in Germany. So far, only the cities of San Francisco, New York, Los Angeles, Chicago and Miami captured by Apple’s camera car have been supported.
Face tracking has also been expanded in ARKit 4 and now supports the front camera of every Apple device with the A12 Bionic chip and later, including the new iPhone SE. However, a device with a TrueDepth camera is required to track up to three faces at the same time, e.g. for memojis or Snapchat features.
The detection of hand bones (Vision API, 21 data points) and several people in one scene as a whole is also possible with iOS 14 in real time. There are a few subtleties hidden here. Like ARKit, the vision framework can identify a total of 19 measuring points on a human body. In contrast to ARKit, however, the vision framework provides an accuracy value (how secure the framework is) and can also be applied to images and videos that are, for example, in the camera library.
It may be that ARKit 4 only really brings performance to the road in conjunction with the LiDAR sensor installed in the new iPad Pro. But in a few months, iPhones that are equipped with this sensor will probably also come onto the market. Other devices are also under discussion, for example ominous AR glasses (Apple Glass) could only have a LiDAR sensor instead of a conventional camera. The advantage is obvious: the wearer – if there will ever be these glasses – could not make video recordings of his surroundings. All of this is, of course, a dream of the future, initially ARKit 4 remains great news for iPads and maybe soon for iPhones. (mb)