Apple devices like MacBooks have supported eye-tracking technology for a while, but it always needed extra hardware. Now, thanks to AI advancements, iPhone and iPad users can control their devices without any additional gadgets.
Apple Eye Tracking uses the front-facing camera to follow eye movements. As users look at different parts of the screen, the interactive elements highlight. When users' gaze lingers on an element, it registers as a tap, a feature called "Dwell Control." This can also imitate physical button presses and swipe gestures. Since Eye Tracking is integrated into the operating system, it works with any iPhone or iPad app.
Another new feature is Vocal Shortcuts, which provides hands-free control. Apple hasn't explained it in detail yet, but it appears simpler to use than the current Shortcuts system, which requires manual setup for automating tasks. Standard Shortcuts can be difficult because of the manual programming needed. Having more ready-made shortcuts from third-party providers would make it more useful. While Verbal Shortcuts seem easier to set up, more details are needed to understand if it just adds voice commands to the existing Shortcuts functionality.
Additionally, Apple introduced "Listen for Atypical Speech." This feature allows Apple's voice recognition technology to learn and understand the speech patterns of users with speech difficulties due to conditions like ALS, cerebral palsy, or stroke. According to Mark Hasegawa-Johnson from the University of Illinois Urbana-Champaign, AI can significantly improve speech recognition for people with atypical speech.
These new AI-powered features use onboard machine learning, meaning biometric data is securely stored and processed on the device, never sent to Apple or iCloud.
Apple also has other upcoming features, such as vehicle motion cues to reduce car sickness, voice control, color filters, and sound recognition for CarPlay. Existing accessibility features like VoiceOver, Magnifier, Personal Voice, and Live Speech are also getting upgrades.
Apple hasn't given a specific timeline for the release of these features but mentioned they will be available before the end of the year. Since the company highlights accessibility in May, it is likely the new features will launch soon, as developers are likely finishing the final touches.