I put ChatGPT on my Ray-Ban Meta glasses — and it completely changed how I used them

I've been using my Ray-Ban Meta display glasses for various tasks, such as shopping and watching a music performance. However, there is one drawback: I am limited to using Meta AI. While Meta AI is a suitable choice, I often prefer ChatGPT, a different AI assistant. This led me to wonder if I could utilize ChatGPT on my glasses.

It turns out that this experiment might give us insight into the future of ChatGPT. OpenAI is collaborating with the designer Jony Ive on a new wave of AI hardware, including a portable, screenless assistant equipped with cameras and sensors capable of understanding its surroundings. Although it's uncertain whether smart glasses will be a part of this new lineup, OpenAI is considering a future where ChatGPT doesn't require users to take out their phones and open an app.

To test this concept, I used my iPhone as an intermediary between the camera on my glasses and ChatGPT. The outcome was essentially ChatGPT with a view. The process involved two main components: voice and vision. Voice was relatively straightforward, as the Ray-Ban Meta glasses can handle audio from phone calls, music, and other apps by connecting to an iPhone via Bluetooth.

However, vision required more effort. Meta's Wearables Device Access Toolkit offers developers access to features on compatible Meta glasses, including their first-person camera. This allowed an iPhone app to receive images from the glasses instead of relying on the iPhone's built-in camera. My iPhone functioned as the intermediary in this setup.

To use ChatGPT on my glasses, I simply pointed the camera at something, and the app captured an image from the glasses' perspective, sending it along with my question to ChatGPT. ChatGPT then analyzed the image and provided a response through the glasses. While this method is slower and relies heavily on the iPhone, it functions effectively.

It's important to note that I did not install ChatGPT directly onto the Ray-Bans, nor did saying "Hey Meta" summon ChatGPT. Once I was back in my pocket, using the phone felt almost like having ChatGPT glasses. With both ChatGPT and Meta AI, I posed the same questions, and the experiences differed. When using Meta AI on my phone, I needed to take deliberate actions, such as opening the app and pointing the camera at an unknown object before asking about it.

In contrast, wearing the glasses allowed the camera to capture whatever I was looking at, enabling me to ask questions without the need for additional steps. I could examine a product, have ChatGPT analyze it, and then ask follow-up questions. Personally, I found ChatGPT's voice and responses more amicable and felt more like conversing with someone who knew me, whereas Meta AI tended to be more direct and concise.

However, Meta AI had one significant advantage: its seamless integration on the glasses. Since Meta controls everything, the camera, microphones, speakers, wake word, and AI were designed to work together effortlessly. With ChatGPT on my glasses, I faced additional steps and potential issues, such as latency and connection problems.

While Meta AI would likely outperform ChatGPT in scenarios like determining what I'm looking at, my main goal was experiencing ChatGPT's unique perspective. ChatGPT on my glasses provided a different, more friendly experience that I wish I had tried earlier. This experiment allowed me to understand the difference of using a distinct AI on my glasses, and I believe the real takeaway is that I finally discovered how it feels to wear glasses with a different AI.

The future of AI hardware, as hinted by OpenAI's collaboration with Jony Ive, appears promising, and this experiment showcased the potential for wearables to offer a fresh, personalized experience with ChatGPT.