Meta AI Smart Glasses: What Can They Really Do?

Hadley

Meta AI Smart Glasses: What Can They Really Do?

Presented by Ricky Enger

Ricky Enger: Meta Glasses have taken the world by storm, but what can they really do for those with vision loss? In this episode, Steve Kelley joins us as we demo Meta Glasses for common tasks. I'm Ricky Enger, and this is Hadley Presents. Did you know you can listen to our podcast plus find how-tos, discussion groups, and more, right from your phone or tablet with the new Hadley app? Search for Hadley Helps in the app store or text APP to 52545 and we'll send you a link to download. And now on with the show. Welcome to the show, Steve.

Steve Kelley: Hey Ricky, glad to be here.

Ricky Enger: Yeah, so great to have you. Also want to welcome Meta Glasses. Hey Meta, how are you doing?

Meta: Hey, going well over here. Thanks. Just hanging out and ready to chat.

Ricky Enger: All right. That was perfect. I think it is not an exaggeration to say that you and I as members of our practical help team here at Hadley, we get a lot of questions about Meta Glasses, right?

Steve Kelley: Oh, for sure. Yes, we do.

Ricky Enger: So, people kind of wonder what can they do? How much are they? How do you work with them? How do you set them up? All that kind of thing. And I think there are a couple of important things to mention. If you're listening and you already have Meta Glasses and you feel pretty comfortable, you might learn a couple of tips and tricks for this episode, but we really are going to keep it pretty basic. So, one thing when people call and wonder, okay, what do I need in order to use these Meta Glasses? Steve, what does somebody need in order to do this?

Steve Kelley: Oh my gosh, you've got to have a smartphone. That's the first critical thing.

Ricky Enger: Yeah, that's essential.

Steve Kelley: And then the next thing you need is you're going to have to have the Meta app installed on the smartphone and have the glasses connect to Bluetooth.

Ricky Enger: Yeah, that's a really important thing right there because some people have connected Bluetooth devices before and there's kind of a standard way that you do it. You go to settings and you open Bluetooth and you go through all that. With Meta Glasses, they are really particular about how they want you to do it, right? So, when you get the Meta app, you will add a pair of glasses and it's going to walk you through doing it in the app. So definitely you want to do it from there, right?

Steve Kelley: Yeah. And for those of us who sometimes don't pay attention to directions and just plow right ahead, I think it would be best in this case to pay attention to the directions because we've had people where they're actually connected, but somehow they're not talking to one another. So, you do have to do it the Meta way.

Ricky Enger: Yeah. And if you do that, it's pretty straightforward. It'll walk you through what to do. And if something's not working, that app will tell you right off, "Oh no, I can't find your glasses," or what have you. So good to walk through it from that app. And then once you do have the glasses paired, you can do a number of things in the settings for the glasses. A couple of things I would recommend is if you can go to accessibility and number one, you can connect to the Be My Eyes service, which we'll demo in just a bit. And there's another thing that you can turn on, which is detailed responses so Meta can give you short responses or it can be a lot more descriptive. And I have found that very useful in terms of getting the environment described and things like that. Have you turned this on for yourself, Steve, or are you still kind of at default?

Steve Kelley: No, I turned it on for myself too. And I found that sometimes it does go on a little bit longer than I would like, but in general, I think I like the longer, more detailed descriptions and that's kind of what you get from that.

Ricky Enger: Yeah, exactly. So, once you get through that hurdle of getting the glasses paired and you're all ready to take them for a spin, or maybe you're kind of thinking ahead for Christmas or you're just like, "I keep hearing about these things and I want to get them, but I don't really know everything that they do." They have a camera on them and the AI in the camera can tell you some things about what it sees. So that's a very quick overview of what they do. But I think it makes sense to just go through some of the common things that people ask us about. Can they do this? Can they do that? And then maybe we'll demonstrate, well, let's see how they do that thing. So, would you say there's the most common thing that people tend to ask about with these glasses?

Steve Kelley: I think one of the biggest things is people want to know how to use them for reading. And there's actually two parts to that, Ricky. Number one, I think because they're called glasses, I think a lot of people assume, oh, I'm going to be looking through them or somehow there's going to be some magnification. That's not so. It's all artificial intelligence, AI. So yeah, they do have a camera and it's going to take a picture of say printed text and then it will read that text back to you. They're not going to be doing any magnifying for you or helping you visually to see the text.

Ricky Enger: Yeah, that's a good point. And so, we actually have a demo of Meta reading, and I just want to show a couple of things that can happen when you ask it to read some things. So, this demo is a little bit long, but I think it does illustrate some important points about how Meta approaches reading text. Hey Meta, look and read what's in front of me.

Meta: Hey, sure thing. I see a mailer for a Capital One card. It mentions 0% intro APR on purchases, a $250 one-time cash bonus, unlimited 1.5% cashback, no annual fee, and a QR code or apply online at getmyoffercap. There's also a note to see the enclosed terms for eligibility and fees.

Ricky Enger: Thanks. Read it from top to bottom.

Meta: You got it. From top down, it says, "Apply for your card now. Get 0% intro APR on purchases, then $250 one-time cash bonus, unlimited 1.5% cash back, no annual fee, scan the code or go to GetMyOfferCap, plus see enclosed terms for eligibility and fees."

Ricky Enger: Look and read the text verbatim.

Meta: Got it. Reading it top to bottom, I see P 0% intro. Enjoy a $250 one-time cash bond. 0% intro APR on purchases and balance December 2027 with E18.499 to 28.49%. Earn a $250 one-time cash bonus. Unlimited 7.5% cashback on every Q-N-I-C-K-S-I-L-V-E-R, no annual fee, cash back that won't expire for the life of why. Shop with rewards at amazon.com or where from online stores with PayPal. Scan your QR code or apply online at Lonowodair. See the enclosed terms for eligibility and rate fee and other costs. Information Wanzers on Niranza.

Ricky Enger: So that's way more information than I ever wanted to know about Capital One, but I asked Meta to read in three different ways, like three different phrases. Did you notice anything about how Meta handled that, Steve?

Steve Kelley: Yeah. I noticed that oftentimes, unless you're specific as you were in the very last example, Meta wants to, by default, summarize, which can be great if you're just trying to sort mail or something like that. But if you're trying to read a page of text or something, you do want it to read from top to bottom verbatim.

Ricky Enger: Right. And it did include even some of the kind of misrecognized text in there, which is telling me that, oh, it really is reading what it saw verbatim rather than trying to guess. So just think about that as you're trying to read something. How important is it for it to be accurate? And I guess it's like time versus accuracy sort of thing. Do you want a summary, or do you want it to be exact? But it did a pretty good job, I think.

Steve Kelley: Oh yeah, it did. One of the things that I want to mention, and I often mentioned it to callers too, is there does seem to be a limit practically speaking. It seems like about 200 words is where it stops. And that may be just because of the technology, but I say that just to caution people, if you're trying to read a book or something like that, if it's pretty dense or there are a lot of words per page or something like that, be cautious because it may just read 200 words and you may end up missing a lot if you're not careful.

Ricky Enger: Yeah, that's a weird place for that page to end. That's exactly right.

Ricky Enger: So, another thing I think people ask about, certainly people who are really new to vision loss and even maybe those who are not using any sort of mobility aid yet, they don't have a white cane, they don't have a dog, or maybe they do, but haven't really been trained well. Have you had people ask about this? What can Meta do for me outside or getting around?

Steve Kelley: They have, and there are some that I really cringe at. Anything related to street crossings or traffic or anything like that, I just want to put my head in my hands. It's very scary.

Ricky Enger: Exactly. You have got to be really cautious about what you're hoping Meta will do and what it can actually do. And so, this next demo I did inside my house, which I know very well, so I didn't need to trust the feedback from the glasses and definitely no traffic. So, let's take a listen and see how that went. Hey Meta, start live AI.

Meta: Starting live AI.

That's a cozy kitchen space. What's on your mind?

Ricky Enger: What do you see?

Meta: Yep. Dim little kitchen, wood cabinets, tile floor, fridge on the right, window over the sink, letting in some light. Feels kind of cozy, quiet in there.

Ricky Enger: Do you see a door from here?

Meta: Nah, no door in view. Just cabinets, the sink under that window and the fridge up close. All walls and counters from this angle.

Ricky Enger: Tell me when you see a door.

Meta: Yep. Right there now. Far right by the sink. Glass panel doors in view.

Ricky Enger: Now where is it from where I'm standing?

Meta: From where you're at now, it's over on your left. Those glass panel double doors by the tile floor just past the table.

Ricky Enger: How about now?

Meta: Yep. Now it's dead ahead. Those glass panel doors are right in front of you, filling the view.

Ricky Enger: Great. Now I want to find some stairs. Can you tell me when you see them?

Meta: No stairs in sight right now. Just the kitchen counters, stove and sink area. I'll holler if any steps pop up.

Ricky Enger: What about now?

Meta: Yep. Right now, there's a carpeted stairway leading down with a railing on the left and a white door at the bottom. Stairs are definitely in view.

Ricky Enger: So, Meta didn't holler when she saw the stairs. I was actually standing at the top of them for a few seconds just waiting to see what would happen and nothing did.

Steve Kelley: Yeah. What's interesting about that, Ricky, is that on the one hand, it sounds very believable like, "Oh yeah, I see the door now and it's right in front of you and oh yeah, there are stairs right here. All I can think about is in a busy airport or train station or something like that, you've got people all around you and you're thinking maybe the Meta Glasses are going to point out the escalator or the stairs or something and it's not going to be reliable for those sorts of things, I don't think.

Ricky Enger: Yeah. I think you do have to be careful about what you ask it to do and maybe you continued walking forward when you said, "Hey, where are the stairs?" And you might have passed them before Meta even looked. So there's a lot that can go wrong there, but I do think that if you're deliberate about some things and you want to know where you are or you're trying to look at street signs or something, how does Meta do with that?

Steve Kelley: I really love the initial description where it was telling you what was going on in the room and giving you an overall sense of your area. And by the same token, I think if you're on the street or something like that and you stop and you look around and want to get a sense of, okay, well what shops are in front of me or something like that, yes, I think that's the sort of environment where Meta is going to excel.

Ricky Enger: Let's see how it does with asking about signs and such. Hey Meta, I'm trying to figure out where I am. Can you read the sign in front of me?

Meta: Hey, the sign above the store reads Sephora in big white letters. Inside, I can see a great mall storefront, shelves of products, a 50% off sign, and a person walking near the entrance.

Ricky Enger: Great, thanks. I have to say that 50% off sign was very tempting. If I had had more time, I would've gone in that Sephora because I love that place.

Steve Kelley: And that's the sort of thing that you may not notice unless you had a sighted partner with you or you were wearing the Meta Glasses, you just breeze right on by it and miss that big old sale right there.

Ricky Enger: All right. So, what else do people tend to ask about when it comes to Meta?

Steve Kelley: A lot of times when you are in the kitchen, I just think about finding stuff in the grocery store or finding stuff in the kitchen or say the refrigerator. It's like I'm putting together my shopping list or something like that. So, what do I have? What might be missing? And then again, in the grocery store, it's like you're going down the aisle or you're looking for particular things. I think that that's another one of those areas. And I know that for some things it can be really good for looking at cans or something like that, but kind of curious what you discovered.

Ricky Enger: Yeah. So, we're going to take a listen to that. And one thing I didn't show was looking at smaller bottles because that's where things kind of get a little more difficult because the text may be smaller and it's harder to get in view, but let's take a listen to what it's like just in a general real-world situation where I'm holding something. I don't know which way it's facing. I just picked it up, right? Hey Meta, start live AI.

Meta:. Starting live AI.

What a cozy kitchen space. What's on your mind?

Ricky Enger: What am I holding?

Meta: You've got a jar in your hand. Looks like a glass jar with a dark lid, probably holding some kind of sauce. The label's got text on it, but it's facing away a bit, so I can't catch the words from here. Feels like a typical pantry size jar, maybe a couple inches wide.

Ricky Enger: Can you read it now?

Meta: Yep. Now I can make it out clearer. The label says tika masala, likely a sauce with Pataks up top on the jar. That's the brand name you're holding.

Ricky Enger: What am I holding?

Meta: You're holding a glass jar about the size you'd expect for sauce with a metal lid. The label's got Pataks Tika masala on it, so it's a reddish-brown sauce jar, easy to grip with one hand.

Ricky Enger: I have switched jars now. Look again. Tell me what I'm holding.

Meta: Yep, you've switched. This one's a Raos jar now. Glass, metal lid, label's got Raos clear on the front, holding a reddish sauce inside. Feels like the same pantry size jar, easy to hold in one hand.

Ricky Enger: So, I noticed a couple of things on this. Number one, if you're listening and you notice sometimes I'm saying start live AI and sometimes I'm not, I tend to start the live AI when I know I'm going to be doing a number of things in a row and I might pause for a while as I'm looking for another jar or something like that. And I don't want to have to say, "Hey Meta” again to get things started. So, if I know I'm going to have kind of a session of it, then I will turn on live AI. Otherwise, I'll just say, "Hey Meta," and keep it going like that. But the other thing I noticed, did you notice this, Steve, where I had switched jars, but it was still telling me about the first one until I said, "No, no, it's different now.”

Steve Kelley: I had a similar experience when I was doing some spice jars. Yeah, I didn't have live AI on, so I wasn't experiencing quite to the extent that you were because each time I was prompting it like, "Hey Meta," it was its own new session. So, I think that that might be the workaround or as you did, "I've switched jars, by the way, take another look."

Ricky Enger: Yeah. And if you are doing the, “Hey Meta” approach, it's still worth noting that with your first Hey Meta, Meta will still keep that conversation open for a good 30 to 45 seconds after it has stopped speaking. So that can still happen where it's referencing the previous image. And all of that to say that if you suspect Meta maybe didn't look again, you can just ask it, "Hey, look again and make sure whatever you just told me." So, kitchen, yeah, I'd say it works pretty well as long as you kind of know the tips and tricks. What else do people ask about?

Steve Kelley: Well, as long as we're in the kitchen, I think one of the other things too is how are they going to work with appliances? So, we've got those wonderful push button LED screens that we're dealing with these days, and it's easy sometimes to put some bump dots and stuff like that on it, so each time you press the button, it increases the temperature on the oven, that sort of thing. I'm kind of curious how Meta works for something like that for appliances.

Ricky Enger: Yeah, it seems like it should work pretty well, right? Let's take a listen and see if that is true. Hey Meta, look and tell me what cycle my dryer is on.

Meta: Hey, from the panel, it looks like the dial set on mixed load with the display showing 26 minutes left. So, you're on a mixed load cycle with about 26 minutes remaining.

Ricky Enger: Look again and tell me what cycle my dryer is on.

Meta: Hey, reading the panel, the dial set on mixed load and the display shows 33 minutes remaining. So, it looks like your dryer's running a mixed load cycle with about 33 minutes left.

Ricky Enger: Hey Meta, look and tell me what button I'm pointing at.

Meta: Hey, your finger's right on the start button. It's the one just below power on the left side of the GE control panel. That's what kicks the cycle off.

Ricky Enger: Look and tell me what button I'm pointing at now.

Meta: Hey, you're pointing at the start button. It's the lower left pad right under the display. That's the one you press to kick off or resume the cycle.

Ricky Enger: Look again and tell me what button I'm pointing at.

Meta: Hey, your finger's on the time button. That's the lower center pad labeled time. It usually adjusts the cycle time up or down.

Ricky Enger: It sounded pretty confident about all of that, and the bad news is that it got the buttons right three out of four times and the cycles right zero out of a number of times.

Steve Kelley: Oh no. I was feeling so hopeful as I was listening to it.

I was thinking, oh my gosh, you could tell the dial position, but oh my gosh, it did not and sounded like it did.

Ricky Enger: It did not. And you would never know, right? Because it said, "Oh, it's pointing at this." It didn't say, "I think it's on this." It was very sure of itself, and it was also very wrong. So, speaking of clothing, let's take a quick look at how it might do with maybe organizing your closet. Hey Meta, describe what's in front of me.

Meta: Hey, right in front of you are two pairs of pants laid flat on a blue quilted bedspread, each clipped to a clear plastic hanger at the waistband. The left pair looks dark gray or black. The right one, a deep navy, both with smooth, slightly shiny fabric and straight wide legs.

Ricky Enger: It was correct and it could tell the difference between black and navy, which sometimes humans can't do.

Steve Kelley: Exactly. That's super helpful.

Ricky Enger: Yeah. So, I've really found that it shines in this area. I don't think I've ever come across a hallucination when it comes to describing a piece of clothing. I don't know why it's so good at that, but it really is. But with all of that said, there are some things that an AI is just not going to be good at. Apparently reading device dials is one of them. And then there are other things like if you're trying to find something that you dropped or maybe you need very accurate product directions and the AI is having a hard time reading those or any number of things where you could be looking around if you're traveling and you're trying to find the thermostat, which could be anywhere. If that AI doesn't spot it immediately, you could be going through a lot of prompting for, “Look along this wall, now what do you see?

I think that sometimes a human is going to do a much better job. So, on the Meta Glasses, there's an app called Be My Eyes, which you can use from your phone as well, and you can do this hands-free because you're wearing the glasses and you can get a human to help where the AI is not going to do that. So, I think the glasses can be really useful for that. And for people who haven't heard how that works, let's just get a quick demo. Hey Meta, Be My Eyes.

Meta: Hey, happy to help. I'm kicking off a Be My Eyes call with a volunteer now, so you'll get live visual assistance.

Be My Eyes is now using your glasses camera.

Ricky Enger: Hello. How are you doing?

Be My Eyes Volunteer: Good.

Ricky Enger: Awesome. So, I've dropped a pill somewhere and I can't see to find it.

Be My Eyes Volunteer: Can you get a little closer to the carpet?

Ricky Enger: Yeah, I think it's somewhere…

Be My Eyes Volunteer: Okay, I do see it.

Ricky Enger: Oh, good.

Be My Eyes Volunteer: It was to the right a teeny bit for the camera. A little bit back.

Ricky Enger: Okay. Oh, there it is. Yes. There you go. I got it.

Be My Eyes Volunteer: Awesome.

Ricky Enger: Awesome. That is exactly what I needed. Thank you so much.

Be My Eyes Volunteer: No problem.

Ricky Enger: I hope you have a good day.

Be My Eyes Volunteer: You too. Bye-bye. Bye.

Meta: Be My Eyes stopped using your glasses camera.

Ricky Enger: So just like that, some random person answered my call and was able to help me find the pill I dropped, which is pretty cool. And if I had had to do that from the phone, then I would've been holding the phone with one hand and searching with the other, which is not quite as easy. So, I think this works out great.

Steve Kelley: Oh yeah, that's totally awesome. I've used it for the thermostat and stuff like that, and it's just super handy and very seamless too, the way it places the call.

Ricky Enger: Yeah. And you notice how happy she was to say, "Okay, let's do it." And then it was done and we both kind of hung up feeling great about, "Cool, that was accomplished."

Steve Kelley: Yeah. So, both your days were made in approximately 30 seconds. That’s pretty awesome.

Ricky Enger: Yes. Well, any final things that you want to mention? We've done a lot of demos here and Meta, thanks for being on the show. It's really been good to have you. You're still learning some things, but we appreciate you, nonetheless. Anything else, Steve, that people might ask about or that you want to mention about these that people should know?

Steve Kelley: No, but I think from the demonstrations that you shared, it's really clear that there are certain things that Meta is going to do well with, and also there are certain things where you need to proceed with a little bit of caution. Just personally, I found that one of the coolest things about the Meta Glasses was walking into a restaurant and asking if there was something on the menu or something like that. I still get amazed when it tells me, "Oh yeah, there's a Caesar salad right there. It's 5.99 or whatever." And it does it so quickly.

Ricky Enger: Yeah. There's a lot that these can do so well. And as you say, there's some things that you want a different tool in the toolbox. And if you're having some trouble or you have questions about, well, what tools in the toolbox do I need or how would I approach this with the Meta Glasses or any of that, we're happy to talk that through with you. So, I hope you'll contact us and we're glad to have that conversation. Thanks so much, Steve, for joining me. As always, it's been a lot of fun, and I appreciate your being here.

Steve Kelley: Oh, what a pleasure. Thanks, Ricky.

Ricky Enger: Got something to say? Share your thoughts about this episode of Hadley Presents or make suggestions for future episodes. We'd love to hear from you. Send us an email at [email protected]. That's P-O-D-C-A-S-T at HadleyHelps. Org, or leave us a message at 847-784-2870. Thanks for listening.