Transcript 0:00 We have this particular design method called user-centered design approach, so we really engage young childrens and their parents and even teachers in those community centers, and then observe how they interact with the, the toys they play, how do they engage with object manipulation and also how do they respond to a mirrored view of themselves? 0:29 [upbeat music] Welcome back to The Peggy Smedley Show, your voice for our connected world, with your host, Peggy Smedley. Hello, listeners, um, welcome back to The Peggy Smedley Show. I'm your host, Peggy Smedley. 0:46 My next guest has developed several technologies in supporting socio-emotional learning skills such as perspective taking communication and curiosity for children with special needs and from marginalized backgrounds in STEM. 1:01 Her research focuses on creating novel learning technologies through augmented reality, embodied conversational agent, and tangible user interface. 1:10 In addition to her teaching role, she co-leads the Rochester Human Computer Interaction Research Group at the Department of Computer Science, University of Rochester. 1:21 Please welcome Jen Bai, assistant professor at the University of Rochester. Jen, welcome to the show. Thank you, Peggy. It's a pleasure to be invited to talk with you on the show. 1:34 So Jen, I'm really excited because, you know, this is a subject I think a lot of us need to understand your work in augmented reality and how that technology right now is really im- improving a lot of people's lives. 1:47 And, and I'm really interested- Mm-hmm... in, um, what your thoughts are right now and how it's advancing because I think it's a, it's an important technology, but you're using it in lots of different ways. 1:57 Can you explain that a little bit? Yeah. 2:00 So augmented reality is the interactive technology that can, you know, instead of bringing the users to another space or to other devices to get access to information, it actually superimpose information right in front of the physical environment surrounding us. 2:21 So basically, for example, if you are... You just got a fancy new coffee machine and you want to learn how to operate it. 2:28 So instead of, you know, open up this menu and then trying to figure out all the, all the operations that you can do with this machine, using augmented reality can directly super, uh, impose like a, a arrow, uh, near the turn-on button to tell you, "Okay, this is where you turn on your machine." 2:46 And then once you've done that, it will show you, "Okay, here's an arrow in front of you, uh, right on top of the lid where you add the coffee beans." 2:54 So [laughs] basically, you don't have to switch your at- attention frequently between the instructions or the knowledge you want to access to, um, versus the, the actual thing you're doing. 3:05 So that is the real power of augmented reality, is the situatedness of it, that you don't have to, you don't have to, you don't have to learn something elsewhere and then apply to the physical, um, activity that you get engaged in. 3:20 So as we see all the situational awareness that we're getting through it, it's almost like AR has given us this ability to have more storytelling. 3:28 We'll be able to do more things with it, and we've talked a lot on, on this show about where AI- AR is going. Mm-hmm. 3:36 How often do you see the term AR and VR as an umbrella term to what makes them different i- in, in the marketplace right now, and the way we get to see them and how we're gonna use them more in, in our, in our personal lives and our professional lives? 3:54 Right. So AR and VR are having both, um, its own advantages. So virtual reality, 4:04 it generate a entire, uh, computer-synthesized, uh, environment, so which we call the virtual environment, and it brings people to this virtual environment. 4:16 Um, the advantage of it is that it doesn't really have to follow the physics law, right? [laughs] You can do a lot of, uh, imaginative and creative activities in this virtual environment. 4:26 Um, AR on the other hand, it keeps people remaining in their familiar physical and social environment. You can continue interacting with the space. 4:37 You can continue interacting, manipulating with the objects in front of you. You can continue have a conversation with people in front of you. You're basically remaining... 4:47 Your, your, your physical activity is basically remaining, uh, intact d- while using the augment- the AR technology. 4:56 And as I have mentioned before, this, particularly if we talked about in the learning context, y- a learning context or training context in, in industry, it's really mitigate the gap of knowledge transfer or what we call transfer learning because you, you don't have to learn something first and then apply it to the, to the, the activity you are engaged in. 5:22 You, you really just follow the instruction or the follow the, the, the, the, the support or scaffolding that's been visualized right in front of you. 5:31 And these instructions are not, you know, just a purely, like a 2D, 2D image like you watch on the, on the, on the screen, but it's actually very cleverly embed into this 3D space. 5:44 It's, it's almost like, you know, you see those advertisement board in, uh, on the street, and, uh, they are all, you know, like nicely stick to the surface of the building, right? 5:56 And with augmented reality, it can understand that this 3D surface of the building, and then stick those virtual information in according to the, to the first person perspective. 6:07 So it's really trying to mitigate the perception effort by blending it into the, the 3D perspective and 3D, um, location.I love the way AR and VR is, is ev- evolved and the way you describe it now. 6:22 And one of the things I find really intriguing is, is what you've done with other needs right now, special needs, autism. Mm-hmm. 6:30 Because I think when we look at this technology and how it can help the other communities out there, talk to me about that because I think there's some exciting advances that you've done. W- help me understand that. 6:44 Yeah, absolutely. So as, as, 6:47 uh, as, uh, probably, uh, you have introduced to the audience that lots of the current AR technologies are about offering information, offering instructions to help, you know, field workers to, to, to, um, view the, you know, the, the, the, the construction site information. 7:05 Uh, those are very objective information. What my work, uh, what the work that I have done with children with autism is actually trying to externalize their internal mental, uh, internal mental process. 7:21 Or in other word, you can think about how to externalize their thought into a way that they can clearly convey to each other. 7:31 So that's, that's why I started with children with autism because they, uh, one of the diagnostic criteria for autism in early childhood is that they have this, uh, they experience this difficulty in engaging with social imaginative play. 7:49 So where they do pretend, they pretend one object as another, they pretend that they're in a situation that is not currently the case with their, the, the physical environment. 8:00 So children with autism especially struggle with that particular and important aspect of, uh, uh, play and learning. 8:08 And so what I have been, um, noticed is this, um, really interesting analogy between pretense and augmented overlay because in pretense you have to have this what we call object substitution. 8:25 So you have to, oh, I pretend this banana is a telephone, right? So in your, in your mind you have to have this, uh, coupling process to replace the representation of a, a banana with a telephone. 8:39 And with augmented reality we also have this clever association with the physical object of a banana and a, a virtual object, a telephone. 8:49 So I was thinking, okay, if children with autism have this, you know, inflexibility of doing this mental, uh, replacement or mental substitution and more, more specifically to share those substitution to, to, to articulate it to the playmates that they're engaged with, right? 9:08 Okay, now let's pretend this, this banana's a phone. So how can we use augmented reality to facilitate, to give them a, a, a support to be better engaged with this type of rich, um, play experience? 9:22 So I built, um, I built tangible toys, so where the kids can, um, say, you know, those, uh, abstract shapes, like a loop or a, a, um, a, a, a semi-sphere, uh, shape, and then they can assign different, uh, virtual, uh, textures and colors to those shape. 9:44 Okay, if I combine a, a blue, uh, color into a loop, a physical loop, and then I, I claim it, okay, now I'm pretending this as a pond. Okay, let's play a pirate ship game. 9:56 So that's, that way, um, through this, um, physical association of the virtual information with the physical object right in front of them between their play with their playmates, um, so the, the, the idea is that they can be more specifically communicating with their intention of this pretense process. 10:21 So i- uh, this is really exciting to see how you've helped, you know, those with special needs, the autism children, to be able to see the virtual with the physical. 10:31 I'd have to imagine that this is really exciting for you that you've been able to do this. How, how's the response been? 10:38 Um, so we have been, uh, running studies with children as young as four year old who are diagnosed with autism spectrum condition, and, uh, we have, we have, the study shows that, uh, kids tend to carry out more pretend play using, by interacting with this AR augmented toys compared to, uh, the same set of toy but when there's no AR augmentation. 11:05 And, uh, I'm also very, um, it's very interesting to see that how young learners and also learners with special need can adopt this AR technology, and they actually get really curious about the technology. 11:21 They want to see how the camera works, how do they see their, th- themselves, and h- they want to record. They want to record their, their puppet play, and they want to show to their friends. 11:31 So to really see the, the engagement and the potential, um, of AR, particularly, you know, not, uh, constraining the, the, the advantage of AR in augmenting knowledge or instructions, but more to external people's thought and facilitate communication between people. 11:53 When I hear you talk about this I always think there's, with, with trying to get autism children to overcome these, there's gotta be challenges with all of this. 12:01 Are you experiencing some of that just in the, in trying to get them more to embrace some of this technology?Of absolutely. 12:09 So there's, uh, working with, um, um, you know, user with special need is always, um, is always, uh, challenging, but also brings up a lot of opportunities. 12:21 Uh, for example, mm, to i- like, in human computer interaction, we have this, um, particular design method called user-centered design approach. 12:32 So we really engage, um, young childrens and their parents and even teachers in those community centers, and then observe how they interact with the, the toys they play, how do they engage with, um, object manipulation, and also how do they respond to a mirrored view of themselves. 12:54 Um, that's all, that's all kind of, uh, uh, plays into the, the, the design of how to introduce something quite new to young children. 13:07 And one thing I, uh, would a- admit that for technology like AR, which often have this, uh, novelty effect that children, you know, initially really engage with it because they are fascinated about the technology. 13:20 So, uh, one thing, one takeaway that I get from the study is that longitudinal study is, um, definitely required, possibly in clinical environment, to see once those novelty effect has faded away, and how children respond to it, and how can we, mm, 13:40 utilize the observation, like utilizing how the, the system understand the environment, understand how kids play, to establish a lifelong learning style. 13:52 So that's, that's one advantage that I think AR has a, like a vast potential, is that, uh, we can really, it can really help us to model the pace and trajectory of child learning, and then create this intelligent adaptation to provide personalized, uh, feedback and learning scaffolding. 14:14 So there's a lot right now that you're being able to see as you get all of this different type of reasoning that you're coming out of this. It's really, really exciting to hear you talk about this. 14:24 I love the idea that you're being able to see all of these children with special needs and advancing technology to be able to open more doors. It's really exciting. Thank you for sharing that with me. Zhen Bai- Mm-hmm... 14:36 assistant professor at the University of Rochester. Thank you for your time. Where can our listeners go to learn all the amazing things that you're doing right now? Thank you very much. It's been my pleasure. 14:48 Yeah, you can find more of my work to my personal website, which is www.zhenbai.io, or you can simply go to the homepage of the Computer Science Department at University of Rochester to find more about my work. 15:05 Thank you so much. Thank you very much. [upbeat music] Okay, listeners, that's all the time we have for today. Remember, uh, you can share your thoughts with me on Twitter, Connected World. 15:15 We are broadcast live every Tuesday at 12 PM Central. Check out our website at connectedworld.com, our show website at thepeggysmethshow.com. All right. 15:22 This is The Peggy Smedley Show, your voice for our connected world, and remember, with great technology comes great responsibility. Have an awesome week. [upbeat music]
