Over the past 10 years, researchers have used virtual reality (VR) as a useful tool in therapy for a wide variety of applications, including treatment for anxiety disorders and pain distraction. VR allows users to interact with a computer-simulated environment. Typically, small head-mounted display (HMD) goggles provide the user a 3-D visual interface with the environment. A head-tracking system, computer joystick or mouse allows the user to navigate through a 360-degree view of the environment. Stereo earphones also allow sound effects to be integrated into the experience.

One of the newest and most exciting applications of VR, is in the field of rehabilitation. Research in VR applications – for stroke patients in particular – has made considerable progress. Functional MRI studies of VR used in conjunction with traditional physical therapy in stroke patients, have showed neuroplastic changes in the brain and corresponding improvements in motor functions.

Mixed Reality (MR) is a parallel technology to VR, with many possibilities in healthcare. MR is generally viewed as being along a continuum from virtual reality to physical reality and extends VR by blending the real and the virtual into a seamless landscape. Wearing a video see-through head-mounted display (VST-HMD), a user can see a virtual environment blended into their view of the real environment, effectively “mixing” the two realities. The paramount advantage of MR is that it creates an altered or augmented reality without losing the benefits of the physical setting – touch, smell, hearing, taste, and visual contact with other humans.

The University of Central Florida’s Media Convergence Lab at the Institute of Simulation and Training (UCF MCL) in Orlando, Fla., is currently partnered with The Virtual Reality Medical Center (VRMC), a professional medical corporation based in San Diego, to develop a new rehabilitation system that takes advantage of the unique capabilities of MR for physical and cognitive therapy. UCF MCL already successfully used virtual enhancements, overlays, and contexts to create MR scenarios for a variety of applications, including a study in which participants with severe stuttering sit at a table in a virtual restaurant, reading a real menu and interacting with other live attendants, to see how they might react when placed under stress in an everyday setting.

In a study funded by the National Science Foundation (NSF), UCF MCL and VRMC are developing a Mixed Reality Rehabilitation System, or MRRS, which they hope will enable stroke patients to receive rehabilitation of the upper extremities both in the office and at home.

There is a large and growing need for stroke rehabilitation, with approximately 700,000 Americans affected by stroke annually, costing an estimated $62 billion in 2008. At the age of 55, the risk of having a stroke is one out of every six. This risk doubles for every decade after 55.

Once the Baby Boomer generation starts turning 65 in 2011, the risk will exceed the growth rate of the total population. More than 50 percent of stroke victims suffer from impairment of the upper limb, hindering their ability to carry out simple tasks such as dressing, picking up objects, and preparing meals. One challenge to rehabilitation is that, after an initial period of recovery, stroke victims often reach a plateau in their recovery. Other challenges include limited rehabilitation resources and low levels of interest and participation.

Pushing the Industry

The driving idea behind the MRRS is to provide an interesting, engaging rehabilitation tool that patients will want to use. Recent evidence shows that stroke patients can improve even a year after their strokes, with intensive and repetitive rehabilitation. The team hopes that providing a social aspect and game environment will not only make rehabilitation more fun and less isolating, but also help divert patients’ attention away from the physical pain that is often associated with physical therapy.

“Physical therapy is painful, and there’s nothing in our research that can make it less so.  However, it can be made less isolating and boring,” says Eileen Smith, associate director of UCF MCL. “How can we enhance the traditional tools of rehab to allow engaging, fun game environments to explore how increased socialization and motivation might increase results? That’s what our research is exploring.” Smith is collaborating with Charles Hughes, PhD, director of MCL and the project’s principal investigator. Smith is also working with VRMC’s president, Mark Wiederhold, MD, PhD, FACP, a physician and co-investigator, as well as an advising physical therapist and occupational therapist.

Using MR, the researchers are able to design and test a variety of hypotheses within a safe, controlled, and naturalistic environment. During their preliminary work, Hughes and Smith found that MCL could produce a viable and reproducible computer program with software to track patients’ progress. In the lab’s controlled environment, they can collect quantitative data about the MR user’s experience, including task performance and neurophsyiological responses to stimuli.

In the first phase of the project, the lab created MR experiences called “SmashMe” and “CatchMe” that use game-based techniques to motivate people to comply with their therapy regimens. Wearing the specialized HMD, users see a large, cartoonish virtual insect flying around in the MR environment. The objective is to catch and smash bugs, and each activity is designed to extend the patient’s range of movement in personalized ways established by a therapist. The application was conceptualized to increase patients’ range of motion by moving the bugs farther way as the patient improves. A scoring system and the ability for patients to challenge other players add motivational aspects to the game.

“Our goal is to engage physical rehabilitation patients with scenarios that will alleviate the boredom and isolation of painful therapy,” says Smith. “Our research is looking at how family and friends might be able to participate in an engaging, fun experience with the patient.”

Changing the Field

There are few – if any – currently marketed products aimed at improving the upper extremities in disabled patients. In addition to stroke patients, there are other groups, such as people with traumatic brain injury (TBI), who could benefit from the system. A recent RAND survey found that 19.5 percent (over 320,000) of service members may have experienced at least a mild traumatic brain injury (mTBI) while deployed. Multiple redeployments, unprecedented in this all-volunteer U.S. military, may compound the risk for physical and psychological injuries, potentially resulting in more severe and chronic mental health problems.

“This technology is very valuable in the evaluation and treatment of blast injuries, one of the major wounds occurring in Iraq and Afghanistan,” says Wiederhold.

Cognitive assessment for TBI, as well as assessment for stress injuries such as post-traumatic stress disorder, could be achieved in MR through capturing and analyzing a user’s reaction while experiencing the environment. For example, scenarios can be created in which users carry out tasks that require sequencing, planning, and organizing in a contextually meaningful environment with commonly-encountered visual and aural distracters. Visual distracters might include subtle ones – people walking – and more abrupt ones – a stack of cartons falling. Aural distracters might include gentle ones – fans whirring – and more aggressive ones – the sound of an automobile backfiring.

Also, the intensity of these distracters during a single experience can be dynamically controlled by a therapist or software that takes into account the user’s perceived level of stress. Intensity can also be controlled between experiences. For example, one may increase the stress level in a subsequent session, attempting to improve a person’s tolerance, thereby helping the individual to better respond to stimuli in real-world situations.

Assessment of progress can include absolute measurements such as neurophysiological metrics (stress indicators) and the performance of meaningful tasks (accuracy and adherence to time constraints). As a result, by accurately capturing a user’s interaction with the environment in the context of simulation events, MR may be an effective tool for assessment and rehabilitation planning for returning military members.

“If the patient was exposed to a blast or had a traumatic brain injury, these scenarios, coupled with the data collection capability, would help treat the injury or help us determine if they need further evaluation,” says Wiederhold.

Phase I served as a proof of principle, in which the team built a prototype system with data capturing capability. The product is currently being refined and tested in Phase II, which will lead to the commercialization of new software and hardware that can be used for further technological developments in MR systems, including those for other applications, such as prosthetic limb rehabilitation for amputees. Success in this area will add to the scientific knowledge base of what is known about mental practice in rehabilitation.

Mentally rehearsing physical movement, a technique known as mental practice, is an underutilized form of therapy that has great potential for improving rehabilitation. Studies with healthy volunteers demonstrate that mental practice can produce effects in the body, such as increased muscle strength and improved performance, that are similar to those achieved when conducting physical practice. More recent studies have demonstrated that stroke patients who mentally rehearse movements during physical rehabilitation have improved outcomes.

For patients to practice limited body movements, the researchers plan to add video capture-based body movement scenarios. Because physical therapists who do utilize mental practice ask patients to imagine their own body doing the movement, the video perspective will be in the first person. The end result will be a program in which therapists can select which scenario they want the patient to mentally rehearse that day, based on the physical therapy treatment that they are receiving.

But in order for the technology to be widely useful, it needs to be “scalable” – that is, deliverable within the constraints that exist outside the lab. “Even when MR experiences are effective, they rarely scale,” says Hughes. He explained that scaling refers to the process of taking an experience developed in a research laboratory and delivering it within the budget constraints required for large- or even medium-scale adoption.

This has many dimensions, but one of particular interest to the research team is to scale an experience that uses a VST-HMD and high-end audio to one that can run on projected surfaces, a desktop computer, a game console or even a cell phone. In the current NSF MR rehabilitation system study, the researchers hope to remove the head-mounted displays entirely in Phase II, introducing a version that will display the scenario via screen projection, without losing the advantages of a 3-D experience.


Another important goal is to develop a set of tools that can be used to analyze the user experience with respect to objective metrics, something rarely found in MR systems. Other tasks to be completed include ensuring the system’s safety and submitting the system for regulatory approval. The team at UCF MCL is developing prototypes using MR technology, and VRMC will develop them into products. The university will wrap up its primary contributions to the project by the close of 2009 and provide a license for commercialization to VRMC. Eventually, VRMC will provide rehabilitation clinics outfitted with the technology.

The impact of the project will be the potential for stroke or TBI patients to borrow or rent a take-home version of the system from telerehabilitation-enabled facilities to continue intensive rehabilitation on their own under the guidance of a therapist via the Internet. Another possibility is to market the product as a take-home program that patients could run on their home computers or hand-held devices. The ability to collect data for assessments as treatment progresses would provide therapists with important information regarding treatment efficacy and the need for additional clinic-based treatments or booster sessions.

The top priority of the team developing this MR rehabilitation system is to extend the reach and generalizability of its research to both domestic and vocational rehabilitation. Over 12 million families in the U.S. alone have members with a physical disability. Therefore, success in this project will pave the development and commercialization of future rehabilitation systems to help this broad and underserved population.

By increasing stroke patients’ upper extremity mobility and rate of recovery, this system will also increase their activities of daily living, enable at-home physical therapy, relieve some of the burden of caregivers, and decrease costs in lost productivity and length of hospital stay. Expanded uses of the system for cognitive assessment and cognitive rehabilitation will allow an even greater number of people to benefit from the technology.

— Nancy Ahmann is associate editor at the Virtual Reality Medical Center in San Diego. Questions and comments can be directed to editorial@therapytimes.com.