Smart home appliance control using natural user interfaces: occupational therapy imperative


Many patients recovering from stroke need Occupational Therapy to perform daily living activities in a normal manner. However,

·  Occupational Therapy requires external help.

·  Many disabled people cannot move around the house to perform daily chores.


·  Can we develop a system that allows a disabled person to control appliances in the house without leaving her seat?

·  Can we convert this daily life activity into an occupational therapy activity that can be monitored and recorded?

·  Can we measure the progress of the patient?


·  We have developed a smart home solution that allows a person to control home appliances such as light bulbs, fans and air conditioners remotely from her seat by making real gestures to control virtual knobs and switches.

·  The activity performed by the patient is silently recorded by the system to measure recovery metrics.

·  Our software platform runs on both PC and smartphone.


·  Smart home control allows a disabled person to manage home appliances without leaving her seat.

·  The smartphone based solution is highly mobile and can be easily used from anywhere inside the house.

·  The system supports a variety of sensors to detect gestures, such as Kinect2, Leap Motion and Myo.

·  A large number of gestures can be detected by the system supporting different kinds of movements.

·  A wide array of controllable hardware is supported by the software platform, such as Philips Hue and LIFX devices.

·  Visual feedback allows the person to correct wrong gestures immediately.

·  Encouraging interface artifacts are incorporated into the environment to make the OT experience enjoyable and to speed up the recovery process.

·  Quantification of the recovery process allows gamification of the therapy environment, which increases the immersiveness of the experience, resulting in increased adherence of the patient to the proper therapy procedure.

·  Integration with social media allows patients to create a collaborative environment for support and recovery.

·  Recovery metrics are recorded seamlessly without intervention in the patient’s daily work routine.

·  Therapists can monitor patients’ recovery metrics remotely.

·  Big Data back end can track millions of patients simultaneously.

·  Big Data analytics can help researchers study and discover new recovery patterns and metrics.

Our Patents Related to This Demo

1. Multimedia-based Browsing of Clustered Multiple Points of Interests (POI) at single GPS location

2. HoloHouse: A 3D collaborative home design environment

3. Next-Generation In-store Shopping Experience through Business Intelligence

4. A Mobile Cloud-based Big Data Framework to Connect Patients with the Doctors

5. Holographic visualization of Hajj Scenario for Disabled

6. Physical Therapy with Mixed Reality

7. Should you go for Hajj this year – A Smart Recommendation System for Physically Challenged

8. BCI (Brain Computer Interface) based VR (Virtual Reality) Therapy System for Paralyzed patients to recover muscle functions

9. Therapist-Assistant: Virtual Reality based Live Augmented Skeletal Therapy Management System

10. A serious game generator for disability persons based on patient disability level and/or therapist recommendation using non-invasive motion sensors

11. Gestpass: a multimodal gesture-based login system for providing assisted living services to physically challenged individuals

12. Mapping virtual 3d human model with the joint-motions of a disabled: a web-based 3d human model with natural user interface-based interaction capability for designing physical therapy

13. Immersive telepresence as occupational therapy for hemiplegic patients through VR technology

14. Hajjgeocaching - a real-life massive spatio-temporal online game for pilgrims with Motor Disability·

15. An electromyography (emg) signal and augmented reality based occupational hand therapy framework for people with hemiplegia

16. A gesture controlled mobile based camera-less smart home management solution for providing assisted living services to physically challenged individuals with low mobility

17. A non-invasive healthgis multimedia serious game therapy environment for providing ranges for motion of full-body joints to hemiplegic patient through forward and inverse kinematics

18. System and method for detecting and visualizing live kinetic and kinematic data for the musculoskeletal system

19. A web-based non-invasive way of detecting, tracking, recognizing and 3d visualizing live kinematic hand therapy data of hemiplegic patienth hemiplegia