Dr Sungchul Jung
Effective telediagnosis for sensitive situations
A tragedy in Dr Sungchul Jung’s homeland of South Korea has inspired a project seeking to leverage virtual reality (VR) technology to help doctors more effectively treat patients.
When respected suicide prevention psychiatrist Lim was attacked and killed by a patient at the Kangbuk Samsung Hospital in Seoul in late 2018, the people of South Korea were shocked. Among them was Dr Jung, visiting home on a break away from his research at the Human Interface Technology Laboratory New Zealand (HIT Lab NZ) at the University of Canterbury.
The 30 year-old patient suffered bipolar disorder and turned on his doctor during a routine consultation. What if Lim had been able to ascertain the patient’s troubled mental state using a ‘telepresence’ tool that kept himself safe in the process?
That was the question that went through Dr Jung’s mind as he started pondering how to apply his PhD research in virtual reality (VR) to safe and effective mental health therapy and treatment.
VR technology to measure emotional states
Telepresence in virtual reality is a fast-growing field, allowing people wearing VR headsets to meet in a virtual space, with their digital avatars interacting and conversing. With newer VR headsets and handheld controllers using cameras and sensors to track head and hand movements, Dr Jung, who joined the HIT Lab NZ after completing his PhD at the Orlando at the University of Central Florida, is looking to harness the technology to ascertain a person’s emotional state – in real-time. He explains:
“Previous research showed that human behaviour such as verbal pattern, eye-gaze, body gesture, and hand gesture are closely related to the human mental status.”
“Also, physiological signals, especially an electroencephalogram (EEG), which represents brain-activity, or an electrocardiogram (ECG), which represents heart activity, could be critical indicators for human emotion, cognition, and perceptual states,” he adds.
Dr Jung’s Science for Technological Innovation (SfTI) two-year Seed project, sees him developing the platform to gather all of that gesture data and biological data, and presenting them in a format that is useful to doctors or therapists treating patients.
Central to that will be using deep learning techniques to build a reliable dataset of verbal and non-verbal behaviour cues. Dr Jung says:
“For improving the validity, creating a training dataset is a challenging part of this project.”
Improving the accuracy of facial expression recognition
Dr Jung will draw on existing work on sentiment analysis in fields such as education, but the project will also explore ways of improving the accuracy of facial expression recognition, which currently sits at around 75 per cent in computer vision.
Ultimately, Dr Jung’s goal is to develop software that can power a VR meeting between a doctor and patient, gathering data that delivers mental status updates to the doctor via a dashboard.
Most treatment programmes involve numerous consultations over time. The software would track the patient’s mental health status and inform the doctor’s treatment programme.
Meeting a market need for VR meeting technology
Dr Jung says the technology could apply to other areas, such as meetings in the legal field, military training and as a tool for scientific research.
“Just as use of Zoom has significantly grown during the COVID-19, we are expecting the telepresence market to grow not only for medical but also for education, entertainment, and training as well.”
Top image: Dr Sungchul Jung in the University of Canterbury HIT Lab, holding a VR headset.