ECR Bolt-On: Turning phones into mixed reality devices for Matariki

SfTI researchers recently designed and built the Ātea Tele-co-presence (3D Video calling) system that adheres to Māori customs and values. While wearing an immersive VR headset, users can virtually explore a 3D replication of Te Rau Aroha Marae in Bluff. Through this system, users can virtually access the stories embedded within the tukutuku and whakairo panels of the wharenui, which is told in the presence of the kaumatua in the manner that is kanohi ki te kanohi. More importantly, live users can have a “face-to-face” conversation despite being physically located in different places. 

Image: A tele-co-presence session between Bubba Thompson and Noel Park having a kōrero in a shared virtual wharenui space. Both were able to have a natural conversation despite the physical distance (in this case between Dunedin and Bluff). Virtually both people are captured in real-time and displayed as voxels, that is 3D (volumetric) pixels. [Photo credit: Stuart Duncan]

Now, as part of SfTI’s ‘bolt-on’ projects, early career researchers from Ātea have successfully been funded to extend this work and are building a mobile phone-compatible Mixed Reality (MR) based tātai arorangi (Māori astronomy) experience with immersive content about Matariki. The Early Career Researcher ‘bolt-on’ projects are designed to further the work of existing SfTI projects whilst giving leadership opportunities to young researchers. 

With the recent introduction of the Matariki public holiday, there has been increasing interest around the country in learning about the science and the story of Matariki, and this project will enable an astronomy experience independent of time, weather, and location.

Image: Noel Park (University of Otago) wearing a virtual reality headset to experience content during a recording session. 

The Matariki Hunga nui project is co-led by PhD students Kris Tong (University of Canterbury) and Noel Park (University of Otago). Stu Duncan of the University of Otago will continue to advance the voxel capture technology used in Ātea. Tiriana Anderson (He Kura Tūī He Kura Ika) will join as a tikanga advisor. Postdoctoral Research Fellow Dr Rory Clifford (Kāi Tahu, Kāti Māmoe) will also be an advisor on this project.

The project uses mahi ngātahi (co-design approach with end-users) and kaupapa Māori guidelines. Once complete, the system has the potential to provide several experiences such as enabling celestial observations, teaching, sharing of mātauranga and educational materials; but most importantly access to everyone in both non-immersive and immersive mediums. During the Ātea project, the project team developed a strong relationship with Te Rūnanga o Awarua and with the kaikōrero and tohunga. The team will again work with these key-knowledge holders to collect targeted kōrero about Matariki to inform the tātai arorangi MR experience.

“The Matariki MR experience is primarily for mobile devices, so that we can capture as wide as possible segment of the public, and enable easy access to the experience,” said Kris Tong, one of the co-leaders of the project. 

The initial stage of the project uses an existing voxel-based tātai arorangi kōrero (from Ātea) and make it portable on mobile devices. In the second stage, the researchers are combining 360-degree videos and volumetric technologies to capture kōrero on Matariki. They are working with key knowledge holders and practitioners from Te Rūnanga o Awarua and Massey University.

Image: Hēmi Whaanga (Ngāti Kahungunu, Ngāi Tahu, Ngāti Mamoe, Waitaha) sharing his knowledge during a recording session for the bolt-on project.  

These kaikōrero are captured explaining the mātauranga and activities and ceremonies involved in the celebration on Matariki in 360-degree videos and volumetric videos. This content will then be merged and loaded into the prototype from the first stage as an updated experience. The final stage of the project will finalize the MR experience with an appropriate user interface and more extendibility to future content.

Image: 360-degree video and volumetric video capture of Pauline Harris sharing her knowledge of Matariki. 

Image: The Ātea team used a 360-degree camera (on tripod in the image) to record one of the kaikōrero (Dean Whaanga) telling stories related to the carvings in the Tahu-pōtiki, the wharenui of the Te Rau Aroha Marae in Bluff.

Image: Early work in Ātea showing recordings sessions of the Tama Rereti story. Bubba Thompson is captured in 3D using the voxel capture system. (Above) Shows Bubba acting the session and dialogue and (Below) Replicated 3D virtual storytelling experience in VR with aurora depicting one of the scenes.

“The aim is for people to either watch this on their regular mobile phone screens or use a virtual reality headset if they have one, so they can experience Matariki in mixed reality,” said Kris Tong.

This project is funded for $200,000 to 2024.