VR Classrooms
Oct 2022-Mar 2023
College
Node Spring Unreal Engine 5 MySQL VR Application Demo Website Front-end Git Backend-Git Download report
This is possibly my most favorite project ever
For my final year project for university, I wanted to go beyond and push myself techically to develop something completly unique that was also practical and viable MVP within the next 4-5 years. My main limitation during this projects were time - the techical scope of the project is large and I had final year university courses to do. I also had to conform to university requirements and fitting what they were looking for and spend time on reports, meetings etc.. However, this is still a usuable MVP that can be improved significantly with a bit more skill and experience (maybe I will in the future when I have more time :D)If you're interested in reading about market research, implementation discussion, system design and more, read the report!
I came up with the idea for VR Classrooms - VR will no doubt be a major technology in the coming years, the only delay is waiting for hardware to catch up. When I came up with this project, we'd barely ended online learning as primary form of teaching and it has many issues including not being able to see body language, stress of having camera on, being distracted by your enviroment and generally disengaged from the class.
All problems Virtual Reality is perfect to solve.
I did some research and analysis on the current market (read my paper for more details), and came up a solution to have a VR Classroom where students and teachers would connect and hold a class virtually. They should be able to move, talk to people near them and also have access to tools for modern day classes like a way to display pdfs on screen, have a whiteboard to draw on and more. After I finalized the idea, I recurited a team-mate to manage the multiplayer aspects while I handled the rest of it so, I'll only be discussing my work on it.
System design
On a techical level, due to my inexperience at the time with spring security and time constraint of the project combined with it's size, I chose firebase as the main back-end to support my MVP. To build apis, I chose SpringBoot for my main back-end to interact with firebase, and basic node server to connect to the vr headset and spring api. .NET would have been a viable alternative, but spring was chosen for comfort. For Node, I had to choose a language that supported firebase, so node was an apporitate choice. For the Spring api, I could use the admin sdk but the node api would interact with the headset, so it needed the client sdk.
For the front-end, because the website would not be worth the same as the other parts of the project according to university, and to save time, a vanilla HTML/CSS/JS combo was sufficent for design with bootstrap to make it passable. In VR headsets, I chose MetaQuest 2 based on it's popularity, accessibtly and future support of that ecosystem. And finally for the most fun part, the VR App, it was made with Unreal Engine 4, and later adpated to UE5 which released during this time. Unity would also be a good selection, but due to the time constraint and learning curve, it would be difficult to fit it in. MetaQuest has a SDK for Unreal Engine that isn't great, but good enough for a MVP.
Cool things I accomplished
I came up with a use case where a teacher could login into to the website, open the page of thier class, upload a pdf and instantly, have that file available inside the VR Classroom and show it in class, go page by page and annotate it for everyone to see. - All within a 100ms avg reponse time (check report to see exact specifics on testing) from the VR Api and the VR Headset to serve each page instantly when the teacher would click next page.The way it worked was the pdf would be stored with in firebase object storage (with apporitate security that only teachers could modify) - this pdf would be downloaded directly to the vr api (node), be converted to images for each page and this page would be sent to the headset. On the headset, in unreal engine, I had limited functionality, doing something like wasn't possible on device easily - the file system on the headset was being accessed from unreal engine to a translation to android app which accessed the device file system so doing anything complicated there was out of question. Based on the pdf size, the breaking of files could take a few seconds, but nothing longer than 30-40s, also, each page would be generated right away so the teacher could browse while the pages were being generated inside the headset. If there was no request for that file on the vr api for a while, it would be deleted automatically till the next request was made.
I made a whiteboard that you could draw on and other people can see - I used masking materials with overlays and some vector maths so you could point with your hand, and a line would be generated from the direction of your hand in the forward vector direction that would be checked if it was intersecting with whiteboard, and if you clicked the trigger, it would start drawing on intersection. Then I combined this with dynamic material generation with some C++ tricks to change the set the base material as the current page of whatever file was loaded by the teacher.
Some other facts about project
- Security of the database, keys, file storage, roles (for teacher/student) and oauth was implemented throughout the chain.
- Each application did go through testing for performance, especially the VR application since maintain constant fps on a VR Application is very important
- Different rendering techiques, foveation levels, graphical fidelity were all tweaked for performance
- Firebase is a public cloud technology, and the apis were all hosted on fly.io public cloud as well with 256 or 512 MB of ram and shared single core cpu often when measuring metrics.
- User feedback was split between 2 different movement systems, so both were implemented :D
- The night before the demo, the hong kong region for fly.io went down and all images had to be migrated to singapore