Apps utilized for this project:
In 2021 I began to explore v-tubing which is live streaming while controlling a 3D avatar using VR and face tracking equipment. I have always been a fan of VR/AR applications so it's no surprise I took an interest in this growing niche of content creators.
My wife and I also enjoy karaoke from to time. So one day I had an idea, and with the help of Blender 3D and Unity, I set off to build ourselves our own little slice of virtual karaoke heaven, inspired by real karaoke booths found in Japan, which became our VTuber Karaoke App:
Once the model is ready, I set up the 52 face shape keys needed to work with Apple's ARKit, after which I am able to start testing movement in Unity.
After the first set of avatars was complete, I was able to create relatively quickly some additional outfits that followed current trends. With a few extra lines of code in Unity its possible to change these outfits on-the-fly during live streaming.
I also created my own headgear to hold the iPhone tracking our face while we move around and sing.
My wife and I also enjoy karaoke from to time. So one day I had an idea, and with the help of Blender 3D and Unity, I set off to build ourselves our own little slice of virtual karaoke heaven, inspired by real karaoke booths found in Japan, which became our VTuber Karaoke App:
Avatar Setup
Getting started with avatar creation was pretty straight forward. First I block out the character's head, then fuse everything together into one mesh to sculpt and clean up, then re-topologize for a low poly count. I also make sure to use edge loops as much as possible for smooth animation.Once the model is ready, I set up the 52 face shape keys needed to work with Apple's ARKit, after which I am able to start testing movement in Unity.
After the first set of avatars was complete, I was able to create relatively quickly some additional outfits that followed current trends. With a few extra lines of code in Unity its possible to change these outfits on-the-fly during live streaming.
I also created my own headgear to hold the iPhone tracking our face while we move around and sing.
The Karaoke Room
Not only did I want to re-create the experience of singing in a real booth in Japan by carefully crafting the surrounding environment, I also wanted to add cool features to the room that really enhance the immersion and feel magical.I was able to import my Blender 3D karaoke room file directly into Unity and start using immediately. I set up a few cameras, set up the TVs inside the room to capture my PC screen for YouTube, set up the speakers to bounce to the sounds, imported a couple of avatar models and we were in business!
Depending on the time of year, in this case winter/holidays, I added some extra eye candy such as snow on the floor and falling snowflakes.
Depending on the time of year, in this case winter/holidays, I added some extra eye candy such as snow on the floor and falling snowflakes.
Immersive Features
YouTube On-Screen
Utilizing NDI Screen Capture technology as a component in Unity, I was able to capture the screen of my PC monitor playing karaoke videos from YouTube and apply them as a "video texture" to the TVs in the 3D karaoke room, giving me live control to play virtually any karaoke song I could practically imagine.
Utilizing NDI Screen Capture technology as a component in Unity, I was able to capture the screen of my PC monitor playing karaoke videos from YouTube and apply them as a "video texture" to the TVs in the 3D karaoke room, giving me live control to play virtually any karaoke song I could practically imagine.
Bouncing Speakers
Using WASAPI Loopback Audio I was able to monitor the sound levels coming from my PC and use that data to bounce the speakers to the rhythm of the music.
Using WASAPI Loopback Audio I was able to monitor the sound levels coming from my PC and use that data to bounce the speakers to the rhythm of the music.
Comments and Gifting
Using the API of the selected streaming service (in this case TwitCast) I was able to grab recent comments and gifted items so I can do cool things like make them bounce around the screen.
Using the API of the selected streaming service (in this case TwitCast) I was able to grab recent comments and gifted items so I can do cool things like make them bounce around the screen.
"Star Power"
When enough gifts are sent by viewers, we enter "Star Power" mode where the entire room dissolves away into a whole new world depending on the season. We see here a winter wonderland during Holidays and New Years. Lyrics from the YouTube video can still be seen floating in the sky and highlighting as they're being sung.
When enough gifts are sent by viewers, we enter "Star Power" mode where the entire room dissolves away into a whole new world depending on the season. We see here a winter wonderland during Holidays and New Years. Lyrics from the YouTube video can still be seen floating in the sky and highlighting as they're being sung.
Final Product
This project is by no means complete because of the sheer amount of potential here! But for now we leave you with a few more videos from our real live TwitCast streams:
johnromerocreative.com
copyright John Romero
copyright John Romero