For the past couple of months, I’ve been working on a brilliantly fun Unity project.
It primarily provides an interactive 3D virtual background that is used during the production of a Twitch Livestream. It combines many other features, integrations and automations.
A primary feature is an integration to the Twitch API that opens the door for a whole bunch of cool interactions with the stream.
Unlike the blow-by-blow instructions that you’ll find in my other articles, this “Projects” section is intended to be a tour of some of the projects that I’ve been working on. If you want to replicate these projects, you’ll need to figure out the implementation details yourself, but hopefully, they can be a source of useful hints and inspiration for folks.
So, what exactly is the Virtual Studio solution?
BTW, that reads Virtual Studio - not Visual Studio!
Folks who watch Twitch streams are quite accustomed to streamers using a greenscreen to composite themselves into a scene that uses a static image as a background.
During a Twitch stream, we often get viewers kindly mention that “they love the Twitch Theme” - assuming it’s just a static background. It’s often delightful to then read their subsequent reaction - when they realise that the background is actually a realtime rendered 3D scene that is reacting to channel activities.
Note: A common question we get is “is the camera mounted on a computer-controlled slide?” - it’s not a physically moving camera, but instead a smoke and mirrors trick with how the camera feed is composited.
The Virtual Studio system does a lot of things, but here is a rough overview:
- Realtime composition of multiple NDI camera feeds into the virtual environment.
- Superimposing chromakey-cutout of the presenter into the virtual scene.
- Integrating screen capture feeds into various virtual props (e.g. a laptop screen or a sci-fi style holographic projector).
At time of writing there are three distinct themed virtual environments - but I have plans for many more.
- For the “penthouse” scene I’ve used a mixture of home-made and purchased 3rd-party models. I’m a novice 3D modeller, so for those times when my skill didn’t match my ambition (e.g. plants and similar organic shapes), it made sense to buy the assets in.
- For the “halloween” scene, I was more ambitious with my modelling, creating most of what you see myself, using blender.
- The “winter” scene continued to push the envelope in terms of on-screen visuals. This time around, because of the huge amount of complex modelling,I purchased a bunch of tree and rock assets. There was still plenty for me to do myself though, as I created the dynamically lighting “sunset” and “snowstorm” effects myself.
- There is a “virtual desk” prop which is intended to be re-used across multiple scenes. It forms part of a strategy to centralise and re-use many of the features.
An integration with Twitch, via the TwitchAPI, is heavily used to react to a wide range of events.
- In some respects, the solution could be thought of as a TwitchBot on steroids.
- Twitch “subscriptions” trigger “fun animated events”, such as sound effects, a shower of party balloons. Subscriptions are visualised as “gift boxes” which are spawned and sit on a desktop for the duration of the stream.
- Twitch “cheers” trigger a virtual coin-drop, which is a similar effect to the subscription gift-boxes. The coins pile-up on a desktop.
- “Channel Point redeems” trigger weird pop-culture effects - such as Darth Vader popping up from behind the sofa.
- “Channel commands” can be used to trigger special effects - for example, I can make it rain in the scene.
There is a two-way websocket integration with OBS itself.
- Specific scene changes in OBS trigger a change in the Unity app, changing view perspectives etc.
- Events in the Scene, typically raised from Twitch commands, can trigger OBS scene changes.
- For example, we have a “!dox” command. “Doxing” is a term used to describe the act of accidentally revealing sensitive data to the viewers (e.g. credentials). The feature I have added to the VirtualStudio solution lets channel moderators intervene and trigger an OBS scene change.
- For important events, such as a viewer gifting a subscription, we want to always reward the viewer with the special effects - meaning that we can cause OBS to display the virtual room scene, even if we’re currently looking at a “live coding” view.
The app uses the Unity HDRP pipeline and can take advantage of next-gen graphics such as DX12 realtime raytracing.
- For example, the “penthouse scene” reflects objects in the glass - it’s possible to see reflections of the back-of-the-sofa.
- Realtime raytracing absolutely murders the GPU. Initially, we have had to turn off the next-gen effects and run the system in a DX11 mode. Looking forward, there is some expensive NVidia hardware on back-order that should get things working better.
- Volumetric lighting and fog effects were used in the Halloween-themed graveyard scene to great effect.
- One benefit of this system not being a general-release game, is that it doesn’t need to be performance-optimised for many users. I’ve been able to be fairly cavalier with throwing expensive effects onscreen, because I know I have been targeting a fairly high end gaming system to run the show.
How did the project come about?
The “penthouse” scene actually has its roots in original artwork created by superb Tunisian artist LifeDesignz.
You can still see this 2D artwork in use as the “starting soon” scenes on Layla’s Twitch stream.
I wanted to practice my 3D modelling and use the artwork as inspiration for a 3D realisation. Initially I only intended it to be a static image rendered in Blender.
Around the same time, I had also been doing a tonne of work using web-based TwitchBot integrations and, separately, various augmented reality projects using Unity. Having a (somewhat eclectic) mix of skillsets at hand, the idea formed to try and bring everything together and create a fun interactive environment.
I’ve not watched the Twitch Channel, what does it all look like?
Here is a small selection of screenshots from the project