This article was first published in November 2020 and has been refreshed in April 2021.

For the better part of the past year, I’ve been working on a brilliantly fun, but brutally hard-going, Unity project.

The solution primarily provides an interactive 3D virtual background that is used during the production of a Twitch Livestream. It combines many other features, integrations and automations.

A primary feature is an integration to the Twitch API that opens the door for a whole bunch of cool interactions with the stream.

This project has formed part of my ongoing support for @laylacodesit and her Live-Coding Twitch channel.

Unlike the blow-by-blow instructions that you’ll find in many of my other articles, this “Projects” section is intended to be a tour of some of the stuff that I’ve been working on. So to be up-front in managing reader expectations, this is NOT a how-to guide! If you want to replicate these projects, you’ll need to know the tools already and figure out the implementation details yourself - but hopefully, they can be a source of useful hints and inspiration for folks.

So, what exactly is the Virtual Studio solution?

BTW, that reads Virtual Studio - not Visual Studio!

Folks who watch Twitch streams are quite accustomed to streamers using a greenscreen to composite themselves into a scene that uses a static image as a background. That’s mainstream - pretty much everyone does that.

What we’ve been doing with @LaylaCodesIt ’s Twitch stream, since around summer 2020, is to try something a bit more ambitious and try compositing video feeds into a realtime rendered 3D scene.

During a Twitch stream, we often get viewers mention that “they love the Twitch Theme” - assuming it’s just a static background with regular OBS greenscreen compositing - it’s so much more than that!

Note: A common question we get is “is the camera mounted on a computer-controlled slide?” - it’s not a physically moving camera, but instead a smoke and mirrors trick with how the camera feed is composited.

The Virtual Studio system does a lot of things, but here is a rough overview:

• Realtime composition of multiple NDI camera feeds into the virtual environment.
  • Superimposing chromakey-cutout of the presenter onto 3D plane within the virtual scene.
  • Integrating video capture feeds into various virtual props (e.g. a laptop screen or a sci-fi style holographic projector).

• At time of writing there are four distinct themed virtual environments - but I have plans for many more.

  • For the “penthouse” scene I’ve used a mixture of home-made and purchased 3rd-party models. I’m a completely novice 3D modeller, so for those times when my skill didn’t match my ambition (e.g. plants and similar organic shapes), it made sense to buy the assets in. Other stuff like the room itself and furniture I modelled.
  • For the “halloween” scene, I was more ambitious with my modelling, creating pretty much everything that you see myself.
  • The “winter” scene continued to push the envelope in terms of on-screen visuals. This time around, because of the huge amount of complex modelling, I purchased an asset pack for the trees and rocks. There was still plenty for me to do myself though, as I created the dynamically lighting “sunset” and “snowstorm” effects myself.
  • Inspired by the holiday season 2020 release of the videogame “Cyberpunk 2077” we had a stab at creating something reminiscent. This originated from yet another purchased asset, but was fairly heavily adapted and modified to support HDRP and various custom textures etc.

• There is a “virtual desk” prop which is intended to be re-used across multiple scenes. It forms part of a strategy to centralise and re-use many of the features.

• An integration with Twitch, via the TwitchAPI, is heavily used to react to a wide range of events.

  • In some respects, the solution could be thought of as a TwitchBot on steroids.
  • Twitch “subscriptions” trigger “fun animated events”, such as sound effects, a shower of party balloons. Subscriptions are visualised as “gift boxes” which are spawned and sit on a desktop for the duration of the stream.
  • Twitch “cheers” trigger a virtual coin-drop, which is a similar effect to the subscription gift-boxes. The coins pile-up on a desktop.
  • “Channel Point redeems” trigger weird pop-culture effects - such as Darth Vader popping up from behind the sofa.
  • “Channel commands” can be used to trigger special effects - for example, I can make it rain in the scene.

• There is a two-way websocket integration with OBS itself.

  • Specific scene changes in OBS trigger a change in the Unity app, changing view perspectives etc.
  • Events in the Scene, typically raised from Twitch commands, can trigger OBS scene changes.
    • For example, we have a “!dox” command. “Doxing” is a term used to describe the act of accidentally revealing sensitive data to the viewers (e.g. credentials). The feature I have added to the VirtualStudio solution lets channel moderators intervene and trigger an OBS scene change.
    • For important events, such as a viewer gifting a subscription, we want to always reward the viewer with the special effects - meaning that we can cause OBS to display the virtual room scene, even if we’re currently looking at a “live coding” view.

• The app uses the Unity HDRP pipeline and can take advantage of next-gen graphics such as DX12 real-time raytracing.

  • For example, the “penthouse scene” reflects objects in the glass - it’s possible to see reflections of the back-of-the-sofa.
  • Volumetric lighting and fog effects are used in several scenes to great effect.
  • Raytraced shadows are used to create the real-time shadow effect behind Layla.
    
  • Realtime raytracing absolutely murders the GPU. Initially, in summer 2020, we have had to turn off the next-gen effects and run the system in a DX11 mode. When NVidia later released their new series of GPU’s, in the holiday season 2020, we shelled out big-time and picked up a top of the range (as of the time of writing) RTX3090. With hardware support for raytracing effects, this has unlocked a huge amount of visual fidelity.
  • One benefit of this system not being a general-release game, is that it doesn’t need to be performance-optimised for many users. I’ve been able to be fairly cavalier with throwing expensive effects onscreen, because I know I have been targeting a fairly high-end gaming system to run the show.

How did the project come about?

The “penthouse” scene actually has its roots in original artwork created by superb Tunisian artist LifeDesignz.

You can still see this 2D artwork in use as the “starting soon” scenes on Layla’s Twitch stream.

I wanted to practice my 3D modelling and use the artwork as inspiration for a 3D realisation. Initially I only intended it to be a static image rendered in Blender.

Around the same time, I had also been doing a tonne of work using web-based TwitchBot integrations and, separately, various augmented reality projects using Unity. Having a (somewhat eclectic) mix of skillsets at hand, the idea formed to try and bring everything together and create a fun interactive environment.

I’ve not watched the Twitch Channel, what does it all look like?

Here is a small selection of screenshots from the project

Problems

I’ve personally found this project to be really hard-gong at times. I don’t have a background or experience in the video game industry (nor did I have anyone to ask for help), so much of the problem-solving has meant just persevering.

This has translated to spending hours and hours just slowly trying to unravel problems and pretty much figure it out as I went along. Some problems could have had me stumped for 2-weeks straight :-(

Here are some of the highlights:

• The Unity video reprojection uses virtual NDI feeds. Out of the box, Unity does not play with OBS virtual cameras. Initially we used OBS VirtualCam OBS plugin. This functionality was more recently added directly into OBS, so that project has been deprecated. Making it work with Unity was partly addressed by a “known issue” link at the of the project Readme. Making it work required a registry edit. What isn’t documented, is that to make it work with multiple Virtual Cameras, you need to add additional registry entries for each feed.

• The project uses TwitchLib for Unity. A problem I faced (which has since been updated by the community) was that the last version of this library has been published in 2018 - a whole two years before I started the project.

  • Although the main TwitchLib library has continued to be updated, this meant that I was stuck with a Unity version that simply didn’t work for features such as Twitch API subscription events.
  • Unable to find support to update the TwitchLib for unity library, I ended up recompiling my own custom version using the CLI tool “ILMerge.exe” to cherry-pick bits from each project.

• It works here, but not there. The classic “but it works on my computer!” problem almost killed the project off. I was faced with a problem where the project worked 100% fine in the Unity editor, but the player version was deeply unstable.

  • There were no meaningful error logs or patterns to figure out … just a nebulous “UnityPlayer.dll caused an Access Violation (0xc0000005)”. Most of the time it would crash on startup, occasionally it would fire up but then crash randomly a few minutes in.
  • There was no way to google the answer - countless game-players had reported the same error when playing multiple other Unity-created games, but no Unity Dev had chimed in with suggestions for a fix. A large part of this diagnosis difficulty was that the error was completely generic. The problem is a low-level issue within Unity itself.
  • The crashing behaviour was wildly different on two almost identical PCs - one crashed 96% of the time (literally, it worked like 2 in 50 attempts), whilst it worked ok most of the time with the other machine.
  • Identifying the cause of instability was like finding a needle in a haystack. At one point I thought I had solved the problem, by reducing scenery complexity. Anecdotal this seemed to solve the problem, but this was just a red herring.
  • The “fix” I eventually settled on, that for me restored stability, was to deactivate the “Static Batching” option in the player build settings. This should be a performance option, but it was killing my project.
  • Finding this was literally a case of “fiddling with knobs and buttons” until something worked. I can’t claim to have figured it out logically, I literally stumbled on it after sticking with it for weeks.

• Performance Sucks. Even on a really strong gaming rig, the performance of my solution is horrible and trying to optimize the project has taken countless days. Unity’s HDRP render pipeline is expensive at the best of times, but keeping a lid on things such as dynamic lighting has caused me problems.

  • A major source of the performance problems in this project, relate to the multiple chroma-cutout processing stages. Passing a relatively high-resolution video feed from OBS and into Unity as a constantly updating material texture. This causes a massive CPU-related bottleneck in the render pipeline.
  • Because of the tight integration with video feeds, it’s not possible to separate and spread the workload between separate machines.

• Project Management. This is similar to the problems every developer faces when dealing with large projects, but is magnified by the file size that Unity projects can expand to, when factoring in various Assets etc.

  • I had initially tried to share everything in a single large project. The idea was that shared resources (such as “Layla’s Desk” and all the attached game cameras) could be re-used across each scene.
  • This plan unravelled as the project grew with more scenes and the project grew to around 12Gb.
    
    • I found that I could be waiting for up to 10 minutes just for Unity to finish loading the project (yes, that is with 32Gb of Ram and an M.2 nvme drive).
      
    • I also was tripping up - and wasting a tonne of time - over making breaking changes to shared assets. e.g. an animation file that was altered to work in one scene, but inadvertently broke it for another scene.
  • Around Jan 2021, I recognised that I would have to “bite the bullet” and separate each scene into its own project. This:
    • Made the projects actually workable again, and brought the startup back down to around 1 minute.
    • Stopped me from making silly breaking changes. Introducing scene-specific tweaks didn’t cause problems elsewhere.
    • Did introduce the new problem that any changes I made in one project, that were needed in the other scenes, would have to be manually copied and configured elsewhere. This is a problem that will only get worse over time, so I’ll need to re-visit this with a strategy for sharing components between scenes.