WebViewStream

WebViewStream is a Unity-based HoloLens 2 application for high-quality, low-latency WebRTC streaming from multiple endpoints, optimized with Microsoft's Mixed Reality Toolkit (MRTK) and WebView package. It offers an extensible, high-performance streaming setup that can scale to multiple streams, limited only by the HoloLens 2 hardware.

Features

• MRTK-Native: Built on MRTK for optimized HoloLens 2 functionality.
• Optimized performance: Configured and fine-tuned for smooth, low-latency streaming on the HoloLens 2.
• WebRTC-based streaming: Uses WebRTC for low-latency, high-quality streaming.
• Dynamic endpoint management: Reads streaming endpoints from a REST API GetEndpoints, simplifying endpoint updates without typing long URLs directly on the HoloLens.
• mDNS client: Compliant with RFC 6762 and RFC 6763 for service discovery.
• Extendable streaming capacity: Supports two streams by default and can scale up, limited only by HoloLens 2's hardware.
• Hand menu and speech commands: Offers an intuitive HandMenu and voice control for hands-free operation.
• Recording: Allows streaming recording, with minor frame drops under high-load conditions.

Prerequisites

• Unity Hub
• Unity 2021.3.37f1 (LTS) with the following modules:
  • Universal Windows Platform Build Support
  • Windows Build Support (IL2CPP)
• Visual Studio 2022 with the following features (workloads):
  • .NET desktop development
  • Desktop Development with C++7
  • Universal Windows Platform development
  • Game Development with Unity
• HoloLens 2

Installation

1. Clone the repository.
2. Open the project with Unity Hub.
3. Build the application. Make sure to build for UWP with the following configurations:
   • Architecture: ARM 64-bit
   • Build Type: D3D project
   • Build Configuration: Master (if you are developing, you can choose Debug or Release instead).
     Then, click Build and select a directory. Create a folder called Build within the project for better organization. If, for some reason, you get an error such as "IOException: Failed to move ... from ... to ...", you will likely need to exclude that folder from your antivirus. Exclude it and rebuild the project; it should work now.
4. Open the Visual Studio solution called WebViewStream.sln. It will be in the Build folder.
5. Connect the HoloLens via USB or Wi-Fi. Ensure Developer Mode is enabled on the HoloLens.
   • USB: Choose the following options at the top: Master, ARM64, and Device. Then click the Device button.
   • Wi-Fi: Choose Remote Machine, then click on the WebViewStream Debug Properties by selecting the arrow on the right of the button. Next, click Debugging, and under Machine Name, enter the HoloLens IP, for example: 192.168.150.4. Exit the menu and click the Remote Machine button.

Technical details

• Platform: HoloLens 2 (UWP)
• Language: C# (Unity)
• Required packages:
  • com.microsoft.mixedreality.openxr
  • com.microsoft.mixedreality.toolkit.foundation
  • com.microsoft.mixedreality.toolkit.standardassets
  • com.microsoft.mixedreality.webview.unity

Note: Install dependencies from my mixedreality repository. Only use the specific libraries and versions hosted there, as they have been patched for this project's requirements. Using different versions may lead to errors and compatibility issues.

API requirements

The app requires an external API for endpoint retrieval (GetEndpoints), expected to be accessible within the HoloLens 2's network environment.

The application supports local service discovery using UDP broadcasting. This is particularly useful in scenarios where DNS is down or when setting up a DNS record is not feasible.