Your UI in VR

Most people simply say “VR”. Developers say immersive VR for the full 360-degree 3D view inside the headset: the viewer is inside the whole 3D world and can look all around. This page uses the term because what a viewer can see and use changes completely at that moment.

A headset does not go into immersive VR by itself. When the viewer opens your link in the headset's browser (for example on a Meta Quest), the stream first appears as a normal page in a browser window — the non-immersive mode. Only when the viewer presses the VR button does the headset switch into immersive VR. Where that button is, and how the viewer leaves again, works a little differently on each headset.

The 360-degree 3D world cannot show any 2D UI or 2D menus. And when UI is shown there, seeing it does not mean the viewer can use it: using it is a separate piece of work.

When people ask about UI in a streamed app, they usually mean one of three different things. Working messages do not make a button visible, and a visible button is not automatically one you can press. These are three separate things, and each one has to be implemented and handled separately:

There are two UIs in Pixel Streaming

And three topics

1. Visibility — can the viewer see the UI? In immersive VR (the 360-degree 3D view inside the headset), anything 2D is not shown: neither the web page nor the app's normal 2D menus. To be seen in VR, UI has to become a 3D object in the world.
2. Interaction — can the viewer press it? Seeing a UI does not mean you can use it. In VR there is no mouse, so pressing a 3D menu needs extra work: something that acts as the mouse inside the 3D world.
3. Messaging — can the web page and the app exchange data? These are messages sent by code, between the web page and the Unreal, Unity or Omniverse app, in both directions. The viewer does not touch anything for them. They travel with the video, so they work the same with or without VR.

So “do messages still work in VR?” and “can my viewers use my UI in VR?” are different questions. The first is yes, with no work. The second needs the UI to be made visible and usable in the 3D world, as the rest of this page explains.

In a non-VR stream (normal Pixel Streaming), a menu is a flat picture drawn on top of the 3D view. A VR headset draws a separate 3D picture for each eye, so that the world has depth. A flat overlay has no depth, so it cannot be shown correctly to both eyes — Unreal does not draw 2D UI in stereo at all. And while the viewer is inside the headset's immersive VR mode (the 360-degree 3D view inside the headset), the browser page around the stream is not shown either.

So the messages are not the problem: your page and your app can keep talking in VR. What is missing is a control the viewer can see and press while in the headset. Every control has to become a 3D object — or the viewer has to step out of VR to use it.

Yes. Messages between your web page and your Unreal app travel over the same connection as the video, so they keep working while the viewer is in immersive VR (the 360-degree 3D view inside the headset), exactly as in a non-VR stream. Your page can still send to the app, and the app can still send to your page.

What changes in VR is only what the viewer can see and press: your page's buttons are not visible inside the headset. If you use messages so that viewers can work your UI, the sections below show how to give them UI they can use in VR.

Unreal solves this with 3D UI: the same UMG widget you already built, shown on a flat panel that is placed in the world instead of on the screen. Two components do it:

• Widget Component — puts a Widget Blueprint into the level as a 3D panel.
• Widget Interaction — added to the motion controller, it casts a ray like a laser pointer and turns a trigger press into a click on whatever widget it hits.

How it works: a 2D UMG menu becomes a 3D object

Your Unreal menu is a UMG widget — a 2D UI. On its own it can only be drawn flat on the screen, which is why it disappears in immersive VR (the 360-degree 3D view inside the headset). A 3D widget gives that same 2D UI a place in the world. This is what happens, in order:

1. You add a Widget Component to an actor. Any actor can carry one — a sign, a kiosk, a stand in the room, or the viewer's own pawn.
2. You pick which UMG it shows. The Widget Component has a field, Widget Class, where you select your existing Widget Blueprint (the 2D UMG menu). Nothing in the menu itself has to change.
3. You place the actor in the world. When the actor is created, its Widget Component creates a 3D surface at that spot. On its own the surface is hollow — it has nothing on it.
4. The UMG you picked fills that hollow surface. Unreal draws your 2D menu onto it, and from then on the menu renders like any other 3D object in the level: it has a position, a size and a depth, and it is drawn for both eyes.
5. Now every viewer can see it — in immersive VR (the 360-degree 3D view inside the headset), in the headset's normal browser window, and in a normal non-VR stream.
6. Seeing is not clicking. In the 3D world there is no mouse cursor, so the menu cannot be pressed yet. The player needs a Widget Interaction component as well: attached to the player (usually its motion controller), it works like a mouse for the 3D world — it points a ray at the 3D widget, and a trigger press becomes a click on whatever button the ray touches.

With all three in place, the 2D UI of Unreal — UMG — is available inside immersive VR (the 360-degree 3D view inside the headset). The steps below set it up.

Put the menu into the world

1. Keep your existing Widget Blueprint (your menu). No need to rebuild it.
2. Open the actor that should carry the menu — a stand in the room, or the VRPawn if the menu should follow the viewer — and Add Component → Widget.
3. In the component's details set Widget Class to your Widget Blueprint and Space to World. World space is what makes it a 3D object both eyes can see.
4. Set Draw Size to the menu's pixel size (for example 1280 × 720) and scale the component down so the panel is a comfortable size about 1–2 metres in front of the viewer.
5. Leave Receive Hardware Input off — Epic's guidance for VR. Input comes from the controller below instead.

Let the controller click it

1. In the VRPawn, add a Widget Interaction component as a child of the right (or left) motion controller, so it points where the controller points.
2. Set Interaction Distance to a few metres and, while testing, turn on Show Debug to see the ray.
3. On the controller's trigger pressed event call Press Pointer Key (Left Mouse Button); on released call Release Pointer Key. Buttons in the panel now react like mouse clicks.
4. Give each hand its own Pointer Index if both hands get a pointer.

Keep your web page informed

When a button in the 3D panel does something your page needs to know — a comment saved, a finish changed — send the same message to the page that you send today. Messages work in VR exactly as in a non-VR stream, so your page and your back end stay in step even though the viewer used the in-world panel.

The quickest solution needs no work at all. The viewer leaves immersive VR (the 360-degree 3D view inside the headset) — on a Meta Quest by pressing the controller's menu (Meta) button and choosing to exit, or with the browser's own exit control — uses your web page as normal, then enters VR again with the same button they used the first time. The stream keeps running in between.

It works, but every switch takes the viewer out of the experience, so it suits actions that happen rarely: picking a different house, saving a summary at the end. Tell viewers about it on the page, for example “Press the menu button to leave VR and use these controls”.

With our iframe and Web SDK integrations, the page around the stream is yours, so you can shape it for VR viewers: put the choices that matter before the viewer enters VR (house, finish set, starting room) on a clear start screen, and send them to the app as messages before the viewer enters immersive VR (the 360-degree 3D view inside the headset). Controls inside the 360-degree view itself still have to come from the app (way 1) — building 3D controls in the web page is web VR development work.

Because the iframe and the Web SDK leave the page to you, a developer can also build a 3D (WebXR) version of the page's HTML controls. Eagle 3D Streaming does not have a demo of this yet; if we build one, we will publish it here. Today the way we can show you is the 3D widget inside your Unreal app (way 1).

• Controls used during the visit (comments, materials, floor cut-through): build them as a 3D panel (way 1). This is the experience VR viewers expect.
• Choices made once (which project, which variant): a start screen on your page (way 3).
• Everything else, or while you build way 1: let viewers step out of VR (way 2).

Setting up the VR project itself is covered in Set up a VR project, and testing it in a headset in Test VR locally.