Merge pull request #142 from EpicGames/master

Update 5.2 from master

Frontend/Docs/Customizing Player Input Options.md

@@ -1,8 +1,10 @@
-**TO DO**: Update this information to match the new front end.
+## Configuring library behaviour
+
+The frontend library exposes a number of configuration options through the [Config](/Frontend/library/src/Config/Config.ts) class. The values of these options can be modified to tweak certain inbuilt behaviour of the frontend.
 
-## Customizing Player Input Options
+The following options are available in the frontend library:
 
-The `app.js` file offers some JavaScript configuration parameters that you can override in your custom player page to control the way the player widget responds to user interactions. The `inputOptions` object exposes the following properties:
+**TO DO**: Update this information to match the new front end.
 
 | Property | Default | Description |
 | --- | --- | --- |

Frontend/Docs/Settings Panel.md

@@ -0,0 +1,80 @@
+# Frontend Setting Panel
+
+The Pixel Streaming frontend contains a settings panel that allows you to configure your Pixel Streaming user experience as needed - whether that be toggling features or connecting to different signalling servers.
+
+This page will be updated with new features and commands as they become available.
+
+
+## Settings
+![Settings Panel](Resources/Images/settings-panel.png)
+
+### Pixel Streaming
+
+| **Setting** | **Description** |
+| --- | --- |
+| **Signalling URL** | The URL of the signalling server. |
+| **Streamer ID** | Allows you to select which streamer to stream. |
+| **Auto connect to stream** | Browser will automatically connect to the stream when loaded. Prevents having to click to start |
+| **Auto play video** | When the stream is ready, starts playing video immediately instead of showing a play button. |
+| **Browser send offer** | The browser will start the WebRTC handshake instead of the Unreal Engine application. This is an advanced setting for users customising the frontend. Primarily for backwards compatibility for 4.x versions of the engine. |
+| **Use microphone** | Will start receiving audio input from your microphone and transmit it to the Unreal Engine. |
+| **Start video muted** | Muted audio when the stream starts. |
+| **Prefer SFU** | Will attempt to use the Selective Forwarding Unit (SFU), if you have one running. |
+| **Is quality controller?** | Makes the encoder of the Pixel Streaming Plugin use the current browser connection to determine the bandwidth available, and therefore the quality of the stream encoding. **See notes below** |
+| **Force mono audio** | Force the browser to request mono audio in the SDP. |
+| **Force TURN** | Will attempt to connect exclusively via the TURN server. Will not work without an active TURN server. |
+| **Suppress browser keys** | Suppress or allow certain keys we use in UE, for example F5 to show shader complexity instead of refreshing the page. |
+| **AFK if Idle** | Timeout the connection if no input is detected for a period of time. |
+| **AFK timeout** | Allows you to specify the AFK timeout period. |
+
+
+### UI
+| **Setting** | **Description** |
+| --- | --- |
+| **Match viewport resolution** | Resizes the Unreal Engine application resolution to match the browser's video element size.|
+| **Control scheme** | If the scheme is `locked mouse` the browser will use `pointerlock` to capture your mouse, whereas if the scheme is `hovering mouse` you will retain your OS/browser cursor. |
+| **Color scheme** | Allows you to switch between light mode and dark mode. |
+
+### Input
+| **Setting** | **Description** |
+| --- | --- |
+| **Keyboard input** | If enabled, captures and sends keyboard events to the Unreal Engine application. |
+| **Mouse input** | If enabled, captures and sends mouse events to the Unreal Engine application. |
+| **Touch input** | If enabled, captures and sends touch events to the Unreal Engine application. |
+| **Gamepad input** | If enabled, captures and sends gamepad events to the Unreal Engine application. |
+| **XR controller input** | If enabled, captures and sends XR controller events to the Unreal Engine application. |
+
+### Encoder
+| **Setting** | **Description** |
+| --- | --- |
+| **Min QP** | The lower bound of quantization parameter (QP) of the encoder. 0 = best quality, 51 = worst quality. |
+| **Max QP** | The upper bound of quantization parameter (QP) of the encoder. 0 = best quality, 51 = worst quality. |
+| **Preferred codec** | The preferred codec to be used during codec negotiation. |
+
+### WebRTC
+| **Setting** | **Description** |
+| --- | --- |
+| **Max FPS** | The maximum FPS WebRTC will attempt to transmit frames at. |
+| **Min Bitrate (kbps)** | The minimum bitrate WebRTC should use. |
+| **Max Bitrate (kbps)** | The maximum bitrate WebRTC should use. |
+
+
+### Commands
+| **Setting** | **Description** |
+| --- | --- |
+| **Show FPS** | Will display the current FPS |
+| **Request Keyframe** | Will ask the stream for a keyframe. This is helpful if your stream is choppy and needs to catch up.  |
+| **Restart Stream** | Restarts the stream by disconnecting and reconnecting the websocket connection. |
+
+
+### Notes
+
+**Quality Controller**
+Although Pixel Streaming adapts the quality of the stream to match the available bandwidth, when using H.264, the video frames are only encoded once by the Pixel Streaming Plugin. That one encoding is used for all clients. Therefore, only one client connection can "own" the quality used for adaptive streaming. If the other clients have a much better connection to the server, they may end up seeing a lower quality stream than necessary. On the other hand, if other clients have a much worse connection to the server, they may end up with lag or jitter. **Note**, this quality controller setting is completely irrelevant if you are streaming using any codec other than H.264 (such as VP8 or VP9).
+
+
+## Legal
+
+Copyright © 2022, Epic Games. Licensed under the MIT License, see the file [LICENSE](../../LICENSE.md) for details.
+
+

Frontend/README.md

@@ -1,63 +1,63 @@
-# Pixel Streaming Frontend Library
+# Pixel Streaming Frontend
 
-This is frontend javascript library for Epic Games' Pixel Streaming. 
+The **frontend** refers to the HTML, CSS, images, and JavaScript/TypeScript code that runs in web browsers and allows them to connect to Unreal Engine Pixel Streaming applications and interact with them. The frontend library is the foundation that developers can modify and extend to suit the needs of their Pixel Streaming experience.
 
+The frontend consists of two packages:
 
-The core features of this library are:
+1. [lib-pixelstreamingfrontend](/Frontend/library/): the core Pixel Streaming frontend for WebRTC, settings, input, and general functionality.
+2. [lib-pixelstreamingfrontend-ui](/Frontend/implementations/EpicGames): the reference UI that users can either optionally apply on top of the core library or build on top of.
 
-- Create a websocket connection to communicate with the signalling server.
-- Create a WebRTC player that displays the Unreal Engine video and audio.
-- Handling of input from the user and transmitting it back to Unreal Engine.
-- Opens a datachannel connection sending and receiving custom data (in addition to input).
+
+## Docs
+- [Customizing the Player Webpage](Docs\Customizing%20the%20Player%20Webpage.md)
+- [HTML Page Requirements](Docs\HTML%20Page%20Requirements.md)
+  - [Player File Location and URL](Docs\HTML%20Page%20Requirements.md)
+- [Customizing Player Input Options](Docs/Customizing%20Player%20Input%20Options.md)
+  - [Disabling User Input](Docs/Customizing%20Player%20Input%20Options.md)
+- [Customizing the Player Widget Style](Docs/Customizing%20the%20Player%20Widget%20Style.md)
+- [Accessing the Pixel Streaming Blueprint API](Docs/Accessing%20the%20Pixel%20Streaming%20Blueprint%20API.md)
+- [Communicating from the Player Page to UE5](Docs/Communicating%20from%20the%20Player%20Page%20to%20UE5.md)
+  - [Using the emitCommand Function](Docs/Communicating%20from%20the%20Player%20Page%20to%20UE5.md)
+  - [Using the emitUIInteraction Function](Docs/Communicating%20from%20the%20Player%20Page%20to%20UE5.md)
+- [Communicating from UE5 to the Player Page](Docs/Communicating%20from%20UE5%20to%20the%20Player%20Page.md)
+- [Timing Out Inactive Connections](Docs/Timing%20Out%20Inactive%20Connections.md)
+
+## Integrating the libraries into your project
+Tthe TypeScript libraries are provided as both an [NPM](https://www.npmjs.com/settings/epicgames-ps/packages) package and a [UMD](https://github.com/umdjs/umd) module (available via [unpkg](https://unpkg.com/)), making it easy to consume the libraries from either TypeScript code or plain JavaScript code using modern web development tools and workflows.
 
 ## Usage from source
 
 When developing your own Pixel Streaming experience the intent is you will start with this library and extend it through the use of 
-its public API. We have provided an example of this workflow in our implementations/EpicGames, which is an implementation of this library.
+its public API. We have provided an example of this workflow in our [implementations/EpicGames](/Frontend/implementations/EpicGames), which is an implementation of this library.
 
 ## Contributing
 
 If part of the library is not exposed and you wish to extend it, please do so in your own branch and open a pull request with your change for our consideration.
 
 ## Developing
 
-Changes to the library occur in the /library directory and require you to have NodeJS installed as part of your development environment.
-Once you have NodeJS installed, 
+Changes to the library occur in the [/library](/Frontend/library) directory and require you to have NodeJS installed as part of your development environment.
+Once you have NodeJS installed: 
 
 - `cd library`
 - `npm install`
 - `npm run build`
 
-The default user interface is provided in /ui-library directory. You can either use it or provide your own user interface. To build the default UI, run
+The default user interface is provided in [/ui-library](/Frontend/ui-library) directory. You can either use it or provide your own user interface. To build the default UI, run:
 - `cd ui-library`
 - `npm install`
 - `npm run build`
 
+This will produce `player.js` under the `SignallingWebServer/Public` directory - this is the default UI.
 
-If you are developing your implementation based on the library, the process is similar:
+### Making your own UI
 
+We recommend studying [/ui-library](/Frontend/ui-library) and [player.ts](/Frontend/implementations/EpicGames/src/player.ts)/[player.html](/Frontend/implementations/EpicGames/src/player.html), then once you have copied and modified the [package.json](/Frontend/implementations/EpicGames/package.json) and `.ts` into your own `implementation/your_implementation` directory, the process is similar:
 
 - `cd implementation/your_implementation`
 - `npm build-all`
 
 
 ## Legal
 
-Copyright © 2022, Epic Games. Licensed under the MIT License, see the file [LICENSE](./LICENSE) for details.
-
-# Table of Contents
-- [Customizing the Player Webpage](Docs\Customizing%20the%20Player%20Webpage.md)
-- [HTML Page Requirements](Docs\HTML%20Page%20Requirements.md)
-  - [Player File Location and URL](Docs\HTML%20Page%20Requirements.md)
-- [Customizing Player Input Options](Docs/Customizing%20Player%20Input%20Options.md)
-  - [Disabling User Input](Docs/Customizing%20Player%20Input%20Options.md)
-- [Customizing the Player Widget Style](Docs/Customizing%20the%20Player%20Widget%20Style.md)
-- [Accessing the Pixel Streaming Blueprint API](Docs/Accessing%20the%20Pixel%20Streaming%20Blueprint%20API.md)
-- [Communicating from the Player Page to UE5](Docs/Communicating%20from%20the%20Player%20Page%20to%20UE5.md)
-  - [Using the emitCommand Function](Docs/Communicating%20from%20the%20Player%20Page%20to%20UE5.md)
-  - [Using the emitUIInteraction Function](Docs/Communicating%20from%20the%20Player%20Page%20to%20UE5.md)
-- [Communicating from UE5 to the Player Page](Docs/Communicating%20from%20UE5%20to%20the%20Player%20Page.md)
-- [Timing Out Inactive Connections](Docs/Timing%20Out%20Inactive%20Connections.md)
-
-
-
+Copyright © 2023, Epic Games. Licensed under the MIT License, see the file [LICENSE](./LICENSE) for details.

Frontend/implementations/EpicGames/docs/configuration.md

@@ -0,0 +1,3 @@
+# Configuration of the Frontend UI
+
+Todo

Frontend/implementations/EpicGames/docs/images/images.md

@@ -0,0 +1 @@
+The images directory

Frontend/implementations/EpicGames/readme.md

@@ -0,0 +1,17 @@
+## lib-pixelstreamingfrontend-ui
+
+A plugin library that can be optionally applied on top of the core library to create a default (but fully featured) Pixel Streaming experience. 
+
+**This is great starting point for building your UI or studying the Pixel Streaming feature set.**
+
+![Frontend](/Frontend/implementations/EpicGames/docs/images/frontend.jpg)
+
+### Key features
+- An info panel (screen right) that provides a UI for displaying live statistics to the user.
+- A settings panel (screen right) that provides a UI for all the options inside [config.ts](/Frontend/library/src/Config/Config.ts).
+- A set of controls (screen left) to maximize the video, open the settings panel, open the info panel, and enter VR mode.
+- Ability to display overlays that present information or errors to the user, or present prompts for the user to interact with.
+
+
+### Adding it to your project
+`npm i @epicgames-ps/lib-pixelstreamingfrontend-ui-ue5.2`

Frontend/implementations/EpicGames/src/player.ts

@@ -2,9 +2,9 @@
 
 import { Config, PixelStreaming } from '@epicgames-ps/lib-pixelstreamingfrontend-ue5.2';
 import { Application, PixelStreamingApplicationStyle } from '@epicgames-ps/lib-pixelstreamingfrontend-ui-ue5.2';
-export const PixelStreamingApplicationStyles =
+const PixelStreamingApplicationStyles =
     new PixelStreamingApplicationStyle();
-
+PixelStreamingApplicationStyles.applyStyleSheet();
 
 document.body.onload = function() {
 	// Example of how to set the logger level
@@ -15,7 +15,10 @@ document.body.onload = function() {
 
 	// Create a Native DOM delegate instance that implements the Delegate interface class
 	const stream = new PixelStreaming(config);
-	const application = new Application({ stream });
+	const application = new Application({
+		stream,
+		onColorModeChanged: (isLightMode) => PixelStreamingApplicationStyles.setColorMode(isLightMode)
+	});
 	// document.getElementById("centrebox").appendChild(application.rootElement);
 	document.body.appendChild(application.rootElement);
 }

Frontend/implementations/EpicGames/src/stresstest.ts

@@ -2,8 +2,9 @@
 
 import { Config, Flags, PixelStreaming } from '@epicgames-ps/lib-pixelstreamingfrontend-ue5.2';
 import { Application, PixelStreamingApplicationStyle } from '@epicgames-ps/lib-pixelstreamingfrontend-ui-ue5.2';
-export const PixelStreamingApplicationStyles =
+const PixelStreamingApplicationStyles =
     new PixelStreamingApplicationStyle();
+PixelStreamingApplicationStyles.applyStyleSheet();
 
 // This is the entrypoint to the stress test, all setup happens here
 export class StressTester {
@@ -146,7 +147,10 @@ export class StressTester {
 
 		// Create a Native DOM delegate instance that implements the Delegate interface class
 		const stream = new PixelStreaming(config);
-		const application = new Application({ stream });
+		const application = new Application({
+			stream,
+			onColorModeChanged: (isLightMode) => PixelStreamingApplicationStyles.setColorMode(isLightMode)
+		});
 		streamFrame.appendChild(application.rootElement);
 		return streamFrame;
 	}

Frontend/library/readme.md

@@ -0,0 +1,15 @@
+## lib-pixelstreamingfrontend
+
+The core library for the browser/client side of Pixel Streaming experiences. **This library contains no UI.**
+
+See [lib-pixelstreamingfrontend-ui](/Frontend/implementations/EpicGames) for an example of how to build UI on top of this library.
+
+### Key features
+- Create a websocket connection to communicate with the signalling server.
+- Create a WebRTC player that displays the Unreal Engine video and audio.
+- Handling of input from the user and transmitting it back to Unreal Engine.
+- Opens a datachannel connection sending and receiving custom data (in addition to input).
+
+### Adding it to your project
+`npm i @epicgames-ps/lib-pixelstreamingfrontend-ue5.2`
+

Frontend/library/src/Config/Config.ts

@@ -27,6 +27,11 @@ export class Flags {
     static StartVideoMuted = 'StartVideoMuted' as const;
     static SuppressBrowserKeys = 'SuppressBrowserKeys' as const;
     static UseMic = 'UseMic' as const;
+    static KeyboardInput = 'KeyboardInput' as const;
+    static MouseInput = 'MouseInput' as const;
+    static TouchInput = 'TouchInput' as const;
+    static GamepadInput = 'GamepadInput' as const;
+    static XRControllerInput = 'XRControllerInput' as const;
 }
 
 export type FlagsKeys = Exclude<keyof typeof Flags, 'prototype'>;
@@ -398,6 +403,61 @@ export class Config {
             )
         );
 
+        this.flags.set(
+            Flags.KeyboardInput,
+            new SettingFlag(
+                Flags.KeyboardInput,
+                'Keyboard input',
+                'If enabled, send keyboard events to streamer',
+                true,
+                useUrlParams
+            )
+        );
+
+        this.flags.set(
+            Flags.MouseInput,
+            new SettingFlag(
+                Flags.MouseInput,
+                'Mouse input',
+                'If enabled, send mouse events to streamer',
+                true,
+                useUrlParams
+            )
+        );
+
+        this.flags.set(
+            Flags.TouchInput,
+            new SettingFlag(
+                Flags.TouchInput,
+                'Touch input',
+                'If enabled, send touch events to streamer',
+                true,
+                useUrlParams
+            )
+        );
+
+        this.flags.set(
+            Flags.GamepadInput,
+            new SettingFlag(
+                Flags.GamepadInput,
+                'Gamepad input',
+                'If enabled, send gamepad events to streamer',
+                true,
+                useUrlParams
+            )
+        );
+
+        this.flags.set(
+            Flags.XRControllerInput,
+            new SettingFlag(
+                Flags.XRControllerInput,
+                'XR controller input',
+                'If enabled, send XR controller events to streamer',
+                true,
+                useUrlParams
+            )
+        );
+
         /**
          * Numeric parameters
          */