Decouple UI from the frontend library (#110)

This PR splits the Frontend/library code into two parts: 1) /library - that provides an API for establishing Pixel Streaming sessions 2) /ui-library - that contains all the UI components like settings overlay panel, connection strength indicator, buttons, etc. The library code is intended to be used as a library through the Javascript/Typescript API, and it can be used programmatically without the default UI implemented in ui-library. This allows the developers to bring their own UI if they wish to customize the user experience, or even start a Pixel Streaming session without any overlay UI. The wish is to keep the library API stable and try to not make breaking changes if possible. If new non-breaking features are introduced to the API, it would be great if it was reflected in the version numbering following semantic versioning. Breaking changes should increase the major version number, while non-breaking changes increase only the minor or patch version number. * add jss dependencies to UI project * move PixelStreamingApplicationStyles to the UI project * remove jss dependencies from library project * add pixel streaming lib as ui dependency * renamed lib Application -> PixelStreaming * expose AFK callbacks from API * expose callbacks for overlays * onDismissAfk function added * fix possible NPE in statistics panel (Firefox only) * build both library projects in setup.sh|.bat * extracted overlays from PixelStreaming library to UI * bind callbacks to have access to `this` * emit onStatsReceived callback * replaced on\* callbacks with a typed event emitter * reuse types in on() and emit() * fix web-xr on Firefox (no navigator.xr available) * latency test API and callback events * move stats panel to UI project * move settings panel to UI project * extract video quality indicator to UI project * move buttons to UI project * events for freezeFrame functionality * import UI project in stresstest.ts * move UI wrapper elements to UI project * optional override for videoParentElement * created uiless.html and uiless.ts, which together are a sample UI application with no overlay UI * let -> const everywhere in sample applications * used named imports everywhere, not import * as libfrontend * document overrides * added onOpen, onClose, onError handlers for data channels * events for webRtc data channel open, close, error * added missing javadocs for classes and functions * renamed webRtcDisconnect -> webRtcDisconnected * added disconnect function to the API * use EventTarget to emit events * emit streamerListMessage, move new streamer select overlay to UI side * Emit config change events * API for changing settings * add getSettings() for symmetry * set partial initial settings in Config constructor * configure initial params in uiless.ts to auto start muted * add missing type exports * extracted config UI from the config components * configurable parameter saving to URL * save config params to URL only on demand * removed console.log * removed unused import * Extract light/dark mode config out of Pixel Streaming library * update labels for custom settings * Build the new library in setup scripts * added a fallback click-to-play handler into uiless.ts * hide non-public attributes in Config * make the public API more compact by hiding private attributes * added documentation for the public functions * underscored some methods in Config * make useUrlParameters read-only * underscore for Config event listeners: public API users use settingsChanged events * underscore for Config event listeners * fix imports: @epicgames-ps/lib-pixelstreamingfrontend-ue5.2, not -dev * added a comment to request function API docs: expected to be changed later * reorganized directory structure * ui-library build now uses NPM dependency, build-all linked filesystem dependency * added ui-library build to GH release * added github workflow for ui-library publishing * Link base library when building ui-library * Bring dispatchEvent, addEventListener and removeEventListener to the top level PixelStreaming class, simplifying API usage * Allow settings to take a default onChange listener. This listener has two params of the new value as well as the calling settings, allowing you to modify the underlying settings members in this callback * Fix overlay to properly notify is none or multiple streamers are connected * Make dark mode the default color scheme regardless of browser preference. * Make numeric settings spinner width take us less space. * Make SettingUIOption match styling width of numeric settings. * renamed EpicGames build-all-dev -> build-dev-all to match other projects * added Click to play overlay in uiless.html and show/hide it when needed * revised documentation for Config/*.ts * revised documentation in PixelStreaming.ts * documented EventEmitter events and event parameters * emit webRtcConnected event when connected * Fix max bitrate not being set correctly. * Update package-lock.json for ui-library * Refactor sendEncoderSettings to instead be sendEncoderMinQP and sendEncoderMaxQP * Bump version numbers of library NPM package to 0.1.0 and bump 5.2 release to 0.2.0 as all changes here a non-breaking as far as we have tested. --------- Co-authored-by: William Belcher <william.belcher@xa.epicgames.com> Co-authored-by: Luke Bermingham <1215582+lukehb@users.noreply.github.com>
EpicGames · Feb 22, 2023 · 7816672 · 7816672
1 parent 359b804
commit 7816672
Show file tree

Hide file tree

Showing 72 changed files with 10,226 additions and 1,845 deletions.
diff --git a/.github/workflows/create-gh-release.yml b/.github/workflows/create-gh-release.yml
@@ -36,6 +36,14 @@ jobs:
       working-directory: ./Frontend/library
       run: npm run build
 
+    - name: Install ui-library deps
+      working-directory: ./Frontend/ui-library
+      run: npm ci
+
+    - name: Build frontend ui-library
+      working-directory: ./Frontend/ui-library
+      run: npm run build-all
+
     - name: Install implementations/EpicGames deps
       working-directory: ./Frontend/implementations/EpicGames
       run: npm ci
@@ -54,7 +62,7 @@ jobs:
         path: 'PixelStreamingInfrastructure-${{ github.ref_name }}-${{ steps.getversion.outputs.version }}'
         type: 'tar'
         filename: '${{ github.ref_name }}-${{ steps.getversion.outputs.version }}.tar.gz'
-        exclusions: '.git .github output Frontend/Docs Frontend/library/dist Frontend/library/types Frontend/library/node_modules Frontend/implementations/EpicGames/node_modules'
+        exclusions: '.git .github output Frontend/Docs Frontend/library/dist Frontend/library/types Frontend/library/node_modules Frontend/ui-library/dist Frontend/ui-library/types Frontend/ui-library/node_modules Frontend/implementations/EpicGames/node_modules'
 
     - name: Archive Release .zip
       uses: thedoctor0/zip-release@0.7.1

diff --git a/.github/workflows/publish-ui-library-to-npm.yml b/.github/workflows/publish-ui-library-to-npm.yml
@@ -0,0 +1,22 @@
+name: Publish ui-library package to npmjs
+on:
+  push:
+    branches: ['UE5.2']
+    paths: ['Frontend/ui-library/package.json']
+jobs:
+  build:
+    runs-on: ubuntu-latest
+    defaults:
+      run:
+        working-directory: Frontend/ui-library
+    steps:
+      - uses: actions/checkout@v3
+      - uses: actions/setup-node@v3
+        with:
+          node-version: '16.x'
+          registry-url: 'https://registry.npmjs.org'
+      - run: npm ci
+      - run: npm run build
+      - run: npm publish --access public
+        env:
+          NODE_AUTH_TOKEN: ${{ secrets.NPM_TOKEN }}
diff --git a/Frontend/README.md b/Frontend/README.md
@@ -28,6 +28,11 @@ Once you have NodeJS installed,
 - `npm install`
 - `npm run build`
 
+The default user interface is provided in /ui-library directory. You can either use it or provide your own user interface. To build the default UI, run
+- `cd ui-library`
+- `npm install`
+- `npm run build`
+
 
 If you are developing your implementation based on the library, the process is similar:
 

diff --git a/Frontend/implementations/EpicGames/package-lock.json b/Frontend/implementations/EpicGames/package-lock.json
diff --git a/Frontend/implementations/EpicGames/package.json b/Frontend/implementations/EpicGames/package.json
@@ -9,8 +9,8 @@
     "watch": "npx webpack --watch",
     "serve": "webpack serve --config webpack.dev.js",
     "serve-prod": "webpack serve --config webpack.prod.js",
-    "build-all": "cd ../../library && npm run build && cd ../implementations/EpicGames && npm link ../../library && npm run build",
-    "build-all-dev": "cd ../../library && npm run build-dev && cd ../implementations/EpicGames && npm link ../../library && npm run build-dev"
+    "build-all": "cd ../../library && npm run build && cd ../ui-library && npm run build-all && cd ../implementations/EpicGames && npm link ../../library ../../ui-library && npm run build",
+    "build-dev-all": "cd ../../library && npm run build-dev && cd ../ui-library && npm run build-dev-all && cd ../implementations/EpicGames && npm link ../../library ../../ui-library && npm run build-dev"
   },
   "devDependencies": {
     "webpack-cli": "^5.0.1",

diff --git a/Frontend/implementations/EpicGames/src/player.ts b/Frontend/implementations/EpicGames/src/player.ts
@@ -1,17 +1,21 @@
 // Copyright Epic Games, Inc. All Rights Reserved.
 
-import * as libfrontend from '@epicgames-ps/lib-pixelstreamingfrontend-ue5.2';
+import { Config, PixelStreaming } from '@epicgames-ps/lib-pixelstreamingfrontend-ue5.2';
+import { Application, PixelStreamingApplicationStyle } from '@epicgames-ps/lib-pixelstreamingfrontend-ui-ue5.2';
+export const PixelStreamingApplicationStyles =
+    new PixelStreamingApplicationStyle();
 
 
 document.body.onload = function() {
 	// Example of how to set the logger level
-	//libfrontend.Logger.SetLoggerVerbosity(10);
+	// Logger.SetLoggerVerbosity(10);
 
 	// Create a config object
-	let config = new libfrontend.Config();
+	const config = new Config({ useUrlParams: true });
 
 	// Create a Native DOM delegate instance that implements the Delegate interface class
-	let application = new libfrontend.Application(config);
+	const pixelStreaming = new PixelStreaming(config);
+	const application = new Application({ pixelStreaming });
 	// document.getElementById("centrebox").appendChild(application.rootElement);
 	document.body.appendChild(application.rootElement);
 }
diff --git a/Frontend/implementations/EpicGames/src/stresstest.ts b/Frontend/implementations/EpicGames/src/stresstest.ts
@@ -1,6 +1,9 @@
 // Copyright Epic Games, Inc. All Rights Reserved.
 
-import * as libfrontend from '@epicgames-ps/lib-pixelstreamingfrontend-ue5.2';
+import { Config, Flags, PixelStreaming } from '@epicgames-ps/lib-pixelstreamingfrontend-ue5.2';
+import { Application, PixelStreamingApplicationStyle } from '@epicgames-ps/lib-pixelstreamingfrontend-ui-ue5.2';
+export const PixelStreamingApplicationStyles =
+    new PixelStreamingApplicationStyle();
 
 // This is the entrypoint to the stress test, all setup happens here
 export class StressTester {
@@ -34,27 +37,27 @@ export class StressTester {
 		this.setupPlayPause();
 
 		document.getElementById("creationIntervalInput").onchange = (event : Event) => {
-			let inputElem = document.getElementById("creationIntervalInput") as HTMLInputElement;
-			let parsedValue = Number.parseInt(inputElem.value);
+			const inputElem = document.getElementById("creationIntervalInput") as HTMLInputElement;
+			const parsedValue = Number.parseInt(inputElem.value);
 			if(!Number.isNaN(parsedValue)) {
 				this.streamCreationIntervalMs = parsedValue * 1000.0;
 				this.startStreamCreation();
 			}
 		}
 
 		document.getElementById("deletionIntervalInput").onchange = (event: Event) => {
-			let inputElem = document.getElementById("deletionIntervalInput") as HTMLInputElement;
-			let parsedValue = Number.parseInt(inputElem.value);
+			const inputElem = document.getElementById("deletionIntervalInput") as HTMLInputElement;
+			const parsedValue = Number.parseInt(inputElem.value);
 			if (!Number.isNaN(parsedValue)) {
 				this.streamDeletionIntervalMs = parsedValue * 1000.0;
 				this.startStreamDeletion();
 			}
 		}
 
-		let creationIntervalInput = document.getElementById("creationIntervalInput") as HTMLInputElement;
+		const creationIntervalInput = document.getElementById("creationIntervalInput") as HTMLInputElement;
 		creationIntervalInput.value = (this.streamCreationIntervalMs / 1000.0).toString();
 
-		let deletionIntervalInput = document.getElementById("deletionIntervalInput") as HTMLInputElement;
+		const deletionIntervalInput = document.getElementById("deletionIntervalInput") as HTMLInputElement;
 		deletionIntervalInput.value = (this.streamDeletionIntervalMs / 1000.0).toString();
 	}
 
@@ -85,15 +88,15 @@ export class StressTester {
 
 		this.creationIntervalHandle = setInterval(() => {
 			if(this.play) {
-				let curNPeers = this.pixelStreamingFrames.length;
+				const curNPeers = this.pixelStreamingFrames.length;
 				if(curNPeers >= this.maxPeers) return;
 
-				let maxPeersToCreate = this.maxPeers - curNPeers;
-				let nPeersToCreate = Math.ceil(Math.random() * maxPeersToCreate);
+				const maxPeersToCreate = this.maxPeers - curNPeers;
+				const nPeersToCreate = Math.ceil(Math.random() * maxPeersToCreate);
 
 				for(let i = 0; i < nPeersToCreate; i++) {
-					let frame = this.createPixelStreamingFrame();
-					let n = this.pixelStreamingFrames.length;
+					const frame = this.createPixelStreamingFrame();
+					const n = this.pixelStreamingFrames.length;
 					frame.id = `PixelStreamingFrame_${n + 1}`;
 					this.streamsContainer.append(frame);
 					this.pixelStreamingFrames.push(frame);
@@ -112,19 +115,19 @@ export class StressTester {
 		this.deletionIntervalHandle = setInterval(() => {
 			if(!this.play) return;
 
-			let curNPeers = this.pixelStreamingFrames.length;
+			const curNPeers = this.pixelStreamingFrames.length;
 			if(curNPeers === 0) return;
 
-			let nPeersToDelete = Math.ceil(Math.random() * curNPeers);
+			const nPeersToDelete = Math.ceil(Math.random() * curNPeers);
 			for(let i = 0; i < nPeersToDelete; i++) {
-				let frame = this.pixelStreamingFrames.shift();
+				const frame = this.pixelStreamingFrames.shift();
 				frame.parentNode.removeChild(frame);
 			}
 		}, this.streamDeletionIntervalMs);
 	}
 
 	private setupPlayPause() : void {
-		let playPauseBtn = document.getElementById("playPause");
+		const playPauseBtn = document.getElementById("playPause");
 		playPauseBtn.innerHTML = this.play ? "Pause" : "Play";
 
 		playPauseBtn.onclick = (event : Event) => {
@@ -134,24 +137,25 @@ export class StressTester {
 	}
 
 	private createPixelStreamingFrame() : HTMLElement {
-		let streamFrame = document.createElement("div");
+		const streamFrame = document.createElement("div");
 
-		let config = new libfrontend.Config();
-		config.setFlagEnabled(libfrontend.Flags.AutoConnect, true);
-		config.setFlagEnabled(libfrontend.Flags.AutoPlayVideo, true);
-		config.setFlagEnabled(libfrontend.Flags.StartVideoMuted, true);
+		const config = new Config();
+		config.setFlagEnabled(Flags.AutoConnect, true);
+		config.setFlagEnabled(Flags.AutoPlayVideo, true);
+		config.setFlagEnabled(Flags.StartVideoMuted, true);
 
 		// Create a Native DOM delegate instance that implements the Delegate interface class
-		let application = new libfrontend.Application(config);
+		const pixelStreaming = new PixelStreaming(config);
+		const application = new Application({ pixelStreaming });
 		streamFrame.appendChild(application.rootElement);
 		return streamFrame;
 	}
 
 	private updateTotalStreams() : void {
-		let nStreamsLabel = document.getElementById("nStreamsLabel");
+		const nStreamsLabel = document.getElementById("nStreamsLabel");
 		nStreamsLabel.innerHTML = this.totalStreams.toString();
 	}
 }
 
-let tester = new StressTester();
+const tester = new StressTester();
 tester.startStressTest();
diff --git a/Frontend/implementations/EpicGames/src/uiless.html b/Frontend/implementations/EpicGames/src/uiless.html
@@ -0,0 +1,48 @@
+<!-- Copyright Epic Games, Inc. All Rights Reserved. -->
+<!DOCTYPE HTML>
+<html style="width: 100%; height: 100%">
+<head>
+    <meta charset="utf-8">
+    <meta name="viewport" content="width=device-width, initial-scale=1">
+
+    <!-- Optional: apply a font -->
+    <link rel="preconnect" href="https://fonts.googleapis.com">
+	<link rel="preconnect" href="https://fonts.gstatic.com" crossorigin>
+	<link href="https://fonts.googleapis.com/css2?family=Michroma&family=Montserrat:wght@600&display=swap" rel="stylesheet">
+
+    <!-- Optional: set some favicons -->
+    <link rel="shortcut icon" href="./assets/images/favicon.ico" type="image/x-icon">
+	<link rel="icon" type="image/png" sizes="96x96" href="./assets/images/favicon-96x96.png">
+	<link rel="icon" type="image/png" sizes="32x32" href="./assets/images/favicon-32x32.png">
+	<link rel="icon" type="image/png" sizes="16x16" href="./assets/images/favicon-16x16.png">
+
+    <style>
+        #clickToPlayElement.visible {
+            display: flex;
+            align-items: center;
+            justify-content: center;
+            cursor: pointer;
+        }
+        #clickToPlayElement {
+            position: absolute;
+            top: 0;
+            left: 0;
+            width: 100%;
+            height: 100%;
+            display: none;
+        };
+    </style>
+
+    <!-- Optional: set a title for your page -->
+    <title>Pixel Streaming</title>
+</head>
+
+<!-- The Pixel Streaming player fills 100% of its parent element but body has a 0px height unless filled with content. As such, we explicitly force the body to be 100% of the viewport height -->
+<body style="width: 100vw; height: 100vh; min-height: -webkit-fill-available; font-family: 'Montserrat'; margin: 0px">
+    <div id="videoParentElement"></div>
+    <div id="clickToPlayElement">
+        <div>Click to play</div>
+    </div>
+</body>
+
+</html>
diff --git a/Frontend/implementations/EpicGames/src/uiless.ts b/Frontend/implementations/EpicGames/src/uiless.ts
@@ -0,0 +1,32 @@
+// Copyright Epic Games, Inc. All Rights Reserved.
+
+import { Config, PixelStreaming } from '@epicgames-ps/lib-pixelstreamingfrontend-ue5.2';
+
+document.body.onload = function() {
+	// Example of how to set the logger level
+	// Logger.SetLoggerVerbosity(10);
+
+	// Create a config object
+	const config = new Config({
+		initialSettings: {
+			AutoPlayVideo: true,
+			AutoConnect: true,
+			ss: "ws://localhost:80",
+			StartVideoMuted: true,
+		}
+	});
+
+	// Create a PixelStreaming instance and attach the video element to an existing parent div
+	const pixelStreaming = new PixelStreaming(config, { videoElementParent: document.getElementById("videoParentElement")});
+
+	// If browser denies autoplay, show "Click to play" and register a click-to-play handler
+	pixelStreaming.addEventListener("playStreamRejected", () => {
+		const clickToPlay = document.getElementById("clickToPlayElement");
+		clickToPlay.className = "visible";
+		clickToPlay.onclick = () => {
+			pixelStreaming.play();
+			clickToPlay.className = "";
+			clickToPlay.onclick = undefined;
+		}
+	})
+}