Skip to content
This repository has been archived by the owner on Mar 1, 2024. It is now read-only.

Commit

Permalink
Bring changes from master to gh-packages branch (#174)
Browse files Browse the repository at this point in the history 
* Downgrade ts-jest and jest to support older Node version (#168)

* Downgrade ts-jest and jest to support older Node version

* bumbed ts-jest to 27.1.5 (latest 27.x.y)

* Bump lib-pixelstreamingfrontend to 0.3.0

Signed-off-by: Luke Bermingham <1215582+lukehb@users.noreply.github.com>

* Bump lib-pixelstreamingfrontend-ui to 0.2.0 update dep versions

Signed-off-by: Luke Bermingham <1215582+lukehb@users.noreply.github.com>

* Bump versions in package-lock.json for ui-library

* Bump ui-library to 0.2.0 and force NPM publish

* Update UE version printed when Cirrus starts

Signed-off-by: Luke Bermingham <1215582+lukehb@users.noreply.github.com>

* Bump Github release of 5.2 infra to 0.4.0

Signed-off-by: Luke Bermingham <1215582+lukehb@users.noreply.github.com>

* Fix: sfu player would try subscribing when sfu disconnected

* Updating the Frontend Docs (#152)

---------

Signed-off-by: Luke Bermingham <1215582+lukehb@users.noreply.github.com>
Co-authored-by: hmuurine <hmuurine@users.noreply.github.com>
Co-authored-by: William Belcher <william.belcher@xa.epicgames.com>
Co-authored-by: Michael Stopa <michael.stopa@xa.epicgames.com>
  • Loading branch information
@lukehb @hmuurine
@Belchy06 @StomyPX
4 people committed Mar 23, 2023
1 parent 344e338 commit 146e4c5
Show file tree
Hide file tree
Showing 25 changed files with 1,604 additions and 1,662 deletions.
34 changes: 17 additions & 17 deletions Frontend/Docs/Communicating from UE5 to the Player Page.md
View file
Original file line number Diff line number Diff line change
@@ -1,31 +1,31 @@
**TO DO**: Update this information to match the new front end.

## Communicating from UE5 to the Player Page

You can make your Unreal Engine application emit custom events to all connected player HTML pages, which you can respond to in the player's JavaScript environment. This lets you change your web page UI in response to gameplay events.

To set this up:

1. In your Unreal Engine application, any time you want to emit an event to the player page, use the **Pixel Streaming > Send Pixel Streaming Response** node. Specify a custom string argument to the node to indicate to the player page what event has happened.
1. In your Unreal Engine application, any time you want to emit an event to the player page, you need an actor with a **Pixel Streaming Input** component. This component has access to the **Send Pixel Streaming Response** node. Specify a custom string argument to the node to send a message to the web frontend.


<p align="center">
<img src="Resources\Images\pixelstreaming-send-game-event.JPG" alt="Send game event">
<img src="Resources\Images\pixelstreaming-send-game-event.png" alt="Send game event">
</p>

2. In the JavaScript of your player page, you'll need to write a custom event handler function that will be invoked each time the page receives a response event from the Unreal Engine application. It will be passed the original string argument that was sent by the **Send Pixel Streaming Response** node. For example:
2. In your TypeScript frontend implementation these messages are consumed by an event listener. This event will be invoked every time the frontend receives a custom message from the Unreal Engine application. The original string argument given to the **Send Pixel Streaming Response** node will be passed to the function as the `response` argument. For example:

function myHandleResponseFunction(data) {
console.warn("Response received!");
switch (data) {
case "MyCustomEvent":
... // handle one type of event
case "AnotherEvent":
... // handle another event
}
}
```typescript
public myHandleResponseFunction(response: string) => void {
Logger.Info(Logger.GetStackTrace(), "Response received!");
switch (response) {
case "MyCustomEvent":
... // handle one type of event
case "AnotherEvent":
... // handle another event
}
}
```

3. Register your listener function by calling the `addResponseEventListener` function provided by `app.js`. You pass this function a unique name for your event listener, and your function. For example:
3. Register your listener function by using the `addResponseEventListener` function provided by the `PixelStreaming` object, found in PixelStreaming/PixelStreaming.ts. You pass this function a unique name for your event listener, and your function. For example:

addResponseEventListener("handle_responses", myHandleResponseFunction);

Expand All @@ -34,11 +34,11 @@ To set this up:
removeResponseEventListener("handle_responses");

**_Tip:_**
If you want to pass more complex data, you can format the string you pass to the **Send Pixel Streaming Response** node as JSON. For example:
If you want to pass more complex data, you can format the string you pass to the **Pixel Streaming Input -> Send Pixel Streaming Response** node as JSON. For example:

<p align="center">
<img src="Resources\Images\pixelstreaming-send-game-event-json.png" alt="Send Pixel Streaming response using JSON">
</p>

Then, in your JavaScript event handler function, use `JSON.parse(data)` to decode the string back into a JavaScript object.
Then, in your response event handler function, use `JSON.parse(data)` to decode the string back into a TypeScript object.

72 changes: 44 additions & 28 deletions Frontend/Docs/Communicating from the Player Page to UE5.md
View file
Original file line number Diff line number Diff line change
@@ -1,69 +1,85 @@
**TO DO**: Update this information to match the new front end.

## Communicating from the Player Page to UE5

The `app.js` file provides two JavaScript functions that you can call in your HTML player page to allow the user to send events and commands from the browser to the Unreal Engine application:
The frontend provides three functions that you can call in your HTML player page to allow the user to pass events and commands from the browser to your Unreal Engine application:

* `emitCommand` Allows sending arbitrary commands to the game in the form of JSON objects.
* You can use `emitConsoleCommand` to send console commands back to Unreal Engine. For example, `stat fps` to show the frame rate. See [Using the emitCommand Function below](#using-the-emitcommand-function).
* `emitUIInteraction` sends any arbitrary string or object to the game. Use this function to send your own custom commands from your player UI, which you can respond to in your gameplay logic to produce any effect you need in your application. See [Using the emitUIInteraction Function below](#using-the-emituiinteraction-function).

* You can use `emitCommand` to send console commands back to Unreal Engine. For example, `stat fps` to show the frame rate. See [Using the emitCommand Function below](#using-the-emitcommand-function).
* `emitUIInteraction` sends any arbitrary string or JavaScript object to the game. Use this function to send your own custom commands from your player UI, which you can respond to in your gameplay logic to produce any effect you need in your application. See [Using the emitUIInteraction Function below](#using-the-emituiinteraction-function).

### Using the emitCommand Function

When you call the `emitCommand` function, you must pass it a JavaScript object. This object must contain a key that matches one of the following strings:
When you call the `emitCommand` function, you pass it an object which then gets converted to JSON and sent onward to the game. This can then be used for implementing any sort of custom functionality you may require.

#### Reserved Keywords

Though you are given free reign to organize commands how you see fit, if the object contains a key with one of the following reserved keywords it will be interpreted by the PixelStreaming module instead:

* `ConsoleCommand` \- Use this key to execute a console command on the remote Unreal Engine application. The value of this key should be a string that contains the command you want to run, along with any parameters it needs. For example:

let descriptor = {
ConsoleCommand: 'stat fps'
}
emitCommand(descriptor);
```typescript
let descriptor = {
ConsoleCommand: 'stat fps'
}
emitCommand(descriptor);
```

**_NOTE:_**
Due to the power of the Unreal Engine console commands, the `emitCommand` function can present a security risk. In order for this function to work, you also need to provide the `-AllowPixelStreamingCommands` parameter on the command line when you launch your Unreal Engine application or start it from the Unreal Editor using the Standalone Game option.
Due to the power of the Unreal Engine console commands, the `emitConsoleCommand` function can present a security risk. In order for this function to work, you also need to provide the `-AllowPixelStreamingCommands` parameter on the command line when you launch your Unreal Engine application or start it from the Unreal Editor using the Standalone Game option.


### Using the emitConsoleCommand Function

You may also use `emitConsoleCommand` to pass Unreal Engine console commands as a string. Like passing commands via `emitCommand`, you must provide `-AllowPixelStreamingCommands` on the command line when you launch your Unreal Engine application in order for these commands to be fielded.


### Using the emitUIInteraction Function

When you call the `emitUIInteraction` function, you can pass it a single string or JavaScript object. For example:
When you call the `emitUIInteraction` function, you can pass it a single string or object. For example:

emitUIInteraction("MyCustomCommand");
```typescript
emitUIInteraction("MyCustomCommand");
```

or

let descriptor = {
LoadLevel: "/Game/Maps/Level_2"
PlayerCharacter: {
Name: "Shinbi"
Skin: "Dynasty"
}
}
emitUIInteraction(descriptor);
```typescript
const descriptor = {
LoadLevel: "/Game/Maps/Level_2",
PlayerCharacter: {
Name: "Shinobi",
Skin: "Dynasty",
},
};
this.stream.emitUIInteraction(descriptor);
```

If you pass a JavaScript object, the `emitUIInteraction` function converts it to a JSON string internally. It then passes the resulting string back to the Pixel Streaming Plugin in your Unreal Engine application, which raises an event on the input controller. In your application's gameplay logic, you bind your own custom event to handle these inputs, using the **Bind Event to OnPixelStreamingInputEvent** node. For example:
If you pass an object, the `emitUIInteraction` function converts it to a JSON string internally. It then passes the resulting string back to the Pixel Streaming Plugin in your Unreal Engine application, which raises an event on the input controller. In your application's gameplay logic, you bind your own custom event to handle these inputs, using the **Pixel Streaming Input > Bind Event to On Input Event** node. For example:

<p align="center">
<img src="Resources\Images\pixelstreaming-uiinteractionrespond.JPG" alt="Bind Event to OnPixelStreamingInputEvent">
<img src="Resources/Images/pixelstreaming-ui-interaction-event.png" alt="Bind Event to On Input Event">
</p>

You need to bind this event once, typically at the start of your game. Each time any player HTML page connected to an instance of your Unreal Engine application calls the `emitUIInteraction`function, your custom event is automatically invoked, regardless of the input passed to `emitUIInteraction`.
You need to bind this event once, typically at the start of your game. Each time any player HTML page connected to an instance of your Unreal Engine application calls the `emitUIInteraction` function, your custom event is automatically invoked, regardless of the input passed to `emitUIInteraction`.

The custom event you assign (for example, the **UI Interaction** node in the image above) has an output named **Descriptor**, which you can use to retrieve the string that was sent to your Unreal Engine application by the `emitUIInteraction` function. You can use that value to determine how your gameplay code needs to respond each time `emitUIInteraction` is called.

For example, the following Blueprint tests to see whether the input given to `emitUIInteraction` contains the string "MyCustomCommand", and calls a custom function to handle the event:


<p align="center">
<img src="Resources\Images\pixelstreaming-respond-searchsubstring.JPG" alt="Search for substring">
<img src="Resources/Images/pixelstreaming-ui-interaction-search-substring.png" alt="Search for substring">
</p>

If you originally passed a JavaScript object to `emitUIInteraction`, you can retrieve the value of any key from that JSON object using the **Pixel Streaming > Get Json String Value** node. For example, the following Blueprint tests for a key named LoadLevel. If that key is present, it calls a custom function to handle the event:


<p align="center">
<img src="Resources\Images\pixelstreaming-respond-json.JPG" alt="Get a JSON field value">
<img src="Resources/Images/pixelstreaming-ui-interaction-extract-json.png" alt="Get a JSON field value">
</p>


**_Tip:_**
If you need to retrieve a nested key, use the dot notation common in JavaScript for your key.
For example, `PlayerCharacter.Name` or `PlayerCharacter.Skin`.
If you need to retrieve a nested key, use the dot notation common in TypeScript for your key.
For example, `PlayerCharacter.Name` or `PlayerCharacter.Skin`.

60 changes: 29 additions & 31 deletions Frontend/Docs/Customizing Player Input Options.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -2,46 +2,44 @@

The frontend library exposes a number of configuration options through the [Config](/Frontend/library/src/Config/Config.ts) class. The values of these options can be modified to tweak certain inbuilt behaviour of the frontend.

The following options are available in the frontend library:
The following options are available in the frontend library to customize input:

**TO DO**: Update this information to match the new front end.
| Property | Default | Description |
| --- | --- | --- |
| HoveringMouseMode | false | Determines whether or not the video element captures and locks the mouse when the player interacts with the widget. When enabled, the mouse cursor hovers over the player widget without interacting with it. In order to send the mouse movements to the input controller of the Unreal Engine application, the user needs to click and hold the left button of the mouse. Otherwise, clicking on the player widget causes it to capture and lock the mouse cursor. Any further movements of the mouse are passed immediately to the input controller in the Unreal Engine application. This typically allows the user to move and rotate the camera by simply dragging the mouse. To release the cursor from the control of the player widget, the user can press the **Esc** key. |
| SuppressBrowserKeys | true | When this setting is enabled, the player widget will intercept function keys (**F1** to **F12**) and the **Tab** key, and pass those keypress events through to the Unreal Engine application rather than allowing the browser to process them normally.| This means, for example, that while this setting is active, pressing **F5** will not refresh the player page in the browser. Instead, that event is passed through to the Unreal Engine application, and has its usual function of switching the view to visualize shader complexity.
| FakeMouseWithTouches | false | When this option is enabled and the user is viewing the stream on a device with a touch screen such as a smartphone or tablet, this setting causes single-finger touch events to be interpreted by the Unreal Engine application as mouse clicks and drag events. Enabling this setting can provide users on mobile devices with the ability to partially control your Unreal Engine application, even when the application's input controller does not specifically handle touch input events. |

| Property | Default | Description |
| --- | --- | --- |
| controlScheme | `ControlSchemeType.LockedMouse` | Determines whether or not the player widget captures and locks the mouse when the player interacts with the widget.|
| suppressBrowserKeys | true |When this setting is enabled, the player widget will intercept function keys (**F1** to **F12**) and the **Tab** key, and pass those keypress events through to the Unreal Engine application rather than allowing the browser to process them normally.| This means, for example, that while this setting is active, pressing **F5** will not refresh the player page in the browser. Instead, that event is passed through to the Unreal Engine application, and has its usual function of switching the view to visualize shader complexity.
| fakeMouseWithTouches | false | When this option is enabled, and the user is viewing the stream on a device with a touch screen such as a smartphone or tablet, this setting causes single-finger touch events to be interpreted by the Unreal Engine application as mouse clicks and drag events. Enabling this setting can provide users on mobile devices with the ability to partially control your Unreal Engine application, even when the application's input controller does not specifically handle touch input events. |
When creating a frontend implementation, these options are visible via the [`Config`](/Frontend/library/src/Config/Config.ts) object required in order to create a [`PixelStreaming`](/Frontend/library/src/PixelStreaming/PixelStreaming.ts) stream for your frontend application. Simply set the values you want before initializing the stream object.

```typescript
const config = new Config({ useUrlParams: true });
config.setFlagEnabled(Flags.HoveringMouseMode, true);
config.setFlagEnabled(Flags.FakeMouseWithTouches, true);

**_NOTE:_** The controlScheme accepts the following values:
* `ControlSchemeType.LockedMouse` - When this control scheme is active, clicking on the player widget causes it to capture and lock the mouse cursor. Any further movements of the mouse are passed immediately to the input controller in the Unreal Engine application. This typically allows the user to move and rotate the camera by simply dragging the mouse. To release the cursor from the control of the player widget, the user can press the **Esc** key.
* `ControlSchemeType.HoveringMouse` - When this control scheme is active, the mouse cursor hovers over the player widget without interacting with it. In order to send the mouse movements to the input controller of the Unreal Engine application, the user needs to click and hold the left button of the mouse.


You can set these values in your player page by including a code block like the following. Make sure that you run this code any time after you load the `app.js` file into the page, but before you call its `load` function.

<script>
inputOptions.controlScheme = ControlSchemeType.HoveringMouse;
inputOptions.fakeMouseWithTouches = true;
</script>
const stream = new PixelStreaming(config);
```

### Disabling User Input

To disable user input entirely for one or more types of input device, you can override the following functions in the JavaScript environment for your player page with empty implementations:
User input can be disabled entirely for one or more types of input device. This is controlled by the following input flags, all of which are enabled by default.

* **registerHoveringMouseEvents** - Disables all input mouse events when the inputOptions.controlScheme is set to ControlSchemeType.HoveringMouse.
* **registerLockedMouseEvents** - Disables all input mouse events when the inputOptions.controlScheme is set to ControlSchemeType.LockedMouse.
* **registerTouchEvents** - Disables touch events on mobile devices and tablets.
* **registerKeyboardEvents** - Disables all keyboard events.
* **MouseInput** - Allows mouse input events.
* **KeyboardInput** - Allows keyboard input events.
* **TouchInput** - Allows touch events on mobile devices and tablets.
* **GamepadInput** - Allows input events from gamepad controllers.
* **XRControllerInput** - Allows input events from XR controllers for use with AR and VR setups.

For example, you could include this block of JavaScript in your player HTML page to disable all inputs. As above, run this code any time after you load the `app.js` file into the page, but before you call its `load` function.
For example, you could set all of these flags to false in the `Config` object in order to block user input altogether.

<script>
registerHoveringMouseEvents = function() {}
registerLockedMouseEvents = function() {}
registerTouchEvents = function() {}
registerKeyboardEvents = function() {}
</script>
```typescript
const config = new Config({ useUrlParams: true });
config.setFlagEnabled(Flags.MouseInput, false);
config.setFlagEnabled(Flags.KeyboardInput, false);
config.setFlagEnabled(Flags.TouchInput, false);
config.setFlagEnabled(Flags.GamepadInput, false);
config.setFlagEnabled(Flags.XRControllerInput, false);

To keep one or more types of inputs active, comment out or remove the line that corresponds to the type of input you want to keep.
const stream = new PixelStreaming(config);
```

Loading

0 comments on commit 146e4c5

Please sign in to comment.