Early TAG review request for Playout Statistics API for WebAudio

[Hernqvist]

こんにちは TAG-さん!

I'm requesting a TAG review of Playout Statistics API for WebAudio.

There is currently no way to detect whether WebAudio playout has glitches (gaps in the played audio, which typically happens due to underperformance in the audio pipeline). There is an existing way to measure the instantaneous playout latency using AudioContext.outputLatency, but no simple way to measure average/minimum/maximum latency over time. With this API, we want to propose a way to be able to measure the delay of that audio and the glitchiness of the audio.

• Explainer¹ (minimally containing user needs and example code): https://github.com/WICG/web_audio_playout
• User research: none
• Security and Privacy self-review²: https://docs.google.com/document/d/1wGv_mr7Lgg2w-6PuKDrcScoa8IvYAOW3PMTFW85O3Gw/edit
• GitHub repo (if you prefer feedback filed there):
• Primary contacts (and their relationship to the specification):
  • Fredrik Hernqvist (Hernqvist), Google
  • Palak Agarwal (palak8669), Google
  • Guido Urdaneta (guidou), Google
  • Olga Sharonova (o1ka), Google
• Organization/project driving the design: Google
• External status/issue trackers for this feature (publicly visible, e.g. Chrome Status): https://chromestatus.com/feature/5172818344148992

Further details:

• I have reviewed the TAG's Web Platform Design Principles
• The group where the incubation/design work on this is being done (or is intended to be done in the future): WICG
• The group where standardization of this work is intended to be done ("unknown" if not known): W3C Audio Working Group
• Existing major pieces of multi-stakeholder review or discussion of this design:
• Major unresolved issues with or opposition to this design:
• This work is being funded by: Google

We'd prefer the TAG provide feedback as (please delete all but the desired option):

💬 leave review feedback as a comment in this issue and @-notify Hernqvist, palak8669

[martinthomson]

Hi @Hernqvist, we're looking at #843 (Audio Render Capacity) and are trying to understand how this approach fits with that. In particular, I'm interested in how this approach deals with the compute side channel leaks. On face value, the information that this API proposes is higher fidelity, if post-hoc, which carries some amount of risk.

[Hernqvist]

Hi @martinthomson! We believe that this proposed API gives less detailed information about CPU load than #843 (Audio Render Capacity). Since audio should always run on realtime threads, general high CPU load should not impact audio. When CPU load is high enough to impact realtime threads to the point of causing glitches, the Render Capacity API should already show underruns and very high load.

As for the risk of side channeling, we have investigated this risk more closely since filing the issue. We have found that the API we are proposing is not necessary to set up an audio glitch-based side channel. One way to set up this side channel using only currently implemented APIs is the following:

• Website 1 creates multiple computationally expensive AudioContexts. I tested this with 3 AudioContexts that each contained one OscillatorNode and 4000 BiquadFilterNodes.
• Website 2 creates a single AudioContext with a ScriptProcessorNode that measures the time between consecutive calls to process.
• Website 2 will observe longer maximum intervals between AudioProcessingEvents while Website 1 is active. Website 1 can now send messages to Website 2 by creating and stopping AudioContexts.

I have tested this scheme (on Mac) on Chrome, Safari and Firefox and it works on all of them (though you might have to change the number of AudioContexts or Nodes a bit depending on browser). So this side channel already exists, and we do not believe it is made worse by the proposed API.

[matatk]

We're trying to understand the difference between this and #843 - this seems to be geared towards statistics, whereas RenderCapacity seems more about adaptivity, but they both seem to be giving similar information - have you explored whether they could be combined?

We'd like to see some further documentation around the abuse cases relating to the side channel that you describe.

Do you have any information on the position of other implementers on this proposal?

[matatk]

Thank you for all the info in your last comment, @Hernqvist, and sorry for my slow reply. We are discussing the spec as a whole still, but for now I have a pointer regarding the side channel that may be relevant.

The Compute Pressure API had a similar potential side channel, and some mitigations were advised, following a PING review; here are the details: w3c/compute-pressure#197

[martinthomson]

@Hernqvist, that piece you have here about side channels probably belongs in the explainer. Having it here is helpful, but this issue will be closed and therefore hard to find. (The same is possibly true for some of the other notes here.)

[matatk]

Another request regarding the explainer: we were discussing your work again this week, and were wondering if you could add a section to the explainer to contrast this spec (e.g. in terms of the scope, when would one use it) with the RenderCapacity API? Thanks!

[Hernqvist]

Hi @martinthomson and @matatk ! Sorry for the slow reply. We have updated the explainer with the main points about the side channel and a comparison to the RenderCapacity API, as requested.

@matatk thank you for the pointer to the Compute Pressure API discussion. We looked at the Compute Pressure mitigations, and we think that the Rate Limitation mitigation could be applied here, which would mitigate the risk that the (preexisting) glitch based side channel is made worse.

[matatk]

Thanks for the extra info you added to the explainer. We don't see any architectural problems with this. This should go to the web audio working group for further work. Please also talk to the PING group regarding the risk mitigation.