Windows WASAPI & ASIO Issues

[Play-AV]

First off, thanks a ton for the script. It absolutely works decently on Windows. However, there's some weird issue with the RTP timestamp resync. The problem I have is 'micro' stutter / dropouts that seem to coincide directly with the console message of the RTP Timestamp Resync. I know you're not on Windows so I doubt you'll be able to help here but perhaps you have some ideas?

Lets talk about WASAPI first. So it does work, I setup the AES67-Sender-Enhanced on my PC with an audio sink (tried a few of the VB Cable stuff including Banana but more on that later) and sure enough, my Ravenna linux machine picks up the stream, just like it picks up streams sent from other machines both Windows & Linux running Ravenna's driver. However, while it doesn't happen on EVERY RTP Resync, occasionally, I'll get this little pop (clearly audible while playing pink noise) that seems to coincide with a few of those messages. The offset is comfortably around 8-9ms although occasionally it drops down to 4ms in which case the little dropout is almost certain to happen, though it CAN happen when the value is firmly around 8-9ms.
Okay so, I changed the sync time to every 10s and every 100s, the dropouts are understandably way less frequent and lipsync is still working even after an hour long session.
Cool but shouldn't I be able to re-sync whenever? So I figure okay, lets try Process Hacker and make everything related to the specific node process 'real-time'. No difference.

I took apart the Dante Virtual Soundcard a bit to see if it has any different flags in terms of 'real time' on windows and no, it's ptp process has the realtime flag but I was able to give the same flag to the 'node' process. I figured, look at something that works and see if they're doing anything different right?

Next thing I tried was forcing node by default to try and set a higher priority.

console.log("setting priority for" + " the current process to -20"); try{ // Setting priority of current process os.setPriority(-20); }catch(err){ // Printing error message if any console.log(": error occurred"+err); }
It does correctly set the priority on windows but it doesn't make a difference.

Finally, the ASIO backend is weird, by changing the Audify binary build to something that worked (the 1.7.1 mentioned above) I'm able to get ASIO working correctly (I think 1.8 is broken for ASIO) but I get a massive difference in reported timestamp differences, going from 7-9ms to 400ms and most of the time is just a distorted mess, so there's really something weird with it. I tried adding the 'framesize' option to replace fpp (so replacing fpp with 0) to no effect. I can't really understand why the timestamp difference goes up so dramatically ?

RtAudioStreamFlags=0x8 Added that in too just in case to try and tell RTAudio to use the realtime flag?

I'm a little at a loss here.
I'd love some ideas. I guess it technically works, especially if you change the resync time to only do it like every 10 or 100s.
I'm 99% sure this is either an audify issue or a node + windows for 'real time' stuff issue but that's way outside of my knowledge.

Thanks again

[teletype1]

Hey Play-AV - thanks for testing this out and for reporting on this stuff.
I have to admit I'm not a windows guy (at all) so you're right, this is a bit foreign to me.
However, I will say that in my experience, printing text to the screen is one of the most computationally intensive things you can do - when I was writing the program, I was doing it over an SSH connection and when I needed to dump out a lot of text data, the computer I was using would lag in a really awful way. That is to say, the machine that was running the node script was fine, but the Mac running Terminal could barely keep up with text being dumped to the screen.

One thing to try first - use verbose mode to get your settings right, and then run the script again with verbosity turned off. Does it still give you the blips?

One thing that is coming to mind is - what is your PTP master on your network? If it's another computer, that could be the cause. In the original version, Phillip Hartung was running on an RPi and the clock was much worse than on my old Mac. If it's a switch or actual Dante device (like mine is) it's going to be a lot more accurate, and probably need to be synched less often. I think this is somewhat of a failing of AES67 that it requires such a precise clock. I mean, I get it for pro audio use, and mixed networks with devices from several manufacturers, but there should be stipulations for running without a PTP grandmaster.
I agree that RTAudio is limited - but it's what the original author used, so I just went with it. This software really should be written in C, but Node is just so convenient!

[Play-AV]

Thanks for the quick reply Matthew! Can't believe it slipped my mind to try it without console output. I'll report in if there's a difference.

So for clocking, this was my first major hurdle to get over. I didn't want to buy a time appliance to play with this stuff, and frankly my goals are to have something that's built on my own terms. Honestly, I'm not that smart of a woman but I try. I spent like half a month pouring through documentation for ptp4l and piecing together the details I'd need to support Ravenna. My whole schtick with this is, I don't have a Merging Hapi, I'd love one but $6K is a lot. However, Ravenna's implementation of AoIP is just fantastic. I figure, if I can make their legit drivers play nice with the stuff I build, that's a good foundation.

Phil has done amazing work on the AES67 front, seeing that he got it working was the kick in the butt I needed to try doing it from the ground up. I actually only came across his ptp4l gm config AFTER I had built my own and honestly, I don't think his version of the ptp4l command is 'ideal'.

Working backwards through documentation about Ravenna's stuff and snooping around, I was able to get all the correct flags and stuff to have Ravenna 'like' the GM.
Currently, my GM runs on a Pi CM4.
This is really important, a regular Pi is not a good source of time, it lacks the correct NIC to do hardware timestamping, furthermore if I get more 'specific' about the idea of clocking, the CM4 allows proper GNSS PPS 'tempering' allowing you to use that very consistent GPS 'pulse' as a reference point.

Frankly, I understand your frustration about the clocking, it should be more accessible. However, in taking apart the various pieces of existing AES67 software solutions, it's definitely not 'easy' to overcome. There's a number of benefits, even in the playback realm when you have this GM just sitting there. I've been looking into GStreamer as a video renderer that happens to render it's audio to AES67 actually.

I still think it's highly possible it's my GM but I don't really have the same issue on Ravenna's legit driver on PC. I might try to spin up a GM with my i210 machine and see if it's any better.
The TimeBeat guys are really helpful, I'm pretty set on moving to their software solution vs ptp4l. In fact, in the coming weeks, I'll probably get their software up and running to see if there's any difference. I put it on hold over the holidays after I had a great call with Ian but since you're skeptical of the clock (as am I) I think it's worth a second shot to see if it can do what I need to. When I tried quickly in late December, it was 'working' but Ravenna did not 'like' it (it would see the grandmaster but never lock on properly) so there's clearly just some config problem as I had a similar issue with ptp4l before I tweaked my settings.

Day to day, like personally, I have a mini Dante network for my desktop audio, production machines and the cinema renderer that handles my film playback.
Dante VSC + Via is a weirdly rock solid combo, like absolutely blew my mind. I can honestly say that analog D/A sounds far better when it's 'decoupled' from machines doing extremely heavy processing (like scaling video for playback). For music, I use HQPlayer + it's NAA.

What's your grandmaster / clocking situation?

Sorry to talk your ear off, it's just rad to see other people tinkering with AES67,

• Evelyn

[Play-AV]

Perhaps when I get TimeBeat up and running I'll write a guide laying out the 'ingredients' and some config details needed for a GM that doesn't just work with 'community' AES67 implementations, rather one that satisfies all the requirements of more locked down or picky commercial software / hardware.

In the mean time, here's an incredibly unorganized, overview of some hardware considerations re: AoIP & Clocking.

The first thing to wrap your head around is that your bog standard NIC probably doesn't support H/W time stamping (Apple is so far ahead of the curve on this AoIP transport stuff) and that's pretty important for the commercial implementations of AES67 AND generating a Grandmaster.

Enter the CM4, it's not ideal in certain respects but even at the vastly inflated prices (I spent id say 75% more than MSRP) it's cheaper than alternatives and contains a NIC that does support hardware timestamping, and the potential for PPS from GNSS.

For x86 systems, the cheapest reliable NIC is the i210. Not every i210 is going to be the most ideal on every OS, one of my fanless single board systems with an i210 on board uses a different 'identifier' than is common and thus the Ravenna Windows driver (which needs to 'take over' from the original NIC driver) refuses to work even with modification.
On Linux however it does indeed support HW timestamping and claims to have a PPS port (somewhere in it's poorly documented GPIO).

There are newer Intel NICs that support H/W time stamping, so if you're after a small, single board / mini solution, you might do well digging around into spec sheets to see what NIC some of these systems use since it's not always obvious or easily 'searchable'.

i210 cards don't always have the SDP pins which can be helpful for PPS or debugging, I got lucky with a random Amazon order and received a brand new HP i210 for like $30, it happened to have those SDP pins but more generic / cheap i210 options may lack them.

Finally, there is a wealth of information out there, scattered about the internet on how PTP and IEEE1558 works, how you can get it setup etc, the thing is alot of it isn't directly relevant to audio, and it can be very dense.

I still have a huge amount of research and experimentation to do on this front.

[Play-AV]

Resycing PTP and RTP timestamp. Offset was 8.75ms. [ 209341, 967791072 ] Resycing PTP and RTP timestamp. Offset was 8.563ms. [ 209342, 967995672 ] Resycing PTP and RTP timestamp. Offset was 8.896ms. [ 209343, 968172672 ] Resycing PTP and RTP timestamp. Offset was 8.688ms. [ 209344, 969796572 ] Resycing PTP and RTP timestamp. Offset was 8.292ms. [ 209345, 969670772 ] Resycing PTP and RTP timestamp. Offset was 8.833ms. [ 209346, 969689472 ] Resycing PTP and RTP timestamp. Offset was 8.625ms. [ 209347, 969841272 ] Resycing PTP and RTP timestamp. Offset was 8.583ms. [ 209348, 969682072 ] Resycing PTP and RTP timestamp. Offset was 8.708ms. [ 209349, 969932120 ] Resycing PTP and RTP timestamp. Offset was 8.333ms. [ 209350, 969922220 ] Resycing PTP and RTP timestamp. Offset was 9.271ms. [ 209351, 970280320 ] Resycing PTP and RTP timestamp. Offset was 8.625ms. [ 209352, 969892020 ] Resycing PTP and RTP timestamp. Offset was 8.208ms. [ 209353, 969946820 ] Resycing PTP and RTP timestamp. Offset was 9.625ms. [ 209354, 970306620 ]

I've setup the script to output the ptpv2.ptp_time() alongside the offset in verbose mode.

There's something interesting here, whenever it 'pops', it seem to correspond with the second # jumping around.
I wonder what that means?

[teletype1]

Hi Evelyn - please keep sharing this stuff!

Perhaps we should change the code to not just give an offset, but a positive or negative offset. I can't remember at this moment if the script prints out an absolute value but it might be helpful to see if the aes67 sender machine is consistently dragging, rushing, or if it's all over the place.

My grandmaster clock is a Soundcraft Si Impact with a Dante card in it, at least down at the venue where I developed the script. I've never tried it on a network with a software solution. Even if the Pi CM can do hardware timestamping, that doesn't necessarily mean the clock is good on it :) If you have access to a cheap actual Dante device it might be worthwhile to see if it will clock better - that would at least illuminate one thing. Those little 2-channel Audinate I/O dongles are $200, which is not really in the DIY spirit, but by design they can act as a master clock, at least I'm pretty sure they can :)

[Play-AV]

Hi Evelyn - please keep sharing this stuff!

Perhaps we should change the code to not just give an offset, but a positive or negative offset. I can't remember at this moment if the script prints out an absolute value but it might be helpful to see if the aes67 sender machine is consistently dragging, rushing, or if it's all over the place.

My grandmaster clock is a Soundcraft Si Impact with a Dante card in it, at least down at the venue where I developed the script. I've never tried it on a network with a software solution. Even if the Pi CM can do hardware timestamping, that doesn't necessarily mean the clock is good on it :) If you have access to a cheap actual Dante device it might be worthwhile to see if it will clock better - that would at least illuminate one thing. Those little 2-channel Audinate I/O dongles are $200, which is not really in the DIY spirit, but by design they can act as a master clock, at least I'm pretty sure they can :)

That's actually my next step, trying one of the cheap audinate I/Os, the problem for me is that I believe they only output PTPv1, which limits their integration with Ravenna IIRC.
There's "AudioCom" on Aliexpress with some wild AES67 stuff. There's also HASSEB. All of these devices can act as a GM unless I'm mistaken.

I've also got another i210 arriving, I'll try it in a more capable system and see if it generating a grandmaster instead, overcomes the limitations of the CM4. I have no problem running a proper computer as a GM if that's what it needs. I wouldn't be surprised if that fixes it to be honest.

The thing is, I've seen a number of people claiming they're doing just fine with the CM4 as a GM.
I think the good answer is TimeBeat with a GNSS PPS source, but I was hoping to avoid it since in AES67, within a single location you don't need 'globally' accurate time, you just need stable time.

It's definitely the most promising software approach (regardless of hardware) for this, they've been really helpful so far.
We're just stuck at a point where for some reason, unlike ptp4l, TimeBeat decides to appear as a boundary clock. That's not a problem for this script, nor for the Ravenna Linux drivers but, it refuses to work on the Windows Ravenna driver for me, for whatever reason, I think it's a hardcoded limit since the actual ptp process used by the Ravenna driver sees the Boundary Clock.

[Play-AV]

Strange thing I noticed, while using my version that prints ptpv2.ptp_time() on each re-sync. I occasionally get a .5 decimal?
"1673814626, 1497606859.5"

I need to re-read Phil's ptpv2 for node but I can't understand why there'd be a decimal at all?

On the plus side, I tried the version of the script that resyncs every 100s vs 1s, for about 4 hours last night, while loading my system heavily with a game and the tv show I was watching and it was pretty much rock solid the entire time. I mean this is 23/24fps video, the latency requirements aren't absurd. Dante VSC + Via to it's credit absolutely can be close enough to handle a 120hz game

For the above, the source system is a relatively powerful Windows 10 PC with an i210 NIC.

The target system is a fanless PC with a J4125 (so nothing spectacular but just enough) and a pair of i210s.

The target system runs Bondagit's modified Ravenna driver + his implementation of a REST interface to manage this. I use his Web UI + my own. The reason for this is that I wanted something pretty that also handles some other functions related to orchestrating this whole setup.

Which leads into how I actually get sound. I have a Topping Dx1 connected to the target system, There's a reason for this specific DAC + Head Amp. So it uses the XMOS XU208, now this has solid Linux support but more importantly, everyone and their mum is using these XMOS chips, and I use a more capable, XU216 on my serious multichannel setup. I see it as a good baseline.

So once this AES67 sender is running, we can just hop into the web UI on the target machine, see the discovered SDP and add it in the "sources" tab

Now you may need to utilize the ignore GM ID.

Don't worry, if you've set things up right, they are the same GM ID (You should confirm this first of course). The problem is the generated SDP ends up being lower case and Ravenna's driver very much only understands it's current GM ID in uppercase. Thus, they don't match.

We've got our source sink, we need to actually output it. I've been playing around and I think alsaloop is the right idea but there's a few other options out there.
sudo taskset -c 1 chrt 99 alsaloop -b -c 2 -f s32_le -C hw:RAVENNA -P hw:DX1 -s -v -S 0 -l 512

Audio should start playing, you may get some errors at first, if they persist past a few seconds, stop and look at that last # in the command. That's your buffer. Try changing it to 1024. Now this buffer WILL add latency so it's important to try and dial it in correctly.

[Play-AV]

Out of curiosity, would you mind grabbing the output of your working system with this modification?

// Interval for timestamp correction calculation // the original aes67.js by Phil Hartung did this every 100ms, but that seems too often, changed to 1 sec setInterval(function(){ let avg = Math.round(offsetSum / count); if(avg > fpp){ correctTimestamp = true; let offsetMS = Math.round(avg / fpp * 1000) / 1000; var loggingPtpTime = ptpv2.ptp_time(); logger('Resycing PTP and RTP timestamp. Offset was '+offsetMS+'ms.', loggingPtpTime); } offsetSum = 0; count = 0; }, 1000);

Thanks <3

[teletype1]

Hi Evelyn - I'm trying that mod now, to print out the PTP time. I'm pulling 32 channels from the console in the performance venue downstairs, and I let it run for about 10 minutes.

Selected CoreAudio as audio API.
Selected 192.168.0.37 as network interface
Channel input map from the sound card is [
   0,  1,  2,  3,  4,  5,  6,  7,  8,
   9, 10, 11, 12, 13, 14, 15, 16, 17,
  18, 19, 20, 21, 22, 23, 24, 25, 26,
  27, 28, 29, 30, 31
]
Selected device KLARK TEKNIK: DN32-LIVE with  32  input channels

We are trying to put  32  channels on the network
This will require  4  AES67 flow(s).

Selected the following MultiCast Addresses:  [ '239.69.1.1', '239.69.1.2', '239.69.1.3', '239.69.1.4' ]
Selected the following names for the AES67 flows:  [
  'SchwobTech-Pierre-Green-Room.local-Bank-1',
  'SchwobTech-Pierre-Green-Room.local-Bank-2',
  'SchwobTech-Pierre-Green-Room.local-Bank-3',
  'SchwobTech-Pierre-Green-Room.local-Bank-4'
]
Opening audio stream.
Trying to sync to PTP leader.
Initializing PTP client
Synced to 00-1d-c1-ff-fe-12-f2-18:0 successfully
Starting SAP annoucements and audio stream.
Resycing PTP and RTP timestamp. Offset was 30.146ms. [ 2856, 207294124.5 ]
Resycing PTP and RTP timestamp. Offset was 1.063ms. [ 2917, 206401669.5 ]
Resycing PTP and RTP timestamp. Offset was 1.083ms. [ 2977, 205017265.5 ]
Resycing PTP and RTP timestamp. Offset was 1.021ms. [ 2985, 204174207.5 ]
Resycing PTP and RTP timestamp. Offset was 1.104ms. [ 3097, 202985012.5 ]
Resycing PTP and RTP timestamp. Offset was 1.021ms. [ 3208, 200149769 ]
Resycing PTP and RTP timestamp. Offset was 1.042ms. [ 3209, 200163177 ]
Resycing PTP and RTP timestamp. Offset was 1.083ms. [ 3268, 201064705 ]
Resycing PTP and RTP timestamp. Offset was 1.021ms. [ 3276, 201109695 ]

I'm not too familiar with the guts of PTP but you might be right that the .5 shouldn't be there, I'll have to investigate that. I just took Phil's code and assumed it was OK.

One thing though, the iMac that is running Node had been on for 70 days, and initially the clock was constantly nearly 80ms off, it was resyncing very often. Maybe something was stale or leaking memory and it was swapping, I'm not sure - but I rebooted it and above is what I got.