-
Notifications
You must be signed in to change notification settings - Fork 6
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Feature Request - Proposed keyword optimization Improvement to pva.py and start.sh #9
Comments
Vosk runs ~ 60% max on a Pinephone with a 4core arm gpu with the small-de-model, so either you are running vosk on an even less powerful cpu, or you use the large model. In both cases, I suggest to change it on the hw level. You solution has one major design flaw: It party disables functionality ;) "What is your name" won't work anymore, same as all of the reaction stuff and this includes the ability to answere the phone, which needs to have pre-supplied reactions, because callers do not know how to address the bot they are calling ;) I suggest to either:
But in no way will I limit this pva to only keyword based interactions, as i will get killed by my family if i do so :D They love to talk to their PC's :) BTW: the main config at /etc/pva/conf.d/01-default.conf explicit sets "carola" as keyword, you don't need to hardcode it in pva.py if you handled it correctly in "pva" . |
Thanks for the reply! Here are the current specs that I am running PVA on: Hardware: When running vosk in separate tests, I get similar results of vosk getting bogged down with background noise. (AC or fan running in background etc.) When it gets bogged down for me, one of my CPU cores gets pinned at 85% and it can take from 10s of seconds to minutes for the recognition to "catch up". All that being said, I would hate to take away the functionality to be able to casually talk to your computer! ; ) However I think having this functionality as an option is extremely useful, especially for laptops or devices that are used in noisy environments. I believe the best option would be having the two loops. That way you would be able to toggle between using conversational interaction and keyword interaction at startup. Maybe adding a "hibernate" option into the /etc/pva/conf.d/01-default.conf for toggling would make sense for this kind of functionality? something like:
where the config default would be no. This could then be used by start.sh and passed to pva.py as a flag to allow for keyword functionality to be toggled at startup. I will see if I can provide to you a patch for pva.py start.sh and /etc/pva/conf.d/01-default.conf that allows for you to toggle this functionality using a flag set in the conf file. With regards to the hard-coding "carola" into the pva.py. Thanks again for your time and effort on this project! |
I have a Surface Pro 4 with i7-6th running with the same kind of problem. It also drains the battery, so I disabled the pva on that device too. see alphacep/vosk-api#1059 pls add your sightings there, and hopefully it gives them something to think about. Regarding the config option: That only works on (re)start of pva script... hey.. wait.. it technically could also work on runtime, now that we have a pva task and a STT task. My python is terrible, do you think you can rebuild pva.py to the two loop version, depending on the option in /etc/pva/conf.d/01-default.conf This will work fine:
I will add the needed other config options and functionality. |
add a watch inside the loop for changes in ~/.config/pva/pva.conf . That way I don't need to kill and restart the python script. |
pva code & config prepared, it's up to you:
|
Wow! this looks great. I am leaving for a weekend trip today, so I should be able to implement something on Monday! With regards to the performance issues, I will see if I can post my findings at the link that you mentioned. Finally, I have some other minor proposals for adding some other simple features. I have already implemented some of these features to my personal install of PVA, but I think they could be useful to the rest of the people using PVA. Thank you so much for your time and patience with me on this matter! |
Open Issues as "Request: ..." And for what purposes do you need such functionality? Maybe you wanne check with "DeepIn" Desktop Environment ? AFAIK, they have deeply integration for apps into the desktop itself, to support i.e. assistants to interact with everything. I never tested it, but heared interesting ideas, in special, if we wanne integrate PVA in handicapped peoples lifes. |
okay, I think I have something that provides basic proof of concept of switching between keyword based recognition and full recognition on the fly. Give this below code a try, and let me know if it works for you. #!/usr/bin/env python3
# This Script is modified for PVA usage
# it's originally shipped by the VOSK package from Alpha Cephi itself.
#
# Modified to have more effiencent keyword recognition.
# Date: 2022-07-13
#
#Modified to have live toggling of keyword recognition and full recognition loops.
#Date: 2022-07-19
#
# TODO:
# DONE - Add flag that accepts a keyword as an argument.
# DONE - have start.sh pass the keyword into pva.py as a flag.
# - Maybe have path to TTS program passed to pva.py as a flag instead of using straight espeak.
#
#
import argparse
import os
import queue
import sounddevice as sd
import vosk
import sys
### used for randomized feedback responses.
from random import randint
q = queue.Queue()
#keyword = "computer"
### Specify config file for pva.py to listen to for changes.
HOME_DIR = os.environ.get("HOME")
config_file_path = HOME_DIR+"/.config/pva/pva.conf"
def int_or_str(text):
"""Helper function for argument parsing."""
try:
return int(text)
except ValueError:
return text
def callback(indata, frames, time, status):
"""This is called (from a separate thread) for each audio block."""
if status:
print(status, file=sys.stderr)
q.put(bytes(indata))
parser = argparse.ArgumentParser(add_help=False)
parser.add_argument(
'-l', '--list-devices', action='store_true',
help='show list of audio devices and exit')
args, remaining = parser.parse_known_args()
if args.list_devices:
print(sd.query_devices())
parser.exit(0)
parser = argparse.ArgumentParser(
description=__doc__,
formatter_class=argparse.RawDescriptionHelpFormatter,
parents=[parser])
parser.add_argument(
'-m', '--model', type=str, metavar='MODEL_PATH',
help='Path to the model')
parser.add_argument(
'-d', '--device', type=int_or_str,
help='input device (numeric ID or substring)')
parser.add_argument(
'-r', '--samplerate', type=int, help='sampling rate')
## Added in flag to accpet keyword from start.sh file
parser.add_argument(
'-k', '--keyword', type=str, help='keyword')
args = parser.parse_args(remaining)
def check_config_hibernate(conf_file_path=config_file_path):
"""This function gets the status of the hibernate configuration option at the specified config file location. It will return True or False based on the option in the configured file."""
try:
file = open(conf_file_path,'rt')
file_contents = file.readlines()
file.close()
except:
print("[!] Error! File at config location of: "+conf_file_path+" \n Could not be found or opened!")
try:
conf = [ l for l in file_contents if l.startswith('conf:') ]
hibernate_status = [ l for l in conf if 'conf:"hibernate"' in l ][0].split(",")[1].strip('\n').strip('"').lower()
if (hibernate_status == 'yes' or hibernate_status == 'true'):
return True
if (hibernate_status == 'no' or hibernate_status == 'false'):
return False
except:
print('\n[!] could not find conf:"hibernate","no" parameter in '+conf_file_path+' defaulting to "hibernate","no"!\n')
return False
try:
if args.model is None:
args.model = "model"
if not os.path.exists(args.model):
print ("Please download a model for your language from https://alphacephei.com/vosk/models")
print ("and unpack as 'model' in the current folder.")
parser.exit(0)
if args.samplerate is None:
device_info = sd.query_devices(args.device, 'input')
# soundfile expects an int, sounddevice provides a float:
args.samplerate = int(device_info['default_samplerate'])
if args.keyword is None:
print("[!] No keyword specified!")
# args.keyword = "computer"
keyword = args.keyword
model = vosk.Model(args.model)
with sd.RawInputStream(samplerate=args.samplerate, blocksize = 8000, device=args.device, dtype='int16',
channels=1, callback=callback):
print('#' * 80)
print('Press Ctrl+C to stop the recording')
print('#' * 80)
rec = vosk.KaldiRecognizer(model, args.samplerate)
## List of randomized responses to be used as feedback.
## That way you know when full VTS is avalible to say a command.
response_lst = ["Yes sir?","Yes?","Your command?","what is it?"]
kywrd_rec = vosk.KaldiRecognizer(model, args.samplerate, '[ "'+keyword+'", "[unk]" ]')
def full_speech_recognition():
"""This function runs the speech recognition algorithim"""
print("[+] Initalized full Speech recognition loop function.")
sys.stdout.flush()
config_mod_time = os.stat(config_file_path).st_mtime
while True:
data = q.get()
if rec.AcceptWaveform(data):
# print(rec.Result() // instead of printing the final sentence, we give it to pva as JSON
## because we are identifying the keyword seperatly, we need to re-add it back to the JSON for pva to use.
str = rec.Result() #.replace(' : "',' : "'+keyword+' ');
os.system( "java PVA '"+ str.replace("'","") +"'");
if config_mod_time != os.stat(config_file_path).st_mtime:
print("[!] config has been modified! now checking if hibernate parameter has been changed.")
sys.stdout.flush()
if check_config_hibernate(config_file_path):
print(' [!!] conf:"hibernate" parameter has been set to yes! now switching to keyword recognition loop!!')
sys.stdout.flush()
break
else:
print(' [=] conf:"hibernate" paramter is still no, continuing full speech recognition model.')
sys.stdout.flush()
config_mod_time = os.stat(config_file_path).st_mtime
def keyword_optimized_recognition(keyword):
"""Listens for only specified keyword, once keyword is recognized, then full recognition is initiated."""
print("[+] Initialized keyword Speech recognition loop function.")
sys.stdout.flush()
## List of randomized responses to be used as feedback.
## That way you know when full VTS is avalible to say a command.
response_lst = ["Yes sir?","Yes?","Your command?","what is it?"]
config_mod_time = os.stat(config_file_path).st_mtime
while True:
data2 = q.get()
if kywrd_rec.AcceptWaveform(data2):
if kywrd_rec.Result()[14:-3] == keyword:
print("[+] Keyword of ["+keyword+"] recognized!")
sys.stdout.flush()
## send a random response to a TTS engine, so you know when you can give commands to pva.
os.system("espeak '"+response_lst[randint(0,(len(response_lst) -1))]+"'")
## Begin full VTS
while True:
data = q.get()
if rec.AcceptWaveform(data):
# print(rec.Result() // instead of printing the final sentence, we give it to pva as JSON
## because we are identifying the keyword seperatly, we need to re-add it back to the JSON for pva to use.
str = rec.Result().replace(' : "',' : "'+keyword+' ');
os.system( "java PVA '"+ str.replace("'","") +"'");
break
else:
print(" [!] Got voice data, but keyword was not recognized!")
sys.stdout.flush()
if config_mod_time != os.stat(config_file_path).st_mtime:
print("[!] config has been modified! now checking if hibernate parameter has been changed.")
sys.stdout.flush()
if check_config_hibernate():
print(' [=] conf:"hibernate" parameter is still yes, continuing keyword voice recognition.')
sys.stdout.flush()
config_mod_time = os.stat(config_file_path).st_mtime
else:
print(' [!!] conf:"hibernate" parameter has been set to no, toggling to full speech recognition model.')
sys.stdout.flush()
break
while True:
if check_config_hibernate(config_file_path):
print("[+] Hibernation option in config has been set to yes. Now starting keyword optimized based recognition.")
sys.stdout.flush()
keyword_optimized_recognition(keyword)
else:
print("[+] Hibernation option in config has been set to no. Now starting full recognition.")
sys.stdout.flush()
full_speech_recognition()
except KeyboardInterrupt:
print('\nDone')
parser.exit(0)
except Exception as e:
parser.exit(type(e).__name__ + ': ' + str(e)) Hopefully this example should work for the solution that you are looking to have. As always, thank you for your time and patience with me on this matter! |
Why did you add your own responses into the loop code. This means, everyone would need to listen to espeak and it's always english and not optional and it's outdated already, as we don't use
anymore. BTW, it's not helpful for the user, if your loop response says "Yes, Sir" and the final result is "I have no idea what you said". It's best to remove this from the pva.py loops completely. They are simply the supply of the spoken wordstream, nothing more. |
any progress? |
Hello,
I really love this project and the work that you guys are doing here!
I would like to propose an improvement optimization to the pva.py file on this project.
I had downloaded and setup PVA on my machine, and noticed that in noisy environments; the vosk Speech recognition will start to bog down my machine. (it seems to pin one of my CPU cores are nearly 100%)
When this happens the recognition can take 30 sec to up to a minute plus.
This seems to be due to vosk trying to make sense of all audio at all times.
So I propose that a change be made to the pva.py, and the start.sh files.
My apologies if I am presenting this in a format that is incorrect, I am relatively new to using git.
First a new keyword flag should be added to the pva.py file.
Then in the start.sh file, grab the keyword from the config file and pass it to pva.py:
This keyword flag would be used for a 2nd vosk model instance that can be initialized to only recognize the specified keyword.
This method drastically reduces CPU overhead as it only listens for the specified keyword.
This should then run in a loop surrounding the primary audio grab loop.
That way vosk will only do full Speech to Text recognition after the keyword has been identified.
Thus saving and reserving processing power for when actual commands want to be relayed via vosk.
Here is a working proof of concept that I am currently using on my machine:
NOTE: I had implemented a quick hack to allow for the pva.py file to be able to provide auditory feedback via espeak, but this can be changed to use a different method of feedback or tts engine if needed.
pva.py:
start.sh:
If you have any questions for me, or need any additional information on this matter, please feel free to contact me!
Thank you for your time and work on this awesome project!
The text was updated successfully, but these errors were encountered: