feat: new voices and wakewords #88

mikejgray · 2024-05-17T02:11:20Z

Description

Allows users to use one of three presets for voice/wakeword combinations:

Neon ("Hey Neon", custom Neon Coqui voice model)
Classic Mycroft ("Hey Mycroft", Piper Alan Pope voice model)
The Butler ("Hey Jarvis", Piper Alan Pope voice model)

Issues

#87

Other Notes

N/A

locale/en-us/dialog/ask_enable.dialog

locale/en-us/dialog/neon_confirmation.dialog

locale/en-us/intent/classic_mycroft.intent

__init__.py

NeonDaniel · 2024-06-06T16:18:07Z

__init__.py

+
+    def initialize(self):
+        self.bus.on("mycroft.ready", self._speak_restart_dialog)
+        if self.dialog_to_speak and self.pending_audio_restart:


If this happens before the audio module is ready, I believe the dialog may be lost.. The on("mycroft.ready") should handle it in any case so this appears to be redundant anyways

See above, audio should always be ready before skills load in a Mark 2, so this is the primary and mycroft.ready is the fallback. mycroft.ready hasn't always been consistent for me so that's why I made this primary

__init__.py

NeonDaniel · 2024-06-29T00:28:36Z

Regarding the test failure, can you log wake_word_config when this assertion fails? I suspect something with the test listeners is causing the config to not update properly in this test case, though I don't see an obvious cause in your changes

NeonDaniel · 2024-07-02T22:51:25Z

I got a "sorry, something has gone wrong. I will continue using-" followed by services restarting.

audio.log

2024-07-02 15:42:10.046 - audio - neon_audio.tts.neon:execute:373 - DEBUG - responses={'en-us': {'sentence': "Okay, I'll be back shortly.", 'translated': False, 'phonemes': None, 'genders': ['female'], 'female': '/home/neon/.cache/neon/tts/CoquiRemoteTTS/en-us/female/8f18e5d69575daf28d21830251e1a820.wav'}}
2024-07-02 15:42:10.249 - audio - ovos_bus_client.session:get:605 - DEBUG - No message, use default session
2024-07-02 15:42:12.954 - audio - neon_audio.tts.neon:_play:185 - INFO - Played 1719960128.2668295
2024-07-02 15:42:12.957 - audio - neon_audio.tts.neon:_sort_timing_metrics:149 - DEBUG - Parsed timing context: {'timestamps': {'handle_utterance': 1719960107.7861273, 'speech_start': 1719960128.266869, 'audio_begin': 1719960130.2339814, 'audio_end': 1719960132.8814085}, 'durations': {'transform_audio': 3.1948089599609375e-05, 'get_stt': 5.132228374481201, 'save_transcript': 1.049041748046875e-05, 'text_parsers': 0.01896810531616211, 'transform_utterance': 0.01896810531616211, 'get_tts': 1.5334551334381104}}
2024-07-02 15:42:12.986 - audio - neon_audio.service:handle_speak:148 - DEBUG - Playback completed for: 1719960128.2668295
2024-07-02 15:42:38.631 - audio - ovos_audio.service:execute_tts:381 - INFO - Speak: Sorry, something went wrong. I will continue using the old wake word.
2024-07-02 15:42:38.726 - audio - neon_audio.tts.neon:get_requested_tts_languages:79 - DEBUG - Got profiles: [{'user': {'first_name': '', 'middle_name': '', 'last_name': '', 'preferred_name': '', 'full_name': '', 'dob': 'YYYY/MM/DD', 'age': '', 'email': 'daniel@neon.ai', 'username': 'local', 'password': '', 'picture': '', 'about': '', 'phone': '', 'phone_verified': False, 'email_verified': False}, 'brands': {'ignored_brands': {}, 'favorite_brands': {}, 'specially_requested': {}}, 'speech': {'stt_language': 'en-us', 'alt_languages': ['en-us', 'uk-ua'], 'tts_language': 'en-us', 'tts_gender': 'female', 'neon_voice': '', 'secondary_tts_language': '', 'secondary_tts_gender': 'male', 'secondary_neon_voice': '', 'speed_multiplier': 1.0}, 'units': {'time': 12, 'date': 'MDY', 'measure': 'imperial'}, 'location': {'lat': 47.6765382, 'lng': -122.2070775, 'city': 'Kirkland', 'state': 'Washington', 'country': 'United States', 'tz': 'America/Los_Angeles', 'utc': -8.0, 'country_code': 'us'}, 'response_mode': {'speed_mode': 'quick', 'hesitation': False, 'limit_dialog': False}, 'privacy': {'save_audio': True, 'save_text': True}}]
2024-07-02 15:42:38.729 - audio - neon_audio.tts.neon:get_requested_tts_languages:87 - DEBUG - local requesting female en-us
2024-07-02 15:42:38.731 - audio - neon_audio.tts.neon:get_requested_tts_languages:132 - DEBUG - Got 1 TTS Voice Requests
2024-07-02 15:42:38.734 - audio - neon_audio.tts.neon:get_multiple_tts:300 - DEBUG - tts_requested=[{'speaker': 'Neon', 'language': 'en-us', 'gender': 'female', 'voice': None}]
2024-07-02 15:42:38.739 - audio - neon_audio.tts.neon:get_multiple_tts:304 - DEBUG - utterance_lang=en-us
2024-07-02 15:42:38.799 - audio - ovos_config.models:load_local:112 - DEBUG - Configuration /home/neon/.config/neon/neon.yaml loaded
2024-07-02 15:42:38.801 - audio - ovos_config.config:_on_file_change:315 - INFO - /home/neon/.config/neon/neon.yaml changed on disk
2024-07-02 15:42:38.804 - audio - neon_audio.tts.neon:_get_tts:268 - INFO - Legacy Neon TTS signature found (CoquiRemoteTTS)
2024-07-02 15:42:38.825 - audio - ovos_config.config:_on_file_change:316 - DEBUG - Calling 1 callbacks
2024-07-02 15:42:38.834 - audio - ovos_audio.service:_maybe_reload_tts:360 - DEBUG - Skipping fallback TTS init
2024-07-02 15:42:39.037 - audio - ovos_audio.service:execute_tts:381 - INFO - Speak: Wake word hey neon is still enabled. Check the logs.
2024-07-02 15:42:39.352 - audio - ovos_config.models:load_local:112 - DEBUG - Configuration /home/neon/.config/neon/neon.yaml loaded
2024-07-02 15:42:39.412 - audio - ovos_config.config:_on_file_change:315 - INFO - /home/neon/.config/neon/neon.yaml changed on disk
2024-07-02 15:42:39.419 - audio - ovos_config.config:_on_file_change:316 - DEBUG - Calling 1 callbacks
2024-07-02 15:42:39.504 - audio - neon_audio.tts.neon:execute:373 - DEBUG - responses={'en-us': {'sentence': 'Sorry, something went wrong. I will continue using the old wake word.', 'translated': False, 'phonemes': None, 'genders': ['female'], 'female': '/home/neon/.cache/neon/tts/CoquiRemoteTTS/en-us/female/ea369b8d5507d03365d8a4c806a801bb.wav'}}
2024-07-02 15:42:39.617 - audio - neon_audio.tts.neon:get_requested_tts_languages:79 - DEBUG - Got profiles: [{'user': {'first_name': '', 'middle_name': '', 'last_name': '', 'preferred_name': '', 'full_name': '', 'dob': 'YYYY/MM/DD', 'age': '', 'email': 'daniel@neon.ai', 'username': 'local', 'password': '', 'picture': '', 'about': '', 'phone': '', 'phone_verified': False, 'email_verified': False}, 'brands': {'ignored_brands': {}, 'favorite_brands': {}, 'specially_requested': {}}, 'speech': {'stt_language': 'en-us', 'alt_languages': ['en-us', 'uk-ua'], 'tts_language': 'en-us', 'tts_gender': 'female', 'neon_voice': '', 'secondary_tts_language': '', 'secondary_tts_gender': 'male', 'secondary_neon_voice': '', 'speed_multiplier': 1.0}, 'units': {'time': 12, 'date': 'MDY', 'measure': 'imperial'}, 'location': {'lat': 47.6765382, 'lng': -122.2070775, 'city': 'Kirkland', 'state': 'Washington', 'country': 'United States', 'tz': 'America/Los_Angeles', 'utc': -8.0, 'country_code': 'us'}, 'response_mode': {'speed_mode': 'quick', 'hesitation': False, 'limit_dialog': False}, 'privacy': {'save_audio': True, 'save_text': True}}]
2024-07-02 15:42:39.621 - audio - neon_audio.tts.neon:get_requested_tts_languages:87 - DEBUG - local requesting female en-us
2024-07-02 15:42:39.628 - audio - neon_audio.tts.neon:get_requested_tts_languages:132 - DEBUG - Got 1 TTS Voice Requests
2024-07-02 15:42:39.658 - audio - neon_audio.tts.neon:get_multiple_tts:300 - DEBUG - tts_requested=[{'speaker': 'Neon', 'language': 'en-us', 'gender': 'female', 'voice': None}]
2024-07-02 15:42:39.662 - audio - neon_audio.tts.neon:get_multiple_tts:304 - DEBUG - utterance_lang=en-us
2024-07-02 15:42:39.668 - audio - neon_audio.tts.neon:_get_tts:268 - INFO - Legacy Neon TTS signature found (CoquiRemoteTTS)
2024-07-02 15:42:39.737 - audio - ovos_bus_client.session:get:605 - DEBUG - No message, use default session
2024-07-02 15:42:41.984 - audio - neon_audio.tts.neon:execute:373 - DEBUG - responses={'en-us': {'sentence': 'Wake word hey neon is still enabled. Check the logs.', 'translated': False, 'phonemes': None, 'genders': ['female'], 'female': '/home/neon/.cache/neon/tts/CoquiRemoteTTS/en-us/female/7f492da5f1189046714154697d04426e.wav'}}
2024-07-02 15:42:41.988 - audio - ovos_audio.service:_maybe_reload_tts:349 - INFO - (re)loading TTS engine
2024-07-02 15:42:41.991 - audio - ovos_plugin_manager.utils.config:get_plugin_config:40 - DEBUG - Loaded configuration: {'lang': 'en-us', 'voice': 'alan-low', 'module': 'ovos-tts-plugin-piper', 'pulse_duck': False, 'preload_fallback': False, 'fallback_module': 'coqui'}
2024-07-02 15:42:42.150 - audio - ovos_plugin_manager.utils:load_plugin:161 - WARNING - Could not find the plugin mycroft.plugin.tts.ovos-tts-plugin-piper
2024-07-02 15:42:42.155 - audio - neon_audio.tts:create:59 - ERROR - Could not find plugin: ovos-tts-plugin-piper
2024-07-02 15:42:42.159 - audio - ovos_config.config:_on_file_change:321 - ERROR - Error in config update callback handler
Traceback (most recent call last):
  File "/home/neon/venv/lib/python3.10/site-packages/ovos_config/config.py", line 319, in _on_file_change
    handler()
  File "/home/neon/venv/lib/python3.10/site-packages/ovos_audio/service.py", line 351, in _maybe_reload_tts
    self.tts.init(self.bus, self.playback_thread)
AttributeError: 'NoneType' object has no attribute 'init'
2024-07-02 15:42:43.525 - audio - neon_audio.tts.neon:_play:185 - INFO - Played 1719960158.5661793
2024-07-02 15:42:43.528 - audio - neon_audio.tts.neon:_sort_timing_metrics:149 - DEBUG - Parsed timing context: {'timestamps': {'handle_utterance': 1719960107.7861273, 'speech_start': 1719960158.5661988, 'audio_begin': 1719960159.7325459, 'audio_end': 1719960163.4334536}, 'durations': {'transform_audio': 3.1948089599609375e-05, 'get_stt': 5.132228374481201, 'save_transcript': 1.049041748046875e-05, 'text_parsers': 0.01896810531616211, 'transform_utterance': 0.01896810531616211, 'get_tts': 0.7698671817779541}}
2024-07-02 15:42:43.566 - audio - neon_audio.service:handle_speak:148 - DEBUG - Playback completed for: 1719960158.5661793
2024-07-02 15:42:49.745 - audio - ovos_utils:wait_for_exit_signal:183 - DEBUG - Exiting on KeyboardInterrupt
2024-07-02 15:42:50.137 - audio - ovos_config.config:_on_file_change:312 - DEBUG - Ignoring non-config file change: /home/neon/.config/neon/schedule.json
2024-07-02 15:42:51.317 - audio - ovos_bus_client.client.client:on_error:122 - ERROR - === RuntimeError('cannot schedule new futures after shutdown') ===
Traceback (most recent call last):
  File "/home/neon/venv/lib/python3.10/site-packages/websocket/_app.py", line 672, in _callback
    callback(self, *args)
  File "/home/neon/venv/lib/python3.10/site-packages/ovos_bus_client/client/client.py", line 160, in on_message
    self.emitter.emit(parsed_message.msg_type, parsed_message)
  File "/home/neon/venv/lib/python3.10/site-packages/pyee/_base.py", line 113, in emit
    handled = self._call_handlers(event, args, kwargs)
  File "/home/neon/venv/lib/python3.10/site-packages/pyee/_base.py", line 96, in _call_handlers
    self._emit_run(f, args, kwargs)
  File "/home/neon/venv/lib/python3.10/site-packages/pyee/_executor.py", line 50, in _emit_run
    future = self._executor.submit(f, *args, **kwargs)
  File "/usr/local/lib/python3.10/concurrent/futures/thread.py", line 167, in submit
    raise RuntimeError('cannot schedule new futures after shutdown')
RuntimeError: cannot schedule new futures after shutdown
2024-07-02 15:42:51.375 - audio - ovos_bus_client.client.client:on_error:126 - ERROR - Failed to emit error event: cannot schedule new futures after shutdown
Traceback (most recent call last):
  File "/home/neon/venv/lib/python3.10/site-packages/ovos_bus_client/client/client.py", line 124, in on_error
    self.emitter.emit('error', error)
  File "/home/neon/venv/lib/python3.10/site-packages/pyee/_base.py", line 116, in emit
    self._emit_handle_potential_error(event, args[0] if args else None)
  File "/home/neon/venv/lib/python3.10/site-packages/pyee/_base.py", line 86, in _emit_handle_potential_error
    raise error
  File "/home/neon/venv/lib/python3.10/site-packages/websocket/_app.py", line 672, in _callback
    callback(self, *args)
  File "/home/neon/venv/lib/python3.10/site-packages/ovos_bus_client/client/client.py", line 160, in on_message
    self.emitter.emit(parsed_message.msg_type, parsed_message)
  File "/home/neon/venv/lib/python3.10/site-packages/pyee/_base.py", line 113, in emit
    handled = self._call_handlers(event, args, kwargs)
  File "/home/neon/venv/lib/python3.10/site-packages/pyee/_base.py", line 96, in _call_handlers
    self._emit_run(f, args, kwargs)
  File "/home/neon/venv/lib/python3.10/site-packages/pyee/_executor.py", line 50, in _emit_run
    future = self._executor.submit(f, *args, **kwargs)
  File "/usr/local/lib/python3.10/concurrent/futures/thread.py", line 167, in submit
    raise RuntimeError('cannot schedule new futures after shutdown')
RuntimeError: cannot schedule new futures after shutdown
2024-07-02 15:42:51.835 - audio - ovos_bus_client.client.client:on_error:134 - WARNING - Message Bus Client will reconnect in 5.0 seconds.
2024-07-02 15:42:56.850 - audio - ovos_bus_client.client.client:on_open:93 - DEBUG - Connected

skills.log

2024-07-02 15:42:08.292 - skills - neon_utils.skills.neon_skill:speak:646 - DEBUG - Skill speak! {'utterance': "Okay, I'll be back shortly.", 'lang': 'en-us', 'expect_response': False, 'meta': {'dialog': 'confirm_change', 'data': {}, 'skill': 'DeviceControlCenterSkill'}, 'speaker': None, 'speak_ident': '1719960128.2668295'}
2024-07-02 15:42:08.295 - skills - neon_utils.skills.neon_skill:speak:647 - DEBUG - speak
2024-07-02 15:42:08.303 - skills - skill_device_controls:_set_user_tts_settings:565 - DEBUG - Patching user ngi config for persona jarvis
2024-07-02 15:42:08.308 - skills - skill_device_controls:_retrieve_ngi_config:583 - DEBUG - Patching user ngi config for Neon TTS
2024-07-02 15:42:08.339 - skills - ovos_bus_client.client.client:on_default_session_update:166 - DEBUG - synced default_session
2024-07-02 15:42:08.538 - skills - neon_utils.configuration_utils:_write_yaml_file:344 - DEBUG - YAML updated ngi_user_info
2024-07-02 15:42:08.540 - skill-device_controls.neongeckocom - DEBUG - Attempting to enable WW: hey_jarvis
2024-07-02 15:42:10.389 - skills - ovos_bus_client.util.scheduler:check_state:186 - DEBUG - Call scheduled event: skill-ovos-homescreen.openvoiceos:skill-ovos-homescreen.openvoiceosupdate_dt
2024-07-02 15:42:20.408 - skills - ovos_bus_client.util.scheduler:check_state:186 - DEBUG - Call scheduled event: skill-ovos-homescreen.openvoiceos:skill-ovos-homescreen.openvoiceosupdate_dt
2024-07-02 15:42:30.429 - skills - ovos_bus_client.util.scheduler:check_state:186 - DEBUG - Call scheduled event: skill-ovos-homescreen.openvoiceos:skill-ovos-homescreen.openvoiceosupdate_dt
2024-07-02 15:42:38.548 - skills - skill_device_controls:_emit_enable_ww_message:594 - ERROR - No response to WW enable request for hey_jarvis!
2024-07-02 15:42:38.551 - skills - skill_device_controls:_enable_wake_word:435 - ERROR - No response to WW enable request for hey_jarvis!
2024-07-02 15:42:38.555 - skills - neon_utils.skills.neon_skill:speak_dialog:680 - DEBUG - data={}
2024-07-02 15:42:38.568 - skills - neon_utils.skills.neon_skill:speak:646 - DEBUG - Skill speak! {'utterance': 'Sorry, something went wrong. I will continue using the old wake word.', 'lang': 'en-us', 'expect_response': False, 'meta': {'dialog': 'error_ww_change_failed', 'data': {}, 'skill': 'DeviceControlCenterSkill'}, 'speaker': None, 'speak_ident': '1719960158.5661793'}
2024-07-02 15:42:38.571 - skills - neon_utils.skills.neon_skill:speak:647 - DEBUG - speak
2024-07-02 15:42:38.752 - skill-device_controls.neongeckocom - DEBUG - Found available WW: {'hey_mycroft': {'active': False, 'module': 'ovos-ww-plugin-precise-lite', 'listen': True, 'sound': 'snd/start_listening.wav', 'version': '0.3', 'model': 'https://github.com/OpenVoiceOS/precise-lite-models/raw/master/wakewords/en/hey_mycroft.tflite', 'expected_duration': 3, 'trigger_level': 3, 'sensitivity': 0.5}, 'hey_neon': {'module': 'ovos-ww-plugin-vosk', 'listen': True, 'sound': 'snd/start_listening.wav', 'debug': False, 'rule': 'fuzzy', 'lang': 'en', 'samples': ['hey neon', '[unk]'], 'model_folder': '/home/neon/.local/share/neon/vosk-model-small-en-us-0.15', 'active': True}, 'hey_jarvis': {'module': 'ovos-ww-plugin-openwakeword', 'active': True, 'listen': True}}
2024-07-02 15:42:38.754 - skill-device_controls.neongeckocom - DEBUG - Found enabled WWs: ['hey_neon', 'hey_jarvis']
2024-07-02 15:42:38.755 - skill-device_controls.neongeckocom - DEBUG - Disabling WW: hey_neon
2024-07-02 15:42:38.927 - skills - skill_device_controls:_disable_wake_word:466 - WARNING - WW disable failed with response: {'error': 'only one active ww', 'active': True, 'wake_word': 'hey_neon'}
2024-07-02 15:42:38.933 - skills - neon_utils.skills.neon_skill:speak_dialog:680 - DEBUG - data={'ww': 'hey neon'}

It seems there's something going on with disabling the old WW before enabling the new one which causes an error to avoid a state where there is no active WW.

UPDATE:
After restarting, it looks like the Audio module is failing to load due to a missing plugin:

2024-07-02 15:46:18.874 - audio - neon_audio.tts:create:59 - ERROR - Could not find plugin: ovos-tts-plugin-piper
2024-07-02 15:46:18.880 - audio - ovos_audio.service:__init__:95 - ERROR - 'NoneType' object has no attribute 'init'
AttributeError: 'NoneType' object has no attribute 'init'
2024-07-02 15:47:41.265 - audio - ovos_utils.process_utils:exists:353 - ERROR - Failed to kill PID <_io.TextIOWrapper name='/tmp/neon/audio.pid' mode='r' encoding='UTF-8'>: [Errno 3] No such process
2024-07-02 15:47:57.173 - audio - neon_audio.tts:create:59 - ERROR - Could not find plugin: ovos-tts-plugin-piper
2024-07-02 15:47:57.176 - audio - ovos_audio.service:__init__:95 - ERROR - 'NoneType' object has no attribute 'init'
AttributeError: 'NoneType' object has no attribute 'init'

I'll install that and try again

Allows NeonGeckoCom/skill-device_controls#88 to succeed, among other things

mikejgray · 2024-07-11T14:51:50Z

@NeonDaniel NeonGeckoCom/NeonCore#702

mikejgray added 14 commits May 16, 2024 21:07

checking in, need to get tests running

edb03ca

add missing private method

42bddb4

work on original tests passing

7031a22

get existing tests passing before adding new ones

c323c3b

add locale files

4d8a936

add tests

eccc57b

fixes from local qa

d7594da

test adjustments

c538067

correction to callback

39f046e

register intent

db3c197

attempt to fix message on restart

cb4e22d

add confirmation and gui feedback

3b6b6d2

align keyword

54bb642

further confirmation

095637b

NeonDaniel reviewed Jun 6, 2024

View reviewed changes

JarbasAl reviewed Jun 6, 2024

View reviewed changes

__init__.py Outdated Show resolved Hide resolved

mikejgray added 3 commits June 6, 2024 23:20

requested changes

85c7171

add missing dialog

71b7dda

more missing dialogs

0011998

mikejgray marked this pull request as ready for review June 7, 2024 04:42

mikejgray requested review from NeonDaniel and JarbasAl June 7, 2024 04:42

This was referenced Jun 7, 2024

Adding another Voice #87

Open

feat: config to support additional voices and wakewords NeonGeckoCom/NeonCore#688

Merged

mikejgray added a commit to mikejgray/NeonCore that referenced this pull request Jul 11, 2024

feat: piper TTS plugin

90768b3

Allows NeonGeckoCom/skill-device_controls#88 to succeed, among other things

mikejgray mentioned this pull request Jul 11, 2024

feat: piper TTS plugin NeonGeckoCom/NeonCore#702

Merged

Merge branch 'dev' into FEAT_NewVoice

34365bb

mikejgray added 3 commits July 11, 2024 10:57

mark test as flaky so it doesn't block the suite

696a2db

tweak flaky

405ba3d

log

2483d31

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat: new voices and wakewords #88

feat: new voices and wakewords #88

mikejgray commented May 17, 2024 •

edited

Loading

NeonDaniel Jun 6, 2024

mikejgray Jun 6, 2024

NeonDaniel commented Jun 29, 2024

NeonDaniel commented Jul 2, 2024 •

edited

Loading

mikejgray commented Jul 11, 2024

feat: new voices and wakewords #88

Are you sure you want to change the base?

feat: new voices and wakewords #88

Conversation

mikejgray commented May 17, 2024 • edited Loading

Description

Issues

Other Notes

NeonDaniel Jun 6, 2024

Choose a reason for hiding this comment

mikejgray Jun 6, 2024

Choose a reason for hiding this comment

NeonDaniel commented Jun 29, 2024

NeonDaniel commented Jul 2, 2024 • edited Loading

mikejgray commented Jul 11, 2024

mikejgray commented May 17, 2024 •

edited

Loading

NeonDaniel commented Jul 2, 2024 •

edited

Loading