Caster as a Package: Master Thread

[LexiconCode]

This is an issue that will help track only various issues, pull requests and documentation related to the transition to a the Caster Voice pip package. #374 implemented caster as a PIP package. Once we are comfortable with the pull request to this issue will be closed.

Each issue or sub issue will have boxes to indicate whether their Status.

Legend

• ✔️ : Checkmark indicates complete
• ⭕️ : issue has a pull request but has not been merged.
• ❗️ : issue has only been documented.
• ❓ : Items that need to be investigated that may lead to documented issue.

Issues

• ✔️ Invalid links due to directory name change #379 Readme links are broken in the development branch.

  • ✔️ : Utilize underscores instead of white spaces for .md for filenames.Utilize underscores instead of white spaces for .md for filenames. #376
  • ✔️ : Not includ version number in CasterQuickReference0.5.8.pdf file name. Not includ version number in CasterQuickReference0.5.8.pdf file name. #375
  • ✔️ : Are there broken links in rest of the documentation?
  • ✔️ : Update broken links

• Issues with configdebug.txt

  • ✔️ : beb208a Converted imports to castervoice
  • ✔️ : Move configdebug.txt file to user space  #378 Move configdebug.txt file to user space

• Issues with PIP castervoice package

  • ✔️ : IOError: [Errno 2] No such file or directory: 'README.md'CasterVoice PIP package fails to install: [Errno 2] No such file #381
  • ❗️ : Move CCR and Apps in to user space Move CCR and Apps in to user space #384
  • ✔️ : Post install script for setup.py to install _caster.py Post install script for setup.py to install _caster.py #388

• Bugs related to transition

  • ✔️ : Userspace rules are not merged. Userspace rules are not merged. #399
  • ✔️: bringme.toml is blank on new installations caster bringme.toml is blank on new installations of caster #400
  • ✔️: ReadTheDocs does not rebuild automatically ReadTheDocs is broken #431

Other

• ✔️ : Caster Settings Window: Timeout while waiting for window Caster Settings Window: Timeout while waiting for window  #377.
• ❓ : Update install instructions
• ✔️ : Implemented TerminalCommand and a update mechanism. Implemented TerminalCommand and a update mechanism. #369

[mrob95]

My 2c: I think that if the py files for CCR rules and apps are going to be taken out of user space then there need to be better and clearer ways of adding or modifying commands than are currently available, otherwise we are taking a lot of power out of the hands of users.

As we've seen recently, almost the first thing most users, especially programmers, want to do when they get started is to see how the commands are defined and work out how they can add new ones and modify existing ones. At the moment it's as simple as opening a couple of .py files in locations which the user is aware of. It doesn't even need documentation because the code is self-explanatory. How is this going to work when all of the files are buried in the Python directory and vulnerable to being overridden during updates?

While I think it's fine to take mechanism like CCRmerger and hide it away, in principle users should have complete control over policy.

[LexiconCode]

Those are some good thoughts, thank you. First I would like to try to restate the issues you've brought forth in my own words. That gives you a chance to verify and clarify my understanding.

• Access to source code has a higher barrier to entry due to being buried in the Python directory.
  • The process for customizing commands requires access to source code for the user to understand implementation.
  • Documentation isn't really necessary due to the simplicity of source code especially CCR and applications.
• The process managing custom commands are not well defined
  • Customize existing CCR or applications
  • Add completely new CCR or applications
• The update process will override user customizations if there implemented in source code.

[synkarius]

This is a very valid concern. The issue with destroying customizations during updates was always a problem. The old /user package was a poor solution. (Most people have just customized the app and CCR "starter" rules and then don't update.) The .caster/rules directory is better, but @mrob95 is right that it makes this ability to customize the "starter" rules very difficult. Here's my (first) suggested solution: the first time Caster is run, copy all the starter rules into the .caster/rules directory and from then on, load them from there.

[LexiconCode]

So when were talking about starter rules were talking about applications and CCR. They would be copied over into the .caster/rules then loaded from then on from there. Okay, so that would be a functional stopgap measure. At least until we figure out how to merge updates with the customizations made in the starter rules?

[kendonB]

Is there a simple words.txt option still? This (at least in principle) was a good way to fix up problem default words in my specific accent

[LexiconCode]

Is there a simple words.txt option still? This (at least in principle) was a good way to fix up problem default words in my specific accent

Place words.txt in C:\Users\%USERNAME%\.caster\data

[synkarius]

@LexiconCode I don't know if it actually needs to be a stopgap measure. If one of the starter rules gets updated and someone (A) is aware of the update and (B) wants it, they can just copy the file down from Github, right? Probably the "modify and never update" strategy is far more common than the people who've been doing commits and resolving merge conflicts. Thoughts?

[Versatilus]

I agree with you both. The files should be copied over to the user directory if they are absent and the user should be nagged to death if there's a potential conflict with the changes.

[LexiconCode]

Hopefully we can improve the situation in the future. We should continue looking for a solution that doesn't edit source code but survives updates for user customizations. This would still help group A and B.

However for now, we can move CCR and Apps into the user directory. This option should be able to be turned off in the user settings for those that are comfortable enough with filter rules.

If we do move forward with this issue there may be things that change in the package that aren't updated in the CCR or applications that might break.

I'm also curious about your work with python.toml

[mrob95]

@LexiconCode I think TOML may be at least a partial solution, 95% of what users do on a daily basis can be done with just Key and Text objects, which can easily be defined as strings in a config file. My Python rule looks like this:

BINDINGS = utilities.load_toml_relative("config/python.toml")

class Python(MergeRule):
    non = PythonNon
    pronunciation = BINDINGS["pronunciation"]

    mapping = {
        "<command>":
            Function(execution.alternating_command),

        BINDINGS["function_prefix"] + " <fun>":
            Text("%(fun)s()") + Key("left"),
    }

    extras = [
        Choice("fun", BINDINGS["functions"]),
        Choice("command", BINDINGS["commands"]),
    ]

And the toml file looks like this:

[commands]
"value false" = "False"
"with open" = ["with open() as f:", "left:7"]

[functions]
"(int | integer)" = "int"
"append" = "append"
"clear" = "clear"
"character" = "chr"
"copy" = "copy"

Alternating command just executes the first string as text, second as keys, etc.

One of the major advantages of this is that it allows for basic addition/editing of commands without rebooting Dragon. Doing a merger.wipe(), reloading the CCR modules and then doing a boot merge is enough.

Doing this for every rule would be a fairly major project though so it's probably something to think about for the long term.

[LexiconCode]

I have an alternative proposal. Although we can certainly do what is been mentioned previously depending in how we proceed. Although if you guys don't think the following proposal worth it, that's okay. Here my thoughts. Starting off with some Terminology.

For every application and CCR file that's in the source code a corresponding template file would be generated upon start up if absent in the user space. The CCR would work the same way as the apps. I'll use the app folder as a example.

Currently castervoice\apps the directory looks like...

atom.py

cmd.py

sublime.py ... And so on

Then in the user space .caster\rules\apps an apps file is created. For each application found in castervoice\apps A template would be generated during caster start. If a new app is present it would be generated upon first startup after updating caster.

atom.py

cmd.py

sublime.py ... And so on

So what would be in these template files contained in .caster\rules\apps?

Each file would be a template for two purposes.

1. New User commands: First we could create a merge rule based on each application that is merged with the source code application specific grammar. Similarly how CCR merged into global grammars but application-specific. This would be the space where users could add their own content and functions. On boot time it would merge or unless we can come up with a clever solution.

• class SublimeRule(MergeRule)

new user command - "custom command [<n2>]": R(Text("c-d/10"), rdescript="Sublime: Trigger command n")*Repeat(extra="n2"),

2A. Customized commands from source code: When users customize they altar the spec or the action. Not both otherwise it would be a new Command. So if we add a special class that compares the spec and action during to each source code command during merge.

• -- For instance source code spec in sublime.py

  class SublimeRule(MergeRule) as represented in current source code.

original source code - "edit next [<n3>]": R(Key("c-d/10"), rdescript="Sublime: Edit Next n")*Repeat(extra="n3"),

• class SublimeRule(OverrideRule)

Customized Spec - "edit tab [<n3>]": R(Key("c-d/10"), rdescript="Sublime: Edit Next n")*Repeat(extra="n3"),

-- or --

Customized Action - "edit next [<n3>]": R(Key("c-f1"), rdescript="Sublime: Edit Next n")*Repeat(extra="n3"),

2.B Another tack would be to utilize MergeRulewith something like

OverridMapping

OverrideExtras

OverrideDefaults

If either a customized Spec or Action matches part of an existing command It will be overridden with the user's customization. As an edge case, If for some reason both action and rule are different is treated as a completely new command.

How is this better than the current implementation and process for user customization?

• It matches very closely to the dragonfly API and the source code.

• It guides the user to add code to existing applications/ccr. That would make it as simple as opening up .py file and adding edits or additions.

• Custom functions are no different than the source code.

• User Customized commands are no different than any other command except for the class.

• User made apps would simply be dropped in .caster\rules\apps the apps folder and loaded. Same for CCR.

  The end result is we have a update resilient one file per CCR or App that contains both custom additions and overrides the don't edit source code. A user experience that is guides and that falls in line with the dragonfly API. So programmers can reference the source code in editor we could add a command ' show application commands or CCR'. The command open up the corresponding .py within python directory for reference.

@mrob95 While I think your solution is very neat but there are some things that need to be considered.

• Moving to the toml format for customizations on that's type of scale makes the dragonfly API obscured and especially the source code. Thus the documentation even more confusing to a new user. It's no longer as simple as opening up a .py file.

• Users you lose out on the ability to leverage the IDE's because toml not python format.

Ultimately I think reload ability needs to be addressed in Dragonfly and then Caster.

That's just my perception and I'm totally open to whatever the consensus of our current users.

[synkarius]

@LexiconCode , that solution comes very close to reinventing Vocola.

Reloadability is achievable without text files. Ultimately the text files should go away. In order to achieve better reloadability, we ought to be actually managing modules via importlib and watching directories for file changes, possibly with fallback to previously loaded modules if the changed files have errors.

As for the updates/customizations issue, this sidesteps it in the same way that copying all the full Python files does, except it adds the complexity of needing to know the template format.

[mrob95]

I think that just copying the files is a reasonable solution for now. It's probably not all that often that changes are made to the existing grammar definitions. Adding new App/CCR grammars will be fine, since the new files will just be created in the user's grammar directory.

[LexiconCode]

Excellent so the consensus is to add move the CCR and apps over.

So I'll go ahead and create two separate issues regarding moving apps/CCR and the ability to reload grammars.

Reloadability of grammars #385
Move CCR and Apps in to user space #384

[LexiconCode]

One other issue at hand to be discussed. Currently the PIP install doesn't include _caster.py. which makes sense.

I was wondering if there was possibility to create a post install script that runs after PIP that would detect c:\NatLink\NatLink\MacroSystem and drop _caster.py and if the folder does not exist place it on the desktop for wsr. It could possibly do other things as well.

I could make the script and integrate it in to setup.py. Post-install script with Python setuptools

thoughts?

[synkarius]

All three are good ideas. I can tackle #384 and #385 if you want to assign those to me.

[synkarius]

That said, if you put that in setup.py, will it run on every update? I think we only want to copy the files to the user directory once.

[LexiconCode]

That said, if you put that in setup.py, will it run on every update? I think we only want to copy the files to the user directory once.

setup.py will run every update. This is for making sure that the _caster.py stays up-to-date with when pip updates caster. However my intent wasn't to use setup.py move/copy CCR/apps. I figured we could trigger that during startup checking Caster version versus previous version. This could to be defined in settings.toml indicate version numbers.

Thinking on it more we could use setup.py for that purpose as well but it's up to you.

It's a little bit more than a simple one time copy. We don't want to overwrite files in the user directory that already exist. In an update scenario where a new application or CCR is added. Those could be safely added in the user space as well.

[synkarius]

Yes, I agree with that design on all points:

• setup.py to install _caster.py and even overwrite it
• one-time copy without overwrites of any "starter" rules, including new ones

And yes, I'd rather keep the starter rules setup out of setup.py.

[LexiconCode]

The last issue

Move CCR and Apps in to user space

Will be integrated in the the roadmap. Therefore this issue will close.