Switching template system from Mustache to Handlebars (or something else)

[wing328]

Description

There were discussions about switching from Mustache to Handlebars (or something else) before. The goal was to simplify the templates as Mustache is a logicless template system and therefore we will need to create a lot of mustache tag(e.g. isInteger, isHttpBasic, etc) every time we need to compare values.

As discussed in swagger-api/swagger-codegen#6077, Handlebars seems to be a good replacement. Here are a couple of questions we need to address:

• are we going to support both at the same time?
• when will we remove Mustache template support?
• Is any tool/script to help migrate the templates from Mustache to Handlebar? (my understanding is that Handlebar is not 100% backward compatible with Mustache)
• for new generators, when will we recommend using Handlebar as the default?
• if users have already customized the Mustache templates, how can they migrate those to Handlebars? (in other words, are we going to provide a tool/script or just a migration guide and users need to manually update the customized Mustache templates)
• let's say we switch the template system from Mustache to Handlebars in v4.0.0, how are we going to sync enhancements, bug fixes from v3.x to v4.0.0 before v4.0.0 is officially released?

The goal of this discussion is to collect feedback from the community and formula a plan to replace Mustache if needed

openapi-generator version

Future major release (4.x or later)

Related issues/PRs

It was previously discussed in swagger-api/swagger-codegen#6077

[wing328]

Here is my view:

• are we going to support both at the same time? Yes with mustache as the primary to start with and later switch to Handlebars as the primary
• when will we remove Mustache template support? If 5.x uses Handlebars as the primary template system, then we should Mustache template support in 6.x.
• Is any tool/script to help migrate the templates from Mustache to Handlebar? (my understanding is that Handlebar is not 100% backward compatible with Mustache) Not that I'm aware of. We can create scripts to ease the migration
• for new generators, when will we recommend using Handlebar as the default? Only after making Handlebar as the primary (default) template system
• if users have already customized the Mustache templates, how can they migrate those to Handlebars? (in other words, are we going to provide a tool/script or just a migration guide and users need to manually update the customized Mustache templates) Ideally we should provide a tool. Otherwise it's unlikely users will upgrade to the latst version of OpenAPI Generator
• let's say we switch the template system from Mustache to Handlebars in v4.0.0, how are we going to sync enhancements, bug fixes from v3.x to v4.0.0 before v4.0.0 is officially released? Ideally we should add the handlebar support and migrate the templates with a tool/script right before the 4.0.0 release so that we can continue to sync enhancements/bug fixes between major versions before that

[grokify]

Here are my thoughts:

• Should we switch to another templating system?
  • There are important benefits.
    • Both the Java and template code will be cleaner and easier to follow. Logicless templates are great when you actually have no logic, but when you start putting a lot of view logic in your code, it makes sense to consider more powerful templating systems.
    • More people can participate in building generators because less Java experience will be necessary for template builders. Template syntax is easier to learn and picking a solution that exists in many languages will may adoption easier (see next).
• Should we switch to Handlebars or something else?
  • An important characteristic for this project is that the template system be supported in many languages. This makes it more likely to be available in a template builder's language of choice and means it is more likely they are familiar with it and willing to invest their time using it, vs. a template system they otherwise would not use. Both Handlebars and Mustache are available in many languages but I'm not aware of other template systems like this.
• are we going to support both at the same time?
  • It's going to take a while for to transition all the generators and there may be things we learn along the way so supporting both makes a lot of sense.
• when will we remove Mustache template support?
  • Agree this should be 1 major release after Handlebars support is GA (generally available / non-experimental).
• Is any tool/script to help migrate the templates from Mustache to Handlebar? (my understanding is that Handlebar is not 100% backward compatible with Mustache)
  • There are a few areas of known incompatibility. We should investigate whether they affect our implementations or not. If they do affect us and there's no automatic transition, we should be able to document them.
• for new generators, when will we recommend using Handlebar as the default?
  • Recommendation can occur when Handlebars support is GA. I think it's ok for new generators to be built on Handlebars ahead of time if there is desire.
• if users have already customized the Mustache templates, how can they migrate those to Handlebars? (in other words, are we going to provide a tool/script or just a migration guide and users need to manually update the customized Mustache templates)
  • This is the hardest item for me. Do we know how many people have made customizations and how intensive they are?
  • The easier we can make the transition the better, but for some things like a tool/script, we should understand if it will help or not do the job. It may be worthwhile to transition some generators ourselves to get more information on what a tool/script could do. It may also be worthwhile to put out a guide and then see what the issues in transition are and how we could assist.
  • Some additional thoughts:
    • If the maintainers are available and have time, they may be more than happy to move to a system that is easier to maintain and understand.
    • If the maintainers are available and don't have time, they may have the same issue with merging current template modifications. It would be good to know if they are taking current modifications.
    • If the maintainer is no longer there, I've seen SDKs get deprecated.
• let's say we switch the template system from Mustache to Handlebars in v4.0.0, how are we going to sync enhancements, bug fixes from v3.x to v4.0.0 before v4.0.0 is officially released?
  • Since we are going to "add" Handlebars instead of switching, could we just add it in a 3.x release once it was functional? It could be experimental in 3.x and in beta in 4.x.

[jmini]

I hope @jimschubert will join this discussion because I think that he has some vision to change the architecture of the responsabilities between the component.

For the moment we have:

• DefaultGenerator: the main class that reads the spec, process them (delegating to the language specific layer), run the template engine and write the files. This is not Language specific.
• Generator classes (like JavaClientCodegen, GoClientCodegen, ...) that are a way to do language specific stuff.
• Codegen Classes like CodegenModel, CodegenParameter, ... that respresent the OpenAPI spec in a way that it is easy to write files (some treatment are made)

If I remember well, @jimschubert wanted to organize responsibilities in an other way.
If you want to introduce multiple template engine, thinking about who is responsible for what make sense.

[grokify]

If the interface to the generators, like JavaClientCodegen, GoClientCodegen, changes and we decide we need to have old and new systems running simultaneously because not all generators should be required to move at the same time, then one decision is whether a new template system would (a) only be on the new architecture or (b) be on the old and new architectures.

If the new template system is only on the new architecture, then it becomes more important to know the timing of the re-architecture.

Some of the re-architecture discussion in #503. Should we have some wiki or repo markdown pages to discuss?

[chatelao]

Is Handlebars in the 4.0.x branch already available?

[grokify]

Is Handlebars in the 4.0.x branch already available?

You can see work being done on this in PR #690 which hasn't been merged yet.

For now, you can get this in the rienafairefr/openapi-generator:templating branch.

[cognifloyd]

What about adding additional template engines that are never meant to become the default?

I'm in DevOps at a Python+Java (+ some JS though I haven't touched that yet) shop and rarely work with mustache or handlebars. I had to learn mustache to write the templates in #2270. Plus, the handlebars templates I've seen in swagger-codegen seem even more obtuse to me than mustache. Both feel clunky even if there are implementations in many languages.

I work with Jinja almost every day thanks to my DevOps work with Ansible, StackStorm, and other DevOps tools. So, for the python generators (this is the one I'm writing: #2270) in particular, I would prefer to work with Jinja templates to craft the python code. So for a python-focused generator I would like to use HubSpot/jinjava or something like that, but only for that generator. Of course, it's possible to write awful templates no matter the language, but allowing different languages to choose different template engines (assuming a java implementation is available) would be really nice.

[jimschubert]

@cognifloyd the long-term idea of #690 is to allow users to extend the "custom" template engines with whatever they prefer. For instance, to use Jinja this would require the user to create an adapter (or pull one from the community and add to claaspath), then explicitly select the template engine at execution time.

#690 gets us a lot closer to that use case than we are today. I plan to get #690 into master this week, with a note that only Mustache is supported for embedded templates.

[dkarlovi]

Since #2657 was merged, does that mean Handlebars is now fully supported?

[lindgrenfredrik]

I looking at migrating from Swagger-generator to Openapi-gen due to creating a issue on their git and not receiving any response for months

Currently have Handlebar templates for project so looking at running with it as template engine but found a issue with it:

In generateSupportingFiles in DefaultGenerator.java it looks if files end with correct template extension before compiling them as templates otherwise just writing them as files.
This works fine when mustache files registered for the different languages end with correct mustache but when switching engine it expects handlebars extension which isn't whats registered by "language" generator
The name conversion is done later in current setup (that is why it works for i.e. model files), which never gets called for Supporting files.

Any ideas on how to solve this properly?

• Allow mustache extension for any template engine?
• Convert names earlier?
  -- Register template files in some other manner to append correct ending?
• Run compile for all supporting files regardless of extension?

Should I write separate issue for this?

My temporary workaround is to allow mustache named targets to pass into compile section
master...lindgrenfredrik:aspnetcore_handlebars#diff-bd1652e990ffe227072a5c8908fd3054

[behrangsa]

Looks like this is merged to master. Is there a version based on master available?

[jimschubert]

@behrangsa I'm not sure what you're referring to as merged. We've had handlebars support throughout the 4.0 release. We don't have any built-in handlebars templates, and no real plans on the horizon to convert from mustache to handlebars as this would be an enormous effort.

[spacether]

When switching from mustache to handlebars one must:

• change all of your mustache extension templates to end in handlebars using a tool like mustache_to_handlebars mentioned in the next post. Using this tool is necessary to convert mustache #someTag to handlebars #if someTag, #each someTag, or #with someTag
• in the YourGeneratorNameCodegen.java file, all .mustache extensions must be changed to .handlebars
• invoke generate -e handlebars ... to switch our openapi generator templating engine to handlebars

[spacether]

I have made a tool that will allow us to automate conversion from mustache to handlebars templates
It is here: https://github.com/spacether/mustache_to_handlebars

If people want to use this tool to switch their openapi generator templates, they can invoke it like:

mustache_to_handlebars /Users/yourusername/programming/openapi-generator/modules/openapi-generator/src/main/resources/python-experimental -handlebars_if_tags="hasProduces isString hasItems hasHttpSignatureMethods hasAuthMethods isBodyParam isHeaderParam isNullable isKeyInHeader isDouble hasBearerMethods isKeyInQuery isFloat isDateTime hasHttpBasicMethods hasRequiredVars isOAuth isUnboundedInteger isByteArray isCookieParam hasVars getIsNull isEnum isDate isBinary isAnyType isPathParam hasMore isFormParam hasConsumes isNumber getHasRequired isBasic hasApiKeyMethods useNose isHttpSignature hasValidation hasRequired isArray isBasicBearer isAlias isBasicBasic isShort isReadOnly isMap isLong isBoolean hasOAuthMethods isApiKey isQueryParam getHasDiscriminatorWithNonEmptyMapping isKeyInCookie indent appName appDescription version infoEmail complexType appDescriptionWithNewLines produces consumes minItems maxItems notes getUniqueItems minProperties pattern maxLength exclusiveMinimum maximum multipleOf returnTypeIsPrimitive minLength required additionalPropertiesIsAnyType arrayModelType recursionLimit httpUserAgent licenseInfo baseType infoName returnType defaultValue asyncio tornado additionalPropertiesType infoUrl exclusiveMaximum minimum nameInSnakeCase description collectionFormat bearerFormat summary vendorExtensions.x-regex maxProperties" -handlebars_each_tags="servers scopes vendorExtensions.x-auth-id-alias getComposedSchemas enumVars optionalParams optionalVars headers imports mappedModels requiredVars vars models apis authMethods requiredParams enumValues vendorExtensions.x-modifiers variables allParams responses allOf oneOf anyOf operation" -handlebars_with_tags="items additionalProperties model apiInfo operations allowableValues discriminator"

[spacether]

The python-experimental generator now successfully uses the handlebars templating engine by default. All templates for that generator are handlebars and must be handlebars for that generator to work. I specifically used that templating engine because it allows recursive indented partials. I use that to render schema definitions at any depth of inlining.

[dziedrius]

I've tried to use https://github.com/spacether/mustache_to_handlebars for typescript fetch template - it does some work, but there's more work needed to be do manually.

Where I got stuck and decided not to go further - seems that you need to register partials in code that is running templating engine - so in generator's javascript, which takes away lots of flexibility, like we're using partials to isolate custom code from original template, so that template upgrade would be much easier. Probably that could be easily solved by scanning template directory and registering all partials dynamically, but intent was not to touch generator's code.

Without support from others and mass migration to handlebars, I don't see this initiative as viable.

[spacether]

partials templates should work without registering them. If you have specific helper functions in java that are custom, they need to be registered. When using partials, partial templates need to indicate where they are relative to the root folder containing that generators templating files.
So values like:

{{> components/schemas/_helper_getschemas }}

where components + schemas are folders and _helper_getschemas.hbs is the partial file

The only helpers that I use are these:

        Handlebars handlebars = new Handlebars(loader);
        handlebars.registerHelperMissing((obj, options) -> {
            LOGGER.warn(String.format(Locale.ROOT, "Unregistered helper name '%s', processing template:%n%s", options.helperName, options.fn.text()));
            return "";
        });
        handlebars.registerHelper("json", Jackson2Helper.INSTANCE);
        StringHelpers.register(handlebars);
        handlebars.registerHelpers(ConditionalHelpers.class);
        handlebars.registerHelpers(org.openapijsonschematools.codegen.templating.handlebars.StringHelpers.class);
        handlebars.registerHelpers(CustomHelpers.class);
        handlebars.setInfiniteLoops(infiniteLoops);
        handlebars.setPrettyPrint(prettyPrint);
        Template tmpl = handlebars.compile(templateFile);
        return tmpl.apply(context);

[bodograumann]

Personally I have never missed logic in mustache when writing/modifying generator templates. (That doesn't mean it cannot improve things.) What I find more difficult, is knowing the data structure that I am actually working with in the templates.

I was thinking about mustache in a different context and wanted to mention the following point here.
Mustache, like many templating tools, is mainly intended for generating HTML. That means

• it escapes as html by default,
• it does not care about whitespace and
• it uses curly braces for place holders, because these are rare in html.

We are not only generating HTML here, but mostly source code. So wouldn't it be better to switch to a templating language that is geared towards code generation instead of website generation?

[dobromyslov]

This bug makes it impossible migrating to Handlebars - jknack/handlebars.java#940 (comment)

It seems to me that HandlebarsEngineAdapter uses resolvers in incorrect order:

• MapValueResolver,
• JavaBeanValueResolver,
• FieldValueResolver,
• MethodValueResolver

Context context = Context
        .newBuilder(bundle)
        .resolver(
                MapValueResolver.INSTANCE,
                JavaBeanValueResolver.INSTANCE,
                MY_FIELD_VALUE_RESOLVER.INSTANCE,
                MethodValueResolver.INSTANCE)
        .build();

Handlebars maintainer said that FieldValueResolver must be the last one and be used to access public fields to avoid illegal access exception:

Caused by: java.lang.IllegalStateException: Shouldn't be illegal to access field 'size'
at com.github.jknack.handlebars.context.FieldValueResolver.invokeMember (FieldValueResolver.java:217)
Caused by: java.lang.IllegalAccessException: class com.github.jknack.handlebars.context.FieldValueResolver$FieldMember cannot access a member of class java.util.HashMap (in module java.base) with modifiers "transient"
at com.github.jknack.handlebars.context.FieldValueResolver$FieldMember.get (FieldValueResolver.java:135)