better "Defaults file" documentation

[allefeld]

The documentation under Defaults should be improved, for example like this:

I can share the underlying Markdown code, and also extend it to include all possible entries.

[jgm]

Yes, please do share.

[allefeld]

Will do, but I have to extend it to all entries first.

[allefeld]

Here it is. A complete list of command line options (according to pandoc --help), and an almost complete list of their defaults file counterparts.

Markdown source and pdf:
defaults.txt
defaults.pdf

It would be good if you could have a look, there are a few things I could not figure out, and others where I'm not certain I got it right. Let me know, and I'll make a new version.

One thing that is confusing me particularly: If I understand the YAML specification correctly, an omitted value is interpreted as null, which in a boolean setting in most languages is treated as equivalent to false. Consistent with that, highlight-style: and highlight-style: null mean --no-highlight. Yet output-file: has the meaning --output=-. Why not just require output-file: -? Or would that violate YAML syntax?

I also see some (unnecessary?) changes from command line options to defaults file entries. Generally, --option=VALUE translates to option: VALUE. If an option can be repeated on the command line, it instead translates to

options:
  - VALUE

So far, so logical.

Deviating from that:

• --read (which already has the alias --from) translates into readER:. Same for --write.
• The non-repeatable --resource-path=, which takes a string with : as separator, translates into a YAML list, not needing a separator. Similar for --indented-code-classes=.
• Different options --filter= and --lua-filter= translate into the same filters:.
• The repeatable options --css, --include-in-header, --include-before-body, --include-after-body, and --metadata do not have an appended s in translation. Granted, that would be awkward. But logical. :)

A more consistently logical translation could make this table unnecessary.

On the other hand, this table could serve not only as a documentation of defaults files, but as a general option reference. For that purpose, I would separate it into several sections, e.g. options affecting the reader, affecting the writer, affecting variables / metadata, and helper options (the last 11).

And I still think it should be an "options file", option --options=.

[jgm]

The defaults file corresponds closely to the structure of the Opts data structure used in convertWithOpts. In some places this diverges from command line option structure for good reasons. E.g. with filters, we need to preserve the order of the filters, which may be a mix of lua and json filters. If we just had lua-filters: and filters:, there'd be no way to specify that e.g. a lua filter needs to be run between two json filters.

--read (which already has the alias --from) translates into readER:. Same for --write.

This was me not remembering what the option actually was (since I never use these variants).
Probably good to support both read and reader, write and writer.

The non-repeatable --resource-path=, which takes a string with : as separator, translates into a YAML list, not needing a separator. Similar for --indented-code-classes=.

I think we should keep allowing the list, but also allow a non-list value (treating it like in options).

Different options --filter= and --lua-filter= translate into the same filters:.

See above.

The repeatable options --css, --include-in-header, --include-before-body, --include-after-body, and --metadata do not have an appended s in translation. Granted, that would be awkward. But logical. :)

I think this is fine the way it is. "metadatas" is not a word anyway ("metadata" already being plural).

For that purpose, I would separate it into several sections, e.g. options affecting the reader, affecting the writer, affecting variables / metadata, and helper options (the last 11).

That's what we've done for the options in the manual; keeping the same order for this would make sense I think.

And I still think it should be an "options file", option --options=.

This was discussed in the original issue for the defaults file. Two reasons I don't like options: -o is already taken; and not all of these things correspond to command-line options -- some (input-files) correspond to command-line arguments.

[jgm]

To answer your questions:

--quiet = verbosity: ERROR
--verbose = verbosity: INFO

--id-prefix PREFIX = identifier-prefix: PREFIX
(Here we should probably allow id-prefix as a synonym.)

--base-header-level=N = shift-heading-level-by: N-1
(or just omit this since it's a deprecated option)

One other thing: currently this table is too wide to fit into the standard of
80 column width for the manual. It seems to me we could make it narrower by leaving off the options that are (n/a) and leaving off the alternatives like block|section|document (which maybe aren't needed here as they're documented with the options?).

[jgm]

Oh, and aside from fitting into 80 columns in the markdown source, we need the table to render well in 80 columns for man page use, and to render in LaTeX in portrait mode. (Unless this is to be a separate document.)

[jgm]

I'm working on cleaning this up now. Never mind, I ran into difficulties. One is that the man writer creates indentation for code blocks, so things that need multiple lines don't line up well.

[allefeld]

I could use a definition list instead of a table, that should fit better?

[allefeld]

This was discussed in the original issue for the defaults file. Two reasons I don't like options: -o is already taken;

How about -O?

and not all of these things correspond to command-line options -- some (input-files) correspond to command-line arguments.

Good point. But command-line options are arguments, too, right? Then --arguments=, -a?

[jgm]

Technically you're right, they're all arguments: I guess I meant "mere arguments."
But anyway, I think --defaults is going to make more sense to people than --arguments -- especially because in this file you're not merely listing arguments as they'd appear on the command line.

[allefeld]

I think --defaults is going to make more sense to people

Hm, but that's exactly my problem: "defaults" doesn't make sense to me. In my understanding, a "default" is a fallback. Just like Pandoc's default templates: They are used if the user does not explicitly request a specific template. But there is no falling back upon "defaults files", they are functionally equivalent to command line arguments, explicitly requesting something.

There is some sense in applying the name "default" to the files in $DATA-DIR/defaults/, because they are not specific to a given input, but even those have to be explictly requested. Unlike e.g. $DATA-DIR/templates/default.latex, which is actually a fallback (which in turn overrules the built-in default template).

There could be "default arguments" proper, if Pandoc were to automatically use e.g. $DATA-DIR/defaults/latex.yaml for writing YAML. (Note, I'm not proposing such a feature.) But even then they would be "default arguments" as opposed to "default templates".

Sorry if I'm being overly pedantic...

[allefeld]

A few updates:

1. I added the missing options incl. the command line argument [FILES]. I think it should now be complete.
   defaults.pdf
   defaults.txt

[allefeld]

2. I replaced explicit option values by VALUE, because as you say the values are already documented, removed the deprecated options --base-header-level and --strip-empty-paragraphs, and two extra comments, and replaced "(as list)" by "(list)". As a result, the markdown source of the table is now at 81 characters width. That means, rendered for the man page it should be below 80 characters, and in LaTeX at 10 pt it fits into 16 cm width (portrait A4 with 1" margins).
   defaults-narrow.pdf
   defaults-narrow.txt

[jgm]

I do think of these things as fallbacks. For example: normally, pandoc creates a fragment unless -s is specified. But you can change that fallback by saying standalone: true in the defaults file.

Note that the fallbacks specified in a defaults file can be overridden by further options on the command line. So, if your defaults file says output-file: foo.pdf and you add -o bar.pdf on your command line, output will be to bar.pdf.

[jgm]

I didn't like having a single yaml entry spanning two table cells, so I redid this with code blocks in a grid table.

+----------------------------------+----------------------------------+
| command line                     | defaults file                    |
+:=================================+:=================================+
| ```                              | ``` yaml                         |
| foo.md                           | input-file: foo.md               |
| ```                              | ```                              |
+----------------------------------+----------------------------------+
| ```                               ``` yaml                          |
| foo.md bar.md                    | input-files:                     |
|                                  |   - foo.md                       |
|                                  |   - bar.md                       |
| ```                              | ```                              |
+----------------------------------+----------------------------------+
| ```                               ``` yaml                          |
| --from markdown+emoji            | from: markdown+emoji             |
| ```                              | ```                              |
+----------------------------------+----------------------------------+
| ```                               ``` yaml                          |
| --to markdown+hard_line_breaks   | to: markdown+hard_line_breaks    |
| ```                              | ```                              |
+----------------------------------+----------------------------------+
| ```                               ``` yaml                          |
| --output foo.pdf                 | output-file: foo.pdf             |
| ```                              | ```                              |
+----------------------------------+----------------------------------+
| ```                               ``` yaml                          |
| --output -                       | output-file:                     |
| ```                              | ```                              |
+----------------------------------+----------------------------------+
| ```                               ``` yaml                          |
| --data-dir dir                   | data-dir: dir                    |
| ```                              | ```                              |
+----------------------------------+----------------------------------+
| ```                               ``` yaml                          |
| --metadata key=value \           | metadata:                        |
|  --metadata key2                 |   key: value                     |
|                                  |   key2: true                     |
| ```                              | ```                              |
+----------------------------------+----------------------------------+
| ```                               ``` yaml                          |
| --metadata-file meta.yaml        | metadata-files:                  |
|                                  |   - meta.yaml                    |
| ```                              | ```                              |
+----------------------------------+----------------------------------+
| ```                               ``` yaml                          |
| --file-scope                     | file-scope: true                 |
| ```                              | ```                              |
+----------------------------------+----------------------------------+
| ```                               ``` yaml                          |
| --standalone                     | standalone: true                 |
| ```                              | ```                              |
+----------------------------------+----------------------------------+
| ```                               ``` yaml                          |
| --template letter                | template: letter                 |
| ```                              | ```                              |
+----------------------------------+----------------------------------+
| ```                               ``` yaml                          |
| --variable key=val \             | variables:                       |
|   --variable key2                |   key: val                       |
|                                  |   key2: true                     |
| ```                              | ```                              |
+----------------------------------+----------------------------------+
| ```                               ``` yaml                          |
| --wrap 60                        | wrap: 60                         |
| ```                              | ```                              |
+----------------------------------+----------------------------------+
| ```                               ``` yaml                          |
| --ascii                          | ascii: true                      |
| ```                              | ```                              |
+----------------------------------+----------------------------------+
| ```                               ``` yaml                          |
| --toc                            | toc: true                        |
| ```                              | ```                              |
+----------------------------------+----------------------------------+
| ```                               ``` yaml                          |
| --toc-depth 3                    | toc-depth: 3                     |
| ```                              | ```                              |
+----------------------------------+----------------------------------+
| ```                               ``` yaml                          |
| --number-sections                | number-sections: true            |
| ```                              | ```                              |
+----------------------------------+----------------------------------+
| ```                               ``` yaml                          |
| --number-offset=1,4              | number-offset: \[1,4\]           |
| ```                              | ```                              |
+----------------------------------+----------------------------------+
| ```                               ``` yaml                          |
| --top-level-division chapter     | top-level-division: chapter      |
| ```                              | ```                              |
+----------------------------------+----------------------------------+
| ```                               ``` yaml                          |
| --extract-media dir              | extract-media: dir               |
| ```                              | ```                              |
+----------------------------------+----------------------------------+
| ```                               ``` yaml                          |
| --resource-path .:foo            | resource-path: ['.','foo']       |
| ```                              | ```                              |
+----------------------------------+----------------------------------+
| ```                               ``` yaml                          |
| --include-in-header inc.tex      | include-in-header:               |
|                                  |   - inc.tex                      |
| ```                              | ```                              |
+----------------------------------+----------------------------------+
| ```                               ``` yaml                          |
| --include-before-body inc.tex    | include-before-body:             |
|                                  |   - inc.tex                      |
| ```                              | ```                              |
+----------------------------------+----------------------------------+
| ```                               ``` yaml                          |
| --include-after-body inc.tex     | include-after-body:              |
|                                  |   - inc.tex                      |
| ```                              | ```                              |
+----------------------------------+----------------------------------+
| ```                               ``` yaml                          |
| --no-highlight                   | highlight-style: null            |
| ```                              | ```                              |
+----------------------------------+----------------------------------+
| ```                               ``` yaml                          |
| --highlight-style kate           | highlight-style: kate            |
| ```                              | ```                              |
+----------------------------------+----------------------------------+
| ```                               ``` yaml                          |
| --syntax-definition mylang.xml   | syntax-definitions:              |
|                                  |   - mylang.xml                   |
| ```                              | ```                              |
+----------------------------------+----------------------------------+
| ```                               ``` yaml                          |
| --dpi 300                        | dpi: 300                         |
| ```                              | ```                              |
+----------------------------------+----------------------------------+
| ```                               ``` yaml                          |
| --eol nl                         | eol: nl                          |
| ```                              | ```                              |
+----------------------------------+----------------------------------+
| ```                               ``` yaml                          |
| --columns 72                     | columns: 72                      |
| ```                              | ```                              |
+----------------------------------+----------------------------------+
| ```                               ``` yaml                          |
| --preserve-tabs                  | preserve-tabs: true              |
| ```                              | ```                              |
+----------------------------------+----------------------------------+
| ```                               ``` yaml                          |
| --tab-stop 8                     | tab-stop: 8                      |
| ```                              | ```                              |
+----------------------------------+----------------------------------+
| ```                               ``` yaml                          |
| --pdf-engine xelatex             | pdf-engine: xelatex              |
| ```                              | ```                              |
+----------------------------------+----------------------------------+
| ```                               ``` yaml                          |
| --pdf-engine-opt=--shell-escape  | pdf-engine-opts:                 |
|                                  |   - '--shell-escape'             |
| ```                              | ```                              |
+----------------------------------+----------------------------------+
| ```                               ``` yaml                          |
| --reference-doc my.docx          | reference-doc: my.docx           |
| ```                              | ```                              |
+----------------------------------+----------------------------------+
| ```                               ``` yaml                          |
| --self-contained                 | self-contained: true             |
| ```                              | ```                              |
+----------------------------------+----------------------------------+
| ```                               ``` yaml                          |
| --request-header foo:bar         | request-headers:                 |
|                                  |   - ['foo', 'bar']               |
| ```                              | ```                              |
+----------------------------------+----------------------------------+
| ```                               ``` yaml                          |
| --abbreviations abbrevs.txt      | abbreviations: abbrevs.txt       |
| ```                              | ```                              |
+----------------------------------+----------------------------------+
| ```                               ``` yaml                          |
| --indented-code-classes python   | indented-code-classes:           |
|                                  |   - python                       |
| ```                              | ```                              |
+----------------------------------+----------------------------------+
| ```                               ``` yaml                          |
| --default-image-extension ".jpg" | default-image-extension: '.jpg'  |
| ```                              | ```                              |
+----------------------------------+----------------------------------+
| ```                               ``` yaml                          |
| --filter pandoc-citeproc \       | filters:                         |
|  --lua-filter count-words.lua \  |   - pandoc-citeproc              |
|  --filter special.lua            |   - count-words.lua              |
|                                  |   - type: json                   |
|                                  |     path: special.lua            |
| ```                              | ```                              |
+----------------------------------+----------------------------------+
| ```                               ``` yaml                          |
| --shift-heading-level-by -1      | shift-heading-level-by: -1       |
| ```                              | ```                              |
+----------------------------------+----------------------------------+
| ```                               ``` yaml                          |
| --track-changes accept           | track-changes: accept            |
| ```                              | ```                              |
+----------------------------------+----------------------------------+
| ```                               ``` yaml                          |
| --strip-comments                 | strip-comments: true             |
| ```                              | ```                              |
+----------------------------------+----------------------------------+
| ```                               ``` yaml                          |
| --reference-links                | reference-links: true            |
| ```                              | ```                              |
+----------------------------------+----------------------------------+
| ```                               ``` yaml                          |
| --reference-location block       | reference-location: block        |
| ```                              | ```                              |
+----------------------------------+----------------------------------+
| ```                               ``` yaml                          |
| --atx-headers                    | atx-headers: true                |
| ```                              | ```                              |
+----------------------------------+----------------------------------+
| ```                               ``` yaml                          |
| --listings                       | listings: true                   |
| ```                              | ```                              |
+----------------------------------+----------------------------------+
| ```                               ``` yaml                          |
| --incremental                    | incremental: true                |
| ```                              | ```                              |
+----------------------------------+----------------------------------+
| ```                               ``` yaml                          |
| --slide-level 2                  | slide-level: 2                   |
| ```                              | ```                              |
+----------------------------------+----------------------------------+
| ```                               ``` yaml                          |
| --section-divs                   | section-divs: true               |
| ```                              | ```                              |
+----------------------------------+----------------------------------+
| ```                               ``` yaml                          |
| --html-q-tags                    | html-q-tags: true                |
| ```                              | ```                              |
+----------------------------------+----------------------------------+
| ```                               ``` yaml                          |
| --email-obfuscation references   | email-obfuscation: references    |
| ```                              | ```                              |
+----------------------------------+----------------------------------+
| ```                               ``` yaml                          |
| --id-prefix ch1                  | identifier-prefix: ch1           |
| ```                              | ```                              |
+----------------------------------+----------------------------------+
| ```                               ``` yaml                          |
| --title-prefix MySite            | title-prefix: MySite             |
| ```                              | ```                              |
+----------------------------------+----------------------------------+
| ```                               ``` yaml                          |
| --css styles/screen.css  \       | css:                             |
|   --css styles/special.css       |   - styles/screen.css            |
|                                  |   - styles/special.css           |
| ```                              | ```                              |
+----------------------------------+----------------------------------+
| ```                               ``` yaml                          |
| --epub-subdirectory=""           | epub-subdirectory: ''            |
| ```                              | ```                              |
+----------------------------------+----------------------------------+
| ```                               ``` yaml                          |
| --epub-cover-image cover.jpg     | epub-cover-image: cover.jpg      |
| ```                              | ```                              |
+----------------------------------+----------------------------------+
| ```                               ``` yaml                          |
| --epub-metadata meta.xml         | epub-metadata: meta.xml          |
| ```                              | ```                              |
+----------------------------------+----------------------------------+
| ```                               ``` yaml                          |
| --epub-embed-font special.otf \  | epub-fonts:                      |
|   --epub-embed-font headline.otf |   - special.otf                  |
|                                  |   - headline.otf                 |
| ```                              | ```                              |
+----------------------------------+----------------------------------+
| ```                               ``` yaml                          |
| --epub-chapter-level 2           | epub-chapter-level: 2            |
| ```                              | ```                              |
+----------------------------------+----------------------------------+
| ```                               ``` yaml                          |
| --ipynb-output best              | ipynb-output: best               |
| ```                              | ```                              |
+----------------------------------+----------------------------------+
| ```                               ``` yaml                          |
| --bibliography logic.bib         | metadata:                        |
|                                  |   bibliography: logic.bib        |
| ```                              | ```                              |
+----------------------------------+----------------------------------+
| ```                               ``` yaml                          |
| --csl ieee.csl                   | metadata:                        |
|                                  |   csl: ieee.csl                  |
| ```                              | ```                              |
+----------------------------------+----------------------------------+
| ```                               ``` yaml                          |
| --citation-abbreviations ab.json | metadata:                        |
|                                  |   citation-abbreviations: ab.json|
| ```                              | ```                              |
+----------------------------------+----------------------------------+
| ```                               ``` yaml                          |
| --natbib                         | cite-method: natbib              |
| ```                              | ```                              |
+----------------------------------+----------------------------------+
| ```                               ``` yaml                          |
| --biblatex                       | cite-method: biblatex            |
| ```                              | ```                              |
+----------------------------------+----------------------------------+
| ```                               ``` yaml                          |
| --mathml                         | html-math-method:                |
|                                  |   method: mathml                 |
| ```                              | ```                              |
+----------------------------------+----------------------------------+
| ```                               ``` yaml                          |
| --webtex=URL                     | html-math-method:                |
|                                  |   method: webtex                 |
|                                  |   url: URL                       |
| ```                              | ```                              |
+----------------------------------+----------------------------------+
| ```                               ``` yaml                          |
| --mathjax                        | html-math-method:                |
|                                  |   method: mathjax                |
| ```                              | ```                              |
+----------------------------------+----------------------------------+
| ```                               ``` yaml                          |
| --katex                          | html-math-method:                |
|                                  |   method: katex                  |
| ```                              | ```                              |
+----------------------------------+----------------------------------+
| ```                               ``` yaml                          |
| --gladtex=URL                    | html-math-method:                |
|                                  |   method: gladtex                |
|                                  |   url: URL                       |
| ```                              | ```                              |
+----------------------------------+----------------------------------+
| ```                               ``` yaml                          |
| --trace                          | trace: true                      |
| ```                              | ```                              |
+----------------------------------+----------------------------------+
| ```                               ``` yaml                          |
| --dump-args                      | dump-args: true                  |
| ```                              | ```                              |
+----------------------------------+----------------------------------+
| ```                               ``` yaml                          |
| --ignore-args                    | ignore-args: true                |
| ```                              | ```                              |
+----------------------------------+----------------------------------+
| ```                               ``` yaml                          |
| --verbose                        | verbosity: INFO                  |
| ```                              | ```                              |
+----------------------------------+----------------------------------+
| ```                               ``` yaml                          |
| --quiet                          | verbosity: ERROR                 |
| ```                              | ```                              |
+----------------------------------+----------------------------------+
| ```                               ``` yaml                          |
| --fail-if-warnings               | fail-if-warnings: true           |
| ```                              | ```                              |
+----------------------------------+----------------------------------+
| ```                               ``` yaml                          |
| --log=FILE                       | log-file: FILE                   |
| ```                              | ```                              |
+----------------------------------+----------------------------------+

[allefeld]

Good! Are you going to integrate this version into the manual?

[jgm]

I've been having trouble making it work. The problem is the large margin we currently get with block-level content in table cells in man page output. I've been experimenting with some changes
to the man writer and template but haven't been able to get anything workable yet.

[jgm]

To get a sense for it, save this as dn.1 and do man ./dn.1:

.TH "" "" "" "" ""
.hy
.RS
.PP
hi
.RE
.SH test
.PP
.TS
tab(@);
lw(34.0n) lw(34.0n).
T{
.RE 0
command line
T}@T{
.RE 0
defaults file
T}
_
T{
.RE 0
.IP
.nf
.ft CW
foo.md
.ft R
.fi
.EE
T}@T{
.RE 0
.IP
.nf
\f[C]
input-file: foo.md
\f[R]
.fi
T}
T{
.RE 0
.IP
.nf
\f[C]
foo.md bar.md

\f[R]
.fi
T}@T{
.RE 0
.IP
.nf
\f[C]
 input-files:
 \ \ -\ foo.md
   - bar.md
\f[R]
.fi
T}
T{
.RE 0
.IP
.nf
\f[C]
--from markdown+emoji
\f[R]
.fi
T}@T{
.RE 0
.IP
.nf
\f[C]
 from: markdown+emoji
\f[R]
.fi
T}
.TE

[jgm]

Interestingly, if you comment out lines 3-6, the table looks good.

[allefeld]

Somehow missed your reply...

You're right, there is a lot of wasted space.

Maybe revisit the idea of using a description list instead of a table?

[jgm]

I'd like to figure out how to make the man table output better if possible.
A description list would take a lot of space because the defaults file version will appear underneath the command-line option.

[jgm]

Here's a slightly modified test case.
If you comment out lines 3 and 6 it will work properly. So, somehow, the presence of indented parts earlier in the document is messing up the tables, even though we're explicitly resetting indent inside the table cells. I couldn't find the answer by googling -- maybe need a roff expert to help.

.TH "" "" "" "" ""
.hy
.RS
.PP
hi
.RE
.PP
.TS
tab(@);
lw(34n) lw(34n).
T{
command line
T}@T{
defaults file
T}
_
T{
.RE 0
.IP
.nf
.ft CW
foo.md
.ft R
.fi
T}@T{
.RE 0
.IP
.nf
\f[C]
input-file: foo.md
\f[R]
.fi
T}
T{
.RE 0
.IP
.nf
\f[C]
foo.md bar.md

\f[R]
.fi
T}@T{
.RE 0
.IP
.nf
\f[C]
input-files:
\ \ -\ foo.md
  - bar.md
\f[R]
.fi
T}
T{
.RE 0
.IP
.nf
\f[C]
--from markdown+emoji
\f[R]
.fi
T}@T{
.RE 0
.IP
.nf
\f[C]
from: markdown+emoji
\f[R]
.fi
T}
.TE

[jgm]

I suppose we could just use a single code block in tabular form...

[allefeld]

I'm afraid I don't know the first thing about roff.

The roff code is generated from the Markdown by Pandoc I assume? If so, that sounds like there is a problem with the roff writer.

There are three people incl. email adresses listed on the groff page:
https://www.gnu.org/software/groff/
Maybe they would be willing to help?

[jgm]

Note: bibliography and csl can now be used directly in the defaults file.

[allefeld]

Thanks for the update!

After thinking about it again, I believe now that it would be better to mention the defaults-file version at the same place where the command line option is described. For example like this:

For this I just hacked a bit of MANUAL.html, for proper implementation we would need to come up with the proper Markdown version. And of course the text would have to be adjusted a bit, too.

What do you think?

[jgm]

I'd prefer not to depart this far from standard practice in presenting option lists; I think that might be confusing.

[brainchild0]

I do think of these things as fallbacks. For example: normally, pandoc creates a fragment unless -s is specified. But you can change that fallback by saying standalone: true in the defaults file.

Note that the fallbacks specified in a defaults file can be overridden by further options on the command line. So, if your defaults file says output-file: foo.pdf and you add -o bar.pdf on your command line, output will be to bar.pdf.

Fallback data is normally that used when a preferred set is unavailable or unusable. Yet the so-called defaults would often be given as a primary set of data to apply. Although command-line options would take precedent, they would be more often used to override select fields for a certain use, not to represent a complete set of options that hopefully remove the need for any other data set designated as fallback.

I tend to agree with @allefeld that the conceptualization as defaults or a fallback is confusing and possibly misleading.

---

This was discussed in the original issue for the defaults file. Two reasons I don't like options: -o is already taken;

How about -O?

and not all of these things correspond to command-line options -- some (input-files) correspond to command-line arguments.

Good point. But command-line options are arguments, too, right? Then --arguments=, -a?

The --arguments/-a suggestion may suffer from vagueness, but a near-identical suggestion that might evade such liability is --arguments-file/-a.

Previously the choice was made to choose a word whose initial was not currently used in lower-case form in the set of short options. I am having growing doubts that this restrictions is having a good effect on final design.

I further suggest that accepting that the file name as an undecorated (positional) argument, and inferring its meaning based on inclusion of some designated extension (e.g. .pd), would represent a highly user-friendly option and would avoid any conundrum over letters and words.

[the-solipsist]

I tend to agree with @allefeld that the conceptualization as defaults or a fallback is confusing and possibly misleading.

This is true. I did some testing, and I found that the values in the defaults files aren't just a fallback.

I tried adding author: metadata in 5 places: the in-file pandoc-style metadata (using %, which can only be added in the very beginning of the file, i.e., before the YAML metadata), in-file YAML (after the pandoc-style metadata), -M author=name, in the defaults file using metadata:, and in a metadata.yaml file invoked using --metadata-file. What I found is this order of precedence where > indicates overwriting and = indicates appending:

--metadata = --defaults > in-file YAML > in-file % > --metadata-file

This indicates that --defaults doesn't just act as a fallback, but overrides in-file YAML, in-file %, and even a command-line option (--metadata-file)

Further, there is inconsistency in this order of precedence depending on the flag/variables/metadata (CSS, for instance, behaves differently).

Full comment here: #3139 (comment)

[jgm]

"Fallback" isn't quite right, that's true. It's more like a replacement for command-line options you could specify on the command line; they agglomerate when the corresponding command line options would. I don't know if it would be a good idea to change the terminology at this point, though.

[brainchild0]

Building on John's comments, and trying to be fair to what I understand as the original intention, fallback and defaults would express a relation limited to that between the contents of such files and command-line arguments. Strictly, rules for processing metadata fields are not in conflict with this relation.

Overall, of course, nuances of such kind are difficult for a user to capture without making a serious study of them.

[the-solipsist]

I see no problem in the term "defaults". But I do think orders of precedence + when the items aggregate, when they take the left-most, when they take the right most, and when they replace, should be clearly captured in the manual. And ideally, there should be one standard rule that can be described in the manual, with any exception being for clearly stated reasons and also described accordingly.

…

On Tue, 13 Oct, 2020, 04:50 brainchild0, ***@***.***> wrote: Building on John's comments, and trying to be fair to what I understand as the original intention, *fallback* and *defaults* would express a relation limited to that between the contents of such files and command-line arguments. Strictly, rules for processing metadata fields are not in conflict with this relation. Overall, of course, nuances of such kind are difficult for a user to capture without making a serious study of them. — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#5990 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAXQE6FERZHUB6MTACQI5UDSKOFKDANCNFSM4J2W6SNQ> .

[brainchild0]

The earlier part of the manual probably should be revised so that defaults is not given to the user only as some later revelation.

(I have been away from this subject for some time, but last I remember, questions of the sort concerning stable aggregation rules had not been firmly resolved.)

[the-solipsist]

I'd prefer not to depart this far from standard practice in presenting option lists; I think that might be confusing.

Perhaps you're right. But it would be useful to provide a three-way comparison between valid command-line options, in-file metadata / metadata file options, and the defaults file option.

(I tripped up today, trying to use variables: in a metadata file before recalling I had to change the syntax.) I don't have a suggestion as to where this should be placed, but I feel it would be helpful for users.

[Kangie]

Hey, so following along this path (and to help with some automation on my end) I've been working on a JSONSchema for pandoc defaults files. Thanks for the documentation here - It made it a much easier process.

I've run into a few inconsistencies between the latest output in this issue (albeit from Dec 2019) and the pandoc defaults file example in the manual. Would someone be able to chime in here with which is more current / correct?

   [defaults-narrow.txt](https://github.com/jgm/pandoc/files/3993007/defaults-narrow.txt)

defaults-narrow.txt:

metadata:
  bibliography: string
  csl: string
  citation-abbreviations: string 

pandoc manual

csl: ieee
bibliography:
- foobar.bib
- barbaz.json
citation-abbreviations: abbrevs.json

In short, should they be keys under metadata, or just root keys?

And is bibliography a string or an array?

Either way, thanks again for the documentation. It has been quite helpful working with some CI systems lately!

[jgm]

In short, should they be keys under metadata, or just root keys?

If they're at root, they behave like the corresponding command line options. But what the command-line options do is set a metadata field. So, the answer is: either!

And is bibliography a string or an array?

Either will work.

[hoclun-rigsep]

Since this discussion did veer somewhat toward argument precedence, I hope my question will not be out of place:

If I specify, e.g., standalone or self-contained in my defaults file, is there a way to cancel that on the command line? Without, of course, editing the defaults file?

[brainchild0]

If I specify, e.g., standalone or self-contained in my defaults file, is there a way to cancel that on the command line?

I suppose this effect would require expanding the user interface, for example, to support arguments of the kind --no-standalone or --standalone=false. Then, perhaps the most appropriate course is to request a feature in a new issue.

[allefeld]

I would like to bump this issue, since I keep forgetting the defaults file entries.
If there's something I can do to help, let me know.

[allefeld]

I have created an updated version of the mapping table, based on jgm's version from by now more than 2 years ago: defaults.md
I fixed a few minor errors, made changes corresponding to changes in Pandoc since then, and broke the table into sections corresponding to those for options in the manual. (An unfortunate side effect of that is that the tables have different column widths; I don't know how to fix that across output formats.)

In doing so, I found a few inconsistencies: There are two command line options without apparent defaults file equivalent: --sandbox, --no-check-certificate, and there is a command line option whose equivalent is mentioned, but which is not documented itself: --trace.

@jgm I'm not sure whether you are still interested in integrating this into the manual?

[jgm]

@allefeld this would be welcome, yes. I think it would be an improvement over what we have now, although there are a few comments in the commented defaults file we now include that may be useful to include somehow (footnotes?, endnotes?).

I don't know why those options are missing. --trace is for use by developers, so I probably intentionally excluded it from the documentation. The others should be in defaults files, I think. I can add them, but don't let that stop you from submitting a PR, we can always modify later.

[jgm]

(An unfortunate side effect of that is that the tables have different column widths; I don't know how to fix that across output formats.)

If you use grid tables, column widths should be the same if they are the same in the source.

[jgm]

And, maybe submit another issue asking for sandbox and no-check-certificate (or perhaps check-certificate?) in defaults files, so we don't forget.

[jgm]

This is closed by #7872.