Skip to content

TestSuite

Jump to bottom
fred-wang edited this page Apr 13, 2012 · 3 revisions

TestSuite

Note: this page was initially written as a plan for a test suite. Since that time, most of the tests mentioned and many others have been written.

Current Status

Currently, the test/ directory contains various test pages. Here is those available on Sean's repository:

  • installation tests

    • install.html: browser's version and webfonts, including Firefox issues with font versions and same-origin policy.
    • install-images.html: tests image fallback. Warning for Internet Explorer 6 with alpha-channel.
    • install-diag.html: Provides installation diagnostics: errors or performance issues ; MathJax script loads ; MathJax configuration ; file downloads ; scripts and font-files compressions ; font-files enabled for cross-domain requests.
    • page-diag.html: Provides a few basic checks of a MathJax enabled page. Also takes a comprehensive snapshot of browser, window and MathJax configuration which may be posted to a support group for analysis. Provides basic timing charts for a MathJax enabled page. These can give an indication of where a page seems to fail or be slow.

  • sample pages (can be used by page-diag)

    • sample.html: a page containing math in TeX format
    • sample-tex.html: a basic page containing math in TeX format
    • sample-mml.html: a basic page containing math in MathML format. The appearance should be the same as sample-tex.html
    • sample-tex2mml.html: a basic page containing math in TeX format. If the browser has native support for MathML then the rendering is delegated to the browser. Otherwise the appearance should be the same as sample-tex.html
    • sample-signals.html: a basic page demonstrating how to use MathJax's signal mechanism to find out about what MathJax is doing, and to hook into various events as they occur.
    • sample-dynamic.html: a simple demonstraton of programming with MathJax to update math on the page in response to user input.

Terminology and general remarks

  • reftest: two webpages (test and reference) which are expected to provide the same output. This output can be the visual rendering, the tree built etc. The reftest can be == (test and reference are expected to be equal) or != (test and reference are expected to be different).
  • crashtest / "load" test: a single webpage which ends by a "success" javascript event. The test consists in loading the patch and listening for the "success" event (test passed). The test is interrupted after a timeout or a browser crash (test fails).
  • unit test: the test is only testing a single feature. This is generally the case of reftests/crashtests.
  • scenario test: similar to "load" test but with interaction between a virtual user (for instance using Selenium actions) and the browser. Some steps may involve reftest-like checkings.

Some remarks:

  • For visual tests of pages whose rendering is changing in function of the time, it may be difficult to have check the accurate timing so we should use large time intervals.
  • For menus whose positions are determined by an offset with respect to a reference element, visual tests may also be difficult.
  • Some results depend on the browsers, the fonts installed etc and tests may be hard to designed for them.
  • To make the tests independent from the testing framework, it may be best to write it with webpages using only javascript. For instance, one can simulate mouse events or compare tree structures using only javascript. The testing framework would be used only to collect the results, take screenshots of visual renderings or other features not available in javascript.

Accessibility

  • copy-and-paste: a scenario test which uses MathJax menu to copy the LaTeX/MathML source and paste it in a textarea. The content of the textarea is then compared against the original source.
  • scale all math: several scenario tests which use MathJax menu to set the scale factor and compare the rendering against reference pages (using zscale).
  • zoom: several scenario tests which use MathJax menu to set the Zoom trigger/factor and compare the rendering against reference pages (using zoom, CTRL).
  • about mathjax: scenario test which check if a box is displayed and if we can follow the link to http://www.mathjax.org/
  • MathJax help: scenario test to check whether we are redirected to http://www.mathjax.org/help/user/
  • screen readers: scenario test or reftest based on "sound" comparaison?
  • rendering mode: a scenario test which uses MathJax menu to select the MathRenderer and compare and compare the rendering against reference pages using different output modes.

What about I18N support?

MathML/LaTeX

  • reftests covering all the LaTeX commands supported
  • reftests covering all the MathML elements and attributes supported
  • import Mozilla MathML reftests when possible (done)
  • import MathML3 testsuite reftests when possible

Configuration

Contexts

  • HTML/XHTML pages. Two pages xhtml-1.xhtml (in XHTML format) basic-ref.html (in HTML format) with a same basic formula (== reftest).
  • pages served with/without application/xhtml+xml MIME-type. Two pages xhtml-2.html and basic-ref.html with the same set of formula (== reftest). Use a .htaccess to map the first one to the application/xhtml+xml MIME type. If the reftest is not run on a server, the test will just be passed.
  • with/without namespaces on &l;math/>: namespace-1.html (without the namespace), basic-ref.html (== reftest)
  • with/without namespace prefixes on <<math/>: namespace-2.html (with the prefix), basic-ref.html (== reftest)
  • hardcoded <script src="MathJax.js"/> or inserted dynamically. script-location-1.html (script inserted dynamically) and basic-ref.html (== reftest)
  • In the <head/> or <body/> of the document. dynamically. script-location-i.html (several tests with script at various locations in the body) and basic-ref.html (== reftest)
  • inside script type="math/mml" or type="math/tex". math-location-i.html (math written inside a script element) and basic.html (== reftests).
  • <foreignObject/> in SVG diagrams (not mentioned in the doc, not sure it works). svg-1.html, svg-1-ref.html (!= reftest). The test has MathML in a foreignObject, the reference has not.

Main configurations

  • extensions "tex2math.js","TeX/noErrors.js","TeX/noUndefined.js", "TeX/AMSmath.js","TeX/AMSsymbols.js", "mml2jax.js". Different pages using macros specific to each package (load test).
  • config: "MMLorHTML.js". mml-or-html-1.html, mml-or-html-1-ref.html. Compare the result against a reference page using "output/HTML-CSS" and "output/NativeMML" according to the browser tested. (== reftest)
  • tex2jax "inlineMath", "displayMath", "processEscapes". (see tex2jax preprocessor configuration below)
  • jax "input/TeX", "input/MathML". Reftests against static pages where only the TeX/MathML elements have been replaced by a HTML-CSS output. (== refests)
  • jax "output/HTML-CSS", "output/NativeMML". comparison against a static MathML page. (!= and == reftests)

Other options

  • The Core Configuration Options

    • config: (== reftests). For config, test a local config.js, not the predefined MMLorHTML.js
    • styleSheets, styles: (== reftests)
    • inside script type="math/mml" or type="math/tex". math-location-i.html (math written inside a script element) and basic.html (== reftests). Also test preJax/postJax workaround for IE (take into account regular expression
    • such as preJax: "\\\\").
    • preRemoveClass: one test to check that the MathJax_Preview is present before processing the <script/> tag and one test to check that it is removed afterwards (== reftests).
    • showProcessingMessages, messageStyle: reftests for these options may be difficult. Instead, it is maybe possible to do some javascript to listen changes of values in the message box and compare with those expected according to the options set.
    • displayAlign, displayIndent: (== reftests)
    • delayStartupUntil: a test listening the MathJax startup signals and compare when they happen with respect to the onload event.
    • skipStartupTypeset: a test listening the MathJax signals and checking that no element is processed.
    • menuSettings: a scenario test for the context option which compares the rendering of the contextual menus (!= reftest). Other options can be tested in scenario tests as described in the accessibility section. Simple != reftests for zscale can also easily be written.
    • errorSettings: compare test with incorrect markup and a reference of styled error messages (== reftests). Compare rendering of an incorrect markup with/without this setting (!= reftest).

  • The tex2jax Preprocessor

    • inlineMath, displayMath", processEscapes, element, processEnvironments, skipTags, ignoreClass, processClass: test pages with several formulas. check that only those described by the options are processed by tex2jax. They can be visual == reftests or test listening MathJax signals.
    • preview: test that the TeX code or a custom preview is inserted. (== reftests with skipStartupTypeset)

  • The mml2jax Preprocessor

    • element, preview: same tests as for tex2jax. (test the altext)

  • The jsMath2jax Preprocessor

    • element, preview: same tests as for tex2jax.

  • The TeX input processor

    • TagSide, TagIndent, MultLineWidth: visual reftests
    • Macros: several reftests to compare the effects of the macros with those of the corresponding LaTeX code expected to be generated.

  • The MathML input processor

    • useMathMLspacing: reftests to compare MML/TeX spacing (rules not really described in the doc)

  • The HTML-CSS output processor

    • scale: visual reftests to compare the visual rendering of different scale. Also test the effect of different sizes for the surrounding text.
    • availableFonts, preferredFont, webFont, imageFont: several tests to compare the effect of these options on the fonts used (!= and == reftests).
    • styles: visual reftests with several CSS styles.
    • showMathMenu: scenario test checking whether the math menu is displayed when right clicking on a formula.
    • tooltip: scenario tests checking whether the tooltip is displayed on mactions with the specified delays. offsetX/offsetY may be difficult to test.

  • The NativeMML output processor

    • scale: same as for HTML-CSS (may be more difficult to test since it depends on the renderer)
    • showMathMath, styles: same as for HTML-CSS.

  • The MMLorHTML configuration options

    • prefer: reftests which compare the rendering against static pages with MathJax HTML-CSS enabled/disabled.

  • The MathMenu options

    • delay, helpURL, showRenderer, showContext, showFontMenu, windowSettings, styles: several scenario test to compare the way the menu is displayed with respect to the setting of each individual option.

  • The MathZoom options

    • delay, styles: same as for MathMenu.

  • The FontWarnings options

    • messageStyle, Message, HTML, removeAfter, fadeoutSteps, fadeoutTime: several reftests to see how the warnings are displayed with respect to the setting of each individual option. All these tests may be complicated by the fact that they depend on the environment (browsers, fonts...).

MathJax API

Main API

  • Startup Sequence:
    • listen all the signals and check that they match the standard list (load test).
    • test insertion of actions in the startup process with MathJax.Hub.Queue.
  • MathJax.Ajax.Require:
    • check loading of a file which exists and one which does not.
    • passing arguments (0, 1 or more)
    • passing method
    • passing objects
  • MathJax.Callback.Queue
    • test pages with actions of different duration performed in several orders. For example a long action followed by a short action.
    • pending, running, queue, Push, Process, Suspend, Resume, wait, call
  • MathJax.Callback.Signal
    • name, posted, listeners, Post, Clear, Interest, NoInterest, MessageHook, ExecuteHook
  • Modifying Math on the Page: several unit tests for Typeset, Text, Remove, SourceElement, PreProcess, Process, Update, Reprocess. check that the dynamic updates happen as expected.

Detailed API

  • The MathJax variable
    • MathJax.*: several test pages that check whether the Main MathJax Components exist
    • MathJax.version: a test to see whether this variable exists (not sure what to check here).
    • MathJax.isReady: check that it is null at the beginning and true before typesetting actions.
  • The MathJax.Hub object
    • config, Config: a test checking that the function Config modify the config values as expected.
    • processUpdateTime: may be hard to check with a visual reftest. Listen the changes of the processing box?
    • Browser: several unit tests to compare the values with the actual browser running. These later information may be determined either by javascript (in that case, the result will be quite similar to what MathJax is doing) or by the test framework controlling the execution of the tests.
    • Browser.Select: unit test to see if the actions is selected with respect to the browser running.
    • Register.PreProcessor, Register.MessageHook, Register.StartupHook, Register.LoadHook: unit tests registering callback and checking that they are called as expected.
    • getAllJax, getJaxByType, getJaxByInputType: unit tests to see if we access the expected jax element. the scope of the search can be reduced to a given element.
    • getJaxFor, isJax: unit tests checking whether an element is or is not a jax.
    • Insert: unit tests checking the effect of inserting dst in src. Take into account repeated objects and recursivity.
    • MathJax.Hub.lastError, formatError: unit test to see if this function is called when parsing a page with incorrect markup. Also see if we can customize this function.
  • The MathJax.Ajax object
    • timeout, STATUS.OK, STATUS.ERROR, loaded, loading, Require, Load, loadHook: several tests loading files and checking that they are loaded/loading or that there is an error. find a way to delay the response of the server to check the timeout? Also try to load non-existent files.
    • loadComplete, loadTimeout, loadError: unit tests to see if these functions are called as expected.
    • Styles: unit tests to check that the style object is created as wanted
    • fileURL: unit tests to verify that the substitution is done as expected
  • The MathJax.Message object
    • #MathJax_Message, #MathJax_MSIE_Frame: tests styling of the message box as well as IE bug.
    • Set, Clear, Log: unit tests of addition/removal of messages.
  • The MathJax.HTML object
    • Cookie.prefix, Cookie.expires, Cookie.Set, Cookie.Get: unit tests to see whether MathJax cookies are created and as the specified properties. Test may be prevented by browsers' security.
    • Element, addElement, TextNode, addText: unit tests verifying the creation of DOM elements.
  • The MathJax.CallBack class
    • specifying a callback: unit tests with MathJax.Callback for each possible syntax, including hook and eval string as well as setting hook, data, object properties.
    • isCallback: unit tests on callback and non-callback objects.
    • called, autoReset, reset: check if callbacks can or can not be called several times.
    • Delay, executeHooks: unit tests to verify the correct execution of callbacks.
  • The base Jax class
    • directory, extensionDir: unit tests getting/setting these values
    • name, version, directory, require, config: unit tests getting/setting these values
    • Translate, noTranslate, Register, Config, Startup, loadComplete: unit tests.
  • The MathJax.InputJax class class
    • see base Jax class
    • Remove
  • The MathJax.OutputJax class
    • see base Jax class
  • The MathJax.ElementJax class
    • see base Jax class
    • inputJax, outputJax, inputID, originalText, mimeType
    • Text, Reprocess, Remove, SourceElement
  • The MathJax Object-Oriented Programming Model
    • SUPER, constructor: create several subclasses and check their SUPER class and constructor.
    • Subclass, Augment: unit tests creating a subclass and adding methods/properties. Check that all is created as expected.
    • isa, can, has: similar tests but verifying the creation with these functions.
    • Init: unit test to check that this method is called when a class is instanciated. Also check parameter.
    • accessing the super class: unit tests for both ways to call methods from the superclass. check that the superclass method is really called.

Other tests

  • $x < y$ vs $x < y$ vs $x \lt y$ (reftests)
  • check that MathJax loads each component only when it is needed?
  • self-closing tag: <mspace></mspace> vs <mspace/> (reftests)
  • (== reftest) several test pages where the correct markup (equal to the one in the reference) is only set after a dynamic change and reprocessing.
  • (crashtests): several test pages that check whether MathJax properties/methods exist.
  • regression tests: try to convert the testcases of old bugs into unit tests.
  • check the syntax supported by HTML snippets / CSS style objects

MathJax Wiki

Clone this wiki locally