text processing utilities

Highlight documentation

Highlight manual

Content

  1. Overview
    1. Intended purpose
    2. Feature list
  2. Usage and options
    1. Quick introduction
    2. CLI options
    3. GUI options
    4. Input and output
    5. GNU source-highlight compatibility
    6. Advanced options
    7. Tips and tricks
  3. Configuration
    1. File format
    2. Regular Expressions
    3. Language definitions
    4. Theme definitions
    5. Keyword groups
    6. Plug-ins
    7. File mapping
    8. Config file search

Overview

Highlight converts sourcecode to HTML, XHTML, RTF, LaTeX, TeX, SVG, BBCode and terminal escape sequences with coloured syntax highlighting. Language definitions and colour themes are customizable.

Intended purpose

Highlight was designed to offer a flexible but easy to use syntax highlighter for several output formats. Instead of hardcoding syntax or colouring information, all relevant data is stored in configuration scripts. These scripts may be altered or enhanced with plug-in scripts.

Feature list

Usage and options

Quick introduction

The following examples show how to produce a highlighted HTML file, using an input file called main.cpp:

- Generate HTML:
  highlight -i main.cpp -o main.cpp.html
  highlight < main.cpp > main.cpp.html --syntax cpp

  You will find the HTML file and highlight.css in the working directory.
  If you use IO redirection, you must define the programming language with
  --syntax.

- Generate HTML with embedded CSS definitions and line numbers:
  highlight -i main.cpp -o main.cpp.html --include-style --line-numbers

- Generate HTML with inline CSS definitions:
  highlight -i main.cpp -o main.cpp.html --inline-css

- Generate LaTeX using "horstmann" source formatting style and "neon" colour theme:
  highlight -O latex -i main.cpp -o main.cpp.html --reformat horstmann --style neon

  The following output formats may be used with --out-format:

  html:      HTML5 (Standard)
  xhtml:     XHTML 1.1
  tex:       Plain TeX
  latex:     LaTeX
  rtf:       RTF
  odt:       OpenDocument Text (Flat XML)
  svg:       SVG
  bbcode:    BBCode
  pango:     Pango markup
  ansi:      Terminal 16 color escape codes
  xterm256:  Terminal 256 color escape codes
  truecolor: Terminal 16m color escape codes

- Customize font settings:
  highlight --syntax ada --out-format=xhtml --font-size 12 --font  Consolas,\'Courier\ New\'
  highlight --syntax ada --out-format=latex --font-size tiny --font sffamily

- Define an output directory:
  highlight -d some/target/dir/ *.cpp *.h

CLI options

The command line version of highlight offers following options:

USAGE: highlight [OPTIONS]... [FILES]...

General options:

 -B, --batch-recursive=<wc>     convert all matching files, searches subdirs
                                  (Example: -B '*.cpp')
 -D, --data-dir=<directory>     set path to data directory
     --config-file=<file>       set path to a lang or theme file
 -d, --outdir=<directory>       name of output directory
 -h, --help[=topic]             print this help or a topic description
                                  <topic> = [syntax, theme, plugin, config, test, lsp]
 -i, --input=<file>             name of single input file
 -o, --output=<file>            name of single output file
 -P, --progress                 print progress bar in batch mode
 -q, --quiet                    suppress progress info in batch mode
 -S, --syntax=<type|path>       specify type of source code or syntax file path
     --syntax-by-name=<name>    specify type of source code by given name
                                  will not read a file of this name, useful for stdin
     --syntax-supported         test if the given syntax can be loaded
 -v, --verbose                  print debug info; repeat to show more information
     --force[=syntax]           generate output if input syntax is unknown
     --list-scripts=<type>      list installed scripts
                                  <type> = [langs, themes, plugins]
     --list-cat=<categories>    filter the scripts by the given categories
                                  (example: --list-cat='source;script')
     --max-size=<size>          set maximum input file size
                                  (examples: 512M, 1G; default: 256M)
     --plug-in=<script>         execute Lua plug-in script; repeat option to
                                  execute multiple plug-ins
     --plug-in-param=<value>    set plug-in input parameter
     --print-config             print path configuration
     --print-style              print stylesheet only (see --style-outfile)
     --skip=<list>              ignore listed unknown file types
                                  (Example: --skip='bak;c~;h~')
     --stdout                   output to stdout (batch mode, --print-style)
     --validate-input           test if input is text, remove Unicode BOM
     --version                  print version and copyright information


Output formatting options:

 -O, --out-format=<format>      output file in given format
                                  <format>=[html, xhtml, latex, tex, odt, rtf,
                                  ansi, xterm256, truecolor, bbcode, pango, svg]
 -c, --style-outfile=<file>     name of style file or print to stdout, if
                                  'stdout' is given as file argument
 -e, --style-infile=<file>      to be included in style-outfile (deprecated)
                                  use a plug-in file instead
 -f, --fragment                 omit document header and footer
 -F, --reformat=<style>         reformats and indents output in given style
                                  <style> = [allman, gnu, google, horstmann,
                                  java, kr, linux, lisp, mozilla, otbs, pico,
                                  vtk, ratliff, stroustrup, webkit, whitesmith]
 -I, --include-style            include style definition in output file
 -J, --line-length=<num>        line length before wrapping (see -V, -W)
 -j, --line-number-length=<num> line number width incl. left padding (default: 5)
     --line-range=<start-end>   output only lines from number <start> to <end>
 -k, --font=<font>              set font (specific to output format)
 -K, --font-size=<num?>         set font size (specific to output format)
 -l, --line-numbers             print line numbers in output file
 -m, --line-number-start=<cnt>  start line numbering with cnt (assumes -l)
 -s, --style=<style|path>       set colour style (theme) or theme file path
 -t, --replace-tabs=<num>       replace tabs by <num> spaces
 -T, --doc-title=<title>        document title
 -u, --encoding=<enc>           set output encoding which matches input file
                                  encoding; omit encoding info if set to NONE
 -V, --wrap-simple              wrap lines after 80 (default) characters w/o
                                  indenting function parameters and statements
 -W, --wrap                     wrap lines after 80 (default) characters
     --wrap-no-numbers          omit line numbers of wrapped lines
                                  (assumes -l)
 -z, --zeroes                   pad line numbers with 0's
     --isolate                  output each syntax token separately (verbose output)
     --keep-injections          output plug-in injections in spite of -f
     --kw-case=<case>           change case of case insensitive keywords
                                  <case> =  [upper, lower, capitalize]
     --no-trailing-nl[=mode]    omit trailing newline. If mode is empty-file, omit
                                  only for empty input
     --no-version-info          omit version info comment


(X)HTML output options:

 -a, --anchors                  attach anchor to line numbers
 -y, --anchor-prefix=<str>      set anchor name prefix
 -N, --anchor-filename          use input file name as anchor prefix
 -C, --print-index              print index with hyperlinks to output files
 -n, --ordered-list             print lines as ordered list items
     --class-name=<name>        set CSS class name prefix;
                                  omit class name if set to NONE
     --inline-css               output CSS within each tag (verbose output)
     --enclose-pre              enclose fragmented output with pre tag
                                  (assumes -f)


LaTeX output options:

 -b, --babel                    disable Babel package shorthands
 -r, --replace-quotes           replace double quotes by \dq{}
     --beamer                   adapt output for the Beamer package
     --pretty-symbols           improve appearance of brackets and other symbols


RTF output options:

     --page-color               include page color attributes
 -x, --page-size=<ps>           set page size
                                  <ps> = [a3, a4, a5, b4, b5, b6, letter]
     --char-styles              include character stylesheets


SVG output options:

     --height                   set image height (units allowed)
     --width                    set image width (see --height)


Terminal escape output options (xterm256 or truecolor):

     --canvas[=width]           set background colour padding (default: 80)


Language Server options:

     --ls-profile=<server>      read LSP configuration from lsp.conf
     --ls-delay=<ms>            set server initialization delay
     --ls-exec=<bin>            set server executable name
     --ls-option=<option>       set server CLI option (can be repeated)
     --ls-hover                 execute hover requests (HTML output only)
     --ls-semantic              retrieve semantic token types (requires LSP 3.16)
     --ls-syntax=<lang>         set syntax which is understood by the server
     --ls-syntax-error          retrieve syntax error information
                                  (assumes --ls-hover or --ls-semantic)
     --ls-workspace=<dir>       set workspace directory to init. the server


If no in- or output files are specified, stdin and stdout will be used.
Reading from stdin can also be triggered using the '-' option.
Default output format: xterm256 or truecolor if appropriate, HTML otherwise.
Style definitions are stored in highlight.css (HTML, XHTML, SVG) or
highlight.sty (LaTeX, TeX) if neither -c nor -I is given.
Reformatting code (-F) will only work with C, C++, C# and Java input files.
LSP features require absolute input paths and disable reformatting (-F)
Wrapping lines with -V or -W will cause faulty highlighting of long single
line comments and directives. Using line-range might interfere with multi
line syntax elements. Use with caution.

GUI options

The Graphical User Interface offers a subset of the CLI features. It includes a dynamic preview of the output file's apperarance. Please see screenshots and screencasts.

Input and output

If no input or output file name is defined by --input and --output options, highlight will use stdin and stdout for file processing.

If no input filename is defined by --input or given at the prompt, highlight is not able to determine the language type by means of the file extension (but some scripting languages are determined by the shebang in the first input line). In this case you have to pass highlight the given langage with --syntax (this should be the file suffix of the source file in most cases).

highlight test.py
highlight < test.py --syntax py       # --syntax option necessary
cat test.py | highlight --syntax py

If there exist multiple suffixes (like C, cc, cpp, h for C++ - files), they are mapped to a language definition in $CONF_DIR/filetypes.conf.

Highlight enters the batch processing mode if multiple input files are defined or if --batch-recursive is set. In batch mode, highlight will save the generated files using the original filename, appending the extension of the chosen output type.
If files in the input directories happen to share the same name, the output files will be prefixed with their source path name.
The --out-dir option is recommended in batch mode. Use --quiet to improve performance (recommended for usage in shell scripts).

HTML, TeX, LaTeX and SVG output

The HTML, TeX, LaTeX and SVG output formats allow to reference style definition files which contain the formatting information (stylesheets).

In HTML and SVG output, this file contains CSS definitions and is saved as 'highlight.css'. In LaTeX and TeX, it contains macro definitions, and is saved as 'highlight.sty'.

Name and path of the stylesheet may be modified with --style-outfile. If the --outdir option is given, all generated output, including stylesheets, are stored in this directory.

Use --include-style to embed style information in the output documents without referencing a stylesheet.

Referenced style definitions have the advantage to share all formatting information in a single file, which affects all referencing documents.

With --style-infile you define a file to be included in the final formatting information of the document. This way you enhance or redefine the default highlight style definitions without editing generated code.
Note: Using a plug-in script is the preferred way to enhance styling.

GNU source-highlight compatibility

The command line interface is extensively harmonised with source-highlight (http://www.gnu.org/software/src-highlite/).
The following highlight options have the same meaning as in source-highlight:

 --input, --output, --help, --version, --out-format, --title, --data-dir,
 --verbose, --quiet, --ctags-file

These options were added to enhance compatibility:

 --css, --doc, --failsafe, --line-number, --line-number-ref, --no-doc, --tab,
 --output-dir, --src-lang

These switches provide a common highlighter interface for scripts, plugins etc.

Advanced options

Prevent parsing of binary input files

If highlight could be invoked with all kinds of input, you can disable parsing of binary files using --validate-input. This flag causes highlight to match the input file header with a list of magic numbers. If a binary file type is detected, highlight quits with an error message.

Highlight nested code without starting delimiter

If a file starts with an embedded code section which misses the starting delimiter, the --start-nested option will switch to the nested language mode. This can happen with LuaTeX files:

highlight luatex.tex --latex --start-nested=inc_luatex

Inc_luatex is a Lua definition with TeX line comments. Note that the nested code section has to end with the ending delimiter defined in the host language definition.

Tips and tricks

Test new configuration scripts

The option --config-file helps to test new config files before installing them. The given file must be a lang or theme file.

highlight --config-file xxx.lang --config-file yyy.theme -I

Debug language definitions

Use --verbose to display the Lua and syntax data.

→ See the wiki for more information

Remove an UTF-8 BOM

Use --validate-input to get rid of the UTF-8 byte order mark.

Configuration

File format

Configuration files are Lua scripts. These constructs are sufficient to edit the scripts:

Variable assigment:
name = value
(variables have no type, only values have)

Strings
string1="string literal with escape: \n"
string2=[[raw string without escape sequence]]

If raw string value starts with "[" or ends with "]", pad the paranthesis with
space to avoid a syntax error. Highlight will strip the string.

Comments
-- line comment
--[[ block comment ]]

Arrays
array = { first=1, second="2", 3, { 4,5 } }

Please refer to the Lua manual for more details about the Lua syntax.

Regular Expressions

Please see Regular Expressions for the supported regex constructs.

Language definitions

A language definition describes all elements of a programming language which will be highlighted by different colours and font types. Save the new file in langDefs/, using the following name convention:

<usual extension of sourcecode files>.lang

Examples: PHP -> php.lang, Java -> java.lang If there exist multiple suffixes, list them in filetypes.conf.

Keywords = { { Id, List|Regex, Group?, Priority?, Constraints? } }

  Id:          Integer, keyword group id (can be reused for several groups).
               Default themes support 4 and base16 themes 6 groups.
  List:        List, list of keywords
  Regex:       String, regular expression
  Group:       Integer, capturing group id of regular expression, defines part of
               regex which should be returned as keyword (optional; if not set,
               the match with the highest group number is returned (counts from
               left to right))
  Priority:    Integer, if not zero no more regexes will be evaluated if this
               regex matches
  Constraints: table consisting of:
               Line: Integer, limit match to line number,
               Filename: String, limit match to input file name

Regular Expressions are evaluated in the their order within Keywords. If a regex
does not appear to match, there might be a conflicting expression listed before.


Comments = { {Block, Nested?, Delimiter={Open, Close?} }

  Block:     Boolean, true if comment is a block comment
  Nested:    Boolean, true if block comments can be nested (optional)
  Delimiter: List, contains open delimiter regex (line comment) or open and close
             delimiter regexes (block comment)


Strings = { Delimiter|DelimiterPairs={Open, Close, Raw?}, Escape?, Interpolation?,
            RawPrefix?, AssertEqualLength? }

  Delimiter:         String, regular expression which describes string delimiters
  DelimiterPairs:    List, includes open and close delimiter expressions if not
                     equal, includes optional Raw flag as boolean which marks
                     delimiter pair to contain a raw string
  Escape:            String, regex of escape sequences (optional)
  Interpolation:     String, regex of interpolation sequences (optional)
  RawPrefix:         String, defines raw string indicator (optional)
  AssertEqualLength: Boolean, set true if delimiters must have the same length


PreProcessor = { Prefix, Continuation? }

  Prefix:        String, regular expression which describes open delimiter
  Continuation:  String, contains line continuation character (optional).


NestedSections = {Lang, Delimiter= {} }

  Lang:      String, name of nested language
  Delimiter: List, contains open and close delimiters of the code section


KeywordFormatHints={ { Id, Bold?, Italic?, Underline? } }
  Id:         Integer, keyword group id whose attributes should be changed
  Bold:       Boolean, font weight property
  Italic:     Boolean, font style property
  Underline:  Boolean, font decoration property

These hints may have no effect if multiple syntax types are highlighted in batch
mode without --include-style.

Description:       String, Defines syntax description

Categories:        Table, List of categories (config, source, script, etc)

Digits:            String, Regular expression which defines digits (optional)

Identifiers:       String, Regular expression which defines identifiers
                   (optional)

Operators:         String, Regular expression which defines operators

EnableIndentation: Boolean, set true if syntax may be reformatted and indented

IgnoreCase:        Boolean, set true if keyword case should be ignored


Script Environment:

The following variables are defined when a script is executed:

hl_lang_dir: current path of language definitions (use with dofile)

Identifiers: Default regex for identifiers;
Digits:      Default reegx for numbers

The following variables are integers which represent the internal highlighting
states:

HL_STANDARD
HL_STRING
HL_NUMBER
HL_LINE_COMMENT
HL_BLOCK_COMMENT
HL_ESC_SEQ
HL_PREPROC
HL_PREPROC_STRING
HL_OPERATOR
HL_LINENUMBER
HL_KEYWORD
HL_STRING_END
HL_LINE_COMMENT_END
HL_BLOCK_COMMENT_END
HL_ESC_SEQ_END
HL_PREPROC_END
HL_OPERATOR_END
HL_KEYWORD_END
HL_EMBEDDED_CODE_BEGIN
HL_EMBEDDED_CODE_END
HL_IDENTIFIER_BEGIN
HL_IDENTIFIER_END
HL_UNKNOWN
HL_REJECT

The function OnStateChange:

This function is a hook which is called if an internal state changes (e.g. from
HL_STANDARD to HL_KEYWORD if a keyword is found). It can be used to alter
the new state or to manipulate syntax elements like keyword lists.

OnStateChange(oldState, newState, token, kwGroupID, lineno, column)

  Hook Event: Highlighting parser state change
  Parameters: oldState:  old state
              newState:  intended new state
              token:     the current token which triggered the new state
              kwGroupID: if newState is HL_KEYWORD, the parameter
                         contains the keyword group ID
              lineno:    line number (since 3.50)
              column:    line column (since 3.50)
  Returns:    Correct state to continue OR HL_REJECT

Return HL_REJECT if the recognized token and state should be discarded; the
first character of token will be outputted and highlighted as "oldState".

Example:

01 Description="C and C++"
02 
03 Keywords={
04   {  Id=1,
05    List={"goto", "break", "return", "continue", "asm", "case", "default",
06          -- [..]
07         }
08   },
09   -- [..]
10 }
11 
12 Strings = {
13   Delimiter=[["|']],
14   RawPrefix="R",
15 }
16 
17 Comments = {
18    { Block=true,
19      Nested=false,
20      Delimiter = { [[\/\*]], [[\*\/]] }  },
21    { Block=false,
22      Delimiter = { [[//]] } }
23 }
24 
25 IgnoreCase=false
26 
27 PreProcessor = {
28   Prefix=[[#]],
29   Continuation="\\",
30 }
31 
32 Operators=[[\(|\)|\[|\]|\{|\}|\,|\;|\.|\:|\&|<|>|\!|\=|\/|\*|\%|\+|\-|\~]]
33 
34 EnableIndentation=true

→ See the wiki for more information

Theme definitions

Colour themes contain the formatting information of the language elements which are described in language definitions.

The files have to be stored as *.theme in themes/. Apply a style with the --style option.

Format attributes:

Attributes = {Colour, Bold?, Italic?, Underline? }

Colour:    String, defines colour in HTML hex notation ("#rrggbb")
Bold:      Boolean, true if font should be bold (optional)
Italic:    Boolean, true if font should be italic (optional)
Underline: Boolean, true if font should be underlined (optional)


Theme elements:

Description:   = String, Defines theme description
Categories     = Table, List of categories (dark, light, etc)
Default        = Attributes (Colour of unspecified text)
Canvas         = Attributes (Background colour)
Number         = Attributes (numbers)
Escape         = Attributes (escape sequences)
String         = Attributes (strings)
Interpolation  = Attributes (interpolation sequences)
PreProcessor   = Attributes (preprocessor directives)
StringPreProc  = Attributes (strings within preprocessor directives)
BlockComment   = Attributes (block comments)
LineComment    = Attributes (line comments)
LineNum        = Attributes (line numbers)
Operator       = Attributes (operators)
Hover          = Attributes (LSP Hover elements)
Error          = Attributes (LSP syntax errors)
ErrorMessage   = Attributes (LSP error descriptions)

Keywords= {
  Attributes1,
  Attributes2,
  Attributes3,
  Attributes4,
  Attributes5,
  Attributes6,
}

AttributesN: Formatting of keyword group N.

SemanticAttributesN: An array consisting of:
                     `Type`: Token Identifier of the LS protocol (V 3.16)
                     `Style`: formatting of the token

Example:

 1 
 2 Description = "vim zmrok"
 3 
 4 Categories = {"dark", "vim"}
 5 
 6 Default = { Colour="#F8F8F8" }
 7 Canvas = { Colour="#141414" }
 8 Number = { Colour="#FACE43" }
 9 Escape = { Colour="#ffa500" }
10 String = { Colour="#D9FF77" }
11 BlockComment = { Colour="#8a8a8a" }
12 PreProcessor = { Colour="#8b864e" }
13 LineNum = { Colour="#777777" }
14 StringPreProc = String
15 LineComment   = BlockComment
16 Operator      = { Colour="#888888" }
17 Interpolation = { Colour="#D084CE" }
18 
19 Keywords = {
20   { Colour="#A56A30" , Bold=true},
21   { Colour="#C7CA87" },
22   { Colour="#30a630" },
23   { Colour="#3b84cc" },
24   { Colour= "#d484aa" },
25   { Colour= "#ae84d4" },
26 }
27 
28 -- new LSP based elements:
29 
30 SemanticTokenTypes  = {
31   { Type = 'keyword', Style = Keywords[1] },
32   { Type = 'type', Style = Keywords[2] },
33   { Type = 'function', Style = Keywords[4] },
34   { Type = 'method', Style = Keywords[4] },
35   { Type = 'class', Style =  Keywords[1] },
36   { Type = 'struct', Style = Keywords[2] },
37   { Type = 'parameter', Style = Keywords[6] },
38   { Type = 'variable', Style = Keywords[5] },
39   { Type = 'number', Style = Number },
40   { Type = 'regexp', Style = String },
41   { Type = 'operator', Style = Operator },
42 }


→ See the wiki for more information

Keyword groups

You may define custom keyword groups and corresponding highlighting styles. This is useful if you want to highlight functions of a third party library, macros, constants etc.

You define a new group in two steps:

 1. Define a new group in your language definition (lang file):

    Keywords = {
      -- add your keyword description:
      {Id=5, List = {"ERROR", "DEBUG", "WARN"} }
    }

 2. Add a corresponding highlighting style in your colour theme (theme file):

    Keywords= {
      --add your keyword style as 5th item in the list:
      { Colour= "#ff0000", Bold=true },
    }

It is recommended to define keyword groups in user-defined plugin scripts to avoid editing of original highlight files.

Plug-ins

The --plug-in option receives the name of a Lua script which can override and enhance the settings of theme and language definition files. Plug-ins make it possible to apply costum settings without editing installed highlight configuration files.
See Plug-Ins for file format and examples.

File mapping

The script filetypes.conf assigns file extensions and shebang descriptions to language definitions.

Format:

FileMapping={
  {  Lang, Extensions|Shebang },
}

Lang:       String, name of language definition
Extensions: list of strings, contains file extensions referring to "Lang"
Shebang:    String, Regular expression which matches the first line of the input
            file

Edit the file gui_files/ext/fileopenfilter.conf to add new syntax types to the file open filter of the GUI.

Config file search

Configuration scripts are searched in the following directories:

1. ~/.highlight/
2. value of the environment variable HIGHLIGHT_DATADIR
3. user defined directory set with --data-dir
4. /usr/share/highlight/
5. /etc/highlight/ (location of filetypes.conf)
6. current working directory (fallback)

These subdirectories are expected to contain the corresponding scripts:

-langDefs: *.lang
-themes: *.theme
-plugins: *.lua

A custom filetypes.conf may be placed directly in ~/.highlight/. This search order enables you to enhance the installed scripts without the need to copy all preinstalled files somewhere else.
As the --plug-in option of older releases accepted absolute paths only, the given plugin scripts will be searched in the directories above only if the absolute file path access fails.

→ See the wiki for more information

Tupel7