1 | AsciiDoc Tests |
---|
2 | ============== |
---|
3 | |
---|
4 | AsciiDoc includes 'testasciidoc', a Python script runs a set of |
---|
5 | AsciiDoc conformance tests. 'testasciidoc' runs through a list of |
---|
6 | AsciiDoc source files, generates backend outputs and then compares |
---|
7 | them to expected result files. The whole process is driven by a |
---|
8 | configuration file containing a list of user configurable test |
---|
9 | specifications. |
---|
10 | |
---|
11 | |
---|
12 | Rationale |
---|
13 | --------- |
---|
14 | When modifying AsciiDoc configuration files or source code it's very |
---|
15 | easy to introduce regression errors. 'testasciidoc' is a tool for |
---|
16 | writing many and varied test cases that can be run after changes have |
---|
17 | been made in order to verify that existing behavior has not been |
---|
18 | broken. |
---|
19 | |
---|
20 | The 'testasciidoc' 'update' command automates the (re)generation of |
---|
21 | expected test result data. Result data regeneration has to be |
---|
22 | performed after changes to test case source files or after legitimate |
---|
23 | changes to the AsciiDoc output formats -- doing this manually is |
---|
24 | prohibitively tedious. |
---|
25 | |
---|
26 | |
---|
27 | Running testasciidoc |
---|
28 | -------------------- |
---|
29 | The `testasciidoc.py` script and the default `testasciidoc.conf` |
---|
30 | configuration file are located in the AsciiDoc distribution `tests` |
---|
31 | directory. |
---|
32 | |
---|
33 | To view the command usage run: |
---|
34 | |
---|
35 | --------------------------------------------------------------------- |
---|
36 | $ python tests/testasciidoc.py |
---|
37 | Usage: testasciidoc.py [OPTIONS] COMMAND |
---|
38 | |
---|
39 | Run AsciiDoc conformance tests specified in configuration FILE. |
---|
40 | |
---|
41 | Commands: |
---|
42 | list List tests |
---|
43 | run [NUMBER] [BACKEND] Execute tests |
---|
44 | update [NUMBER] [BACKEND] Regenerate and update test data |
---|
45 | |
---|
46 | Options: |
---|
47 | -f, --conf-file=CONF_FILE |
---|
48 | Use configuration file CONF_FILE (default configuration file is |
---|
49 | testasciidoc.conf in testasciidoc.py directory) |
---|
50 | --force |
---|
51 | Update all test data overwriting existing data |
---|
52 | --------------------------------------------------------------------- |
---|
53 | |
---|
54 | To view the list of tests in the default `testasciidoc.conf` |
---|
55 | configuration file run the 'list' command: |
---|
56 | |
---|
57 | --------------------------------------------------------------------- |
---|
58 | $ python tests/testasciidoc.py list |
---|
59 | 1: Test cases |
---|
60 | 2: Tables |
---|
61 | 3: Source highlighter |
---|
62 | 4: Example article |
---|
63 | 5: Example book |
---|
64 | 6: Example multi-part book |
---|
65 | 7: !User Guide |
---|
66 | --------------------------------------------------------------------- |
---|
67 | |
---|
68 | The '!' prefix signals that a test is currently disabled. |
---|
69 | |
---|
70 | Before running the tests you will need to regenerate the expected |
---|
71 | outputs by running the 'update' command: |
---|
72 | |
---|
73 | --------------------------------------------------------------------- |
---|
74 | $ python tests/testasciidoc.py update |
---|
75 | WRITING: tests/data/testcases-html4.html |
---|
76 | WRITING: tests/data/testcases-xhtml11.html |
---|
77 | WRITING: tests/data/testcases-docbook.xml |
---|
78 | : |
---|
79 | WRITING: tests/data/book-multi-docbook.xml |
---|
80 | --------------------------------------------------------------------- |
---|
81 | |
---|
82 | Now you can run the tests: |
---|
83 | |
---|
84 | --------------------------------------------------------------------- |
---|
85 | $ python tests/testasciidoc.py run |
---|
86 | 1: Test cases |
---|
87 | SOURCE: asciidoc: tests/data/testcases.txt |
---|
88 | PASSED: html4: tests/data/testcases-html4.html |
---|
89 | PASSED: xhtml11: tests/data/testcases-xhtml11.html |
---|
90 | PASSED: docbook: tests/data/testcases-docbook.xml |
---|
91 | : |
---|
92 | 6: Example multi-part book |
---|
93 | SOURCE: asciidoc: doc/book-multi.txt |
---|
94 | PASSED: html4: tests/data/book-multi-html4.html |
---|
95 | PASSED: xhtml11: tests/data/book-multi-xhtml11.html |
---|
96 | PASSED: docbook: tests/data/book-multi-docbook.xml |
---|
97 | |
---|
98 | TOTAL PASSED: 18 |
---|
99 | --------------------------------------------------------------------- |
---|
100 | |
---|
101 | [NOTE] |
---|
102 | ===================================================================== |
---|
103 | - 'testasciidoc' requires AsciiDoc is located in your environment |
---|
104 | search path, if not you'll need to set the `ASCIIDOC_PY` environment |
---|
105 | variable to point to `asciidoc.py`. |
---|
106 | - If you don't have GNU source-highlight installed you should disable |
---|
107 | the 'Tables' and 'Source highlighter' tests in the |
---|
108 | `tests/testasciidoc.conf` configuration file. |
---|
109 | ===================================================================== |
---|
110 | |
---|
111 | |
---|
112 | testasciidoc commands |
---|
113 | --------------------- |
---|
114 | 'list':: |
---|
115 | List test numbers and titles. A '!' title prefix signals that a |
---|
116 | test is currently disabled. |
---|
117 | |
---|
118 | 'run':: |
---|
119 | Read and execute tests from the test configuration file. A test |
---|
120 | specifies AsciiDoc test case source file and command options. The |
---|
121 | test compares generated outputs to expected outputs and any |
---|
122 | differences displayed as a diff. You can run selected tests by |
---|
123 | specifying the test number and/or backend after the `run` command. |
---|
124 | Examples: |
---|
125 | |
---|
126 | python tests/testasciidoc.py run |
---|
127 | python tests/testasciidoc.py run 3 |
---|
128 | python tests/testasciidoc.py run html4 |
---|
129 | python tests/testasciidoc.py run 3 html4 |
---|
130 | |
---|
131 | 'update':: |
---|
132 | Generates and updates missing and out of date test output data |
---|
133 | files, this eliminates one of the most time consuming aspect of test |
---|
134 | management. Use the `--force` option to force updates. |
---|
135 | Examples: |
---|
136 | |
---|
137 | python tests/testasciidoc.py update |
---|
138 | python tests/testasciidoc.py --force update 4 |
---|
139 | |
---|
140 | NOTE: You can run or update disabled tests by explicitly specifying |
---|
141 | the test number. |
---|
142 | |
---|
143 | |
---|
144 | Test configuration file |
---|
145 | ----------------------- |
---|
146 | The tests configuration file format consists of one or more test specs |
---|
147 | separated by a line of one or more percent characters. Each test spec |
---|
148 | consists of an optional test title and description followed by one or |
---|
149 | more optional directives (a directive starts with a percent |
---|
150 | character). A 'directive' consists begins with a line containing a '%' |
---|
151 | character followed by a directive name followed by zero or more lines |
---|
152 | of directive data. |
---|
153 | |
---|
154 | Test spec format |
---|
155 | ~~~~~~~~~~~~~~~~ |
---|
156 | --------------------------------------------------------------------- |
---|
157 | Optional test title |
---|
158 | Optional test description... |
---|
159 | |
---|
160 | % name |
---|
161 | Optional base output file name. Defaults to base source file name. |
---|
162 | |
---|
163 | % source |
---|
164 | AsciiDoc source file name. |
---|
165 | |
---|
166 | % backends |
---|
167 | Optional list of backends to be tested(default is all backends). |
---|
168 | % options |
---|
169 | Optional list of command-line option tuples. |
---|
170 | |
---|
171 | % attributes |
---|
172 | Optional dictionary of attribute values. |
---|
173 | |
---|
174 | --------------------------------------------------------------------- |
---|
175 | |
---|
176 | Example test spec: |
---|
177 | |
---|
178 | --------------------------------------------------------------------- |
---|
179 | Example book |
---|
180 | |
---|
181 | % options |
---|
182 | ['--section-numbers',('--doctype','book')] |
---|
183 | |
---|
184 | % attributes |
---|
185 | # Exclude date from document footer. |
---|
186 | {'docdate':None} |
---|
187 | |
---|
188 | % source |
---|
189 | ../doc/book.txt |
---|
190 | --------------------------------------------------------------------- |
---|
191 | |
---|
192 | TIP: Take a look at the default `tests/testasciidoc.conf` |
---|
193 | configuration file that comes with AsciiDoc. |
---|
194 | |
---|
195 | - Tests can be disabled by prefixing the test title with an |
---|
196 | exclamation '!' character. |
---|
197 | - All relative file names are relative to the configuration file |
---|
198 | directory. |
---|
199 | - Multiple tests must by separated by a `%` separator line (one or |
---|
200 | more percent characters). |
---|
201 | - Lines starting with a percent character specify a test 'directive' |
---|
202 | and may be followed by zero or more lines of directive data. |
---|
203 | - Directive data lines cannot start with a percent character. |
---|
204 | - Lines starting with a `#` hash character are ignored. |
---|
205 | - The 'source' directive data is a single line containing the |
---|
206 | AsciiDoc source file name. |
---|
207 | - The 'options' directive data is a Python list of `(name,value)` |
---|
208 | tuples specifying AsciiDoc command-line options. A string item is |
---|
209 | equivalent to a `(name,None)` tuple. |
---|
210 | - The 'attributes' directive data specifies a Python dictionary |
---|
211 | containing AsciiDoc attributes to be passed to AsciiDoc. |
---|
212 | |
---|
213 | globals directive |
---|
214 | ~~~~~~~~~~~~~~~~~ |
---|
215 | An optional 'globals' directive can precede all test specs, the |
---|
216 | globals directive data is a Python dictionary containing global |
---|
217 | values. Currently the only global is 'datadir', the directory |
---|
218 | containing expected output files (defaults to configuration file |
---|
219 | directory). For example: |
---|
220 | |
---|
221 | --------------------------------------------------------------------- |
---|
222 | % globals |
---|
223 | {'datadir': 'data'} |
---|
224 | --------------------------------------------------------------------- |
---|
225 | |
---|
226 | Expected output test data files are stored in the 'datadir' and are |
---|
227 | named after the corresponding AsciiDoc input source file. For example |
---|
228 | if the AsciiDoc file name is `article.txt` then the corresponding |
---|
229 | backend output files will be `article-html4.html`, |
---|
230 | `article-xhtml11.html`, `article-docbook.xml` (stored in the 'datadir' |
---|
231 | directory). |
---|