Problem: tests: cannot run single syntax tests
Solution: Support running a subset of syntax tests
(Aliaksei Budavei)
Two methods of assembling a subset of test filenames for
selection are provided:
* Filename and filetype Make targets will be generated, and
multiple such targets can be passed before the mandated
trailing "test" target, e.g. "make html markdown test".
* Filenames and their parts can be specified as a regular
expression that is assigned to a "VIM_SYNTAX_TEST_FILTER"
environment variable, and used with the test Make target,
e.g. "VIM_SYNTAX_TEST_FILTER=html\\\|markdown make test".
(This variable will be ignored and the whole suite will be
run when Make is GNU Make and a parent Makefile is used.)
Methods can be used alone or together, with the Make targets
having the higher precedence. Neither method will influence
the order of test execution.
closes: #15670
Signed-off-by: Aliaksei Budavei <0x000c70@gmail.com>
Signed-off-by: Christian Brabandt <cb@256bit.org>
Problem: tests: Analyzing failed screendumps is hard
Solution: Facilitate the viewing of rendered screendumps under src/
add some documentation on how to use the viewdumps.vim
script (Aliaksei Budavei)
With the submitted "viewdumps.vim" script, a few manual
steps in typical workflows (see below) can be automated.
The updated "README.txt" contains additional information.
============================================================
Reviewing LOCAL failed screendump tests can be arranged as
follows:
1) Run tests and generate screendumps:
------------------------------------------------------------
cd /path/to/fork/src/testdir
make
------------------------------------------------------------
2) Examine the screendumps from the "failed" directory:
------------------------------------------------------------
../vim -u NONE -S viewdumps.vim
------------------------------------------------------------
============================================================
Reviewing UPLOADED failed screendump tests can be arranged
as follows (it can be further locally scripted):
1) Fetch an artifact with failed screendumps from
"github.com/vim/vim/actions/runs/A_ID/artifacts/B_ID".
2) Extract the archived files:
------------------------------------------------------------
unzip /tmp/failed-tests.zip -d /tmp
------------------------------------------------------------
3) Set up the "dumps" directory. Create a symlink to
"/path/to/fork/dirs/dumps" in the extracted directories so
that term_dumpdiff() can be used. (The lookup algorithm
resolves "dumps" for every loaded filename. So, with
"/tmp/src/testdir/failed/*.dump" files passed as script
arguments, the algorithm will make the files in
"/tmp/src/testdir/dumps" queried.)
------------------------------------------------------------
cd /path/to/fork
ln -s $(pwd)/src/testdir/dumps /tmp/src/testdir/dumps
------------------------------------------------------------
4) Examine the extracted screendumps:
------------------------------------------------------------
./src/vim -u NONE -S src/testdir/viewdumps.vim \
/tmp/src/testdir/failed/*.dump
------------------------------------------------------------
5) Clean up:
------------------------------------------------------------
unlink /tmp/src/testdir/dumps
rm -rf /tmp/src
------------------------------------------------------------
============================================================
Reviewing SUBMITTED FOR PULL REQUEST screendump tests can be
arranged as follows (it can be further locally scripted):
1) List the fetched changeset and write the changed "dumps"
filenames to "/tmp/filelist":
------------------------------------------------------------
cd /path/to/fork
git switch prs/1234
git diff-index --relative=src/testdir/dumps/ \
--name-only prs/1234~1 > /tmp/filelist
------------------------------------------------------------
2) Reconcile relative filepaths, and copy next-to-be-updated
"dumps" files in the "failed" directory (note the missing
new screendumps, if any):
------------------------------------------------------------
git switch master
cd src/testdir/dumps
test -d ../failed || mkdir ../failed
cp -t ../failed $(cat /tmp/filelist)
------------------------------------------------------------
3) Remember about the introduced INVERTED relation between
"dumps" and "failed", i.e. the files to be committed are in
"dumps" already and their previous versions are in "failed";
therefore, copy the missing new screendumps from "dumps" to
"failed" (otherwise these won't be shown):
------------------------------------------------------------
git switch prs/1234
cp -t ../failed foo_10.dump foo_11.dump foo_12.dump
------------------------------------------------------------
4) Examine the screendumps from the "failed" directory (new
screendumps will be shown with no difference between their
versions):
------------------------------------------------------------
cd ..
../vim -u NONE -S viewdumps.vim
------------------------------------------------------------
closes: #15515
Signed-off-by: Aliaksei Budavei <0x000c70@gmail.com>
Signed-off-by: Christian Brabandt <cb@256bit.org>
With the submitted "viewdumps.vim" script, a few manual
steps in typical workflows (see below) can be automated.
The updated "README.txt" contains additional information.
============================================================
Reviewing LOCAL failed syntax tests can be arranged as
follows:
1) Run tests and generate screendumps:
------------------------------------------------------------
cd /path/to/fork/runtime/syntax
make clean test
------------------------------------------------------------
2) Examine the screendumps from the "failed" directory:
------------------------------------------------------------
../../src/vim --clean -S testdir/viewdumps.vim
------------------------------------------------------------
============================================================
Reviewing UPLOADED failed syntax tests can be arranged as
follows (it can be further locally scripted):
1) Fetch an artifact with failed screendumps from
"github.com/vim/vim/actions/runs/A_ID/artifacts/B_ID".
2) Extract the archived files:
------------------------------------------------------------
unzip /tmp/failed-tests.zip -d /tmp
------------------------------------------------------------
3) Set up the "dumps" directory. Create a symlink to
"/path/to/fork/dirs/dumps" in the extracted directories so
that term_dumpdiff() can be used. (The lookup algorithm
resolves "dumps" for every loaded filename. So, with
"/tmp/runtime/syntax/testdir/failed/*.dump" files passed
as script arguments, the algorithm will make the files in
"/tmp/runtime/syntax/testdir/dumps" queried.)
------------------------------------------------------------
cd /path/to/fork
ln -s $(pwd)/runtime/syntax/testdir/dumps \
/tmp/runtime/syntax/testdir/dumps
------------------------------------------------------------
4) Examine the extracted screendumps:
------------------------------------------------------------
./src/vim --clean -S runtime/syntax/testdir/viewdumps.vim \
/tmp/runtime/syntax/testdir/failed/*.dump
------------------------------------------------------------
5) Clean up:
------------------------------------------------------------
unlink /tmp/runtime/syntax/testdir/dumps
rm -rf /tmp/runtime
------------------------------------------------------------
============================================================
Reviewing SUBMITTED FOR PULL REQUEST syntax tests can be
arranged as follows (it can be further locally scripted):
1) List the fetched changeset and write the changed "dumps"
filenames to "/tmp/filelist":
------------------------------------------------------------
cd /path/to/fork
git switch prs/1234
git diff-index --relative=runtime/syntax/testdir/dumps/ \
--name-only prs/1234~1 > /tmp/filelist
------------------------------------------------------------
2) Reconcile relative filepaths, and copy next-to-be-updated
"dumps" files in the "failed" directory (note the missing
new screendumps, if any):
------------------------------------------------------------
git switch master
cd runtime/syntax/testdir/dumps
cp -t ../failed $(cat /tmp/filelist)
------------------------------------------------------------
3) Remember about the introduced INVERTED relation between
"dumps" and "failed", i.e. the files to be committed are in
"dumps" already and their previous versions are in "failed";
therefore, copy the missing new screendumps from "dumps" to
"failed" (otherwise these won't be shown):
------------------------------------------------------------
git switch prs/1234
cp -t ../failed foo_10.dump foo_11.dump foo_12.dump
------------------------------------------------------------
4) Examine the screendumps from the "failed" directory (new
screendumps will be shown with no difference between their
versions):
------------------------------------------------------------
cd ..
../../../src/vim --clean -S viewdumps.vim
------------------------------------------------------------
closes: #15476
Signed-off-by: Aliaksei Budavei <0x000c70@gmail.com>
Signed-off-by: Christian Brabandt <cb@256bit.org>
- Remove the code handling their generation.
- Remove the code written for the shell filetype to clear up
the buffer screen before such a file can be generated.
- Remove all but "yaml.yaml_99.dump" of such generated files
(since there is no published "input/yaml.yaml.yaml", let
its maintainer regenerate screen dump files and remove
redundant ones).
The new algorithm turns a "*_99.dump" file obsolete because
all lines written in such a file will already be present in
lower-numbered "*.dump" files.
Signed-off-by: Aliaksei Budavei <0x000c70@gmail.com>
Signed-off-by: Christian Brabandt <cb@256bit.org>
The current implementation falls short for syntax test files
on two accounts:
1. With folded lines -- some lines before folded lines are
unnecessarily repeated in generated dump files because
closed folded lines are always treated as opened for the
cursor to move _in_ instead of to move _over_ them.
2. With wrapped lines (longer than 75 columns) -- some lines
are omitted in generated dump files because calculations
for the cursor progress and its movement commands only
refer to file lines and not their layout within a 20x75
buffer (less &cmdheight).
As an alternative, we abandon deterministic (and inaccurate
at times) calculations for the cursor progress and, instead,
advance the cursor by as much as before for a single dump
file, but now rely on marking the last visible line and
additional movement to position lines at desired offsets,
carefully preserving compatibility for the &scrolloff and
&ruler values inherited from defaults.vim. The parent Vim
process will keep track of progress through a syntax test
file made by its child process ("terminal") by reading the
rightmost end of the ruler line from the terminal buffer,
looking for " All " or " Bot " for its cue to finish dump
file generation.
With these changes applied, the lossless line length limit
will be raised from 75 to 1425 (for a 19x75 view) columns.
Also, prefer "lastline" to "truncate" for &display; hiding
the content of any last _long_ line in a view goes against
the purpose of syntax file testing -- all lines should be
recorded.
related: #15150fixes: #14245
Signed-off-by: Aliaksei Budavei <0x000c70@gmail.com>
Signed-off-by: Christian Brabandt <cb@256bit.org>
Not all programming languages support comments; without such
support, the TEST_SETUP functionality that offers embeddable
Vim Ex commands may no longer be applicable.
We can achieve similar functionality by storing Ex commands
in a Vim file with its basename matching the basename of the
test file, and having this Vim file sourced, and then having
the test file loaded and tested. When such a Vim file would
be used for a language that has comments and whose matching
test file has embedded TEST_SETUP lines, we will accommodate
it by letting the TEST_SETUP lines augment and/or overrule
sourced configuration.
Further details can be found in the discussion thread at
https://github.com/vim/vim/discussions/14117.
related: #14215
Signed-off-by: Aliaksei Budavei <0x000c70@gmail.com>
Signed-off-by: Christian Brabandt <cb@256bit.org>
Currently, the very same syntax file for which a test can
be written is the only place where global variables can be
defined so that the file parts guarded with such variables
can be read during screen dump generation. This approach
would lead to littering the syntax file with test-related
queries.
Instead, we could borrow the idea of comment-based mechanism
for test setup from the indent test runner. With it, the
first 20 lines of each test file would be ALWAYS scanned in
search of the TEST_SETUP markers and, when found, the part
between the end of the marker and the end of the line would
be treated as a Vim Ex command.
Note that with these changes, runtime/defaults.vim is no
longer sourced for screen dump generation; however, some of
its functionality is reintroduced.
Further details can be found in the discussion thread at
https://github.com/vim/vim/discussions/14117.
related: #14215
Signed-off-by: Aliaksei Budavei <0x000c70@gmail.com>
Signed-off-by: Christian Brabandt <cb@256bit.org>
