Problem: Vim tests are slow and flaky at the same time due to reliance
on timeouts which are unreliable.
Solution: improve Vim test performance and reduce flakiness
(Yee Cheng Chin)
A lot of Vim tests currently rely on waiting a specific amount of time
before asserting a condition. This is bad because 1) it is slow, as the
timeout is hardcoded, 2) it's unreliable as a resource-starved runner
may overshoot the timeout. Also, there are a lot of builtin sleep
commands in commonly used utilities like VerifyScreenDump and WaitFor()
which leads to a lot of unnecessary idle time.
Fix these issues by doing the following:
1. Make utilities like VerifyScreenDump and WaitFor use the lowest wait
time possible (1 ms). This essentially turns it into a spin wait. On
fast machines, these will finish very quickly. For existing tests
that had an implicit reliance on the old timeouts (e.g.
VerifyScreenDump had a 50ms wait before), fix the tests to wait that
specific amount explicitly.
2. Fix tests that sleep or wait for long amounts of time to instead
explicitly use a callback mechanism to be notified when a child
terminal job has finished. This allows the test to only take as much
time as possible instead of having to hard code an unreliable
timeout.
With these fixes, tests should 1) completely quickly on fast machines,
and 2) on slow machines they will still run to completion albeit slowly.
Note that previoulsy both were not true. The hardcoded timeouts meant
that on fast machines the tests were mostly idling wasting time, whereas
on slow machines, the timeouts often were not generous enough to allow
them to run to completion.
closes: #16615
Signed-off-by: Yee Cheng Chin <ychin.git@gmail.com>
Signed-off-by: Christian Brabandt <cb@256bit.org>
Problem: tests: timeout might be a bit too small
Solution: increase the test timeout from 30 to 45 seconds
related: #16599
Signed-off-by: Christian Brabandt <cb@256bit.org>
Problem: GVim: default font size a bit too small
Solution: increase guifont size to 12 pt on GTK builds
of gVim (matveyt).
fixes: #16172closes: #16178
Signed-off-by: matveyt <matthewtarasov@yandex.ru>
Signed-off-by: Christian Brabandt <cb@256bit.org>
Problem: CI: recent ASAN changes do not work for indent tests
Solution: Move code to runtime/indent/testdir/runtest.vim
(Aliaksei Budavei)
closes: #15981
Signed-off-by: Aliaksei Budavei <0x000c70@gmail.com>
Signed-off-by: Christian Brabandt <cb@256bit.org>
Problem: CI: tests can be improved
Solution: collect failed indent tests, raise timeout for search()
functions when using ASAN/Valgrind (Aliaksei Budavei)
ASan-instrumented Vim builds tend to run slower (x2) than
non-instrumented Vim builds and occasionally make indent
tests fail when "search*()" functions time out and give up
further execution.
Reference:
https://github.com/google/sanitizers/wiki/AddressSanitizercloses: #15974
Co-authored-by: Christian Brabandt <cb@256bit.org>
Signed-off-by: Aliaksei Budavei <0x000c70@gmail.com>
Signed-off-by: Christian Brabandt <cb@256bit.org>
Problem: tests: Mac OS tests are too flaky
Solution: Increase max test timeout to 25 minutes,
allow up to 10 retries on Mac OS runners,
refactor runtest.vim (Milly).
closes: #15940
Co-authored-by: K.Takata <kentkt@csc.jp>
Signed-off-by: Milly <milly.ca@gmail.com>
Signed-off-by: Christian Brabandt <cb@256bit.org>
Problem: tests: messages files contains ANSI escape sequences
Solution: filter those out in runtests.vim
Also, since we are setting $LC_ALL, we don't need to set $LANG and
$LANGUAGE since those are overridden by $LC_ALL anyhow.
closes: #15788
Signed-off-by: Christian Brabandt <cb@256bit.org>
Problem: tests: avoid error when no swap files exist
Solution: use unlet! so that no error message is reported
in case the variable does not exists
When s:GetSwapFileList() does not find any swapfiles, it will return an
empty list []. This means, that the variable 'name' will not be
declared, cause the following unlet command to fail and causing a 1 sec
delay on running the tests.
So let's instead use the :unlet! command which simply skips reporting an
error when the variable given as parameter does not exists.
closes: #13396
Signed-off-by: Christian Brabandt <cb@256bit.org>
Co-authored-by: zeertzjq <zeertzjq@outlook.com>
Problem: tests: checking for swap files takes time
Solution: don't check for swap files when test has been skipped
Check for swap files takes a considerable about of time, so don't do
that for skipped tests to avoid wasting time.
closes: #13371
Signed-off-by: Christian Brabandt <cb@256bit.org>
Co-authored-by: zeertzjq <zeertzjq@outlook.com>
Problem: Tests may leave leftover files around
Solution: Clean up tests and remove files
There were a few failures in 'linux (huge, gcc, testgui, true, true)'
e.g. here: https://github.com/vim/vim/actions/runs/5497376153/jobs/10018060156
,----
| Error detected while processing command line..script /home/runner/work/vim/vim/src/testdir/runtest.vim[585]..function RunTheTest[54]..Test_lvimgrep_crash[16]..TestTimeout[12]..VimLeavePre Autocommands for "*"..function EarlyExit[7]..FinishTesting:
| line 70:
| E445: Other window contains changes
| E937: Attempt to delete a buffer that is in use: Xtest_stable_xxd.txt
| E937: Attempt to delete a buffer that is in use: Xtest_stable_xxd.txt
| E937: Attempt to delete a buffer that is in use: Xtest_stable_xxd.txtmalloc(): unsorted double linked list corrupted
`----
Which is puzzling, because the Xtest_stable_xxd file should have been
long gone after test_crypt.vim is run (and definitely no longer be
staying around in test_quickfix.vim).
So try to clean up properly after a test script is run, just in case any
X<file> is still around. During testing, a found a few leftover files,
which I also fixed in the relevant test-file.
Unfortunately, the test workflow 'linux (huge, gcc, testgui, true,
true)' now seems to fail with 'E1230: Encryption: sodium_mlock()' in
test_crypt.vim. Hopefully this is only temporary.
Problem: Errors from the codestyle test are a bit confusing.
Solution: Use assert_report() with a clearer message. Avoid a warning for
an existing swap file.
Problem: There is no way to get a list of swap file names.
Solution: Add the swapfilelist() function. Use it in the test script to
clean up. Remove deleting individual swap files.
Problem: Reproducing memory access errors can be difficult.
Solution: When testing, copy each line to allocated memory, so that valgrind
can detect accessing memory before and/or after it. Fix uncovered
problems.
Problem: Interrupt not caught in test.
Solution: Consider an exception thrown in the current try/catch when got_int
is set. Also catch early exit when not using try/catch.
Problem: Vim9: still allows :let for declarations.
Solution: Make the default for v:disallow_let one. It can still be set to
zero to allow for using :let.
Problem: Not all tests are executed on Github Actions.
Solution: Copy "src" to "src2" earlier. Recognize "src2" in a couple more
places. Add two tests to the list of flaky tests. (Ken Takata,
closes#6798)
