Problem: Vim tests are slow and flaky at the same time due to reliance
on timeouts which are unreliable.
Solution: improve Vim test performance and reduce flakiness
(Yee Cheng Chin)
A lot of Vim tests currently rely on waiting a specific amount of time
before asserting a condition. This is bad because 1) it is slow, as the
timeout is hardcoded, 2) it's unreliable as a resource-starved runner
may overshoot the timeout. Also, there are a lot of builtin sleep
commands in commonly used utilities like VerifyScreenDump and WaitFor()
which leads to a lot of unnecessary idle time.
Fix these issues by doing the following:
1. Make utilities like VerifyScreenDump and WaitFor use the lowest wait
time possible (1 ms). This essentially turns it into a spin wait. On
fast machines, these will finish very quickly. For existing tests
that had an implicit reliance on the old timeouts (e.g.
VerifyScreenDump had a 50ms wait before), fix the tests to wait that
specific amount explicitly.
2. Fix tests that sleep or wait for long amounts of time to instead
explicitly use a callback mechanism to be notified when a child
terminal job has finished. This allows the test to only take as much
time as possible instead of having to hard code an unreliable
timeout.
With these fixes, tests should 1) completely quickly on fast machines,
and 2) on slow machines they will still run to completion albeit slowly.
Note that previoulsy both were not true. The hardcoded timeouts meant
that on fast machines the tests were mostly idling wasting time, whereas
on slow machines, the timeouts often were not generous enough to allow
them to run to completion.
closes: #16615
Signed-off-by: Yee Cheng Chin <ychin.git@gmail.com>
Signed-off-by: Christian Brabandt <cb@256bit.org>
Problem: tests: still preferring python2 over python3
Solution: prefer Python 3 when picking a Python program in Vim tests,
by checking for the more specific python version first and
only when python3 not found, check for the python binary
(Yee Cheng Chin)
Most OSes have Python 3 mapped to `python3` instead of `python`. Vim
tests should prioritize using that instead of Python 2 in case that is
still installed on the host system.
closes: #15986
Signed-off-by: Yee Cheng Chin <ychin.git@gmail.com>
Signed-off-by: Christian Brabandt <cb@256bit.org>
Problem: CI: tests can be improved
Solution: collect failed indent tests, raise timeout for search()
functions when using ASAN/Valgrind (Aliaksei Budavei)
ASan-instrumented Vim builds tend to run slower (x2) than
non-instrumented Vim builds and occasionally make indent
tests fail when "search*()" functions time out and give up
further execution.
Reference:
https://github.com/google/sanitizers/wiki/AddressSanitizercloses: #15974
Co-authored-by: Christian Brabandt <cb@256bit.org>
Signed-off-by: Aliaksei Budavei <0x000c70@gmail.com>
Signed-off-by: Christian Brabandt <cb@256bit.org>
Problem: tests: tests may fail on Windows environment
Solution: use shellcmdflag=/D to skip executing autorun from
the registry (Milly)
closes: #15900
Signed-off-by: Milly <milly.ca@gmail.com>
Signed-off-by: Christian Brabandt <cb@256bit.org>
Problem: Add support for using a class type of itself in an object
method (thinca)
Solution: Vim9: Add support for using a class type of itself in an
object method (Yegappan Lakshmanan)
fixes: #12369closes: #14165
Signed-off-by: Yegappan Lakshmanan <yegappan@yahoo.com>
Signed-off-by: Christian Brabandt <cb@256bit.org>
Problem: tests: test_restricted() fails
(after: v9.1.0091)
Solution: Add a space before the pipecmd and the actual Vim command to
run
Signed-off-by: Christian Brabandt <cb@256bit.org>
Problem: Syntax test fails when run with non C locale
Solution: Run syntax tests with C locale, clean up Xtestscript file,
strip environment variables from GetVimCommand()
(h-east)
closes: #14007
Co-authored-by: h-east <h.east.727@gmail.com>
Signed-off-by: h-east <h.east.727@gmail.com>
Signed-off-by: Christian Brabandt <cb@256bit.org>
Problem: test_channel may fail because of IPv6 config issues
Solution: Catch and skip the test, if getaddrinfo() fails with
'Address family not supported'
Mark tests as skipped when ch_open encounters E901
On some of the Debian build systems, the IPv6 channel tests fail because
`ch_open('[::1]:<port>', ...)` raises the error "E901: getaddrinfo() in
channel_open(): Address family for hostname not supported".
This appears to happen because getaddrinfo() can't perform the reverse
lookup for the ::1, which is a config issue on that system. Therefore,
instead of reporting a test failure, mark the test as skipped due to the
bad network config
closes: #13473
Signed-off-by: James McCoy <jamessan@jamessan.com>
Signed-off-by: Christian Brabandt <cb@256bit.org>
Problem: Cannot detect whether modifyOtherKeys is enabled.
Solution: Use XTQMODKEYS introduced by xterm version 377 to request the
modifyOtherKeys level. Update the keycode check results.
Problem: MS-Windows: build failure with MSVC.
Solution: Adjust the list of distributed files. Add hint about python.
Adjust path for reading runtime files.
Problem: When using valgrind a Vim command started by a test uses the same
log file name which gets overwritten.
Solution: Fix regexp to rename the log file.
Problem: Screenshot tests may use a different encoding. (Dominique Pelle)
Solution: Always set 'encoding' to "utf-8" when running Vim in a terminal.
(closes#4884)
Problem: When testing in the GUI may try to run gvim in a terminal.
Solution: Add the -v argument. (Yee Cheng Chin, closes#4605) Don't skip
tests that work now.
Problem: Some tests are flaky or fail on some systems.
Solution: Increase waiting time for port number. Use "cmd /c" to execute
"echo" on win32. (Ken Takata, closes#3534)
Problem: Terminal windows in a session are not properly restored.
Solution: Add "terminal" in 'sessionoptions'. When possible restore the
command running in a terminal.
Problem: Search test can be flaky.
Solution: Use WaitFor() instead of a delay. Make it possible to pass a
funcref to WaitFor() to avoid the need for global variables.
(James McCoy, closes#2282)
Problem: No error when WaitFor() gets an invalid wrong expression.
Solution: Do not ignore errors in evaluationg the expression. Fix places
where the expression was wrong.
Problem: When WaitFor() has a wrong expression it just waits a second,
which goes unnoticed. (James McCoy)
Solution: When WaitFor() times out throw an exception. Fix places where the
expression was wrong.
Problem: Test_popup_and_window_resize() does not always pass.
Solution: Do not use $VIMPROG, pass the Vim executable in the vimcmd file.
(Ozaki Kiichi, closes#2186)
Problem: Popup menu drawing problem when resizing terminal.
Solution: Redraw after resizing also when a popup menu is visible. (Ozaki
Kiichi, closes#2110)
