Problem: Vim tests are slow and flaky at the same time due to reliance
on timeouts which are unreliable.
Solution: improve Vim test performance and reduce flakiness
(Yee Cheng Chin)
A lot of Vim tests currently rely on waiting a specific amount of time
before asserting a condition. This is bad because 1) it is slow, as the
timeout is hardcoded, 2) it's unreliable as a resource-starved runner
may overshoot the timeout. Also, there are a lot of builtin sleep
commands in commonly used utilities like VerifyScreenDump and WaitFor()
which leads to a lot of unnecessary idle time.
Fix these issues by doing the following:
1. Make utilities like VerifyScreenDump and WaitFor use the lowest wait
time possible (1 ms). This essentially turns it into a spin wait. On
fast machines, these will finish very quickly. For existing tests
that had an implicit reliance on the old timeouts (e.g.
VerifyScreenDump had a 50ms wait before), fix the tests to wait that
specific amount explicitly.
2. Fix tests that sleep or wait for long amounts of time to instead
explicitly use a callback mechanism to be notified when a child
terminal job has finished. This allows the test to only take as much
time as possible instead of having to hard code an unreliable
timeout.
With these fixes, tests should 1) completely quickly on fast machines,
and 2) on slow machines they will still run to completion albeit slowly.
Note that previoulsy both were not true. The hardcoded timeouts meant
that on fast machines the tests were mostly idling wasting time, whereas
on slow machines, the timeouts often were not generous enough to allow
them to run to completion.
closes: #16615
Signed-off-by: Yee Cheng Chin <ychin.git@gmail.com>
Signed-off-by: Christian Brabandt <cb@256bit.org>
Problem: tests: no support for env variables when running Vim in
terminal
Solution: support the "env" argument in RunVimInTerminal(),
close swapfiles properly in test_termcodes,
use CheckFeature in test_termencoding
Signed-off-by: Christian Brabandt <cb@256bit.org>
Problem: tests: tests may fail on Windows environment
Solution: use shellcmdflag=/D to skip executing autorun from
the registry (Milly)
closes: #15900
Signed-off-by: Milly <milly.ca@gmail.com>
Signed-off-by: Christian Brabandt <cb@256bit.org>
Problem: xxd: no color support
Solution: Add color support using xxd -R
Add some basic color support for xxd
The hex-value and value are both colored with the same color depending
on the hex-value, e.g.:
0x00 = white
0xff = blue
printable = green
non-printable = red
tabs and linebreaks = yellow
Each character needs 11 more bytes to contain color. (Same color in a
row could contain only one overhead but the logic how xxd creates colums
must be then changed.) Size of colored output is increased by factor of
~6. Also grepping the output will break when colors is used.
Flag for color is "-R", because less uses "-R".
Color uses parameters auto,always,never same as less and grep (among
others).
E.g.
xxd -R always $FILE | less -R
Add some screen-tests (that currently on work on linux) to verify the
feature works as expected.
closes: #12131
Signed-off-by: Christian Brabandt <cb@256bit.org>
Co-authored-by: Aapo Rantalainen <aapo.rantalainen@gmail.com>
Problem: Terminal test for current directory not used on FreeBSD.
Solution: Make it work on FreeBSD. (Ozaki Kiichi, closes#9516) Add
TermWait() inside Run_shell_in_terminal() as a generic solution.
Problem: Test sometimes fails waiting for shell in terminal.
Solution: Use WaitForAssert() so we can see the actual job status. Use
Run_shell_in_terminal().
Problem: Some terminals misinterpret the code for getting cursor style.
Solution: Send a sequence to the terminal and check the result. (IWAMOTO
Kouichi, closes#2126) Merged with current code.
Problem: Tests using term_wait() can still be flaky.
Solution: Increase the wait time when rerunning a test. (James McCoy,
closes#5899) Halve the initial times to make tests run faster
when there is no rerun.
Problem: When testing in the GUI may try to run gvim in a terminal.
Solution: Add the -v argument. (Yee Cheng Chin, closes#4605) Don't skip
tests that work now.
