Yee Cheng Chin
e70587dbdb
patch 9.1.1110: Vim tests are slow and flaky
...
Problem: Vim tests are slow and flaky at the same time due to reliance
on timeouts which are unreliable.
Solution: improve Vim test performance and reduce flakiness
(Yee Cheng Chin)
A lot of Vim tests currently rely on waiting a specific amount of time
before asserting a condition. This is bad because 1) it is slow, as the
timeout is hardcoded, 2) it's unreliable as a resource-starved runner
may overshoot the timeout. Also, there are a lot of builtin sleep
commands in commonly used utilities like VerifyScreenDump and WaitFor()
which leads to a lot of unnecessary idle time.
Fix these issues by doing the following:
1. Make utilities like VerifyScreenDump and WaitFor use the lowest wait
time possible (1 ms). This essentially turns it into a spin wait. On
fast machines, these will finish very quickly. For existing tests
that had an implicit reliance on the old timeouts (e.g.
VerifyScreenDump had a 50ms wait before), fix the tests to wait that
specific amount explicitly.
2. Fix tests that sleep or wait for long amounts of time to instead
explicitly use a callback mechanism to be notified when a child
terminal job has finished. This allows the test to only take as much
time as possible instead of having to hard code an unreliable
timeout.
With these fixes, tests should 1) completely quickly on fast machines,
and 2) on slow machines they will still run to completion albeit slowly.
Note that previoulsy both were not true. The hardcoded timeouts meant
that on fast machines the tests were mostly idling wasting time, whereas
on slow machines, the timeouts often were not generous enough to allow
them to run to completion.
closes : #16615
Signed-off-by: Yee Cheng Chin <ychin.git@gmail.com>
Signed-off-by: Christian Brabandt <cb@256bit.org>
2025-02-13 20:55:45 +01:00
Bram Moolenaar
29b4c513b1
patch 9.0.1592: not all timer tests are marked as flaky
...
Problem: Not all timer tests are marked as flaky.
Solution: Set the flaky flag for all timer tests. (closes #12355 )
2023-05-30 15:34:50 +01:00
Bram Moolenaar
94722c5107
patch 9.0.1257: code style is not check in test scripts
...
Problem: Code style is not check in test scripts.
Solution: Add basic code style check for test files.
2023-01-28 19:19:03 +00:00
Bram Moolenaar
c4860bdd28
patch 9.0.0767: too many delete() calls in tests
...
Problem: Too many delete() calls in tests.
Solution: Use deferred delete where possible.
2022-10-15 20:52:26 +01:00
Bram Moolenaar
cf3d0eaf47
patch 9.0.0682: crash when popup with deleted timer is closed
...
Problem: Crash when popup with deleted timer is closed. (Igbanam
Ogbuluijah)
Solution: Check the timer still exists. (closes #11301 )
2022-10-07 11:20:29 +01:00
Bram Moolenaar
865bf2ed30
patch 9.0.0578: one timer test fails on Mac M1
...
Problem: One timer test fails on Mac M1.
Solution: Skip the test on Mac M1.
2022-09-24 17:44:22 +01:00
Bram Moolenaar
140f6d0eda
patch 9.0.0574: timer garbage collect test hangs on Mac M1
...
Problem: Timer garbage collect test hangs on Mac M1.
Solution: Properly check for Mac M1 and skip the test.
2022-09-24 14:49:07 +01:00
Bram Moolenaar
0056ca7137
patch 9.0.0564: a few tests keep failing on MacOS M1
...
Problem: A few tests keep failing on MacOS M1.
Solution: Add a test check CheckNotMacM1. Fix timer tests.
2022-09-23 21:26:39 +01:00
Bram Moolenaar
4ecf16bbf9
patch 9.0.0559: timer test may get stuck at hit-enter prompt
...
Problem: Timer test may get stuck at hit-enter prompt.
Solution: Feed some more characters.
2022-09-23 18:22:21 +01:00
Yegappan Lakshmanan
8deb2b30c7
patch 9.0.0359: error message for wrong argument type is not specific
...
Problem: Error message for wrong argument type is not specific.
Solution: Include more information in the error. (Yegappan Lakshmanan,
closes #11037 )
2022-09-02 15:15:27 +01:00
Yegappan Lakshmanan
04c4c5746e
patch 9.0.0335: checks for Dictionary argument often give a vague error
...
Problem: Checks for Dictionary argument often give a vague error message.
Solution: Give a useful error message. (Yegappan Lakshmanan, closes #11009 )
2022-08-30 19:48:24 +01:00
zeertzjq
bb404f5ad5
patch 9.0.0059: test file has wrong name
...
Problem: Test file has wrong name.
Solution: Rename the file. Various small fixes. (closes #10674 )
2022-07-23 06:25:29 +01:00
zeertzjq
eb273cd7b0
patch 9.0.0019: timers test not run where possible
...
Problem: Timers test not run where possible.
Solution: Adjust platform checks. (closes #10645 )
2022-07-01 19:11:23 +01:00
Bram Moolenaar
e615db0604
patch 8.2.4166: undo synced when switching buffer in another window
...
Problem: Undo synced when switching buffer in another window.
Solution: Do not sync undo when not needed. (closes #9575 )
2022-01-20 21:00:54 +00:00
Bram Moolenaar
ff39a650b2
patch 8.2.3772: timer info test fails on slow machine
...
Problem: Timer info test fails on slow machine.
Solution: Use WaitForAssert().
2021-12-10 10:57:08 +00:00
Bram Moolenaar
95b2dd0c00
patch 8.2.3768: timer_info() has the wrong repeat value in a timer callback
...
Problem: timer_info() has the wrong repeat value in a timer callback.
Solution: Do not add one to the repeat value when in the callback.
(closes #9294 )
2021-12-09 18:42:57 +00:00
Bram Moolenaar
f08b0eb869
patch 8.2.3518: Test_xrestore sometimes fails
...
Problem: Test_xrestore sometimes fails.
Solution: Mark the test as flayky. Move marking test as flaky to the test
instead of listing them in runtest.
2021-10-16 13:00:14 +01:00
Bram Moolenaar
9f28416e54
patch 8.2.2800: after a timer displays text a hit-enter prompt is given
...
Problem: After a timer displays text a hit-enter prompt is given.
Solution: Reset msg_didany and need_wait_return. (closes #8136 )
2021-04-22 21:39:30 +02:00
Bram Moolenaar
e2e4075fad
patch 8.2.1593: tests do not check the error number properly
...
Problem: Tests do not check the error number properly.0
Solution: Add a colon after the error number. (closes #6869 )
2020-09-04 21:18:46 +02:00
Bram Moolenaar
3fffa97159
patch 8.2.0908: crash when changing the function table while listing it
...
Problem: Crash when changing the function table while listing it.
Solution: Bail out when the function table changes. (closes #6209 )
2020-06-05 21:06:10 +02:00
Bram Moolenaar
494e9069cb
patch 8.2.0866: not enough tests for buffer writing
...
Problem: Not enough tests for buffer writing.
Solution: Add more tests. Use CheckRunVimInTerminal in more places.
(Yegappan Lakshmanan, closes #6167 )
2020-05-31 21:28:02 +02:00
Bram Moolenaar
6a2c5a7dd5
patch 8.2.0533: tests using term_wait() can still be flaky
...
Problem: Tests using term_wait() can still be flaky.
Solution: Increase the wait time when rerunning a test. (James McCoy,
closes #5899 ) Halve the initial times to make tests run faster
when there is no rerun.
2020-04-08 21:50:25 +02:00
Bram Moolenaar
14e57909e6
patch 8.2.0361: internal error when using "0" for a callback
...
Problem: Internal error when using "0" for a callback.
Solution: Give a normal error. (closes #5743 )
2020-03-07 16:59:25 +01:00
Bram Moolenaar
e20b9ececa
patch 8.2.0203: :helptags and some other functionality not tested
...
Problem: :helptags and some other functionality not tested.
Solution: Add more tests. (Yegappan Lakshmanan, closes #5567 )
2020-02-03 21:40:04 +01:00
Bram Moolenaar
5feabe00c4
patch 8.2.0183: tests fail when the float feature is disabled
...
Problem: Tests fail when the float feature is disabled.
Solution: Skip tests that don't work without float support.
2020-01-30 18:24:53 +01:00
Bram Moolenaar
5c463a28ff
patch 8.2.0048: another timers test is flaky on Travis for Mac
...
Problem: Another timers test is flaky on Travis for Mac.
Solution: Increase maximum expected time.
2019-12-27 17:14:33 +01:00
Bram Moolenaar
818fed7a5e
patch 8.2.0043: timers test is still flaky on Travis for Mac
...
Problem: Timers test is still flaky on Travis for Mac.
Solution: Increase maximum expected time.
2019-12-25 15:47:14 +01:00
Bram Moolenaar
bc28e9f15b
patch 8.2.0021: timer test fails too often on Travis with MacOS
...
Problem: Timer test fails too often on Travis with MacOS.
Solution: Be less strict with the time.
2019-12-18 20:10:23 +01:00
Bram Moolenaar
f92e58cadb
patch 8.1.2013: more functions can be used as methods
...
Problem: More functions can be used as methods.
Solution: Make various functions usable as a method.
2019-09-08 21:51:41 +02:00
Bram Moolenaar
ce90e36f59
patch 8.1.2011: more functions can be used as methods
...
Problem: More functions can be used as methods.
Solution: Make various functions usable as a method. Make the window
command test faster.
2019-09-08 18:58:44 +02:00
Bram Moolenaar
315244d85b
patch 8.1.1867: still a timer test that is flaky on Mac
...
Problem: Still a timer test that is flaky on Mac.
Solution: Loop with a sleep instead of one fixed sleep.
2019-08-17 13:18:16 +02:00
Bram Moolenaar
413c04e8d5
patch 8.1.1864: still a timer test that is flaky on Mac
...
Problem: Still a timer test that is flaky on Mac.
Solution: Adjust the sleep times.
2019-08-16 22:29:18 +02:00
Bram Moolenaar
427dddf014
patch 8.1.1859: timer test sometimes fails on Mac
...
Problem: Timer test sometimes fails on Mac.
Solution: Show more info when it fails.
2019-08-16 21:22:41 +02:00
Bram Moolenaar
9a2fddcf04
patch 8.1.1855: another failing timer test
...
Problem: Another failing timer test.
Solution: Assert that timers are finished by the end of the test. Rename
test functions to make them easier to find.
2019-08-16 11:26:06 +02:00
Bram Moolenaar
7e6feb9eeb
patch 8.1.1852: timers test is flaky
...
Problem: Timers test is flaky.
Solution: Accept a larger count. Add test to list of flaky tests.
2019-08-15 23:42:21 +02:00
Bram Moolenaar
8c5a278fc5
patch 8.1.1826: tests use hand coded feature and option checks
...
Problem: Tests use hand coded feature and option checks.
Solution: Use the commands from check.vim in more tests.
2019-08-07 23:07:07 +02:00
Bram Moolenaar
0d702028fe
patch 8.1.1624: when testing in the GUI may try to run gvim in a terminal
...
Problem: When testing in the GUI may try to run gvim in a terminal.
Solution: Add the -v argument. (Yee Cheng Chin, closes #4605 ) Don't skip
tests that work now.
2019-07-04 14:20:41 +02:00
Bram Moolenaar
435f9f06ca
patch 8.1.1619: tests are not run with GUI on Travis
...
Problem: Tests are not run with GUI on Travis.
Solution: Add a testgui job. (Ozaki Kiichi, closes #4609 )
2019-07-03 21:40:16 +02:00
Bram Moolenaar
7d491c4253
patch 8.1.1591: on error garbage collection may free memory in use
...
Problem: On error garbage collection may free memory in use.
Solution: Reset may_garbage_collect when evaluating expression mapping.
Add tests. (Ozaki Kiichi, closes #4579 )
2019-06-25 06:28:02 +02:00
Bram Moolenaar
7a39dd7f00
patch 8.1.1581: shared functions for testing are disorganised
...
Problem: Shared functions for testing are disorganised.
Solution: Group finctions in script files. (Ozaki Kiichi, closes #4573 )
2019-06-23 00:50:15 +02:00
Bram Moolenaar
adc6714aac
patch 8.1.1579: dict and list could be GC'ed while displaying error
...
Problem: Dict and list could be GC'ed while displaying error in a timer.
(Yasuhiro Matsumoto)
Solution: Block garbage collection when executing a timer. Add
test_garbagecollect_soon(). Add "no_wait_return" to
test_override(). (closes #4571 )
2019-06-22 01:40:42 +02:00
Bram Moolenaar
b46fecd345
patch 8.1.1544: some balloon tests don't run when they can
...
Problem: Some balloon tests don't run when they can.
Solution: Split GUI balloon tests off into a separate file. (Ozaki Kiichi,
closes #4538 ) Change the feature check into a command for
consistency.
2019-06-15 17:58:09 +02:00
Bram Moolenaar
b0f94c1ff3
patch 8.1.1524: tests are silently skipped
...
Problem: Tests are silently skipped.
Solution: Throw an exception for skipped tests in more places.
2019-06-13 22:19:53 +02:00
Bram Moolenaar
5d30ff1964
patch 8.1.1483: skipped tests are not properly listed
...
Problem: Skipped tests are not properly listed.
Solution: Throw a "Skipped" exception instead of using ":finish" or ":return".
2019-06-06 16:12:12 +02:00
Bram Moolenaar
8d4ce56a19
patch 8.1.0851: feedkeys() with "L" does not work properly
...
Problem: feedkeys() with "L" does not work properly.
Solution: Do not set typebuf_was_filled when using "L". (Ozaki Kiichi,
closes #3885 )
2019-01-30 22:01:40 +01:00
Bram Moolenaar
50948e4ac2
patch 8.1.0844: when timer fails test will hang forever
...
Problem: When timer fails test will hang forever.
Solution: Use reltime() to limit waiting time. (Ozaki Kiichi, closes #3878 )
2019-01-29 20:36:56 +01:00
Bram Moolenaar
cb908a813c
patch 8.1.0842: getchar_zero test fails on MS-Windows
...
Problem: getchar_zero test fails on MS-Windows.
Solution: Disable the test for now.
2019-01-28 23:20:04 +01:00
Bram Moolenaar
12dfc9eef1
patch 8.1.0840: getchar(0) never returns a character in the terminal
...
Problem: getchar(0) never returns a character in the terminal.
Solution: Call wait_func() at least once.
2019-01-28 22:32:58 +01:00
Bram Moolenaar
b0f42ba60d
patch 8.0.1817: a timer may change v:count unexpectedly
...
Problem: A timer may change v:count unexpectedly.
Solution: Save and restore v:count and similar variables when a timer
callback is invoked. (closes #2897 )
2018-05-12 15:38:26 +02:00
Bram Moolenaar
0e9d1ae321
patch 8.0.1776: in tests, when WaitFor() fails it doesn't say why
...
Problem: In tests, when WaitFor() fails it doesn't say why.
Solution: Turn a few more WaitFor() into WaitForAssert().
2018-04-30 14:28:24 +02:00