1
0
forked from aniani/vim

73 Commits

Author SHA1 Message Date
Yee Cheng Chin
e70587dbdb
patch 9.1.1110: Vim tests are slow and flaky
Problem:  Vim tests are slow and flaky at the same time due to reliance
          on timeouts which are unreliable.
Solution: improve Vim test performance and reduce flakiness
          (Yee Cheng Chin)

A lot of Vim tests currently rely on waiting a specific amount of time
before asserting a condition. This is bad because 1) it is slow, as the
timeout is hardcoded, 2) it's unreliable as a resource-starved runner
may overshoot the timeout. Also, there are a lot of builtin sleep
commands in commonly used utilities like VerifyScreenDump and WaitFor()
which leads to a lot of unnecessary idle time.

Fix these issues by doing the following:
1. Make utilities like VerifyScreenDump and WaitFor use the lowest wait
   time possible (1 ms). This essentially turns it into a spin wait. On
   fast machines, these will finish very quickly. For existing tests
   that had an implicit reliance on the old timeouts (e.g.
   VerifyScreenDump had a 50ms wait before), fix the tests to wait that
   specific amount explicitly.
2. Fix tests that sleep or wait for long amounts of time to instead
   explicitly use a callback mechanism to be notified when a child
   terminal job has finished. This allows the test to only take as much
   time as possible instead of having to hard code an unreliable
   timeout.

With these fixes, tests should 1) completely quickly on fast machines,
and 2) on slow machines they will still run to completion albeit slowly.
Note that previoulsy both were not true. The hardcoded timeouts meant
that on fast machines the tests were mostly idling wasting time, whereas
on slow machines, the timeouts often were not generous enough to allow
them to run to completion.

closes: #16615

Signed-off-by: Yee Cheng Chin <ychin.git@gmail.com>
Signed-off-by: Christian Brabandt <cb@256bit.org>
2025-02-13 20:55:45 +01:00
Bram Moolenaar
29b4c513b1 patch 9.0.1592: not all timer tests are marked as flaky
Problem:    Not all timer tests are marked as flaky.
Solution:   Set the flaky flag for all timer tests. (closes #12355)
2023-05-30 15:34:50 +01:00
Bram Moolenaar
94722c5107 patch 9.0.1257: code style is not check in test scripts
Problem:    Code style is not check in test scripts.
Solution:   Add basic code style check for test files.
2023-01-28 19:19:03 +00:00
Bram Moolenaar
c4860bdd28 patch 9.0.0767: too many delete() calls in tests
Problem:    Too many delete() calls in tests.
Solution:   Use deferred delete where possible.
2022-10-15 20:52:26 +01:00
Bram Moolenaar
cf3d0eaf47 patch 9.0.0682: crash when popup with deleted timer is closed
Problem:    Crash when popup with deleted timer is closed. (Igbanam
            Ogbuluijah)
Solution:   Check the timer still exists. (closes #11301)
2022-10-07 11:20:29 +01:00
Bram Moolenaar
865bf2ed30 patch 9.0.0578: one timer test fails on Mac M1
Problem:    One timer test fails on Mac M1.
Solution:   Skip the test on Mac M1.
2022-09-24 17:44:22 +01:00
Bram Moolenaar
140f6d0eda patch 9.0.0574: timer garbage collect test hangs on Mac M1
Problem:    Timer garbage collect test hangs on Mac M1.
Solution:   Properly check for Mac M1 and skip the test.
2022-09-24 14:49:07 +01:00
Bram Moolenaar
0056ca7137 patch 9.0.0564: a few tests keep failing on MacOS M1
Problem:    A few tests keep failing on MacOS M1.
Solution:   Add a test check CheckNotMacM1.  Fix timer tests.
2022-09-23 21:26:39 +01:00
Bram Moolenaar
4ecf16bbf9 patch 9.0.0559: timer test may get stuck at hit-enter prompt
Problem:    Timer test may get stuck at hit-enter prompt.
Solution:   Feed some more characters.
2022-09-23 18:22:21 +01:00
Yegappan Lakshmanan
8deb2b30c7 patch 9.0.0359: error message for wrong argument type is not specific
Problem:    Error message for wrong argument type is not specific.
Solution:   Include more information in the error. (Yegappan Lakshmanan,
            closes #11037)
2022-09-02 15:15:27 +01:00
Yegappan Lakshmanan
04c4c5746e patch 9.0.0335: checks for Dictionary argument often give a vague error
Problem:    Checks for Dictionary argument often give a vague error message.
Solution:   Give a useful error message. (Yegappan Lakshmanan, closes #11009)
2022-08-30 19:48:24 +01:00
zeertzjq
bb404f5ad5 patch 9.0.0059: test file has wrong name
Problem:    Test file has wrong name.
Solution:   Rename the file.  Various small fixes. (closes #10674)
2022-07-23 06:25:29 +01:00
zeertzjq
eb273cd7b0 patch 9.0.0019: timers test not run where possible
Problem:    Timers test not run where possible.
Solution:   Adjust platform checks. (closes #10645)
2022-07-01 19:11:23 +01:00
Bram Moolenaar
e615db0604 patch 8.2.4166: undo synced when switching buffer in another window
Problem:    Undo synced when switching buffer in another window.
Solution:   Do not sync undo when not needed. (closes #9575)
2022-01-20 21:00:54 +00:00
Bram Moolenaar
ff39a650b2 patch 8.2.3772: timer info test fails on slow machine
Problem:    Timer info test fails on slow machine.
Solution:   Use WaitForAssert().
2021-12-10 10:57:08 +00:00
Bram Moolenaar
95b2dd0c00 patch 8.2.3768: timer_info() has the wrong repeat value in a timer callback
Problem:    timer_info() has the wrong repeat value in a timer callback.
Solution:   Do not add one to the repeat value when in the callback.
            (closes #9294)
2021-12-09 18:42:57 +00:00
Bram Moolenaar
f08b0eb869 patch 8.2.3518: Test_xrestore sometimes fails
Problem:    Test_xrestore sometimes fails.
Solution:   Mark the test as flayky.  Move marking test as flaky to the test
            instead of listing them in runtest.
2021-10-16 13:00:14 +01:00
Bram Moolenaar
9f28416e54 patch 8.2.2800: after a timer displays text a hit-enter prompt is given
Problem:    After a timer displays text a hit-enter prompt is given.
Solution:   Reset msg_didany and need_wait_return. (closes #8136)
2021-04-22 21:39:30 +02:00
Bram Moolenaar
e2e4075fad patch 8.2.1593: tests do not check the error number properly
Problem:    Tests do not check the error number properly.0
Solution:   Add a colon after the error number. (closes #6869)
2020-09-04 21:18:46 +02:00
Bram Moolenaar
3fffa97159 patch 8.2.0908: crash when changing the function table while listing it
Problem:    Crash when changing the function table while listing it.
Solution:   Bail out when the function table changes. (closes #6209)
2020-06-05 21:06:10 +02:00
Bram Moolenaar
494e9069cb patch 8.2.0866: not enough tests for buffer writing
Problem:    Not enough tests for buffer writing.
Solution:   Add more tests. Use CheckRunVimInTerminal in more places.
            (Yegappan Lakshmanan, closes #6167)
2020-05-31 21:28:02 +02:00
Bram Moolenaar
6a2c5a7dd5 patch 8.2.0533: tests using term_wait() can still be flaky
Problem:    Tests using term_wait() can still be flaky.
Solution:   Increase the wait time when rerunning a test. (James McCoy,
            closes #5899)  Halve the initial times to make tests run faster
            when there is no rerun.
2020-04-08 21:50:25 +02:00
Bram Moolenaar
14e57909e6 patch 8.2.0361: internal error when using "0" for a callback
Problem:    Internal error when using "0" for a callback.
Solution:   Give a normal error. (closes #5743)
2020-03-07 16:59:25 +01:00
Bram Moolenaar
e20b9ececa patch 8.2.0203: :helptags and some other functionality not tested
Problem:    :helptags and some other functionality not tested.
Solution:   Add more tests. (Yegappan Lakshmanan, closes #5567)
2020-02-03 21:40:04 +01:00
Bram Moolenaar
5feabe00c4 patch 8.2.0183: tests fail when the float feature is disabled
Problem:    Tests fail when the float feature is disabled.
Solution:   Skip tests that don't work without float support.
2020-01-30 18:24:53 +01:00
Bram Moolenaar
5c463a28ff patch 8.2.0048: another timers test is flaky on Travis for Mac
Problem:    Another timers test is flaky on Travis for Mac.
Solution:   Increase maximum expected time.
2019-12-27 17:14:33 +01:00
Bram Moolenaar
818fed7a5e patch 8.2.0043: timers test is still flaky on Travis for Mac
Problem:    Timers test is still flaky on Travis for Mac.
Solution:   Increase maximum expected time.
2019-12-25 15:47:14 +01:00
Bram Moolenaar
bc28e9f15b patch 8.2.0021: timer test fails too often on Travis with MacOS
Problem:    Timer test fails too often on Travis with MacOS.
Solution:   Be less strict with the time.
2019-12-18 20:10:23 +01:00
Bram Moolenaar
f92e58cadb patch 8.1.2013: more functions can be used as methods
Problem:    More functions can be used as methods.
Solution:   Make various functions usable as a method.
2019-09-08 21:51:41 +02:00
Bram Moolenaar
ce90e36f59 patch 8.1.2011: more functions can be used as methods
Problem:    More functions can be used as methods.
Solution:   Make various functions usable as a method.  Make the window
            command test faster.
2019-09-08 18:58:44 +02:00
Bram Moolenaar
315244d85b patch 8.1.1867: still a timer test that is flaky on Mac
Problem:    Still a timer test that is flaky on Mac.
Solution:   Loop with a sleep instead of one fixed sleep.
2019-08-17 13:18:16 +02:00
Bram Moolenaar
413c04e8d5 patch 8.1.1864: still a timer test that is flaky on Mac
Problem:    Still a timer test that is flaky on Mac.
Solution:   Adjust the sleep times.
2019-08-16 22:29:18 +02:00
Bram Moolenaar
427dddf014 patch 8.1.1859: timer test sometimes fails on Mac
Problem:    Timer test sometimes fails on Mac.
Solution:   Show more info when it fails.
2019-08-16 21:22:41 +02:00
Bram Moolenaar
9a2fddcf04 patch 8.1.1855: another failing timer test
Problem:    Another failing timer test.
Solution:   Assert that timers are finished by the end of the test.  Rename
            test functions to make them easier to find.
2019-08-16 11:26:06 +02:00
Bram Moolenaar
7e6feb9eeb patch 8.1.1852: timers test is flaky
Problem:    Timers test is flaky.
Solution:   Accept a larger count.  Add test to list of flaky tests.
2019-08-15 23:42:21 +02:00
Bram Moolenaar
8c5a278fc5 patch 8.1.1826: tests use hand coded feature and option checks
Problem:    Tests use hand coded feature and option checks.
Solution:   Use the commands from check.vim in more tests.
2019-08-07 23:07:07 +02:00
Bram Moolenaar
0d702028fe patch 8.1.1624: when testing in the GUI may try to run gvim in a terminal
Problem:    When testing in the GUI may try to run gvim in a terminal.
Solution:   Add the -v argument. (Yee Cheng Chin, closes #4605)  Don't skip
            tests that work now.
2019-07-04 14:20:41 +02:00
Bram Moolenaar
435f9f06ca patch 8.1.1619: tests are not run with GUI on Travis
Problem:    Tests are not run with GUI on Travis.
Solution:   Add a testgui job. (Ozaki Kiichi, closes #4609)
2019-07-03 21:40:16 +02:00
Bram Moolenaar
7d491c4253 patch 8.1.1591: on error garbage collection may free memory in use
Problem:    On error garbage collection may free memory in use.
Solution:   Reset may_garbage_collect when evaluating expression mapping.
            Add tests. (Ozaki Kiichi, closes #4579)
2019-06-25 06:28:02 +02:00
Bram Moolenaar
7a39dd7f00 patch 8.1.1581: shared functions for testing are disorganised
Problem:    Shared functions for testing are disorganised.
Solution:   Group finctions in script files. (Ozaki Kiichi, closes #4573)
2019-06-23 00:50:15 +02:00
Bram Moolenaar
adc6714aac patch 8.1.1579: dict and list could be GC'ed while displaying error
Problem:    Dict and list could be GC'ed while displaying error in a timer.
            (Yasuhiro Matsumoto)
Solution:   Block garbage collection when executing a timer.  Add
            test_garbagecollect_soon(). Add "no_wait_return" to
            test_override(). (closes #4571)
2019-06-22 01:40:42 +02:00
Bram Moolenaar
b46fecd345 patch 8.1.1544: some balloon tests don't run when they can
Problem:    Some balloon tests don't run when they can.
Solution:   Split GUI balloon tests off into a separate file. (Ozaki Kiichi,
            closes #4538)  Change the feature check into a command for
            consistency.
2019-06-15 17:58:09 +02:00
Bram Moolenaar
b0f94c1ff3 patch 8.1.1524: tests are silently skipped
Problem:    Tests are silently skipped.
Solution:   Throw an exception for skipped tests in more places.
2019-06-13 22:19:53 +02:00
Bram Moolenaar
5d30ff1964 patch 8.1.1483: skipped tests are not properly listed
Problem:    Skipped tests are not properly listed.
Solution:   Throw a "Skipped" exception instead of using ":finish" or ":return".
2019-06-06 16:12:12 +02:00
Bram Moolenaar
8d4ce56a19 patch 8.1.0851: feedkeys() with "L" does not work properly
Problem:    feedkeys() with "L" does not work properly.
Solution:   Do not set typebuf_was_filled when using "L". (Ozaki Kiichi,
            closes #3885)
2019-01-30 22:01:40 +01:00
Bram Moolenaar
50948e4ac2 patch 8.1.0844: when timer fails test will hang forever
Problem:    When timer fails test will hang forever.
Solution:   Use reltime() to limit waiting time. (Ozaki Kiichi, closes #3878)
2019-01-29 20:36:56 +01:00
Bram Moolenaar
cb908a813c patch 8.1.0842: getchar_zero test fails on MS-Windows
Problem:    getchar_zero test fails on MS-Windows.
Solution:   Disable the test for now.
2019-01-28 23:20:04 +01:00
Bram Moolenaar
12dfc9eef1 patch 8.1.0840: getchar(0) never returns a character in the terminal
Problem:    getchar(0) never returns a character in the terminal.
Solution:   Call wait_func() at least once.
2019-01-28 22:32:58 +01:00
Bram Moolenaar
b0f42ba60d patch 8.0.1817: a timer may change v:count unexpectedly
Problem:    A timer may change v:count unexpectedly.
Solution:   Save and restore v:count and similar variables when a timer
            callback is invoked. (closes #2897)
2018-05-12 15:38:26 +02:00
Bram Moolenaar
0e9d1ae321 patch 8.0.1776: in tests, when WaitFor() fails it doesn't say why
Problem:    In tests, when WaitFor() fails it doesn't say why.
Solution:   Turn a few more WaitFor() into WaitForAssert().
2018-04-30 14:28:24 +02:00