0
0
mirror of https://github.com/vim/vim.git synced 2025-07-04 23:07:33 -04:00

161 Commits

Author SHA1 Message Date
Yee Cheng Chin
2ce070c27a
patch 9.0.1913: if_python: undefined behaviour for function pointers
Problem:  if_python: undefined behaviour for function pointers
Solution: Fix if_python undefined behavior for function pointer casts

Identified by clang 17 UBSAN (see #12745). Make sure to cast function
pointers with the same signature only.

closes: #13122

Signed-off-by: Christian Brabandt <cb@256bit.org>
Co-authored-by: Yee Cheng Chin <ychin.git@gmail.com>
2023-09-19 20:30:22 +02:00
Yee Cheng Chin
c13b3d1350
patch 9.0.1776: No support for stable Python 3 ABI
Problem:  No support for stable Python 3 ABI
Solution: Support Python 3 stable ABI

Commits:
1) Support Python 3 stable ABI to allow mixed version interoperatbility

Vim currently supports embedding Python for use with plugins, and the
"dynamic" linking option allows the user to specify a locally installed
version of Python by setting `pythonthreedll`. However, one caveat is
that the Python 3 libs are not binary compatible across minor versions,
and mixing versions can potentially be dangerous (e.g. let's say Vim was
linked against the Python 3.10 SDK, but the user sets `pythonthreedll`
to a 3.11 lib). Usually, nothing bad happens, but in theory this could
lead to crashes, memory corruption, and other unpredictable behaviors.
It's also difficult for the user to tell something is wrong because Vim
has no way of reporting what Python 3 version Vim was linked with.

For Vim installed via a package manager, this usually isn't an issue
because all the dependencies would already be figured out. For prebuilt
Vim binaries like MacVim (my motivation for working on this), AppImage,
and Win32 installer this could potentially be an issue as usually a
single binary is distributed. This is more tricky when a new Python
version is released, as there's a chicken-and-egg issue with deciding
what Python version to build against and hard to keep in sync when a new
Python version just drops and we have a mix of users of different Python
versions, and a user just blindly upgrading to a new Python could lead to
bad interactions with Vim.

Python 3 does have a solution for this problem: stable ABI / limited API
(see https://docs.python.org/3/c-api/stable.html). The C SDK limits the
API to a set of functions that are promised to be stable across
versions. This pull request adds an ifdef config that allows us to turn
it on when building Vim. Vim binaries built with this option should be
safe to freely link with any Python 3 libraies without having the
constraint of having to use the same minor version.

Note: Python 2 has no such concept and this doesn't change how Python 2
integration works (not that there is going to be a new version of Python
2 that would cause compatibility issues in the future anyway).

---

Technical details:
======

The stable ABI can be accessed when we compile with the Python 3 limited
API (by defining `Py_LIMITED_API`). The Python 3 code (in `if_python3.c`
and `if_py_both.h`) would now handle this and switch to limited API
mode. Without it set, Vim will still use the full API as before so this
is an opt-in change.

The main difference is that `PyType_Object` is now an opaque struct that
we can't directly create "static types" out of, and we have to create
type objects as "heap types" instead. This is because the struct is not
stable and changes from version to version (e.g. 3.8 added a
`tp_vectorcall` field to it). I had to change all the types to be
allocated on the heap instead with just a pointer to them.

Other functions are also simply missing in limited API, or they are
introduced too late (e.g. `PyUnicode_AsUTF8AndSize` in 3.10) to it that
we need some other ways to do the same thing, so I had to abstract a few
things into macros, and sometimes re-implement functions like
`PyObject_NEW`.

One caveat is that in limited API, `OutputType` (used for replacing
`sys.stdout`) no longer inherits from `PyStdPrinter_Type` which I don't
think has any real issue other than minor differences in how they
convert to a string and missing a couple functions like `mode()` and
`fileno()`.

Also fixed an existing bug where `tp_basicsize` was set incorrectly for
`BufferObject`, `TabListObject, `WinListObject`.

Technically, there could be a small performance drop, there is a little
more indirection with accessing type objects, and some APIs like
`PyUnicode_AsUTF8AndSize` are missing, but in practice I didn't see any
difference, and any well-written Python plugin should try to avoid
excessing callbacks to the `vim` module in Python anyway.

I only tested limited API mode down to Python 3.7, which seemes to
compile and work fine. I haven't tried earlier Python versions.

2) Fix PyIter_Check on older Python vers / type##Ptr unused warning

For PyIter_Check, older versions exposed them as either macros (used in
full API), or a function (for use in limited API). A previous change
exposed PyIter_Check to the dynamic build because Python just moved it
to function-only in 3.10 anyway. Because of that, just make sure we
always grab the function in dynamic builds in earlier versions since
that's what Python eventually did anyway.

3) Move Py_LIMITED_API define to configure script

Can now use --with-python-stable-abi flag to customize what stable ABI
version to target. Can also use an env var to do so as well.

4) Show +python/dyn-stable in :version, and allow has() feature query

Not sure if the "/dyn-stable" suffix would break things, or whether we
should do it another way. Or just don't show it in version and rely on
has() feature checking.

5) Documentation first draft. Still need to implement v:python3_version

6) Fix PyIter_Check build breaks when compiling against Python 3.8

7) Add CI coverage stable ABI on Linux/Windows / make configurable on Windows

This adds configurable options for Windows make files (both MinGW and
MSVC). CI will also now exercise both traditional full API and stable
ABI for Linux and Windows in the matrix for coverage.

Also added a "dynamic" option to Linux matrix as a drive-by change to
make other scripting languages like Ruby / Perl testable under both
static and dynamic builds.

8) Fix inaccuracy in Windows docs

Python's own docs are confusing but you don't actually want to use
`python3.dll` for the dynamic linkage.

9) Add generated autoconf file

10) Add v:python3_version support

This variable indicates the version of Python3 that Vim was built
against (PY_VERSION_HEX), and will be useful to check whether the Python
library you are loading in dynamically actually fits it. When built with
stable ABI, it will be the limited ABI version instead
(`Py_LIMITED_API`), which indicates the minimum version of Python 3 the
user should have, rather than the exact match. When stable ABI is used,
we won't be exposing PY_VERSION_HEX in this var because it just doesn't
seem necessary to do so (the whole point of stable ABI is the promise
that it will work across versions), and I don't want to confuse the user
with too many variables.

Also, cleaned up some documentation, and added help tags.

11) Fix Python 3.7 compat issues

Fix a couple issues when using limited API < 3.8

- Crash on exit: In Python 3.7, if a heap-allocated type is destroyed
  before all instances are, it would cause a crash later. This happens
  when we destroyed `OptionsType` before calling `Py_Finalize` when
  using the limited API. To make it worse, later versions changed the
  semantics and now each instance has a strong reference to its own type
  and the recommendation has changed to have each instance de-ref its
  own type and have its type in GC traversal. To avoid dealing with
  these cross-version variations, we just don't free the heap type. They
  are static types in non-limited-API anyway and are designed to last
  through the entirety of the app, and we also don't restart the Python
  runtime and therefore do not need it to have absolutely 0 leaks.

  See:
  - https://docs.python.org/3/whatsnew/3.8.html#changes-in-the-c-api
  - https://docs.python.org/3/whatsnew/3.9.html#changes-in-the-c-api

- PyIter_Check: This function is not provided in limited APIs older than
  3.8. Previously I was trying to mock it out using manual
  PyType_GetSlot() but it was brittle and also does not actually work
  properly for static types (it will generate a Python error). Just
  return false. It does mean using limited API < 3.8 is not recommended
  as you lose the functionality to handle iterators, but from playing
  with plugins I couldn't find it to be an issue.

- Fix loading of PyIter_Check so it will be done when limited API < 3.8.
  Otherwise loading a 3.7 Python lib will fail even if limited API was
  specified to use it.

12) Make sure to only load `PyUnicode_AsUTF8AndSize` in needed in limited API

We don't use this function unless limited API >= 3.10, but we were
loading it regardless. Usually it's ok in Unix-like systems where Python
just has a single lib that we load from, but in Windows where there is a
separate python3.dll this would not work as the symbol would not have
been exposed in this more limited DLL file. This makes it much clearer
under what condition is this function needed.

closes: #12032

Signed-off-by: Christian Brabandt <cb@256bit.org>
Co-authored-by: Yee Cheng Chin <ychin.git@gmail.com>
2023-08-20 21:18:38 +02:00
Zdenek Dohnal
15a0a0281a
patch 9.0.1716: Windows build with python 3.12 and clang fails
Problem: Windows build with python 3.12 and clang fails
Solution: Remove the PyBool_Type function pointer for python 3.12

Clang on Win doesn't like non-static functions in static struct with
Python 3.12 - removing the new obfuscated function and protecting the
old PyBool function for older Pythons fixes the issue.

closes: #12800

Signed-off-by: Christian Brabandt <cb@256bit.org>
Co-authored-by: Zdenek Dohnal <zdohnal@redhat.com>
2023-08-15 22:52:01 +02:00
Zdenek Dohnal
e5e4709a6a
patch 9.0.1709: dynamic build with python 3.12 breaks
Problem: dynamic build with python 3.12 breaks
Solution: if_python3.c: Fix building dynamic Python3 interpreter

There are new extern global variables defined in python3 development
files, which types are within python3 library, so they break dynamic
python3 interpret Vim plugin.

Since the variables are used in macro `Py_SIZE` which is used in other
python3 headers, the dummy variables have to defined before including
Python.h.

closes: #12660

Signed-off-by: Christian Brabandt <cb@256bit.org>
Co-authored-by: Zdenek Dohnal <zdohnal@redhat.com>
2023-08-13 19:39:27 +02:00
Philip H
422b9dcbfa
patch 9.0.1685: silence Python 3.11 depreciations for gcc
Problem: Python 3.11 interface throws deprecation warnings
Solution: ignore those warnings for gcc and clang

Python 3.11 deprecation warnings are already silenced for clang using
the pragma
```
 # pragma clang diagnostic ignored "-Wdeprecated-declarations"
```

However those warnings are also emitted when using gcc. To avoid them
for both compilers, change use the __GNUC__ ifdef, which is defined for
gcc as well as clang.

Additionally, instead of using the "clang diagnostic ignored" pragma,
let's make use of 'GCC diagnostic ignored' which is again supported by
clang and GCC

closes: #12610

Signed-off-by: Christian Brabandt <cb@256bit.org>
Co-authored-by: Philip H <47042125+pheiduck@users.noreply.github.com>
2023-08-11 22:38:48 +02:00
Bram Moolenaar
ebfec1c531 patch 9.0.1234: the code style has to be checked manually
Problem:    The code style has to be checked manually.
Solution:   Add basic code style checks in a test.  Fix or avoid uncovered
            problems.
2023-01-22 21:14:53 +00:00
Yegappan Lakshmanan
0233bdfa2b patch 9.0.1183: code is indented more than necessary
Problem:    Code is indented more than necessary.
Solution:   Use an early return where it makes sense. (Yegappan Lakshmanan,
            closes #11805)
2023-01-12 12:33:30 +00:00
K.Takata
a7fbaa43b7 patch 9.0.1102: complicated use of #ifdef
Problem:    Complicated use of #ifdef.
Solution:   Simplify #ifdef use. (Ken Takata, closes #11745)
2022-12-26 14:46:51 +00:00
K.Takata
161b6ac04f patch 9.0.0880: preprocessor indenting is off
Problem:    Preprocessor indenting is off.
Solution:   Adjust preprocessor indentation. (Ken Takata, closes #11546)
2022-11-14 15:31:07 +00:00
Philip H
5a5f17f9b3 patch 9.0.0832: deprecation warning causes build failure
Problem:    Deprecation warning causes build failure.
Solution:   Suppress deprecation warning. (closes #11503)
2022-11-05 14:05:31 +00:00
ichizok
6c3d3e6904 patch 9.0.0831: compiler warning for redefining HAVE_DUP
Problem:    Compiler warning for redefining HAVE_DUP.
Solution:   Undefine HAVE_DUP if needed. (Ozaki Kiichi, closes #11484)
2022-11-04 22:38:11 +00:00
Zdenek Dohnal
fee511c1d3 patch 8.2.5168: cannot build with Python 3.11
Problem:    Cannot build with Python 3.11.
Solution:   Adjust define for _PyObject_TypeCheck. (Zdenek Dohnal,
            closes #10627)
2022-06-27 13:59:00 +01:00
Bram Moolenaar
63ff72aab9 patch 8.2.4317: MS-Windows: Vim exits when Python 3 initialisation fails
Problem:    MS-Windows: Vim exits when Python 3 initialisation fails.
Solution:   Hook into the exit() function to recover from the failure.
            (Ken Takata, closes #9710)
2022-02-07 13:54:01 +00:00
Bram Moolenaar
9d00e4a814 patch 8.2.4010: error messages are spread out
Problem:    Error messages are spread out.
Solution:   Move more error messages to errors.h.
2022-01-05 17:49:15 +00:00
Bram Moolenaar
9a846fbaa5 patch 8.2.3977: error messages are spread out
Problem:    Error messages are spread out.
Solution:   Move more error messages to errors.h.
2022-01-01 21:59:18 +00:00
Bram Moolenaar
460ae5dfca patch 8.2.3967: error messages are spread out
Problem:    Error messages are spread out.
Solution:   Move more errors to errors.h.
2022-01-01 14:19:49 +00:00
Dominique Pelle
af4a61a85d patch 8.2.3914: various spelling mistakes in comments
Problem:    Various spelling mistakes in comments.
Solution:   Fix the mistakes. (Dominique Pellé, closes #9416)
2021-12-27 17:21:41 +00:00
Martin Tournoij
1a3e5747b7 patch 8.2.3208: dynamic library load error does not mention why it failed
Problem:    Dynamic library load error does not mention why it failed.
Solution:   Add the error message. (Martin Tournoij, closes #8621)
2021-07-24 13:57:29 +02:00
Zdenek Dohnal
90478f35a8 patch 8.2.2995: linker errors with dynamic Python 3.10
Problem:    Linker errors with dynamic Python 3.10.
Solution:   Add a couple of library entries. (Zdenek Dohnal, closes #8381,
            closes #8356)
2021-06-14 15:08:30 +02:00
Bram Moolenaar
b2f9e0e2c5 patch 8.2.2211: MS-Windows: can't load Python dll if not in the path
Problem:    MS-Windows: can't load Python dll if not in the path.
Solution:   Use the InstallPath registry entry. (Kelvin Lee, closes #7540)
2020-12-25 13:52:37 +01:00
Bram Moolenaar
2e2f52a4a0 patch 8.2.2178: Python 3: non-utf8 character cannot be handled
Problem:    Python 3: non-utf8 character cannot be handled.
Solution:   Change the string decode. (Björn Linse, closes #1053)
2020-12-21 16:03:02 +01:00
Bram Moolenaar
4ce5fe4c87 patch 8.2.1883: compiler warnings when using Python
Problem:    Compiler warnings when using Python.
Solution:   Adjust PyCFunction to also have the second argument.  Use "int"
            return type for some functions.  Insert "(void *)" to get rid of
            the remaining warnings.
2020-10-21 21:01:59 +02:00
Bram Moolenaar
efc0d94afc patch 8.2.1834: PyEval_InitThreads() is deprecated in Python 3.9
Problem:    PyEval_InitThreads() is deprecated in Python 3.9.
Solution:   Do not call PyEval_InitThreads in Python 3.9 and later. (Ken
            Takata, closes #7113)  Avoid warnings for functions.
2020-10-11 18:05:02 +02:00
Bram Moolenaar
c6ed254d9f patch 8.2.1830: MS-Windows: Python3 issue with stdin
Problem:    MS-Windows: Python3 issue with stdin.
Solution:   Check if stdin is readable. (Ken Takata, closes #7106)
2020-10-10 23:26:28 +02:00
Bram Moolenaar
253b16a4ab patch 8.2.1806: MS-Windows with Python: Vim freezes after import command
Problem:    MS-Windows with Python: Vim freezes after import command.
Solution:   Use either "NUL" or "CONIN$" when reopening stdin. (Yasuhiro
            Matsumoto, closes #7083)
2020-10-06 19:59:06 +02:00
Bram Moolenaar
794771cfd8 patch 8.2.1790: MS-Windows with Python: crash when executed from Vifm
Problem:    MS-Windows with Python: crash when executed from Vifm.
Solution:   Use NUL instead of CONIN. (Ken Takata, closes #7061, closes #7053)
2020-10-03 13:57:13 +02:00
Bram Moolenaar
56b8dc331d patch 8.2.1381: MS-Windows: crash with Python 3.5 when stdin is redirected
Problem:    MS-Windows: crash with Python 3.5 when stdin is redirected.
Solution:   Reconnect stdin. (Yasuhiro Matsumoto, Ken Takata, closes #6641)
2020-08-06 21:47:11 +02:00
Bram Moolenaar
ee1b93169d patch 8.2.1225: linker errors when building with dynamic Python 3.9
Problem:    Linker errors when building with dynamic Python 3.9.
Solution:   Add #defined items. (closes #6461)
2020-07-16 22:15:53 +02:00
Bram Moolenaar
ab58946384 patch 8.2.1146: not enough testing for Python
Problem:    Not enough testing for Python.
Solution:   Add more tests.  Fix uncovered problems. (Yegappan Lakshmanan,
            closes #6392)
2020-07-06 21:03:06 +02:00
Bram Moolenaar
2027973b5b patch 8.2.0479: unloading shared libraries on exit has no purpose
Problem:    Unloading shared libraries on exit has no purpose.
Solution:   Do not unload shared libraries on exit.
2020-03-29 20:51:07 +02:00
Bram Moolenaar
a65bb53514 patch 8.2.0354: Python 3.9 does not define _Py_DEC_REFTOTAL
Problem:    Python 3.9 does not define _Py_DEC_REFTOTAL. (Zdenek Dohnal)
Solution:   Remove it, it was only for debugging.
2020-03-04 22:06:07 +01:00
Bram Moolenaar
d672dde584 patch 8.2.0319: file missing in distribution, comments outdated
Problem:    File missing in distribution, comments outdated.
Solution:   Correct path of README file.  Update comments.
2020-02-26 13:43:51 +01:00
Bram Moolenaar
2ab2e8608f patch 8.1.2387: using old C style comments
Problem:    Using old C style comments.
Solution:   Use // comments where appropriate.
2019-12-04 21:24:53 +01:00
Bram Moolenaar
13a1f3fb0c patch 8.1.2201: cannot build with dynamically linked Python 3.8
Problem:    Cannot build with dynamically linked Python 3.8.
Solution:   Implement py3__Py_DECREF() and py3__Py_XDECREF(). (Ken Takata,
            closes #4080)
2019-10-23 21:37:25 +02:00
Bram Moolenaar
c799fe206e patch 8.1.1414: alloc() returning "char_u *" causes a lot of type casts
Problem:    Alloc() returning "char_u *" causes a lot of type casts.
Solution:   Have it return "void *". (Mike Williams)  Define ALLOC_ONE() to
            check the simple allocations.
2019-05-28 23:08:19 +02:00
Bram Moolenaar
964b3746b9 patch 8.1.1384: using "int" for alloc() often results in compiler warnings
Problem:    Using "int" for alloc() often results in compiler warnings.
Solution:   Use "size_t" and remove type casts.  Remove alloc_check(), Vim
            only works with 32 bit ints anyway.
2019-05-24 18:54:09 +02:00
Bram Moolenaar
abab0b0fdd patch 8.1.1086: too many curly braces
Problem:    Too many curly braces.
Solution:   Remove curly braces where they are not needed. (Hirohito Higashi,
            closes #3982)
2019-03-30 18:47:01 +01:00
Bram Moolenaar
8e9a24a127 patch 8.1.1021: pyeval() and py3eval() leak memory
Problem:    pyeval() and py3eval() leak memory.
Solution:   Do not increase the reference count twice. (Ozaki Kiichi,
            closes #4129)
2019-03-19 22:22:55 +01:00
Bram Moolenaar
828bff1f9b patch 8.1.1020: compiler warning for Python3 interface
Problem:    Compiler warning for Python3 interface.
Solution:   Add type cast. (Ozaki Kiichi, closes #4128, closes #4103)
2019-03-19 22:11:41 +01:00
Bram Moolenaar
14816ad6e5 patch 8.1.0950: using :python sets 'pyxversion' even when not executed
Problem:    Using :python sets 'pyxversion' even when not executed.
Solution:   Check the "skip" flag. (Shane Harper, closes #3995)
2019-02-18 22:04:56 +01:00
Bram Moolenaar
4f97475d32 patch 8.1.0941: macros for MS-Windows are inconsistent
Problem:    Macros for MS-Windows are inconsistent, using "32", "3264 and
            others.
Solution:   Use MSWIN for all MS-Windows builds.  Use FEAT_GUI_MSWIN for the
            GUI build. (Hirohito Higashi, closes #3932)
2019-02-17 17:44:42 +01:00
Bram Moolenaar
f9e3e09fdc patch 8.1.0743: giving error messages is not flexible
Problem:    Giving error messages is not flexible.
Solution:   Add semsg().  Change argument from "char_u *" to "char *", also
            for msg() and get rid of most MSG macros. (Ozaki Kiichi, closes
            #3302)  Also make emsg() accept a "char *" argument.  Get rid of
            an enormous number of type casts.
2019-01-13 23:38:42 +01:00
Bram Moolenaar
6e5ea8d2a9 patch 8.1.0735: cannot handle binary data
Problem:    Cannot handle binary data.
Solution:   Add the Blob type. (Yasuhiro Matsumoto, closes #3638)
2019-01-12 22:47:31 +01:00
Bram Moolenaar
6dff58f15c patch 8.1.0443: unnecessary static function prototypes
Problem:    Unnecessary static function prototypes.
Solution:   Remove unnecessary prototypes.
2018-09-30 21:43:26 +02:00
Bram Moolenaar
3b48b11c07 patch 8.1.0147: compiler warning when building with Python 3.7
Problem:    Compiler warning when building with Python 3.7.
Solution:   #undef PySlice_GetIndicesEx before redefining it. (Ozaki Kiichi,
            closes #3153)
2018-07-04 22:03:25 +02:00
Bram Moolenaar
d4a8c98ef6 patch 8.0.1846: Python interface is incompatible with lldb
Problem:    Python interface is incompatible with lldb.
Solution:   For OutputType set the base to be PyFile_Type. (Boxu Zhang)
            Partly disabled to avoid a crash.
2018-05-15 22:31:18 +02:00
Bram Moolenaar
2e324950b8 patch 8.0.1709: some non-C89 code may slip through
Problem:    Some non-C89 code may slip through.
Solution:   Enforce C89 in configure. Fix detected problems. (James McCoy,
            closes #2735)
2018-04-14 14:37:07 +02:00
Bram Moolenaar
7b24ce08fe patch 8.0.1654: warnings for conversion of void to function pointer
Problem:    Warnings for conversion of void to function pointer.
Solution:   Use a temp variable that is a function pointer.
2018-03-29 18:15:26 +02:00
Bram Moolenaar
16d7eced1a patch 8.0.1635: undefining _POSIX_THREADS causes problems with Python 3
Problem:    Undefining _POSIX_THREADS causes problems with Python 3. (Micah
            Bucy, closes #2748)
Solution:   Remove the lines.
2018-03-24 14:06:15 +01:00
Bram Moolenaar
94073167e3 patch 8.0.1451: difficult to set the python home directories properly
Problem:    It is difficult to set the python home directory properly for
            Python 2.7 and 3.5 since both use $PYTHONHOME.
Solution:   Add the 'pythonhome' and 'pythonthreehome' options. (Kazuki
            Sakamoto, closes #1266)
2018-01-31 21:49:05 +01:00