Hello all,
I'm a contributor to the Wine project. To summarize the following mail,
Wine needs special versions of some of its normal dependencies, such as libfreetype and libgnutls, built using the MinGW cross-compiler, and I'm sending out a mail to major distributions in order to get some feedback
from our packagers on how these should be built and packaged.
For a long time Wine has built all of its Win32 libraries (DLLs and
EXEs) as ELF binaries. For various reasons related to application compatibility, we have started building our binaries as PE instead,
using the MinGW cross-compiler. It is our intent to expand this to some
of our dependencies as well. The list of dependencies that we intend to
build using MinGW is not quite fixed yet, but we expect it to include
and be mostly limited to the following:
* libvkd3d
* libFAudio
* libgnutls
* zlib (currently included via manual source import)
* libmpg123
* libgsm
* libpng
* libjpeg-turbo
* libtiff
* libfreetype
* liblcms2
* jxrlib
and dependencies of the above packages (not including CRT dependencies,
which Wine provides).
There is currently some internal discussion about how these dependencies should be built and linked. There are essentially three questions I see
that need to be resolved, and while these resolutions have a significant impact on the Wine building and development process, they also have an
impact on distributions, and accordingly I'd like to get input from our packagers to ensure that their considerations are accurately taken into account.
(1) Should we build via source import, or link statically, or dynamically?
Source imports are dispreferred by Debian [1], on the grounds that they
cause duplication of libraries on disk and in memory, and make it harder
to handle security updates. They also make building and bisecting
harder. Static libraries don't seem to be expressly discouraged, but
share most of the same downsides (see also [2]).
Note however that if they are linked dynamically, we need to make sure
that we load our packages instead of MinGW builds of open-source
libraries with applications ship with. There's some internal discussion
about whether this is possible while using "stock" builds of MinGW
libraries, but, due to the way the Win32 loader works, we will probably
need to compile each library, and its dependencies, with a separate, wine-specific name, e.g. "libwinefreetype-6.dll" and
"libwineharfbuzz.dll". For a detailed explantion see footnote [3]. Note
that all we actually need to change is the name; we don't need to patch
the source.
Accordingly, although static linking and source imports are generally disprefered, it may quite likely be preferable in our case. We don't get
the benefits of on-disk deduplication, since Wine is essentially the
only piece of software which needs these libraries.
(2) If we use dynamic libraries, should dependencies be included in the
main wine package, or packaged separately?
This is mostly a question for packagers, although it also relates to (3).
I expect that Debian will want to answer "packaged separately" here, on
the grounds that this lets them update (say) Wine's libgnutls
separately, and in sync with ELF libgnutls, if some security fix is
needed. There is a snag, though: we need libraries to be copied into the prefix (there's some internal effort to allow using something like
symlinks instead, but this hard and not done yet). Normally we perform
this copy every time Wine is updated, but if Wine and its dependencies
aren't updated on the same schedule, we may end up loading an old
version of a dependency in the prefix.
(3) If dependencies are packaged separately, should Wine build them as
part of its build tree (e.g. using submodules), or find and link
(statically or dynamically) to existing binaries?
Linking to existing binaries is generally preferable: it avoids
duplication on disk; it reduces compile times when compiling a single
package from source (especially the first time). However, we aren't
going to benefit from on-disk duplication. And, most importantly, unlike
with ELF dependencies, there is no standardized way to locate MinGW libraries—especially if it comes to Wine-specific libraries. We would
need a way for Wine's configure script to find these packages—and
ideally find them automatically, or else fall back to a submodule-based approach.
If we rely on distributions to provide our dependencies, the best idea I
have here would be something like a x86_64-w64-mingw32-pkg-config
(Fedora actually already ships this, but I think Fedora is the only
one). And if we use shared libraries rather than static, things get
worse: we need to know the exact path of each library and its
dependencies at runtime so that we can copy (or symlink) them into a
user's WINEPREFIX. For ELF this is the job of ld.so. For MinGW there is
no standardized install location across distributions
For what it's worth, the current proposed solution (which has the
support of the Wine maintainer) involves source imports and submodules. There's probably room for changing our approach even after things are committed, but I'd still like to get early feedback from distributions,
and make sure that their interests are accurately represented, before we commit.
ἔρρωσθε,
Zebediah
(she/her)
[1]
https://www.debian.org/doc/debian-policy/ch-source.html#embedded-code-copies
[2] https://wiki.debian.org/StaticLinking
[3] The basic problem is that applications can and often do ship with PE builds of cross-platform libraries. These libraries can be ahead of
Wine's system libraries, behind them, or even built with custom patches. Accordingly we really don't want to load "our" freetype in place of
"their" freetype, or "theirs" in place of "ours". But because of the way
the Win32 loader works, you can generally only have one DLL of a given
name loaded in a process, and further attempts to dlopen() [as it were] "libfreetype-6.dll" will return the handle to the already loaded
library, potentially breaking either Wine or the application. There
*may* be ways we can hack around this internally, but it's not clear
that it's feasible yet, and so it's probably best to assume that we'll
need special builds of dynamic libraries.
<div dir="auto"><div class="gmail_quote"><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
I'm a contributor to the Wine project. To summarize the following mail,
Wine needs special versions of some of its normal dependencies, such as libfreetype and libgnutls, built using the MinGW cross-compiler, and I'm sending out a mail to major distributions in order to get some feedback
from our packagers on how these should be built and packaged.
For a long time Wine has built all of its Win32 libraries (DLLs and
EXEs) as ELF binaries. For various reasons related to application compatibility, we have started building our binaries as PE instead,
using the MinGW cross-compiler. It is our intent to expand this to some
of our dependencies as well. The list of dependencies that we intend to build using MinGW is not quite fixed yet, but we expect it to include
and be mostly limited to the following:
* libvkd3d
* libFAudio
* libgnutls
* zlib (currently included via manual source import)
* libmpg123
* libgsm
* libpng
* libjpeg-turbo
* libtiff
* libfreetype
* liblcms2
* jxrlib
and dependencies of the above packages (not including CRT dependencies, which Wine provides).
There is currently some internal discussion about how these dependencies should be built and linked. There are essentially three questions I see
that need to be resolved, and while these resolutions have a significant impact on the Wine building and development process, they also have an impact on distributions, and accordingly I'd like to get input from our packagers to ensure that their considerations are accurately taken into account.
(1) Should we build via source import, or link statically, or dynamically?
Source imports are dispreferred by Debian [1], on the grounds that they cause duplication of libraries on disk and in memory, and make it harder
to handle security updates. They also make building and bisecting
harder. Static libraries don't seem to be expressly discouraged, but
share most of the same downsides (see also [2]).
Note however that if they are linked dynamically, we need to make sure
that we load our packages instead of MinGW builds of open-source
libraries with applications ship with. There's some internal discussion about whether this is possible while using "stock" builds of MinGW libraries, but, due to the way the Win32 loader works, we will probably
need to compile each library, and its dependencies, with a separate, wine-specific name, e.g. "libwinefreetype-6.dll" and
"libwineharfbuzz.dll". For a detailed explantion see footnote [3]. Note
that all we actually need to change is the name; we don't need to patch
the source.
Accordingly, although static linking and source imports are generally disprefered, it may quite likely be preferable in our case. We don't get
the benefits of on-disk deduplication, since Wine is essentially the
only piece of software which needs these libraries.
(2) If we use dynamic libraries, should dependencies be included in the
main wine package, or packaged separately?
This is mostly a question for packagers, although it also relates to (3).
I expect that Debian will want to answer "packaged separately" here, on
the grounds that this lets them update (say) Wine's libgnutls
separately, and in sync with ELF libgnutls, if some security fix is
needed. There is a snag, though: we need libraries to be copied into the prefix (there's some internal effort to allow using something like
symlinks instead, but this hard and not done yet). Normally we perform
this copy every time Wine is updated, but if Wine and its dependencies aren't updated on the same schedule, we may end up loading an old
version of a dependency in the prefix.
(3) If dependencies are packaged separately, should Wine build them as
part of its build tree (e.g. using submodules), or find and link
(statically or dynamically) to existing binaries?
Linking to existing binaries is generally preferable: it avoids
duplication on disk; it reduces compile times when compiling a single package from source (especially the first time). However, we aren't
going to benefit from on-disk duplication. And, most importantly, unlike with ELF dependencies, there is no standardized way to locate MinGW libraries—especially if it comes to Wine-specific libraries. We would
need a way for Wine's configure script to find these packages—and
ideally find them automatically, or else fall back to a submodule-based approach.
If we rely on distributions to provide our dependencies, the best idea I have here would be something like a x86_64-w64-mingw32-pkg-config
(Fedora actually already ships this, but I think Fedora is the only
one). And if we use shared libraries rather than static, things get
worse: we need to know the exact path of each library and its
dependencies at runtime so that we can copy (or symlink) them into a
user's WINEPREFIX. For ELF this is the job of ld.so. For MinGW there is
no standardized install location across distributions.
For what it's worth, the current proposed solution (which has the
support of the Wine maintainer) involves source imports and submodules. There's probably room for changing our approach even after things are committed, but I'd still like to get early feedback from distributions,
and make sure that their interests are accurately represented, before we commit.
Le dim. 5 sept. 2021 à 03:34, Zebediah Figura <zfigura@codeweavers.com> a écrit :
I'm a contributor to the Wine project. To summarize the following mail, Wine needs special versions of some of its normal dependencies, such as libfreetype and libgnutls, built using the MinGW cross-compiler, and I'm sending out a mail to major distributions in order to get some feedback from our packagers on how these should be built and packaged.
For a long time Wine has built all of its Win32 libraries (DLLs and
EXEs) as ELF binaries. For various reasons related to application compatibility, we have started building our binaries as PE instead,
using the MinGW cross-compiler. It is our intent to expand this to some
of our dependencies as well. The list of dependencies that we intend to build using MinGW is not quite fixed yet, but we expect it to include
and be mostly limited to the following:
* libvkd3d
* libFAudio
* libgnutls
* zlib (currently included via manual source import)
* libmpg123
* libgsm
* libpng
* libjpeg-turbo
* libtiff
* libfreetype
* liblcms2
* jxrlib
and dependencies of the above packages (not including CRT dependencies, which Wine provides).
There is currently some internal discussion about how these dependencies should be built and linked. There are essentially three questions I see that need to be resolved, and while these resolutions have a significant impact on the Wine building and development process, they also have an impact on distributions, and accordingly I'd like to get input from our packagers to ensure that their considerations are accurately taken into account.
(1) Should we build via source import, or link statically, or dynamically?
Source imports are dispreferred by Debian [1], on the grounds that they cause duplication of libraries on disk and in memory, and make it harder
to handle security updates. They also make building and bisecting
harder. Static libraries don't seem to be expressly discouraged, but
share most of the same downsides (see also [2]).
Note however that if they are linked dynamically, we need to make sure
that we load our packages instead of MinGW builds of open-source
libraries with applications ship with. There's some internal discussion about whether this is possible while using "stock" builds of MinGW libraries, but, due to the way the Win32 loader works, we will probably need to compile each library, and its dependencies, with a separate, wine-specific name, e.g. "libwinefreetype-6.dll" and
"libwineharfbuzz.dll". For a detailed explantion see footnote [3]. Note that all we actually need to change is the name; we don't need to patch
the source.
Accordingly, although static linking and source imports are generally disprefered, it may quite likely be preferable in our case. We don't get the benefits of on-disk deduplication, since Wine is essentially the
only piece of software which needs these libraries.
(2) If we use dynamic libraries, should dependencies be included in the main wine package, or packaged separately?
Improve dpkg to support partial arch. I volonteer to implement none arch
but i am waiting from guillem here.
Hi Zebediah,
On Sat, 4 Sep 2021 20:17:53 -0500, Zebediah Figura <zfigura@codeweavers.com> wrote:
I'm a contributor to the Wine project. To summarize the following mail,
Wine needs special versions of some of its normal dependencies, such as
libfreetype and libgnutls, built using the MinGW cross-compiler, and I'm
sending out a mail to major distributions in order to get some feedback
from our packagers on how these should be built and packaged.
For a long time Wine has built all of its Win32 libraries (DLLs and
EXEs) as ELF binaries. For various reasons related to application
compatibility, we have started building our binaries as PE instead,
using the MinGW cross-compiler. It is our intent to expand this to some
of our dependencies as well. The list of dependencies that we intend to
build using MinGW is not quite fixed yet, but we expect it to include
and be mostly limited to the following:
* libvkd3d
* libFAudio
* libgnutls
* zlib (currently included via manual source import)
* libmpg123
* libgsm
* libpng
* libjpeg-turbo
* libtiff
* libfreetype
* liblcms2
* jxrlib
and dependencies of the above packages (not including CRT dependencies,
which Wine provides).
There is currently some internal discussion about how these dependencies
should be built and linked. There are essentially three questions I see
that need to be resolved, and while these resolutions have a significant
impact on the Wine building and development process, they also have an
impact on distributions, and accordingly I'd like to get input from our
packagers to ensure that their considerations are accurately taken into
account.
(1) Should we build via source import, or link statically, or dynamically? >>
Source imports are dispreferred by Debian [1], on the grounds that they
cause duplication of libraries on disk and in memory, and make it harder
to handle security updates. They also make building and bisecting
harder. Static libraries don't seem to be expressly discouraged, but
share most of the same downsides (see also [2]).
Note however that if they are linked dynamically, we need to make sure
that we load our packages instead of MinGW builds of open-source
libraries with applications ship with. There's some internal discussion
about whether this is possible while using "stock" builds of MinGW
libraries, but, due to the way the Win32 loader works, we will probably
need to compile each library, and its dependencies, with a separate,
wine-specific name, e.g. "libwinefreetype-6.dll" and
"libwineharfbuzz.dll". For a detailed explantion see footnote [3]. Note
that all we actually need to change is the name; we don't need to patch
the source.
Assuming Debian provides the dependencies (which is currently true only for zlib), this could be handled in packaging by providing symlinks, couldn’t it?
Not in the Wine prefixes, but elsewhere.
(The Wine team also maintains libvkd3d and libFaudio, so we can take care of those at least.)
Accordingly, although static linking and source imports are generally
disprefered, it may quite likely be preferable in our case. We don't get
the benefits of on-disk deduplication, since Wine is essentially the
only piece of software which needs these libraries.
(2) If we use dynamic libraries, should dependencies be included in the
main wine package, or packaged separately?
This is mostly a question for packagers, although it also relates to (3).
I expect that Debian will want to answer "packaged separately" here, on
the grounds that this lets them update (say) Wine's libgnutls
separately, and in sync with ELF libgnutls, if some security fix is
needed. There is a snag, though: we need libraries to be copied into the
prefix (there's some internal effort to allow using something like
symlinks instead, but this hard and not done yet). Normally we perform
this copy every time Wine is updated, but if Wine and its dependencies
aren't updated on the same schedule, we may end up loading an old
version of a dependency in the prefix.
Debian packaging doesn’t touch anything in users’ home directories, so this
would have to be handled in Wine itself, perhaps in a similar fashion to existing provisions for Gecko and Mono.
(3) If dependencies are packaged separately, should Wine build them as
part of its build tree (e.g. using submodules), or find and link
(statically or dynamically) to existing binaries?
Linking to existing binaries is generally preferable: it avoids
duplication on disk; it reduces compile times when compiling a single
package from source (especially the first time). However, we aren't
going to benefit from on-disk duplication. And, most importantly, unlike
with ELF dependencies, there is no standardized way to locate MinGW
libraries—especially if it comes to Wine-specific libraries. We would
need a way for Wine's configure script to find these packages—and
ideally find them automatically, or else fall back to a submodule-based
approach.
If we rely on distributions to provide our dependencies, the best idea I
have here would be something like a x86_64-w64-mingw32-pkg-config
(Fedora actually already ships this, but I think Fedora is the only
one). And if we use shared libraries rather than static, things get
worse: we need to know the exact path of each library and its
dependencies at runtime so that we can copy (or symlink) them into a
user's WINEPREFIX. For ELF this is the job of ld.so. For MinGW there is
no standardized install location across distributions.
This also works in Debian:
$ sudo apt install libz-mingw-w64-dev mingw-w64-tools
$ x86_64-w64-mingw32-pkg-config --libs zlib
-L/usr/x86_64-w64-mingw32/lib -lz
For what it's worth, the current proposed solution (which has the
support of the Wine maintainer) involves source imports and submodules.
There's probably room for changing our approach even after things are
committed, but I'd still like to get early feedback from distributions,
and make sure that their interests are accurately represented, before we
commit.
Realistically, I think this is the best approach for now. As Debian adds support for PE libraries, we can replace the source imports in the Wine source code; this is done in many other Debian packages for projects which vendor dependencies.
Regards,
Stephen
On 9/5/21 11:19 AM, Stephen Kitt wrote:
On Sat, 4 Sep 2021 20:17:53 -0500, Zebediah Figura <zfigura@codeweavers.com> wrote:
I'm a contributor to the Wine project. To summarize the following mail,
Wine needs special versions of some of its normal dependencies, such as
libfreetype and libgnutls, built using the MinGW cross-compiler, and I'm >> sending out a mail to major distributions in order to get some feedback
from our packagers on how these should be built and packaged.
For a long time Wine has built all of its Win32 libraries (DLLs and
EXEs) as ELF binaries. For various reasons related to application
compatibility, we have started building our binaries as PE instead,
using the MinGW cross-compiler. It is our intent to expand this to some
of our dependencies as well. The list of dependencies that we intend to
build using MinGW is not quite fixed yet, but we expect it to include
and be mostly limited to the following:
* libvkd3d
* libFAudio
* libgnutls
* zlib (currently included via manual source import)
* libmpg123
* libgsm
* libpng
* libjpeg-turbo
* libtiff
* libfreetype
* liblcms2
* jxrlib
and dependencies of the above packages (not including CRT dependencies,
which Wine provides).
There is currently some internal discussion about how these dependencies >> should be built and linked. There are essentially three questions I see
that need to be resolved, and while these resolutions have a significant >> impact on the Wine building and development process, they also have an
impact on distributions, and accordingly I'd like to get input from our
packagers to ensure that their considerations are accurately taken into
account.
(1) Should we build via source import, or link statically, or
dynamically?
Source imports are dispreferred by Debian [1], on the grounds that they
cause duplication of libraries on disk and in memory, and make it harder >> to handle security updates. They also make building and bisecting
harder. Static libraries don't seem to be expressly discouraged, but
share most of the same downsides (see also [2]).
Note however that if they are linked dynamically, we need to make sure
that we load our packages instead of MinGW builds of open-source
libraries with applications ship with. There's some internal discussion
about whether this is possible while using "stock" builds of MinGW
libraries, but, due to the way the Win32 loader works, we will probably
need to compile each library, and its dependencies, with a separate,
wine-specific name, e.g. "libwinefreetype-6.dll" and
"libwineharfbuzz.dll". For a detailed explantion see footnote [3]. Note
that all we actually need to change is the name; we don't need to patch
the source.
Assuming Debian provides the dependencies (which is currently true only
for zlib), this could be handled in packaging by providing symlinks, couldn’t it? Not in the Wine prefixes, but elsewhere.
Almost :-/
Copying/symlinking libfreetype-1.dll to libwinefreetype-1.dll is easy.
The problem is that libwinefreetype-1.dll is still going to link to libharfbuzz-1.dll, but we need it to link to libwineharfbuzz-1.dll.
This also works in Debian:
$ sudo apt install libz-mingw-w64-dev mingw-w64-tools
$ x86_64-w64-mingw32-pkg-config --libs zlib
-L/usr/x86_64-w64-mingw32/lib -lz
Ah, cool, I looked for that and somehow missed it.
Note that's the wrong path for dynamic libraries, though; those go in /usr/x86_64-w64-mingw32/bin/. We'll need a way to find those. Maybe hardcoding a list won't be too painful, though...
For what it's worth, the current proposed solution (which has the
support of the Wine maintainer) involves source imports and submodules.
There's probably room for changing our approach even after things are
committed, but I'd still like to get early feedback from distributions,
and make sure that their interests are accurately represented, before we >> commit.
Realistically, I think this is the best approach for now. As Debian adds support for PE libraries, we can replace the source imports in the Wine source code; this is done in many other Debian packages for projects which vendor dependencies.
On Sun, 5 Sep 2021 12:14:47 -0500, Zebediah Figura <zfigura@codeweavers.com> wrote:
On 9/5/21 11:19 AM, Stephen Kitt wrote:
On Sat, 4 Sep 2021 20:17:53 -0500, Zebediah Figura <zfigura@codeweavers.com> wrote:
I'm a contributor to the Wine project. To summarize the following mail, >> Wine needs special versions of some of its normal dependencies, such as >> libfreetype and libgnutls, built using the MinGW cross-compiler, and I'm >> sending out a mail to major distributions in order to get some feedback >> from our packagers on how these should be built and packaged.
For a long time Wine has built all of its Win32 libraries (DLLs and
EXEs) as ELF binaries. For various reasons related to application
compatibility, we have started building our binaries as PE instead,
using the MinGW cross-compiler. It is our intent to expand this to some >> of our dependencies as well. The list of dependencies that we intend to >> build using MinGW is not quite fixed yet, but we expect it to include
and be mostly limited to the following:
* libvkd3d
* libFAudio
* libgnutls
* zlib (currently included via manual source import)
* libmpg123
* libgsm
* libpng
* libjpeg-turbo
* libtiff
* libfreetype
* liblcms2
* jxrlib
and dependencies of the above packages (not including CRT dependencies, >> which Wine provides).
There is currently some internal discussion about how these dependencies >> should be built and linked. There are essentially three questions I see >> that need to be resolved, and while these resolutions have a significant >> impact on the Wine building and development process, they also have an >> impact on distributions, and accordingly I'd like to get input from our >> packagers to ensure that their considerations are accurately taken into >> account.
(1) Should we build via source import, or link statically, or
dynamically?
Source imports are dispreferred by Debian [1], on the grounds that they >> cause duplication of libraries on disk and in memory, and make it harder >> to handle security updates. They also make building and bisecting
harder. Static libraries don't seem to be expressly discouraged, but
share most of the same downsides (see also [2]).
Note however that if they are linked dynamically, we need to make sure >> that we load our packages instead of MinGW builds of open-source
libraries with applications ship with. There's some internal discussion >> about whether this is possible while using "stock" builds of MinGW
libraries, but, due to the way the Win32 loader works, we will probably >> need to compile each library, and its dependencies, with a separate,
wine-specific name, e.g. "libwinefreetype-6.dll" and
"libwineharfbuzz.dll". For a detailed explantion see footnote [3]. Note >> that all we actually need to change is the name; we don't need to patch >> the source.
Assuming Debian provides the dependencies (which is currently true only for zlib), this could be handled in packaging by providing symlinks, couldn’t it? Not in the Wine prefixes, but elsewhere.
Almost :-/
Copying/symlinking libfreetype-1.dll to libwinefreetype-1.dll is easy.
The problem is that libwinefreetype-1.dll is still going to link to libharfbuzz-1.dll, but we need it to link to libwineharfbuzz-1.dll.
Ah yes, I hadn’t thought it through. So really Wine needs its own parallel ecosystem of DLLs in any case, which effectively means building them along with Wine (from Wine’s perspective, ignoring distribution constraints and preferences).
This also works in Debian:
$ sudo apt install libz-mingw-w64-dev mingw-w64-tools
$ x86_64-w64-mingw32-pkg-config --libs zlib
-L/usr/x86_64-w64-mingw32/lib -lz
Ah, cool, I looked for that and somehow missed it.
Note that's the wrong path for dynamic libraries, though; those go in /usr/x86_64-w64-mingw32/bin/. We'll need a way to find those. Maybe hardcoding a list won't be too painful, though...
In Debian they go in /usr/x86_64-w64-mingw32/lib currently, which is why
the .pc file points there. But as you point out, that doesn’t help at runtime.
Also, having pkg-config support doesn’t really help with a parallel set of DLLs, does it?
For what it's worth, the current proposed solution (which has the
support of the Wine maintainer) involves source imports and submodules. >> There's probably room for changing our approach even after things are
committed, but I'd still like to get early feedback from distributions, >> and make sure that their interests are accurately represented, before we >> commit.
Realistically, I think this is the best approach for now. As Debian adds support for PE libraries, we can replace the source imports in the Wine source code; this is done in many other Debian packages for projects which
vendor dependencies.
I still think this is true. With requirement for a Wine-specific set of DLLs, improving the situation in Debian would involve supporting source build-dependencies, i.e. being able to say that a given package wants access to the source code of another package. That’s something that’s been brought
up previously, and is worked around by providing binary packages containing package source code (e.g. binutils-source, gcc-11-source etc.), but isn’t really upstream’s concern, in that it’s a Debian implementation detail.
Going back to your original three questions, I think that the best approach for you as upstream is to focus on providing a complete set of source code (including dependencies) which works, and to make it friendlier to distributions, make the build process capable of handling alternative locations for the dependencies’ source code or even build artifacts. (This has a number of knock-on effects — in particular, you should ensure that the
upstream source code for all your dependencies works with Wine, i.e. that Wine doesn’t require Wine-specific patches to any of its dependencies.)
Given how varied MinGW-w64 handling is in different distributions, pushing things further risks making it easier for one distribution and harder for the others...
Regards,
Stephen
On Sun, 5 Sep 2021 12:14:47 -0500, Zebediah Figura <zfigura@codeweavers.com> wrote:
On 9/5/21 11:19 AM, Stephen Kitt wrote:
On Sat, 4 Sep 2021 20:17:53 -0500, Zebediah Figura
<zfigura@codeweavers.com> wrote:
I'm a contributor to the Wine project. To summarize the following mail, >>>> Wine needs special versions of some of its normal dependencies, such as >>>> libfreetype and libgnutls, built using the MinGW cross-compiler, and I'm >>>> sending out a mail to major distributions in order to get some feedback >>>> from our packagers on how these should be built and packaged.
For a long time Wine has built all of its Win32 libraries (DLLs and
EXEs) as ELF binaries. For various reasons related to application
compatibility, we have started building our binaries as PE instead,
using the MinGW cross-compiler. It is our intent to expand this to some >>>> of our dependencies as well. The list of dependencies that we intend to >>>> build using MinGW is not quite fixed yet, but we expect it to include
and be mostly limited to the following:
* libvkd3d
* libFAudio
* libgnutls
* zlib (currently included via manual source import)
* libmpg123
* libgsm
* libpng
* libjpeg-turbo
* libtiff
* libfreetype
* liblcms2
* jxrlib
and dependencies of the above packages (not including CRT dependencies, >>>> which Wine provides).
There is currently some internal discussion about how these dependencies >>>> should be built and linked. There are essentially three questions I see >>>> that need to be resolved, and while these resolutions have a significant >>>> impact on the Wine building and development process, they also have an >>>> impact on distributions, and accordingly I'd like to get input from our >>>> packagers to ensure that their considerations are accurately taken into >>>> account.
(1) Should we build via source import, or link statically, or
dynamically?
Source imports are dispreferred by Debian [1], on the grounds that they >>>> cause duplication of libraries on disk and in memory, and make it harder >>>> to handle security updates. They also make building and bisecting
harder. Static libraries don't seem to be expressly discouraged, but
share most of the same downsides (see also [2]).
Note however that if they are linked dynamically, we need to make sure >>>> that we load our packages instead of MinGW builds of open-source
libraries with applications ship with. There's some internal discussion >>>> about whether this is possible while using "stock" builds of MinGW
libraries, but, due to the way the Win32 loader works, we will probably >>>> need to compile each library, and its dependencies, with a separate,
wine-specific name, e.g. "libwinefreetype-6.dll" and
"libwineharfbuzz.dll". For a detailed explantion see footnote [3]. Note >>>> that all we actually need to change is the name; we don't need to patch >>>> the source.
Assuming Debian provides the dependencies (which is currently true only
for zlib), this could be handled in packaging by providing symlinks,
couldn’t it? Not in the Wine prefixes, but elsewhere.
Almost :-/
Copying/symlinking libfreetype-1.dll to libwinefreetype-1.dll is easy.
The problem is that libwinefreetype-1.dll is still going to link to
libharfbuzz-1.dll, but we need it to link to libwineharfbuzz-1.dll.
Ah yes, I hadn’t thought it through. So really Wine needs its own parallel ecosystem of DLLs in any case, which effectively means building them along with Wine (from Wine’s perspective, ignoring distribution constraints and preferences).
This also works in Debian:
$ sudo apt install libz-mingw-w64-dev mingw-w64-tools
$ x86_64-w64-mingw32-pkg-config --libs zlib
-L/usr/x86_64-w64-mingw32/lib -lz
Ah, cool, I looked for that and somehow missed it.
Note that's the wrong path for dynamic libraries, though; those go in
/usr/x86_64-w64-mingw32/bin/. We'll need a way to find those. Maybe
hardcoding a list won't be too painful, though...
In Debian they go in /usr/x86_64-w64-mingw32/lib currently, which is why
the .pc file points there. But as you point out, that doesn’t help at runtime.
Also, having pkg-config support doesn’t really help with a parallel set of DLLs, does it?
For what it's worth, the current proposed solution (which has the
support of the Wine maintainer) involves source imports and submodules. >>>> There's probably room for changing our approach even after things are
committed, but I'd still like to get early feedback from distributions, >>>> and make sure that their interests are accurately represented, before we >>>> commit.
Realistically, I think this is the best approach for now. As Debian adds >>> support for PE libraries, we can replace the source imports in the Wine
source code; this is done in many other Debian packages for projects which >>> vendor dependencies.
I still think this is true. With requirement for a Wine-specific set of DLLs, improving the situation in Debian would involve supporting source build-dependencies, i.e. being able to say that a given package wants access to the source code of another package. That’s something that’s been brought
up previously, and is worked around by providing binary packages containing package source code (e.g. binutils-source, gcc-11-source etc.), but isn’t really upstream’s concern, in that it’s a Debian implementation detail.
Going back to your original three questions, I think that the best approach for you as upstream is to focus on providing a complete set of source code (including dependencies) which works, and to make it friendlier to distributions, make the build process capable of handling alternative locations for the dependencies’ source code or even build artifacts. (This has a number of knock-on effects — in particular, you should ensure that the
upstream source code for all your dependencies works with Wine, i.e. that Wine doesn’t require Wine-specific patches to any of its dependencies.)
Given how varied MinGW-w64 handling is in different distributions, pushing things further risks making it easier for one distribution and harder for the others...
On 9/6/21 1:57 AM, Stephen Kitt wrote:
On Sun, 5 Sep 2021 12:14:47 -0500, Zebediah Figura <zfigura@codeweavers.com>
wrote:mail,
On 9/5/21 11:19 AM, Stephen Kitt wrote:
On Sat, 4 Sep 2021 20:17:53 -0500, Zebediah Figura
<zfigura@codeweavers.com> wrote:
I'm a contributor to the Wine project. To summarize the following
asWine needs special versions of some of its normal dependencies, such
I'mlibfreetype and libgnutls, built using the MinGW cross-compiler, and
feedbacksending out a mail to major distributions in order to get some
somefrom our packagers on how these should be built and packaged.
For a long time Wine has built all of its Win32 libraries (DLLs and
EXEs) as ELF binaries. For various reasons related to application
compatibility, we have started building our binaries as PE instead,
using the MinGW cross-compiler. It is our intent to expand this to
toof our dependencies as well. The list of dependencies that we intend
dependencies,build using MinGW is not quite fixed yet, but we expect it to include >>>> and be mostly limited to the following:
* libvkd3d
* libFAudio
* libgnutls
* zlib (currently included via manual source import)
* libmpg123
* libgsm
* libpng
* libjpeg-turbo
* libtiff
* libfreetype
* liblcms2
* jxrlib
and dependencies of the above packages (not including CRT
dependencieswhich Wine provides).
There is currently some internal discussion about how these
seeshould be built and linked. There are essentially three questions I
significantthat need to be resolved, and while these resolutions have a
ourimpact on the Wine building and development process, they also have an >>>> impact on distributions, and accordingly I'd like to get input from
intopackagers to ensure that their considerations are accurately taken
theyaccount.
(1) Should we build via source import, or link statically, or
dynamically?
Source imports are dispreferred by Debian [1], on the grounds that
hardercause duplication of libraries on disk and in memory, and make it
discussionto handle security updates. They also make building and bisecting
harder. Static libraries don't seem to be expressly discouraged, but >>>> share most of the same downsides (see also [2]).
Note however that if they are linked dynamically, we need to make sure >>>> that we load our packages instead of MinGW builds of open-source
libraries with applications ship with. There's some internal
probablyabout whether this is possible while using "stock" builds of MinGW
libraries, but, due to the way the Win32 loader works, we will
Noteneed to compile each library, and its dependencies, with a separate, >>>> wine-specific name, e.g. "libwinefreetype-6.dll" and
"libwineharfbuzz.dll". For a detailed explantion see footnote [3].
patchthat all we actually need to change is the name; we don't need to
the source.
Assuming Debian provides the dependencies (which is currently true only >>> for zlib), this could be handled in packaging by providing symlinks,
couldn’t it? Not in the Wine prefixes, but elsewhere.
Almost :-/
Copying/symlinking libfreetype-1.dll to libwinefreetype-1.dll is easy.
The problem is that libwinefreetype-1.dll is still going to link to
libharfbuzz-1.dll, but we need it to link to libwineharfbuzz-1.dll.
Ah yes, I hadn’t thought it through. So really Wine needs its ownparallel
ecosystem of DLLs in any case, which effectively means building themalong
with Wine (from Wine’s perspective, ignoring distribution constraints and preferences).
This also works in Debian:
$ sudo apt install libz-mingw-w64-dev mingw-w64-tools
$ x86_64-w64-mingw32-pkg-config --libs zlib
-L/usr/x86_64-w64-mingw32/lib -lz
Ah, cool, I looked for that and somehow missed it.
Note that's the wrong path for dynamic libraries, though; those go in
/usr/x86_64-w64-mingw32/bin/. We'll need a way to find those. Maybe
hardcoding a list won't be too painful, though...
In Debian they go in /usr/x86_64-w64-mingw32/lib currently, which is why the .pc file points there. But as you point out, that doesn’t help at runtime.
Also, having pkg-config support doesn’t really help with a parallel setof
DLLs, does it?
I mean... eh. In theory you could say "here's a library called libwinefreetype, and to find it you do `pkg-config --cflags --libs libwinefreetype`", but that does strike me as more than a little janky.
submodules.For what it's worth, the current proposed solution (which has the
support of the Wine maintainer) involves source imports and
distributions,There's probably room for changing our approach even after things are >>>> committed, but I'd still like to get early feedback from
weand make sure that their interests are accurately represented, before
addscommit.
Realistically, I think this is the best approach for now. As Debian
support for PE libraries, we can replace the source imports in the Wine >>> source code; this is done in many other Debian packages for projects which
vendor dependencies.
I still think this is true. With requirement for a Wine-specific set ofDLLs,
improving the situation in Debian would involve supporting source build-dependencies, i.e. being able to say that a given package wantsaccess
to the source code of another package. That’s something that’s beenbrought
up previously, and is worked around by providing binary packagescontaining
package source code (e.g. binutils-source, gcc-11-source etc.), but isn’t really upstream’s concern, in that it’s a Debian implementation detail.
Going back to your original three questions, I think that the bestapproach
for you as upstream is to focus on providing a complete set of sourcecode
(including dependencies) which works, and to make it friendlier to distributions, make the build process capable of handling alternative locations for the dependencies’ source code or even build artifacts.(This
has a number of knock-on effects — in particular, you should ensure thatthe
upstream source code for all your dependencies works with Wine, i.e. that Wine doesn’t require Wine-specific patches to any of its dependencies.)
Given how varied MinGW-w64 handling is in different distributions,pushing
things further risks making it easier for one distribution and harderfor the
others...
Thanks for the detailed response!
It's probably worth pointing out that:
(1) if we use static linking, we should be able to use distribution
libraries unmodified. Of course, static linking comes with its own downsides...
(2) Like I mentioned or hinted at in my original email, it *may* be
possible to use distribution dynamic libraries unmodified. It's not
clear that it can be done without breaking compatibility, but if it can, would that change anything?
ἔρρωσθε,
Zebediah
Le lun. 6 sept. 2021 à 18:36, Zebediah Figura <zfigura@codeweavers.com> a écrit :
On 9/6/21 1:57 AM, Stephen Kitt wrote:
On Sun, 5 Sep 2021 12:14:47 -0500, Zebediah Figura <zfigura@codeweavers.com>
wrote:mail,
On 9/5/21 11:19 AM, Stephen Kitt wrote:
On Sat, 4 Sep 2021 20:17:53 -0500, Zebediah Figura
<zfigura@codeweavers.com> wrote:
I'm a contributor to the Wine project. To summarize the following
asWine needs special versions of some of its normal dependencies, such
I'mlibfreetype and libgnutls, built using the MinGW cross-compiler, and
feedbacksending out a mail to major distributions in order to get some
somefrom our packagers on how these should be built and packaged.
For a long time Wine has built all of its Win32 libraries (DLLs and >>>>>> EXEs) as ELF binaries. For various reasons related to application
compatibility, we have started building our binaries as PE instead, >>>>>> using the MinGW cross-compiler. It is our intent to expand this to
toof our dependencies as well. The list of dependencies that we intend
dependencies,build using MinGW is not quite fixed yet, but we expect it to include >>>>>> and be mostly limited to the following:
* libvkd3d
* libFAudio
* libgnutls
* zlib (currently included via manual source import)
* libmpg123
* libgsm
* libpng
* libjpeg-turbo
* libtiff
* libfreetype
* liblcms2
* jxrlib
and dependencies of the above packages (not including CRT
dependencieswhich Wine provides).
There is currently some internal discussion about how these
seeshould be built and linked. There are essentially three questions I
significantthat need to be resolved, and while these resolutions have a
ourimpact on the Wine building and development process, they also have an >>>>>> impact on distributions, and accordingly I'd like to get input from
intopackagers to ensure that their considerations are accurately taken
theyaccount.
(1) Should we build via source import, or link statically, or
dynamically?
Source imports are dispreferred by Debian [1], on the grounds that
hardercause duplication of libraries on disk and in memory, and make it
discussionto handle security updates. They also make building and bisecting
harder. Static libraries don't seem to be expressly discouraged, but >>>>>> share most of the same downsides (see also [2]).
Note however that if they are linked dynamically, we need to make sure >>>>>> that we load our packages instead of MinGW builds of open-source
libraries with applications ship with. There's some internal
probablyabout whether this is possible while using "stock" builds of MinGW >>>>>> libraries, but, due to the way the Win32 loader works, we will
Noteneed to compile each library, and its dependencies, with a separate, >>>>>> wine-specific name, e.g. "libwinefreetype-6.dll" and
"libwineharfbuzz.dll". For a detailed explantion see footnote [3].
patchthat all we actually need to change is the name; we don't need to
parallelthe source.
Assuming Debian provides the dependencies (which is currently true only >>>>> for zlib), this could be handled in packaging by providing symlinks, >>>>> couldn’t it? Not in the Wine prefixes, but elsewhere.
Almost :-/
Copying/symlinking libfreetype-1.dll to libwinefreetype-1.dll is easy. >>>> The problem is that libwinefreetype-1.dll is still going to link to
libharfbuzz-1.dll, but we need it to link to libwineharfbuzz-1.dll.
Ah yes, I hadn’t thought it through. So really Wine needs its own
ecosystem of DLLs in any case, which effectively means building themalong
with Wine (from Wine’s perspective, ignoring distribution constraints and >>> preferences).
This also works in Debian:
$ sudo apt install libz-mingw-w64-dev mingw-w64-tools
$ x86_64-w64-mingw32-pkg-config --libs zlib
-L/usr/x86_64-w64-mingw32/lib -lz
Ah, cool, I looked for that and somehow missed it.
Note that's the wrong path for dynamic libraries, though; those go in
/usr/x86_64-w64-mingw32/bin/. We'll need a way to find those. Maybe
hardcoding a list won't be too painful, though...
In Debian they go in /usr/x86_64-w64-mingw32/lib currently, which is why >>> the .pc file points there. But as you point out, that doesn’t help at
runtime.
Also, having pkg-config support doesn’t really help with a parallel set >> of
DLLs, does it?
I mean... eh. In theory you could say "here's a library called
libwinefreetype, and to find it you do `pkg-config --cflags --libs
libwinefreetype`", but that does strike me as more than a little janky.
submodules.
For what it's worth, the current proposed solution (which has the
support of the Wine maintainer) involves source imports and
distributions,There's probably room for changing our approach even after things are >>>>>> committed, but I'd still like to get early feedback from
weand make sure that their interests are accurately represented, before
addscommit.
Realistically, I think this is the best approach for now. As Debian
whichsupport for PE libraries, we can replace the source imports in the Wine >>>>> source code; this is done in many other Debian packages for projects
DLLs,vendor dependencies.
I still think this is true. With requirement for a Wine-specific set of
improving the situation in Debian would involve supporting sourceaccess
build-dependencies, i.e. being able to say that a given package wants
to the source code of another package. That’s something that’s beenbrought
up previously, and is worked around by providing binary packagescontaining
package source code (e.g. binutils-source, gcc-11-source etc.), but isn’t >>> really upstream’s concern, in that it’s a Debian implementation detail. >>>approach
Going back to your original three questions, I think that the best
for you as upstream is to focus on providing a complete set of sourcecode
(including dependencies) which works, and to make it friendlier to(This
distributions, make the build process capable of handling alternative
locations for the dependencies’ source code or even build artifacts.
has a number of knock-on effects — in particular, you should ensure that >> thepushing
upstream source code for all your dependencies works with Wine, i.e. that >>> Wine doesn’t require Wine-specific patches to any of its dependencies.) >>>
Given how varied MinGW-w64 handling is in different distributions,
things further risks making it easier for one distribution and harderfor the
others...
Thanks for the detailed response!
It's probably worth pointing out that:
(1) if we use static linking, we should be able to use distribution
libraries unmodified. Of course, static linking comes with its own
downsides...
(2) Like I mentioned or hinted at in my original email, it *may* be
possible to use distribution dynamic libraries unmodified. It's not
clear that it can be done without breaking compatibility, but if it can,
would that change anything?
Yes i really prefer (2) it means we could use multiarch easilly....
Please try to document thé blocking point.
The basic problem is that applications can and often do ship with PE
builds of cross-platform libraries. These libraries can be ahead of
Wine's system libraries, behind them, or even built with custom patches. Accordingly we really don't want to load "our" freetype in place of
"their" freetype, or "theirs" in place of "ours". But because of the way
the Win32 loader works, you can generally only have one DLL of a given
name loaded in a process, and further attempts to dlopen() [as it were] "libfreetype-6.dll" will return the handle to the already loaded
library, potentially breaking either Wine or the application.
On Mon, Sep 6, 2021 at 9:54 PM Zebediah Figura wrote:I agree with pabs, implementing name space is a good solution
The basic problem is that applications can and often do ship with PE
builds of cross-platform libraries. These libraries can be ahead of
Wine's system libraries, behind them, or even built with custom patches. Accordingly we really don't want to load "our" freetype in place of
"their" freetype, or "theirs" in place of "ours". But because of the way the Win32 loader works, you can generally only have one DLL of a given
name loaded in a process, and further attempts to dlopen() [as it were] "libfreetype-6.dll" will return the handle to the already loaded
library, potentially breaking either Wine or the application.
I don't know the details here, but my immediate thought when reading
this is that you need some kind of namespace. I then found that linker namespaces are a thing, perhaps they would provide the solution for
you. It sounds like the OpenGL shim loader solution listed on the
glibc wiki might work for your use-case. Or perhaps the LD_AUDIT
feature would also work.
https://www.sourceware.org/glibc/wiki/LinkerNamespaces
Le mardi 7 septembre 2021, 00:44:31 UTC Paul Wise a écrit :
On Mon, Sep 6, 2021 at 9:54 PM Zebediah Figura wrote:I agree with pabs, implementing name space is a good solution
The basic problem is that applications can and often do ship with PE
builds of cross-platform libraries. These libraries can be ahead of
Wine's system libraries, behind them, or even built with custom patches. >>> Accordingly we really don't want to load "our" freetype in place of
"their" freetype, or "theirs" in place of "ours". But because of the way >>> the Win32 loader works, you can generally only have one DLL of a given
name loaded in a process, and further attempts to dlopen() [as it were]
"libfreetype-6.dll" will return the handle to the already loaded
library, potentially breaking either Wine or the application.
I don't know the details here, but my immediate thought when reading
this is that you need some kind of namespace. I then found that linker
namespaces are a thing, perhaps they would provide the solution for
you. It sounds like the OpenGL shim loader solution listed on the
glibc wiki might work for your use-case. Or perhaps the LD_AUDIT
feature would also work.
https://www.sourceware.org/glibc/wiki/LinkerNamespaces
and will allow to be distrib friendly.
Moreover it will be best if marking a library as wine system one could be done
post build. We will prefer to not modify upstream (like libpng) source.
It is easier for us to apply small patch to upstream, or better in your case to add a post link linker script or even some compiler flag that will add some
notes section to dll in order to mark it as use namespace system, or so on.
The problem on windows side is well known as dll hell
Renaming of the lib could be done post install. Moreover a crazy idea why not renaming system dll instead of libpng-2.0.0.dll as libpng-2.0.0.sll (note the version is manadaroty) Therefore we could use upstream source. Then add a small linker script that will rewrite the depends of libpng-2.0.0.sll to .sll file (aka binary sed s/.dll/.sll/g).
then add a small wraper (shim) like said pab that load the .sll library, the best pratice will be a command tool that take the name of the sll and create magically the dll
Therefore from distrib point of side we only changes the way we build the package, it could be automated, it is scalable and it does not need to patch package by package. Only to add some linker script magic
On 9/7/21 5:16 AM, Bastien Roucariès wrote:
Le mardi 7 septembre 2021, 00:44:31 UTC Paul Wise a écrit :patches.
On Mon, Sep 6, 2021 at 9:54 PM Zebediah Figura wrote:
The basic problem is that applications can and often do ship with PE
builds of cross-platform libraries. These libraries can be ahead of
Wine's system libraries, behind them, or even built with custom
wayAccordingly we really don't want to load "our" freetype in place of
"their" freetype, or "theirs" in place of "ours". But because of the
I agree with pabs, implementing name space is a good solutionthe Win32 loader works, you can generally only have one DLL of a given >>> name loaded in a process, and further attempts to dlopen() [as it were] >>> "libfreetype-6.dll" will return the handle to the already loaded
library, potentially breaking either Wine or the application.
I don't know the details here, but my immediate thought when reading
this is that you need some kind of namespace. I then found that linker
namespaces are a thing, perhaps they would provide the solution for
you. It sounds like the OpenGL shim loader solution listed on the
glibc wiki might work for your use-case. Or perhaps the LD_AUDIT
feature would also work.
https://www.sourceware.org/glibc/wiki/LinkerNamespaces
and will allow to be distrib friendly.
Moreover it will be best if marking a library as wine system one couldbe done
post build. We will prefer to not modify upstream (like libpng) source.
It is easier for us to apply small patch to upstream, or better in yourcase
to add a post link linker script or even some compiler flag that willadd some
notes section to dll in order to mark it as use namespace system, or soon.
The problem on windows side is well known as dll hell
Renaming of the lib could be done post install. Moreover a crazy ideawhy not
renaming system dll instead of libpng-2.0.0.dll as libpng-2.0.0.sll(note the
version is manadaroty) Therefore we could use upstream source. Then adda
small linker script that will rewrite the depends of libpng-2.0.0.sllto
.sll file (aka binary sed s/.dll/.sll/g).
then add a small wraper (shim) like said pab that load the .sll library,the
best pratice will be a command tool that take the name of the sll andcreate
magically the dll
Therefore from distrib point of side we only changes the way we build the package, it could be automated, it is scalable and it does not need topatch
package by package. Only to add some linker script magic
The problem isn't particularly about detecting what is and isn't a
system library; I think we can come up with a reliable heuristic for
that, without even needing linker namespaces or anything.
The outstanding problem seems to be more about potentially breaking applications because they see two identically named DLLs loaded in the
same process. Applications can and do trawl the internal loader state, although the Win32 loader also exposes some APIs so they don't even have
to.
It may also be that the aforementioned heuristic is too hacky to be
accepted upstream into Wine.
I disagree.
Le mar. 7 sept. 2021 à 17:48, Zebediah Figura <zfigura@codeweavers.com> a écrit :
On 9/7/21 5:16 AM, Bastien Roucariès wrote:I do not propose to change the loader, i propose to change the link or
Le mardi 7 septembre 2021, 00:44:31 UTC Paul Wise a écrit :patches.
On Mon, Sep 6, 2021 at 9:54 PM Zebediah Figura wrote:
The basic problem is that applications can and often do ship with PE
builds of cross-platform libraries. These libraries can be ahead of
Wine's system libraries, behind them, or even built with custom
wayAccordingly we really don't want to load "our" freetype in place of
"their" freetype, or "theirs" in place of "ours". But because of the
were]the Win32 loader works, you can generally only have one DLL of a given >> >>> name loaded in a process, and further attempts to dlopen() [as it
be doneI agree with pabs, implementing name space is a good solution"libfreetype-6.dll" will return the handle to the already loaded
library, potentially breaking either Wine or the application.
I don't know the details here, but my immediate thought when reading
this is that you need some kind of namespace. I then found that linker
namespaces are a thing, perhaps they would provide the solution for
you. It sounds like the OpenGL shim loader solution listed on the
glibc wiki might work for your use-case. Or perhaps the LD_AUDIT
feature would also work.
https://www.sourceware.org/glibc/wiki/LinkerNamespaces
and will allow to be distrib friendly.
Moreover it will be best if marking a library as wine system one could
post build. We will prefer to not modify upstream (like libpng) source.case
It is easier for us to apply small patch to upstream, or better in your
to add a post link linker script or even some compiler flag that willadd some
notes section to dll in order to mark it as use namespace system, or soon.
why not
The problem on windows side is well known as dll hell
Renaming of the lib could be done post install. Moreover a crazy idea
renaming system dll instead of libpng-2.0.0.dll as libpng-2.0.0.sll(note the
version is manadaroty) Therefore we could use upstream source. Thenadd a
small linker script that will rewrite the depends of libpng-2.0.0.sllto
.sll file (aka binary sed s/.dll/.sll/g).library, the
then add a small wraper (shim) like said pab that load the .sll
best pratice will be a command tool that take the name of the sll andcreate
magically the dllthe
Therefore from distrib point of side we only changes the way we build
package, it could be automated, it is scalable and it does not need topatch
package by package. Only to add some linker script magic
The problem isn't particularly about detecting what is and isn't a
system library; I think we can come up with a reliable heuristic for
that, without even needing linker namespaces or anything.
The outstanding problem seems to be more about potentially breaking
applications because they see two identically named DLLs loaded in the
same process. Applications can and do trawl the internal loader state,
although the Win32 loader also exposes some APIs so they don't even have
to.
It may also be that the aforementioned heuristic is too hacky to be
accepted upstream into Wine.
post link stage of system library. You could rename the lib and change the dépend to new name after build. So you will have an unique name per system lib without changing the way you build these lib.
It is équivalent to use a namespace resolved at late build time.
I disagree.
Le mar. 7 sept. 2021 à 17:48, Zebediah Figura <zfigura@codeweavers.com> a écrit :
On 9/7/21 5:16 AM, Bastien Roucariès wrote:I do not propose to change the loader, i propose to change the link or post link stage of system library. You could rename the lib and change the
Le mardi 7 septembre 2021, 00:44:31 UTC Paul Wise a écrit :patches.
On Mon, Sep 6, 2021 at 9:54 PM Zebediah Figura wrote:
The basic problem is that applications can and often do ship with PE >>>>> builds of cross-platform libraries. These libraries can be ahead of
Wine's system libraries, behind them, or even built with custom
wayAccordingly we really don't want to load "our" freetype in place of
"their" freetype, or "theirs" in place of "ours". But because of the
be doneI agree with pabs, implementing name space is a good solutionthe Win32 loader works, you can generally only have one DLL of a given >>>>> name loaded in a process, and further attempts to dlopen() [as it were] >>>>> "libfreetype-6.dll" will return the handle to the already loaded
library, potentially breaking either Wine or the application.
I don't know the details here, but my immediate thought when reading
this is that you need some kind of namespace. I then found that linker >>>> namespaces are a thing, perhaps they would provide the solution for
you. It sounds like the OpenGL shim loader solution listed on the
glibc wiki might work for your use-case. Or perhaps the LD_AUDIT
feature would also work.
https://www.sourceware.org/glibc/wiki/LinkerNamespaces
and will allow to be distrib friendly.
Moreover it will be best if marking a library as wine system one could
post build. We will prefer to not modify upstream (like libpng) source.case
It is easier for us to apply small patch to upstream, or better in your
to add a post link linker script or even some compiler flag that willadd some
notes section to dll in order to mark it as use namespace system, or soon.
why not
The problem on windows side is well known as dll hell
Renaming of the lib could be done post install. Moreover a crazy idea
renaming system dll instead of libpng-2.0.0.dll as libpng-2.0.0.sll(note the
version is manadaroty) Therefore we could use upstream source. Then adda
small linker script that will rewrite the depends of libpng-2.0.0.sllto
.sll file (aka binary sed s/.dll/.sll/g).the
then add a small wraper (shim) like said pab that load the .sll library,
best pratice will be a command tool that take the name of the sll andcreate
magically the dllpatch
Therefore from distrib point of side we only changes the way we build the >>> package, it could be automated, it is scalable and it does not need to
package by package. Only to add some linker script magic
The problem isn't particularly about detecting what is and isn't a
system library; I think we can come up with a reliable heuristic for
that, without even needing linker namespaces or anything.
The outstanding problem seems to be more about potentially breaking
applications because they see two identically named DLLs loaded in the
same process. Applications can and do trawl the internal loader state,
although the Win32 loader also exposes some APIs so they don't even have
to.
It may also be that the aforementioned heuristic is too hacky to be
accepted upstream into Wine.
dépend to new name after build. So you will have an unique name per system lib without changing the way you build these lib.
It is équivalent to use a namespace resolved at late build time.
The outstanding problem seems to be more about potentially breaking applications because they see two identically named DLLs loaded in the
same process. Applications can and do trawl the internal loader state, although the Win32 loader also exposes some APIs so they don't even have to.
On 9/7/21 12:05 PM, Bastien ROUCARIES wrote:
I disagree.
Le mar. 7 sept. 2021 à 17:48, Zebediah Figura <zfigura@codeweavers.com> a écrit :
On 9/7/21 5:16 AM, Bastien Roucariès wrote:I do not propose to change the loader, i propose to change the link or post link stage of system library. You could rename the lib and change the dépend to new name after build. So you will have an unique name per system lib without changing the way you build these lib.
Le mardi 7 septembre 2021, 00:44:31 UTC Paul Wise a écrit :patches.
On Mon, Sep 6, 2021 at 9:54 PM Zebediah Figura wrote:
The basic problem is that applications can and often do ship with PE >>>>> builds of cross-platform libraries. These libraries can be ahead of >>>>> Wine's system libraries, behind them, or even built with custom
wayAccordingly we really don't want to load "our" freetype in place of >>>>> "their" freetype, or "theirs" in place of "ours". But because of the
be doneI agree with pabs, implementing name space is a good solutionthe Win32 loader works, you can generally only have one DLL of a given >>>>> name loaded in a process, and further attempts to dlopen() [as it were] >>>>> "libfreetype-6.dll" will return the handle to the already loaded
library, potentially breaking either Wine or the application.
I don't know the details here, but my immediate thought when reading >>>> this is that you need some kind of namespace. I then found that linker >>>> namespaces are a thing, perhaps they would provide the solution for
you. It sounds like the OpenGL shim loader solution listed on the
glibc wiki might work for your use-case. Or perhaps the LD_AUDIT
feature would also work.
https://www.sourceware.org/glibc/wiki/LinkerNamespaces
and will allow to be distrib friendly.
Moreover it will be best if marking a library as wine system one could
post build. We will prefer to not modify upstream (like libpng) source. >>>add some
It is easier for us to apply small patch to upstream, or better in your >> case
to add a post link linker script or even some compiler flag that will
notes section to dll in order to mark it as use namespace system, or so >> on.why not
The problem on windows side is well known as dll hell
Renaming of the lib could be done post install. Moreover a crazy idea
renaming system dll instead of libpng-2.0.0.dll as libpng-2.0.0.sll(note the
version is manadaroty) Therefore we could use upstream source. Then add >> ato
small linker script that will rewrite the depends of libpng-2.0.0.sll
.sll file (aka binary sed s/.dll/.sll/g).create
then add a small wraper (shim) like said pab that load the .sll library, >> the
best pratice will be a command tool that take the name of the sll and
magically the dllpatch
Therefore from distrib point of side we only changes the way we build the >>> package, it could be automated, it is scalable and it does not need to
package by package. Only to add some linker script magic
The problem isn't particularly about detecting what is and isn't a
system library; I think we can come up with a reliable heuristic for
that, without even needing linker namespaces or anything.
The outstanding problem seems to be more about potentially breaking
applications because they see two identically named DLLs loaded in the
same process. Applications can and do trawl the internal loader state,
although the Win32 loader also exposes some APIs so they don't even have >> to.
It may also be that the aforementioned heuristic is too hacky to be
accepted upstream into Wine.
It is équivalent to use a namespace resolved at late build time.
Ah, so what you're proposing is that we do some sort of objcopy-likeYes sort of obcopy patching under linux it is called patchelf...
patching at runtime, e.g. when copying the dependency into the prefix.
It probably won't be easy, but it might be more feasible than modifying
the loader. I'll look into it...
On Tue, 2021-09-07 at 10:48 -0500, Zebediah Figura wrote:Yes but it does not work for PE because win native application do
The outstanding problem seems to be more about potentially breaking applications because they see two identically named DLLs loaded in the
same process. Applications can and do trawl the internal loader state, although the Win32 loader also exposes some APIs so they don't even have to.
I might be wrong but I got the impression that runtime linker
namespaces & LD_AUDIT are designed to solve this exact issue.
--
bye,
pabs
https://wiki.debian.org/PaulWise
Le mar. 7 sept. 2021 à 21:16, Zebediah Figura
<zfigura@codeweavers.com> a écrit :
On 9/7/21 12:05 PM, Bastien ROUCARIES wrote:
I disagree.
Le mar. 7 sept. 2021 à 17:48, Zebediah Figura <zfigura@codeweavers.com> a
écrit :
On 9/7/21 5:16 AM, Bastien Roucariès wrote:I do not propose to change the loader, i propose to change the link or post
Le mardi 7 septembre 2021, 00:44:31 UTC Paul Wise a écrit :patches.
On Mon, Sep 6, 2021 at 9:54 PM Zebediah Figura wrote:
The basic problem is that applications can and often do ship with PE >>>>> builds of cross-platform libraries. These libraries can be ahead of >>>>> Wine's system libraries, behind them, or even built with custom
(note theI agree with pabs, implementing name space is a good solutionAccordingly we really don't want to load "our" freetype in place of >>>>> "their" freetype, or "theirs" in place of "ours". But because of the >> way
the Win32 loader works, you can generally only have one DLL of a given
name loaded in a process, and further attempts to dlopen() [as it were]
"libfreetype-6.dll" will return the handle to the already loaded >>>>> library, potentially breaking either Wine or the application.
I don't know the details here, but my immediate thought when reading >>>> this is that you need some kind of namespace. I then found that linker >>>> namespaces are a thing, perhaps they would provide the solution for >>>> you. It sounds like the OpenGL shim loader solution listed on the
glibc wiki might work for your use-case. Or perhaps the LD_AUDIT
feature would also work.
https://www.sourceware.org/glibc/wiki/LinkerNamespaces
and will allow to be distrib friendly.
Moreover it will be best if marking a library as wine system one could >> be done
post build. We will prefer to not modify upstream (like libpng) source. >>>
It is easier for us to apply small patch to upstream, or better in your >> case
to add a post link linker script or even some compiler flag that will >> add some
notes section to dll in order to mark it as use namespace system, or so >> on.
The problem on windows side is well known as dll hell
Renaming of the lib could be done post install. Moreover a crazy idea >> why not
renaming system dll instead of libpng-2.0.0.dll as libpng-2.0.0.sll
version is manadaroty) Therefore we could use upstream source. Then adda
small linker script that will rewrite the depends of libpng-2.0.0.sll >> tothe
.sll file (aka binary sed s/.dll/.sll/g).
then add a small wraper (shim) like said pab that load the .sll library,
best pratice will be a command tool that take the name of the sll and >> create
magically the dll
Therefore from distrib point of side we only changes the way we build the
package, it could be automated, it is scalable and it does not need to >> patch
package by package. Only to add some linker script magic
The problem isn't particularly about detecting what is and isn't a
system library; I think we can come up with a reliable heuristic for
that, without even needing linker namespaces or anything.
The outstanding problem seems to be more about potentially breaking
applications because they see two identically named DLLs loaded in the >> same process. Applications can and do trawl the internal loader state, >> although the Win32 loader also exposes some APIs so they don't even have >> to.
It may also be that the aforementioned heuristic is too hacky to be
accepted upstream into Wine.
link stage of system library. You could rename the lib and change the dépend to new name after build. So you will have an unique name per system
lib without changing the way you build these lib.
It is équivalent to use a namespace resolved at late build time.
Ah, so what you're proposing is that we do some sort of objcopy-like patching at runtime, e.g. when copying the dependency into the prefix.
It probably won't be easy, but it might be more feasible than modifyingYes sort of obcopy patching under linux it is called patchelf...
the loader. I'll look into it...
If it work, it could be latter done in memory like paul wise suggest implementing namespace and LT_AUDIT (but it is a long term goal)
1. capture library loading
2. do the patching at loading time (using the same code then patchelf
for pe replacing file read/write by mmap operation)
And voila
bastien
Simon, do you think you could implement a version of libcapasule for PE object ?
On Wed, 08 Sep 2021 at 07:31:59 +0000, Bastien ROUCARIES wrote:
Simon, do you think you could implement a version of libcapasule for PE object ?
Given that libcapsule is very glibc- and ELF-specific, doesn't work
properly without new glibc feature work that isn't in Debian yet (I've
lost track of whether/when it got merged upstream, it might be in 2.34)
and wasn't written by me, the answer is likely to be no.
As far as I understand it, the PE loader used for Wine is part of Wine,
so it has total control over the libraries that it loads and how it loads them. This means that if Wine developers (the experts on this codebase)
have decided a libcapsule-like approach isn't feasible or isn't desirable, they are probably right.
More generally, if Wine developers wanted Wine's dependency libraries
to be in one version of reality, and Windows DLLs to be in a different version of reality, then there's a straightforward way to do that:
use ELF .so dependency libraries, like older versions of Wine did, and
don't give them a representation in the PE layer. Presumably they have
put in the work to move from ELF to PE dependency libraries because it
has some compelling advantage, rather than creating work for themselves
for no reason!
As far as I understand it, the PE loader used for Wine is part of Wine,
so it has total control over the libraries that it loads and how it loads them. This means that if Wine developers (the experts on this codebase)
have decided a libcapsule-like approach isn't feasible or isn't desirable, they are probably right.
The basic reason is that Wine is essentially trying to split itself into "PE" and "Unix" parts. The idea is that all of the "user" code is built
in PE format, and only calls into unix code by making a sort of fake syscall. The "kernel" code is built in ELF format.
There are a few basic reasons for doing this:
On 9/8/21 3:13 AM, Simon McVittie wrote:
As far as I understand it, the PE loader used for Wine is part of Wine,
so it has total control over the libraries that it loads and how it loads
them. This means that if Wine developers (the experts on this codebase)
have decided a libcapsule-like approach isn't feasible or isn't desirable, >> they are probably right.
Could the PE loader add support for some sort of namespaces and then be
told to load Wine dependencies within a Wine namespace?
Does the Microsoft Windows loader support anything like this?
Recompiling every PE library with special flags or renaming every PE
library with a Wine prefix seems a bit tedious if just the PE loader
and anything that requests Wine deps can use a namespace/capsule/etc.
On Wed, 2021-09-08 at 23:47 -0500, Zebediah Figura wrote:
Unfortunately, while thinking about the answer to this question, I
realized another snag, which I think really does make using
identically-named dynamic libraries impossible: if system library A
loads system library B dynamically, i.e. does the equivalent of a
dlopen(), we have no way of knowing which namespace to apply to it.
[AFAICT dlmopen() has this problem too, and just isn't meant for that
use case.] Even rewriting the import table doesn't help here.
While loading library A into a namespace Foo, the loader could set a
process global variable containing the current namespace to load into,
and then have the dlopen equivalent use namespace Foo in preference to
the default namespace for the process. Then when library B is loaded it
will go into namespace Foo. Once loading library B and A has completed
the loader could revert to using the default namespace.
Unfortunately, while thinking about the answer to this question, I
realized another snag, which I think really does make using identically-named dynamic libraries impossible: if system library A
loads system library B dynamically, i.e. does the equivalent of a
dlopen(), we have no way of knowing which namespace to apply to it.
[AFAICT dlmopen() has this problem too, and just isn't meant for that
use case.] Even rewriting the import table doesn't help here.
Right, but we don't have any guarantee that library A will load library
B in its constructor routines. In fact, if it's loading library B dynamically, it's probably not doing that.
On Thu, 2021-09-09 at 00:39 -0500, Zebediah Figura wrote:
Right, but we don't have any guarantee that library A will load library
B in its constructor routines. In fact, if it's loading library B
dynamically, it's probably not doing that.
Can the loader tell which library asked it to load a library?
Then the namespace could be inherited from the parent library.
Unfortunately, no. We have no way of knowing the caller.
On Thu, 2021-09-09 at 00:59 -0500, Zebediah Figura wrote:
Unfortunately, no. We have no way of knowing the caller.
Can the PE loading mechanism do something like inject a fake dlopen
function available only in the Wine namespace that just passes the Wine namespace to the dlmopen function? Or the same but for each library?
--
bye,
pabs
https://wiki.debian.org/PaulWise
The problem is that windows apps particularly games try to check if
mapped ram exec pages are from dll from disk and not modified in
memory.
You propose:
Hello all,
I'm a contributor to the Wine project. To summarize the following mail, Wine needs special versions of some of its normal dependencies, such as libfreetype and libgnutls, built using the MinGW cross-compiler, and I'm sending out a mail to major distributions in order to get some feedback from our packagers on how these should be built and packaged.
For a long time Wine has built all of its Win32 libraries (DLLs and EXEs) as ELF binaries. For various reasons related to application compatibility, we have started building our binaries as PE instead, using the MinGW cross-compiler. It is our intent to expand this to some of our dependencies as well. The list of dependencies that we intend to build using MinGW is not quite fixed yet, but we expect it to include and be mostly limited to the following:
* libvkd3d
* libFAudio
* libgnutls
* zlib (currently included via manual source import)
* libmpg123
* libgsm
* libpng
* libjpeg-turbo
* libtiff
* libfreetype
* liblcms2
* jxrlib
and dependencies of the above packages (not including CRT dependencies,
which Wine provides).
...
Accordingly, although static linking and source imports are generally disprefered, it may quite likely be preferable in our case. We don't get the benefits of on-disk deduplication, since Wine is essentially the only piece of software which needs these libraries.
...
On Sat, Sep 04, 2021 at 08:17:53PM -0500, Zebediah Figura wrote:
Hello all,
I'm a contributor to the Wine project. To summarize the following mail,Wine
needs special versions of some of its normal dependencies, such as libfreetype and libgnutls, built using the MinGW cross-compiler, and I'm sending out a mail to major distributions in order to get some feedbackfrom
our packagers on how these should be built and packaged.
For a long time Wine has built all of its Win32 libraries (DLLs andEXEs) as
ELF binaries. For various reasons related to application compatibility,we
have started building our binaries as PE instead, using the MinGW cross-compiler. It is our intent to expand this to some of ourdependencies
as well. The list of dependencies that we intend to build using MinGW isnot
quite fixed yet, but we expect it to include and be mostly limited to the following:
* libvkd3d
* libFAudio
* libgnutls
* zlib (currently included via manual source import)
* libmpg123
* libgsm
* libpng
* libjpeg-turbo
* libtiff
* libfreetype
* liblcms2
* jxrlib
and dependencies of the above packages (not including CRT dependencies, which Wine provides).the
...
Accordingly, although static linking and source imports are generally disprefered, it may quite likely be preferable in our case. We don't get
benefits of on-disk deduplication, since Wine is essentially the onlypiece
of software which needs these libraries.
...
How are distributions supposed to provide security support for Wine?
As an example, Debian 10 that is security supported by Debian
until next summer ships Wine 4.0.
The libgnutls in Debian 10 has already been patched several times
by Debian for CVE fixes.
Having to patch several different versions of the same library
in different packages multiplies the effort required to provide
security support for a library.
cu
Adrian
...
Improve dpkg to support partial arch. I volonteer to implement none arch
but i am waiting from guillem here.
...
On Sun, Sep 05, 2021 at 08:53:49AM +0200, Bastien ROUCARIES wrote:we have at least 3 architectures that need uefi none arch...
...
Improve dpkg to support partial arch. I volonteer to implement none arch but i am waiting from guillem here.
...
There is also plenty of infrastructure on the buildd, archive and
release team sides that would likely need changes for supporting
anything like that.
A multilib based approach might be more realistic for bookworm.
What benefits would a "none arch" even bring compared to building
binary-all packages?
The ability to binNMU is the only one that comes into my mind.
cu
Adrian
Le dim. 12 sept. 2021 à 07:38, Adrian Bunk <bunk@debian.org> a écrit :
On Sun, Sep 05, 2021 at 08:53:49AM +0200, Bastien ROUCARIES wrote:
...
Improve dpkg to support partial arch. I volonteer to implement none arch but i am waiting from guillem here.
...
There is also plenty of infrastructure on the buildd, archive and
release team sides that would likely need changes for supporting
anything like that.
A multilib based approach might be more realistic for bookworm.
What benefits would a "none arch" even bring compared to building binary-all packages?we have at least 3 architectures that need uefi none arch...
The ability to binNMU is the only one that comes into my mind.
So we build three arch-all package with paterning name.
I think you misunderstand: https://wiki.debian.org/Teams/Dpkg/Spec/FreestandingArches
They are a full color gradiant between:
- freestanding arches pure cross compile without any depends except arch:all - partial cross built arch
- partial arch
- full arch
I believe the first step to get partial cross built arch is to begin
by freestanding arch.
Bastien
On Sun, Sep 12, 2021 at 01:18:11PM +0000, Bastien ROUCARIES wrote:
Le dim. 12 sept. 2021 à 07:38, Adrian Bunk <bunk@debian.org> a écrit :
On Sun, Sep 05, 2021 at 08:53:49AM +0200, Bastien ROUCARIES wrote:
...
Improve dpkg to support partial arch. I volonteer to implement none arch
but i am waiting from guillem here.
...
There is also plenty of infrastructure on the buildd, archive and
release team sides that would likely need changes for supporting
anything like that.
A multilib based approach might be more realistic for bookworm.
What benefits would a "none arch" even bring compared to building binary-all packages?we have at least 3 architectures that need uefi none arch...
The ability to binNMU is the only one that comes into my mind.
So we build three arch-all package with paterning name.
I might be misunderstanding what you are trying to do.
What release architecture would "none" packages be in the Debian
archive, and on what buildds would they be built?
Debian buildds are not cross-compiling packages for a different Debian architecture, a partial architecture would still require DSA-maintained buildds doing native building on this partial architecture.
cu
Adrian
Disclaimer: I know precisely zero of the details here nor
if the PE loader can support any of the below features.
On Fri, 2021-09-10 at 09:23 +0000, Bastien ROUCARIES wrote:
The problem is that windows apps particularly games try to check if
mapped ram exec pages are from dll from disk and not modified in
memory.
The fake dlopen could presumably be provided by an on-disk DLL?
The PE loader would just load the fake dlopen DLL first, like
glibc does when you set the LD_PRELOAD environment variable.
You propose:
I'm not proposing any on-disk symlinks, renaming DLLs or any other
changes to any PE DLLs, just loading into memory like dlmopen.
The solution I'm thinking of looks something like this:
src:libpng -> libpng16-16:$winarch -> /usr/lib/$wintriplet/libpng16.dll (requires Debian to support cross-only, partial and Windows arch types)
PE loader sets up in-memory Wine namespace.
PE loader loads into Wine namespace: /usr/lib/$windowstriplet/wine/fake-dlopen.dll
PE loader loads Wine dependencies into Wine namespace.
Some of the Wine dependencies call the Windows equivalent of dlopen,
which resolves to the fake dlopen equivalent within the namespace,
which calls the Windows equivalent of dlmopen("Wine", dll), which makes
the PE loader load those libraries into the Wine namespace.
PE loader finishes loading Wine dependencies and proceeds loading the
PE file and its dependencies into the default namespace.
Windows apps check mapped RAM pages and they show up correctly.
The main issue would be that apps could check they run in Wine.
OTOH the PE loader could also just hide the pages somehow?
--
bye,
pabs
https://wiki.debian.org/PaulWise
Le dim. 12 sept. 2021 à 14:16, Adrian Bunk <bunk@debian.org> a écrit :
On Sun, Sep 12, 2021 at 01:57:44PM +0000, Bastien ROUCARIES wrote:
I think you misunderstand: https://wiki.debian.org/Teams/Dpkg/Spec/FreestandingArches
They are a full color gradiant between:
- freestanding arches pure cross compile without any depends except arch:all
- partial cross built arch
- partial arch
- full arch
I believe the first step to get partial cross built arch is to begin
by freestanding arch.
I believe this is mostly irrelevant for the discussion about Wine.
Making Wine in Debian depend on such lofty ideas that might be available very far in the future (if ever) would imply no Wine in the next Debian releases.
Debian 12 will be released mid-2023, which is only 2 years away.
Debian 13 will be released mid-2025, which is only 4 years away.
If your partial cross built arch is ever fully implemented and supported
by Debian it would be a good idea to migrate Wine to using that.
But a realistic solution for Wine in Debian 12 has to be multilib,Yes but current solution used by wine should be compatible with
not multiarch.
multiarch. And It was since 2012 that multiarch exists. Maybe time to
speed up
bastien
On Sun, Sep 05, 2021 at 08:53:49AM +0200, Bastien ROUCARIES wrote:
...
Improve dpkg to support partial arch. I volonteer to implement none arch but i am waiting from guillem here.
...
There is also plenty of infrastructure on the buildd, archive and
release team sides that would likely need changes for supporting
anything like that.
A multilib based approach might be more realistic for bookworm.
What benefits would a "none arch" even bring compared to building
binary-all packages?
The ability to binNMU is the only one that comes into my mind.
On Sun, Sep 12, 2021 at 05:31:41PM +0200, Stephen Kitt wrote:
...
While one could imagine adding support to all the appropriate
source packages to build similar “Architecture: all” packages, that would
require convincing all the relevant maintainers,
Adding a new release architecture (partial or not) requires convincing
the ftp team and the release team.
It would also require many software changes.
How would for example the dependencies of wine:amd64 be fulfilled?
Just supporting that package dependencies might only be fulfilled by
also using packages from a different architecture would require changes
in many places, like for example the testing migration scripts that
ensure installability after migration.
and it would end up tying the testing migrations to MinGW-w64...
If this partial arch (and Wine) should be part of Debian releases,
then testing migrations would have to be tied to it in any case.
...
The buildd situation isn’t necessarily that much of an obstacle: it seems to me we could have “Windows” buildds which are really Linux amd64 systems, that cross-build for Windows.
The first obstacle is that if you want to be the first who does cross building packages on the buildds, there is likely a lot of work and bugfixing ahead for you for getting that working.
...
While one could imagine adding support to all the appropriate
source packages to build similar “Architecture: all” packages, that would require convincing all the relevant maintainers,
and it would end up tying the testing migrations to MinGW-w64...
...
The buildd situation isn’t necessarily that much of an obstacle: it seems to
me we could have “Windows” buildds which are really Linux amd64 systems, that
cross-build for Windows.
Regards,
Stephen
On Sun, 12 Sep 2021 19:20:16 +0300, Adrian Bunk <bunk@debian.org> wrote:
On Sun, Sep 12, 2021 at 05:31:41PM +0200, Stephen Kitt wrote:
...
While one could imagine adding support to all the appropriate
source packages to build similar “Architecture: all” packages, that would
require convincing all the relevant maintainers,
Adding a new release architecture (partial or not) requires convincing
the ftp team and the release team.
It would also require many software changes.
Yes, it would. However, it requires convincing a well-defined set of people *once*. If we handle Windows dependencies using “Architecture: all” packages,
we would either end up with something like Fedora’s MinGW-w64 SIG (with a complete set of independent packages), or we would end up having to convince a huge set of package maintainers over and over again.
How would for example the dependencies of wine:amd64 be fulfilled?
Just supporting that package dependencies might only be fulfilled by
also using packages from a different architecture would require changes
in many places, like for example the testing migration scripts that
ensure installability after migration.
In the same way as the none-arch packages would be. Yes, it’s a lot of work,
but it’s useful for more than just Wine, and it can be done once for all the
interested architectures.
...
Regards,
Stephen
They are a full color gradiant between:
- freestanding arches pure cross compile without any depends except arch:all - partial cross built arch
- partial arch
- full arch
I believe the first step to get partial cross built arch is to begin
by freestanding arch.
On Sun, Sep 12, 2021 at 07:03:54PM +0200, Stephen Kitt wrote:
On Sun, 12 Sep 2021 19:20:16 +0300, Adrian Bunk <bunk@debian.org> wrote:
On Sun, Sep 12, 2021 at 05:31:41PM +0200, Stephen Kitt wrote:
...
While one could imagine adding support to all the appropriate
source packages to build similar “Architecture: all” packages, that would require convincing all the relevant maintainers,
Adding a new release architecture (partial or not) requires convincing the ftp team and the release team.
It would also require many software changes.
Yes, it would. However, it requires convincing a well-defined set of
people *once*. If we handle Windows dependencies using “Architecture: all” packages, we would either end up with something like Fedora’s MinGW-w64 SIG (with a complete set of independent packages), or we would end up having to convince a huge set of package maintainers over and over again.
Based on the package list provided in the initial email in this thread
(and guessing the amount of transitive dependencies) I would have guessed that there are perhaps ~ 30 packages that would have to be touched once.
And after that things will just continue working, just like with udebs
which are in some ways similar to what is being discussed here.
How would for example the dependencies of wine:amd64 be fulfilled?
Just supporting that package dependencies might only be fulfilled by also using packages from a different architecture would require changes in many places, like for example the testing migration scripts that ensure installability after migration.
In the same way as the none-arch packages would be. Yes, it’s a lot of work, but it’s useful for more than just Wine, and it can be done once for all the interested architectures.
It is also the kind of change where it might be required that it is
fully supported in one release before it can be used by packages in the
next release, if for example any of dpkg/apt/aptitude/... in the
previous stable gives an error when seeing dependencies to another architecture.
I believe I can speak with my "main cross building porter" hat on.
On Sun, Sep 12, 2021 at 01:57:44PM +0000, Bastien ROUCARIES wrote:
They are a full color gradiant between:
- freestanding arches pure cross compile without any depends except arch:all
- partial cross built arch
- partial arch
- full arch
I believe the first step to get partial cross built arch is to begin
by freestanding arch.
Certainly not. As much as I would love to see this happen, it is very unrealistic. wine can quite simply use -$arch-cross:all packages like
very other bare metal target and switch to our pipe dream once it is
ready (if ever).
cross building is very far off becoming the thing you want it to be. It presently has a bus factor of around 1.5. Around 59% of source packages
are cross-satisfiable. Around 78% of cross-satisfiable packages are
cross buildable. In other words: less than 50% of packages are cross buildable. Beyond that, the cross porters cannot keep up with the
current rate of regressions. Fractions are falling.
To make cross building a viable thing we'd need at least three more
porters determined to make it happen. Talk doesn't make it happen.
Patches do. Let me know if you want to help. I'll get you started.
Is the process documented anywhere? Or does one simply pick a failure from http://crossqa.debian.net and figure out what’s going wrong (and hope that pulling the threads doesn’t reveal a monster...)?
...
For Wine (and even a wider MinGW-w64
ecosystem) we don’t need all that many source packages to be cross-satisfiable for the whole endeavour to be useful...
The regressions are significant though: if packages can’t stay cross-satisfiable for Debian cross-targets, there’s little hope they can stay
cross-satisfiable for Windows! This means that separate source packages are probably the only viable option.
...
Regards,
Stephen
On Mon, Sep 13, 2021 at 08:19:31PM +0200, Stephen Kitt wrote:
...
For Wine (and even a wider MinGW-w64
ecosystem) we don’t need all that many source packages to be cross-satisfiable for the whole endeavour to be useful...
But you would still need to create and maintain the whole
infrastructure.
The larger picture is that there are at least 3 big topics:
- cross-architecture dependencies
- partial release architectures
- cross-building a release architecture
Every single of these topics would be a huge amount of work.
If you need all 3 as basis for a solution for Wine,
then that's simply not realistic in the foreseeable future.
The regressions are significant though: if packages can’t stay cross-satisfiable for Debian cross-targets, there’s little hope they can stay
cross-satisfiable for Windows! This means that separate source packages are probably the only viable option.
...
Separate source packages don't bring any real benefits compared to
just bundling and building everything in src:wine.
Either would likely imply that Wine in Debian comes without security
support, and the Release Notes stating that Wine should only be used
for trusted local contents.
Regards,
Stephen
cu
Adrian
Le lun. 13 sept. 2021 à 20:24, Adrian Bunk <bunk@debian.org> a écrit :
On Mon, Sep 13, 2021 at 08:19:31PM +0200, Stephen Kitt wrote:...
The regressions are significant though: if packages can’t stay cross-satisfiable for Debian cross-targets, there’s little hope they can stay
cross-satisfiable for Windows! This means that separate source packages are
probably the only viable option.
...
Separate source packages don't bring any real benefits compared to
just bundling and building everything in src:wine.
No please do not do that. Vendoring is bad and I fight on the
javascript side since four years, against it.
...
| Sysop: | Keyop |
|---|---|
| Location: | Huddersfield, West Yorkshire, UK |
| Users: | 293 |
| Nodes: | 16 (2 / 14) |
| Uptime: | 228:01:04 |
| Calls: | 6,624 |
| Calls today: | 6 |
| Files: | 12,171 |
| Messages: | 5,318,859 |