Hi all,
I'd like us to move from
/etc/apt/sources.list
to
/etc/apt/sources.list.d/debian.sources
in bookworm.
# deb822 intro
The deb822 format can be shorter and easier to read, to quote the sources.list manual page:
As an example, the sources for your distribution could look like this in one-line-style format:
deb http://deb.debian.org/debian bullseye main contrib non-free
deb http://security.debian.org bullseye-security main contrib non-free
or like this in deb822 style format:
Types: deb
URIs: http://deb.debian.org/debian
Suites: bullseye
Components: main contrib non-free
Types: deb
URIs: http://security.debian.org
Suites: bullseye-security
Components: main contrib non-free
Now if you mix unstable and testing, you could just have that it one paragraph.
The big advantage of deb822 sources is that you can embed larger data:
Types: deb
URIs: https://deb.debian.org
Suites: stable
Components: main contrib non-free
Signed-By:
-----BEGIN PGP PUBLIC KEY BLOCK-----
.
mDMEYCQjIxYJKwYBBAHaRw8BAQdAD/P5Nvvnvk66SxBBHDbhRml9ORg1WV5CvzKY
CuMfoIS0BmFiY2RlZoiQBBMWCgA4FiEErCIG1VhKWMWo2yfAREZd5NfO31cFAmAk
IyMCGyMFCwkIBwMFFQoJCAsFFgIDAQACHgECF4AACgkQREZd5NfO31fbOwD6ArzS
dM0Dkd5h2Ujy1b6KcAaVW9FOa5UNfJ9FFBtjLQEBAJ7UyWD3dZzhvlaAwunsk7DG
3bHcln8DMpIJVXht78sL
=IE0r
-----END PGP PUBLIC KEY BLOCK-----
Oh yeah, a standalone sources file with embedded key - making third-party repository management substantially more convenient.
# issues
several software does not support deb822 currently. I plan to work on
adding deb822 support to python-apt in the upcoming months, I don't know
what else is affected.
There is some software "parsing" sources.list on its own, most of that
is better served by `apt-get indextargets` (and for downloading stuff
based on the urls, `apt-helper download-file`, such that it respects
proxies and supports all transports users may use in sources.list)
In terms of setting up system, I guess debootstrap and d-i's apt-setup
module need changes? I admit I do not have a total overview.
# timeline
I do not know the total outcome of this. My hope would be that
we can switch the default in Summer 2022 and see what breaks,
although I don't know who's going to install from testing
d-i :D
docker images probably can move earlier as they don't have
as much interactive users that use tools that might be broken
(e.g. software-properties). Possibly others, there's no need
to roll out in multiple places at the same time, as long as
we converge by freeze time.
I invite people to test this out already on their own systems,
I know several people do; and extrepo also builds deb822 sources
files, but several people I guess do not know about it yet. Please
test and report bugs.
--
debian developer - deb.li/jak | jak-linux.org - free software dev
ubuntu core developer i speak de, en
I'd like us to move from[...]
/etc/apt/sources.list
to
/etc/apt/sources.list.d/debian.sources
#timeline
Hi Julian,
this sounds like a nice and useful plan and feature(s), thank you!
just one question:
On Wed, Nov 03, 2021 at 04:45:15PM +0100, Julian Andres Klode wrote:
I'd like us to move from[...]
/etc/apt/sources.list
to
/etc/apt/sources.list.d/debian.sources
#timeline
You didn't say so explicitly, but do you plan to support old style /etc/apt/sources.list until forever? ;) Or do you envision automatic migration of that file? Or?
I don't know, to be honest, have not thought about it yet.
I think an automatic migration might be to painful what with all the
juju and ansible and saltstack (I feel like it'd be nice to have
those tools migrate config to new formats).
There is some software "parsing" sources.list on its own, most of that
is better served by `apt-get indextargets` (and for downloading stuff
based on the urls, `apt-helper download-file`, such that it respects
proxies and supports all transports users may use in sources.list)
Hi Julian,
On 03-11-2021 16:45, Julian Andres Klode wrote:
There is some software "parsing" sources.list on its own, most of that
is better served by `apt-get indextargets` (and for downloading stuff
based on the urls, `apt-helper download-file`, such that it respects proxies and supports all transports users may use in sources.list)
Like autopkgtest. When I was working on it to support Debian's migration testing, I looked at python-apt and because that didn't support it,
stopped thinking. With indextargets and download-file I guess we could
work on it again. When were those introduced? Ubuntu needs it on old
releases so before autopkgtest can change it, we'd need support for a while.
On Wed, Nov 03, 2021 at 05:32:53PM +0100, Julian Andres Klode wrote:
I don't know, to be honest, have not thought about it yet.
many thanks for your verbose reply! /me likes this timeline.
I think an automatic migration might be to painful what with all the
juju and ansible and saltstack (I feel like it'd be nice to have
those tools migrate config to new formats).
I guess it would be nice if those tools (and users not using those
tools) could run one standard tool doing the job :)
On 03-11-2021 16:45, Julian Andres Klode wrote:
There is some software "parsing" sources.list on its own, most of that
is better served by `apt-get indextargets` (and for downloading stuff
based on the urls, `apt-helper download-file`, such that it respects proxies and supports all transports users may use in sources.list)
Like autopkgtest. When I was working on it to support Debian's migration testing, I looked at python-apt and because that didn't support it,
stopped thinking. With indextargets and download-file I guess we could
work on it again. When were those introduced? Ubuntu needs it on old
releases so before autopkgtest can change it, we'd need support for a while.
On Wed, Nov 03, 2021 at 04:23:52PM +0000, Holger Levsen wrote:
You didn't say so explicitly, but do you plan to support old style /etc/apt/sources.list until forever? ;) Or do you envision automatic migration of that file? Or?
I don't know, to be honest, have not thought about it yet.
With APT's 5 year interface stability and major version bump,
The first time we could remove support for old releases would be
trixie/apt 3.0 in 2025 (that schedule just happens because I don't
want to release .10 versions, but keep the 2nd component single
digit :D)
I think an automatic migration might be to painful what with all the
juju and ansible and saltstack (I feel like it'd be nice to have
those tools migrate config to new formats).
Of course, once everyone and their dog has migrated, I might feel
different and complain about legacy sources.list and deprecate
them.
On Wed, Nov 03, 2021 at 08:53:15PM +0100, Paul Gevers wrote:
On 03-11-2021 16:45, Julian Andres Klode wrote:
There is some software "parsing" sources.list on its own, most of that
is better served by `apt-get indextargets` (and for downloading stuff based on the urls, `apt-helper download-file`, such that it respects proxies and supports all transports users may use in sources.list)
Like autopkgtest. When I was working on it to support Debian's migration testing, I looked at python-apt and because that didn't support it,
stopped thinking. With indextargets and download-file I guess we could
work on it again. When were those introduced? Ubuntu needs it on old releases so before autopkgtest can change it, we'd need support for a while.
`apt-get indextargets` is from 2015 and a part of the acquire-additional- files feature used mainly by apt-file and appstream to have apt download files it isn't using itself, so those tools don't have to implement it.
The job of indextargets is it mostly to give access to metadata (and crucially filenames) for those previously configured and hopefully now downloaded files. apt-file e.g. asks for the Contents files in this way
to avoid exposing file naming logic and location to other tools.
So, for the filenames of all (downloaded) Packages files:
apt-get indextargets --format '$(FILENAME)' 'Identifier: Packages'
(the default output is deb822 stanzas you could grep with more powerful
tools than the simple inbuilt line-based filter)
Note that you either have to implement opening compressed files yourself
or use `/usr/lib/apt/apt-helper cat-file`.
That was historically the most common reason to fiddle with sources.list parsing, hence Julian referring to it, but this seems not what autpkgtest
is aiming for. On a casual look (well, grep) I see only:
* lib/adt_testbed.py apt-pocket codepath seems to want to construct new
sources.list entries based on existing ones. That should be possible
with some indextargets busywork in general, but I am not completely
sure what is going on here and the gymnastics should be similar to…
* setup-commands/setup-testbed tries to find the mirror (and release)
used for target distribution based on your current system. I am
a bit surprised that works actually…
Anyway, the later could perhaps be implement with:
apt-get indextargets --format '$(ORIGIN)|$(REPO_URI)' | sort -u | \
grep -i -e "^$(. /etc/os-release; echo "$ID")" | cut -d'|' -f 2
(That is one line only for posterity – as you see, I am trying to fix
the too general search by checking against Origin as defined by the
Release file of a repository, but that would need work still to
eliminate same-origin-but-different-repo cases)
Parsing of the sources files is not really indextargets job through, so
it might not always work for that task: It e.g. doesn't work if the data files are not on disk which might or might not be okay for you (there is 'guess' mode, but that of course has no metadata extracted from the
Release files – the Origin I was using above).
The apt family doesn't really have a publicly exposed way of reasoning
about sources.list (or .sources) files and I am not quite sure it really
can as subtil differences between repositories make it hard to give them
all a common interface which makes sense. (I will probably be proven
wrong by Julian though.)
Like, for example, if stable is in the sources, make sure there
is also updates and security there and/or add them. What for Debian are
three distinct repositories might for others very well be components.
Assuming you even know which line refers to Debian: I was using Origin
above for this task as we can't really guess based on the URI. And even then… that logic above finds the tor+mirror+file source I am using,
that won't work for autopkgtest, but I am special and this just
a default fallback, so I might be thinking way too much about it…
On Wed, Nov 3, 2021 at 11:45 PM Julian Andres Klode <jak@debian.org> wrote:
Hi all,
I'd like us to move from
/etc/apt/sources.list
to
/etc/apt/sources.list.d/debian.sources
While it's really a nice feature for the third-party repository, I
don't see the benefits to change the default one, especially the path.
I had to admit that I have countless scripts which run `sed /etc/apt/souces.list`, to change the default mirror, as well as in the Dockerfile.
On Thu, Nov 04, 2021 at 12:13:48AM +0800, Shengjing Zhu wrote:
On Wed, Nov 3, 2021 at 11:45 PM Julian Andres Klode <jak@debian.org> wrote:
Hi all,
I'd like us to move from
/etc/apt/sources.list
to
/etc/apt/sources.list.d/debian.sources
While it's really a nice feature for the third-party repository, I
don't see the benefits to change the default one, especially the path.
I had to admit that I have countless scripts which run `sed /etc/apt/souces.list`, to change the default mirror, as well as in the Dockerfile.
There's a technical limitation in that we get the format from the file extension. It's a bit annoying.
docker images probably can move earlier as they don't have
as much interactive users that use tools that might be broken
(e.g. software-properties).
| Sysop: | Keyop |
|---|---|
| Location: | Huddersfield, West Yorkshire, UK |
| Users: | 293 |
| Nodes: | 16 (2 / 14) |
| Uptime: | 216:39:10 |
| Calls: | 6,621 |
| Calls today: | 3 |
| Files: | 12,169 |
| Messages: | 5,317,614 |