Hello,

The cloud team wants to make folks aware of a possible change to the cloud images. The team plans to register a new domain, debian.cloud, for mirrors inside of cloud provider infrastructure. For such mirrors, sources.list will look like:
deb http://<provider>.debian.cloud/debian/ bullseye main

Hosting mirrors inside the cloud infrastructure provides users with faster, cheaper access to the archive. And since it saves the providers money, they're often willing to provide the hosting infrastructure for free. Our image build process allows customizing sources.list with these mirrors when possible. All of this is great!

But some of the hostnames are controlled by the cloud providers. Mostly, this has happened when the name is assigned by a CDN. This isn't optimal: if that name ever changes, users with the old hostname will be unable to install packages.

These names have been very stable. But in some providers, they're tied to cloud accounts. This makes it impossible to move the mirror to another account without losing the name. And of course, for reasons [1], we need to move some of these mirrors.

Since a migration is required, we'd like to adopt debian-controlled hostnames in sources.list. This way, we can never lose the hostnames that appear in sources.list.

Our first choice would be a subzone of debian.org. But we are not in DSA, and haven't been able to get help with this request. So in the interest of making progress, a new domain is the simplest alternative.

I don't know when this work will be complete - hopefully, all of the new infrastructure will be ready to go for the next stable release.

Thanks,
Ross

[1] - Briefly: some of Debian's cloud accounts are technically owned by individual developers, or consulting houses that work on Debian.
Unfortunately, we can't just transfer the accounts in question to SPI, since some also host other things. Thus the team has slowly been transitioning workloads into new accounts owned by SPI.

--- SoupGate-Win32 v1.05
* Origin: fsxNet Usenet Gateway (21:1/5)

On Tue, 25 Jan 2022 21:47:49 -0800, Ross Vandegrift
<rvandegrift@debian.org> wrote:

The cloud team wants to make folks aware of a possible change to the cloud >images. The team plans to register a new domain, debian.cloud, for mirrors >inside of cloud provider infrastructure. For such mirrors, sources.list will >look like:
deb http://<provider>.debian.cloud/debian/ bullseye main

Are the IP ranges of the Cloud Providers registered that badly that deb.debian.org wouldn't reliably point to the mirrors inside the
provider's infrastructure? Or are the cloud providers' mirrors
differnet from what we expect from a Debian mirror?

Greetings
Marc
--
-------------------------------------- !! No courtesy copies, please !! ----- Marc Haber | " Questions are the | Mailadresse im Header Mannheim, Germany | Beginning of Wisdom " |
Nordisch by Nature | Lt. Worf, TNG "Rightful Heir" | Fon: *49 621 72739834

--- SoupGate-Win32 v1.05
* Origin: fsxNet Usenet Gateway (21:1/5)

On Wed, Jan 26, 2022 at 07:25:18AM +0100, Marc Haber wrote:

On Tue, 25 Jan 2022 21:47:49 -0800, Ross Vandegrift
<rvandegrift@debian.org> wrote:

The cloud team wants to make folks aware of a possible change to the cloud >images. The team plans to register a new domain, debian.cloud, for mirrors >inside of cloud provider infrastructure. For such mirrors, sources.list will
look like:
deb http://<provider>.debian.cloud/debian/ bullseye main

Are the IP ranges of the Cloud Providers registered that badly that deb.debian.org wouldn't reliably point to the mirrors inside the
provider's infrastructure? Or are the cloud providers' mirrors
differnet from what we expect from a Debian mirror?

deb.debian.org is served from fastly and AWS CDNs - so it's outside of most cloud provider's infrastructure.

Ross

--- SoupGate-Win32 v1.05
* Origin: fsxNet Usenet Gateway (21:1/5)

On Tue, 25 Jan 2022 22:38:00 -0800, Ross Vandegrift
<rvandegrift@debian.org> wrote:

On Wed, Jan 26, 2022 at 07:25:18AM +0100, Marc Haber wrote:

On Tue, 25 Jan 2022 21:47:49 -0800, Ross Vandegrift
<rvandegrift@debian.org> wrote:

The cloud team wants to make folks aware of a possible change to the cloud >> >images. The team plans to register a new domain, debian.cloud, for mirrors >> >inside of cloud provider infrastructure. For such mirrors, sources.list will
look like:
deb http://<provider>.debian.cloud/debian/ bullseye main

Are the IP ranges of the Cloud Providers registered that badly that
deb.debian.org wouldn't reliably point to the mirrors inside the
provider's infrastructure? Or are the cloud providers' mirrors
differnet from what we expect from a Debian mirror?

deb.debian.org is served from fastly and AWS CDNs - so it's outside of most >cloud provider's infrastructure.

So it is not possible to hook arbitrary mirrors into deb.debian.org
and we're dependent on Fastly and AWS here?

I thought it was something more flexible.

Greetings
Marc
--
-------------------------------------- !! No courtesy copies, please !! ----- Marc Haber | " Questions are the | Mailadresse im Header Mannheim, Germany | Beginning of Wisdom " |
Nordisch by Nature | Lt. Worf, TNG "Rightful Heir" | Fon: *49 621 72739834

--- SoupGate-Win32 v1.05
* Origin: fsxNet Usenet Gateway (21:1/5)

Hi Ross (2022.01.26_05:47:49_+0000)

Our first choice would be a subzone of debian.org. But we are not in DSA, and
haven't been able to get help with this request. So in the interest of making
progress, a new domain is the simplest alternative.

FWIW, DSA has the ability to host domains in DSA's DNS and delegate git
zone access to non-DSA DDs. This is how debconf.org DNS is handled.
So, there is a path to bring external domains under the DSA umbrella,
later.

SR

--
Stefano Rivera
http://tumbleweed.org.za/
+1 415 683 3272

--- SoupGate-Win32 v1.05
* Origin: fsxNet Usenet Gateway (21:1/5)

Hi Marc

On Wed, Jan 26, 2022 at 07:25:18AM +0100, Marc Haber wrote:

Are the IP ranges of the Cloud Providers registered that badly that deb.debian.org wouldn't reliably point to the mirrors inside the
provider's infrastructure? Or are the cloud providers' mirrors
differnet from what we expect from a Debian mirror?

I wonder, which mechanism would you propose to do so?

Bastian

--
Witch! Witch! They'll burn ya!
-- Hag, "Tomorrow is Yesterday", stardate unknown

--- SoupGate-Win32 v1.05
* Origin: fsxNet Usenet Gateway (21:1/5)

On 1/26/22 10:04, Marc Haber wrote:

On Tue, 25 Jan 2022 22:38:00 -0800, Ross Vandegrift
<rvandegrift@debian.org> wrote:

On Wed, Jan 26, 2022 at 07:25:18AM +0100, Marc Haber wrote:

On Tue, 25 Jan 2022 21:47:49 -0800, Ross Vandegrift
<rvandegrift@debian.org> wrote:

The cloud team wants to make folks aware of a possible change to the cloud >>>> images. The team plans to register a new domain, debian.cloud, for mirrors
inside of cloud provider infrastructure. For such mirrors, sources.list will
look like:
deb http://<provider>.debian.cloud/debian/ bullseye main

Are the IP ranges of the Cloud Providers registered that badly that
deb.debian.org wouldn't reliably point to the mirrors inside the
provider's infrastructure? Or are the cloud providers' mirrors
differnet from what we expect from a Debian mirror?

deb.debian.org is served from fastly and AWS CDNs - so it's outside of most >> cloud provider's infrastructure.

So it is not possible to hook arbitrary mirrors into deb.debian.org
and we're dependent on Fastly and AWS here?

Correct.

Cheers,

Thomas Goirand (zigo)

--- SoupGate-Win32 v1.05
* Origin: fsxNet Usenet Gateway (21:1/5)

On Wed, Jan 26, 2022 at 10:04:47AM +0100, Marc Haber wrote:

The cloud team wants to make folks aware of a possible change to the cloud
images. The team plans to register a new domain, debian.cloud, for mirrors
inside of cloud provider infrastructure. For such mirrors, sources.list will
look like:
deb http://<provider>.debian.cloud/debian/ bullseye main

Are the IP ranges of the Cloud Providers registered that badly that
deb.debian.org wouldn't reliably point to the mirrors inside the
provider's infrastructure? Or are the cloud providers' mirrors
differnet from what we expect from a Debian mirror?

deb.debian.org is served from fastly and AWS CDNs - so it's outside of most >cloud provider's infrastructure.

So it is not possible to hook arbitrary mirrors into deb.debian.org
and we're dependent on Fastly and AWS here?

I thought it was something more flexible.

I could be misremembering the conversation, but I believe deb.debian.org
is only fastly at the moment. It would be technically possible to
direct some clients to other mirrors/CDNs, but the mirror admins are
hesitant to introduce that level of complexity at the moment, as it
would make troubleshooting significantly more difficult. If fastly
becomes unreliable for some reason, then deb.debian.org would be
repointed to some other back end.

noah

--- SoupGate-Win32 v1.05
* Origin: fsxNet Usenet Gateway (21:1/5)

On Tue, Jan 25, 2022 at 09:47:49PM -0800, Ross Vandegrift wrote:

Hello,

The cloud team wants to make folks aware of a possible change to the cloud images. The team plans to register a new domain, debian.cloud, for mirrors inside of cloud provider infrastructure. For such mirrors, sources.list will look like:
deb http://<provider>.debian.cloud/debian/ bullseye main

Hosting mirrors inside the cloud infrastructure provides users with faster, cheaper access to the archive. And since it saves the providers money, they're
often willing to provide the hosting infrastructure for free. Our image build
process allows customizing sources.list with these mirrors when possible. All
of this is great!

But some of the hostnames are controlled by the cloud providers. Mostly, this
has happened when the name is assigned by a CDN. This isn't optimal: if that name ever changes, users with the old hostname will be unable to install packages.

These names have been very stable. But in some providers, they're tied to cloud accounts. This makes it impossible to move the mirror to another account
without losing the name. And of course, for reasons [1], we need to move some
of these mirrors.

Since a migration is required, we'd like to adopt debian-controlled hostnames in sources.list. This way, we can never lose the hostnames that appear in sources.list.

Our first choice would be a subzone of debian.org. But we are not in DSA, and
haven't been able to get help with this request. So in the interest of making
progress, a new domain is the simplest alternative.

I don't know when this work will be complete - hopefully, all of the new infrastructure will be ready to go for the next stable release.

Hi,

I think we (DSA) have been reluctant to add new third-party-run services
under debian.org, and it's not clear to me if that infrastructure would
be run by the cloud team on behalf of debian, or if the cloud team would control the names but point them at mirrors run by the cloud providers themselves.

Either way as Stefano said if you go for a new domain name it should be possible to use the same setup as our other domains if you want.

Cheers,
Julien

--- SoupGate-Win32 v1.05
* Origin: fsxNet Usenet Gateway (21:1/5)

❦ 26 January 2022 10:04 +01, Marc Haber:

Are the IP ranges of the Cloud Providers registered that badly that
deb.debian.org wouldn't reliably point to the mirrors inside the
provider's infrastructure? Or are the cloud providers' mirrors
differnet from what we expect from a Debian mirror?

deb.debian.org is served from fastly and AWS CDNs - so it's outside of most >>cloud provider's infrastructure.

So it is not possible to hook arbitrary mirrors into deb.debian.org
and we're dependent on Fastly and AWS here?

I thought it was something more flexible.

This was redir.debian.org. I was very happy with it. I never understood
why we replaced it by something centralized. There were problems with it
and nobody was fixing them, but I think we have never been told exactly
what the problems were. But I can understand how using an external CDN
is less a burden than maintaining a redirector like our customn one or something like MirrorBrain (not packaged in Debian but used by many open
source projects).

deb.debian.org is just a CNAME to Fastly.

At my location (France, 1st ISP, FTTH; was the same in Switzerland),
Fastly is very slow from time to time (once every two months? less than
100 KB/s). Their support fix it in a day or two once you tell them. I
have switched back to geographic mirrors: ftp.fr and ftp.ch never failed
to deliver good performance.
--
Use the fundamental control flow constructs.
- The Elements of Programming Style (Kernighan & Plauger)

--- SoupGate-Win32 v1.05
* Origin: fsxNet Usenet Gateway (21:1/5)

On Wed, Jan 26, 2022 at 07:58:23PM +0100, Julien Cristau wrote:

I think we (DSA) have been reluctant to add new third-party-run services under debian.org, and it's not clear to me if that infrastructure would
be run by the cloud team on behalf of debian, or if the cloud team would control the names but point them at mirrors run by the cloud providers themselves.

It depends on the provider, and what you mean by "infrastructure". Many cases fall into a grey area between your options:

- AWS: the mirror will be a CloudFront distribution. SPI will own the account,
the cloud team will create the CDN, but the infrastructure will be AWS.
- Azure, Infomaniak: they contract or employ folks to run mirrors for them.
Cloud team folks are responsible in both cases.

Speaking for myself only, I'd be be open to providing a name for a provider that doesn't work with us, and runs their own mirror. This would definitely not meet your requirement.

Either way as Stefano said if you go for a new domain name it should be possible to use the same setup as our other domains if you want.

Great, thanks for the offer - we haven't settled on any mechanisms, but will keep this in mind.

Thanks,
Ross

--- SoupGate-Win32 v1.05
* Origin: fsxNet Usenet Gateway (21:1/5)

On Wed, 26 Jan 2022 15:27:53 +0100, Bastian Blank <waldi@debian.org>
wrote:

On Wed, Jan 26, 2022 at 07:25:18AM +0100, Marc Haber wrote:

Are the IP ranges of the Cloud Providers registered that badly that
deb.debian.org wouldn't reliably point to the mirrors inside the
provider's infrastructure? Or are the cloud providers' mirrors
differnet from what we expect from a Debian mirror?

I wonder, which mechanism would you propose to do so?

I have no idea but I would expect that a GeoIP-similar mechanism would
allow to point clients to a local mirror inside the vendor's cloud infrastructure.

Greetings
Marc
--
-------------------------------------- !! No courtesy copies, please !! ----- Marc Haber | " Questions are the | Mailadresse im Header Mannheim, Germany | Beginning of Wisdom " |
Nordisch by Nature | Lt. Worf, TNG "Rightful Heir" | Fon: *49 621 72739834

--- SoupGate-Win32 v1.05
* Origin: fsxNet Usenet Gateway (21:1/5)

Hi Julien

On Wed, Jan 26, 2022 at 07:58:23PM +0100, Julien Cristau wrote:

I think we (DSA) have been reluctant to add new third-party-run services under debian.org,

Just being curious: what is your definition of "third-party-run"? As
example: deb.debian.org. It uses Fastly, which is shared
responsibility. It is run by someone else but provides a product that
Debian configures. But where would be the limit?

and it's not clear to me if that infrastructure would
be run by the cloud team on behalf of debian, or if the cloud team would control the names but point them at mirrors run by the cloud providers themselves.

I doubt that there will be any reason for us to point Debian names to
mirrors the provider controls.

There is one provider providing it's own mirrors: Hetzner. They use an
already existing mechanism to configure the mirror to their own if not overriden by the user. So there is no name controlled by Debian
required.

The whole reason for this stunt is to protect Debian. Protect Debian
and it's users from screwups by
- the providers themselves,
- a laps in the agreement that currently provides access to Debian
mirrors on Azure and AWS and
- single Debian developers controlling resources.

Bastian

--
I've already got a female to worry about. Her name is the Enterprise.
-- Kirk, "The Corbomite Maneuver", stardate 1514.0

--- SoupGate-Win32 v1.05
* Origin: fsxNet Usenet Gateway (21:1/5)

On Wed, Jan 26, 2022 at 08:56:55PM +0100, Vincent Bernat wrote:

This was redir.debian.org. I was very happy with it. I never understood
why we replaced it by something centralized. There were problems with it
and nobody was fixing them, but I think we have never been told exactly
what the problems were.

From my memory: one apt session may requires many http requests. The redirector could send those requests to different mirrors. If two of those mirrors were in different states, apt would interpret that as bad data. This caused unreliable updates.

Ross

--- SoupGate-Win32 v1.05
* Origin: fsxNet Usenet Gateway (21:1/5)

On Wed, Jan 26, 2022 at 09:43:16PM -0800, Ross Vandegrift wrote:

On Wed, Jan 26, 2022 at 07:58:23PM +0100, Julien Cristau wrote:

I think we (DSA) have been reluctant to add new third-party-run services under debian.org, and it's not clear to me if that infrastructure would
be run by the cloud team on behalf of debian, or if the cloud team would control the names but point them at mirrors run by the cloud providers themselves.

It depends on the provider, and what you mean by "infrastructure". Many cases
fall into a grey area between your options:

- AWS: the mirror will be a CloudFront distribution. SPI will own the account,
the cloud team will create the CDN, but the infrastructure will be AWS.
- Azure, Infomaniak: they contract or employ folks to run mirrors for them.
Cloud team folks are responsible in both cases.

Speaking for myself only, I'd be be open to providing a name for a provider that doesn't work with us, and runs their own mirror. This would definitely not meet your requirement.

Sorry for being unclear. For these purposes the first case would count
as debian-run, the second probably not.

Either way a couple of things to think about and maybe document is what
are the criteria for providing a name for a vendor, and what happens
when the criteria are no longer met.

Cheers,
Julien

--- SoupGate-Win32 v1.05
* Origin: fsxNet Usenet Gateway (21:1/5)