I don't see the OP's messages because of all the spam from Google Groups.

Michael Grant wrote:

I would definitely lay some fault on sendmail here. It could have
complained about the local cert was depending on an expired cert in its
chain and that would have forced me to fix the problem without hours of

Sendmail is in good company; Apache HTTP Server, NginX, Postfix, and IIS
will also happily start with expired certs somewhere in the chain.

Most daemons that I use trust that the system administrator knows what
they are doing and as such might have a good reason to start using an
expired cert that is beyond the daemon's understanding.



--
Grant. . . .

--- SoupGate-Win32 v1.05
* Origin: fsxNet Usenet Gateway (21:1/5)

On 1/23/24 21:45, Grant Taylor wrote:

I don't see the OP's messages because of all the spam from Google Groups.

Thank you for emailing me directly Michael.

I'm replying here in the hopes that what I write can help multiple people.

On 1/24/24 19:12, Michael Grant wrote:

When I send mail from one of my servers to the other (both using
sendmail 8.17.2), I am seeing this in my mail logs:

I believe I ran into a similar issue between some (then) 2-4 year old
Sendmail installs on a pair of my systems a few years ago.

STARTTLS=client, cert-subject=/CN=mail.networkguild.org, cert-issuer=/C=US/O=Let's+20Encrypt/CN=R3, verifymsg=unable to get
issuer certificate

This looks very familiar to me.

If I verify the cert by hand using:

openssl s_client -starttls smtp -showcerts -connect
mail.networkguild.org:25 -servername mail.networkguild.org

Please elaborate on what "verify" means in this context. Are you
checking / comparing things by hand / script (grep, et al.)? Or are you
using the `openssl verify` (sub)command?

% time (openssl s_client -starttls smtp -showcerts -connect mail.networkguild.org:25 -servername mail.networkguild.org < /dev/null
|& openssl verify)
stdin: OK
( openssl s_client -starttls smtp -showcerts -connect
mail.networkguild.org:2) 0.01s user 0.00s system 0% cpu 13.132 total

Aside: I'm surprised that it's taking 13 seconds to do this. I can
only assume that you have something akin to pre-greeting traffic
filtering. Though I would think that would be after the TCP+TLS
connection establishes and before the SMTP greeting banner.

it verifies OK and sees all the certificates in the chain.

This seems remarkably similar to an issue I had between the two systems
running Sendmail alluded to above.

From memory, what I found after hours of digging is that one of the
root certificates that was then used to cross sign the Let's Encrypt certificates had expired.

Like you I was providing the full chain.

But something about the OpenSSL install I was using still wasn't happy.

I don't have notes at hand, but I remember one of the root certificate
files in /etc/ssl/certs contained the current LE root /and/ the expired
cross signing cert (full chain of sorts). This expired cross signing
cert was being found and used by OpenSSL /before/ the current cross
signing cert in the same /etc/ssl/certs directory.

...

Yep, looking back at the script that I'm using to manage certificates
collected by acme.sh (I don't let acme.sh install things for me) I see
that the ca.cer (PEM) file that it downloads contains two certificates:

1st what is probably best described as an intermediate certificate:

subject= /C=US/O=Let's Encrypt/CN=R3
notBefore=Sep 4 00:00:00 2020 GMT
notAfter=Sep 15 16:00:00 2025 GMT
serial=912B084ACF0C18A753F6D62E25A75F5A
issuer= /C=US/O=Internet Security Research Group/CN=ISRG Root X1

2nd what is probably best described as the root certificate:

subject= /C=US/O=Internet Security Research Group/CN=ISRG Root X1
notBefore=Jan 20 19:14:03 2021 GMT
notAfter=Sep 30 18:14:03 2024 GMT
serial=4001772137D4E942B8EE76AA3C640AB7
issuer= /O=Digital Signature Trust Co./CN=DST Root CA X3

The above output came from the following command:

cat ca.cer | awk 'BEGIN{cmd="openssl x509 -noout -subject -dates -serial -issuer -extensions subjectAltName; echo"}/BEGIN CERTIFICATE/,/END CERTIFICATE/{print | cmd}/END CERTIFICATE/{close(cmd)}'

So, I'm having openssl read in ca.cer and write out the first
certificate to another file. -- I then use /just/ the one certificate (currently serial=912B084ACF0C18A753F6D62E25A75F5A) and my certificate
to generate a new chain that doesn't include the other (expired at the
time I worked on this) certs in the chain.

As best as I could tell, this was all about things in the full chain
file from acme.sh (I don't know if it amalgamated it or if it acme.sh
simply downloaded it from somewhere else, maybe Let's Encrypt
themselves) being expired and the expired root cert being used /before/
OpenSSL would use the (then) current / new root cert in /etc/ssl/certs.

Upon spot checking this, I found that there was a smattering of
different numbers of certs in different files in /etc/ssl/certs.

Fortunately for me I was able to work around the problem for Let's
Encrypt which is what I cared about.

I am indeed using the fullchain.pem file in my sendmail config. I've
tried using both fullchain.pem and cert.pem as the CAcert file,
result is the same.

I currently have the following in my sendmail.cf file (set via sendmail.mc):

# CA directory
O CACertPath=/etc/ssl/certs
# CA file
O CACertFile=/etc/mail/tls/letsencrypt-ca.cer

I want to say that the CACertFile wasn't strictly necessary after I
fixed things in the CACertPath.

N.B. the CACertFile is the file I extract just the 1st certificate
(currently serial=912B084ACF0C18A753F6D62E25A75F5A) into.

My 'CN=mail.networkguild.org' depends on the 'CN=R3' cert. The 'CN=R3'
cert depends on a cert named 'CN = ISRG Root X1'. I definitely see
that cert both in the chain my mail server presents and in /etc/ssl.

This seems strikingly familiar.

I suspect that if (when) you dig, you will find that there may be an
expired certificate in another file, possibly the full chain file, that
is interfering with you.

I'm not sure if this is a debian issue, a sendmail issue, or a cert
issue. The fact that my cert verifies from other sites seems to
indicate it may be a sendmail sending issue, but I'm not sure.

The systems that I ran into this issue on was a (now) ~5 year old Ubuntu install. So it may be an upstream Debian issue that Ubuntu is inheriting.

Or it may have been a bug in the acme.sh client version I was using at
the time.

Anyone have any ideas how to get sendmail to say this cert is valid?

As you can see above, the system that I'm talking about thinks that your server's certificate /is/ valid and /does/ verify.

This is a VERY DEEP in the weeds issue. I spent hours over a few days debugging this.

I spent the time debugging this because my servers are using their
certificate when they are sending / relaying through each other and I'm allowing relay based on CERTISSUER+CERTSUBJECT in AccessDB:

# Allow <SERVER> to relay based on client TLS certificate.
CERTISSUER:/C=US/O=Let's+20Encrypt/CN=R3 SUBJECT
CERTSUBJECT:/CN=<SERVER1 subject> RELAY
CERTSUBJECT:/CN=<SERVER2 subject> RELAY

Please feel free to email me again if you'd like to discuss any private
command output / investigation.

I hope my write is helpful.



--
Grant. . . .

--- SoupGate-Win32 v1.05
* Origin: fsxNet Usenet Gateway (21:1/5)