• [gentoo-user] net-libs/gnutls-3.7.2 fails to verify some certificates (

    From Branko =?UTF-8?Q?Grubi=C4=87?=@21:1/5 to All on Tue Nov 23 20:50:02 2021
    Hi,

    I have few applications which use webkit-gtk and gnutls behind as far
    as I know, recently I noticed that RSS feeds for some distrowatch.com subscriptions I had started to fail, initially I did ignore them I
    thought something is wrong on the server side and it was not critical.

    But since it wasn't fixed I started to investigate a little bit more.

    So, in the end it seems to be related to gnutls on Gentoo (I'm running
    ~amd64)

    net-libs/gnutls-3.7.2 abi_x86_64 cxx idn nls openssl seccomp tls-
    heartbeat tools

    Important note, websites using Let's Encrypt certificates work fine,
    except this one (only example known to me). Based on the output of
    `gnutls-cli` it seems that server certificate is served twice compared
    to other working ones (I could be wrong).

    Example output:
    $ gnutls-cli distrowatch.com:443
    Processed 130 CA certificate(s).
    Resolving 'distrowatch.com:443'...
    Connecting to '82.103.129.71:443'...
    - Certificate type: X.509
    - Got a certificate list of 4 certificates.
    - Certificate[0] info:
    - subject `CN=distrowatch.com', issuer `CN=R3,O=Let's Encrypt,C=US',
    serial 0x0408fd5a5ae26286bed92e97da0c830f623c, RSA key 2048 bits,
    signed using RSA-SHA256, activated `2021-09-15 03:49:15 UTC', expires `2021-12-14 03:49:14 UTC', pin- sha256="QoW1tiDGE8S3FLukw86yRL8IfevROPxnx0qwVuu/rUI="
    Public Key ID:
    sha1:fcd2b25ac6ffd73fce3ef65211defd25331dc151
    sha256:4285b5b620c613c4b714bba4c3ceb244bf087debd138fc6 7c74ab056ebbfad42
    Public Key PIN:
    pin-
    sha256:QoW1tiDGE8S3FLukw86yRL8IfevROPxnx0qwVuu/rUI=

    - Certificate[1] info:
    - subject `CN=distrowatch.com', issuer `CN=R3,O=Let's Encrypt,C=US',
    serial 0x0408fd5a5ae26286bed92e97da0c830f623c, RSA key 2048 bits,
    signed using RSA-SHA256, activated `2021-09-15 03:49:15 UTC', expires `2021-12-14 03:49:14 UTC', pin- sha256="QoW1tiDGE8S3FLukw86yRL8IfevROPxnx0qwVuu/rUI="
    - Certificate[2] info:
    - subject `CN=R3,O=Let's Encrypt,C=US', issuer `CN=ISRG Root
    X1,O=Internet Security Research Group,C=US', serial 0x00912b084acf0c18a753f6d62e25a75f5a, RSA key 2048 bits, signed using RSA-SHA256, activated `2020-09-04 00:00:00 UTC', expires `2025-09-15
    16:00:00 UTC', pin-
    sha256="jQJTbIh0grw0/1TkHSumWb+Fs0Ggogr621gT3PvPKG0="
    - Certificate[3] info:
    - subject `CN=ISRG Root X1,O=Internet Security Research Group,C=US',
    issuer `CN=DST Root CA X3,O=Digital Signature Trust Co.', serial 0x4001772137d4e942b8ee76aa3c640ab7, RSA key 4096 bits, signed using
    RSA-SHA256, activated `2021-01-20 19:14:03 UTC', expires `2024-09-30
    18:14:03 UTC', pin-
    sha256="C5+lpZ7tcVwmwQIMcRtPbsQtWLABXhQzejna0wHFr8M="
    - Status: The certificate is NOT trusted. The certificate issuer is
    unknown.
    *** PKI verification of server certificate failed...
    *** Fatal error: Error in the certificate.


    Firefox and Chrome open website just fine, no complains. Also openssl
    client doesn't complain if I read the output right.


    I have tested this on Fedora 35 as well using gnutls-cli, it comes with
    same gnutls release, and has no issues connecting to problematic host.
    So I suspect it's something to do with my system, Gentoo ebuild, or
    combination of libraries used for gnutls on my Gentoo system.

    I have found an interesting (similar) bug[1] which was fixed in the
    current release (fix is included in 3.7.2 based on the NEWS/Release
    notes) where gnutls would fail if Root CA certificate is present twice
    in the chain.

    Can anyone confirm it happening on their system as well, I was not sure
    should I open a Gentoo bug.

    Regards,
    Branko


    [1] https://gitlab.com/gnutls/gnutls/-/issues/1131

    --- SoupGate-Win32 v1.05
    * Origin: fsxNet Usenet Gateway (21:1/5)
  • From Jack@21:1/5 to All on Tue Nov 23 23:20:01 2021
    On 2021.11.23 14:43, Branko Grubić wrote:
    Hi,

    I have few applications which use webkit-gtk and gnutls behind as far
    as I know, recently I noticed that RSS feeds for some distrowatch.com subscriptions I had started to fail, initially I did ignore them I
    thought something is wrong on the server side and it was not critical.

    But since it wasn't fixed I started to investigate a little bit more.

    So, in the end it seems to be related to gnutls on Gentoo (I'm running ~amd64)

    net-libs/gnutls-3.7.2 abi_x86_64 cxx idn nls openssl seccomp tls-
    heartbeat tools

    Important note, websites using Let's Encrypt certificates work fine,
    except this one (only example known to me). Based on the output of `gnutls-cli` it seems that server certificate is served twice compared
    to other working ones (I could be wrong).

    Example output:
    $ gnutls-cli distrowatch.com:443
    Processed 130 CA certificate(s).
    Resolving 'distrowatch.com:443'...
    Connecting to '82.103.129.71:443'...
    - Certificate type: X.509
    - Got a certificate list of 4 certificates.
    - Certificate[0] info:
    - subject `CN=distrowatch.com', issuer `CN=R3,O=Let's Encrypt,C=US',
    serial 0x0408fd5a5ae26286bed92e97da0c830f623c, RSA key 2048 bits,
    signed using RSA-SHA256, activated `2021-09-15 03:49:15 UTC', expires `2021-12-14 03:49:14 UTC', pin- sha256="QoW1tiDGE8S3FLukw86yRL8IfevROPxnx0qwVuu/rUI="
    Public Key ID:
    sha1:fcd2b25ac6ffd73fce3ef65211defd25331dc151
    sha256:4285b5b620c613c4b714bba4c3ceb244bf087debd138fc6 7c74ab056ebbfad42
    Public Key PIN:
    pin-
    sha256:QoW1tiDGE8S3FLukw86yRL8IfevROPxnx0qwVuu/rUI=

    - Certificate[1] info:
    - subject `CN=distrowatch.com', issuer `CN=R3,O=Let's Encrypt,C=US',
    serial 0x0408fd5a5ae26286bed92e97da0c830f623c, RSA key 2048 bits,
    signed using RSA-SHA256, activated `2021-09-15 03:49:15 UTC', expires `2021-12-14 03:49:14 UTC', pin- sha256="QoW1tiDGE8S3FLukw86yRL8IfevROPxnx0qwVuu/rUI="
    - Certificate[2] info:
    - subject `CN=R3,O=Let's Encrypt,C=US', issuer `CN=ISRG Root
    X1,O=Internet Security Research Group,C=US', serial 0x00912b084acf0c18a753f6d62e25a75f5a, RSA key 2048 bits, signed using RSA-SHA256, activated `2020-09-04 00:00:00 UTC', expires `2025-09-15
    16:00:00 UTC', pin-
    sha256="jQJTbIh0grw0/1TkHSumWb+Fs0Ggogr621gT3PvPKG0="
    - Certificate[3] info:
    - subject `CN=ISRG Root X1,O=Internet Security Research Group,C=US',
    issuer `CN=DST Root CA X3,O=Digital Signature Trust Co.', serial 0x4001772137d4e942b8ee76aa3c640ab7, RSA key 4096 bits, signed using RSA-SHA256, activated `2021-01-20 19:14:03 UTC', expires `2024-09-30
    18:14:03 UTC', pin-
    sha256="C5+lpZ7tcVwmwQIMcRtPbsQtWLABXhQzejna0wHFr8M="
    - Status: The certificate is NOT trusted. The certificate issuer is
    unknown.
    *** PKI verification of server certificate failed...
    *** Fatal error: Error in the certificate.


    Firefox and Chrome open website just fine, no complains. Also openssl
    client doesn't complain if I read the output right.


    I have tested this on Fedora 35 as well using gnutls-cli, it comes
    with
    same gnutls release, and has no issues connecting to problematic host.
    So I suspect it's something to do with my system, Gentoo ebuild, or combination of libraries used for gnutls on my Gentoo system.

    I have found an interesting (similar) bug[1] which was fixed in the
    current release (fix is included in 3.7.2 based on the NEWS/Release
    notes) where gnutls would fail if Root CA certificate is present twice
    in the chain.

    Can anyone confirm it happening on their system as well, I was not
    sure
    should I open a Gentoo bug.

    Regards,
    Branko


    [1] https://gitlab.com/gnutls/gnutls/-/issues/1131
    Works fine for me, after recompiling gnutls to add nls and tools. I
    haven't compared the outputs completely, but first pass I don't see any differences, although I could easily have missed one.

    $gnutls-cli distrowatch.com:443
    Processed 128 CA certificate(s).
    Resolving 'distrowatch.com:443'...
    Connecting to '82.103.129.71:443'...
    - Certificate type: X.509
    - Got a certificate list of 4 certificates.
    - Certificate[0] info:
    - subject `CN=distrowatch.com', issuer `CN=R3,O=Let's Encrypt,C=US',
    serial 0x0408fd5a5ae26286bed92e97da0c830f623c, RSA key 2048 bits,
    signed using RSA-SHA256, activated `2021-09-15 03:49:15 UTC', expires `2021-12-14 03:49:14 UTC', pin-sha256="QoW1tiDGE8S3FLukw86yRL8IfevROPxnx0qwVuu/rUI="
    Public Key ID:
    sha1:fcd2b25ac6ffd73fce3ef65211defd25331dc151

    sha256:4285b5b620c613c4b714bba4c3ceb244bf087debd138fc67c74ab056ebbfad42
    Public Key PIN:
    pin-sha256:QoW1tiDGE8S3FLukw86yRL8IfevROPxnx0qwVuu/rUI=

    - Certificate[1] info:
    - subject `CN=distrowatch.com', issuer `CN=R3,O=Let's Encrypt,C=US',
    serial 0x0408fd5a5ae26286bed92e97da0c830f623c, RSA key 2048 bits,
    signed using RSA-SHA256, activated `2021-09-15 03:49:15 UTC', expires `2021-12-14 03:49:14 UTC', pin-sha256="QoW1tiDGE8S3FLukw86yRL8IfevROPxnx0qwVuu/rUI="
    - Certificate[2] info:
    - subject `CN=R3,O=Let's Encrypt,C=US', issuer `CN=ISRG Root
    X1,O=Internet Security Research Group,C=US', serial 0x00912b084acf0c18a753f6d62e25a75f5a, RSA key 2048 bits, signed using RSA-SHA256, activated `2020-09-04 00:00:00 UTC', expires `2025-09-15
    16:00:00 UTC', pin-sha256="jQJTbIh0grw0/1TkHSumWb+Fs0Ggogr621gT3PvPKG0="
    - Certificate[3] info:
    - subject `CN=ISRG Root X1,O=Internet Security Research Group,C=US',
    issuer `CN=DST Root CA X3,O=Digital Signature Trust Co.', serial 0x4001772137d4e942b8ee76aa3c640ab7, RSA key 4096 bits, signed using RSA-SHA256, activated `2021-01-20 19:14:03 UTC', expires `2024-09-30
    18:14:03 UTC', pin-sha256="C5+lpZ7tcVwmwQIMcRtPbsQtWLABXhQzejna0wHFr8M="
    - Status: The certificate is trusted.
    - Description: (TLS1.3-X.509)-(ECDHE-SECP256R1)-(RSA-PSS-RSAE-SHA256)-(AES-256-GCM)
    - Session ID: 83:62:97:D1:C0:77:19:76:F8:2F:41:7E:DD:CD:C5:A6:35:2A:5D:4C:39:B4:F5:12:CA:09:0F:07:26:BA:83:5F
    - Options:
    - Handshake was completed

    - Simple Client Mode:

    - Peer has closed the GnuTLS connection

    --- SoupGate-Win32 v1.05
    * Origin: fsxNet Usenet Gateway (21:1/5)
  • From Branko =?UTF-8?Q?Grubi=C4=87?=@21:1/5 to Jack on Wed Nov 24 00:00:01 2021
    On Tue, 2021-11-23 at 17:26 -0500, Jack wrote:
    On 2021.11.23 14:43, Branko Grubić wrote:
    Hi,
    [1] https://gitlab.com/gnutls/gnutls/-/issues/1131
    OK, diff of your output to mine:
    1,2c1,2
    < $ gnutls-cli  distrowatch.com:443
    < Processed 130 CA certificate(s).
    ---
    $gnutls-cli  distrowatch.com:443
    Processed 128 CA certificate(s).
    21,23c21,29
    < - Status: The certificate is NOT trusted. The certificate issuer
    is 
    unknown.
    < *** PKI verification of server certificate failed...
    < *** Fatal error: Error in the certificate.
    ---
    - Status: The certificate is trusted.
    - Description:  (TLS1.3-X.509)-(ECDHE-SECP256R1)-(RSA-PSS-RSAE-SHA256)-(AES-256-
    GCM)
    - Session ID:  83:62:97:D1:C0:77:19:76:F8:2F:41:7E:DD:CD:C5:A6:35:2A:5D:4C:39:B4:F 5:12:CA:09:0F:07:26:BA:83:5F
    - Options:
    - Handshake was completed

    - Simple Client Mode:

    - Peer has closed the GnuTLS connection

    So I have two fewer CA Certs than you do, but somehow I know the
    issuer 
    and you do not.
    I have app-misc/ca-certificates 20210119.3.66, but there are two ~  versions.  I wonder if something got dropped (whether intentionally
    or 
    not.)


    Hi,

    Thanks for testing. Not happy that it's only me :D

    I don't think my issue is related to ca-certificates, btw. I'm "fully"
    on ~amd64 (unstable/testing).
    Here is what's installed here:

    app-misc/ca-certificates-20211016.3.72 -cacert

    Also, as I said Let's Encrypt certificates from other sites work fine
    with gnutls-cli (and other clients using gnutls), which have similar certificate chain's (same, except the server certificate, same
    intermediate and same "Root" certificate), so I don't think it's
    related to a "missing trust".

    Could be something specific to my system or ~arch (unstable), that some
    library gnutls uses is causing issues (but I cannot remember now when
    it started happening to pinpoint after which update it stopped
    working).

    Regards,
    Branko

    --- SoupGate-Win32 v1.05
    * Origin: fsxNet Usenet Gateway (21:1/5)
  • From Jack@21:1/5 to All on Tue Nov 23 23:30:02 2021
    On 2021.11.23 14:43, Branko Grubić wrote:
    Hi,

    I have few applications which use webkit-gtk and gnutls behind as far
    as I know, recently I noticed that RSS feeds for some distrowatch.com subscriptions I had started to fail, initially I did ignore them I
    thought something is wrong on the server side and it was not critical.

    But since it wasn't fixed I started to investigate a little bit more.

    So, in the end it seems to be related to gnutls on Gentoo (I'm running ~amd64)

    net-libs/gnutls-3.7.2 abi_x86_64 cxx idn nls openssl seccomp tls-
    heartbeat tools

    Important note, websites using Let's Encrypt certificates work fine,
    except this one (only example known to me). Based on the output of `gnutls-cli` it seems that server certificate is served twice compared
    to other working ones (I could be wrong).

    Example output:
    $ gnutls-cli distrowatch.com:443
    Processed 130 CA certificate(s).
    Resolving 'distrowatch.com:443'...
    Connecting to '82.103.129.71:443'...
    - Certificate type: X.509
    - Got a certificate list of 4 certificates.
    - Certificate[0] info:
    - subject `CN=distrowatch.com', issuer `CN=R3,O=Let's Encrypt,C=US',
    serial 0x0408fd5a5ae26286bed92e97da0c830f623c, RSA key 2048 bits,
    signed using RSA-SHA256, activated `2021-09-15 03:49:15 UTC', expires `2021-12-14 03:49:14 UTC', pin- sha256="QoW1tiDGE8S3FLukw86yRL8IfevROPxnx0qwVuu/rUI="
    Public Key ID:
    sha1:fcd2b25ac6ffd73fce3ef65211defd25331dc151
    sha256:4285b5b620c613c4b714bba4c3ceb244bf087debd138fc6 7c74ab056ebbfad42
    Public Key PIN:
    pin-
    sha256:QoW1tiDGE8S3FLukw86yRL8IfevROPxnx0qwVuu/rUI=

    - Certificate[1] info:
    - subject `CN=distrowatch.com', issuer `CN=R3,O=Let's Encrypt,C=US',
    serial 0x0408fd5a5ae26286bed92e97da0c830f623c, RSA key 2048 bits,
    signed using RSA-SHA256, activated `2021-09-15 03:49:15 UTC', expires `2021-12-14 03:49:14 UTC', pin- sha256="QoW1tiDGE8S3FLukw86yRL8IfevROPxnx0qwVuu/rUI="
    - Certificate[2] info:
    - subject `CN=R3,O=Let's Encrypt,C=US', issuer `CN=ISRG Root
    X1,O=Internet Security Research Group,C=US', serial 0x00912b084acf0c18a753f6d62e25a75f5a, RSA key 2048 bits, signed using RSA-SHA256, activated `2020-09-04 00:00:00 UTC', expires `2025-09-15
    16:00:00 UTC', pin-
    sha256="jQJTbIh0grw0/1TkHSumWb+Fs0Ggogr621gT3PvPKG0="
    - Certificate[3] info:
    - subject `CN=ISRG Root X1,O=Internet Security Research Group,C=US',
    issuer `CN=DST Root CA X3,O=Digital Signature Trust Co.', serial 0x4001772137d4e942b8ee76aa3c640ab7, RSA key 4096 bits, signed using RSA-SHA256, activated `2021-01-20 19:14:03 UTC', expires `2024-09-30
    18:14:03 UTC', pin-
    sha256="C5+lpZ7tcVwmwQIMcRtPbsQtWLABXhQzejna0wHFr8M="
    - Status: The certificate is NOT trusted. The certificate issuer is
    unknown.
    *** PKI verification of server certificate failed...
    *** Fatal error: Error in the certificate.


    Firefox and Chrome open website just fine, no complains. Also openssl
    client doesn't complain if I read the output right.


    I have tested this on Fedora 35 as well using gnutls-cli, it comes
    with
    same gnutls release, and has no issues connecting to problematic host.
    So I suspect it's something to do with my system, Gentoo ebuild, or combination of libraries used for gnutls on my Gentoo system.

    I have found an interesting (similar) bug[1] which was fixed in the
    current release (fix is included in 3.7.2 based on the NEWS/Release
    notes) where gnutls would fail if Root CA certificate is present twice
    in the chain.

    Can anyone confirm it happening on their system as well, I was not
    sure
    should I open a Gentoo bug.

    Regards,
    Branko


    [1] https://gitlab.com/gnutls/gnutls/-/issues/1131
    OK, diff of your output to mine:
    1,2c1,2
    < $ gnutls-cli distrowatch.com:443
    < Processed 130 CA certificate(s).
    ---
    $gnutls-cli distrowatch.com:443
    Processed 128 CA certificate(s).
    21,23c21,29
    < - Status: The certificate is NOT trusted. The certificate issuer is unknown.
    < *** PKI verification of server certificate failed...
    < *** Fatal error: Error in the certificate.
    ---
    - Status: The certificate is trusted.
    - Description: (TLS1.3-X.509)-(ECDHE-SECP256R1)-(RSA-PSS-RSAE-SHA256)-(AES-256-GCM)
    - Session ID: 83:62:97:D1:C0:77:19:76:F8:2F:41:7E:DD:CD:C5:A6:35:2A:5D:4C:39:B4:F5:12:CA:09:0F:07:26:BA:83:5F
    - Options:
    - Handshake was completed

    - Simple Client Mode:

    - Peer has closed the GnuTLS connection

    So I have two fewer CA Certs than you do, but somehow I know the issuer
    and you do not.
    I have app-misc/ca-certificates 20210119.3.66, but there are two ~
    versions. I wonder if something got dropped (whether intentionally or
    not.)

    --- SoupGate-Win32 v1.05
    * Origin: fsxNet Usenet Gateway (21:1/5)
  • From Jack@21:1/5 to All on Wed Nov 24 00:20:02 2021
    OK, here's something.

    I changed my stable version of ca-certificates from -cacert to cacert,
    and now I get the same failure you do. So - it's due to either
    something in nss-cacert-class1-class3-r2.patch which only gets applied
    if that USE flag is set, or to something else only done when that USE
    flag is set.

    I don't understand it, but it's a place to start - and note the note in
    the ebuild:

    # When triaging user reports, refer to our wiki for tips:
    # https://wiki.gentoo.org/wiki/Certificates#Debugging_certificate_issues

    --- SoupGate-Win32 v1.05
    * Origin: fsxNet Usenet Gateway (21:1/5)
  • From Branko =?UTF-8?Q?Grubi=C4=87?=@21:1/5 to Jack on Fri Nov 26 11:40:02 2021
    On Tue, 2021-11-23 at 18:14 -0500, Jack wrote:
    OK, here's something.

    I changed my stable version of ca-certificates from -cacert to
    cacert, 
    and now I get the same failure you do.  So - it's due to either 
    something in nss-cacert-class1-class3-r2.patch which only gets
    applied 
    if that USE flag is set, or to something else only done when that
    USE 
    flag is set.

    I don't understand it, but it's a place to start - and note the note
    in 
    the ebuild:

    # When triaging user reports, refer to our wiki for tips:
    #
    https://wiki.gentoo.org/wiki/Certificates#Debugging_certificate_issues


    Thanks Jack.

    Interesting that it breaks for you as well now. Btw I haven't used
    USE="cacert" (could be that I copied it wrong here), it's a additional Certificate Authority which is not included by default in Mozilla
    database as far as I understand. I have tried to change USE="cacert"
    and rebuild app-misc/ca-certificates but no change when tested, which
    to me is expected, but who knows what could be an issue.

    I'll try to dig deeper into this. Initially I was hoping that someone
    more familiar with the topic could jump in and suggest what to do next.

    I did look at the wiki, but wiki uses openssl tools for debugging, and
    I have no issues with openssl client connecting to this server :/ (so I
    don't think it's useful in this case)

    Regards,
    Branko

    --- SoupGate-Win32 v1.05
    * Origin: fsxNet Usenet Gateway (21:1/5)
  • From Branko =?UTF-8?Q?Grubi=C4=87?=@21:1/5 to Jack on Sat Nov 27 15:20:02 2021
    On Tue, 2021-11-23 at 18:14 -0500, Jack wrote:
    OK, here's something.

    I changed my stable version of ca-certificates from -cacert to
    cacert, 
    and now I get the same failure you do.  So - it's due to either 
    something in nss-cacert-class1-class3-r2.patch which only gets
    applied 
    if that USE flag is set, or to something else only done when that
    USE 
    flag is set.

    I don't understand it, but it's a place to start - and note the note
    in 
    the ebuild:

    # When triaging user reports, refer to our wiki for tips:
    #
    https://wiki.gentoo.org/wiki/Certificates#Debugging_certificate_issues



    Another update, I have masked ~arch ca-certificates: >app-misc/ca-certificates-20210119.3.66

    Downgraded to stable one, and now certificate verification is
    successful with gnutls-cli on my test example. Weird since it didn't
    fail to verify similar chains with newer app-misc/ca-certificates. I
    will file a bug report, but still not sure which component app-mist/ca- certificates or net-libs/gnutls.

    Regards,
    Branko

    --- SoupGate-Win32 v1.05
    * Origin: fsxNet Usenet Gateway (21:1/5)
  • From Branko =?UTF-8?Q?Grubi=C4=87?=@21:1/5 to All on Sun Nov 28 07:00:01 2021
    On Tue, 2021-11-23 at 20:43 +0100, Branko Grubić wrote:
    Hi,

    I have few applications which use webkit-gtk and gnutls behind as far
    ...


    Filed a bug report[1]


    [1] https://bugs.gentoo.org/827718

    --- SoupGate-Win32 v1.05
    * Origin: fsxNet Usenet Gateway (21:1/5)