• Debugging why KRB5_KTNAME isn't working

    From Brian J. Murrell@21:1/5 to All on Thu Jan 27 12:01:48 2022
    I am trying to debug why having KRB5_KTNAME set in the environment of a
    process is not actually making that process use that keytab file but
    instead is using the default /etc/krb5.keytab.

    The process is Postfix's SMTP deamon (smtpd).

    I have confirmed that the method of setting the environment variable is working:

    # ps -ef | grep smtpd
    postfix 3726845 3515138 0 11:56 ? 00:00:00 smtpd -n smtp -t inet -u -o stress= -s 2
    # tr '\0' '\n' < /proc/3726845/environ
    KRB5_KTNAME=/etc/postfix/smtp.keytab KRB5_CLIENT_KTNAME=/etc/postfix/smtp.keytab
    KRB5_TRACE=/tmp/smtpd_krb5_trace
    GENERATION=55

    However when looking at what the process is actually trying to open
    with strace, it's clear that it's not using /etc/postfix/smtp.keytab:

    # grep keytab /tmp/smtpd.strace6
    3726845 openat(AT_FDCWD, "/etc/krb5.keytab", O_RDONLY) = -1 EACCES (Permission denied)
    3726845 openat(AT_FDCWD, "/etc/krb5.keytab", O_RDONLY) = -1 EACCES (Permission denied)

    As you can see, I did attempt to try to trace the kerberos library with

    KRB5_TRACE=/tmp/smtpd_krb5_trace

    but that file does not actually get anything written to it:

    -rwxrwxrwx. 1 root root 0 Jan 27 11:27 smtpd_krb5_trace

    and the above strace doesn't show any sign of trying to open or even
    stat the file:

    # grep smtpd_krb5_trace /tmp/smtpd.strace6
    [nothing]

    Kerberos version appears to be MIT 1.18.2.

    Any thoughts/ideas?

    Cheers,
    b.

    --- SoupGate-Win32 v1.05
    * Origin: fsxNet Usenet Gateway (21:1/5)
  • From Ken Hornstein@21:1/5 to Brian J. Murrell on Thu Jan 27 13:03:56 2022
    Copy: kerberos@mit.edu

    I am trying to debug why having KRB5_KTNAME set in the environment of a >process is not actually making that process use that keytab file but
    instead is using the default /etc/krb5.keytab.

    The process is Postfix's SMTP deamon (smtpd).
    [...]
    Any thoughts/ideas?

    Is it possible Postfix is clearing out the environment at startup? I'm
    a little unclear if the "environ" file in proc shows the _current_
    environment or just the environment at process start (a very brief
    test suggests to me that "environ" contains the environment at process
    start and modification of the current environment isn't reflected there,
    so if Postfix was resetting the environment you wouldn't know it).

    --Ken

    --- SoupGate-Win32 v1.05
    * Origin: fsxNet Usenet Gateway (21:1/5)
  • From Greg Hudson@21:1/5 to Brian J. Murrell on Thu Jan 27 13:41:38 2022
    To: kerberos@mit.edu

    On 1/27/22 12:01 PM, Brian J. Murrell wrote:
    I am trying to debug why having KRB5_KTNAME set in the environment of a process is not actually making that process use that keytab file but
    instead is using the default /etc/krb5.keytab.

    There are three possible reasons why environment variables might be
    ignored. First, Postfix might be asking for a secure krb5 context (krb5_init_secure_context()). Second (and I think the most likely), the process may be running with elevated privilege as determined by secure_getenv(). A setuid or setgid bit on the executable could be
    enough to trigger this. Third, as Ken suggested, the program might
    clean up its own environment.

    If any of these are true, then you have limited options to affect the
    program behavior from outside of the process. You can change the
    default keytab location in /etc/krb5.conf, but that would be global (and
    of course you can't point the program at a different config file via environment variable because those are ignored).

    Of course, the program itself can provide configuration for the keytab
    file. I couldn't find any gss_ or krb5_ calls in the Postfix source
    code (looking at Viktor Dukhovni's git mirror), so I don't have any
    immediate insight as to whether that's currently possible or what would
    need to change.

    --- SoupGate-Win32 v1.05
    * Origin: fsxNet Usenet Gateway (21:1/5)
  • From Brian J. Murrell@21:1/5 to Ken Hornstein on Thu Jan 27 13:27:07 2022
    On Thu, 2022-01-27 at 13:03 -0500, Ken Hornstein wrote:


    Is it possible Postfix is clearing out the environment at startup?

    As anything, I suppose it is possible. It would be doing so in
    violation of exactly the purpose of the mechanism that is being used to
    set the environment though. Meaning, the environment is not just being
    set by the caller and Postfix is clearing it as a matter of good
    housekeeping. The variable is actually being specified in a for-
    purpose Postfix configuration mechanism. This mechanism is
    specifically defined to set environment variables in Postfix processes.

    I will ask the Postfix mailing list in any case.

    a very brief
    test suggests to me that "environ" contains the environment at
    process
    start and modification of the current environment isn't reflected
    there,

    That's the result of my testing too. I must admit to being a little
    surprised at that though. Seems to significantly diminish the purpose/usefulness of that particular /proc entry.

    so if Postfix was resetting the environment you wouldn't know it

    Indeed.

    I wonder if you can suggest a simple test that would verify/demonstrate
    the functionality of the KRB5_KTNAME env. var. that I can use in my
    problem report to the Postfix devs.

    Cheers,
    b.

    --- SoupGate-Win32 v1.05
    * Origin: fsxNet Usenet Gateway (21:1/5)
  • From Ken Hornstein@21:1/5 to Brian J. Murrell on Thu Jan 27 13:45:38 2022
    Copy: kerberos@mit.edu

    Is it possible Postfix is clearing out the environment at startup?

    As anything, I suppose it is possible. It would be doing so in
    violation of exactly the purpose of the mechanism that is being used to
    set the environment though.

    Hm. From postconf(5):

    import_environment (default: see postconf -d output)
    The list of environment parameters that a privileged Postfix process
    will import from a non-Postfix parent process, or name=value environ-
    ment overrides. Unprivileged utilities will enforce the name=value
    overrides, but otherwise will not change their process environment.

    Is that what you're using? It looks to me that if the variable isn't
    listed in the import_environment configuration entry, it doesn't make
    it very far and is removed by the function clean_env().

    (If you want to demonstrate to others how KRB5_KTNAME is supposed to
    work, just include the output of "env KRB5_KTNAME=/dev/stdout kinit" or
    some other Kerberos program).

    --Ken

    --- SoupGate-Win32 v1.05
    * Origin: fsxNet Usenet Gateway (21:1/5)
  • From Jochen Kellner@21:1/5 to Greg Hudson on Thu Jan 27 20:31:25 2022
    Copy: brian@interlinx.bc.ca (Brian J. Murrell)
    Copy: kerberos@mit.edu

    Greg Hudson <ghudson@mit.edu> writes:

    Of course, the program itself can provide configuration for the keytab
    file. I couldn't find any gss_ or krb5_ calls in the Postfix source
    code (looking at Viktor Dukhovni's git mirror), so I don't have any
    immediate insight as to whether that's currently possible or what would
    need to change.

    I once configured postfix to uses sasl:

    main.cf:83:smtpd_sasl_auth_enable = yes

    And in /etc/postfix/sasl/smtpd.conf:

    pwcheck_method: auxprop saslauthd
    #pwcheck_method: saslauthd
    mech_list: plain login gssapi
    log_level: 0
    keytab: /etc/smtp.keytab

    That at least worked some time ago...

    Jochen

    --
    This space is intentionally left blank.

    --- SoupGate-Win32 v1.05
    * Origin: fsxNet Usenet Gateway (21:1/5)
  • From Brian J. Murrell@21:1/5 to Ken Hornstein on Thu Jan 27 14:25:32 2022
    On Thu, 2022-01-27 at 13:45 -0500, Ken Hornstein wrote:


    import_environment (default: see postconf -d output)

    Is that what you're using?

    Yes. That is the "for-purpose" mechanism that I alluded to earlier
    which is why I posited that if smtpd was clearing the environment it
    was doing so in violation of the specific mechanism that was supposed
    to make this all work.

    It looks to me that if the variable isn't
    listed in the import_environment configuration entry, it doesn't make
    it very far and is removed by the function clean_env().

    In my case, I am using the "name=value" variant so that KRB5_KTNAME is
    supposed to be getting an explict value even, rather than relying on
    the environment already having the variable set.

    (If you want to demonstrate to others how KRB5_KTNAME is supposed to
    work, just include the output of "env KRB5_KTNAME=/dev/stdout kinit"
    or
    some other Kerberos program).

    Indeed. I used as my example:

    # KRB5_KTNAME=/etc/postfix/smtp.keytab klist -k
    Keytab name: FILE:/etc/postfix/smtp.keytab
    KVNO Principal
    ---- --------------------------------------------------------------------------
    1 smtp/server.example.com@EXAMPLE.COM

    Cheers,
    b.

    --- SoupGate-Win32 v1.05
    * Origin: fsxNet Usenet Gateway (21:1/5)
  • From Ken Hornstein@21:1/5 to Brian J. Murrell on Thu Jan 27 14:55:43 2022
    Copy: kerberos@mit.edu

    Yes. That is the "for-purpose" mechanism that I alluded to earlier
    which is why I posited that if smtpd was clearing the environment it
    was doing so in violation of the specific mechanism that was supposed
    to make this all work.

    Oh, hm. I might be reading the code wrong, but it looks like the
    SASL library (which is what eventually calls the Kerberos library) is initialized _before_ the environment is reset. So if you're not also
    setting those variables in the environment in the "traditional" way then
    it might not see them.

    (And Jochen's suggestion is also good, I forgot about the SASL config
    file).

    --Ken

    --- SoupGate-Win32 v1.05
    * Origin: fsxNet Usenet Gateway (21:1/5)
  • From Simo Sorce@21:1/5 to Brian J. Murrell on Thu Jan 27 15:53:10 2022
    To: kerberos@mit.edu

    On Thu, 2022-01-27 at 15:34 -0500, Brian J. Murrell wrote:
    On Thu, 2022-01-27 at 20:31 +0100, Jochen Kellner wrote:

    I once configured postfix to uses sasl:

    main.cf:83:smtpd_sasl_auth_enable = yes

    I do have that already.

    And inĀ  /etc/postfix/sasl/smtpd.conf:

    Hrm. I don't have this file. But I never did and this all worked
    prior to a few days ago when the machine was upgraded from EL7 to EL8,
    which unsurprisingly upgrades a lot of things in big jumps. So maybe
    this is now necessary.

    Ahh. Looking at smtpd's strace output, it seems it's looking in /etc/sasl2/smtpd.conf on my machine and I do have that file with:

    pwcheck_method: saslauthd
    mech_list: gssapi plain login

    keytab: /etc/smtp.keytab

    And indeed, winner winner, chicken dinner! Adding a "keytab: /etc/postfix/smtp.keytab" to that file is making smtpd use the correct
    keytab file now.

    So this must all be new behavior in some upgraded versions.

    The keytab option for the cyrus-sasl gssapi plugin is somewhat new
    (considering that RHEL-8 is almost 3 years old now) and is probably
    causing the change in behavior wrt environment variables that you are experiencing.

    --
    Simo Sorce
    RHEL Crypto Team
    Red Hat, Inc

    --- SoupGate-Win32 v1.05
    * Origin: fsxNet Usenet Gateway (21:1/5)
  • From Brian J. Murrell@21:1/5 to Jochen Kellner on Thu Jan 27 15:34:13 2022
    On Thu, 2022-01-27 at 20:31 +0100, Jochen Kellner wrote:

    I once configured postfix to uses sasl:

    main.cf:83:smtpd_sasl_auth_enable = yes

    I do have that already.

    And in  /etc/postfix/sasl/smtpd.conf:

    Hrm. I don't have this file. But I never did and this all worked
    prior to a few days ago when the machine was upgraded from EL7 to EL8,
    which unsurprisingly upgrades a lot of things in big jumps. So maybe
    this is now necessary.

    Ahh. Looking at smtpd's strace output, it seems it's looking in /etc/sasl2/smtpd.conf on my machine and I do have that file with:

    pwcheck_method: saslauthd
    mech_list: gssapi plain login

    keytab: /etc/smtp.keytab

    And indeed, winner winner, chicken dinner! Adding a "keytab: /etc/postfix/smtp.keytab" to that file is making smtpd use the correct
    keytab file now.

    So this must all be new behavior in some upgraded versions.

    Cheers,
    b.

    --- SoupGate-Win32 v1.05
    * Origin: fsxNet Usenet Gateway (21:1/5)