On Tue, Feb 08 2022 at 12:28:21 -0600, Matt Zagrabelny via Kerberos scribbled
in "Re: unexpected failure for GSS Pg server":
On Tue, Feb 8, 2022 at 11:54 AM Matt Zagrabelny <mzagrabe@d.umn.edu> wrote:
Greetings,
I'm experiencing a failure between a GSS enabled Postgresql server and my CLI client.
To my knowledge nothing has changed on the system to create this
failure. I did modify some puppet configs, but according to the
puppet log output (and stat'ing
/etc/postgresql-common/krb5.keytab) file contents did not change.
Here are the errors on on the Pg side:
2022-02-08 11:29:59.304 CST [19401] [unknown]@[unknown] FATAL:
unsupported frontend protocol 1234.5680: server supports 2.0 to 3.0 2022-02-08 11:29:59.315 CST [19402] mzagrabe@somedb FATAL: accepting GSS security context failed
2022-02-08 11:29:59.315 CST [19402] mzagrabe@somedb DETAIL: Unspecified GSS failure. Minor code may provide more information: Request ticket server postgres/db.example.com@EXAMPLE.COM kvno 3 not found in keytab; keytab is likely out of date
NOTE: I still have some Pg servers where my GSS authentication is
not failing. On those systems, Pg still logs the first line above: "unsupported frontend protocol 1234.5680: server supports 2.0 to
3.0". I only included it above for full disclosure.
Here is the error on the client side:
GSSAPI continuation error: Unspecified GSS failure. Minor code may
provide more information: Key version is not available
Here is what klist has to say...
Valid starting Expires Service principal
02/08/2022 11:10:57 02/08/2022 21:10:57 krbtgt/EXAMPLE.COM@EXAMPLE.COM
renew until 02/09/2022 11:10:46
02/08/2022 11:11:05 02/08/2022 21:10:57 postgres/ db.example.com@EXAMPLE.COM
renew until 02/09/2022 11:10:46
Looking at the krb5kdc logs, I see no differences between successful GSS
Pg auths and the failure mentioned above:
Feb 8 11:36:42 auth krb5kdc[434]: TGS_REQ (8 etypes {18 17 20 19 16 23 25 26}) [IPV6 redacted]: ISSUE: authtime 1644340257, etypes {rep=18 tkt=18 ses=18}, mzagrabe@EXAMPLE.COM for postgres/db.example.com@EXAMPLE.COM
Feb 8 11:36:42 auth krb5kdc[434]: closing down fd 14
Feb 8 11:42:45 auth krb5kdc[434]: TGS_REQ (8 etypes {18 17 20 19 16 23 25 26}) [IPV6 redacted]: ISSUE: authtime 1644340257, etypes {rep=18 tkt=18 ses=18}, mzagrabe@EXAMPLE.COM for postgres/ successful.example.com@EXAMPLE.COM
Feb 8 11:42:45 auth krb5kdc[434]: closing down fd 14
Does anyone have any ideas of where to look to start to debug this issue?
After thinking a bit more on the problem. I believe the issue is that I
have different kvno (key version numbers).
Based on the server-side logs from your initial post I would
completely agree with your recent thoughts.
To confirm the state of things there are two (or three) commands that
you will need to run, one on a client system, and one on the
postgresql server (and possibly, just for good measure, a third you
your master KDC):
* On a client run `kvno postgres/
db.example.com@EXAMPLE.COM`. This
should return the key version number of the
"postgres/db.example/com" principal as fetched from the KDC.
Going by the your quoted logs, I would expect this to be "3", at
least at the time you captured those logs.
* On the postgres server run
`klist -ek /etc/postgresql-common/krb5.keytab`. This should
provide output that looks something like the following:
Keytab name: FILE:/etc/postgresql-common/krb5.keytab
KVNO Principal
---- -------------------------------------------------------------
2 postgres/
db.example.com@EXAMPLE.COM (aes256-cts-hmac-sha1-96)
2 postgres/
db.example.com@EXAMPLE.COM (aes128-cts-hmac-sha1-96)
2 postgres/
db.example.com@EXAMPLE.COM (arcfour-hmac)
which may have a different selection of keytypes for your keytab,
and possibly a different value in the KVNO column. I would
guess that at the time you captured the logs, the KVNO listed
would have been less than 3 (certainly should not be more, unless
something rather weird has happened).
* Optionally, you could also use `kadmin` to connect to your KDC,
and run the subcommand: `get_principal postgres/db.example.com`.
This will also give you a list of the current kvno (for each
keytype), which should match the value from the first command you
ran on the client. More importantly or usefully though, it will
also list when it was "Last modified", and by whom, which may shed
light on why it has changed.
Armed with that information, the most likely solution would be to
extract a fresh keytab (using either the kadmin "ktadd" subcommand, or
the handy `k5srvutil` command).
One caveat however (and something that has caught many sysadmins out
at one time or another): If you're running a clustered system which is attempting to use the same principal on more than one server, creating
a fresh keytab will increment the kvno in the KDC again, meaning it
will be out of date on any other system that may be using the
principal. In this case you'd have to copy the keytab to those
affected systems by hand.
For now (before anyone spends any time on my initial request for
help) I would say that I need to do more research and confirm that
my process is working as expected.
One request, where is the documentation that describes how/where the kvno
is used between the kdc and principals?
It may appear a bit old, but the O'Reilly book is still a classic
resource for becoming familiar with Kerberos and how it functions.
Cheers.
Dameon.
--
<> ><> ><> ><> ><> ><> ooOoo <>< <>< <>< <>< <>< <><
Dr. Dameon Wagner, Unix Platform Services
IT Services, University of Oxford
<> ><> ><> ><> ><> ><> ooOoo <>< <>< <>< <>< <>< <><
--- SoupGate-Win32 v1.05
* Origin: fsxNet Usenet Gateway (21:1/5)