We have a Kerberised Hadoop Cloudera Custer. KDC Admin server is on one of the nodes. We don't have a failover node for KDC server yet. On the KDC
admin server while doing a clean up activity for unwanted kdc principals, I deleted the master key principal (K/M@REALM.COM) We never took a kdc dump
of the master key. So we don't have a backup to restore from.
Is there any way I can restore the master key principal?
I basically need an advice on an ongoing issue I am currently stuck on.
We have a Kerberised Hadoop Cloudera Custer. KDC Admin server is on one of the nodes. We don't have a failover node for KDC server yet. On the KDC
admin server while doing a clean up activity for unwanted kdc principals, I deleted the master key principal (K/M@REALM.COM) We never took a kdc dump
of the master key. So we don't have a backup to restore from.
Is there any way I can restore the master key principal?
I have tried creating with kdb5_util add_mkey but the error says that KDC
DB is not able to find a master key credential. I assume this would only
work when you want to create another master key without deleting the
primary key.
Another option for me would be to de-kerberise the cluster and create the same REALM and kerberise the cluster again. But there could be serious
issues if this doesn't fix as this is a live cluster where people are using this on a daily basis.
Maybe dump the core of the running process so you don't accidentally crash
it while trying to debug it live? But that would make finding it in memory even harder...
On 16 Jun 2020, at 04:07, D'Angelo, Jeff C <jcd@psu.edu> wrote:
Would the stash file help here (if it exists)?
--
Jeff
From: kerberos-bounces@mit.edu <kerberos-bounces@mit.edu> on behalf of Chris Hecker <checker@d6.com>
Sent: Thursday, June 11, 2020 6:54 PM
To: Nico Williams <nico@cryptonector.com>
Cc: Harshawardhan Kulkarni <harshawardhan.rk@gmail.com>; kerberos@mit.edu <kerberos@mit.edu>
Subject: Re[2]: MIT Kerberos Master principal deletion
I don't think it would make it harder.
I just mean because you won't be able to set a breakpoint at a function
that uses the key, you'll have to actually chase it around in memory (assuming you use something like gcore to dump it as fast as possible without regard to where it is executing when it's dumped).
If I was doing this live, I'd set a breakpoint on some function that
used the key to decrypt and then inspect there, but with a core file
you'll need to make sure you can find all the structures first.
Is realm_mkey in the kdc_realm_data struct the one he wants?
Chris
------ Original Message ------
From: "Nico Williams" <nico@cryptonector.com>
To: "Chris Hecker" <checker@d6.com>
Cc: "Harshawardhan Kulkarni" <harshawardhan.rk@gmail.com>; "kerberos@mit.edu" <kerberos@mit.edu>
Sent: 2020-06-11 15:31:28
Subject: Re: MIT Kerberos Master principal deletion
On Thu, Jun 11, 2020 at 10:19:39PM +0000, Chris Hecker wrote:
Maybe dump the core of the running process so you don't accidentally crash
it while trying to debug it live? But that would make finding it in memory
even harder...
I don't think it would make it harder.
BTW, we should make it much harder to delete important principals...
________________________________________________sdata=slErWkRJAvfE0nd%2BMESCEFY5Ucx8c79mIpMN%2BwFBMz8%3D&reserved=0
Kerberos mailing list Kerberos@mit.edu https://nam01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fmailman.mit.edu%2Fmailman%2Flistinfo%2Fkerberos&data=02%7C01%7Cjcd%40psu.edu%7C5ecb0ae46a0f4206310108d80e5b131f%7C7cf48d453ddb4389a9c1c115526eb52e%7C0%7C0%7C637275131630535798&
Hi Team,
I basically need an advice on an ongoing issue I am currently stuck on.
We have a Kerberised Hadoop Cloudera Custer. KDC Admin server is on one of the nodes. We don't have a failover node for KDC server yet. On the KDC
admin server while doing a clean up activity for unwanted kdc principals, I deleted the master key principal (K/M@REALM.COM) We never took a kdc dump
of the master key. So we don't have a backup to restore from.
Is there any way I can restore the master key principal?
I have tried creating with kdb5_util add_mkey but the error says that KDC
DB is not able to find a master key credential. I assume this would only
work when you want to create another master key without deleting the
primary key.
Another option for me would be to de-kerberise the cluster and create the same REALM and kerberise the cluster again. But there could be serious
issues if this doesn't fix as this is a live cluster where people are using this on a daily basis.
Can anyone help me here? Looking forward for your reply.
Thanks,
Harsh Kulkarni
Hi Team,
I basically need an advice on an ongoing issue I am currently stuck on.
We have a Kerberised Hadoop Cloudera Custer. KDC Admin server is on one of the nodes. We don't have a failover node for KDC server yet. On the KDC
admin server while doing a clean up activity for unwanted kdc principals, I deleted the master key principal (K/M@REALM.COM) We never took a kdc dump
of the master key. So we don't have a backup to restore from.
Is there any way I can restore the master key principal?
I have tried creating with kdb5_util add_mkey but the error says that KDC
DB is not able to find a master key credential. I assume this would only
work when you want to create another master key without deleting the
primary key.
Another option for me would be to de-kerberise the cluster and create the same REALM and kerberise the cluster again. But there could be serious
issues if this doesn't fix as this is a live cluster where people are using this on a daily basis.
Can anyone help me here? Looking forward for your reply.
Thanks,
Harsh Kulkarni
So doing a simple test on a krb5-1.17 instance I have on a Fedora Linux
box seemed to find a possible solution to this. I'd like to hear from the veterans if this is a good idea or not as I can guess doing this wrong may make things worse before I offer it as a suggestion to try.
I deleted the K/M principal on a test database (note there's a speed bump
in databases created with krb5 versions 1.15+ where the LOCKDOWN_KEYS attribute prevents casual deletion over kadmin/kadmind and one would need kadmin.local to bypass it, so I used kadmin.local to `modprinc
-lockdown_keys K/M` first before `delprinc K/M` in kadmin) and left kadmind and krb5kdc running, which is what I expect matches Harsh's state. This is after I already made backup dump of the database using kdb5_util; let's
call that file "kdb5.dump". For Harsh, I'd be he'd also need to make a
dump of the original db file before continuing (kdb5_util dump -d /path/to/old/var/krb5kdc/principal krb5.dump).
Then I created a shorter dump file of just the header and K/M entry using grep [1]:
sudo sh -c 'grep -E '(kdb5_util|K/M)' kdb5.dump > kdb5.dump.km_restore'
[1] Adding the sudo step here for when you are running a non root shell
in a normal environment that has root ownership restrictions over the db
and dump files.
Make sure it's just those two lines:
sudo cat kdb5.dump.km_restore
Then do a kdb5_util incremental (-update) load with that file:
sudo kdb5_util load -update kdb5.dump.km_restore
Surprisingly, it worked. I guess kdb5_util load would use the K/M it
finds in the dump file instead of the living "principal" database file because it needs to handle the case that it is creating a brand new
database and/or blowing out an exiting one.
Harsh, what version kerb are you running?
Disclaimer: This presumes you haven't changed (rekeyed) K/M since you
created your database (well really since you made that backup copy) and
that you are really sure that backup copy was from an earlier date of this existing db. I'm not sure yet what loading a different K/M would do.
--
Jeff
------------------------------
*From:* kerberos-bounces@mit.edu <kerberos-bounces@mit.edu> on behalf of Harshawardhan Kulkarni <harshawardhan.rk@gmail.com>
*Sent:* Thursday, June 18, 2020 6:27 PM
*To:* kerberos@mit.edu <kerberos@mit.edu>
*Subject:* Re: MIT Kerberos Master principal deletion
Hi Team,
I am reaching out back again with my existing issue regarding master key deletion. I am trying ways to somehow restore it although I don't have a
dump of the KDC.
Re-creating is the last option for me as the cluster is live and a lot of people are using it.
While going through all the KDC related files, I came across all the files which get created while the kdc database was created for the first time.
I was wondering is there any way to restore the master key using either the stash file? or either using the database file which resides in the /var/log/kerberos/krb5kdc location?
We have both the stash files and the principal.db file. When I open the
file although it's not text readable, I can see the K/M@REALM principal details in this file.
So is there any way to restore the master key using this principal.db file
or the .k5.... stash file?
Thanks,
Harsh
On Thu, Jun 11, 2020 at 3:32 AM Harshawardhan Kulkarni < harshawardhan.rk@gmail.com> wrote:sdata=nXkq2krG5q8Shuw6BQ%2FOKIHxS87a%2FrNinLwV%2BOXEk8g%3D&reserved=0
Hi Team,
I basically need an advice on an ongoing issue I am currently stuck on.
We have a Kerberised Hadoop Cloudera Custer. KDC Admin server is on oneof
the nodes. We don't have a failover node for KDC server yet. On the KDC admin server while doing a clean up activity for unwanted kdcprincipals, I
deleted the master key principal (K/M@REALM.COM) We never took a kdcdump
of the master key. So we don't have a backup to restore from.
Is there any way I can restore the master key principal?
I have tried creating with kdb5_util add_mkey but the error says that KDC DB is not able to find a master key credential. I assume this would only work when you want to create another master key without deleting the primary key.
Another option for me would be to de-kerberise the cluster and create the same REALM and kerberise the cluster again. But there could be serious issues if this doesn't fix as this is a live cluster where people areusing
this on a daily basis.
Can anyone help me here? Looking forward for your reply.
Thanks,
Harsh Kulkarni
________________________________________________
Kerberos mailing list Kerberos@mit.edu
https://nam01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fmailman.mit.edu%2Fmailman%2Flistinfo%2Fkerberos&data=02%7C01%7Cjcd%40psu.edu%7C0c15f8ef8a3b49a94a8d08d813dc11fc%7C7cf48d453ddb4389a9c1c115526eb52e%7C0%7C0%7C637281183207940471&
Hi Team,
I basically need an advice on an ongoing issue I am currently stuck on.
We have a Kerberised Hadoop Cloudera Custer. KDC Admin server is on one of the nodes. We don't have a failover node for KDC server yet. On the KDC
admin server while doing a clean up activity for unwanted kdc principals, I deleted the master key principal (K/M@REALM.COM<mailto:M@REALM.COM>) We never took a kdc dump
of the master key. So we don't have a backup to restore from.
Is there any way I can restore the master key principal?
I have tried creating with kdb5_util add_mkey but the error says that KDC
DB is not able to find a master key credential. I assume this would only
work when you want to create another master key without deleting the
primary key.
Another option for me would be to de-kerberise the cluster and create the same REALM and kerberise the cluster again. But there could be serious
issues if this doesn't fix as this is a live cluster where people are using this on a daily basis.
Can anyone help me here? Looking forward for your reply.
Thanks,
Harsh Kulkarni
Hi Team,
I basically need an advice on an ongoing issue I am currently stuck on.
We have a Kerberised Hadoop Cloudera Custer. KDC Admin server is on one of the nodes. We don't have a failover node for KDC server yet. On the KDC
admin server while doing a clean up activity for unwanted kdc principals, I deleted the master key principal (K/M@REALM.COM<mailto:M@REALM.COM>) We never took a kdc dump
of the master key. So we don't have a backup to restore from.
Is there any way I can restore the master key principal?
I have tried creating with kdb5_util add_mkey but the error says that KDC
DB is not able to find a master key credential. I assume this would only
work when you want to create another master key without deleting the
primary key.
Another option for me would be to de-kerberise the cluster and create the same REALM and kerberise the cluster again. But there could be serious
issues if this doesn't fix as this is a live cluster where people are using this on a daily basis.
Can anyone help me here? Looking forward for your reply.
Thanks,
Harsh Kulkarni
Sysop: | Keyop |
---|---|
Location: | Huddersfield, West Yorkshire, UK |
Users: | 296 |
Nodes: | 16 (2 / 14) |
Uptime: | 68:38:37 |
Calls: | 6,655 |
Calls today: | 1 |
Files: | 12,200 |
Messages: | 5,332,040 |
Posted today: | 1 |