Recover Master Password in MIT Kerberos

So I am in the process of updating the KDC at my job ( Open Systems @ University of Florida ) and we ran into a few issues:

1. RHEL does not have a new enough version of MIT Kerberos
2. No one can remember the "Master Password" for the KDC
3. The KDC is running on older Power hardware under AIX

It took a while but it looks like we finally figured out how to get from the old AIX/Power box to RHEL/x86_64.

The following is how we solved each one of the problems from above.

RHEL has an older version of MIT Kerberos

As was mentioned in my previous posts: I hate you RHEL and Kerberos 1.8 on RHEL. I was able to get MIT Kerberos 1.8 compiled, packaged, and install on RHEL5.5. See those posts for my work arounds.

Anyone remember the KDC Master Password?

The KDC was installed back in 1996. At the time, the password was known. The stash file was also created so that the KDC could start automatically. There were about 4 people that knew the Master Password either because they entered it in or they had access to the piece of paper that had the password. The person that created the Master Password no longer worked at UF, so when I needed to know what the Master Password was I had to find someone that had access to the piece of paper. After some searching, It was determined that the piece of paper was long lost and assumed destroyed. So no one knew the Master Password. All we had was the stash file.

We thought, well that sucks but at least we have the stash file so we can move forward. WRONG! Turns out the stash file is not endian safe. The file itself is written in the native endianess of the machine. Since we are going from a Power to and x86_64 machine, taking a KDC dump and taking the stash file will not work.

After much research by one of my co-workers we found an option to kdb5_util dump command that allows you to re-key the principals based on a new Master Password. That option is -mkey_convert.

There is one more wrinkle to this story. The new version of MIT Kerberos allows you to have multiple encyption types on your K/M principal. The default encryption type has also changed. So you will not just need to dump with a new Master Password but you will also need to know the encryption type of your K/M principal. This can be achieved by executing a getprinc on K/M and noting the encryption type.

So now that we had all the tools in place here is the procedure:

1. Dump the KDC database on your primary kdc with the following: kdb5_util dump -mkey_convert kdc.dump. This will ask you for a new Master Password. Set it to the actual Master Password that you want in your new KDC
2. Copy the contents over to the new KDC.
3. Figure out what the encryption type of the K/M principal is: kadmin.local -q "getprinc K/M" and not the encryption type. You will need it later.
4. Create a new KDC: kdb5_util -r create. The password is not important.
5. Add a new Master Key encryption type by doing the following: kdb5_util add_mkey -e where encryption type is set to the encryption type used where you took the KDC dump.
6. Get a list of the Master Keys: kdb5_util list_mkeys. Note the kvno of the newly created key.
7. Switch to using the newly created key: kdb5_util use_mkey where kvno is the kvno of the key with the correct encryption type.
8. Create a stash file: kdb5_util stash
9. Load your dump: kdb5_util -r load kdc.dump

At this point, you have a KDC with a password that you should know that you set in step 1 above. You also have a stash file with two entries.

• The Master Password that you entered in step 4
• The Master Password that you set in step 1

The second is more important. If you wanted to, you could create a new stash file which will remove the first entry.

There you have it. If you follow these directions you should be able to:

• Set a new Master Password on your KDC
• Change your KDC CPU architecture from one to another

You just need to have the following things:

• Have a working stash file for your current KDC
• Know the encryption type of your K/M key

Kerberos 1.8 on RHEL

Turned out the dependency hell was easier to get out of this time. I needed to remove both pam_krb5 and krb5-libs.i386. Weird thing is you can leave the krb5-libs.x86_64 installed and the update will work just fine. Probably has to do with the fact that I did not build i386 versions of krb5-libs-1.8 but only x86_64 version.

Again, Thanks RHEL. That sucked but at least I can move on to the next step.

I hate you RHEL

So I am now tasked with upgrading the Kerberos Key Distribution Center ( KDC ) from an AIX box to a 64bit RHEL5 box. Most of the hard work with respect to the steps needed to be taken for the transition had already been taken by a co-worker of mine. Unfortunately a wrinkle was thrown into the mess. That wrinkle is a version change.

The current KDC is at version 1.6.x. The new KDC we want to go to is version 1.8.x. The reason for the upgrade and transition is new functionality that is only available in the 1.8.x branch which is now considered an enterprise requirement.

No biggie, I thought to myself. I'll just go and get a newer version from RedHat. Nope. RHEL5 is stuck at 1.6.x.

Ok. I guess I'll build a newer package like I have done for other things that I have done with RHEL recently. So I started to build a 1.8.x version and since I do not like reinventing the wheel, I used Fedora Core Rawhide's krb5 source rpm as a starting point.

After ripping out the Fedora-isms, I got the rpm to build. When I tried to install it on RHEL5.5 box it failed dependency checks. Looks like pam_krb5 depends on krb5-libs. Makes sense. Hmm, pam_krb5 depends on the krb4 parts of krb5-libs. That could be a problem.

Every since version 1.7 of MIT Kerberos, krb4 compatibility has been removed. Thats actually a really good thing since:

1. I don't have a need for any parts of krb4
2. MIT has an easier time of maintaining the kerberos distribution

So pam_krb5 is built with krb4 compatibility or at least the RPM depends on the libkrb4 libraries.

At this point I got really frustrated ( read: angry ). I really don't want to go down the RPM dependency graph to much further than a single package.

I think my solution to this problem is going to just remove my need for pam_krb5 from the KDC and install my custom package.

The other alternative is to install an RHEL6 beta on the KDC since it is deployed with at least a 1.7 krb5-libs package. That makes me feel really icky. Deploying a beta distribution for such a critical part of the enterprise makes me feel kinda sick.

EMC World

I'll be heading to EMC world in Boston this Saturday through Thursday. Should be a good week. I've scheduled myself for a bunch of classes about the Celerra and lots of stuff about VMware. Most of it is best practices and performance tuning.

There are some things that I am curious about. One in particular that I have been thinking about is Virtual Computing Environment ( VCE ) Vblock architecture. If I am reading the glossies correctly, this is basically a joint venture between VMware, Cisco, and EMC. They seem to want to take a building block approach to getting people to virtualize their workloads.

The blocks consists of:

• VMware Hypervisor and other software
  
• Cisco switches in both hardware and software form ( MDS and Nexus 1000v )
  
• Cisco compute in the form of UCE
• EMC storage as either a Clariion or Symetrix
• Some software glue
  

It is an interesting concept but I think there are a few flaws to this approach

Interoperability

One of the tenets of VCE is that it makes deployment of these products simpler due to rigorous testing.

Lets looks at all these products. Each one is already certified to work with the other. The "testing" they claim has already been done. All these vendors want/need to make sure that their products work with each other since their customers demand it already. The only added functionality that VCE brings to the block is this software glue that supposedly makes configuring and maintaining this environment simpler. I'll have more to say about that after I see a VCE demo at EMC world.

Support

You can go to one place for support of this entire complex. I have to question this due to my personal experience. When it comes to support of gear from one vendor from another vendor, it is weak at best. Case in point, Cisco MDS 95xx series switch purchased through EMC. When we need support for it, we need to call EMC. The feeling we get from support makes us feel that EMC then basically opens a support case with Cisco. They seem to play the middle man. I wonder how this will work when you have a blade in your UCE that needs to be replaced.

Flexibility

The idea behind these building blocks is that they make it easy for you to manage and maintain your environment. If that was the case why are they only working with VMware, EMC, and Cisco gear? It seems to me the most important part of this whole product is the glue that lets you tie in the management of all this gear in a simple way. I don't see why this must be done with this gear specifically. The hardware and software provided by these vendors has open and public APIs for management. Can the glue software not be abstracted to the point where it would not matter what hardware is behind the scenes?

Hopefully I will get a better idea of why EMC is pushing this approach during EMC world.