The Mysterious ’shareall’ problem…

Hostnames, directory names, etc have been changed to protect my job.

The situation, in brief:

A Solaris 9 system that is serving as an SAP on Oracle database server. The system also acts as an NFS server to SAP application servers. This specific system is currently in use as a test system for end-of-fiscal-year processing. As a result, it gets reconfigured often. Many times, in a hurry.

The problem, in brief:

The NFS server is only exporting the LAST entry in /etc/dfs/dfstab. Or more accurately, the NFS server only exports one filesystem at a time.

Yesterday, I got a request from the SAP BASIS group to reconfigure this server in a hurry. One UFS filesystem (on Veritas Volume Manager) needed to be grown, and a second directory needed to be exported via NFS. I added the new filesystem into the /etc/dfs/dfstab as usual, and as usual, I ran the shareall command. So, I logged into the application server, added the filesystem into my automount map, restarted autofs, then tried to test the new mountpoint:

[root@client root]# ls /mnt/b
Permission denied


Permission denied. Hmm. Running showmount -e on the server showed the following:

[steelmi1@server steelmi1]$ showmount -e
export list for server:
/export/a


Which is interesting, because the /etc/dfs/dfstab reads thusly:

[steelmi1@server steelmi1]$ cat /etc/dfs/dfstab
share -F nfs /export/b
share -F nfs /export/a


In order to make sure that I hadn’t malformed the /export/b entry, I reveresed the two entries in /etc/dfs/dfstab so that it now looks like this:

[steelmi1@server steelmi1]$ cat /etc/dfs/dfstab
share -F nfs /export/a
share -F nfs /export/b
[steelmi1@server steelmi1]$ showmount -e
export list for server:
/export/b


So obviously, the entry isn’t malformed.

The next thing I tried was running the individual share commands. No matter which one I ran, it would export that filesystem, and remove the others.

Now, we UNIX administrators are loath to admit that our systems sometimes need to be rebooted. Ran uptime to see how long this machine had been up, and I saw something strange:

[steelmi1@server ~]$ uptime
1:32pm up 3 users, load average: 0.41, 0.23, 0.15


This is what the output of uptime normally looks like:

[steelmi1@server ~]$ uptime
1:32pm up 13 day(s), 21:51, 3 users, load average: 0.41, 0.23, 0.15


So, the “uptime” field was missing from the output. I knew that during end-of-year testing, we often disable the NetworkTime Protocol daemon, then set the system clock several months forward to more accurately simulate what will happen during closing. I took a quick look at the /var/adm/wtmpx and /var/adm/utmpx files to see if there were any obvious problems with them. First, I copied them to /root, and operated on the copies:

[root@server root]# cp /var/adm/*tmpx /root
[root@server root]# /usr/lib/acct/fwtmp < wtmpx | grep time
old time 0 3 0000 0000 1151077407 0 0 0 Fri Jun 23 10:43:27 2006
new time 0 4 0000 0000 1163001000 0 0 0 Wed Nov 8 09:50:00 2006
old time 0 3 0000 0000 1166355898 0 0 0 Sun Dec 17 05:44:58 2006
new time 0 4 0000 0000 1154431320 0 0 0 Tue Aug 1 06:22:00 2006
old time 0 3 0000 0000 1154956653 0 0 0 Mon Aug 7 08:17:33 2006
new time 0 4 0000 0000 1160226960 0 0 0 Sat Oct 7 08:16:00 2006
old time 0 3 0000 0000 1160227423 0 0 0 Sat Oct 7 08:23:43 2006
new time 0 4 0000 0000 1159881720 0 0 0 Tue Oct 3 08:22:00 2006
[root@server root]# /usr/lib/acct/fwtmp < utmpx
system boot 0 2 0000 0000 1163450999 0 0 0 Mon Nov 13 14:49:59 2006
run-level 3 0 1 0063 0123 1163451051 0 0 0 Mon Nov 13 14:50:51 2006
(OUTPUT TRUNCATED)
[root@server root]# date
Thu Oct 26 13:54:16 CDT 2006


So, since 23 June, the date has been set and reset several times, forward and backward. In fact, the last recorded reboot is about 8 days into the future. As far as the system is concerned, uptime is a negative value!

The Fix, in brief:

With that mystery solved, and with /var/adm/utmpx already backed up, I decided to clear utmpx out, and try running shareall again.

[root@server root]# > /var/adm/utmpx
[root@server root]# shareall; showmount -e
export list for server:
/export/b
/export/a


I do not know why share cares about the system’s uptime. I’m hoping to have time to go poke around at the OpenSolaris source code to figure out exactly where things went wrong. As it stands, I don’t even know if I could reproduce this problem by changing the system date forward several days or weeks, then rebooting, setting the clock backwards, and then trying to export multiple filesystems. It would be worth finding out at some point.

The new title…

Perhaps you will notice that I’ve changed the title of this web log from “My Spot for Venting About Stupid Things No One Cares About” to “The Daft Musings Hall.”

This is largely due to my extensive work with the French Resistance, and the fact that my Series 5 “Allo, Allo” discs have finally arrived… Not that it is hard for me to resist the French, you will understand.

That got me thinking again about the fine broadcasts performed by Mr. Crosby during WWII at the Kraft Music Hall. In deference to that, I’m happy to bring you “The Daft Musings Hall.”

Listening to these recordings makes one thing abundantly clear in my mind. Something has gone terribly wrong in this country over the last 60 years. We will always have struggles as a nation. Today, we fight the same fascist enemy with a different face. We didn’t want to fight then, as we don’t want to fight now. But then, we knew what we had to do, and we did it. And our countrymen, even the entertainers, WANTED us to win!

Today, we are engaged in battle. Many people in this country don’t understand why. Many have forgotten or ignored more than 30 years of attacks and aggression by people who hate us because we are free and successful. Many don’t even believe it is real. Many more people in this country think that we deserve it, or that we caused it, or that we had it coming… precisely because we are free and successful.

Perhaps the biggest failure of the greatest generation was that they were too successful at providing better lives for their children. We have become too soft. Baby-boomers grew up in a more comfortable world than their parents. The next generation grew up with less hardship, still. My generation, with even less. We demand air-conditioning, padded chairs, and perpetual entertainment from TV, iPods, computer games, and the internet. It has gotten to the point that we’re unable to discern right from wrong, because “right and wrong” are anachronistic abstract concepts that have long since been washed out of the common fabric of our nation.

We are being taught from our youngest years to “open” our minds. We are taught that the only true wrong that a person can commit is to be “judgmental.” Political Correctness is eroding our language, as Mr. Eric Blair predicted. Worse still, many people confuse simple crassitude with being Politically Incorrect… Another example of the erosion our language. Our institutions of higher learning are producing astounding numbers of people who think they are smarter than they actually are. I think that it is attributable to my proposition that once you’ve both removed high standards and made people feel guilty about judging anything, you’ve lost the ability to discern brilliance and the flicker of thought from banal platitudes.

I’m not going to pretend that I know how to solve these problems. I like my iPod and my air-conditioning. The only thing that I can do is to point-out the problems I see, and try to identify them explicitly. The first rule of trouble-shooting is that you identify the problem, then cut it in half. I’m not even sure how to do that in this case, but I will at least offer-up the following positive affirmations:

1. It is good and acceptable to turn off your television. You probably won’t die from it.
2. Read a book. You’ll increase your chances of stumbling upon an actual thought.
3. It is good and acceptable to be critical of people who do the wrong things.
4. It is good and acceptable to be proud of your country.
5. It is good and acceptable to want your country to succeed.
6. It is bad and unacceptable to feel guilty over being successful and free.
7. Our way of life is good. Even if we are too soft. It deserves to be protected from those who would destroy it.
8. Spreading success and freedom throughout the world is NOT fascist, nor is it imperialist. If you think that it is, you’ve been watching too much television.