Travelogue

May 8-12 Newark, California: SunUP Network Forum

Wednesday

SunUP didn’t start until 09:00 today. Missed the shuttle bus. No big deal. The campus is only a few miles from the hotel. Got delayed in toll-booth traffic for a few minutes. Drove straight to the right building, no wrong turns. This place is MUCH easier to get around than New York.

The first talk this morning was about the Sun-Fujitsu relationship. Basically it amounts to a way for Sun and Fujitsu to trade some technology for a few years, until they can find a way to screw each other. Diplomacy is the art of saying “Nice Doggy” until you can find a big stick.

Then came the Solaris 10 migration “Lessons Learned” session. This was possibly the best talk of the conference. Some of the highlights were:

• Most applications experience a performance gain simply by upgrading to Solaris 10.
• The IP Stack has been rewritten to vastly improve performance.
• A “Container” == A Zone + Resource Management.
• “Whole Root” local zones!!!
• All Zones in a domain share the same process table. So a fork-bomb in any local zone will crash the global zone. I knew this already, but it is nice to see Sun admit to it.
• Memory leaks in a local zone can also take down the global zone. I didn’t know that, but I suspected.

It would be nice if Sun were actually able to get Zones to be as fine grained and self sufficient as LPARS on an IBM mainframe, but they have a LONG way to go.

This would have been the most appropriate discussion to bring up some of the gripes I had about Solaris, but it didn’t seem right to voice them to the guy who migrated his datacenter to Solaris 10. It would have been REALLY nice to have had access to an actual Solaris Engineer.

Then we talked about the new DIMM replacement policy. Most sites like to replace DIMMs that are throwing Correctable memory errors, under the assumption that soft errors will lead to hard errors. Sun did some research, and found that 70% of these correctable errors were replaced on ’suspicion’ of being bad. They collected 800 of these DIMMs that were throwing correctable errors, and ran them all for 5 months under heavy load. They found that at the end of that 5 month period, they didn’t have a single non-correctable error (read system panic). I know that we replaced a LOT of them on our E10k machines in the first
two years I was here.

The new policy is to replace a DIMM only if it has thrown 24 errors over 24 hours. I’m not sure how this meshes with the new Memory Page Retirement functionality that was introduced in Solaris 10, then back-ported to Solaris 9 and Solaris 8. It seems like MPR would retire pages of memory (essentially a “bad block map” for RAM) before they hit that threshold of 24 in 24, and you’d never see enough errors to replace a failing DIMM. They had a customer testimonial, and the guy said that they don’t bother replacing a DIMM until the memory error is logged as persistent. That is how we’ve treated them for the most part over the last few years, anyway.

Sun also suggested the new cediag. This new and presumably useful tool does not ship with the OS, but
instead the 5.0 version of the explorer package. Talking of which, why isn’t explorer part of the OS by now??

The only choices for technical break-out sessions were “Capacity Management” and “Disaster Recovery.” I stayed for the DR discussion. It wasn’t very useful unfortunately. That being said, I’d like to see more break-out sessions next time, particularly ones with Solaris engineers.

The next discussion was on Time Dependant Reliability (snooze). The guy giving the talk was so far above the heads of the audience it wasn’t funny. The crux of his argument was that MTBF is a poor tool for reliability analysis.

The last thing we did was to plan the next meeting. Hopefully, it will be at Sun’s Broomfield campus. Fat Tire is plentiful near Broomfield because the brewery is less than an hour away. I’ve done the tour, and quite enjoyed it.

Wednesday night, I had dinner with Stephen. As good as it was to see Shannon, it was better to see Stephen because I did get to hang out with Shannon and So Jung over Christmas. Stephen, I hadn’t seen since one week before I got married, very near five years. Stephen didn’t have long. Something about Google working him to death, I suspect. Still, it is incredible to me that with real friends, the passage of time evaporates when you get together. It has been eleven years since high school, and it just didn’t matter. I really appreciate that, since it reassures me that I made the right choices in friends so long ago. We ate at the same steak place I had eaten at on the first night. I had two Lagunitas India Pale Ales which claim to be made with 65 different malts and 43 different types of hops. That is incredible. Needless to say, the first one was so good, I had to have a second. Stephen had to leave early, but it was so good to hang out with him that I didn’t care. Hopefully, I’ll get to go back some time.

Thursday

Got up at 09:30. Checked out at just before 10:30. On the I880 toward San Jose. I only missed one turn going into the airport, mostly due to construction around the airport. Flight was supposed to depart at 12:15 PDT. We had to wait on the plane at the terminal for an hour, while they fixed the plan with duct tape. Seriously. Ok, ok… so the problem was that one of the overhead bins came unhinged, and they had to tape it closed. I really didn’t think I’d make my flight from DFW to HSV, and I was certain my luggage wouldn’t. Fortunately, I got to the gate just as boarding was starting. My luggage also made it to HSV unharmed. All-in-all, long, boring, and full flights, but safe ones. I got to Huntsvegas at about 20:15, made it home by 21:00.

Travelogue

May 8-12 Newark, California: SunUP Network Forum

Tuesday

Set the clock for 06:30. Then 06:45. Then 06:50. Amy called 06:46. Got
up. At least I didn’t have to iron anything.

Got the shuttle-bus to Sun’s Menlo Park campus. Like everything here, it was
beautiful. Nice view of the bay and of the mountains. Nice green trees. Just
like everything here.

Signed in with about 50 or 40 other people. They had packets
on the table with everyone’s names on them. When you got your
packet, they handed out Mikasa crystal wine stoppers in the shape
of grapes. This is wine country, I suppose, but that was the
strangest schwag I’ve gotten since EMC handed out toe nail clippers
in 1999. Once you got in the door, they had a table full of
circa-1999 Sun Blueprints books for grabs. I snagged 6 different
ones. A nice addition to the book shelf, but they are all hopelessly
out-of-date and mostly useless.

This was a long freaking day. We started at 08:30 PDT, and didn’t really
stop until 18:30 PDT. By about 15:45, I was ready to start throwing things.
Really, people. If you have this much junk to cover, make it a three day
conference!

As a result of the long time sitting, I began to think of ways Sun annoys me.

1. Solaris excluded, Sun isn’t any good at making software. They have a
   lot of tools that are either half-finished, half-useful, half-tested, or
   half-assed in some other way I haven’t listed yet. Worse than that, they
   have a lot of products with overlapping functionality, and don’t really seem
   to understand the concept of code reusability. I really wish they’d get
   their act together, particularly with systems management software. They
   spent probably three hours out of the day, showing off several new offerings
   that they were really proud of, but were nothing more than an extension of
   the current mess they have (or in a couple of cases, completely new
   messes. One dude told us about this new program going by the moniker SMC. I
   don’t remember what SMC stands for in this case because Sun already has
   at least three other products called SMC.
2. Product names. Is it Netscape Directory Server? Or maybe iPlanet. Or
   maybe SunOne. Uh… how about Java Directory Server? Sun Management Center
   (SunMC, not to be confused with Solaris Management Console: SMC) apparently
   used to be called Symon. Netconnect is now being renamed into two different
   products (that only half-work at this point). PLEASE stop renaming things.
   Or at very least, remove all of the old names from the documentation.
3. Java AWT(Abstract Window Toolkit)/Swing. It is old.
   It is slow. It looks like it has been beaten with an ugly
   stick. When most people say “Java sucks!” they are talking
   about AWT/Swing applications. For some reason, Sun has
   chosen to make all of their systems management applications
   with Swing, rather than native tools using C and probably
   GTK (since they are now shipping Gnome). These apps look
   like they were written in 1989. They are way too slow to
   be useful to anyone. They only work half the time. If you
   need to update the Java Virtual Machine on your box, they
   probably won’t work at all. If you need to run them over
   a remote X-Windows session, the best thing to do is forget
   it. Curiously enough, Sun are not the only guilty party
   here. Veritas has recently perpetrated this with their
   NetBackup admin client. I think that Apple did it right
   by providing a Java interface into Aqua. As a result, you
   can write Java apps that at least look like native
   applications, though they may not always behave like
   native apps. From what few I’ve seen, they actually perform like native
   apps too. What Sun should to is to hire some GUI designers from Cupertino
   to work on a replacement for Swing (which was a replacement for the AWT), or
   just freaking license the Java + Aqua layers. Yeah, right. I know that UNIX is predominately a text-oriented system. I am
   fine with that. However, if you’re going to provide GUI tools and force us to use them for some tasks, PLEASE make them
   useable.

   In a lot of ways, I think that Apple has ruined things for other UNIX vendors, by proving that UNIX can look good and be functional.

4. Could you guys please make better LDAP server and client configuration
   tools?! This is one thing Microsoft has got right. It is really easy to
   configure a secured Active Directory server, and connect clients to it,
   without ever passing clear text passwords over the wire. Replication to
   redundant servers is apparently not very difficult either. They have had
   this working well enough since NT 4.0, and you haven’t. Period. If system administrators can manage to figure out exactly what documentation they need for the Directory Server, it is possible
   to get an LDAP server running, with clients authenticating to it in
   probably about 48 hours if you’ve never done it before. Now try adding TLS.
   Good-freaking-luck. We don’t need to go dinking around with ten different
   tools for creating self-signed certificates, etc. You should assume that:
   1. We need Transaction Layer Security by default. These days, it is
      NOT acceptable to pass clear-text passwords over the wire, unless I
      specifically tell you to.
   2. Unless I tell you differently, self-signed certificates are OK. Ask
      me if I have a “Real” cert, and if not, CREATE A FREAKING CERTIFICATE
      AUTHORITY. Make the admin tools smart enough to push the proper trusts
      out to the clients when I initialize them. Instructions for working
      with self-signed certificates that say things like: “Open the Netscape
      Web Browser” are NOT ACCEPTIBLE. This needs to be automated, easy, and
      above all needs to “just work.”
   3. pam_ldap needs to be smart enough to allow RSA key authentication
      for password-less logins over SSH. You might be able to talk me out of
      that one. It would at least be nice if a system administrator could
      allow that for specific accounts.
   4. Kerberos integration should be documented and as easy to implement as it is on Windows (nearly invisible).

   In other words right now, the Directory Server that ships with Solaris is just a tool. Sun needs to evolve a little bit by providing the tool integrated with the design, configuration, and deployment tools to make it useful quickly.

5. Jumpstart needs to be updated to be smart enough to
   use DHCP. There NO excuse for this. And “Go download JET” isn’t a good answer either, unless Jet both grows up and gets shipped with the OS. Again, this needs to be automated, easy, and just work. We have to use this tool often, and have committed significant time into customizing it for our environment. We shouldn’t have to dance around RARP any more, since DHCP has been the
   standard for at LEAST 10 years.

The patch management discussion nearly drove me over a cliff. They are
trying to make better tools, but it looks like they are worse and that
is really a shame. I can’t imagine that people are going to want to use
these tools unless something dramatic happens. If Sun are planning to charge for these tools, I will laugh.

The most interesting quote of the day was “You cannot manage
availability. Availability is a result.” That is simple, but quite
profound, and I’m glad they are thinking along those lines. Also
interesting is the statistic they gave of the % chance of a system
administrator inadvertently causing an unplanned outage: 1 in 200. So,
every time I log into a machine, I have a .5% chance of causing down time
such as accidentally rebooting production instead of a development machine,
etc. Excellent.

Over all, I left the first day of the conference a LOT more agitated than
when I went in. Hopefully I will get some opportunity to provide input
(read vent) about some of these things tomorrow.

We did get a chance to tour the iForce center today. That was pretty
neat, but I already have an E25k, and they could have done the tour in 20
minutes instead of more than an hour. It got old fast.

I skipped the free dinner, in favor of going to the Apple
Store in Palo Alto. It wasn’t worth the time. I was very
disappointed in it, as the Mac Resource in Huntsville is
way better than this place. Ate at PF Chang’s in Palo Alto.
Had the Orange Peel Shrimp and a Fat Tire.
All I ask of life is a plate of shrimp(or maybe oysters) and enough Fat Tire
to choke a goat. Maybe one day, New Belgium will expand
enough to be able to ship to Alabama. For that matter,
maybe one day Alabama will change their beer laws to make
that worthwhile.

Got Amy a present, then headed back to the hotel.

Travelogue

May 8-12 Newark, California: SunUP Network Forum

Monday
No required SunUP events today. I
think most people used today as their travel-day since Sunday was
Mother’s day (thanks again, Hallmark). Woke up at about 8:00
PDT/10:00 CDT. Let me tell you… It was nice. I still haven’t
gotten used to DST, so I felt like I was taking my hour back.

Got all of my ironing for the week done. This is always a PITA, but
at least it is done now, and I won’t have to worry about ironing
anything else until I get home. Got out on the road. Took US-101
into San Francisco. Crossed the bottom deck of the Bay Bridge, then
turned around and headed back into SF on the upper deck. That was
a really nice view of the Golden Gate Bridge, Alcatraz, etc…
Drove around down-town SF for about 30 minutes… Seen one city,
seen them all, and what do you know… It looked JUST like it does
on TV.

I’d killed enough time. Now it was 13:00, and time to head to
Monterey. I didn’t bring/buy a paper map. But I got good directions
from Google Maps, plus I had the GPS. I’d done a good job of route
planning, making sure to go through Cupertino. I quickly typed in “1
Infinite Loop” into my Garmin eMap, drove there (there
is a REALLY nice looking Target just around the corner), and took a GPS
waypoint in the parking lot of Apple Computer’s main campus. It was
really a nice campus. Didn’t have much time, so I didn’t dally. Back
on the road.

Talked to Sutton about 15 minutes out of
Monterey. Great timing. Monterey/Pacific Grove was absolutely beautiful.
Seeing Sutton again was awesome. It is amazing to me that it has been more than
10 years since we went our separate ways, and when we get back together, it is
like nothing has changed. Shannon and I have each traveled a boat-load of miles
on a lot of different roads since high school. Despite that fact, I still
regard him as a brother.

Shannon, you have an awesome wife who understands you. I know that because I
can see it. Not many people have that, but I can tell you from experience, that
it makes life enjoyable and worth living. Brandon, you’d better be taking
notes.

Shannon took me on a brief but very productive beer hunt. Didn’t find the
Stone Vertical Epic that I was looking for, but they had Fat Tire (and New
Belgium Abbey). I know that I’m going to Denver next month (it is a
traveling year, after all), but I just couldn’t wait. I know that there are
many good California beers that are worth my time (I bought a bottle of
Lagunitas something-or-other that is still in the rental car). But when New
Belgium brews are available, I just can’t help myself. Fortunately, I
restricted myself to $15. It was difficult, but I managed. We also discovered that Shannon’s dog, Judith, likes dried banana chips. I doubt that our feline would be so inclined, unless someone’s making fish-flavoured bananas now.

Also of note was the really cool tree in Shannon’s front yard that had purple flowers and orange leaves. I wish I knew what kind of tree it is because Amy would love it, and try to plant a dozen of them.

Travelogue

May 8-12 Newark, California: SunUP Network Forum

This is turning out
to be quite the year for travel. I guess that is a good thing
because it certainly makes life more interesting. However, it tends
to generate a lot of extraneous paperwork, particularly when traveling
for work.

Sunday
Not much here. I had a 12:55 flight
from Huntsville to Dallas, to San Jose. Had weather going into
Dallas, but landing wasn’t too bad. I found out that the Scurvy Pirates make really
good nerve-calming music leaving out of Huntsville. I’m kind of
strange about flying. I usually am pretty freaked out during my
first take-off on any given trip. After about 10 minutes in the
air, I usually calm down for the rest of the whole trip. Somehow
listening to “Worse Things than Dying” helped this time.

To get myself into the California state of mind, I dug up all
of the California songs on my iPod at 35,00 feet. Ok, there only
turned out to be two. Red Hot Chili Peppers’ “Californication” and
the Dead Kennedy’s “California Uber Alles.” Fitting.

The approach into San Jose was ridiculously “entertaining.” I’ve had worse landings, but there were a couple of pilots sitting behind me cracking wise the whole way down.

Got my luggage, and drug it out to the shuttle-bus stop.

Douglas Adams wrote somewhere in the Hitchhiker’s Guide to the
Galaxy series, something about having that inner twinge that lets
you know just exactly how far you’ve gone from the place of your
birth, and in that instant, experiencing some small part of the
vastness of the Universe. On Earth, he wrote, most people don’t
really notice it because you can never be more than 36,000-odd miles
from your birth-place at any time. Regardless, while waiting for
the bus to the rental car place, I got the twinge. It isn’t unusual
for me. It was just a small matter of wondering why (oh, why) did
I make the decision to take 10 hours out of my life, to go careering
across the sky at 35,000 feet in a pressurized metal tube, and end
up 2000-odd miles from the relative comfort, safety, and routine
of my home. What possessed me to do such a thing, and why the hell
am I here, when I could be at home? Douglas Adams fans would
probably answer: “Excitement, Adventure! Really Wild Things!!!”

I got to the rental car place, and secured an upgrade… I’m a
careful driver, but I don’t consider compact cars to provide much
protection in the unlikely event of a crash. Besides, it was only
a couple quid more per day. I ended up with a Camry (grey) that
came fully equipped with a Sirius radio. Ahh… right at home then. I
found the hotel very easily, after only a 20 minute drive (and only
one wrong turn). GPS is a lovely thing, and I don’t think I’d ever
like to travel without one ever again (ever).

Found a steak joint across the road from the hotel. Had a nice NY Strip (medium,
thank you), and my very first draft Sierra Nevada. The bottled
version is good, but this was awesome. Google maps tells me that
I am less than 200 miles from the brewery. Nice.

Laugh of the day.

Amy found this today. I guess that it is possible, but I really don't believe it is. Regardless of the fact that in the late 1800s, the Coden area used to be a resort, the Mobile side of the bay has been known as the “Working” side of the bay for a long time. Also, I doubt that there is much that the developers can to to change the water from muddy brown to anything moderately resembling a pretty blue or lovely green, and ceartainly not “crystal clear.”

Travelogue

April 9-10 Oak Mountain State Park, Birmingham, AL.

Tony Cowan was in Birmingham for a Phi Mu Alpha Sinfonia event on Saturday. He was planning on staying the night so he could hang with Robb for a while. So, I suggested the we just camp out in the park. Bought a cheap 8′x8′ Coleman tent the night before… $38 at Wal-mart. Excellent. I haven’t been camping properly since I was about 16. It was LONG past time to do it again. Got to the camp site just at dusk. Got the tent set up, but it was too dark to go dinking around for fire wood. Went to the little camp store, and it was $5 for a bundle of about 7 or 8 bits of split log. I only bought one, which was enough to get the fire started.

It ended up being me, Tony, and a guy called Zack from Mobile. Robb show up after he got off of work… around 9 PM. Knowing that the beer in the metal cup thing had been so successful before, I brought 3 litres of my brown ale. That stuff should be gone soon. All of the Scottish Ale is already gone. So we drank some beer, grilled some food, and poked the fire for a few hours. By midnight, all of the food was gone. So was all of the firewood and charcoal that I had packed with me. The camp store was closed by 9 so, we were pretty much out of luck. Robb had the bright idea to go to a grocery store to find some. We had talked just a few minutes earlier and discovered that both Tony and this Zack guy used to work at Delchamps. So, someone
suggested that we go to Delchamps for firewood. Then Robb made the quote of the night:
Unless you’re driving a Delorean, we’re not going to Delchamps.

Then again, maybe you had to be there.

We found a place open, but they didn’t have any proper fire wood. They had charcoal, but what we needed was light. And charcoal isn’t renowned for burning brightly. We settled on a couple of those freaky firelog things. Turns out, they were just what we needed. Three of them burned all night, and threw enought light to see by. The campground rules said “no open fires” but there didn’t seem to be anybody paying attention to that. We also didn’t pay much attention to the no glass bottles rule. Robb ended up leaving at about 3:30 AM. It was a nice night, so I just layed my sleeping bag out on the picnic table. This was the first time I’ve used one of those mummy style bags. It was a bit clostrophobic for about the first 10 minutes. Then I started to get cold, and it was fine. The bag is rated to -10 F, and I now believe it will take it. Sweet.

At about 10:00 PM Saturday night, Amy’s dad called, so I just called her and told her to call him. Apparently, her grandmother had fallen and broken her hip. So, she called me at 7:00 AM Sunday. I got up, broke camp, then drove home. It was a good trip, and I really enjoyed catching up with Robb and Tony again. Now that I have all the gear, I’ll probably go more often.