A Good Imagination and a Pile of Junk

In mid July, our primary linux development host was unreachable.

Eventually it came out that it was in a data center that was being shut down, and while the infrastructure people had communicated this to our team’s management, that information never quite made it down to the line developers.

After some confusion, scrambling and a couple nastygrams they brought the machine back up for 36 hours while they set up a replacement.

Machines housed in this particular data center are easily identifiable by a prefix on their host name.

Eventually the replacement came up (it wasn’t 36 hours) and they pulled down the original again.

Three weeks ago we had one of our (close to a dozen) development databases disappear. People flipped out. “We were doing critical testing…” etc. On a hunch I went and dug up the host name of this database… yep. Decommissioned box.

Last Friday morning there was a production problem and we needed to simulate a situation in production to duplicate the error.

It took 45 minutes to load the database from the production dump, and testing was off and away. About 3:00 in the afternoon I heard the developer doing the testing, a 20 year man, emit what I could only assume was an expletive in Croatian.

“I can’t connect to the database! Can anyone else get to the database?” It was the Friday before a long weekend so I didn’t stick around to hear the resolution.

Yesterday, the first day back from the long weekend the same guy needed a database, presumably for the same test. As I was working on deployment scripts I said just to take mine. They were worried that I was volunteering something I shouldn’t, but a production problem generally takes presidence over development work any day. They hemmed and hawed about it. “Just take it. It’s not my critical path right now. You need it more than I do.” They nervously looked at each other.

The analyst called our boss, who was on her last day of vacation, and asked if it was ok. I was gobsmacked. She said it was fine, and they were off and running.

This morning I got in and was going through email. There was a back and forth between one of our other developers and database/infrastructure teams asking about why he couldn’t connect to the database HE was using.

And of course the host had been decommissioned. I just started laughing.

Now, with the database vendor we use, database names are all mapped to their host and port names in a single file. I went to this file and scanned it for all hosts that were in the data center being decommissioned. I built that list and sent an email to the team explaining that it was a LITTLE bit silly for us to act surprised when these databases disappear, and by the way, here are the rest of the databases that are going to vaporize.

About a half hour after I sent this email, I heard “Hey Mike, can you get to that database you were using?” (Referring to the one I’d lent him yesterday.) “I can’t seem to connect.”

I went into a five minute gigglefit. “I’ll bet you lunch I know why.”

“Why?”

“Dude, it’s a cmc box. It’s being decommed. It’s gone. Did you see the exchange with Xiao from last night about his database disappearing? Same issue.”

I went and checked (pinged and tracerouted) all the hosts that were under contention, confirming that they were indeed all gone.

Our boss asked him what was going on and he stuttered a bit as I walked over, then pointed to me. I gave her a quick overview; that database boxes are in a decommissioned data center, that we’d known about it for months and that our surprise that these things had all disappeared was a little weak.

“Do you want me to follow up…”
“No, I’ll…”
“Got it.”
“Let me know if you need anything from me on it.” She gave me a half nod and I knew I was dismissed.

I came over here and opened this buffer in emacs to let off some steam. A couple minutes later I saw Vlad’s reflection in the cubicle glass, coming up behind me, so I paused my Johnny Cash (Folsom Prison Blues), pulled my ear buds and wheeled around.

“It would be nice if these infrastructure people would tell us that they were taking down these machines before just doing it.”
“Well, we knew the data center was going away in July. Remember? When they pulled our development box?”
“Yeah, but they should really let us know. Maybe there was a conference call I wasn’t invited to.” I let it go.

We talked for a couple more minutes about the replacement databases we’re supposed to be allocated and how they weren’t going to be very useful (a whole different set of issues. I’ll vent about that when it blows up in a couple days. It’ll be fun, really.) Then he wandered away muttering about the injustice of it all.

Now, believe it or not I actually do appreciate, as the teller of these tales, that I may have a certain kind of narcissistic myopia for which I am more or less famous, depending on how long you’ve known me.

But Nobody on the team has less information than I do. There aren’t emails I received that they didn’t. There were no conference calls they weren’t included on. But there’s no follow-through. I just don’t understand how people can be surprised (to take this particular instance) that the databases disappear.

My head just spins with “What did they think was going to happen?”

I’m left here looking for a sensible epilogue with nothing but the quote from The Bird Cage when Gene Hackman finally begins to grasp what’s going on and just says “I feel like I’m insane.”

It’s clear to me only that I can’t see the forest for the trees somehow. I’ve got a bad primary directive in there someplace. Either I should be able to cope with this (a notion that strikes me as a concession to utter madness) or find a development shop full of people that don’t behave like a bunch of petrified lobotomized cogs.

Believe it or not, what makes me most mad is actually NOT that this behavior is going on. It’s that I know these guys are all smarter than this. We have 3 team members with PhDs in applied math or physics. The ‘long timer’ has been in this division for 20+ years and is a kindred spirit with regards to skunkworks code generation and refactoring projects.

These people aren’t stupid. Nor are they cantankerous curmudgeons. It’s actually a good bunch of guys. But they’re just entirely unwilling to act outside of what they’ve been told.

If it were a bunch of morons I’d bitch about my circumstance, but be more easily able to deal with it. As always, it’s the incongruity that I find so utterly incomprehensible.

As always, any input is thoroughly welcome, even (perhaps especially) if it’s input you think wouldn’t be particularly welcome.

This entry was posted on Wednesday, September 8th, 2010 at 11:25 pm and is filed under Uncategorized. You can follow any responses to this entry through the RSS 2.0 feed. You can leave a response, or trackback from your own site.