Thursday, January 28, 2010

Server Replacement Project

Day 1

This is going to fail and I'm going to be blamed. Competent guys are running around like chickens with their heads cut off. Why do all my vendors lie to me about whether or not their software will run in a new environment? I hate IT and I want to go work at Starbucks. I hate feeling like I am Responsible for things that are out of my control.

Day 2

My team is wicked awesome! These guys can spin straw into gold. My vendors are awesome! One guy stayed late to help me sort out a problem and get his company's application up and running. I am awesome (and competent.) Two SQL applications down and one more to go! Who cares if I have to work late or come in on Saturday, I can't imagine doing anything else for a living.

There's nothing like feeling like a competent member of a competent team.

Day 3

Spent 4.5 hours reinstalling and restoring the most annoying SQL based app in the house. Unlike the others, I did almost all of the heavy lifting-with help from tech support. I had the IT Contractor sit with me and go through the documentation and install with me because this application is Difficult and I had done the previous upgrade myself-with terrible results. This came in handy when the application on the (test environment) client couldn't see the server and the vendor's tech support was stumped. The nice, patient, tech support guy consulted with his colleagues. By the time he came back with "Something about DNS Flush?" it had occurred to one of my technical contractors that the DNS was screwed up and we resolved the problem.

And really, even *I* know that it's flushdns-not DNS Flush.

Working on this project is interesting-not just because we're starting with 1 SBS 2003 server and ending with 1) Server 2008 Domain Controller and document vault repository, 2) 64 bit SQL Server 2008 SQL fish tank for some of our SQL apps 3) Server 2003 with SQL 2000 machine as a fish tank for those applications that just aren't up to playing in a SQL 2008 environment (including the one that took 4.5 hours of my time this afternoon.) But also because I'm working with the person who taught me most of what I know about IT. He and I no longer work as closely as we did in earlier days, and I've learned a decent amount in his (for lack of a better word) absence. Because of the way the company has changed, he hasn't been around as much (we couldn't afford his services and my Boss pissed him off so he's been (understandably) disinclined to work for free.) Because I couldn't just call him any time I got scared-because of an Excel Bug or a Worldox glitch-that's a document management system-I've learned to do a thing or two by myself.

So while I don't claim to understand how SQL apps work (I don't even know what exactly an instance is) and all I know about their backups is that you have to use their native agents to make it work (unlike Access-it's not like you can just copy yesterday's .MDB over and make it work) and while I don't have the knowledge to keep our servers up and running, I know more about what it takes to make my company's apps run than anyone. And the knowledge that I'm missing is just knowledge-it's not that I'm not smart enough to be a proper Sys Admin-it's just that other people have devoted their time to knowing more about That Sort of Thing, whereas I have devoted time to knowing how to reinstall our own weird apps that no one else-outside of the fee-only wealth management environment-is likely to use.

So, my mentor underestimates my IT cred a bit. I can live with that. It's not like I plan on making a living installing and supporting these applications anymore (See also-MBA Program)so I'm not offended that he didn't notice I could do it without him to hold my hand. It was also interesting to see the point at which my mentor decided to call in tech support. It reaffirmed my belief that there is no shame in doing so.

You did what the documentation told you to do, and it didn't work. Not your fault.

There's a lot of work to be done tomorrow, but I'm confident that by the end of the day everything that is within our control will be taken care of.

Day 4-ow

Our "Technology Area" (the room outside the server room that we're allowed to keep locked) finally has the requisite amount and of IT clutter. There's a stack of cardboard boxes from Dell as tall as I am in one corner. In another corner, two non-working Dell towers are stacked on top of each other with the switch that's tying our "test environment" subnet together perched precariously on top of them. Multi-colored Ethernet cables snake out of the back of the switch and connect up various servers and test-client machines. On top of *that* is a box that once held an NAS and the plastic front grate for some server or other.

While I find the clutter respectable, I have to admit that it's a Bad Sign as one of my contractor colleagues is one of the neatest humans I know, and the fact that things have degraded to such an extent means that he must be seriously distracted. Two brand new rack-mountable servers (total value $25,000) sit one on top of the other on one of the desks. I can tell that they were set up in a less frantic stage of the project because my neat-freak colleague (let's call him E) used a few hard drive cages to keep them off of the desk surface to allow air to circulate and also managed to tie up the cables that connect them both to a KVM switch.

By way of contrast, the desk at which I spent most of the day was clearly a later development. Underneath it sits one server (not the server we plan to use in the long run-just the one that's subbing because the server we planned to use needs a new power supply which no-one has had time to install yet) and a desktop. There's a a KVM switch on the desk, but it's just for decoration since one of the devices has PS2 ports and the other has USB ports. So there are two monitors and two sets of keyboards and mice. Oh and a netbook (which has something dreadfully wrong with it-so it's not really a good test client and which I spent most of today threatening to throw out the window.)

I got up at 6:30 this morning and walked to the commuter rail station. It was 3 degrees Fahrenheit out. But I was okay with this all because today we were going to Get 'Er Done. Maybe we'd finish by 4 and we'd all go out for coffee or drinks and sit around congratulating each other and telling stupid stories about the project ("Yeah-I was really scared for a minute when you said that Application X wouldn't come up, but then I slapped you upside the head and we fixed the mapping and it was all good!")

This is not what happened. I got there at 8:30 and the computer genius contractors got there at 9. The three of us worked like fiends all day long. The head contractor went off to Microcenter to pick up a cable and a power supply that couldn't be found in our giant pile of cables, E and I worked on restoring databases, writing a script that would map all of the logical drives in the new environment and making sure we could restore all of the users' desktop shortcuts and background pictures of their pets.

The heat in the building went off at 3 PM. By then we had shut off the AC in the server room and were making nervous jokes about warming our hands over the server when it got too cold. Sometime around 3, the head contractor suggested that we might just have to stick with the old environment and come back again next Saturday to do this again. "Anything but that!" I thought. Our credibility will be ruined. I pared the list of workstations that needed to be operational back to the list of people that would actually be in the office on Monday.

The head contractor went out to get E a tuna melt and me an espresso. I was leaning against things and yawning (I did get up at what can be considered Fuck You o'clock in the morning since it was Saturday)I had made noises suggesting I needed an espresso (and E badly needed to be fed) but when E said "why don't you go get one now?" I explained that it only works for a little while-so I needed to wait for just the right moment. When the double espresso arrived I wished I'd ordered a triple. I was suddenly awake and useful again.

Progress was made. We restored two databases successfully with all of Friday's data and managed to get the logon script to work (this maps all the logical drives for the users) in the test environment.

The head contractor left at 5 because he had to get home to his wife. we all agreed on the steps that needed to be taken to bring everything up and E and I were sure that we could be done by 6 ("What Could Possibly Go Wrong?")

We did everything that we were supposed to do. We still had epic flailure with our CRM. We called the head contractor (as he left he did say "Call me.") and we tried things that he suggested for half an hour or so (I think we managed to piss his wife off too.) And in the end we failed. Things that worked in the test environment didn't work in the real one-no matter how much we replicated the real environment. No matter that we'd tested them two days ago-when the vendors could be gotten on the phone before trying them out in reality and no matter that I was working with the two biggest computer geniuses I know.

Worse, even though it's still not working (and won't be until we can talk to the vendors on Monday), I *still* have to go into work tomorrow. Because there are still things that need to be done before the users show up, and I left at 7 because the next commuter whale train to take me home was at 8:30 and there was more than an hour's worth of work to be done.

I really don't know what the lesson to be learned from today is. I'm going to have a wicked hard time explaining all of this to my Boss (who knows as little about computers as I do about astro-physics) but I'm not sure I'm going to be able to explain it to my computer-savvy colleagues either. We really did bust our asses for several days running. We really did everything we could to try out all the applications in a test environment before trying it for real in the place they have to work. And still it failed.

When we left E asked me "is there anything else that we have to be worried about?" I assume he meant "did we leave any doors unlocked" or "are we going to be able to get out of the building?" but I responded "You mean aside from the fact that our CRM won't work?!" We laughed big, punchy, "we just busted our asses for 9 hours and it is still not working" laughs.

We got on the orange line together. I said "I still need to call the owner who came in today to tell him that the network isn't up and running yet." E said "Well, he can use the internet?" "yeah" I said that's great. He can log in from home and use the internet here. Why would he do that?" More punchy laughs.

I don't understand.

Day 5

There really isn't a lot to say about today. Last night I texted a friend who lives locally and begged her to let me know if she and her boy (and her dog) were going out for breakfast this morning, because I needed to talk to people about something other than DNS problems or cables or SQL. The friend (who I'm recommending for canonization) didn't get my text until this morning but called me twice-until I woke up to say that she and the boy were going out for breakfast and would I join them for brekkie and time with the dog on the beach? I said yes of course-just let me put some clothes on. At breakfast, when I started to ramble on about how badly this project was going the boy said "Cantabridgienne! Cntrl Alt Delete! Start, Run, Cmd, CD..Change conversation!" And then after that I said "oh yeah right". And he sat back and looked pleased with himself and said "I told you I speak geek." That was all much kinder (and probably more effective) than saying "shut up Cantabridgienne!" because I had in fact been spending quality time with the command line-using language that I could have sworn went out of style sometime around Windows 95 (but hey apparently it's still useful to know how to Change Directory.)

The nice friends then took me to the beach with their dog.

And then they took me to the commuter rail and I went back to work. It takes a special type of masochist to go back and do work that needs to be done-even though you know the project as a whole is still not a go. But even though the network applications were still fubarred, I had to at the very least, go in and make all of the client machines join the domain and then run the "files and settings" wizard so that they'd all get their desktops and all the files and shortcuts (and all their IE favorites) back. To do anything else would have been irresponsible.

I arrived to discover an e-note from the head contractor asking for a status update. I responded that I was in house and ready to try out any brilliant ideas he had. He called my cell (he interrupted Shane McGowen singing about some situation that was *much* worse than joining the domain on 15 different computers) I got onto the SQL server and he logged in remotely to try a few things out, and then again on the domain server. And then he had to go somewhere and it was just me with 15 deskktops and 2 databases that didn't work.

While he was on the phone with me I had a foolish hope that he would come up with a solution that I could implement. I looked at the servers and thought about the train home that I'd planned to catch and realized that if he said "go change the DNS settings on all the computers in the office and everything will work" I would have done it-even if it meant that I was there til now (9:00 PM.)

Perhaps it's the result of my exposure to Catholicism at an early age or maybe my parents just raised me this way-if you give me a project and say that 15 people are counting on it being done I will happily stay late, come in on Saturday, sacrifice a chicken or give a pint of my own blood to make it happen. But that's not the way the world functions. Willingness to work hard is not enough.

In this particular instance I have no idea what "enough" is, because I and my colleagues planned, tested, and worked hard. While I was at breakfast this morning, my friend's geeky boyfriend said "I really don't understand why Dell doesn't send people out for server migrations." and then I explained to him that in fact, we had ordered a technician from Dell, but unfortunately, he was an idiot and that the end result of us paying for a technician to come out and install things was that my colleague E got to baby sit the "technician" to make him do his job.

On Friday I remarked to one of my friends at work that I really wished it was Tuesday next. Another friend said "why? isn't it going to be fun?" It's only fun if it works.

Day 6

There is a light at the end of the tunnel-let's hope it's not a train. When I got to work this morning I was as tired as I'd ever been in the past 10 years. I felt like I was an undergrad who had pulled an all-nighter. By 9:10 the document management system was (supposedly) running. I spoke with a tech from the CRM company at 9:30. In spite of the fact that he spoke very softly (and could barely be heard over the noise of two servers) he gave me a list of steps to take on every single workstation to make the software work. I took screen shots, and then proceeded to go through the routine on every single computer with a regular user. This took a long time. People told me the Document Management System wasn't working after all. I told them I'd deal with it next. It's a very special feeling-being dead tired and having a very long list of very important things to do. I told myself, that at least I'd surely get to leave by 5 today (hah!)

By the time I left at 6:15 the list for tomorrow had been reduced to a mostly manageable number of things. But damn. I'm getting awfully tired of stumbling home at 7:40 too brain dead to care what I eat for dinner or do more than sit in front of the computer going "duuuuuh."

Of course tomorrow I can't stay beyond 5 because I have class. But on the other hand I have class-which means getting home at 10:30 feeling brain dead.

Day 7

What day is it? Have I really only been doing this for a week? I was appallingly tired this morning. I briefly considered postponing tomorrow's social engagement because I am running out of things to wear to work and I have less than 5 pairs of clean underwear. But this is one of exactly 2 social engagements I've had in the past week, so I won't postpone it.

I arrived late to class after having failed to do the reading, because I have devoted every spare minute of quality brain time (now doesn't count) to this project for the past week. Literally. It's been a week. I can't remember the last time I devoted this much effort to something. I didn't work like this the week before I took the GMAT, and I certainly didn't put this much effort into getting into grad school.

What am I getting out of this project that is worth the sacrifice of so much time? There's the joy of competence I suppose. My mentor is no longer anywhere near under-estimating me. And I love working as part of this team and just doing pure technical work-no bullshit task-force meetings or phony "book report" presentations in staff meetings or silly projects that really ought to go to the administrative staff.

The most valuable is the knowledge that yes, in fact I can do all of this. I'm not quite sure it's worth the price I've paid so far.

Day 9
Make it stop-oh please make it stop! Yesterday one of the brand new servers crashed. Today I went into work thinking (not for the first time) that I just had a few more calls to make to software vendors and then it would all be over.

Alas no. The Difficult SQL app that we installed in a separate, Server 2003 environment on the C drive (like the vendor insisted we do) died repeatedly. I spent all morning on the phone with them getting it to work. I hadn't planned on giving them that much time, but hey (I told myself) that application is important. We're a Financial Services shop and we need it to place trades-it's not the first time I've suddenly had to spend several hours on this particular application.

But this afternoon I noticed a bug and I called tech support to see if they could help me. While I was on the phone with the rep from tech support, I recognized the problem-we'd had the same issue with the software two releases ago. So 1) apparently no one other than us is using this feature-otherwise they would have fixed it. 2) once again I was in a position where I called a vendor for tech support and discovered that I knew more about how his product was supposed to function than the guy I'd called. I tried to explain the issue to the guy in tech support. I am willing to admit that my "people skills" were not at a high point at 4 PM this afternoon. But for the love of all that's holy-I called about a bug in their software that had been present for two years and at least as many releases. And the guy with whom I spoke had no clue what I was talking about and just assumed that I'd installed it wrong. While he had me on hold, I opened up regedit and found the place I needed to make an edit. When he picked up the phone again I gave him a "K Thanx Bai" Since he hadn't really helped me.

It still isn't done yet, and people are getting a bit grouchy about a few things. I understand that and I've tried to make it clear that I'm working on all of it. I've started polling the users and asking them to keep lists.

It's still not done and it's still kinda painful. But I remind myself-it's more fun to spend time at the command line than it is to spend time in meetings.

1 comment:

akboognish said...

Awesome post!!! When I read it the first time, though, it only had the first two days (did it always have three and I missed the third maybe?). I'll go on record as saying that I liked it better as a two-day post: those two days are beautiful poetry. The third day should be its own post.

I liked this one so much I read it to S, who loved it too, and then I read her the Four Verbals post, which is also delightful.

I'm glad to see that things in Beverly are getting better...

Are you the last great blogger?