The Trouble with Data Mining

I don’t know what the right way to go about this would have been, but the way the professor chose was definitely one of the wrong ones.

The class was data mining. It’s my second to last class in the MSIT program and it’s the first one that has required any coding. The syllabus for the class said that programming experience was a prerequisite but it is not the first syllabus in our course of study to say this. To us it doesn’t really matter what the prerequisites are—we take the courses as they happen. It’s not as if, knowing the prerequisites for this course, any of us would eschew the class (which won’t come around for another 6 months to a year) so that we could take an intro to programming at a community college.

Having prerequisites outside the programs course of study is impractical—like having Wonderland Station on the Blue Line as a prerequisite to get off at Harvard Square on the Red Line*. Which is not to say that the professor was wrong to expect graduate students in an IT program would be able to write programs—just that he went about it the wrong way.

Our data mining professor is a tall, cheerful Indian man of about my age (perhaps he is younger). He is teaching many courses concurrently and so asked us to set the course title in the subject line of any e-mail we sent him. On the first night he asked a version of the standard MSIT professor’s questions of us all—What’s your name, where do you work, what did you get your undergrad degree in and what is your programming language experience?

A third of us (including me) have had no programming experience within the last decade (or none at all.) In spite of this, the professor insisted that programming was an important part of data mining and so we would all have to learn a programming language. On the spur of the moment he decided we should all learn Perl. He recommended getting the O’Reilly books.

Our classes are 6 weeks long—5 weeks really, since there is usually a Monday holiday during any given 6 week period. The idea of learning a programming language along with the course material (which was rather-math heavy) was daunting. At the second class the professor announced that he was giving us a homework assignment to work on over the next few weeks including a few Perl programs we would have to write. The third week of class was the week of Marathon day so there would be no class. I could fit all the Perl he'd taught us by the end of the second week on my thumb nail.

I spent most of the 2013 Marathon weekend trying to learn Perl. To do the homework we had to install Perl/Perl DBI and get them to work (or work on a command line Linux server in the cloud) and then write Perl scripts that would get a bunch of data from a CSV into a SQL database and then pull it out again and report statistics on it (max, min StDev etc.) The second of the two programs was supposed to draw a graph of the data.

This seemed impossible. And that was depressing. This was a graduate level course. I am a professional geek—shouldn’t I be able to rise to this challenge? Haven’t I faced more daunting tasks—situations where I had no idea how to get from A to B and somehow succeeded anyways? Both professionally and academically?  Maybe I wasn’t as smart as I thought I was. Didn’t I want to learn Perl and Perl DBI? Of course I did! So what was I complaining about?

Worse, my study-buddy sent me e-mail the Saturday after the professor gave us the assignment entitled “Don’t procrastinate on the homework.” Shit oh dear. He’d already spent 7 hours on the programming portion and wasn’t done. I had progressed from “Hello World!” to a few simple scripts that did math —but nothing that approached the complexity of the homework assignment. It if took this guy 7 hours I was toast. And if I was toast, what was the rest of the class—that dry piece of bread that caught fire when I stuck it in the microwave?

I admit I was enjoying working my way through the first few chapters of the O'Reilly Perl book, but I felt like I was playing around in the shallow end of the pool and I was not sure I was going to be able to swim in the deep end by next weekend.  There was only one problem in the problem set that involved Perl. I think in the end some of my colleagues gave up and just didn’t do that problem. I never considered that option because I figured this was not the last Perl problem we were going to be assigned. Also, there was my pride. 

It became apparent that most of my cohort had not started the homework on the first weekend—or at any rate hadn’t considered the Perl portion last weekend. E-mail about how hard this was flew back and forth from all members of the cohort. Finally the professor sent out a note reiterating “given that we are in an IT Program, and in a Data Mining course, it is reasonable to expect that we will implement Data Mining in code.”  And “Inevitably, some students will end up having a hard time, others will have an easy time, and some will find it to be just right.” I found this statement to be callous. I don't think he had any idea how hard we were finding it to make our way into the problem set he had assigned us. 

But the real show stopper was that he had cleared all of this with the dean. This meant that none of us could appeal to said dean about the difficulty of the homework. 

My problem with the data mining professor was that he decided we should all go off and learn a language on our own—and then do all the data mining coursework involved. I will admit that the Endicott program has spoon-fed the MSIT cohort to a certain extent, so it took me a while to decide if I was upset because this professor was asking something unreasonable or if I was upset because he was asking us to just work a little harder than normal. I decided he was asking us to do something unreasonable (given the 6 week course length.)

I e-mailed the dean and told him that in my opinion, if the MSIT students were expected to program it would be helpful if the course of study included a class in programming.

The dean responded by saying that the last time he had tried that the course had been a disaster. I mentioned this to a study buddy of mine who pointed out that it would be ridiculous to have a $17XX course teaching what anyone could pick up at a community college for $3XX—how much programming can you teach in 6 weeks anyways?  I admitted he had a point, but what is the right solution then—warn MSIT applicants that they will be required to code?

Meanwhile I miraculously found my way into the homework assignments. With a lot of luck (and a while loop that the professor gave me) I went from staring at Perl forums online and whimpering to writing two programs that compiled and ran. They didn't do everything they were supposed to do and they were not elegant, but they moved me from the side of people who couldn't figure the assignment out to the side of people who could. Suddenly I didn't hate the professor so much.

Meanwhile the professor had grasped that he was asking us to do something a little out of the ordinary. He  offered a tutorial before class—starting at 4:30. Most of us were there for it. He then spent most of the class teaching Perl—which was great except that we didn’t learn any data mining. 

In the parking lot after class some of us discussed the situation. We felt a little bad for the professor, but most of our sympathy was reserved for ourselves.  “You can’t just read Perl code to people after 9 PM and expect them to get anything out of it.” I opined.

“Remember what the dean said about the last time they included a programming course?”
 “It was a disaster!” we all said in chorus.  

Over the course of the next week it became clear that we had convinced our Data Mining professor that we really couldn’t write Perl code (or at least that he was not the man to teach it to us.) The professor sent out grades for the Perl assignment (graded on a very lenient curve) and a new assignment with no Perl in it. He also sent out a final project assignment—a research paper on data mining (initially it was supposed to involve getting a data set and doing some data mining).  We had broken his spirit. I am sorry anyone had to get his/her spirit broken but better him than us.

Boston Strong

I have had an interesting couple of weeks. On Monday 04/15/13 I was struggling with an awful problem set for my data mining class when I found out that there had been explosions in the Back Bay a few blocks from my office.

My initial thoughts were mostly annoyance--I need to finish this problem set before the weekend is over! I can't be distracted by drama--I have a box plot to build! I took a walk to clear my head and the implications of what had happened sunk in. I work a few blocks from the finish line of the Boston Marathon, so we usually have the day off. On the other hand, I work with a bunch of workaholics. Also, it was April 15th and I work at a financial services firm, so there were a few people in the building that day. I texted them and found that they had gotten out safely (I later found out that our doorman had to be hauled out of the building by the cops because he didn't want to leave until he knew everyone else had gotten out safely.)

I got most of my info that afternoon second hand from a colleague who owns a TV. But even at second hand with unreliable info from Boston.com "15 blocks around the incident are closed off" stood out. "Are we going to work tomorrow?" I texted my buddy? It seemed unlikely to me that we would and I felt relieved--for two reasons. One was that I did not want to go back to the neighborhood where I had worked for over a decade and see the evidence of explosions.

The other was that I had the worst problem set ever--not only did I have a bunch of statistics problems, but our professor expected us to teach ourselves Perl and Perl DBI. In the grand scheme of things, these are not at all equal thoughts, but at the time the daunting problem set--or rather my fear of being unable to do not just this but all the rest of the work for my current course--was just as big a deal in my head as the fact that a bomb went off near Sugary Heaven--which I walk by almost every week day.

On the day after the marathon I got calls and e-mails from former colleagues, vendors, former vendors (one of whom knew the family of the little boy who was killed) everyone wanted to know we were okay. Everyone wanted to offer whatever assistance they could. In spite of the circumstances I was touched.

 We all worked from home for the next week. This meant that I got to sleep in and take a walk to the beach before starting the work day. It also meant that I got to talk to my sister more often--she could call during the day when some of her children were at school. It gave me more time to obsess about my statistics/Perl problem set (although without making much headway.)  Meanwhile they found the "suspects"--turns out they lived on the same street as I did in Cambridge.

On Sunday I turned in the problem set--the programs I wrote didn't do everything the were supposed to do but I was so very grateful to have written code that did *something* that I didn't care. Monday the professor held a tutorial starting at 4:30 (class starts at 6). I went to the tutorial but was nervous--I am happy to spend more time working on this with the professor--but what if this doesn't help?

Actually, after working at home for a weird week it was nice to encounter a group of people and do something relatively normal (like gripe about the homework) for a while. The tutorial helped a little--but not as much as seeing everyone. Some of it was seeing that almost everyone was as frustrated as I was, but some of it was just seeing everyone--even my study buddy who found last weekend's assignment "fun." I realized that it was probably not the end of the world that I had failed to assimilate Perl on my own--yes it would have been nice, but it didn't mean that I wasn't going to get a degree in June.

On Wednesday they re-opened the Back Bay and we all went back to work. I was grumpy--it was rainy. In the North Station T station the guy who normally sells Bruins and Celtics T Shirts was selling Boston Strong T-shirts that said "never forget 4-15-13." I was repulsed--who sells commemorative bombing t-shirts?

But then I got off the T at Arlington and went into Pret for a croissant and an OJ. "Have you been open all along?" I asked the cashier "No we just opened for the first time today since the accident." "It's good to be back!" I said. The cashier agreed--yes you read that right--accident?

When I got to the office one of the owners came and hugged me as soon as he saw the light in my office go on. The office manager hugged me.  Our NJ office sent us flowers to welcome us back. We all went down to the memorial together at around 9:30. I'm glad we went--before the mobs and the TV stations showed up, but it was more emotional heaviness than I usually have before 10 AM. I was glad to get back to my desk to send out e-mails to various folks to tell them that we were back in the office.

I took a walk with a colleague after lunch to look at the rest of Bolyston Street--apparently so did everyone else in the 617 area code.  At one place where one of the bombs went off--by Marathon Sports and Sugary Heaven--there was concrete drying. There were a few bouquets of flowers and a Boston Marathon medal. There were several people taking pictures and looking (The concrete pouring guys smirked and posed) but everyone kept a respectful distance.

But that wasn't all--all of the restaurants and shops had signs that said "Boston Strong" or "Welcome Back--free coffee!" All of the servers at the outdoor cafes were wearing Sox shirts or Boston Strong shirts. Seeing the t shirts cheered me--maybe they weren't such a bad idea.

Today after work my colleagues and I all went out to Solas for dinner and drinks. (Solas is an Irish bar on Boylston Street.) When we all had our beverages I raised my glass and said "To the Boston Marathon!"And we all toasted.

The Demonologist

In between finishing IT Strategy and starting Data Mining I managed to sneak in a book. I picked up the Demonologist at Harvard Bookstore on Tuesday night and finished it on the train into Boston Friday morning. I highly recommend it. It is still in hardcover though.

The book is the story of David Ullman, a tenured professor of literature at Colombia who specializes in Paradise Lost. Ullman is lured to Venice with his daughter where he encounters a demon. The demon steals his daughter (in such a way that it appears to the rest of the world--including Ullman's soon to be ex-wife) that she drowned.

Ullman pursues the demon across America,  aided by his best friend Elaine O'Brien and using Paradise Lost as his guide. This isn't a coincidence--the demon makes it clear that his interest in Ullman is due to his professional expertise. Ullman finds himself  examining texts that he knows well--this time looking for literal meanings--not figurative ones.

This is plot-driven fiction. I do not recommend it for the beauty of the prose. I do not find the behavior of all of the characters entirely believable--the duct tape that holds the plot together is visible under the velvet drapery. For example Ullman's wife is leaving him, but makes no try to gain custody of their daughter. O'Brien--the female-best-friend-with-whom-the-protagonist-has-never-slept--has bone cancer and only a short while to live when she joins the crusade.

The journeys across country serve no purpose except as a chance for the characters to muse between demonic revelations. They seem to be part of the plot because that's what heros do--go on journeys. The whole story could have remained the bounds of New York City--unless the author chose to have his characters travel because he wished to make some point about the banality of evil--it's not all about heroin addicts in the big city. Maybe he felt that New York City was already too potentially demonic--too like Pandemonium--and that's why he made his characters travel.

In spite of these shortcomings I found it a good read--it made me want to re-read Paradise Lost. Perhaps that's its appeal--it is Paradise Lost lite. There is more to it that that though.  It is another attempt to imagine the motivations of fallen angels and, for a change, one shelved outside of the science fiction section. Don't get me wrong--I love sci fi--but it's nice to read something without the snark now and again.

I feel the author could have done more character development of the demons and demonic possession. The protagonist talks a lot about the Bible and Paradise Lost, but to me this book is a descendant of The Exorcist as much as it is of Paradise Lost.

Perhaps grad school is lowering my internal bar--suddenly anything with a better plot than the Mythical Man Month seems like a page turner.

Belated New Year's Resolution

"I don't like food anymore" Strong Sad 2003

A few years ago after my first full year of grad school I was at my sister's and brother in law's Christmas party apologizing to my sister for the fact that I couldn't bring myself to eat more than a few mouthfuls of the excellent spread her husband had cooked. I found myself saying that I found I just didn't like food anymore. "What are you eating?" she asked. I explained that I mostly subsisted on a diet of canned soup and Odwalla bars "Bleck--there's your problem--you're the vegetarian who doesn't like vegetables."

I'm a picky eater. I'm a vegetarian (although since moving to MA I've included fish in my diet) and even though I tended to eat frozen food or canned soup I like to think it was relatively good frozen food and canned soup--there were no ingredients I needed Wikipedia to help me identify and it's all organic--right? To which my sister has always responded that it's still not real food, made from real ingredients.

Cooking meals from real food may be cheaper than nuking an Amy's dinner, but it costs more time--something that is at a bit of a premium for me at the moment. However, my sister has 4 children. While I can certainly argue that she, feeder of six humans, gets a better return on investment for cooking real food, I can hardly argue that it is easier for her to cook real food than it is for me to do so (no one who has spent an hour in the company of her "piggies" will ever argue that it is easier for her to do anything than it is for me to do so.)

Then there was the weekend of Snowpocolypse. I had a paper to write so I went to the Whole Foods by North Station and bought $80 worth of frozen/canned vegetarian food (along with good coffee and iced cookies.) I was not going to have to worry about cooking at all that weekend. Except, by that Sunday I never wanted to eat another frozen whatsit again.

I decided that I was done eating things that came out of cans--this includes beans and tomatoes.

This year my goal is to stop eating canned things (until such time as I take up canning myself) and to eat no more frozen, pre-prepared food.

We'll see how this goes

Graduate School

I have 14 weeks until I'm done with grad school and they give me a degree! I am so relieved/ impressed with myself that I considered attending commencement. I feel like completing a graduate degree is an accomplishment where as--in my case anyways--completing an undergraduate degree was par for the course*. I got an eviction notice written in Latin and I would have skipped commencement but my parents insisted I attend-"We paid for this--you will walk" said my mom.

I would not trade my undergraduate education for anything (although I might have, in hind sight, done a few things differently) I met some awesome people--including my best friend--at Carleton and studied (or at least pretended to study) a whole bunch of interesting stuff. I was disconsolate when they kicked me out.

My graduate education is another matter. Grad school is fun! You get to learn about all sorts of interesting things--like DNS poisoning or how DHCP really works. You get to meet people who are interested in the same weird things that interest you. Why would you want to spend your Saturday nights drinking beer and watching movies about Moon Nazis with your friends when you can spend them talking to your study buddies about the paper you have to write?

I can't wait to be done with graduate school. But I feel good about having done it. It was't any fun spending July 4th last year trying to understand two's complement or reading the RFC that describes SMTP. These things were hard in a way that getting an undergraduate degree in French literature was not.  In grad school almost every final paper I turned in scared me--I think I understood the concepts but what if I guessed wrong? Aside from working full time the whole thing was hard in a way that college wasn't. Some of this may be because my brain is older. Some of this may be because I'm working full time. I suspect some of it is because I'm making myself learn things that don't come easily to me.

Not all of it is an uphill battle or I'd have given up.  And some parts of the whole thing fell into place as easily and obviously as some of my undergraduate analyses of good literature. This is not to be discounted or overlooked--the joy of discovering that something you find fascinating comes naturally to you is wonderful.

All of it (or almost all of it) falls into the category of "Stuff I want to know" but without grad school to goad me I wouldn't have bothered to *really* learn the history of Flame or consider the difficulties of storing and manipulating very large numbers that must be very,very accurate.

Getting a degree will mean that someone thinks I've understood even the difficult concepts well enough to be validated.





*I recognize that this is because I'm from a middle class background and my parents could afford to send me to college.

Entomology

This is about work--but it's mostly about me so I think it's okay to post about this. I feel like I've gone through a bit of a professional growth spurt lately. I put this mostly down to the fact that I'm almost a year into my MSIT program--in three more courses they give me a degree.

I do things I wouldn't have had the confidence to do a year ago (although some of these are things I was perfectly capable of doing). I find I don't always have to check with the outsourced IT firm before answering questions about why this e-mail message wasn't delivered or why So-and-So got a funny message the first time they opened Worldox today. This is a good change for the most part (assuming I don't get cocky and break something) but "in literature as in life" change can be scary--even good change.

I started noticing this last week when I was sending an e-mail out to the entire staff about changes I'd made to our document management system. "I have made a few changes to Worldox's file-attachment functionality" was the first sentence. I was a little appalled that I had used the word "functionality" in a written sentence. I've always hated this word and I refused to use it for a long time* but I needed to send this e-mail notification out yesterday so...functionality it was.  Luckily I usually ask a colleague to proofread memos before sending them out to the whole staff. The colleague in question rolled his eyes at functionality  "Yeah I know--I hate the word but.."

"Use 'feature'," he suggested. I did. My inner-French Major/Dickens Reader sighed with relief.

Today had to log in as a user for whom the CRM kept crashing and see if I could reproduce the crash. I did. I was alarmed at how satisfied I was when I succeeded in reproducing the crash and at how I could tell that this was the same error code the user had gotten. I've become a connoisseur of Microsoft Dynamics bugs.

I am not complaining about being deeply satisfied for something that someone paid me to do--it's just that I'm not yet comfortable with my new self--I never actually expected to use the phrase "escape sequence" in a sentence (at least not outside of a classroom.) Some of my apprehension is that I worry that some of this is just too far out of the acceptable range of dorkiness--that my new knowledge has pushed me farther along the NGLA spectrum. I know my Beverly pack mates (and most of the other humans I know) are not going to want to hear about Dynamics bugs or e-mail spoofing** over Saturday brunch.  The things I spend time working on have passed out of the range of "stuff I really think everyone aught to know about computers/be able to do for themselves" to the kind of things that don't work well on t-shirts. That's okay. I have my MSIT study buddies and a few nerd friends who tell SQL jokes.

Whenever I started a new romantic relationship, or even fell in love with a band I'd have a period of uncertainty at the beginning (no matter how nuts I had been about the person/band) where I asked myself--"Is this really what you want?" I figure that's what I'm currently feeling about my new, nerdier, self.


*I remember discussing our mutual hatred for this word with a colleague of mine at the tech startup where I worked in 2000. We both agreed that it was a terrible letter-salad and we were discussing whether it was grammatically correct. At the time I opined that it would fade out of usage soon. Hah!

**I was disappointed this wasn't more of a winner--I remember being awed the first time someone showed me how to do this.

Ten Years Ago This Weekend..

On Friday I was shocked when I saw the 2003 protest against the Iraq war listed on Wikipedia's "On This Day" area of the page. Holy sh!t! It's been ten years since then. Time flys-whether you are having fun or not.

I don't remember who's idea it was to go down to the protest in New York City that weekend--if I had to guess I would have said it was Dan's idea. We had all gone to several protests in Boston. I even went to one march by myself on a Sunday. (Most of the people I encountered at that march were sanctimonious prigs.)

We must have rented a car because I know we didn't take Adam's ailing BMW 2002 down to New York. Adam and his girlfriend Stephanie and Sean and I drove down to my parents' house on Staten Island on Friday night 2/14/03. While on the BQE with Sean driving Adam ( a native of the Bay State) yelled "Don't signal! that will let them know what you're doing!"*

My dad was in Idaho visiting his mother because she was ill, but my mom was home. The front door was supposed to be unlocked, but  it wasn't. I called from my cell phone and woke Mom up. She came down to let us in to the house in a robe and then went back upstairs to have a sleep emergency. It reminded me of how Mom's mother--my grandma--would always come downstairs in her nightgown whenever we arrived at her house in Beverly.

Mom had prepared a vegetarian chili for us and left it (in a microwave safe container) in the refrigerator along with some beer. I was happy to have my friends down at my parents' place to show it to them and  shown them to my mom (see Mom--my friends aren't imaginary!)

 I had been a little worried what my parents would think of the idea of the protest. They had lived in the village in the 1960s and (as far as I could tell) still seemed to have missed most of the cool parts of it. They didn't seem to think highly of the anti-war movement (although they were anti-Vietnam war). Dad had gone to one anti war meeting at NYU and had come away unimpressed--"They kept talking about The Movement. It sounded like they all had trouble with their bowels."

So I was pleasantly surprised to find that that not only did they approve of the protest, Mom would be going with some people from her church. Perhaps we could all meet up at the rally? The city of New York had vetoed the idea of a march in front of the UN. Instead there was a park where we could hold a stationary rally. We were going to meet our friend Dan at the rally as well. He'd also come down from Boston but he'd stayed with his brother Friday night.

On Saturday 2/15/3 Mom went in to town earlier than we did. It was Cold that day. Luckily Adam and Sean worked at Hilton's Tent City. They had brought hand warmers for us. We all had ENEMY t-shirts, but you couldn't have seen  them without an X ray. I made the error of wearing just my t-shirt beneath my wind-stopper fleece and hard shell. This was not nearly enough coat and I spent the entire afternoon realizing this.

We took the ferry and the subway up to near the rally site. So did everyone else in the NY, NJ CT area. By the time we got off the train we were in a huge crush--everyone was headed towards the rally point, but there were so many of us we became a march (a very slow one) whether the City wanted it or not. We tried to stay on the sidewalks but there were just too many people. We never did find Mom. Somehow we found Dan. None of us had smartphones with Google Maps (because this was 2003!) and this was a part of New York City I didn't know well, but it didn't really matter because there were so many people that no one was moving quickly, and we were all going the same way whether we liked it or not. We passed a deli where Dan had once worked. I think we passed the Roosevelt Island Tram, but I don't remember for sure. The police were around but everyone was behaving well** People were drumming and chanting "This is what democracy looks like!" They got fire trucks to go down the avenue blocks with their lights and sirens on. People yelled at the firetrucks to stop doing the cops bidding.

We never made it to the "rally" but that didn't matter. The rally was too big to fit in the park and spilled out all over Manhattan.

Eventually we got cold and hungry and went to a bar to eat dinner. We were supposed to meet up with Dan's brother and my sister (because they both lived in New York City.)  I think we got Dan's brother but not my sister (she was working at the Strand that night.) We hit multiple bars. One of them had a juke box and Adam and Stephanie danced. I watched them feeling happy and sad. Sean didn't like to dance and so I'd occasionally danced with Adam, but now Adam had a girlfriend and so I'd lost my dance partner.

I was thrilled to be out so late because we were in a city with all night public transit. Somehow we all--Adam, Stephanie Sean Dan and I made it back to my parents' place on Staten Island and we all went to bed. Adam and Stephanie got the attic, I got my childhood bed, Sean got the bed in my sister's room (we referred to it as "The Slab" because it was like a marble mattress) and Dan got the couch in my parents living room.

The next day Adam and Stephanie went to Brooklyn to see friends of hers. Sean, Dan and I met them at the Brooklyn museum later and then we all went for dinner before driving back to Cambridge. At dinner my Mom called and told us to come back to SI. There was a snow storm coming and she didn't want us driving in it. We decided to go home anyways--we had to work the next day.

I'm sure there are plenty of parts I've forgotten and some I've mis-remembered. The part I remember the best is how very cold I was. I don't think I've been as cold since. It was still a good weekend and I can't believe it was 10 years ago.

Reference
http://en.wikipedia.org/wiki/February_15,_2003_anti-war_protest

*I have since heard other Bay State natives say this. I have yet to figure out why you wouldn't want your fellow drivers know that you were changing lanes.

** As far as we could tell at the time the protest was entirely peaceful. When I went to go buy bagels the next morning I noted that the New York Post had a picture of a cop on a horse with a protester in what *might* have been construed as a violent confrontation. The headline was "FACE OFF." The Post has never been noted for it's liberalism.