Wednesday, August 31, 2005

It's All About New Orleans -- Part II

Note: This is part 2 of a 2-part posting... but for those who want the latest news and don't need a long-winded historical backstory, you can just start here. Part 2 is long-winded enough by itself.

This past Friday, I was hearing reports that Hurricane Katrina was headed for the New Orleans area, and gathering strength as it moved across the Gulf of Mexico. The last serious hurricane that threatened New Orleans was almost exactly a year ago, but the city had been spared the worst of that storm, and my Decatur Street property had fared quite well without any special precautions taken.

Naturally we hoped for similar luck this time, especially since we were not really prepared with plywood for the windows, etc. Since I had hoped to sell the property soon, and I've been so busy with other things, preparing for possible storms had not been a high priority.

Pop, always the frugal one, was living in the upper apartments without a phone. Since I travel with a cell phone and a laptop with wireless internet access, I hadn't bothered to have a phone installed at the property.

During the day on Saturday, the storm was gathering strength, and New Orleans residents were being encouraged to evacuate. I spoke to other friends and family in the New Orleans on Saturday. Most (not all) of them were in the process of getting out. I hoped Pop would call, since I had no way of calling him.

On Saturday evening, I was visiting with some friends when a call came in on my cell phone from a New Orleans number. I tried to answer it, but not quickly enough, and it went to my voicemail. It was Pop calling from a payphone, and he left a message.

Pop said that he had hoped to get out of New Orleans by bus. But when he arrived at the bus station, he found it already closed. No more buses. Of course there were no more trains or planes either. So, always the courageous and eccentric fellow, Pop said that he was thinking of weathering the storm in New Orleans, and he would use the remaining time to do what he could to prepare the property for the upcoming storm.

I had been hearing the reports that the storm was becoming more menacing, and I thought it would be better for him to leave. I started calling friends and family. Most of them had already left town, but my 3rd cousin Nancy was just about to leave with her family. She said she'd be happy to go and get Pop and take him out of town with her family. I explained to her that there was no phone and no doorbell at the property, but that there was a key safe attached to the front door. I gave her the combination to the safe so she could open it, get the keys, and then go inside the property to find Pop.

A while later I got a call from Nancy on her cell phone. They had arrived at the Decatur Street address, found the key safe, opened it, and found it empty. I guess on my last trip to New Orleans, I had kept the keys, and Pop had taken his set inside with him. They shouted, and banged on the door, but there was no answer. It was 1am at that point, and Pop was inside asleep. He was on one of the upper floors, and he's a little hard of hearing. Besides that, our block of Decatur Street was home to quite a number of bars, which tended to stay open late and produce a lot of noise, so whatever noises did reach Pop as he slept were likely to blend in with the rest of the general cacophony on the block.

I considered telling Nancy and her husband Stephen to just kick the door in, but I didn't. At that point, breaking the door still seemed like a slightly drastic measure. I figured there would still be options the following day, including sending Pop to stay with other friends who were planning to stay in town, but who had cars to escape in at the last minute if things started looking worse. And of course I also had some concern for the property. I didn't really want Pop to leave the building with the main entrance ajar when there were still plenty of rowdy revelers around.

On Sunday, I awoke to news that Katrina had been upgraded to a Category 5 hurricane, which would rank it among the half dozen strongest hurricanes ever to make landfall in the USA in recorded history. And it was on course for a direct hit on New Orleans. This would definitely be the worst storm to ever hit New Orleans. It was looking a lot like the doomsday scenario that I'd been thinking and talking about for some time now.

I suddenly began to feel quite urgent about getting Pop out, and wondering how to get in touch with him.

I sent a quick email to my web-host friend Mike at Icorp, urging him (not for the first time) to get all his web-hosting and e-commerce systems backed up on a new server that was outside the area. Although he had systems in multiple locations, I knew they were all in the New Orleans area. My website ( http://www.stockroom.com ) was still up and running on two different servers, one in Metairie (a suburb of New Orleans), and one in the CBD (Central Business District) in Downtown New Orleans. My plea to Mike:
"I really think you/we need to consider the possibility of catastrophic
failure of *all* your internet connectivity and/or all servers. I suggest, if
it isn't already happening, a complete off-site backup of website data and the
VOE. We can give you FTP access to one of our servers and diskspace for the
purpose."
That advice was dismissed, for the time being, as it always had been in the past. Mike has always been a die-hard, one of those who stays in New Orleans when the majority of the residents are fleeing the city.

I turned my mind back to the question of getting Pop out, and how to get in touch with him. I called more friends, now looking for pretty much anyone who could go and retrieve my father from the French Quarter. My friend Bart had evacuated, but his parents were ready to leave within a half hour or so. But they were out in the suburbs, and headed away from the French Quarter. They were feeling time pressure, and it wouldn't make sense for them to backtrack for someone they didn't know. Pop didn't have a car, and there were of course no cabs. There were 3 bicycles that I had bought for the Decatur Street property; Could he bike out to the suburbs in time?Probably not.

Finally Pop called from a payphone again, and we had a point of contact where I could call him back at appointed times while we worked on what to do.

Pop said he had gone out and tried to hitchhike. He made a sign that said "Evacuate Me" and wore it. 100 cars went by, and nobody offered Pop a ride. Their loss, of course, but potentially ours too.

Time was passing. I spoke with Bart again, and he calculated that there wasn't enough time for my father to meet up with Bart's parents, unless my father could sustain a speed of 30 miles per hour on his bicycle. No chance of that.

I talked with Pop. We discussed the possibility of taking shelter in the Superdome, where the city government was encouraging the stragglers to go. But somehow that didn't sound too appetizing.

"If I were you," I said, "I'd hit the road on one of those bicycles."

Pop said he had thought the same thing, but wasn't sure about it. He knows he's a little eccentric sometimes, and he likes to get some validation for some of his more dubious schemes before risking his life on them. So he'd talked to a couple of other people about their evacuation plans, and when he mentioned the bike idea, they thought he was crazy. But I told him at that point, riding the bike is exactly what I would do, and in fact I had that very possibility in mind on the day I bought those 3 bicycles 18 months ago. I remember thinking that if the city was being evacuated and the roads were jammed, a bike just might be able to travel faster than a car.

The roads certainly were jammed now. Pop was heartened to hear that someone else thought the bicycle might be the best option, and not an entirely insane way to flee the storm that was approaching the city at about 15 miles per hour. Some of our shared family traits -- streaks of eccentric rationality and stoical pragmatism -- were showing themselves.

But even if Pop did flee on a bicycle, where should he go, and by what route?

I made another call to Bart, who is a bit of a cyclist himself. I was lucky to get through again, since the phone networks were becoming increasingly unreliable. Bart suggested that Pop head for Baton Rouge. He told me there was a bike trail that Pop could follow that ran along the top Mississippi River levee for about 30 miles. Then he could cross the river and take River Road all the way to Baton Rouge.

I told Pop about the proposed route and destination, which would be more than 80 miles. That is a long distance for anyone to travel on a bicycle, but Pop had gotten into pretty good shape with his recent hiking training and Appalachian Trail walk. We figured if he could easily walk 10 miles in a day, he might be able to bike 80 miles. In any case, even if he ended up 40 miles away hiding under a tree, he would be on higher ground, and inland, away from the most dangerous threat, which was the possibility of catastrophic flooding in New Orleans. And there was the hope of catching a ride along the way, and/or finding shelter somewhere. Pop was ready to hit the road.

If Pop did make it to Baton Rouge, where would he stay? There certainly wouldn't be any hotel rooms that weren't already booked. At that point, it didn't much matter. Whatever could be arranged on the fly would surely be preferable to being trapped under a wall of water in below-sea-level New Orleans. We agreed he would hit the road while I continued working on options in Baton Rouge. I told Pop to stop and try to reach me on pay phones whenever he could. I would go to the office so he could call me on my toll-free 800 number and he wouldn't have to worry about carrying change.

My 3rd cousin Nancy had been headed for Baton Rouge, and I thought if I could get her on her cell phone, they would take him in wherever they were. But her cell number wasn't working. Cell phones were becoming increasingly unreliable with each passing hour, as circuits got jammed.

I spoke with another friend, Louis, who had evacuated the previous day. Baton Rouge? Well, there definitely wouldn't be any vacancies there, but it just so happened that Louis (who lives in New Orleans) has been working on an assignment in Baton Rouge, and he had rented a room at the Extended Stays Hotel there. He and his family had evacuated further inland, and his room was empty, so Pop was welcome to use that. Louis said there was even some food in the refrigerator, enough to make some sandwiches, and Pop was welcome to that as well.

This was a ridiculous stroke of luck, in a situation where there wouldn't be any rooms at any price for most of the refugees.

Pop called me after about ten miles. The weather had been deceptively nice when he started out, but within the first ten miles he'd been rained on pretty hard already, and he had also been hit by some frightening gusts of wind, strong enough at one point to cause some concern about being blown into the Mississippi river. He took shelter on someone's porch until the squall died down, and then got back on the path, which he said he had all to himself for miles and miles while the highways were completely jammed.

I told Pop about the hotel room, and he was encouraged and amused at our good luck. He said he would keep moving.

I talked to Mike at Icorp, trying again to impart my sense of urgency about backing up our web services. Although I knew I could get my website's content relocated to a new server quickly, I would need Mike's help to keep the online ordering system working.

Mike's shopping-cart system, which was such a leading-edge product in 1996, is now a bit antiquated, but still very fast and solid under normal conditions. Mike is the only person who really knows how it works, which makes us dependent on him. We had already identified that as a liability, and have been working on porting our website to a new system so we can be more leading edge again, and less dependent on one person (as well as the New Orleans weather). That project has been going for months, and is expected to be completed within weeks. But it's not ready yet, and so we are still dependent on Mike.

Mike was in his calm-down-the-client-who-is-freaking-out mode, even though I didn't feel freaked out at all. (I felt calm, and was trying to make what I thought were logical and rational arguments in our mutual best strategic interests.) Mike told me again about his multiple servers, his backup generators, and his stores of food and drink. He said he thought things would be fine, pointing out that his downtown servers were on an upper floor that couldn't be flooded. I told him none of that would do any good if the city was inundated with water. I said that if that happened, there was no chance that his internet connections would keep working, even if his servers did stay on and powered up. He was unmoved. He just didn't see things getting that bad.

I looked at my website, and noticed that it was slowing down. I could guess one reason: One of Mike's other clients had a webcam attached to one of his servers that served up an image of Bourbon Street, updated every 5 minutes. Every year during Mardi Gras, or anytime a bad storm hit the city, traffic to that cam would always spike up, causing slowdowns to one of the two servers that was serving up my webpages.

I called my in-house webmaster, Steve, to come into the office for a Sunday crisis session. Steve was quite helpful. He had a friend in Germany, a fellow photographer, who kindly offered us the use of his webserver for our content. We got busy uploading our files and getting ready to move the site. I kept talking to Mike, lobbying for him to back up his shopping cart system on one of our servers, or somewhere else outside the threatened zone. He finally seemed to warm up to the idea just the tiniest bit, but there were technical barriers, not the least of which was that his shopping cart software was built on a relatively obscure variant of the Unix operating system, and all the servers I had access to were running the more popular Linux. Also, although my grasp of technical issues is better than average for the owner of a business like mine, I am not a real IT person, and I didn't have one on site to help me sort all this out.

In the midst of all this, another call from Pop. Someone with a truck had picked him up, and he had a ride the rest of the way to Baton Rouge. Excellent news.

Mike seemed to disappear for a while, and his cell phone wasn't working. Later in the day, I got in touch with him again, and learned that he had monitored the news, seen the light, and fled New Orleans after all. I was glad to hear that he had evacuated, since I thought it was dangerous to stay in the city under the circumstances. But even then he was still trying to calm me down with respect to the prospect of catastrophic outages.

I was able to get Mike the passwords to a new Linux server, where he could poke around and do some testing to see if he could make his software work there.

We got our site content moved over, and prepared to move traffic over to the new server, even as Mike assured me that such drastic measures would probably not be necessary. In testing the site, the complications became apparent. I knew my online ordering functions were at risk, but there were other services and customizations that weren't going to work right on the new server without a fair amount of work. Search functions, chat, our gallery, and most vitally, e-mail. Mike was also handling our Domain Name Services, and I knew little about how to reconstruct those if all his servers went down. Without properly-functioning DNS, nobody can find a website, even if it's up and running on a server somewhere.

I'd given Mike the passwords to a new server, and he had said he'd have a look at it. It was getting late, and I was tired after a long day in crisis mode. I went to sleep on my couch for a few hours, hoping for the best. The hurricane was supposed to get really intense shortly after midnight.

I woke up a little before 4:30am. I jumped up, went to a computer, and checked the website and email situation.

The site was sluggish, and seemed to be going up and down. I saw a collection of "Device Down" and "Device Up" alerts in my inbox, showing me that the site had been going up and down starting about 1:55am, when the storm started hitting the city pretty hard. The last "Device Down" message was at 4:26AM on Monday.

I sent Mike a message at 4:27AM:

Hey.

I'm getting periodic Device Down messages from the server.

It looks like the server that's handling cam traffic may be hosed pretty bad,
and is essentially useless for serving our pages.

I request that you route ALL stockroom traffic to the server that is NOT
handling any cam traffic.

Joel

What I didn't know at that moment, though of course I feared it, was that our New Orleans servers had just gone down for the last time just one minute before.

Shortly thereafter, I lost the use of my stockroom.com mailserver because of the DNS servers going offline. People who tried to email me or any other address in my company would now just get a bounce message.

Mike was now at his girlfriend's family's country home north of Baton Rouge. It took him a while to get internet access, and cellphone services were spotty. Later in the morning he was able to get online, and was working on moving his shopping cart services to one of our Linux servers, when the power went out and he lost his internet access.

We had a discussion about how maybe he could talk me through the rest of the process over the phone. But then, of course, cell phone services quit working too, so I had lost all contact with Mike, and any hope of getting our online ordering services working any time soon.

I found a new DNS server, and gave myself a crash course in DNS configurations, which involved some trial and error. But I was able to remap web traffic to our new server in Germany, so graciously provided by Steve's friend Martin. At one point our site's traffic apparently overwhelmed the server, and it crashed for a while, but Martin got it up and running again.

Mike resurfaced on Tuesday afternoon. He had driven down to into Baton Rouge, and after a difficult and frustrating search for internet access in that city (also ravaged by the storm), he ended up at a crowded and noisy coffee shop, using his laptop with their wireless network.

Mike ran into more snags than expected with Linux, while I placed an order for a new server with one of my vendors that would run his preferred version of Unix. Finally, Mike, exhausted, gave up. So our online ordering services are still down, but we hope to have them back tomorrow, after about 3 days offline, to the tune of about $10,000 in sales per day. (And, mind you, that $10K is not our company profit; most of it is just the daily cash flow that is needed to pay the mortgage, keep the lights on, and make payroll for 35 people!)

We have put up notices on our website explaining the situation, and even offering people a small discount if they will call us and place their order by phone. Since our website does its heaviest business at night, I thought it would be a good idea for me to stay around tonight and cover the phones, and that's what I'm doing right now, at 12:14 am. But nobody is calling. I have taken one call since 10pm, and it was a guy who was only pretending to be interested in product because he wanted to talk to another man about leather sex toys. He went all through the order, asking about the prices, the sizing, the discount, the quality of the leather codpiece he was interested in, whether I had ever worn one myself, etc. He said he wanted to go ahead and place the order, and at the exact moment that I asked him his name, he hung up. This is exactly why we are not normally open at night-- only lonely weirdos call at this hour. I thought tonight might be an exception, but I guess not.

I notice that this blog is all about me in one way or another, and I guess that's not so out of the ordinary-- It's my blog after all, and nobody is making you read it, whoever you may be. I have been told from time to time that I have an interesting life and I should write about it more and share that with the world. So here you go.

But I'm well aware that other people have bigger problems today. Even Mike, who has been a source of frustration for me by not listening to my annoying (and now obviously correct) advice, is faced with some very difficult realities, since his house is almost certainly underwater and the future of his business is in doubt. Many of my friends have lost their homes and businesses. And, as most everyone knows quite well, many have died today. I haven't forgotten that.

And yet, here I am, with a business to see to, and it is in crisis now. And I suppose I am too. My bread-and-butter website isn't working, I am contemplating a possible total loss of my property in New Orleans, and the clock is ticking on an impressive portfolio of mortgage debt. Even before this latest glitch, we were facing some difficult cash-flow realities, because of intensified competitive pressure in our market, and this huge, interesting, expensive building here in L.A. that we recently bought (planning for growth that hasn't yet materialized). And then there are the expensive ongoing renovations on the building, which are threatening to break the quarter-million-dollar mark. To save all of those 35 jobs and grow into this big building that we bought, we will have to pull a few more rabbits out of that same rather well-worn hat that has produced more than its share of rabbits already in the last 17 years. And I'm getting tired.

Meanwhile, on the periphery of all the other things I've been working on, I watched the news from New Orleans. The levees are broken, and the city is filling with water, just as the worst-case scenarios predicted. The French Quarter isn't quite so badly flooded as many neighborhoods, but whatever's left of my 170-year-old building after the surely very serious damage from the intense wind and rains will be standing in water, and probable prey to the looters who are now ravaging the area. Also, since there are refugees in the city now, and the French Quarter is some of the higher ground inside the New Orleans bowl, I would not be at all surprised to find squatters staying there at present. But I won't know for quite some time what's left of the building, because at this point the city government is saying it will be at least a month before anyone is allowed back in, if there's anything left of the flooded city at that point.

It seems as if sometimes the universe decides to erase things... like when I was a kid, playing with an Etch-a-Sketch toy. You could spend hours or days working on an intricate drawing of some kind, and then at some point get tired of it, and decide to start over with a clean slate. You shake it up, and it goes blank again.

The universal version is that this can happen to a person, a city, a civilization, a species, an entire planet, or a star system. It is How Things Work. Powers greater than ourselves. New Orleans has been a fascinating but famously corrupt city for hundreds of years. It maintained its existence on borrowed time, while doing a typical New-Orleans-style half-ass job of staving off its own doom. Remember the line from that 70s commercial? "It's not nice to fool with Mother Nature." In hindsight, New Orleans could have done a MUCH better job of protecting itself, but it would have cost billions of dollars that nobody could find, staggering sums of money that are, ironically, dwarfed by the much-much-larger calculations of the damage that has now been done.

New Orleans was beautiful, byzantine, stratified, aged, haunted, and crooked. And I experienced (and ostensibly "owned") part of it for a while. But does anyone really "own" land or buildings, or do we just imagine that we do? Seriously, friends... it's just a nearly-but-not-quite-unanimously-shared delusion. There are precious few pieces of ground on this earth that haven't been forcibly appropriated from prior human inhabitants at the point of a spear, a sword, or a gun, prior to coming into the hands of the current "owners." Where is the legitimacy, ultimately?

It's hard to predict the future of New Orleans at this point. It might have been nice for me if I'd taken one of those lower offers I got, and sold my Decatur Street property before this all happened. But to wish that were so is merely to wish my troubles onto someone else. Just as well me as anyone. Maybe it's better that this should happen to me, since I have a bit of detachment from these things, and other individuals stand to be more hurt or upset by their attachment to land and buildings and things. (I could've sold the property to a non-individual-- Wal-Mart, McDonald's, or Halliburton. But I don't remember any of those entities making an offer, and I'm not sure how I would've responded if they had.)

As for the difficult times, and the threat of losing most of what I've spent the last 17 years trying to build, well, I have been here before, numerous times. It just keeps happening, at an ever larger scale as it continues its geometric growth. The scale becomes more daunting, the trapeze seeming ever higher. But there is still a net. I have my health, my intelligence, my network of friends and family. And even if I lose some or all of those things, I have what I have come to know spiritually: That there is no end, no death, no separation of anyone or anything or any place or any time from any other person, thing, place or time. So worry and fear are illusions-- and not pretty, Glenda-the-Good-Witch-descending-in-a-bubble illusions either, but useless and limiting ones that will swallow as much of my mind and my life as I surrender to them. If I let my fear rule the day all the time, or even more than half the time, I wouldn't have this interesting life and portfolio of assets and responsibilities and concerns and challenges to work with.

I am feeling that big L.A. earthquake coming any day now. This super-cool old building was built in 1928. It's 3 stories, 30,000 square feet, made of unreinforced masonry. It has been retrofitted as required by law, but it isn't the kind of retrofit stands much chance of saving the building; It's just the kind that gives whoever's inside the building extra time to evacuate it before it falls down, or is condemned and demolished. That scenario would probably force an exit from this business for me. I will flee the country with whatever cash I can scrape together and move to Bombay.

Or, maybe my psychic premonitions are off by a few years or decades, and there will be time yet to get this machine properly tuned again (as it has been in the past at smaller scales), and make it churn out another good run of productivity, relative harmony, and profit. For a while.

We'll see.

Thanks for listening.

Comments: Post a Comment

<< Home

This page is powered by Blogger. Isn't yours?