One of the first lessons I learned in my IT career was it's not about the mistakes you make, it's how you recover from them.
I was working as a new hire at IBM at the newly opened Mercedes Benz plant in Alabama. I was fresh off the graduation stage at the University of Alabama (Roll Tide!) and I was working the night shift along side a friend and fellow graduate as front line support for IT.
Being the proactive, ambitious, eager to prove ourselves people we were we had taken on a project to clean up the Oracle database behind our support ticket system to make it possible to report on the data and deliver our contractually obligated weekly update.
We had direct access to the production data (you can probably see where this is going) and were making line by line updates to ticket records.
Around 2:00 am, I went to get a cup of coffee from the vending machine across the floor from my desk. As I was turning around I saw my friend walking toward me, pale as a ghost eyes wide.
“Well, all of the records are updated.” he said. “They're just all the same value now.”
I paused and waited for the punchline. There wasn't one. Instead he said, “I left the ‘WHERE' clause off the update statement.”
With one fateful keystroke the entire ticket database now held the same data in every row.
We could have panicked. We could have gathered our belongings and headed for the hills. We could have admitted defeat.
But we didn't. Instead, I asked if he had the phone number for the server admin. He did and we called him. “Could you give us access to the latest backup of the ticketing database?”
“Check your email,” he replied and hung up the phone. We had the file location and all the help we were going to get.
I had just completed my Oracle Backup and Recovery Certification and this would be my first real world test. I triple checked the timestamp on the backup, it was fortunately just a few minutes before our shift had started. I took a deep breath and started the restoration.
Within five minutes the restore was complete. My friend hesitantly restarted the ticket application and waited for the result.
Success! All of the tickets were back to their state prior to the update. We breathed a sigh of relief.
But… the toughest part wasn't over.
Our shift ended at six am but we stuck around until our dba arrived at 7:30. We had to own up to the mistake and face the music.
We asked closed ourselves in a conference room and explained to him what had happened. As luck would have it our careers at IBM didn't end that day. Instead, he grilled us on the details. Did some intense investigation and verification of the database and then told us, “Never again!”
I learned a couple lessons that day:
1. When you play with fire you WILL get burned.
2. When you get knocked down, stand back up.
3. Don't Panic!
4. Learn from your mistakes. That's how you will grow.
What mistakes have you made that were major pivot points in your career?
