A portion of the product I was assigned had a nightly scheduled job execution to perform processing tasks to keep the production data system functioning.  Unfortunately, this was implemented quite enthusiastically in a complex series of processes that would of course fail in some fashion each night. 

This required a daily manual routine to fix the triggering data to then manually execute the whole convoluted process again manually.  Sadly, for the labyrinthine processes to have any chance at all of any success, it was critical to pay extreme attention to detail and also have a developer technical aptitude to perform these manual tasks.  Frankly the whole setup was insane, but I digress. 

One day, our development team was informed the customer was going to migrate production support to another contractor.  Normally, for a half-decent product, I would agree that separating production support from development would be a good plan. 

However, I warned, that in this case, it was a terrible idea and bad things could happen. I advised that only until, at a very minimum, we were first allowed to redesign and implement a sane nightly batch job, would this have any chance of success. 

Of course, I lost that battle as usual, and in the end our ‘Senior’ developer (who had wrote most of the mess) trained the incoming team on the ridiculous daily processes. 

Not long after, a significant portion of the production data was destroyed. 

This resulted in the necessity of assigning our best engineer on our team (not me) to spend a month unraveling the problem from backups. 

There was eventually a post-mortem meeting, where the question was posed on why the production team deviated from the outlined process. 

At which point, the hot-shot production team lead got very angry and exclaimed, “What do you expect, we’re just clicking stuff!” 

What a great line and slogan!  To this day, I continue to laugh and laugh from that exchange.