Lessons learned from tough migrations
I wrote this in a very direct way to remind myself of what is right, but you can enjoy it too.
Is the product critical?
Many new engineers will see a product as a legacy, even after 1 year. We engineers want to refactor and migrate things. The previous engineers were bad, but we are good. Business wants more features. Customers love features they don't want.
You will know how well you have balanced your product development after 5-7 years. The product will answer the question of whether it "fits the market" or not.
In the latter case, there will be five options that you can use when your product does not fit the market:
The product isn’t critical. Terminate the whole product.
We think it’s critical. If you lack the required skills, hire specialized people to perform surgical cuts.
Product is critical. Pivot. Build a new product or approach from scratch, but it requires upfront investment.
Stay where you are and carry on as usual.
Try to sell.
Keep your operating costs low
When you ask 1,000 managers to cut costs, most will decide to turn off products used by customers or lay off people. The rest will try to find approaches to optimize without making irreversible decisions.
We associate cost-cutting with how many dollars we can reduce our expenses. More often it's about how to leverage the brains of your people and make sure they're doing the right things.
1. How often do you deliver to production? 3 weeks? Lots of manual work?
Automate process up to production after merge. Change your company policy to allow it. Turn your engineers into operators of their products. It will create a lot of pain but long-term, you won't have to get up at 3 a.m. as often.
After the change product started delivering 30 times more to production. Real demand was 45+ times more and there was room for improvement. The downtimes, which cost $1,300 a minute, weren't caused by the juniors but skewed process.
2. How often do you update and check your dependencies? How many do you have?
Any dependency can cause problems, but some are necessary. The most dangerous are external or critical dependencies.
By choosing GCP you have a good chance that the next day Google will abandon the product you choose. I love AWS for its customer obsession. You can sleep even better with cloud-agnostic approaches in mind.
You built the product on AngularJS and .NET Framework. Google abandoned AngularJS. Microsoft asked you to update the .NET Framework from 4.6.1 to 4.6.2 and gave you 5+ years. Define monthly tasks for engineers to update dependencies. If any of the dependencies need 3+ weeks of work in total per year it's time to consider saying goodbye.
Your product depends on internal products providing APIs. Make sure you have agreements on paper or email. Someone may decide to change priorities or ask who is funding it.
The company built products only on Java, but engineers decided to use Python, C#, Go, and a brand new Blockchain Database! Effective organizations are boring and use one set of tools. You can add something new when it’s critical to support the organization or growth.
Did you ship the Proof Of Concept product to production?
If you kill your product in the next 6 months because its purpose is to gather knowledge, that's okay. Ship fast with low quality. Test the market and technology, and when you get traction, bin the product and start from scratch the right way.
Signs that you are doing it the wrong way:
Brand new technologies that you have never used in your organization.
A product built by juniors with no senior/staff who knows how to do it right
It should have taken 6 months but it took 20 months.
Built by a brilliant person who refuses to work as a team
Not enough investment in Developer Experience
Developer Experience (DevEx) defines techniques to measure productivity. You don't need to have extensive knowledge of psychology and read scientific articles.
It's about how engineers perceive productivity. If you solve a problem but engineers don't perceive it as solved, you haven't solved the problem. It helps with better-perceived ease of software delivery, engagement, satisfaction, and reduced burnout.
Organizations often use DevOps/SRE, and platform teams to remove the toil. I often see that these teams build solutions for themselves not engineers. Engineers don't realize that this will solve their problems. Sometimes it's even difficult for them to onboard such solutions.
Imagine you inherited a product where you have 3 teams. Staffed 50% of the headcount due to attrition. Company employees perceive this product as a "no-go zone".
Such a product was behind. There was no support because it was a new technology for the company even after 4 years. So, I invested in changing that. After a year, 10+ products wanted to join us because they saw solutions to their problems. One such solution was the delivery process mentioned above.
A whispered strategy is a strategy
With the lack of strategy, everything is possible. The worst kind of strategy is when you learn the strategy by talking to people. Yet none of these initiatives company incorporated.
You can complain about the lack of strategy. Better to be first to write the strategy plan proposal and ask for feedback, at most, you will throw it in a bin. At best, you can change the company's trajectory in the right direction.
Health is more important than deadlines
People are like production servers when they are running at above 70% of their capacity all the time. When enough critical servers die, you will invoke the definition of cascading failure without opening Wikipedia when someone asks about it.
Positive stress turns into negative stress which causes lower productivity and health problems. People aren’t containers that you can recreate. It’s about being like a cheetah, focused on one sprint to hunt down its prey. Use the remaining time to regenerate.
I had an engineer from an external team, let's call him Matt. Matt made a change to the SDK library that broke JSON serialization and deserialization. No one could call our APIs and we had to make sure these SDK libraries worked as expected.
Matt wanted to fix it, but he was sick. I convinced him to come home and recover. I did it because I could test disaster recovery mechanisms and succession plans. Also, it was the right thing to do.
My engineer resolved the issue. I agreed with Matt's Principal SWE to use our QA standards. This sped up the feedback loop and security. After Matt recovered, he was eager to work on this initiative.
People may close their laptops or leave work, but work will not leave them. The trick is to cut off the mental roots of your people so that they can recover and not think about work.
Companies fire even outstanding or competent people
Positive politics, such as being able to influence and do the right things are critical. Negative politics, when you work in a very hostile environment is something to hate. If you belong to the latter organization, run away until you take on the challenge of cleaning the house.
Working with a former Amazon VP allowed me to understand my weaknesses. Also, I understood why the CEO fired my Director. He built an entire organization 10 years ago and everyone liked him. CEO fired him after a migration that blocked the company's development for 5 months.
He never had a good Executive Presence. The new CEO didn't trust him and ran the organization on his own. It created more problems and burned many bridges in the industry when I spoke with people.
The migration was necessary because, after over 20 years of software development, major staff shortages due to lack of money, and technical debt knocked on the door like a debt collector and said:
You will pay off the debt, or you see your production for the last time...
When you have to choose between keeping the lights on and keeping the lights on, there is no room to add value. It will backfire and bring you down.
It required upfront investment in platform initiatives to do it with less drama. The only people laid off in this organization were him and me because "my work wasn't critical". 4 years later the company still uses the foundation laid by me which supports over 200 services and 40+ products.
Stay positive!
In Poland, we say without swearing:
It's clear that we're in a hell. The problem is that we are starting to settle here.
Migrations are tough like life. Surround yourself with positive people and avoid negative ones.
The last thing you should remember is that you had a good time with people you enjoyed working with. You won't do it again, but it was f… worth it!