Spoiler alert: If your boss does not need an elephant, he is definitely NOT going to buy one from you. If he will, he will regret it and eventually you will too.
I must appologize to the reader who was expecting to find an advice on selling useless goods to his boss. While I do use a similar technique to get a quarterly raise (no, I don’t), this article is actually about convincing your team, your manager or anyone else who has influence over project’s priorities, that pending system performance optimizations are a priority (assuming, they indeed are). However this headline was not very catchy and way too long, so I decided to go with the elephant instead.
System performance optimization is what I do day to day here at Percona. Looking back at the duration of an optimization project, I find that with bigger companies (bigger here means it’s not a one-man show) it’s not the identification of performance problems that takes most of the time. Nor it is looking for the right solution. Biggest bottleneck in the optimization project is where solution gets approved and prioritized appropriately inside the company that came for performance optimization in the first place. Sometimes I would follow-up with the customer after a few weeks or a month just to find that nothing was done to implement suggested changes. When I would ask why, most of the time the answer is someting along those lines: my manager didn’t schedule/approve it yet.
I don’t want to say that all performance improvements are a priority and should be done right away, not at all. I want to suggest that you can check if optimizations at hand should be prioritized and if so – how to make it happen if you’re not the one who sets priorities.
1. Estimate harm being done
This is the hardest and the most important part of this. The Question that you have to answer is this: How much does it cost your company NOT to have the optimization in place?
This should be expressed in dollars per amount of time (e.g. month). Normally, Percona consultant would tell you how much faster is something going to be when changes are applied. When we can measure it- it will be an accurate number, othertimes – we estimate it (e.g. when buying new hardware or making significant changes to database architecture is needed). What Percona can’t do though is map that to dollars for your company so this is something you will have to work out on your own. These questions might help you:
- How many users are we loosing [per month] because they.. (“get very unstable response times”, “get timeout for every 10th request”, “need to wait 10s for page to load”) ? How much loosing one customer cost us? Then, multiply.
- How much more efficient X dept. could be if they got this report in 5min rather than 4h. Would that give any $$ value to the company? How much per month?
- How many extra servers do we need to run because our systems are not optimized? How much does that cost our company every month?
- How many conversions are we loosing because users turn away due to slow service during registration?
It may be hard to get some of these answered if the company is not very transparent and you may have to go asking around, sometimes guess. That’s fine though, just keep a record of the guess you made so you can recalculate the whole thing easily once you have the number secured. In other cases there’s no way to get a good enough guess as the data is just way too far from you. In that case you either have to make a very rough guess or accept that there’s nothing you can do about it and that you don’t know how important for the company is the work that you do.
By the way, if someone says users don’t care if the website loads in 2s or in 5s, let him read this.
In the end you have to come up with one number. That number will vary greatly depending on many things like size of the company, number of servers system runs on, number of users, importance of the system we are working on in the global picture and what not.
2. Estimate the cost of the solution
Now comes the question of how much does it cost to implement given optimization?
In some cases it’s just a matter of opportunity costs: how much more valuable is it for the company that you work on implementing performance improvement -vs- that other thing? In other cases, you have to buy extra hardware and then see when (and if) the optimization is going to pay off.
When you have the two numbers, you can clearly see what is more beneficial to the company – get the performance improvements implemented now or leave things as they are and just work on the next thing. Few examples I just came up with:
(a) Company “Find that thing” runs a specialized search service with 25k new users monthly. 20% of users cancel their subscription after first month because sometimes the search is so painfully slow that the requests would time out. Because of that, company is loosing $60k/month. Search stability can be improved to all 6 sigmas if you just had 2 extra servers for search, but that is $20k extra for the servers and $1k/month for colocation. Well, that means the company is going to loosing ~$39k this month if nothing gets done and $59k or more month after month until it gets fixed.
(b) Company “Ana Lysis” sells data analysis services. Each customer report takes approx. 2h to generate, so one 24-disk server can only run ~300 reports a day and needs to queue them so no more than 24 such requests run in parallel. Since one customer can run 5 reports in parallel, with no overbooking they can only host 60 users on a single such server and still due to queueing response time would be unstable during peak times. For every 60 new customers, “Ana Lysis” needs to buy a new 24-disk machine which costs $25k. At the moment there are 6 such machines and a plan to buy 4 new ones next month. By moving to different storage engine, average report generation time would drop to 10 minutes hence one server would now be able to serve 12 times more customers all other things being equal, so not only the new hardware won’t be needed but the current one will be underloaded as well. You will save your company at least $100k. And if you know how many new customers come in every month, you can also get the $/month figure.
3. Make it a short and clear statement
Now that you know the most important numbers, go to your manager or whoever sets project priorities and make it clear:
“Our company is wasting $59k every month and I know exactly how to fix this.”
While you may sound funny at first if it’s the first time you talk about money other than your salary, it should still get you the attention – business is all about making money and no business likes to be wasting money. Most managers know this and they are normally welcome to save money. However, beware that different orders of magnitude are relevant for different businesses, so don’t be surprised if it appears that you bring in more than $59k a month to the company by working on that other thing.
Assuming manager happens to be worried about the $59k/month loss, now it is your turn to show exactly how is it wasting that much money and what to do about it.
4. Show the method, 5. the main problem and 6. the solution
Having the attention, I would try to be very thorough here and explain everthing from top to bottom starting with how it occurred that there is a problem at all, how the performance review was done, what were the findings, what’s the impact of other performance problems found and then most importantly, how you came up with the $59/month figure and what were your assumptions and guesses. Of course, don’t make it a 3h explanation – if you can run through this within 15-20 minutes, that’s perfect.
Once you’re clear on that, it’s time to present the solution, why you think it makes sense and how much you think it will cost – either in your hours or in company’s dollars.
7. Overcome any obsticles
At this point it is quite likely that your manager will come up with a few or lots of reasonable arguments against the problem or the solution. Don’t get discouraged – most of the time you have already thought about it and found a solution, you will just have to do it again if you didn’t take notes. On the other hand, managers often see broader picture and they know more than you think they do, therefore it is very important that you (either on your own or together with manager) actually acknowledge and remove every possible obsticle or it will be a show-stopper.
8. Kick it off
If you were able to overcome all the obsticles and unless the problem is fixed, the net result is still a significant waste of money, it’s time to go ahead and get that thing fixed.
Summary
I know this all sounds like a lot of work and indeed it is, however it is very rewarding as you focus on what is the most important thing for virtually any business: making (or saving) money.
Also note that Percona can do a lot of heavy lifting for you. The method that we use – Goal driven performance optimization – is based on the same value mindset: you tell us what would give you the most value and we will help you get there, while you’re busy rolling out that new feature or launching the new project.
Of course, there are certain things we can’t do, like re-architecting application, but often the question is moving to better hardware, optimizing indexes or queries, tuning MySQL etc. which we can offload 80-90% from the team. On the other hand, if the application is so bad it needs to be re-desgined, we are going to let you know.
Alternatively instead of coming up with a cogent argument and playing intra office politics, just install new relic (newrelic.com), then sit back and watch the whole development and management teams break down in tears right before you get approval to finally fix performance issues.
If anyone is wondering, new relic isn’t paying me a dime to make the above statement, though I did get a nice t-shit. Also to give credit where credit is due, It was Peter Zaitsev of Percona who turned me onto New Relic.
Thanks, Brian. While I agree about the awesomeness of New Relic, I am not sure about the negative sound of “playing intra office politics” :). I think everyone should have a right (and, in fact – an urge) to understand the value of their work inside the company as this is what leads to best results regardless of the position one is in. NewRelic does help in achieving this indeed, although you still need to do the numbers to dollars conversion.
Funny thing, I haven’t had to do the cost estimation yet. I find that I get 2 reactions at the same time after I have installed New Relic at a client site, 1 it’s like someone who was blind can now see, and 2 a profound desire to fix and improve what’s there from management down to the entry level coder. So my little side about “intra office politics and cogent arguments” was not really a jab about politics, but the rather profound effect of someone turning on the light in a crowded room full of darkness as opposed to describing what the room would look like if the light was on.