Reengineering Engineering with a new SDLC, with influences from Salesforce.com and Facebook

Our company has made amazing progress over several years and we create some world class products and we had a SDLC (Software Development Lifecycle) that matched.

• We were in a niche market owning about 95% and we were doing excellent.
• We acquired three other companies and placed them under an umbrella corporation.
• We were generating millions in profits after operating expenses.
• For the past five to six years we’ve had a twenty percent profit every year.
• We are the true company that will actual go public. Not just talk about it. You see we’ve been keeping our closet clean and our books show that for the past six years we have profits, a clean SDLC, and a top notch workforce, we continually get on the Inc. 5000 as top 5 fastest growing software company.
• We’ve built up a great culture here. People we hire know how to work hard and we know how to reward that work.

However as the term goes “who moved my cheese” things change. We turned around and had a recession in the automobile industry so we had to adjust. We then had a dozen competitors who were taking our business away from us.

Rant!

We offered a “Premium” package, this means we offered our customers everything. Put our products in your shop. Follow our processes and you WILL make more revenue which pays off our cost and puts a s**t amount more money into your pocket. Just because somebody builds something that looks like clone at a 3rd of the cost, does not mean it equals our product!!. As they soon found out and came back to us. But still! where’s the faith!.

Anyway we got a new presidente that recognized this and ALA Carted our product. Which I think was genius, we now offer completive rates with many behind the covers that our competitors just can’t match.

Reengineering:

Okay so now we want to show these competitors who their dealing with. Lets. We need a new SDLC for the new strategy.  It will be painful and tough to communicate to our offshore teams. A quick look at our old SDLC yielded something like this:

• Milestone I (Define Features)
• Milestone II (Specifications)
• Milestone III (Developer Code Complete)
• Milestone IV (Quality Assurance Complete)
• Milestone V ( User Acceptance Testing)
• General Release

This was built with a vertically integrated, monolithic technology for our onsite client/server architecture (distributed). It served us well and when functionality and reliability were being invented to address the needs of our customers.

Cons of old SDLC:

• It took QA as much time as it took the developer creating the code. Two weeks for a feature, Two weeks for QA.
• There is or can be a lot of finger pointing between QA and Development.
• Development might just code something and throw it over the wall, thus developers were not taking as much ownership over not writing bugs VS allowing somebody to find bugs and then building the cushion for them to fix the bug. This still held up the release date.
• Technical Debt would bite us sometimes because of the all too quick agile design to produce features.
• Million of other projects to add to the existing core products to address other roles in the service departments. But not enough developers as well as cost of training new development into the domain.
• Inaccurate early estimates resulted in missed features and their complete dates.
• Teams compressing their testing schedules to meet the demands of the hard feature date.
• Lack of visibility at all stages of the release.
• Late feedback on features at the end of the release cycle.
• Long and unpredictable release schedules.
• Gradual productivity decline as the team grew.

Research:

Here is some great resources that helps put things in perspective.

1. http://tcrn.ch/qEp6i1
2. http://bit.ly/rntqq6

NOTE: I just want to say in video 2. I love how Eric Schmidt started talking about US workforce and how he believes what America needs to accomplish to help bring it back into the top workforce of the world. I think it starts around 0:45:00. It’s very interesting.

New SDLC:

Outcome of our Company Engineering Standards Meeting.

• Foster Open Communication and over communicate.
• Empower teams.
• Continuous iteration.
• Delivering customer value early (More agile type processes in the form of weekly releases).

So what did we do. We looked at how Facebook operates and focus our SDLC towards their structure. Yes we know we are not Facebook. which is why we don’t adopt everything across the board. But obviously they have some good ideas.

• We eliminated quality assurance and took all those resources with their existing domain knowledge and put it back into the development organization.
• This means quality is now the responsibility of all developers.
• We determine to build a software architecture to be more in line with Salesforce.com and their service offerings.
• The development organization will adopt test driven development that allows for the discovery and resolution of bugs at the top of the project effort instead of at the end. “lean manufacturing” or lean software development.
• The development team has ownership over the features, this means they are responsible for seeing their feature promoted to production in a controlled manner.

Each team is now responsible for their own:

• Browser/OS Test Matrix.
• Repository Updates.
• End-to-End Testing.
• Environment Update Testing.
• Crash Emails.
• IT updates.
• Load and Performance Tests.
• Release Version Tracking.
• Server Setup.
• TSG (Trouble Shooting Guide) Documentation.
• User Acceptance Testing, in combination with other teams.

What we learned:

It has now been a 12 months since we’ve completely revamped our engineering SDLC so what have we learned, what’s the the post mortem?

• We are now a loosely coupled organization (Robust).
• We built a CI server that allows every team to use Ant and PowerShell to build their own CI that works for them.
• Every team now implements acceptance tests driven by the ScrumMaster/PM.
• Every team now has all their Ant/Powershell scripts custom to their unique needs and running on every check.
• Each team has dropped off branches and we focus on a single main branch.
• New teams/projects formed pick the new technology. So although we are primarily a Microsoft workshop. We have Java, Rails, Android, I O/S teams. (After approval. of course.)
• Teams have adopted testing as part of their development processes
• Reduced the number of processes.
• Strengthen those processes that remain so they now have more attention.
• Teams that spent a lot of time manually testing their products, now focus that time allocation towards building a effective acceptance test or integration test when applicable.
• Don’t write up bugs caught during the development life cycle. that just slows you down. Only create the bugs after a project has passed all unit and acceptance tests when a bug is found.
• Teams now focus on team throughput rather than individual productivity.
• Simple agile processes with common vocabulary.
• Prioritized work for every team.
• More defined organization roles (ScrumMaster, Product Owner…).
• Daily metric visibility into each team.
• Weekly Scrum of Scrums that provides visibility to all teams at a higher level.
• Reduction of bugs.
• Potentially release-able products every 30 days.
• Mandatory design meetings, code reviews. Any change no matter what is reviewed by at least one person
• Our code is beginning to look like one single developer wrote everything, not just a multitudes of everybody.

What we would do next time:

• Involve more individual contributors earlier.
• Train PM earlier.
• Work on automation earlier.
• Communicate delayed deadlines to executives due to the reengineering.
• Be more clear to workforce what the new SDLC rules are and ensure the dev managers enforce them more.
• Encourage more peer to peer coaching.
• Create a better dedicated, fully empowered cross functional rollout team.
• Managers need to understand to be patient and expect mistakes.

I hope this information enlightens you as it has me through this process. Much thanks to my old CTO Marcus Daleyand VP of Software Architecting Nate Zobrist. Marcus is a CTO that can keep at least 600 people all on their toes and pushing them all to be better and everybody should thank him for making them better at their career. Nate is a forward thinker and just has a much higher IQ then most anybody I’ve ever met.