Why creating a engineering wiki site. (Living Document) is a good thing.

Recently I’ve started at company that didn’t really have a good solid engineering wiki. Let me state the problems and if you fall in this category, you might want to start thinking about looking at your software development practices.

Problems:

• Knowledge Islands: Anytime and I mean anytime you need answers about something everybody always directs you to somebody else. The alternative is to spend so much time in code that might be so abstracted or not very transparent that although you CAN figure it out, the cost to the company is high. OR It’s pretty much impossible for said individual to move on to another project without constantly being bothered about his “design” six months after he’s left. When I say bothered I mean weekly.
• Knowledge Gaps: Do you have multiple efficient teams but all all of them don’t contribute to the over all “Engine” so that the car all moves in the same direction?
• Very unclear/no requirements: Agile does not mean you do NO documentation.
• Each standup/scrum master does their own thing: On this team you create tasks for unit tests and then you swap to another scrum master and unit tests are atomic and “should” be included in as apart of your tasks already.
• Engine that could: Do you have

Plenty more to add, but you get the idea I think.

Purpose of a Wiki

Some of the benefits of a wiki can be:

• QA can learn how functions changed or work from previous versions.
• PM's can disseminate information to peers, which allows feedback and question to better the existing design.
• PM's can build Releases Notes can be used to communicate outside of engineering about how features work.
• PM's can share with the entire company about their notes on competition, this helps share the knowledge of understanding to all in engineering to help employees understand the market we fight to dominate.
• Managers can use use simple and precise language and general propositions in new or existing processes, or evolve them to fit the reality situation.
• Managers can update status reports for their team, not handled in a TFS Query.
• Developers can reduces the knowledge islands which can can then allow people to move on to the new projects or not leave a knowledge gap.
• Developers to write a very high level outline functional and technical specifications (one page) of a Feature and can do so right after development has taken place. (Not before and not months after..)
• TAMS can build install guides which can be used inside/outside their existing team.
• New employees are empowered to find answers on their own, or become educated enough to ask solid questions.

..... Just to name a few advantages.

Q: Isn't having a wiki kinda anti-agile?

• Document writing is not anti-agile. Fostering open communication and getting feedback from others/customers is a important aspect of being agile. it's more efficient and it allows documentation to evolve from its original to a current working state as well as disseminate that information across the company.

Okay.. so I’ve convinced you to start a wiki for your engineering company. Here are some problems you will want to consider solving.

• If emails are sent out and they link a wiki page. What if that wiki page is changed. Portal sites like Sharepoint have static URLS so you if the page changes, the email link becomes invalid.
• Code Syntax Highlighting, it really helps.
• Flexibility
• Ease of Use
• Can it hook into your Active Directory?
• Does it have a good search?

My recommendation for all this: MediaWiki

Here are some tips for your wiki.

• Prefix all your pages with something like: We call it a “Manual of Style’
  • Design - 
    • (Code is the comment my butt), This page doesn’t capture the design the developer was “going” for. Or the “right” way they intended you to add to their design.
  • Tutorial -
    • How To, this lists step by step how to do something
  • Guide -
    • Best practices
  • Standard -
    • This is a YOU MUST follow this.
  • Process -
    • This is a stream line process. Not something to be blocked on. But it outlines easily what’s expected.  Do this, email this email group. here’s a template that details what we want to capture.
  • ….. etc etc
• Your SDLC (Software Development Lifecycle) should be focused on delivering business value with emphasis on high quality frequent releases. It should be based on the following four values:
  • Over communication
  • Empowering Teams
  • Continuous Iteration
  • Delivering Customer value early
• Build a Category system that fits your company. In my current company we have “many” products that “share” components. So I built categories that are “Features – “ Then everybody binds their page to those Features and all those Feature categories are bound to each Product that uses them. That way if they are shared they can be found easily. It also easily shows a Feature list for each product.
• Build a wiki syntax help page that details how to create a new page, or how to bind to the categories. etc.
• Build a email group of wikihelp and add in as many good solid stakeholders. So when question is asked, all get it and understand the solution. i.e. form a committee
• Organize your companies SDLC and Processes, show in there what’s expected from engineering (QA, Installers, Build, Devs, PMs, Mgr) on a Daily, Weekly, Sprint, Release. This way it’s clear and understood by all.

Adoption:

• Getting a company, it’s employees to switch gears and adopt this culture changing event is going to take time, you need to be active, leadership must adopt this and help enforce and promote it. Eventually it will take as time goes by and people will be able to see the benefits of being able to search for their own answers and understanding clearly what’s going on in the company and what’s expected of them.

Writing more effective emails in your company.

I work at a very email-centric environment and worked under the best CTO I’ve ever had the pleasure to work under. One of his emails he sent out is a great template to follow when writing emails. I wanted to share the email template.

1. Format email responses that consistently repeat the following:

• What is the problem?
• What are the steps that will be taken to identify the fix?
• Who owns driving those steps?
• When will those steps be taken and results published?

2. The above email should be sent daily until the issue is resolved

• A day in SRS time is a week for another company

The second item is clarity of communication.  When we are sending an email that contains release notes, UAT notes/plans, crisis management steps (i.e. what was mentioned above), etc. we need to make sure we are explaining things in layman terms.  Describe the situation like you would to a family member who doesn’t work at the company.  Someone you care about.  This may sound silly, but it’s not.  Terms engineering uses are not universal and are thus a foreign language to others in the company.  Please communicate briefly, but clearly using words that are more likely to appear in Webster’s dictionary than in the wiki.

Software Construction

1.1 What is Software Construction?

Many books, articles will make analogies that software construction is a lot like building development. They port over concepts from building development into the software construction world. Many points are accurate, but some are not. The root problem that everybody should agree to is that computer software can be a complicated process.

To me programming is human thought codified and crystallized. Today's software metaphor normally relate software to buildings. There are others beside this metaphor, programming to math, programming as a model, or programming as rule-following. Given that my premise of programming is just raw human thought, then it stands to reason that some people are more apt towards this field then others are. That doesn't mean we can't train our brains or though process to be better focused towards this field. But it also means that there are those out there with slight advantages as a "master programmer." however the software industry does not only need "master programmers" just because you are a great problem solver does not mean you are somebody that can stand back and look at the larger picture. So in our field you also need the "Architect." Metaphors, design patterns and coding styles cannot replace actual intelligence, but they do help us to communicate to each other what it is we are trying to do. This is what Software Construction is about.

1.2 Why is Software Construction Important?

In our current market of today, construction is a major part of the Software Development Life Cycle. (SDLC), this is the life cycle of a software project with formalized stages for a start to finish solution. In the business world, it is a major part of their process.

Each software project you begin, you must realized there are four factors that effect that project;

• Time
• Resources
• Quality
• Features

At any given moment, you as the developer, own one of those four items. The other two are controlled by external sources. For example, say you have to complete this project in three months. You have customers flying in to train, or trainers flying out to train on your new feature/product. This is a hard date (meaning it can't be moved). Also the quality of the product must work as a finished product with very high quality requirements. In this situation you should be able to control how many resources it takes to get to this milestone.

• For every project you work on you must pay the price of time. Either you pay for that time up front with some good construction design. Making your application scalable or flexible . Then when you do need to be handle changing something, because you designed your application with flexibility, you can easily "hook" into the existing code "interject" some new behavior at the right processing stage, debug, test and ship over to QA. (because we all know you don't just ship out to the customer without QA approval. That is what we call a "Hope Install.", and your customer should not be your first and last QA).

• Most business stakeholders want you to finish the project faster and meet the requirements. Get it done and quickly. These people do not know that choice leads to greater bug time, as the developer will have to either refactor a lot of code to fit in the new changes. Let me put it to you another way business people. You tell me which is cheaper. When would you like to choose to move a elevator shaft if it ends up not being in the proper position. Would you choose to do this on paper while designing, or once it's completed. If you choose to do so once it's completed, it will be very expensive.

However there is a pitfall here. Anybody can sit down with a architect and produce a "specification" that, in truth does not adequately describe what the customer wants or what the requirements needs. This fact may not be apparent until after the customer sees the first demo of the software. Identifying these features and planning on them moving will money and time for all involved.

• Many software projects fail because of weak specifications, not all of them, but enough to say.. Don't make weak specifications.
• Many formally trained developers might over engineer and take way too much time for specifications that are not accurate. So don’t over engineer, just for the sake of making documentation.

My recommendation to you is, take some time to focus on the most critical 20% of your project core that needs to be worked out (thoughts onto paper) and then proceed with the project. This is a good general rule, but depending on the size of the project, may or may not be necessary. Large projects need more documentation, projects with more then three developers need more documentation to coordinate API's with each other. Small projects or one developer don’t need as much. It's a weighted scale and you must identify what is necessary to get the job done.

1.3 Determine the Kind of Software or Industry requirements you are working on.

Ask yourself what category does this project fit into?

• Business Systems
  • Type of Projects: Internal, Intranet
  • Type of Work: As-needed testing and QA Planning, informal requirements, Design and coding are combined, Developers test their own code and little or no testing by separate test group, somewhat informal deployment procedure.

• Core Systems
  • Type of Projects: Embedded Software, Packaged Software, Tools, Web Services
  • Type of Work: Staged delivery, up front planning, basic test planning, semiformal requirements, architectural designs documented, as-needed design reviews, code reviews, developers test their own code, separate testing group, formal deployment procedure.

• Mission-Critical Systems
  • Type of Projects: Core Embedded Frameworks, Required Release Set Software
  • Type of Work: Staged Delivery, Extensive up-front planning, extensive QA test planning, Rigorous change control, Formal requirements specification, Architectural design, formal architecture inspections, formal detailed design, formal check-in procure, code reviews, developers test their own code, separate testing group, formal deployment procedure.

1.4 Defining the Problem

Defining the problem should be described from the user's point of view, not to be stated in technical computer terms. You don't want to tie the problem to specific technologies, because that may not be the best solution to the problem. If you list something like "Refactor stored procedures to solve performance problems." You are focusing on the specific area that may not be the problem at all. Instead the problem should be stated. "Performance problems when running client reports." This way you are not miscommunicating to the developer where the problem lies. Listing stored procedures may or may not be the solution.

1.5 Categorizing the Software Project

1. Small Projects
   1. Less then 100,000 lines of code

1. Medium Projects
   1. Greater than 100,000 lines of code and less than 3 million lines of code

1. Large Projects
   1. Greater than 3 million lines of code

Lines of Codes of other projects

• Windows NT 3.1 ( 4-5 Million)
• Android OS (12 Million)
• Windows Server 2003 (50 million)

• Mac OS X (86 Million)

1.6 Meeting the Business needs of your Software Project

This is where you choose your technology you want to use to solve your software project. This includes the programming language, any third party libraries or technologies. Most times the programming language has been selected for you, so you might want to determine the type of classes you are going to use. Are you going to consistently use LINQ Query Syntax, or use long query syntax. Maybe you will use EJB's for Java, maybe not, or you might use WCF for .NET technologies.

What types of coding conventions do you follow? Good code should look like one developer wrote all of it, not pieced together from multiple developers. When choosing technologies if it's bleeding edge, realize that the community for it will not be mature. VS a mature community of a proven product or technologies which chances are that somebody already has tried what you are attempting to do. Be certain to choose the best software practices and tools that are best suited to your project.

1.7 Quick Construction example

Focus on documenting 20% of the Core requirements, do this by identifying the most critical components to achieve success. Focus on categorizing each component into three categories:

Lets say my company required me to create a listening web service for a partner to push massive amounts of data through so I can parse some information out of it, and build a held message for another project to pull from my storage cloud down to it's domain.

Identify your components

• Focus on documenting 20% of the Core requirements, do this by identifying the most critical components to achieve success. Focus on categorizing each component into three categories:
• Lets say my company required me to create a listening web service for a partner to push massive amounts of data through so I can parse some information out of it, and build a held message for another project to pull from my storage cloud down to it's domain.

Identify my components

1. Listening Web Service: I choose WCF
2. Partner Interface Method:  I choose to follow a standard in my industry name STAR Standard
3. Extracting data - I will use LINQ Query Syntax
4. Storing Message Structure - Build a project that handles dealing with Message Packets.
5. Storage - Utilize Windows Azure Cloud
6. Testing - Utilize a 3rd party tool. SOAPUI to mimic our partners message, use Azure storage explorer to see data in cloud

1. Deploying -
   1. Dev - (Use host file to build a friendly name for dev testing purpose, no need to bother IT with outside dns) Azure also provides the Cloud Compute Emulator to simulate cloud on dev box
   1. Stage - Azure accounts each have staging and production. This can be combined for a "Dev Azure Account for my final testing"
   1. QA - Get a Domain Name alias (friendly name) to give to partner to use. Map to a "QA" Cloud account. Again there is a Staging and Production in the cloud, so no staging is necessary to configure.
   1. Production- Get a domain name alias (friendly name). Deploy to cloud

1. Maintaining - What can I do to minimize having to redeploy for bug fixes? Research this with the cloud to see what I can use. Estimate: 6 hours for research. 3 for designing something if there is anything. After that.

1. Recovery Plan - Research.

Focus on identifying the 20%

• I know what I know (1 - lowest to 5 - strongest)
  1. (4) WCF
  2. (3) STAR Standard
  3. (4) LINQ Query Syntax - extracting data from where clause. No fancy queries foreseeable.
  4. (1) Azure Storage Cloud
  5. (3) SOAP UI
  6. (3) Azure Storage Explorer
• I know what I don't know (this only focuses on what we actually have in the requirements)
  1. Customer data structure
  2. Performance of WCF in Cloud
  3. Integrating WCF with Web Role or Worker Roles of the Azure Cloud
  4. What types of configuration/power should I give to those that maintain this?
  5. Logging in the cloud. (Limitations?)
  1. Recovery if production is wiped out, how to rebuild this easily? Scripts? What?
• I don't know what I don't know (these are the what if scenarios)
  1. Schema validation?
  1. Duplicate messages one to many?

  So from the look of things, I think I should probably focus my detailed documentation on what happens When I extract the data from the customer and how I'm storing the message. I might end up with a rough draft that looks something like this:

•  

  1. Inputs to Public Azure Roles

  • PutMessageV1.svc : Called by IIK Series Partner to send information through. Message will parse information and create a unique address name.
  • ProcessMessageV1.svc: See PutMessageV1, exact same functionality.

  2. Input to Internal Azure Roles

  • ReceiverV1.svc Receives structured data to store until pulled by another project.

  3. Internal Azure Storage

  • Held Packet Table Held messages in Azure Storage.
  • Delivered Packet Table Once message has been delivered it is stored in this table for archiving or audit purposes as well as guaranteed delivery
  • Log Table Log table for storing log entries.