Big Swinging Developer

The Accidental Architect (Part 2)

In the previous post I talked about software architecture and software architects in general.  In this post, I get into some specifics along with recommendations if you find yourself in a similar situation.  It is important to note that the advice that follows is for people who want to build good software in a good organization that supports your effort.  If you feel (or find) that you don't have the support necessary to get your architecture adopted, you may want to read Terrance Ryan's book Driving Technical Change

Have an opinion

If I could give only one piece of advice to a software architect it would be to have an opinion. Wishy-washy architects who speak vaguely in buzzword driven platitudes are the worst because they are inherently un-fixable. You can fix "wrong", but you can't fix "unclear".  Since there are so many possibilities with software the only way you can achieve clarity is through having an opinion and being able to describe/defend it. As an example, it was important to me that the system I designed be asynchronous from the beginning since I knew that we'd need to integrate with some slow external systems.  The opinion I formed is:

When the user clicks Save, their part is done.  The system should allow them to keep working and then notify them (if necessary) when the processing of their action is complete.

This led to a bunch of design decisions that wouldn't have been nearly as clear if I had a generic "the system needs to be fast" bullet point.

Be wrong early

You have your opinion and you start discussing it with others - your peers, your manager, your team, or whoever else will listen.  Be prepared to find out that you're wrong.  "Wrong" can come in a bunch of flavors: You've chosen a technology that's going to be difficult/impossible to get deployed, you've designed for a problem that doesn't exist, you failed to consider something that's critical.  The "be prepared" part means listening to feedback and watching reactions to find out that something could be wrong even if you're not being told exactly what.  It also means separating out general push-back from specific problems, especially from developers.  I once found myself in a surprising discussion with a developer over the design of my auditing strategy because he was worried that the database administrators wouldn't handle the data growth well.  Notice that that's a general fear that has less to do with the system (after all, auditing requires data growth by definition) and more to do with the environment.  We ended up keeping the design after discussing the various ways in which we could address data growth both with and without database administrator support.

Don't ship PowerPoint

If you're an architect, there is a strong possibility that you'll need to put your thoughts and designs into a slide deck to present to others.  This is a great exercise as long as you don't see it as your final deliverable.  The architecture I found myself designing needed to support several business units and perform a wide range of functions.  I had my opinions, I had my designs, but I didn't consider the architecture complete until I'd built at least a prototype.  In this case, that actually meant building the first module and I can tell you that things changed substantially from the time the paper design was done to the time we completed that module.  Even if you can't build an actual portion of the finished product, you should at least build a representative example of what you've designed and demonstrate that it exhibits the properties for which you've designed.

Know your environment

In the opening paragraph I noted that the advice was for people who wanted to build good software in a good organization.  If you're reading this blog, you probably have an idea of what "good software" is (i.e. meets user needs, can be extended, etc.), but you may not know what I mean by "good organization". The cornerstone of a good organization is honesty and most of the organizational problems I've encountered stem from a lack of honesty - anywhere from people kidding themselves to folks outright lying. In the context of development and architecture, you'll know you're in a good place if your opinions are met with open discussion and debate. It's surprisingly satisfying to have someone show you that you're wrong and then watch your design improve as you hash out a better solution together. You'll know you're in the wrong place if your opinions are met with silence, unproductive bickering, or passive aggressive avoidance. If you find yourself in such a place, then once again check out Driving Technical Change or dust off your resume and start to look around.  There are too many good places to let the bad ones grind you down.

The Accidental Architect

2 Years ago today, I wrote about centralized architecture groups in organizations.  A few months after I wrote that post, I was commissioned to write a whitepaper about enterprise architecture by a local boutique consulting firm.  The idea of software architects has been on my mind a lot lately.  While I've been selling myself as a Big Swinging Developer for the past several years, I recently found myself in an architect role accidentally.  I'll go into some specifics in the next post, but first I want to throw out some of my thoughts on software architecture in general and software architects in particular.

WTF is Software Architecture?

It's one of those terms that seems to mean something different every time you hear it.  In the realm of consulting it often translates to "the most expensive PowerPoint slides you've ever seen".  My personal definition is:

"software architecture is the collection of hard-to-change decisions about a system" 

Decisions like whether to use an ORM, or an Enterprise Service Bus, or how security is going to be handled.  These aren't impossible-to-change decisions, but if you've ever been on a project where something of this magnitude was either overlooked or chosen poorly, you know that this kind of mistake threatens to sink a system.

What Does an Architect Really Do?

That depends on two things: The individual and the organization.  The problem with the title "Software Architect" is that there isn't a strong connotation of what will be delivered.  A "Software Developer" better deliver software.  A "Graphic Designer" better deliver designs.  A "Software Architect", however, has no corresponding delivery requirements.  This is left largely to the individual and it's then often left to the organization to decide whether or not to tolerate what they do all day value the role.

In the past 5 years, I've been fortunate enough to work with the full spectrum of architects and the full spectrum of organizational attitude.  My favorite type of architect is one who has a good grasp of how software fits into an organization, the technical ability to address the challenges, and feels a responsibility for the success of the developers who will be implementing the vision.  Former colleague Mike was a great example of this.  He know his stuff, had strong opinions on the things that were important to him, and always provided a reference implementation to show that his design was practical.  My least favorite type of architect is one who generates nothing but documents (PowerPoint being a favorite), dictates "how things should be", and then blames the developers when the vision turns out to be impossible or useless or both. 

But architects don't exist without an environment in which to practice, which is why organizations are almost as accountable for architect quality as the individuals.  The quality of the environment is more easily determined by what it won't tolerate than by what it encourages.  Sure, you want a place that encourages quality work and personal accountability, but that's not enough.  What separates the good from the bad is that the bad organizations will tolerate (or even value) the Big Talkers who spend days preparing for a presentation without ever once vetting the practicality of what they're proposing.  You probably know who I'm talking about, the folks who will show you super-complicated diagrams of their Service Oriented Architecture but can't answer a single question about how to version or end-of-life any of the listed services.  Casting things into "good" and "bad" is a little simplistic, though.  There are really 3 types of environments:

The Consciously Good Organization

These are the folks who have their act together.  Somewhere along the way, they've committed to creating first-rate systems.  If you find yourself in an environment like this, they'll likely have a really good architect who you can learn a lot from.  If you're brought into to be an architect in this type of organization, you are being brought in explicitly for the role and you'll have a good idea of what it means to succeed.  There is money to be made in a consciously good organization since they tend to pay for quality, but note that the competition for the role will be fierce and you'll have to be really good to land the gig.

The Cluelessly Bad Organization

These are folks who want to be good, but don't know how.  Or maybe they don't even know that they want to be good, but they have a feeling that things could be better and are receptive to ideas on how to improve.  Honestly, this is my favorite type of environment.  There is a lot of money to be made in these types of organizations because they are often leaving so much of it on the table with bad systems.  Not only is the money good, but the work will be very pure in that you'll spend much of your time showing them how things can be better and then implementing it.  If you are correct in your analysis and solid in your implementation then your work will easily pay for itself and you can continue until you've built them into a sustainable technology powerhouse.

The Delusionally Bad Organization

This is the danger zone, because these folks think they're good but they aren't.  Remember when I mentioned that you can more easily determine the quality of the organization by what they tolerate?  This is where that little nugget comes in handy.  The delusionally bad organization does some things well (and will point them out as evidence that they're "good"), but they tolerate mediocrity to the point where the good stuff almost doesn't matter.  The way you can tell the delusional from the clueless is that the delusional will actually know a lot about what it means to design and build first-rate systems . . . but then they won't do it.  Instead of building good stuff, everything turns into a messy collection of compromises that's explained away by factors beyond their control.  The good news is that there's money to be made in this type of environment because they will, in fact, need help.  The bad news is that they will make you earn every freakin' penny of it.  The work isn't really the work, the work will be trying to get things done in an environment that thinks it's okay even while things are crumbling around them.  You'll suggest a way of doing something, get told that it's not necessary, and then see the negative effects of not having it . . . all before lunch.

Is an Architect Role Right For You?

As you might imagine, the job of software architect isn't for everyone.  One of the best programmers I know has no desire to be an architect even though he certainly has the technical qualifications.  So how do you know if the position is right for you?  I think that before even attempting the role, you need the following:

• Wide technical experience with some pockets of depth - you should be able to build everything yourself, even though you won't.
• Network operations or sysadmin experience.
• Defensible opinions on how software should be built.
• The ability to convey complex ideas in simple language.
• Ideas about how software and technology can make organizations more successful.

Have I missed any big ones?  If so, leave a message in the comments below!

 

 

What To Do When You F Up Big Time

This post will probably strike you as either common sense or absolute crazy talk.  It is especially written for those in the latter group.

I write a lot about working safely.  After lots of posts on branching, test environments, kitchen analogies, etc. I'm here to recommend some behaviors for those times when you totally screw up.  After all, you may very likely find yourself in an environment without all of the safety nets you want because you were specifically brought in to build the safety nets.  I'm going to assume that you messed up while doing the right thing in the wrong way rather than something criminally stupid like, say, encoding your DVD collection to Divx on the production database server because "it has those really fast drives and all that RAM".

First, and foremost, as soon as you realize that you've screwed up, let someone know.  Do not be tempted to keep things quiet and fix it before anyone notices.  I have yet to see a production issue that didn't get worse with time (and quickly).  Keeping things quiet is outright selfish because you're putting your own comfort ahead of the good of the group.

Secondly, fixing your mistake needs to become your top priority.  Fixing means not only getting things working again, but getting them back to the way they would have been.  Does data need to be re-keyed?  It's now your job to re-key it.  Do numbers need to be verified?  If you're not the one who can do it, be prepared to generate special reports or data dumps to make the job as easy as possible.

Next, take responsibility for your mistakes.  Full responsibility.  You don't get to say, "I deleted the production website, but the slow restore process is what caused the outage to be so long."  Being up the creek without a paddle means that you own the upstream and downstream problems as well.

After things are back to normal, do your own private After Action Review (note: there's a good chance you'll be asked for either a public one or one with your manager).  Take this opportunity to learn from what just happened while it's still fresh.  For a big enough mistake, you'll probably also reflect on it for a day or two.  Having said that, hear me now and believe me later: do not utter the words, "Well it's kind of lucky it happened because...".  Even if there's some fantastically beneficial outcome, you don't get to celebrate the effect, you are still responsible for the action.

Lastly, get over it.  If you've made the kind of mistake that I'm writing about, it will almost certainly affect you emotionally, mentally, and physically.  That's to be understood and will actually help with internalizing the "Don't do that again" lesson.  But don't let it affect you too much for too long or you'll kill your productivity.  If making a huge mistake makes you skittish to the point that you are no longer a high performing contributor, then things aren't back to normal are they?

As a final thought, while things are at their worst you may start wondering, "Are they going to fire me for this?"  I can't answer for certain, but I can tell you this:  When I had headcount, I never fired someone for making a mistake . . . and they pulled some doozies.  If you do get fired for a blunder that you feel comfortable defending (i.e. Doing the right thing the wrong way), then chances are it wasn't the place for you anyway.  The only way you can do truly incredible work is by being willing to take some risks and if your employer squashes any chance of that happening by firing people for mistakes, then you're better off elsewhere.  Just don't make a habit of it: it's easy to explain a one-off, but the second time you get fired for f'ing up big time it starts to look like a trend.

Brownfield Development: How To Peel The Onion Without The Tears

In the Spring of 2008, I decided to become a brownfield development specialist.  Greenfield development is when you're starting a project from scratch and get to design everything with only minimal constraints.  Brownfield development is the opposite of that: it's when a project is n months into development and most of the constraints have been cast in stone.  My guess is that if you were to ask every developer you know whether they'd prefer to work on a greenfield project or a brownfield project that they'd all say greenfield.  Heck, I'd bet that a quarter of them would laugh so hard that Red Bull shot out of their nose just at being asked the question.  Therein lies one of the secrets of becoming a Big Swinging Developer:

You can make money being good at things that other people hate to do.  A lot of money.

When people hate to do something, they don't do it very much.  Since they don't do much of it, they never get particularly good at it.  This makes it harder for them to do it and the cycle starts all over again.  Some common values of "it" for development include: writing tests, documentation, and creating installers.  You can safely add brownfield development to the list since a job description of "Support the big ball of mess that runs our business while adding features and fixing bugs" doesn't usually have folks banging down your door. Throw in the fact that you'll have no relevant documentation and few, if any, tests to guide your way and you can picture weeks of pestering your co-workers with questions just to get to the point where you can fix a simple defect.

But what if brownfield was easy for you? What if, rather than pestering your co-workers with basic questions, you could understand what the code was doing and then ask why it was doing things that way rather than asking what it actually does?  That's what my secret weapon, Visustin, gives you.  You paste in your source code and it'll generate a flowchart for you:

It supports 31 languages including the popular .NET, open source, and SQL variants.  I spent last week throwing hundreds of lines of Python into the tool and tracing through an incredibly complex financial trading system to learn how portfolio valuation is calculated.  I was able to correctly describe the process to my team lead, including a couple gotchas buried in the code even though I don't know Python.  Since I know how the system works, though, I can find the path the code will take and identify where problems are likely to occur while coming up to speed on the language at night.

If you're a developer, I'd highly recommend Visustin.  It's great for code reviews, documentation, and for diving into existing systems.  Developer or not, look around your industry and find the important things that no one wants to do because there's a real opportunity there.  You can become better (or more tolerant) than anyone else simply by identifying the key aspects to the unpleasantness and solving that problem first.

Why Should We Have Centralized Architecture?

I recently had a chance to discuss the role of centralized architecture with a colleague.  When I say "centralized architecture", I'm talking about a group usually found in large companies that is responsible for technology standards across the organization.  In talking with developers over the past several years, I've seen the full spectrum of opinions on central architecture groups: At one end you have the developers who are appreciative of the architects hammering things out so that the organization has a common direction.  In the middle you have the folks that have heard such a group exists, but isn't exactly sure what they do.  At the other end are the developers who see the group actively impeding their progress.

Given the wide range of opinions and reactions to the architecture function, I started thinking about what I would want such a group to do.  I finally boiled it down to this:

The role of central architecture should be to balance the desires of the organization with the desires of the developers.

There are plenty of places where the desires of the organization and the developers are aligned.  The org wants some software, the developers want to write it.  But from there, things get a little tricky.  The org prefers to standardize on certain products in order to reduce cost and simplify maintenance.  Developers want to use whatever they want to use.  Developers focus on their system, or part of the system.  The org sees their system as part of a larger system and wants to make sure it can talk to everything else when need be.  The list goes on and on, but an architecture group can resolve these differences by providing the structure with which technology gets built.  Not only that, they can help developers work outside of the general structure when it makes sense to do so and provide tools that make staying inside the structure desirable.

Are You Busy or Drunk?

There's a difference between "having a lot to do" and "being busy". 

I almost always have a lot to do.  I have a long list of tasks and projects that I'm interested in doing and I can productively work on that list at the drop of a hat.  By the same token, I can usually stop what I'm doing and do something else if the need arises.  There are certainly some tasks that require uninterrupted time, but those are easily accomplished in the "off-hours" of early or late.  Most of the people that I enjoy working with structure their work similarly: they're never "busy", but they are constantly making progress.

The people I least enjoy working with are the "busy" ones.  These are the folks who feel buried by their tasks and often when asked to do something (and almost always when asked why something didn't get done), their response is "I'm very busy".  The worst part is that everyone seems to take being busy as a universal and unquestionable excuse instead of immediately responding, "Well how did you let yourself get into this predicament?".  My friend, and never-busy-colleague, Jimmy C once pointed out that being busy is a choice just like being drunk is a choice.  He likes to say, "Replace the word 'busy' with 'drunk' and see how much weight it carries". 

"Sorry, I can't help you out, I'm too drunk" or "He's a very drunk person" or "I didn't get to it yet, I'm too drunk".

Really takes the teeth out of the statement doesn't it? 

I've spent too much time over too many pints^* trying to figure out why the busy people are the way they are.  Poor time management?  Unrealistic estimates?  A need to feel more important than they are?  Guilt over their bill rate?  I finally realized that answering the question wasn't productive, so I stopped thinking about it since I have a lot to do.

So stay productive and don't get too busy too often.

*I said it was a choice, not a sin.

The Difference Between Knowledge and Belief

Every developer knows that unit tests are a good idea.
Most developers know that using version control is a good idea.
Some developers know that a "one-click" build is a good idea.

As you walk the aisles of the programming section at a bookstore, or browse the online equivalent you'll see knowledge galore.  How-to, what-to, why-to out the wazoo.  With all this knowledge so readily available, why aren't there more good or great developers?

The Big Swinging Developer is rarely judged by knowledge alone.  As always, you have to be at least "good", which involves knowing a thing or two, but it's your beliefs that will set you apart from everyone else.  I happen to believe in isolating changes/problems, repeatable builds, and testing.  Because of these beliefs, I've learned a lot of techniques to help me practice them.

I've been working on a web app this Memorial Day weekend.  I'm a development team of one.  No one is watching me, checking up on me, or even working with me.  I have a one-click NAnt build that deploys to a Virtual Machine and have been branching my releases in my hosted Subversion repository. 

You practice your beliefs because you can't imagine doing things any other way and that defines you as a developer more than any single technology or language ever can.  So, what do you believe?

Surrounded By Tools

Like other craftsmen, the Big Swinging Developer has an above average toolbox.  When a problem rears it's ugly head, the right program, utility, or plug-in is just a download away.  Like a master craftsmen, Big Swinging Developers also realize that tools are there to perform a function and that there's more than one way to complete a task.  When you've reached this level of mastery, you stop complaining that your client uses ClearCase for version control even though you prefer Subversion. You'll gladly work with Oracle, SQL Server, or MySQL . . . after all, databases are just a tool that let you work with data.  You're a fan of Infragistics, but they're a Syncfusion shop?  In the words of Tim Gunn, "Make it work".

This is not to say that you shouldn't have preferences.  And you should certainly put in your two cents when they're in the process of making technology decisions.  After the decisions have been made, however, it's time to get down to business.  Remember: It's a poor craftsman who blames his tools.