Thoughts on "How do you become an Architect?"

There's an interesting post on the Server Side from Jan 26th entitled "How do you become an Architect?" and although the original blog they reference isn't all that great, I was pleased to see that some of the beliefs I have, on what it means to be a "Software Architect", are also shared by others.

Here are some of the quotes and ideas on the topic at hand - "How to become an Architect"

http://www.theserverside.com/news/thread.tss?thread_id=44011#226268
"-Gain the respect from the developers actually implementing solutions. You should be able to jump in and get your hands dirty. I've also found that you need to have a good working relationship with system or business analysts.
-Prove yourself on successful projects. Word travels fast when your design is good.
-Develop diplomacy skills. Lots of managers feel that architecural [sic] decisions from somebody else make them look technically weak. You need to find a way to sell the best idea, and cut through red-tape.
-Think big picture, think strategically, think abstactly [sic] . The challenge of a good architect is to be tactically and strategically proficient. Developers and managers often fall to either side.
-Read, read, read. "

http://www.theserverside.com/news/thread.tss?thread_id=44011#226287
"Wether [sic] you use the term 'architect' or use another term I believe there is a need for a role where the end responsibility lies for the structure and *-ilities of the things you're delivering. "

http://www.theserverside.com/news/thread.tss?thread_id=44011#226303
"I have spent long hard hours thinking about this question :-)
My business card states exactly what I do . . . . I make stuff happen"

http://www.theserverside.com/news/thread.tss?thread_id=44011#226312
"For this discussion, I would also add that the architect must be a leader. Architects rarely have enough authority to truly enforce conformance to their architecture. I've never seen a single system blocked from going into production just because it didn't fit enterprise standards. That means the architect, if they aren't on the individual project teams, work through influence, not control. Hence, the need for leadership and earned (not demanded) respect. "

http://www.theserverside.com/news/thread.tss?thread_id=44011#226339
"The best way to become an Architect is by listening.

- Listen to your business customer. At the end the day your designs are only good if it meets his or her needs.

- Listen to developers. You may have the best blueprints, but they are worth nothing if you don't have developers on board to build them. "

http://www.theserverside.com/news/thread.tss?thread_id=44011#226388
"Being an architect is not about kissing ass with your managers. A lot of times it is about putting
your neck on the line for your developers and explaining to your managers that certain decisions
are detrimental to the project, that a schedule is unreasonable and will result in failure and so on

Lastly an architect should be very approachable. Developers do not like architects with an attitude. Such architect tend to create distrust for their own skills. Be there when your developers need you and please can that attitude. This helps team building and promotes a spirit of cooperativeness. Even the most hardened 'protect my territory' developer will come around if the architect is a approachable sharing member of the team. The cluture [sic] of a team is defined by the attitude of the architect."

Boiling it down to the key aspects . . . .

• An architect *must* code to gain the respect of developers and be approachable
  
• You must prove your ideas are successful and that you can "get it done"
  
• The (technical) buck stops with the Architect
  
• Architects succeed through influence not control
  
• Architects must listen to their two key audiences - the users and the developers

In a sense, being an Architect isn't all about being in a position of power, it's more a position of overall technical responsibility and also serving the needs of your users and your development teams. The concept of "Servant Leadership" is a bit overused on the 'business motivation' circuit but there's a good grain of truth in it. When you realize you can't really control (free) people, you also realize your 'leadership' role then becomes one of enabler rather than enforcer.

Putting the A in SOA

You'd have to have lived in a cave for the last three years to not know that Service-Oriented Architecture (SOA) is hot hot hot. Why is it hot? Well the IT community has sold it as yet another "Silver Bullet" but even so, many CIOs believe SOA will provide a competitive advantage but also CIOs with an SOA are paid on average 57% more than those without! And their budgets are bigger too.

So whatever you believe - SOA will probably touch more of our lives before the hype fades and it becomes "just another tool".

So after the Sales folks have left and you starting asking yourself what's the first step, one of the first key things to do (perhaps after a few quick wins) is to put the focus on the "A" in SOA. Grady Booch had a great blog article on this a few weeks back and touched on many of the same points I'll talk about here.

So here are a few thoughts to "prime the pump" as it were - things to consider for the architectural aspects of your SOA . . .

1) Antipattern - "If you build it they will come"
First off a classic anti-pattern of entrepreneurs and those chasing the hottest technologies - "Its a good idea and someone will like / buy it" - you first services need to be providing some value to the business (at least beyond a proof of concept). That leads nicely to the next point.

2) What's the Value Add?
Exactly what is the value you are adding? More customers? Lower costs? More exposure outside your business unit? Also when will the value outweigh the up-front cost? How will you measure value? Chargebacks? Although this is less an Architectural question than a business question - it's always nice to motivate the technical team by knowing "this will make a big difference"

3) Who will use this?
What technical expertise do your users have? Is it to be consumed by excel-savvy business users or other IT departments and their developers - this knowledge can help guide some of your technology choices. And if the users / uses change how will we change the services?

4) Service Granularity
As you add services will your users be inundated with 270 services they can call or can you make the services fewer but more generic? But will that make them too generic? Tough question.

5) Allow for Quick Wins for your Consumers
Create raving fans by helping them use your services and get them integrated quickly e.g. Provide XSL Templates to integrate quickly, or excel hook-ins, sample code etc.

6) Security
What about Authorization? How do you control which users can access what information? How will you define and maintain roles? With a small number of broad services how will you expose different service definitions to different end-users who can see only certain subsets? Then there's authentication, cryptography etc. Is SSL acceptable? Where are the WS-* security standards?

7) Integration styles
Again back to your users - what kinds of integration styles do you expect them to use e.g. Web 2.0 shops (a preference for REST, JSON) vs. Typical "Big Iron" shops (.Net, Java middle-tiers or COBOL/CICS -- both using SOAP)

8) Display Styles
Similarly does a Web-focus make sense (Ajax) or will most of your consumers be in-house where you they can build a Rich Client (e.g. C#, SWT, Swing)

9) Remember - you are exposing an API
Services you expose are basically APIs - so when designing services remember that all the rules that apply to APIs apply here also (see Josh Bloch's great presentation)

10) Remember and account for the assumptions of networks

• The network is reliable (handle transactions, recovery, retry, dupes etc.)
  
• Latency is zero (handle timeouts, recovery, retry etc.)
  
• Bandwidth is infinite (compression, tight XML schemas - attributes vs. elements)
  
• The network is secure
  
• Topology doesn't change
  
• There is one administrator
  
• Transport cost is zero
  
• The network is homogeneous
  

11) Performance and Scalability

Should you put a queue behind the service to insulate the underlying logic from getting too many requests? How can you load balance?

12) Availability

How do you provide HA and fail-over?

And I haven't even touched on monitoring, manageability, reliability at the service level or extensibility!

Grady's blog article has some other great things to consider - stateful vs. stateless services? How do you define the semantics of the data within the services?

You see SOA is no free ride - once you get beyond the first few quick wins there's a lot to consider or else you'll end up with the same mish-mash except this time it will be exposed more clearly to your end-customers!

Career advice for Java developers for the coming year

Well it would probably apply if it was 2001 as much as 2007 and it's this

"Focus on the data"

Good advice is often surprisingly simple - to some it's unsatisfactory. I believe this advice fits is in that category. It's like "Invest in Index funds" - brilliant investment advice for so many reasons (e.g. diversification, lower taxes, lower costs, avoid bad management) but often people expect a solution much more intricate and involved . . . . . but I digress.

I guess the main point is - by focusing on "the data" you are effectively focusing to a large extent on your customer and the guys with the $$$ paying the bills and your pay check . . . and that's not a bad thing.

Within "data" there is a lot of stuff in Java - let's start with how data is often stored and represented

Representation of data and accessing it

• RDBMS -- There's so much to SQL - the DDL and DML - and of course there's JDBC (CallableStatements, PreparedStatements etc.). Then there's the differences between Sybase, Oracle, MSS etc.
• XML -- Parsing it (DOM, SAX, XPP), formatting it (XSLT), accessing it (XPath), describing it (XML Schema)
  

Mapping of Data to and from Objects
Getting it out of the "persistence device" and into Java. That is:

• Object-Relational Mapping (e.g. Hibernate, iBatis, Toplink)
• XML-Object Mapping (e.g. Castor, Digester)

Transmission of data
Now we probably have to send that data to somewhere - partners, other business units, end customers etc.

• Web Services (JAX-WS and Axis) -- then of course there's SOAP and REST but let's not go there now :-)
• Messaging (JMS -- check out ActiveMQ)
  

Architectures

These are the hot architectures of the moment but they really put the emphasis on data and services to access that data.

• SOA (Service Oriented Architecture)
• ESB (Enterprise Service Bus)

I could also go into the display of data too but that's really getting a little too broad . . .

Either way, that's a lot of stuff - just to address business data-related functionality - but hopefully your career as a developer will be long and successful so you've got time to plug away at it. My advice - start with JDBC and XML as everything builds on that foundation. Know those already? Web Services and Messaging and so forth.

In the end we build most applications to access, update, manipulate and display data because that's what customers and employers will pay for. And if your "value add" as a developer aligns with the needs of your customer / employer then your career as a developer will be bright.

And yet as with all such apparently simple advice - the follow-through, persistence and effort - week-in and week-out is the hard part and it's what often distinguishes the best from the rest.

Big Bad Architectural Smells - #4 Feature X cannot be unit tested

I've heard this twice now in the last two weeks - "Feature X cannot be unit tested" or "Feature Y is not meant to be unit tested" - so to test it or test with it, you have to deploy the whole application in the container. Unfortunately my stuff completely relies on X as a major integration point and I want to unit test my stuff sitting on top of this.
(True I guess I'm writing more of an integration test than a unit test but still . . . .)

Frankly the situation is unacceptable and the developer should be embarrassed. Pretty much everything can be unit tested - it may take a lot of work in some circumstances, but it's doable. True for me, I could write a mock object but that's so much bloody extra work :-)

In the end I figured out a pretty easy way to Unit test both features (and, surprise, surprise, it wasn't that hard). What was *not* surprising for me was that the lack of unit testing in those features was somehow strangely correlated with a significant number of bugs also in those features - funny that! ;-)
Bugs that would have been found with - you guessed it - some simple unit tests!

Being unable to test a feature is a "bad smell" - things like testing database driven apps are hard, yes, but far from impossible and any time invested in helping developers unit test (e.g. mock objects or test database load / unload scripts) is very well worth it IMO.

For those looking beyond just plain old JUnit - I strongly recommend folks take a good look at extensions such as

• HttpUnit for unit testing web interfaces
• DbUnit for a ton of great services to help test database driven apps
• Cactus for container testing
• MockEJB for Mocking EJBs (kind of obvious eh?)
• EasyMock for plain old Mock objects

for nice integration with Eclipse I recommend EclEmma to give you code coverage reports to go with your unit tests (it is surprisingly well featured despite being only at version 0.1.8 at the moment) .

A few more Java Developer Questions

Well here they are - a few more of my favorite questions and again "No" I am not going to give the answers (that's what Google is for!)

J2SE
1) Does every try{} block have to be followed by a catch{} block?
2) Which do you prefer - checked or unchecked exceptions? Why?
3) When you are reading code looking for potential memory leaks, what kinds of things should you look for?
4) Say I have a JavaBean Class that has 10 fields and I have two instances A and B. Write some code to copy all the values from A to B? How do I make sure that A and B are the same?
5) Name some ways in which two Java threads (running on the same VM) can communicate

JDBC
1) In the Statement class, the executeUpdate() method returns an int. What does that int represent?
2) What is a SQLWarning, how is it different from a SQLException?

J2EE
1) True or false, the J2EE spec allows you to create your own Threads within the container? And in practice?
2) What files are needed to define an EJB (1.1 to 2.1)? How about in EJB 3.0?
3) In JMS messaging - what is the difference between "Point to Point" and "Publish Subscribe"?
4) What are the different ways of storing and using a user session in J2EE? Indicate the pros and cons of each method?

Struts Basics
1) Name the XML files needed for Struts to work?
2) What kinds of things go into the Web.xml for a Struts-application?
3) What are the different ways in which Struts support exception handling? Which would you choose and why?

Some Developer Interview Questions

Seeing as my "Favorite Java developer interview questions" entry was such a hit (read "controversial") I thought I'd follow-up with some more of questions I like to ask. I don't think these are nearly as contentious (I hope!) :-)

These however are more generic in nature - not specific to Java or any language/platform. Some may be for more senior developers than others (but I've had kids out of school be able to answer them too)

And once more - I'm not providing answers - hey just use Google ;-)

Basic OO Concepts
1) Define "Polymorphism"
2) Define "Encapsulation"
3) Define "coupling"? Why do we want "loose coupling" in our systems?
4) What is "cohesion"? Why do we want "high levels of cohesion" in the modules of our systems?

Code scalability / efficiency
1) In terms of Order notation, what is the best and worst complexity for "Bubble Sort"?
2) Now "Quick Sort"
3) Now "Heap Sort"

Puzzle Questions (yes I'm starting to ask a few of these - but very software related)
1) You have an array of 10 billion objects, how would you search it most efficiently?
2) How would you implement an LRU Cache in your favorite language?

Miscellaneous
1) What process do you follow for performance tuning of an application?
2) What is "refactoring"? (Note: I'm shocked how few people know this.)
3) Name for me your top 5 practices to deliver high-quality applications on time and on budget.

Don't worry I'm not giving away my "game" - if you get past each of these there's a myriad more drill-down questions to verify you really know it. Some are also pretty "open" questions.

What is Software Architecture?

It's what software architects do / create / make! :-)

Ha! No I can't take that cop out - well at least not here.

But still . . . . we've asked probably the hardest question in the world of software development.

Why? Because no two people can seem to agree. But I think most of us who've been in the industry can say "I know it when I see it" but it's another unsatisfying answer. (Such issues also exist in the legal world http://en.wikipedia.org/wiki/Potter_Stewart)

Why is it so difficult - well think of it this way- Code is not architecture although code is the ultimate expression of the architecture. Requirements are not architecture but they help lead you to the architecture. Design patterns are not architecture but they are building blocks towards the architecture. So what is it?

Well in this blog entry I'll try to bring together different views of what constitutes software architecture. Some I have problems with as they're quite incomplete definitions IMO. Others are on the right track . . .

1) Architecture as Representation.
Wikipedia Definition - See http://en.wikipedia.org/wiki/Software_architecture

"The software architecture of a system consists of software components, their external properties, and their relationships with one another"

First of all Software Architecture is not just a a set of "relationships " - true it can be *expressed* as that but it's so much more than that - there's no talk of the end goal or the rationale for the architecture - a system to be used by people to address a specific goal. The wiki talks about "views" (as in the 4+1 model) but that is really just a slight hint that an architecture has different stakeholders - users (functional view), developers (code view, development view etc), but what also about managers?

2) IEEE Definition - Architecture as Organization with Principles in an Environment

"Architecture is the fundamental organization of a system embodied in its components, their relationships to each other, and to the environment, and the principles guiding its design and evolution. "
"A system is a collection of components organized to accomplish a specific function or set of functions. The term system encompasses individual applications, systems in the traditional sense, subsystems, systems of systems, product lines, product families, whole enterprises, and other aggregations of interest. A system exists to fulfill one or more missions in its environment. "

But this seems to separate the system itself from the "fundamental organization" (i.e. the architecture). But in many ways, a system's architecture is inseparable from the system itself - change the system you change the architecture. Change the architecture and you change the system.

3) Bredemeyer Consulting
http://www.bredemeyer.com/whatis.htm

Too many quotes to list so I suggest you click the link. But these guys are getting very close - they don't just give a blanket single sentence definition. But the idea of components and their interactions (dynamic and static aspects) with different views (users, managers, developers etc.) is pretty darn good. The concept of "decomposability" is emphasized here.
Also they point out that a "vision" is part of an architecture also - as an architecture, at least to my mind, is a growing entity since software is never fully "done". The vision should encapsulate some aspects of how the architecture will evolve over time.

4) IBM Developerworks
The following link is to a great summary from which I have borrowed the quotes that follow
http://www-128.ibm.com/developerworks/rational/library/feb06/eeles/index.html

"An architecture is the set of significant decisions about the organization of a software system, the selection of structural elements and their interfaces by which the system is composed, together with their behavior as specified in the collaborations among those elements, the composition of these elements into progressively larger subsystems, and the architectural style that guides this organization -- these elements and their interfaces, their collaborations, and their composition." [Kruchten]

"The software architecture of a program or computing system is the structure or structures of the system, which comprise software elements, the externally visible properties of those elements, and the relationships among them." [Bass et al.]

"[Architecture is] the organizational structure and associated behavior of a system. An architecture can be recursively decomposed into parts that interact through interfaces, relationships that connect parts, and constraints for assembling parts. Parts that interact through interfaces include classes, components and subsystems." [UML 1.5]

"The software architecture of a system or a collection of systems consists of all the important design decisions about the software structures and the interactions between those structures that comprise the systems. The design decisions support a desired set of qualities that the system should support to be successful. The design decisions provide a conceptual basis for system development, support, and maintenance." [McGovern]

In this summary article Eeles does some great analysis and points out some really great things:

• Firstly all architectures should emanate from some rationale "An important aspect of an architecture is not just the end result, the architecture itself, but the rationale for why it is the way it is. Thus, an important consideration is to ensure that you document the decisions that have led to this architecture and the rationale for those decisions."
• Architectures have an experiential and style elements "Most architectures are derived from systems that share a similar set of concerns. This similarity can be described as an architectural style, which can be thought of as a particular kind of pattern, albeit an often complex and composite pattern (a number of patterns applied together). Like a pattern, an architectural style represents a codification of experience"
• The architecture influences the development team "An architecture influences team structure" and vice-versa "the current team structure and the skills available represent a very real constraint on what is possible". Unmentioned though are other influences of the other resources - time and money
• Also he points out a very important assumption we've all made "An architecture has a particular scope" that is there are software architectures, business architectures, information architectures, hardware architectures etc. which also goes back to the point about views but here not necessarily for the sake of stakeholders - different levels of description.

5) Grady Booch

http://www.booch.com/architecture/index.jsp

"At the highest level of abstraction, every system has an architecture, encompassing the key abstractions and mechanisms that define that system's structure and behavior as seen from the perspective of different stakeholders, each with a different set of concerns"

This is probably the closes to a great one sentence definition I've seen - discussing structure, behavior, viewpoints and the necessary abstractions.

6) SEI Definitions

So even the Software Engineering Institute (SEI) at Carnegie Mellon effectively punts on the definition -http://www.sei.cmu.edu/architecture/definitions.html

Summary

So many definitions - so many aspects and qualities of the architecture, so many constraints and influences on the architecture and constraints/influences emanating from the architecture.

Can you point to a software architecture - with so many viewpoints (stakeholder and scope) the answer is "Yes" and "No"? The architecture is everywhere in the system and yet nowhere. How very Zen / Buddhist.

Will the answers get much better than this? I'm not sure - as the field of software (engineering?) advances I hope so, but there is something poetic about it's current fungibility.

Conflict: Architects vs. Project Managers

Just the other day I was thinking of some things to blog about that were fresh and new and it suddenly hit me that I've never really discussed project management related issues.

But what hit me harder (quite a smack on the head I must say), was what I subsequently realized -- very few of the software / architecture / java blogs I read talk about project management or related concepts? Now I'm talking not just about developer blogs but folks who are tech leads or architects, CIOs and CTOs!

Oh yes we talk about XP and other agile methods but it's all very "developer-centric".
That is, you don't see the following words very much
- "Project Plan"
- "Critical Path"
- "Dependencies & Sequencing "
- "Estimation & Budgeting"
- "Resource Planning"
- "Staffing"
- "Team Development"
- "Scope"
- "Risks"

Are we saying these things aren't relevant or aren't important? If these aren't the words we use when talking to our various project managers they sure as heck are the words THEY use to talk to their bosses and their bosses' bosses! Yeah - the guys with the $$$.

If Architect's don't think what Managers do is all that important, then we've got some conflict brewing. Nothing overt mind you, but lack of common goals and lack of common interests don't lead to the most productive professional relationships.

But how is it we can share the same tacit goals (timely completion, full functionality, happy users etc.) and not really be on the same page? True to an extent, project management and architecture/design are orthogonal issues - along different dimensions.

Also generally the PM, I've found, relies a lot on the architect for guidance and help in things like risk identification / mitigation and validating work estimates. But where do Architects rely on PMs? I have to say, in my experience, we don't - and that doesn't sit right. We can't be as successful as we want (need?) to be if there's a one-way dependency.

Perhaps that's why I've often heard managers express frustration about their technical leadership - if/when you're a PM and your project's success of failure (and thus your professional success/failure) depend on an architect and tech team that just doesn't speak your language and doesn't understand concepts such as how blown estimates really affect project plans -- well frustration must be expected!

True - frustration goes the other way too. Software is still more "art" than "engineering" or "science" and estimates can be way off for quite valid and reasonable reasons (e.g. a bug in the VM can be very expensive to work around). Being expected to meet deadlines when such things happen can be tough to swallow.

And perhaps that's the problem - we're applying project management techniques which we're perfected in the 40s, 50s and 60s when mass industrialization and mass manufacturing took off. But software is not nearly as mature now as those industries were then. Linear system thinking just doesn't apply nearly as neatly to the realm of software development (yet!).

Let's call this the "Management-Technical Impedance mismatch" in honor of two other things that really don't go well together but need to (Objects and Relational DBs)

That said, this mismatch doesn't absolve us from trying to bridge the gap and looking for the synergies. Technical folks need to acknowledge that these project management concepts, and what they represent, are important and ever present in modern business.

The communication needs to go both ways, but where to start? Take two books and call me in the morning! The Mythical Man Month is unparalleled and unmatched for elucidating some of the problems with some great stories. For a more prescriptive and more modern approach there's Steve McConnell's Rapid Development - just an incredible book.

Get copies for everyone on your team and see what happens! Anything that gives us a common frame of reference, with things we can agree upon is a good place to start.

2007 Goals

By now you're probably sick and tired of reading, watching or hearing 2006 retrospectives and 2007 predictions. Of course there's always the perennial (forgive the pun) list of resolutions or goals for the new year.

And well I'm no different here (sorry!). Hopefully though by making my goals public there will be a sense of obligation that results in me sticking to the goals.

1) Learning more about the Java Language
I want to learn more about enums - they seem a nice promising addition but just how do they relate to objects and classes?

2) Writing Better Code
One area I'm less than happy with myself on is Exception Handling. Oh I can already do pretty well but the problem is I'm not always consistent in how I handle them. Sometimes I choose to wrap an exception, sometimes I just throw them in the method declaration, and sometimes I just log them. I've got to come up with a set of guidelines for myself, try them, change them if needed and then try to stick by them.

3) Reading
You'll often find me ranting on about Unit tests and, I have to say, that I still find them hard to write and write well, especially when there's a database involved. Should I load the DB with test data? How do I tell the test which database to point to (property file? VM args?)? Should I use mock objects?

Anyway I definitely want to find time to sit down and read JUnit Recipes from Manning press. I drool over it every time I'm at the bookstore (sorry for the imagery).

From less of a technology point of view I also want to read more of the writings of Peter Drucker. From what I've already read from him, he was a man well before his time with great insights into the nature of business and the emergence and important of the "knowledge worker".

I really should also pick up a copy of Securities Operations: A Guide to Trade and Position Management - I think it's a great idea for architects not just to stay current on technology but also to understand as much about the domain in which their application and users reside. For me that's the financial world of securities trading, compliance, mutual funds etc.

Then there's all the articles I end up reading either directed there from mailing lists or
blogs I read.

4) Writing
After re-reading some of my earlier blog entries, I realize my punctuation could do with a kick in the pants - too few commas and too many exclamation marks!!!!

5) Areas needing development
On the Myers Briggs Personality Type tests I show up as an INTJ and, as the last link makes clear, the small little interpersonal details that contribute to marketing oneself and to "networking" are not exactly the strength of the INTJ, but they're critical to one's success. It's also critical in being able to lead others.

(Normally I would be against pigeon holing anyone, least of all myself, but after taking the MBTI exam and reading about INTJs I've got to say it was really very accurate and also very insightful. As the famous psychologist David Keirsey put it " a word which captures the essence of INTJs is builder - a builder of systems")

Conclusion
Hopefully this time next year I can look back and see how well I've done against these goals.
Anyway, thanks for your patience, normal (not-so-self-serving) service will now resume . . .