7/28/11

The Technology Hype Cycle and why you should bet your Software Career on the Cloud

You'd have to be living under a rock not to be well aware of the hype cycle in technology. Here is a list of hype cycles I've personally lived through and been affected by. I've tried to identify the year where expectations were either rising the fastest or reached their peak

1999 - Y2K
2000 - Y2K (Yes it was still pretty hot)
2001 - Java / J2EE
2002 - Linux (also Enterprise Application Integration was hot here too)
2004 - Open Source & Web Services & BPEL
2007 - SaaS
2008 - Web 2.0 and Mashups

I'm probably off by a year or two in some cases but as you can see each and every year has something new.

Of all of those hyped technologies very few of them delivered on the (marketing) promise. However I think some of them really were a net positive. On a scale of "mostly helpful" to "mostly hype" here are the highlights:

1) Open Source and Linux definitely helped reduced a small chunk of development costs. The problem is most of the costs in development are people - that said any reduction was useful. As a (Java) developer my first preference in most technology choices is Open Source because most of these products "just work" and there is enough documentation and forum help to dig you out of most holes.

2) Java was probably a net drag initially (especially J2EE in its first few incarnations) but after JavaEE 5 things are getting better.

3) SOA was a slight evolution (XML + HTTP?) however if not for the advent of REST, SOA would reside more towards hype than helpful as REST dispelled the myth (pushed by consultants and software firms) that we needed a bunch of heavyweight WS-* protocols to make SOA viable.

4) Virtualization was definitely a net positive for IT organizations and data centers - definitely more helpful than hype saving organizations lots of money.

So what about the cloud, cloud is getting hyped a lot lately. A lot of people don't fully understand it so this may surprise people: the advent of the Cloud is, in my opinion, the biggest event in software development since Tim Berners-Lee proposed the world wide web .

Take that in for a minute. I am willing to bet my public reputation that the Cloud is going to be a HUGE sea change for IT organizations and for developers. Bigger than anything in the past 12+ years.

As a developer if you are not getting on board with this you are gonna be left behind. This is like being a N-Tier C++ programmer in 1994 as the web was starting to gain traction. Get in with the cloud NOW!

Why? Cloud solutions (computing, storage etc.) allows companies (especially large ones) to reduce and perhaps eliminate many of their data centers and supporting IT staff. For smaller companies the "Pay as you go" model (converting CapEx costs to OpEx) and the "elasticity" support to allow you to scale-on-demand are exciting. Venture Capital companies are now demanding the companies they invest in to use the cloud - it helps to reduce costs now and scale more later without large outlays. So who is NOT going to get on board this trend? I can't think of any organization that cares about costs / growth who will not ultimately want to adopt this model.

Also as the number of devices per person (Tablets, laptops, smartphones) continues to accelerate the "elasticity" demands will become ever more prevalent for large enterprises as well as Web 2.0 companies. True you can't use more than one device at a time but devices are getting smarter and creating work themselves (to keep your email, rss feeds, facebook status all up to date etc.).

This is like when Electricity distribution took off with the advent of AC (alternating current) it was critical for economic growth. True the Cloud may take 3 or 4 years to REALLY take off (1994 in the web wasn't nearly as exciting as 1999) but when it does . . . .

And when stuff is hot (like cloud is, or is going to be) the pay goes up up up and the firms that drive value in this space are going to be worth billions.

There are a number of Cloud providers and it's growing almost daily - but if you want to get on board I suggest you start taking a look at Amazon EC2 and Amazon's Elastic Beanstalk (for Java folks). Back in the nineties they used to say "No one was ever fired for choosing IBM" - not that IBM was flawless but it was hard to argue they weren't top of their game for Enterprise software (back then). Today I think the same is true of Amazon - not perfect but compared to the other providers they are clearly on top of their game.

Getting knowledge of AWS and understanding the very different architecture, design and coding strategies (e.g. security, privacy, failover concerns and NoSQL, Hadoop etc.) will help you transfer more quickly over to whatever is "next" when invariably Amazon gets surpassed by someone with a better mousetrap.

I still have a few doubts. Invariably there will some mini-Bubble in cloud company valuations - in the hype cycle there will be a "Peak of expectations" and a "Trough of disillusionment", but in the end the "Plateau of productivity" is going to be a very good (but very different) place to be. And when a large wave of change is coming - it's better to be ready with a surfboard and paddling ahead of the wave, than sitting on the beach waiting for it to come ashore.

7/13/11

Scrum Purity Test

OK so just for fun (malicious fun!) I decided to create a Scrum purity test. The reason is I am pretty sure a LOT of people out there SAY they are using Scrum but aren't really - or they aren't using all the tools and techniques involved. Fair to say, I have never fully utilized Scrum in any place I have worked - mostly for political reasons - usually because of lack of buy-in from Senior Management. That said, we've made it work and I'm sure others have too.

But let me say that I don't think you HAVE to adopt Scrum 100% to get value from it. Small iterations with defined deliverables and constant communication and accountability can go a long way regardless of what you call it.

Anyway here is my take on a "Scrum Purity Test". There is another take here.

FIRST AND FOREMOST
Q: Do you have Sprints at the end of which you deliver (roughly!) shippable functionality?
- No - Then don't do this test - you've already lost. I don't think you can consider yourself agile without small iterations.
- Yes, but the Sprints are 4 weeks long and the code has lots of bugs - zero points
- Yes, the Sprints are under 3 weeks and the code has some bugs - 5 points
- Yes, the Sprints are under 2 weeks and the code is tested (but has some small issues) - 10 points
- Yes, the Sprints are under 2 weeks and we release to production - 20 points

MEETINGS
Q: Do you have a Daily Scrum Standup meeting?
- No - zero points
- Yes but it goes over 15 minutes - 5 points
- Yes, it's under 15 minutes and we answer the three questions - 10 points

Q: Do you have a Sprint Planning meeting?
- No - zero points
- Yes kind of, it's kind of ad hoc - 5 points
- Yes Ken Schwaber would be proud - 10 points

Q: Do you have a Sprint review meeting at the end of a Sprint?
- No - zero points
- Yes - 5 points

Q: Do you have a Sprint Retrospective to identify what can be improved?
- No - zero points
- Yes - 5 points

ARTIFACTS
Q: Does your team maintain a Product backlog?
- No - zero points
- Yes, kind of, it's ad hoc and it lacks priority- 5 points
- Yes its prioritized, updated regularly, visible to all - 10 points

Q: Does your team maintain a Sprint Backlog?
- No - zero points
- Yes, but features aren't broken into tasks - 5 points
- Yes, features are broken down and well understood before beginning - 10 points

Q: Does your team maintain a Burndown chart
- No - zero points
- Yes, kind of, we track it roughly in an ad hoc fashion - 5 points
- Yes, we use some tools such as Greenhopper or Excel and its visible to all 24 x 7 - 10 points


ROLES
Q: Does your team have a Scrummaster?
- No and we could use one - zero points
- No we don't need the protection / impediment removal (e.g. in a startup) - 5 points
- Yes we have one and we also report to her (e.g. a Project Manager) - 5 points
- Yes we have one and it's not a person we report to - 10 points

Q: Does your team have a Product Owner?
- No - zero points
- Yes but they really don't understand the user - 5 points
- Yes and they know our user base inside and out - 10 points


SCRUM-BUTS
Q: Does your team do Estimates?
- No - zero points
- Yes and we estimate using hours and days using MS Project - 1 point
- Yes and we use planning poker and our estimates are within 15-30% - 5 points
- Yes, we use planning poker and our estimates are within 10% - 10 points

Q: What are your Specifications like?
- Big requirements docs - zero points
- Good user stories - 5 points
- Just in time requirements / stories - 10 points

Q: How is testing performed?
- We have No dedicated testers on team - zero points
- Developers Unit test only - points
- The feature is tested soon after Sprint is complete - 3 points
- Software passes acceptance testing as Sprint ends - 8 points
- Software was tested and is deployed to Production - 10 points


Now what is your total - out of 130 points? What grade would you give yourself on the following scale?

110 - 130: A - you are doing a great job following Scrum (just remember to make sure your users and stake holders are happy)
90 - 110: B
70 - 90 : C
50 - 70 : D
< 50: F

I have to say I thought we were doing pretty good but when I scored my team we got a 59 a D! Yeouch! That said we ship on time and everyone is pretty happy but clearly we are not doing great following Scrum.

What do you think? Are there any critical aspects of Scrum that I missed? Any items that deserve more points (or less)? I am being unfair?

POSTSCRIPT
Some folks have been kind enough to get me searching for "the Nokia test" which has thrown up the following

3/10/11

2-Way SSL with Java: The Keystore Strikes Back - Part 1

You'd think with how prevalent SSL/TLS on the web that SSL with Java is easy, well it's not. In addition to having to get to grips with understanding how private/public key, certificates work there's all the networking that goes on in Java.

If you start off trying to get to grips with the in-built Java Secure Sockets Extension you're gonna be stunned by the complexity of it all.



Clear as mud eh? The JSSE guide alone is 73 pages long - but I hate to say that its REALLY a must read. Print it out (double-sided to save some trees) grab a Venti decaff or two and wrestle the information into your brain.

Thankfully Jakarta Commons HTTP client makes it relatively easy code-wise (Side Note: Why the heck does HTTP Components v4.0 seem so much more complicated than v3.0? The v4 docs seems much less clear!). For example:

HttpClient httpclient = new HttpClient();
GetMethod httpget = new GetMethod("https://www.bob.com/");
try {
httpclient.executeMethod(httpget);
System.out.println(httpget.getStatusLine());
}
finally {
httpget.releaseConnection();
}

In any event it's pretty straightforward code-wise with HTTP Client until you have to start to get to grips with three funky things - they are
1) the keystore
2) the truststore
3) keytool to access both

Truststore
Lets start off with a truststore. In a normal 1-Way SSL situation, say on a browser, you need your bank's server to authenticate itself to assure you that you are hitting the REAL bank server and not some spoofed site. So in the 1-Way SSL case the bank server sends your browser a certificate signed by a CA(Certificate Authority) then your browser compares that CA signature against an in-built list of CAs. In a similar way your truststore is your secure store of trusted CA entries or self-signed certs provided from third parties you trust. So essentially the truststore contains the certificates you use to authenticate other servers / processes. Every Java install comes with it's own truststore located in $JAVA_HOME/lib/security/cacerts

Keystore
A keystore is, as you can imagine, is a secure store for keys used in the SSL protocol. Essentially it's where you will put private keys and their associated certificates used to authenticate YOU as client to a server. Typically in a normal web browser transaction you use 1-Way SSL to authenticate the server then you use a login/password combo to authenticate you. Well in B2B systems you can use 2-Way SSL where client and server authenticate each other - and this is where the keystore is critical to authenticate your Java program, the client, to the server.

Typically for client authenticate you create either a self-signed or you purchase a CA signed cert. You store your private key and public cert in the keystore and provide your B2B partner with the self-signed cert - or in the case of a CA-signed cert typically your partner's truststore will trust that CA and you are good to go.

keytool
keytool is the command line tool to maintain these two beasts. And let's just say it's not easy to use keytool. It's like snowboarding - its good when you're used to it but getting started with keytool you're gonna spend a lot of time on your ass or face first! If you're just getting started with keystores and truststores I strongly recommend using Portecle - an open source GUI tool which makes keytool obsolete and should really ship with Java someday.

Then you have to tell java where they keystore and truststore are and the passwords to these stores. Well you can do that with System properties e.g.
 -Djavax.net.ssl.keyStore=keystore
-Djavax.net.ssl.keyStorePassword=password
-Djavax.net.ssl.trustStore=truststore
-Djavax.net.ssl.trustStorePassword=trustword

If standards are good, more is better right?
Getting tired yet? Well this is gonna put you over the edge . . . . when you get keys and certs from your B2B partners how many types of files do you think there are - two? three? four? Let's list the ones I've had to deal with lately.
  1. pfx
  2. pem
  3. p12
  4. cer
  5. crt
  6. der
It's madness! If you want a good description overview of the file formats then this link is a good place to start. This is when openssl becomes your friend when you need to convert from different file formats into formats compatible with keytool - oh and good luck finding documentation on which file formats keytool takes!

In any event so far things are tough but not insurmountable - well so I thought until two B2B partners decided to provide me each with a key for my keystore to authenticate me. A key of the same type (RSA), signed by the same CA. Then all hell broke loose - cos Java don't support that without some major surgery . . . . and that's the topic of Part II.

1/7/11

Migrating from JBoss 4 to JBoss 5

[ Yes I know it has been a while - two years, two very busy years but hey I'm back - for now ]

Anyway I have been using JBoss 4.2.3 for over a year right now and really like it (although the clustering / JMS stuff seems WAY overcomplicated - and needing 80 config files - not fun!). But anyway it is time to upgrade to JBoss 5 and the path has been marred by many stops and starts and has been surprisingly difficult.

In any event I found the following links to be a life saver and thought I would share them

The real kickers have to be
1) Increased adherence to the Java spec that cause WARs / JARs to no longer deploy
2) Changed location of XML files
3) Changed XML filenames
4) Changed XML file contents

Man they don't make it easy do they?

12/9/08

How to Motivate Developers - A three step framework

No . . . . not THOSE three steps!

Anyway, now that we're approaching the end of the calendar year, invariably many folks are filling out self-evaluations and managers are presiding over year-end reviews, and choosing raises and bonuses for their staff.

Managers should also be considering the year ahead and one of the most important aspects - developer productivity. A key element of productivity is motivation and this article tries to answer the question "How to motivate developers".

But first some comments on prior work . . . .

There have been quite a few blog articles written by others on this topic (see some references below). They're pretty solid but they're mostly written from the perspective of
1) Developers or
2) Managers who don't have a major technical background.

The problem with #1 is that self-reporting of what a person wants is very unreliable. Often people don't really know what they want. Or they know what they want but don't truly understand how it will make them feel or for how long (see this really great article on happiness for some more insights)

The issue with #2 is that, without knowing the perspective of the other side (developers understanding what it means to be a manager and vice-versa) the full picture is often lacking.

One thing I noticed in my research for this post is that many of the articles below try to ask "What do developer's want?". That's important but a bit misleading. For example take a more common situation - housework. I don't want to do the dishes - I'd rather be playing Call of Duty 4 - but I *have* to. For many reasons - the sanitary one is obvious - the not so obvious one . . . . my wife will kill me otherwise :-) So what's my motivation - good health and staying alive :-) What motivates me isn't what I want.

There's also another aspect to this that I feel people haven't considered. I am going to venture (without citation . . . I couldn't find one) that the majority of a person's motivation is intrinsically determined. That is - their motivation level is largely driven by who they are, their personalities and things in their lives you have no control over.

So please don't get the idea from this that I believe one can control 100% of a developer's motivation. You can influence it, but only so much. In many respects motivation should be a key aspect when interviewing new team members - because once they're hired you can only impact a relatively small aspect of their motivational make-up. Again that's based on experience - I'm not aware of any scientific studies to confirm this hypothesis.

To repeat, I feel that, to a large extent, people are already largely motivated or not. You can improve things quite a bit though. Say you could make a developer 5-10% more effective - that's like having an extra 2 weeks to a month of their development time each year.

That's a lot of extra bugs fixed or new features produced. Do that across a team of 6-8 developers and you can make a very noticeable improvement.


Anyway after thinking about it - from both a managerial and developer perspective, here's what I've come up with for my framework.

STEP 1) Compensate Well
This is one of those "necessary but not sufficient" conditions you often hear about. You can address every other motivational item but if you don't pay well and give good benefits you're shooting yourself in the foot and worst of all people are going to go elsewhere. Probably most of all you're just not going to be able to attract and retain really good (i.e. already motivated) developers.

If you think of the famous Maslow's Hierarchy of needs, this is one of those fundamental "lower" or more basic needs.



STEP 2) Remove or reduce demotivating aspects
You can pay all the money in the world to people but if you have non-existent specs, unrealistic deadlines, micro-manage etc. people aren't going to be motivated for long and they will quickly become unmotivated or leave.

As I just stated there are several things that many developers experience that drive them batty.
i) Poor requirements - there's nothing worse than building junk but consequently (for me at least) there's nothing better than great (detailed, unambiguous) requirements and the resulting happy users.

ii) Unrealistic deadlines - whether this results in 90 hour+ weeks or buggy software - this is very much a drain on developers, their ego and sense of pride. Developers want to be proud of their work and go home and see their families. Is that too much to ask?

Yes everyone has crunch times and there's invariably some extra unforeseen work but hopefully it's infrequent and as a manager hopefully you don't have to "go to the bank" too often.

iii) Too many meetings - I've found that the more formal a place is (or the less friendly it is) the more meetings you need to just get people to talk to each other. But many meetings are a bad sign - ideally developers would/should be able to talk to each other and meetings should be "organic" and not forced.

iv) Disrespectful or uncooperative colleagues. "Bad apples" can ruin a whole batch. The problem is exacerbated when management is unaware of them or worse, is aware, but refuses to do what it takes to remedy the situation.

v) Processes, Tools or Techniques that slow you down e.g. slow builds, slow unit tests, slow desktops etc. They suck productivity and reduce enthusiasm.

vi) A poor work environment e.g. a place where it's too loud, there's no privacy, you get lots of interruptions etc.. A bad environment can destroy motivation or at the very least be a distraction. And distractions are TERRIBLE for a developer's focus on the problem at hand.

Either way, my recommendation is that before focusing on MOTIVATING factors, I would focus on removal or amelioration of demotivating factors. There's also some evidence that there is some psychological difference between demotivating aspects and motivational factors.

The good thing is by focusing on these e.g. improving requirements, reducing unnecessary meetings, speeding up processes etc. you are also improving team productivity even if there's no motivational boost to be had. But that's unlikely - removing some of those impediments should buy you some good will - but frequently that fades over time. Either way, for the productivity boost alone that's makes good business sense.

STEP 3) Focus on positive motivational aspects.

Here are the things that you really are hoping will translate into more motivated (and more productive and happy) developers

i) Support Developer Learning
Developers love to learn - new languages, new tools, new frameworks. Spring, Ajax, Hibernate, Ruby - that gets their juices flowing. Now the challenge is how do you direct that to produce the product you're paid to . . . . . the key here is to align the new item with key gaps needing filling. Need an Object-Relational Mapping system - Hibernate! Need a rapid prototyping language - Ruby! etc. You also need to factor in the productivity dip as they learn.

ii) Provide Challenges & Problem Solving Opportunities
Developers in general love to solve problems - whether it's how to integrate new features into a legacy system, how best to compare two lists of strings or how to make a system more scalable developers love such problems.

In my experience however there are sub-types - different developers who prefer different kinds of challenges - ideally again you would align the challenge to the person who wants it!

iii) Provide Feedback
Developers want to learn and grow - beyond the technical learning / challenge they also need feedback in other ways to help them grow as individuals and adapt to their organization. Often I find this is where management falls down. Who could blame them / us - we often say we're open to hearing about areas needing improving but not so good in practice. I've found the Sandwich Feedback Technique works quite well though.

iv) Other Ideas
See the section below for other really good ideas from other articles

These are all the things that "developers" want, in the sense of the higher "self actualizing" levels of Maslow's hierarchy. Again, these factors should not be detached from "conditions on the ground" as it were.

For example I've known a few developer's whose idea of self-actualizing was deciding what to eat for lunch that day. They didn't want to learn ANYTHING new - no new languages, no new environments - they know what they have and don't want to change. Their motivation isn't going to take a boost from being asked to learn something - on the contrary.

STEP 4) The Secret Step - Fear as motivator
OK I said there were three steps, but here's that "magic" 4th step. The "nuclear option". Go here only when you must e.g. you or someone on your team might be fired unless some action is taken, a client could lose a lot of money etc.

As with the "doing the dishes" analogy, there comes a point when you have to do what you don't want to do. You do it because the ramifications of not doing it are worse - MUCH worse.

Often if you're seen as the "good cop", you need a "bad cop" to make this work. Sometimes you might need to be the "bad cop". Most folks know the analogy having seen it many times on the various CSI or Law & Order shows on TV.

Basically the line here is "Do this or you'll lose your bonus / won't get a raise / be fired".

As I said you can't rely on using this too often, but you need to be sure also that people know the chance is there. Developers are smart people - if you don't follow through on what you promise you lose credibility. This lesson on follow-through was something that my two year old taught me. "If you don't get down from that chair, you'll get a timeout". Guess what, unless you start handing out "timeouts" they just will continue to stand on that chair.

Again you don't want to do it too often and when you do you need to explain the reasons, it can't be seen to be arbitrary. For example - fire too many people, and people will start updating their resumes and looking elsewhere rather than live in fear. Don't fire anybody and you send the message that there are no consequences and people want to leave because they tire of supporting the slackers. It's a fine line but most of your strong developers already can identify who isn't pulling their weight and will thank you for pulling the trigger and finding someone better.

Anyway there's my thoughts on developer motivation - I look forward to your comments.


REFERENCES TO RELATED WORK

Nine things developers want more than money
1) Being Set up to Succeed
- Good requirements
- Realistic Deadlines
"Being forced to build crap is one of the worst things you can do to a craftsman"
2) Having Excellent Management
3) Learning New Things
4) Exercising Creativity and Solving the right kind of problems
5) Having a voice
"When a developer speaks, someone should listen. When several developers are saying the same thing, someone should listen and act . . .quickly"
6) Being Recognized for Hard Work
7) Building Something that Matters
8) Building Software without an Act of Congress
9) Having Few Legacy Constraints

Top 10 ways to motivate geeks
1) Geeks are curious. Let them feed their desire to learn things
2) Geeks like to be self-sustaining. Let them figure things out on their own.
3) Geeks are creative even if they don't know it. Give them a chance.
4) Geeks need tools, good ones. Give them more than they need.
5) Private, yet collaborative. Geeks need to be left alone, but not too alone.
6) Free stuff. T-shirts, food, desktop widgets, whatever.
7) Control
8) Geeks need recognition
9) Freedom
10) Compensation - Saved this for last, but geeks gotta live too

Motivating Software Developers
1) Allow me to focus
2) Allow me to feel that I am making progress
3) Allow me to make something I can take pride in
4) Allow me to do it for me, not for the money
5) Allow me to work on interesting problems
6) Allow me to be part of a team
7) Allow my ideas to be taken seriously

What motivates software developers?
1) Introduce new technologies and techniques
2) Practice pair programming to encourage communication, sharing of skills and team building
3) Encourage participation in community developer events (user groups, code camps), blogs (share links across the team), books (monthly bookshelf anyone?) and conferences.
4) Avoid generalized training - in my opinion this tends to serve the paycheck programmer more than the dedicated ones.
5) Interesting projects
6) Satisfy your customer

Motivating Software Developers


To Motivate or not to demotivate
"Only taking away the things that make people dissatisfied, will simply result in people having neutral feelings towards their jobs."

Some programming principles: People

Managing and Motivating Developers: Tips for Management Cluefulness

What makes people tick - 10 motivation factors in the workplace


"The first thing to note here is that most people are motivated by:
1. doing interesting stuff
2. feeling recognized and appreciated
3. making an impact"

What motivates programmers?

11 ways to motivate geeks

Five things your manager could be doing better