Web Directions 2015 – Day #2 Likes & Takeaways

This is part two of my series of articles on the Web Directions conference. Part one was a recap of Day #1, this post is about Day #2.

Unfortunately in Day #2, I have been less enthusiastic about the sessions I attended (why didn’t I go to The Two Pillars of JavaScript by Eric Elliot??). Once again I did float between engineering and design, and definitely got the best and the worse of both worlds. Once again I’ll only talk about the talks I found valuable.

Hannah Donovan (Drip) – Souls & Machines
Patrick Hamann (Financial Times) – Embracing the networks
Martin Charlier (Rain Cloud) – Designing connected products
Vitaly Freedman (Smashing Magazine) – Smart Responsive Development

 


 Souls & Machines by Hannah Donovan (@han)

Hannah is a freelance designer who mainly worked in the Music industry at places such as LastFM.com, JAM, MTV, and more recently Drip. Hannah is chasing the interception between product & content: she designs for desires and not needs.

Hannah started with a personal story of when she blind-watched the Blade Runner movie on a rainy Sunday afternoon in 2008. She had absolutely no idea what the movie was about, so she had to stop watching after 10 minutes as she felt completely overwhelmed by the story and graphics. She uses this story as a personal example of how content without context is confusing, and content without perspective has no feeling.

The debate of humans vs computers has been going on for year (e.g. at South By South West). While computers connect us in ways we cannot, human beings definitely connect us is ways computers cannot. For instance the best music recommendations come from friends. With music it is important to provide this connection with people and with time. Only humans also can provide perspective through emotions, and it is important to have a strong point of view when creating an experience.

Nike Town store in NYC

Nike Town store in NYC

Hannah also made her talk very relevant and poignant by showing us how the digital era has affected the music industry, as a proof that the value of content is diminishing. Hence the need to design for a “post-content” world, i.e. design for the space around content by providing, context, connection, and perspective.

Screen Shot 2015-11-22 at 11.39.05 am

The decline of paid music in the USA

 


 Embracing The Networks by Patrick Hamann (@patrickhamann) (Slides)

Patrick’s talk covers techniques he applied while developing high-speed and resilient websites for The Guardian and The Financial Times.

Our web applications are not living up to their contract due to the variances in network as well as trying to shoehorn desktop experiences to mobile. As we have no control on the distribution over the network, Patrick reminds us of The 8 Fallacies of Distributed Computing (Peter Deutsch – Sun Microsystems 1994) as well as Sam Newman’s quote from Building Microservices:

Unreliability of networks forces people to think about system with failure in mind.

Indeed, as websites have grown very large, failure is unavoidable. Below is a picture of the requests made by the ft.com website using this really cool app:

Screen Shot 2015-12-07 at 8.34.07 pm

Patrick provides insight into how to monitor and test failures.

  • He describes Single Point of Failures (SPOFs) as a part of your website which, if failing, will stop the entire site from rendering.
  • Slow or no access to Javascript files is usually the main source of problems with modern websites. It is really important to develop and test your website under no-js conditions by disabling JavaScript to mimic unreliable network connections.
  • Facebook, who relentlessly performance tests its apps for mobile devices, has created 2G Tuesdays where all employees are exposed to a low cellular bandwidth.
  • Flaky network connections can also be simulated with a library like comcast.
  • File load times can be measured with webpagetest.com or better the Resource Timing API.

Beyond testing for failures, web developers and designers should embrace failures. From a design point of view Patrick advises to:

  • Beware that spinners give the perception that things are slow: they are acceptable but always use them in context.
  • Use skeleton screens for displaying content placeholders while they load.
  • Re-use familiar UI patterns with error messages.
  • Visualise the state of the network.

For developers, there are a some features of HTML and JavaScript (and support across – or not – the various browsers) that really make coding for failures possible, such as:

  • Use Resource Hints (preconnect, prefetch, prerender) to optimize connections and fetch or execute resource while the network is available.
  • Do not bundle all your assets together (this is a hack that will become harmful with HTTP/2). Instead group together what changes often or rarely, so that they are not always re-downloaded (e.g. group together libraries and utilities, but do not bundle them with application files).
  • Use a ServiceWorker as a replacement of the tedious AppCache. A ServiceWorker is a javascript file installed on the browser that tunnels all request from your browser. It is like a locally installed proxy that can help providing a real offline experience through the implementation of cache management techniques such as stale-while-revalidate, stale-if-error, timeouts, and dead-letter-queues.

 


 Designing connected products by Martin Charlier (@marcharlier) (Slides)

Martin has a background of industrial design, service design, and UX. This talk is a summary of what he discusses in his book.

Martin starts his talk explaining how the boundary between physical and digital is getting finer, creating a new generation of products with:

  • An extended value proposition through connected devices (e.g. iHealth scale).
  • Digital business models applied to industrial goods (e.g. freemium or micro payments).
  • Digital services going physical (e.g. smart locks, Nespresso Zenius).
  • New kind of products (e.g. Amazon Echo or Dash Button).
  • A device ecosystem (e.g. the battery as delivery mechanism).

Designing for connected products is not just UX (most visible) and industrial design (least visible), it is also being able to:

  • Formulate a value propositionWhat does it do?
  • Describe the conceptual modelHow does it work?
  • Design the interaction modelHow do I use it?

Some design techniques for productisation can help formulate the value proposition early such as writing the press release or sketching the box. For the conceptual model, because connected systems are more complex, designers must help connect technical differences to users (e.g. light bulb with wifi vs. bluetooth hubs): explain your system model or develop a really good way to simplify it.

The last part of Martin’s talk was to introduce the concept of interusability, which comes with focussing product design on:

  • Composition: How functionality is distributed across connected devices. Distributed functionality depending on the context of use (e.g. Tado thermometer, Skydrop sprinkler controller).
  • Consistency: Same terminology, same constructs across devices – while following platforms conventions (e.g. Hackaball).
  • Continuity: Design fluent cross-device interactions.

Finally, Martin concluded by stressing that designing for IoT it hard, and that building the right product is more important that building the product right. Hence the need for lightweight prototyping techniques to validate ideas.

 


 Smart Responsive Development by Vitaly Freedman (@vitalyf)

Vitaly is the Editor in Chief and co-founder of Smashing Magazine. His talk was mainly around providing CSS solutions to very specific UI needs like:

  • Creating a GLITHC effect with DIVs overlay.
  • Having links within links and using <object> for that.
  • Using object-fit to fit an image to a smaller section and keep the ratio (e.g. object-fit: cover).
  • Creating a responsive object map with SVG.
  • How to use display : flex to align vertically input label, field, and button, as well as aligning align text and pictures.
  • Using SVG and svgmask container to display an image over a transparent gradient background (there is an online tool to do that now).
  • How to use currentColor to inherit and re-use the color of parent or surrounding elements.
  • Using word-wrap and -webkit-hyphens to break a long word (like in German).
  • How to use CSS pseudo elements to highlight table cells and rows.
  • How to use rem units in the component root and em units for all sub-component elements to create components that really scale.
  • How to use mix-blend-mode to adapt the color of an element depending on the color of its background (e.g. mix-blend-mode: difference or mix-blend-mode: darken).
  • Using CSS not selector to apply a style to only some elements.
  • Using viewport units (e.g. height: 900vh) to make sure a hero image (landing page full screen) is always above the fold.

Then Vitaly talks about how to use modularity to establish universal styling rules between Designer and Developers:

  1. Establish a vocabulary as foundation of a design system.
  2. Identify and name design modules.
  3. Document the grammar (use lobotomised selectors).

Finally Vitaly finishes with a rant about forms. Stop designing forms. Design User Interfaces instead (www.typeform.com)

Web Directions 2015 – Day #1 Likes & Takeaways

I have just attended Web Directions 2015: a gathering of product designers and web developers for two days of fun and mind hiving.

This year the venue is quite a surprise: Luna Park on the shores of North Sydney. As I was staying on the other side of the river, I walked across the Harbour Bridge to hit the park backstage as they were preparing for Halloween, and a friendly staff showed me the way to the “Big Top“.

IMG_0516 IMG_0498

The conference ran with two tracks: Product+Design and Engineering+Code. It is a shame because its segregates the two by nature. So I decided to switch between the two tracks as much as I could. Here are the sessions I enjoyed the most on day #1:

Cap Watkins (BuzzFeed) – Design Everything
Alisa Lemberg (Twitter) – Building Empathy Through Data
Mark Dagleish (Seek) & Glenn Maddern – CSS in the Component Age
Dan Burka (Google Ventures) – Build for Speed
Maciej Ceglowski (Pinboard) – The Website Obesity

 


 Design Everything by Cap Watkins (@cap)

Cap started with a fairly long introduction of himself. He is the VP of Design at BuzzFeed. He previously worked at Etsy and Amazon. He explains how chance led him to become a designer: until he left college he wanted to be a doctor, then went into writing, and then design. This atypical career path gave him an understanding that product design is more than user flows and A/B testing.

Cap went on describing BuzzFeed’s approach to news and content, testing products, and experimenting. For instance the latest edition of their iOS App now comes with a tomagoshi-like watch app, which was developed during a hackathon.

Screen Shot 2015-10-30 at 10.43.41 pm

He then continued on what he does as a Leader, solving people challenges, to make great teams and bridge the gab between management, design, and engineering. These are his recommendations for making changes happen in an organisation:

  • Find your goonies in life and at work
  • Define the ideal state
  • Know how many fucks you give, be prepared to let go
  • Crystallize team thinking early in the project
  • Designers must learn HTML and CSS (and ship code to production)
  • Leading is empowering people
  • Be emphatic with your staff as you are with your users
  • Be patient

I really enjoyed Cam talk although he did spend a long time talking about himself. He reminded me of my ex-colleague Kent McNeill : self-centered and funny.


 Building Empathy Through Data by Alisa Lemberg (@DataByDesign)

Alisa is a User Researcher currently at Twitter working with video products. Her talk is about conducting user data research with empathy i.e. not by gathering series of statistical numbers, but by relating them to individuals: the users of course, but also the decision makers in your company.

She started describing the concept of extension neglect and why good user research must create empathy.

“The death of one man is a tragedy; the death of millions is a statistic”

Joseph Stalin

Alisa covered the work she does at Twitter to user-test the video auto-play, where they released the feature to only a limited number of users and asked them feedback using quick surveys.

She also did a gig with eBay to understand why they were loosing customers. Surprisingly, eBay did not have a lot of info about their customers except purchasing data. So they had to fill the gap conducting interviews.

She finished on empowering your organisation to make decisions based on the data (and not just chuck the results in a drawer). It is paramount to understand not only what users are (not) doing, but also why they are (not) doing it. In the eBay example, customers where actually not gone: they did not purchase but they were still browsing the site. So they created a personalised feed based on browsing data.

Key takeaways:

  • Empathy allows people to connect, understand, and feel empowered to act.
  • Know what you want to do with your data before you start gathering it.
  • If you do not have the data, fill the gap.
  • Quantitative and live data cannot replace talking to customers directly.
  • Release a feature to a small portion of users, and use quick surveys to understand how users felt.
  • Break the silos to cross-validate insights.
  • User data in three dimensions:
    • Experimental data = what users are doing
    • Survey data = how they feel about what they are doing
    • Qualitative Research = the larger context of their experience

CSS in the Component Age by Mark Dagleish (@markdalgleish) and Glen Maddern (@glenmaddern)

This was the best talk in the dev track IMO. Mark & Glen talked about the techniques that are now available to developers to “componentize” CSS, so as to avoid the one-big-fat-css-file effect. Also – and more interestingly – they explained how the emergence of web components has created options for scoping and composition in CSS. The talk was split in two distinct parts.

Part I – The End of Global CSS (Slides)

Mark starts the first part on the need to address the problem of scoping in CSS, as web applications have become large. Some projects and techniques attempt to solve it: OOCSSSMACSS, BEM, SUITCSS, and Inline Styles in React. Mark likes BEM, a .Block__Element--Modifier { ... } naming convention, to avoid style leaks, even if the HTML markup becomes quite noisy.

But with Web components (Polymers, Angular directives, React components), the focus is now on grouping and abstracting web assets into components. Webpack is a build tool that provides a single dependency tree for all assets associated with a component. Webpack can do cool stuffs to manipulate CSS like JavaScript. For instance you control the scope of a CSS class explicitly using :local(), and you also can load CSS files using require() if you use css-loader. So you can end up with something like that:

/* MyComponent.css */

.MyComponent {
   // some style here
}

.MyComponent__Icon {
   // more style there
}
/* MyComponent.js */

require('./MyComponent.css');

export default () -> {
  <div className="MyComponent">
    <div class="MyComponent__Icon">Icon</div>
  </div>
}

Mark finishes his talk introducing CSS Modules. This framework makes local scoping for CSS the de-facto standard. Prior to its existence Mark used postcss-local-scope to turn all global CSS selectors into local selectors for Webpack css-loader.

Part II – The Rise of Modular Style (Slides)

Glen starts with a bit of history of modules in JavaScript. In 2008, namespacing had to be crafted within a global context (you don’t know what runs before, nor after):

window.NAMESPACE = window.NAMESPACE || {};
window.NAMESPACE.Widgets = window.NAMESPACE.Widgets || {};
window.NAMESPACE.Widgets.foo = function () {
 ... 
};

Then came Google Chrome and V8, and with fast performance came the realisation that JavaScript could be a server-side language. But in 2009, when commonJS was born, there was no concept of modules:

“JavaScript needs a standard way to include other modules and for those modules to live in discreet namespaces. There are easy ways to do namespaces, but there’s no standard programmatic way to load a module (once!)”

Kevin Dangoor – What Server Side JavaScript needs

Following that, there has been an explosion of tools in the JavaScript ecosystem to promote the idea of an explicit and standard module system. Node nailed it, and npm swamped the web with interoperable JavaScript modules (+175K packages!) .

But what about CSS? Despite the emergence of new human interfaces to CSS (Sass, Less, PostCss), the machine interface hasn’t changed. Even if CSS is not JavaScript, the rise of web components is finally getting the CSS community moving in the right direction. Glen introduces the work he has done with Interoperable CSS (ICSS), to provide compilation and linking targets to be able to export and import CSS classes and variables into and fro JavaScript, and leverage existing JavaScript modules (he is a big fan of jspm). Glen continues on providing more insight into the rationale and philosophy behind CSS Modules, as an attempt to solve both the isolation and the re-use problem in CSS. To achieve that CSS Modules made all CSS class names local for isolation, and provide the construct compose: for reuse.

Isolation with CSS Modules

/* submit_button.css */
.normal {
  // styles here
}
/* submit_button.icss */
:export {
  normal: normal_f34f7fa0;
}

.normal_f34f7fa0 {
   // same styles here
}
/* components/submit_button.js */
import styles from submit_button.css'
document.querySelector('...').classList.add(styles.normal);

Composition with CSS Modules

/* BEM */
.SubmitButton .SubmitButton--normal { ... }
.SubmitButton .SubmitButton--danger { ... }
/* CSS Modules */
base {
    // common styles here
}
.normal {
    compose: base;
    // normal styles here
}
.error {
    compose: base;
    // error styles here
}

Somehow Glen finishes with a reference to Brad Frost Atomic Design, to explain that the component abstraction applies at every level: page, template, organism, molecule, atom.

Key takeaways:

  • The age of big fat CSS is long gone.
  • There is still not standard, nor widely adopted framework for writing and sharing modules in CSS.
  • It seems like BEM, Saas, and Webpack are a viable combination.
  • CSS frameworks like BEM or Sass address isolation or re-use, but not both.
  • CSS Modules attempt to provide answers to both, mostly by local scoping all CSS classes by default.
  • The web component area is upon us, the community is exited to see how it will change the way we style the web!

 Build for Speed by Dan Burka (@dburka)

Dan is a designer at Google Ventures, a Silicon Valley venture capitalist and technology services company. His talk is about Design Sprints, which is the idea of doing lightweight user testing without a build.

sprint-diagramDan realised when interviewing entrepreneurs in the UK than all of them worry about customers but none about design. This is the misconception that design is about branding or UI, and not so much ideation. The role of designers is fundamental in testing fresh ideas in the field. The typical Lean Startup Build-Learn-Measure can be a waste of time when you are close to launch and/or know that the build phase will be lengthy. Hence Design Sprints.

238431-944-629While Dan’s idea is not novel, the example he chose for this talk was great and entertaining. They ran a design sprint for Savioke who was 2 weeks before the launch of their Relay concierge robot. In five days, they recruited users from the street (using Craigslist + a screener), identified the KPIs (customer satisfaction index), provided solutions to address the main business challenge (how to make the robot less intimidating) and built experiments around those scenarios (robot knocking, dancing, adding a face, making weird sounds).

Key takeaways:

  • Design, is more than branding or UI, it is what creates a product the customers will enjoy.
  • Agile/Lean Design-Build-Launch-Measure process can be too slow.
  • User testing can be done very cheaply.
  • Gather the right team, make sure the customer is involved (e.g. dot voting on design ideas).
  • When hiring users, having the right people is important.
  • Prototype like it is a prototype (use Keynote or Flinto for mobile app).
  • Risky ideas sometimes work: test them!

 The Website Obesity by Maciej Ceglowski (@baconmeteor)

Maciej closing keynote is a super funny rant on the size of web pages on the internet.

When the average web page size was around 600KB in 2010, it was over 1.6M in July 2014, so it almost tripled in 4 years. Maciej stresses that heavy pages are slow pages, particularly in countries with low internet speed and bandwidth (like Oz). Throughout his talk, Maciej gives examples of the worse culprits, carefully selected from “evil” large corporations, and also compares them to classical Russian literature. One of Facebook Instant Articles features a 43MB national geographic image. Google Accelerated Mobile pages features a carousel that gets loaded over and over again, and is broken on Safari. Maciej suggests using the greatest US president in size (William H. Taft) to replace all websites’ large images.

Screen Shot 2015-11-04 at 10.40.51 pm

Then Maciej goes on the fallacy of the Google Speed Index: it does not measure the total page load (see this article for details), so things could still be loading and sucking resources while visually the page has been rendered.

In Maciej’s fantasy world, internet pages are mostly text, markup, some images, less CSS, and even less JavaScript. In reality they are made of a lot of crap on top of HTML, and at the apex the NSA.
Screen Shot 2015-11-04 at 11.01.43 pm

Then Maciej’s dismantles the economy of online advertisement. He compares online ads to an internet tax where all revenues go to Facebook, Yahoo, and Google. Even small players in the space will end up paying royalties to those big three. Besides, ads massive contributors to pages obesity. The only way is to stop ads. We need surveillance of our ads, to re-gain control.

Maciej finishes on mainstream web designs that feature fat web assets like the Apple public website, or a waste of real estate like the new Gmail front-end. He calls that chickenshit minimalism: the illusion of minimality backed by megabytes of cruft. His last rent before his killing closing sentence (see below) is about the cloud envy: cloud can lead to prohibitive costs while there is this illusion that the cloud will make your site faster.

Thanks Maciej, I had a laugh.

“Let’s break the neck of the surveillance authorities”

Maciej Ceglowski – Web Directions 2015


Thanks to Maggie, Huan Huan, Marzieh, Mindy, Nathan, and Peter for their very enjoyable company on a beautiful and mind buzzing day.

IMG_0520_crop

The Fates of a successful IT project

When Agile came up, it had a high barrier to entry, mainly because it was a real shift from traditional software development methods, but also because of its jargon and ceremonies: stand-up, iteration, card, points, technical debt, owner, customer, retrospective, etc. But now that Agile is mainstream, most companies have tried these things out, more or less successfully, and the initial mindset of challenging the “status quo” is probably long gone.

That is the reason why in this post I would like to come back on the three principles that are the most essential to a healthy IT project, and have proven very valuable in my career as an Iteration Manager or Tech Lead: Transparency, Team, and Waste. While they could be applied to any methodology, I will relate those principles to Agile practices, so that you have something concrete to take away.

Transparency

What if you’re project is not going well at all? What if you have disengaged stakeholders, unhappy customers, low morale within the team, and no clear view as to where the project is going and when it can be completed? In this case people will have the tendency to become defensive, protective, even maybe secretive. Things could go political very quickly.

One of my colleagues once said Agile should simply be called “transparent delivery“. If you want to avoid inflating the bubble, keeping the project stakeholders in touch with the reality using clear representations on the project status from different perspectives (business, budget, technical) is really important. Make sure that project indicators and signs are in place from day one, clearly visible, and seen and understood by all. In an Agile project, there are usually various “information radiators” for that purpose. The project or iteration wall is definitely one of them. Also, having a visible backlog is essential to customers and product owners. Finally, technical indicators like bug counts, build lights, or tech debt are useful for developers and testers.

If you are unlucky and the bubble is already close to bursting, a little bit of courage and honesty can make a difference. In this situation you may find that mid-level or top-level executives are usually much more pragmatic than managers and will be more helpful and supportive if you have a plain honest conversation about the project. In my case I found that inviting them to the showcase and depicting things as they are can pay off, as long as you can propose changes to improve the situation.

Team

The most important thing in an IT project is the team. Let me repeat that: the most important thing is the team. What does it mean? If you are in a project that goes bad, then leave the angry customers, the money, and the managers raised eyebrows aside. Instead, focus on regrouping the team.

Now it is fair to wonder “What makes a winning team?“. In my experience, beyond anything else (talent, salary, environment), the two main ingredients are morale and trust. Tuckman’s forming-storming-norming-performing model details the different phases the team will go through before it can be effective. I believe maintaining morale greatly helps having a “norming” team, while trust is the key to reaching the “performing” state.

To raise or maintain morale, I like to use Appreciative Inquiry during retrospectives. Each participant writes down something she is grateful the person seating next to her did in the last iteration. Notes should be read aloud, and each team member leaves with an energising post-it to stick to its monitor. Another technique is to use humour as often as possible. Below is an example where the team gathered funny statements in the stories management tool Trello.com.

Screen Shot 2013-11-16 at 12.21.09 PM

I love this technique because it proves that people talk and listen to each other, and that they are having fun. It also helps bridge the gap between introvert and extravert.

Establishing trust is probably more difficult. The Wikipedia page about Tuckman’s group development model mentions (about the norming state):

The danger here is that members may be so focused on preventing conflict that they are reluctant to share controversial ideas.

Indeed, an element of gaining trust is to challenge each other, particularly on sore points that make the team inefficient. I always found that people will trust you even more if you disagree with them. Also empowerment is important. Letting the team taking responsibilities for design discussions, organising showcases or important meetings with key stakeholders, or teaching junior developers (through pairing) will provide the reward they need for a both ways trust relationship.

Waste

In the last 5 years, lean principles have become increasingly popular within the IT industry. Preached by Eric Ries in his The Lean Startup book, principles like continuous deployment, lightweight processes, failing fast, or pivoting really help IT organisations refocusing their effort on discovering the shape and form of a product as it is being developed. It dramatically reduces waste by starting with nothing and building cheaply and effectively during the journey.

For me the underlying fundamental principle of a lean and agile team is really to cut the nonsense: do only what is strictly necessary for delivering a good product as early as possible. There are so many things that IT teams produce, which will never see the light of day, or are pure wasted time, it is astounding! To use an analogy with Taxis, it is a bit as if the driver would take you to your destination but going through another city, maybe because he thinks that is where you want to go. If you are in this cab, make sure you stop him at the city outskirts.

Again several techniques are on offer to reduce waste. Early user testing (before the product is built or the story delivered) will assure that you develop the right product. Prioritising the backlog with the customer and/or product owner frequently ensures the team delivers the most important stories first (and maybe only). Story kick-off with most of the team members (and particularly BAs and QAs) ensures that we all understand what needs to be built and the developers do not go on a tangent. Finally having clear and detailed acceptance criteria will guide the developers throughout the delivery of the story, forcing them to focus in fulfilling those criteria only.

Recap

The table below recaps the three fundamental principles of a successful IT team/project and the corresponding techniques I recommend you apply.

Transparency

  • Information Radiators (iteration wall, backlog, technical debt, build lights)
  • Plain and honest showcases

Team

  • Appreciative enquiry during retrospectives
  • Humour, like statements of the week
  • Empower the team with design discussions, showcase organisation, or technical challenges

Waste

  • Lean principles
  • Early user testing
  • Stories prioritisation, kick-off, and precise acceptance criteria

Why I like Go so much, and techniques for managing its configuration.

For the last 5 years, I have tried several Continuous Integration servers: Team City, Jenkins, AnthillPro, and ThoughtWorks Go. I have to admit, even if it may sound bias, that I really like the latter. There are several reasons why I like it: its CCTray XML feed, its UI and model designed around pipelines (i.e. builds) and dependencies, and its security model. But above all, the entire Go configuration can easily be managed as source code: version controlled, built, tested, and automatically deployed. In this post, I will explain what I mean by that.

Go Config Overview

Go is entirely configured using a single XML file, cruise-config.xml (Cruise Control is the old name for Go, prior to version 2). This file is located in /etc/go on Linux and by default in C:\Program Files\Go Server\config on Windows. It is composed of 5 main sections:

  • <server> Global server settings such as license and users
  • <pipelines> Your build pipelines
  • <templates> Your build pipeline templates (to create re-usable pipeline blueprints)
  • <environments> The definition of the hardware/software resources the pipelines will use and act upon
  • <agents> The build agents

Although Go, like the others CI tools, has an Admin UI to configure the general features (e.g. LDAP integration) or pipelines, I much prefer to manipulate the cruise-config.xml file directly. Indeed, if you change it, then the changes are automatically applied to your Go server (if they pass Go validation). So adding a new pipeline is a simple as adding a few lines of XML!

First Pipeline

Let’s for instance write a dumb pipeline that will just output “Hello World” to the console. To do so, simply add the following lines of XML to your Go server cruise-config.xml file (note that I use a personal Github repository as <material>, and you will gave to change it to something else):

<cruise>
  <server>
    [...]
  </server>
  <pipelines group="Demo"> 
     <pipeline name="HelloWorld">
       <materials>
         <git url="http://github.com/jdamore/cruise-config" dest="cruise-config" />
       <materials>
       <stage name="HelloWorld">
         <jobs>
           <job name="SayHelloWorld">
             <tasks>
               <exec command="echo">
                 <arg>Hello World</arg>
               </exec>
             </tasks>
           </job>
         </jobs>
       </stage>
     </pipeline>
  </pipelines>
</cruise>

Screen Shot 2013-06-19 at 10.50.59 PM

For me being able to edit cruise-config.xml, add a few lines of code, save the changes, and see the UI updated with my new pipeline, stage or job is really really cool. But of course why stop there?

Cruise Config Pipeline

Modifying cruise-config.xml in an uncontrolled manner is dangerous. True, Go will backup any version that is rejected because syntactically or logically incorrect, so that you do not loose your latest changes, but what if you have to come back to the version before last? What if you want to revert the last two changes? Of course what I am getting at is: cruise-config.xml must be version controlled. So first thing you must do is stick it in the VCS repo of your choice.  At least you would be able to push new changes or rollback previous changes if you wanted to.

But what if…. what if anytime you commit a change to the cruise-config.xml in your source control repository, it gets automatically applied to your Go Server, instead of having to manually copy the latest file to Go? Is it not what Go is good at, automatic deployment and all that? Sure. So let’s create a CruiseConfig pipeline that will take the cruise-config.xml file from your repo and copy it into the Go Server configuration directory. In the example below, I use a GitHub repo to control my cruise-config.xml. The pipeline below will download the content of the repo and execute a cp command (my Go Server is on Linux) to copy the cruise-config.xml file to /etc/go. Of course, in my case, the Go Agent running this job will have to be a local agent installed on the Go Server machine. If you want to use a remote agent, then you could copy over SSH (scp command) on Unix/Linux or over a shared drive on Windows.

<pipelines group="Config"> 
   <pipeline name="CruiseConfig">
     <materials>
       <git url="http://github.com/jdamore/cruise-config"/>
     </materials>
     <stage name="Deployment">
       <jobs>
         <job name="DeployConfig">
           <tasks>
             <exec command="cp">
               <arg>cruise-config.xml</arg>
               <arg>/etc/go/.</arg>>
             </exec>
           </tasks>
         </job>
       </jobs>
     </stage>
   </pipeline>
</pipelines>

So now I have two pipelines: CruiseConfig and HelloWorld.

Screen Shot 2013-06-19 at 11.28.23 PM

It means that if I want to change my HelloWorld pipeline I can simply edit cruise-config.xml an check it in. As a test, I will add another stage HelloWorldAgain as follows:

<pipeline name="HelloWorld">
  <materials>
    <git url="http://github.com/jdamore/cruise-config" dest="cruise-config" />
  </materials>
  <stage name="HelloWorld">
    <jobs>
      <job name="SayHelloWorld">
        <tasks>
          <exec command="echo">
            <arg>Hello World</arg>
          </exec>
        </tasks>
      </job>
    </jobs>
  </stage>
  <stage name="HelloWorldAgain">
    <jobs>
      <job name="SayHelloWorldAgain">
        <tasks>
          <exec command="echo">
            <arg>Hello World Again</arg>
          </exec>
        </tasks>
      </job>
    </jobs>
  </stage>
</pipeline>

Then I commit the changes.

Screen Shot 2013-06-19 at 11.20.18 PM

Then the CruiseConfig pipeline will automatically be triggered and deploy the changes to Go.

Screen Shot 2013-06-19 at 11.29.51 PM

The CruiseConfig pipeline runs

Screen Shot 2013-06-19 at 11.30.10 PM

The HelloWorld pipeline now has a second stage

Et voila! The HelloWorld pipeline changed instantly and automatically, with the addition of the new stage.

Cruise Config split

I just want to finish this post with what I strongly recommend people do in order to make the cruise-config.xml more manageable. Because the number of pipelines and pipeline groups will grow, particularly if more than one team uses Go, you need to be able to split and isolate the various pipelines and composing elements of the configuration file. I use a Ruby template (erb) to do so as follows:

<?xml version="1.0" encoding="utf-8"?>
<cruise xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:noNamespaceSchemaLocation="cruise-config.xsd" schemaVersion="62">
    <server ...>
        <license ...>
           [...]
        </license>
        <security>
            [...]
        </security>
    </server>
    <%= pipeline_xml 'oloservices'     %>
    <%= pipeline_xml 'olohtml5working' %>
    <%= pipeline_xml 'olohtml5release' %>
    <%= pipeline_xml 'spdev'           %>
    <%= pipeline_xml 'goenvironment'   %>
    <%= section_xml  'templates'       %>
    <%= section_xml  'environments'    %>
    <agents>
        [...]
    </agents>
</cruise>

I just have one line for various sections, the first 5 being <pipelines>, and the last two being the <templates> and <environments> sections. I have implemented two functions, pipeline_xml and section_xml, whose job is to output the XML for the specified pipeline group or section. For instance, pipeline_xml 'spdev' will output the content of the file called pipelines.xml that is located in a directory called spdev. The idea here is to reconstruct the full cruise-config.xml before deploying it to Go. Besides, by doing so, one can also execute acceptance tests before deploying the config changes to Go, to make sure they are harmless. I then ended-up with a GoConfig pipeline that has 3 Stages:

  1. Build – Reconstruct the file
  2. Test – Validate the config
  3. Deploy – Copy the file to the Go config directory

Screen Shot 2013-06-19 at 11.55.13 PM

Story points vs. stories count, plus different ways to look at your project

As I was reading a ThoughtWorks Studios blog post on estimation and story points, it reminded me of a very same experience on a smaller project for a retail client in Australia.

The project was a 6 months long .NET delivery gig, where I did a fair bit of projection/iteration management. We used a Trello board plus a physical card wall, and the stories had all been previously estimated during an Inception we had run a few weeks in advance.

After 3 Iterations, the customer asked me the typical question: “How are we tracking?“, to which I gave the typical answer: “42“. Well, not quite actually, as I had data: the cards had been estimated, I knew what we had delivered in the first three Iterations, so I could show them the beautiful burn-up that follows.

Screen Shot 2013-05-28 at 9.36.45 PM

Unarguably, it does not look very promising (rest assured, this story has a happy ending). But more importantly, by looking at it myself I really challenged whether this graph told the whole story, or even the most important part of the story. So I went back to my Excel (yes, really), and came up with the following two graphs.

Screen Shot 2013-05-28 at 9.40.30 PM

backlog vs. delivered by story size

The two graphs show the backlog with its composing features (above), alongside what has been delivered in the past iterations (below). The first plot splits the stories by T-Shirt size – from the smallest in orange (XS), to the largest in blue (XL) – whereas the second splits them by priority – 1 in the darker green, and 4 in the lightest green.

Screen Shot 2013-05-28 at 9.40.46 PM

backlog vs. delivered by story priority

So, what can we deduce from these two graphs that the burn-up could not tell us? First, we know we have focussed primarily on high priority stuffs, which is always a good sign. In this case we had a pretty good idea of the slice we would deliver first. Second, in the last Iteration we have delivered a similar amount of smaller stories than in Iteration 2. So maybe those stories were not that small after all? Or maybe the Iteration 2 stories were not that large? Hard to say. It turns out we spent most of our time in Iteration 3 battling with the database, and re-writing a previously delivered story to fit misunderstood architectural constraints. But my point is: those questions about the accuracy of our story points sizing are not really important. We are here to deliver software, not playing the accountants.

Now, something else came up. As I said, a lot of those stories had been estimated quite a long time ago, and by a different team. So as soon as we started discussing them in details, we realised that they were too big, or too small, or could be grouped more adequately. So we did spend a lot of time changing/splitting/merging them. That’s great! I love that because you get a shared understanding within the team of the business context and what is to be delivered, instead of just moving the story to In-Dev without thinking about it.

Then I asked the team: “Okay, we need to re-estimate them now.And I was righteously asked: “Why?“. Indeed, the total number of points of all the original stories we have changed/split/merged was probably not far off the total number of points for the new stories. We had done a great job discussing those stories, why spending any more time re-estimating them? Well, I tell you why: “To make me feel secure, and to make the PMs feel secure.” And since I don’t do secure, I thought what the hell, let’s just arbitrarily set the estimates based on my sole knowledge. From then, and for the rest of the project, I never asked anyone to re-estimate a story, unless it was brand new scope (in which case I did the estimation with the Technical Lead). You might think it is a fraud, I would say it is eliminating waste.

So, how did the project go? The following burn-up is up to completion of the project.

Screen Shot 2013-05-28 at 10.02.10 PM

As you can see, we did okay and managed to deliver most fo the scope. Now, the next burn-up is the same one but using a Stories Count (i.e. number of stories delivered per iteration) instead of Story Points (i.e. number of potatoes delivered per Iteration). Look how similar it is.

Screen Shot 2013-05-28 at 10.04.30 PM

That is why I believe velocity and project tracking by Story Points is a bit of a waste. Mike Cohn might have become a little bit richer with his Agile Estimating & Planning book (which confused the hell out of me when I first read it), but I don’t think it can help any estimation getting closer to reality.

On my other point of trying new ways to track progress, let’s have a look at the backlog vs. delivery plot mid project.

Screen Shot 2013-05-28 at 11.04.38 PM

I really like it because it tells you many things:

  • Three epics (or features) still have a significant amount of work required (Product Selection, Customer Engagement, and Global). What about pulling more stories from those into the next Iteration?
  • There is a good split between Large, Medium, and Small stories in the last 4 Iterations.
  • There are only 2 small stories left in the Data Collection bucket. Any reason why they are not done? Maybe we should have a look at them and kill them if possible.
  • At a glance it should take another 4 iteration to deliver the entire backlog.

So to wrap-up an already too long post:

  1. Estimating in story points has often a low return on investment.
  2. Be creative and use your data in an informative way: it is not all about burn-up and deadline.