Clean Agile, XP, and Scrum - Avoiding the "Hangover"

Apr 22 | Henrique F. Teixeira

Is your team's board showing several cards stuck in the same column for days with nothing being delivered? Are business stakeholders frustrated, constantly having to redefine deadlines and strategies? Do you feel like there's a lot of work but little result? Is the atmosphere tense, and the situation only seems to be getting worse?

You might be experiencing what Robert C. Martin (Uncle Bob) named in his book Clean Agile as the 'agile hangover.' It's rare for anyone working with software not to have gone through this, especially when there's a lack of senior people on the team and/or we're dealing with complex software: The source code is spread across various projects, it's too extensive, poorly documented, or badly written.

The idea behind the agile movement is comprehensive, complex (despite appearances), and encompasses various different 'agile methods' scattered around. Similarly, so is software. And the more complex a problem, the harder it is to explain and understand. Furthermore, our knowledge transmission is often flawed (confirmation bias, information overload, and gaps), and each listener can interpret and replicate the same information in different ways, which can end up shaping a reality, a truth, or a social behavior.

Returning to the main topic, it's possible to observe in the market a mix of interpretations of 'agile' being applied simultaneously, in pursuit of an idealized and 'theoretical' result that often doesn't prove true in practice. These interpretations can stray far from the main manifesto and its principles, leading to much frustrated expectation, finger-pointing, and few results.

"The agile methodology took the software industry by storm. But, like a game of telephone, the original ideas of agility were distorted and simplified, arriving at companies as a promise of 'a process to deliver software faster.'

– Robert C. Martin, Clean Agile, p. 170"

Who is Robert C. Martin

Better known as Uncle Bob, Robert C. Martin has been a software engineer since 1970, an international consultant, speaker, author of several IT books, and one of the 17 original signatories of the Agile Manifesto in 2001.

Recently, he decided to revisit the topic of 'agility' and release the book Clean Agile, where he explains the original ideas of the manifesto and offers some critiques on how the market has absorbed them.

Clean Agile Development

According to the book, the definition of agile in its 'clean' form (the original idea, without noise) is:

"A process in which a project is subdivided into iterations. The output of each iteration is calculated and used to continuously evaluate planning. Features are implemented according to the added business value, so that the most valuable things are implemented first. The quality level is kept as high as possible. The schedule is primarily managed by manipulating scope."

– Robert C. Martin, Clean Agile, p. 32

The rest (Scrum, XP, Kanban) are merely tools, methods, or frameworks to facilitate and/or guide the use of agile. Beyond this definition, the twelve principles of agility must be followed:

  1. Our highest priority is to satisfy the customer through early and continuous delivery of valuable software.

  2. Welcome changing requirements, even late in development. Agile processes harness change for the customer's competitive advantage.

  3. Deliver working software frequently, from a couple of weeks to a couple of months, with a preference to the shorter timescale.

  4. Business people and developers must work together daily throughout the project.

  5. Build projects around motivated individuals. Give them the environment and support they need, and trust them to get the job done.

  6. The most efficient and effective method of conveying information to and within a development team is face-to-face conversation.

  7. Working software is the primary measure of progress.

  8. Agile processes promote sustainable development. The sponsors, developers, and users should be able to maintain a constant pace indefinitely.

  9. Continuous attention to technical excellence and good design enhances agility.

  10. Simplicity—the art of maximizing the amount of work not done—is essential.

  11. The best architectures, requirements, and designs emerge from self-organizing teams.

  12. At regular intervals, the team reflects on how to become more effective, then tunes and adjusts its behavior accordingly.

Uncle Bob also states that the XP (Extreme Programming) framework is the "most well-defined, complete, least confusing, and best representative of the essential core of agility," and he uses it as an ideal basis to explain agile concepts.

XP, a brief summary

XP (Extreme Programming) is an agile framework created by Kent Beck that provides practices for software development, represented in the model below:

In the outer circle (Purple), we have 📊 business practices.

  • Acceptance tests: basically means that each demand (stories, tasks, and spikes, for example) must have its acceptance criteria, which are necessary for the completion and acceptance of a demand. These criteria can be validated through acceptance tests and are well-aligned with the concept of Definition of Done.
  • Small releases: we should plan to deliver software incrementally, feature by feature, in a gradual manner.
  • Planning: the act of planning the iteration, defining how we can divide a project into features, epics, stories, tasks, and spikes.
  • Whole team: the team (P.O, P.M, Devs, QA, Managers) must work together from the beginning to the end of the project; communication should be easy between all parties. The result belongs to everyone.

In the middle circle (Dark Blue), we have 👥 team practices.

  • Continuous integration: the act of constantly integrating changes made to the main code (production), one by one. (Do not accumulate sequences of changes by integrating them all at once).
  • Metaphor: similar to the ubiquitous language of DDD (Domain Driven Design), everyone on the team should speak the same language, and the code should reflect this language in how it's written. Personally, I like to say that readable code is one where a non-technical person could easily step in, read, and infer what's happening.
  • Sustainable pace: we should develop at a predictable rhythm. The goal is not to finish quickly, but for the team to maintain a constant and healthy pace.
  • Collective ownership: any team member can check and improve any module of the project at any time. Knowledge should be distributed among everyone.

In the inner circle (Light Blue), we have 💻 technical practices.

  • TDD (Test Driven Development): we should write automated tests and, if possible, practice TDD.
  • Pair programming: we should encourage pair programming, especially in scenarios where there's a need for knowledge exchange, mentoring, and attention to very critical demands. It's a choice for programmers and the team (it's not mandatory).
  • Refactoring: We must deliver code better than we found it. If there's a need or opportunity for refactoring, refactor.
  • Simple Design: "It's the practice of writing only the necessary code with a structure that makes it simpler, smaller, and more expressive":
  1. Run all tests
  2. Express intent
  3. Eliminate duplication
  4. Reduce elements.

A little more about Planning in XP

This is the central practice of the entire framework; planning occurs in a meeting called Iteration Planning where:

  1. We verify the data generated by the last iteration.
  2. Stories are presented by business stakeholders and validated by DoR (Definition of Ready) criteria. The book Clean Agile cites the use of the acronym INVEST (Independent, Negotiable, Valuable, Estimable, Small, and Testable) as an example of Definition of Ready.
  3. Stories can be broken down into technical SMART tasks and estimated by the technical team. Estimates can be made in various ways: in points from 1 to 6, t-shirt sizes (s, m, l), Fibonacci for measuring effort, or a three-point estimate if we want to estimate time.
  4. Spike (or discovery) stories are created if there isn't enough information or technical knowledge to deliver a certain story, or if we can't ensure it is INVEST.
  5. What we will do in the next iteration is defined by the whole team; we use the data generated in previous iterations to support us (such as team velocity).

"The meeting should be scheduled to be one-twentieth the length of the iteration. The IPM (Iteration Planning Meeting) for a two-week iteration should take about half a day."

– Robert C. Martin, Clean Agile, p. 70

Note: We can have a second Iteration Planning meeting after the first, only with the technical team, to break down stories into SMART tasks instead of doing the breakdown in a single meeting.

Planning in a Zero Iteration

In a project's zero iteration, we have a different planning approach, in a meeting called Release Planning. It's essentially the same as Iteration Planning, with the difference that we won't define what we'll do for a single iteration, and we don't yet have generated data. We focus on global issues (at a high level, without much detail) for the project as a whole (all iterations).

Summarizing the flows of an XP project with planning as the central practice, we have:

Scrum vs XP

Scrum is an agile framework developed for use in complex and frequently changing environments. It defines roles with responsibilities (Scrum Master, Developers, Product Owner), events (Sprint, Planning, Daily, Retrospective, Review), artifacts (Product Backlog, Sprint Backlog, Increment), and the relationships between them. Scrum is intentionally incomplete, and ideally, it should be complemented with other practices. For example, the Scrum Guide doesn't define that we must do TDD or refactoring in the system's code, but it does state that 'during the Sprint, quality must not decrease.' Therefore, adding these technical practices from XP to Scrum further enriches the project and aligns with the agile principle: 'Continuous attention to technical excellence and good design enhances agility.'

Similarities between Scrum and XP:

  • Both define an event called planning to plan the iteration.
  • The iteration in XP is analogous to the Sprint in Scrum.
  • Product Backlog items are similar to User Stories.
  • Stand-Up Meetings are similar to Dailies.

XP points that add value to Scrum:

  • Although Scrum mentions that we should choose items that are Ready to work on in the iteration, the concept of Definition of Ready is not explicitly defined.
  • Spike stories are fundamental for working on discovery tasks within the sprint.
  • Technical, team, and business practices can further enhance the quality of deliverables.
  • We can add acceptance criteria to User Stories / Product Backlog Items.

Definition of Done vs. Acceptance Criteria

These two concepts confuse many people. Although they seem similar, Definition of Done (from Scrum) is an item-independent definition, while acceptance criteria (from XP) are item-dependent. Example of DoD: 'Any backlog item, to be considered done, must have test coverage of 95% or more.' Example of acceptance criteria: 'The user must be able to click on the menu item and view the shopping cart page.'

Blending XP with Scrum

Personally, I prefer to avoid very long meetings. I understand that accumulated mental exhaustion can lead to poor decisions. In XP, a lot of responsibility is concentrated in a single meeting (Planning), but following the idea of Scrum Events, we can separate these responsibilities into shorter meetings:

  • The iteration planning and commitment (Sprint goal) part continues in a meeting called Planning.
  • The estimation, creation of stories, tasks, and technical spikes / discovery are handled in a meeting called Backlog Refinement (which happens informally during the iteration, as needed). Ideally, this meeting should occur before planning to have Ready items for the iteration.
  • The review of results and data generated by the iterations takes place in a meeting called Review.

In addition to these, we can also have a specific meeting to reflect on the iteration and seek opportunities for improvement, the Retrospective (usually right after the Review), making the flow more similar to the one below:

If we replace 'Iteration' with 'Sprint' and add the Daily, we have all the Scrum Events defined in the Scrum Guide (https://scrumguides.org/scrum-guide.html#scrum-events).

Furthermore, we can also address the Scrum Planning questions in our Planning:

  • Why is this Sprint valuable?
  • What can be Done this Sprint?
  • How will the chosen work get done?

By using XP framework practices 'seasoned' with Scrum events, we can have an agile model that, respecting the manifesto and all its principles, leverages the benefits of both tools.

Daily Meetings

Regarding the famous 'dailies,' I bring a quote from the book Clean Agile:

"Over the years, there has been much confusion about 'The Daily Scrum' and 'Daily Meetings' (aka Standup Meeting). Let me clarify this confusion. Here's what applies to the Daily Meeting:

• This meeting is optional. Many teams get along just fine without having one.

• It can be less frequent than daily. Choose the schedule that suits your reality.

• It should last about ten minutes, even if the team is large.

• This meeting follows a simple formula.">

– Robert C. Martin, Clean Agile, p. 113

The author continues:

"The basic idea is for team members to stand in a circle and answer the following questions:

1. What did I do since the last meeting?

2. What will I do until the next meeting?

3. Are there any impediments in my way?"

– Robert C. Martin, Clean Agile, p. 113

Although the author's ideas are slightly different from what is defined in the Scrum Guide, based on my professional experience, I can only agree.

Currently, my team's dailies are asynchronous (we send updates via text); it's fast and works very well. If we need to delve deeper into something, we jump on a synchronous call, and it's resolved.

  • We join synchronous calls or chats as needed
  • We save time
  • We stay focused

Avoiding the Hangover

If I could list three main practices that increase a team's predictability and productivity, avoiding a major 'agile hangover,' based on the book 'Clean Agile' and my professional experience, they would be:

1. Attention to software 'design'

Design is an essential stage in the software development process.

"Every iteration in the project, from beginning to end, will have a bit of analysis, design, and implementation. In an agile project, we are always analyzing and designing."

"The activities of analysis, architecture, design, and requirements implementation are constant throughout the iteration."

– Robert C. Martin, Clean Agile, p. 24/25

Everyone who develops software does design, consciously or not.

Before starting to write code, it's necessary to spend time analyzing the demand and the development environment in question to understand business rules, effort, and project patterns. In other words, it's necessary to understand what we will technically need to do and what the best practices to use are. It's necessary to plan the technical part.

Imagine a developer picked up a task with a one-week deadline and after 7 days of development discovered that the task involves much more effort than initially known, and could take up to a month to complete.

To improve attention to design, we can create Spike stories on the board when necessary, or even create a dedicated board just for technical discovery of stories and epics. Spikes are fundamental for defining technical code implementation strategies and breaking down tasks.

It's also important to adopt practices like TDD and Refactoring in daily work.

2. Maintain a sustainable pace

Software development is much more like a 'marathon' than a sprint. If we maintain the pace, we go further.

The important thing is to be predictable, more than fast. This way, we achieve more accurate metrics and deadline estimates.

"Managers willing to encourage developers to work faster are using all the transparency of the process to micromanage them. Agile coaches, without a shred of commercial or technical experience, are training managers and telling development teams what to do. Managers are setting roadmaps, milestones, and shoving them down the throats of development teams."

– Robert C. Martin, Clean Agile, p. 171

If we try to go too fast, we'll eventually burn out.

Being burnt out makes us more prone to causing bugs and design errors in systems.

Bugs and design errors slow down deliveries.

Slowed deliveries cause management problems.

Management problems increase pressure, and the vicious cycle of the 'agile hangover' continues.

3. DoR (Definition of Ready) and DoD (Definition of Done)

The entry of poorly 'refined' demands into the development pipeline causes many problems, including:

  • Rework
  • Feeling of insecurity
  • Waste of time
  • Frustrated expectations

Similarly, 'finishing' a demand without a well-defined 'done' definition can cause the same problems if we deliver something different from what the business team expected, or if important details are missing.

(Note: Understand 'demand' as a story card written by business stakeholders.)

Before committing to a demand in the iteration, set aside time to refine it with the team:

The demand refinement stage can occur in a meeting specifically scheduled for it, during the planning meeting, or at any other time when the team is gathered.

It's important that the entire team is present during demand refinements because this way, everyone involved gains awareness of each demand, has the opportunity to discuss them, offer opinions, and feels more secure if they are later responsible for delivering any of the demands.

Conclusion

Clean Agile is one of the most accessible and precise books I've read on the subject. The author truly captures the current sentiment surrounding agile and brings us 'back to basics' as promised. I classify the book as essential for anyone looking to boost performance and improve the mental health of their team by shifting perspectives, breaking fallacies, and reducing noise related to agile.

Regarding the model I presented, I have contributed to applying it in some teams, and here I offer a graph based on a real 'before and after' experience concerning task lead time from initial to completed state.

Around April and May, we decided to change our mindset and pay more attention to task refinement, spikes, and acceptance criteria, along with other XP ideas combined with a flow and events based on Scrum. We gained in:

  • Predictability (sustainable pace)
  • Trust
  • Mental health

It's worth noting that teams, projects, and companies have their specificities, and continuous improvement learning is also part of the agile model. Therefore, there is no perfect model, but it's always good to start from well-established ideas and solid principles to gain maximum advantages.

Hey, what did you think of this article??

Share and give your opinion by clicking on one of the networks below:


Thank you very much!