Agile audit of agile projects

Studying this technical article and answering the related questions can count towards your verifiable CPD if you are following the unit route to CPD and the content is relevant to your learning and development needs. One hour of learning equates to one unit of CPD. We'd suggest that you use this as a guide when allocating yourself CPD units.

As an auditor I need to know what agile really is, how it works in practice, the audit principles, so that I can plan and do my work. I will achieve this when:

• I understand agile principles and how they differ from other approaches
• I have the confidence to undertake an audit of agile projects
• I know where to go for further information
• I can pass the MCQ CPD test.

What is ‘agile’?

Agile is one of the most mis-used buzz words in modern organisations. It is often taken to mean low cost, cavalier with little or no need for governance or controls. This is not true; the best definition is from the 2001 Agile Manifesto:

‘We are uncovering better ways of developing software by doing it and helping others do it. Through this work we have come to value:

• individuals and interactions over processes and tools
• working software over comprehensive documentation
• customer collaboration over contract negotiation
• responding to change over following a plan
• that is, while there is value in the items on the right, we value the items on the left more.’

Unlike some other approaches (e.g. Prince2) the emphasis is more on the actual outputs - deliverables and improving collaboration team work to achieve it. There is less emphasis on the inputs (tools used) or the process (relying on a plan). As auditors the last sentence is particularly important as it emphasises that there is still value in documentation etc. Interestingly, many organisations omit this sentence when copying the manifesto (BTW - this infringes the manifesto authors’ copyright).

The manifesto is supported by 12 principles. These can be summarised as:

• focus on early, frequent, continuous deliverables (based on business priorities) of useful product
• welcoming change as the project progresses
• business people and developers working together
• motivation of self-organising teams with feedback and improvement
• keeping it simple but still paying continuous attention to technical excellence.

Unlike a traditional waterfall type approach, the focus is on getting a product in use (a ‘minimum viable product’) and then refining it. A waterfall project completes all requirements, design, build, testing and release for a phase together. At any one time in agile, different products for the project may be at different steps of development. There is also an emphasis on change - the project may change in response to changing business needs or technical developments. Waterfall projects are less flexible - they are not agile.

If a waterfall project is abandoned after say six months, you will probably only have some expensive documentation to show for it. If an agile project is abandoned after six months there should be some useful deliverable that is being used by the business to derive benefit, and there will have been information gathered to help with future projects. The real risk is that the waterfall project may not have been identified as a potential failure after just six months and may have consumed even more resources. Agile is about daring to ‘fail quickly’.

There are times when a waterfall approach may be better than agile - for example if there is unlikely to be significant change of requirements or technology, or if there are significant regulatory or compliance requirements for the project. Organisations should set policies as to how each approach is used. Sometimes a hybrid approach will be used (‘wagle’ - or cynically called ‘fragile’). Agile and Prince2 are not mutually exclusive. Prince2 Agile™ has been developed to combine the requirements for governance with the need for flexibility and responsiveness (see references below).

How do you ‘do’ agile?

Now we know what agile is - but how do we do it. Agile should be considered as an approach rather than a strict methodology. Agile can be applied not only for software development projects but is also used in civil engineering and in planning and even in conducting audit and assurance reviews. The most commonly used approach to applying agile for these is the ‘scrum approach’ - the principles are similar in some of the other approaches.

In scrum multiple, independent small teams organise themselves to work intensively (usually in sprints of about 30 days) to produce useable deliverables. We will consider the main steps in the approach and key players.

Key sprint steps

1. Vision / idea
   • As with all projects, there needs to be a vision - some perception of business benefit that can be achieved. This will include a high-level risk assessment and estimation of costs and benefits to be achieved - it may also consider whether the agile / scrum approach is best for this project.
2. Requirements
   • Requirements are stated in user stories (See example ‘as an auditor’ at the top of this article). Each user story states the user, what is to be achieved, why and the acceptance criteria. These are summarised in a prioritised product backlog. 
3. Sprint planning
   • Requirements from the product backlog that are to be delivered in that sprint are known, selected and finessed. 
4. Sprint iteration
   • Procedure for selecting requirement will be designed, built, tested and deployed.
5. Daily sprint
   • A short daily meting to identify any blockages and seek their resolution.
6. Review
   • Often a show and tell - is where the product is demonstrated to key stakeholders.
7. The retrospective
   • A private team meeting for lessons learnt during the sprint and how these can be applied top future sprints
8. Start next sprint (from step 3).

Key sprint players

1. Product owner (PO)
   • The main business representative - embedded within the team who owns the product backlog / deliverable, sets priorities and is responsible for ensuring actual return on investment or business benefit is achieved.
2. Scrum master
   • Acts as a team coach to help the PO and team with applying the scrum approach, ensuring resolution of risk and issues.
3. Scrum team
   • Provide the technical delivery capability and subject matter experts.

How do you audit ‘agile’?

Like all audits, an agile project audit will have the following main phases:

• preparation and planning
• fieldwork and evidence gathering
• QA review
• report publication.

However, the timing and conduct of each of these may differ for an agile project.

Preparation and planning

The auditor should become familiar with their organisations agile approach / policy, methodology / standards etc (even if only in the form of training materials).  This will inform the audit and provide an authority for any findings, including compliance with this process - if these artefacts are not covered - there’s your first audit finding.

Next perform a risk assessment of the project to identify the key audit objectives.  The standard risks for any project still apply, however the impact and likelihood for these will be different:

• cost - in theory agile fixes the costs and any new functionality should only be added when functionality with a lower benefit is removed. Early releases will give an indication of the accuracy of the effectiveness of delegation of budgets
• timing - by looking at planned dates and lengths of scrums etc the auditor will be able to see if releases are effective. Also, as each release is made a burn down rate will show how the product backlog is being cleared by the iterations to date.
• quality of deliverable - by discussion with the product owner and review of functionality delivered the auditor will be able to confirm that useable products are being generated. 

In addition to the above the auditor may wish to consider:

• agility risk - is there evidence of flexibility in approach to meet changing requirements
• technical debt - have some parts of the functionality been delayed to a later stage (including security and control features) - which may result in hidden costs.

Finally agree the terms of reference, including the timing and reporting arrangements. A waterfall audit approach will likely be based upon the key stage gates of the project (e.g. end of design, pre-go live). For an agile project these will be at different times for different releases / iterations or sprints. By the time the last sprint has started detailed planning or design most of the product should be issued and available. Hence a different approach is required - one based on little and often auditing. 

Fieldwork and evidence gathering

As the auditor can expect less documentation there is a need for greater observation and judgement. Evidence may take the form of whiteboards (a camera is useful), slide packs of show and tell sessions, emails of meeting findings.

It is important for the auditor to ensure that this include all regulatory, control or security issues - if they are not included they are unlikely to be built - which could lead to unnecessary risk issues or delays later.

QA review

These can often delay an audit - we need to be agile in our own approach. For example, if we discuss the factual accuracy of the findings with scrum master and product owners, with sufficient caveats, they will probably start to implement these during this or the next iteration - thereby making our audit more effective even before the report is issued.

Report

The report should be brief and ideally in a format to follow the audit approach (for example, user stories can help convey requirements).

About the author

Christopher Wright BSc(hon), CISA, MBCS, MAPM

A certified ScrumMaster, Chris has over 35 years’ experience of providing audit and IT advisory and risk management services, and is a qualified accountant. Sector specialisations include aviation and travel, oil and gas and the public sector. For the past 12 years he has been an independent consultant specialising in GRC for major enterprises. During this time he has seen a significant change from traditional to agile project management and has developed a number of techniques and tools to provide effective audit, control and governance frameworks within these revised approaches. 

He has run agile audit training for a variety of organisations, has spoken at a number of UK and international conferences on agile and has published books including Agile Governance and Audit.