Learning from lessons: a quality perspective

1. Scope

The aim is to provide a methodology for generating learning from lessons encountered during project work which can be relevant to all sizes of organisations, including complex ones that happen in multiple, separate phases. Ideally, this can result in learning shared by all entities participating in a project.

This article also looks at how an emerging technique for processing knowledge, Artificial Intelligence (AI), may be harnessed for systematic learning.

The idea is that data from successful activities (‘Good Practice’) be included in the analysis, not just that from ‘Things gone wrong’. In this way, the results provide a more rounded picture. It can also help ensure that what the entity is good at is taken forward for future activities (promoting the organisation’s culture and knowledge).

2. Introduction

This is an article about how to harness the information from lessons experienced, particularly in a construction context. It looks at how this may be done in relation to the coming availability of new technology, i.e. Artificial Intelligence.

In the wider construction industry, ‘Lessons learned’ is a colloquial term often used to describe the outcome of a review. It is contended here that, for organisations, ‘lessons learned’ are only lessons until the knowledge extracted is applied in all the areas where it is relevant and retained there for future reference (i.e. stays learned).

Many people will be aware of, and use, the Plan-Do-Check-Act cycle (PDCA) [1] as a management tool through which to produce and deliver goods and services. Whilst it is necessary to capture learning at all these stages, it is especially true at the ‘Act’ part of the cycle where the intention is that the learning gained be fed back to the ‘Plan’ phase to enable continuous improvement on future work. However, there can, in less diligent practice, be an unplanned break after ‘Act’ where lessons received, both during and at the close of a project, are not properly fed back into the planning of subsequent work, so do not become learned by the organisation.

This article will address this aspect of actually learning from lessons by looking at methods arising from the structure in Table 1 below.

Table 1 – Three steps to learning from lessons experienced.

In doing so, it is intended to cover how quality approaches to learning can take advantage of the new industrial age brought about by Information Technology, known as ‘Industry 4.0’ [2] and its associated ‘Quality 4.0’ [3] which refers to, ’The leveraging of technology with people to improve the quality of an organisation, its products, its services and the outcomes it creates’. Specifically, the article will consider the potential of Artificial Intelligence (AI) and its sub-set Machine Learning (ML) for data processing to generate learning from lessons received.

Broadly, the idea is to discuss how to systemise an organisation’s collective memory about its performance such that the knowledge within is available and easily retrievable where and when required.

3. How data for lessons can be captured

a) Acquire the lessons

Information on lessons doesn’t need to arise from ‘Things gone wrong’ alone, inputs can be elements from Technical Papers and Best Practice Papers, for instance. These can be papers from previous experience already published or those emerging as a result of learning on the current project.

Gathering data would include lessons, both positive and negative, to feed forward to the benefit of future similar projects. These would include all levels of project execution and may be graded as to their considered importance.

Acquiring the lessons can be difficult. People naturally do not want things that may have gone wrong to be scrutinised, e.g. through embarrassment. Therefore, establishing a culture that is open to sharing this information is essential. Promising that lessons will be anonymised when the information is published can help build confidence.

In the construction industry, a Quality Manager approaching this for the first time could find sources for lessons amongst:

• Incident reports
• Inspection and test results;
• Snag reports;
• Maintenance records;
• Warranty claims;
• Review minutes;
• Nonconformity reports and
• Audit observations.

They can also result from feedback, e.g. complaints, plaudits, customer satisfaction interviews or internal recommendations.

Furthermore, sources can be the result of positive actions such as a citation for something done that exceeds expectation, if the result pays back for the effort made and therefore produces added value.

Often it is the written responsibility of the new Project Manager/Director to address any lessons from previous similar projects, usually by reference to any preceding project reports.

Methods for capturing information can be electronic as well as physical, e.g. using hand-held mobile technology for inspection and testing purposes.

Furthermore, AI has the capability to incorporate imagery pertaining to lessons, e.g. from a drone. It may include geographical information or survey data, or just photographs. Conceivably information from drawings could also be input as could calculations associated with information on the drawings.

Lessons gained from quality incidents may have been recorded for future reference in a library (physical or electronic data management system).

The challenge then is to draw from this information its maximum value and make the knowledge arising (especially actions) available to people where they are working, and for it to be available in a participative format that engages with them in their role, e.g. using handheld device, mobile phone, iPad etc.

b) Approach to capturing knowledge

The learning on a project is usually the responsibility of the Project Manager/Director but often the generation of such reports is delegated to the Project Quality Manager.

There needs to be a structured approach to capturing knowledge that is evident to everyone in the organisation and involves them, otherwise learning outcomes from the process will never be truly maximised.

Publishing a Knowledge Plan is a way of establishing how different types of information from lessons can be shared for the benefit of the organisation by setting out a methodology for this. A good plan should provide a consistent approach to the gathering of data and information. It should:

• Identify where source data can be obtained across the organisation;
• Locating relevant data / papers published outside the organisation, e.g. by associated industry groups;
• In having the broadest trawl for information, it is recommended that a classification system be introduced to identify in what area the data / lesson may apply;
• Explain how knowledge is assimilated from data / lessons; and then,
• How it is carried forward to where it is needed (work positions).

Furthermore, having a Knowledge Plan to gather accurate data that is unexpurgated is vital as in section 4 where the value of AI for data processing is described. Machines enabled by AI to make decisions require structured and ‘clean’ data to reach logical conclusions.

Therefore, to maintain that accuracy, the framework of the Knowledge Plan must provide transparency around the process for managing learning from lessons activity. It may cover:

• How to submit a lesson;
• How to submit the data relevant to the lesson, or identify where it can be found;
• Triage procedure for assessing the value and relevance of the lesson together with the availability of applicable data;

• Unique identification for each lesson for traceability, and use of keywords to tag the information, e.g. to facilitate processing and retrieval;
• Appointment of a lesson owner for each with responsibility for keeping the information up to date (change control);
• How to verify the lessons (input);
• How to establish where the output will be needed;
• How to verify the learning from lessons (output), e.g. for actions arising;

Assessing data: any lessons need to be considered as to whether they will be beneficial and written in such a manner that those engaged in following work can interpret and understand their value. Additionally, any data being fed forward needs to be considered as to its suitability for future projects and should not be too project-specific to the work in hand.

Data should be collated throughout the work and compiled into a Learning Report at the end of the project to avoid losing the advantages of the lessons. This is because by time the project draws to a close, the work is running down and the personnel involved moving on, it makes obtaining recall more difficult.

On large scale projects where the work being undertaken is under a similar programme of work, any lessons, positive or negative, may prove beneficial to other projects within the same programme.

4. How most learning value can be extracted from lessons

a) The potential of AI to generate further learning

You could use a traditional method for knowledge processing such as reporting from a database, but there is something comparatively new with the potential to extract greater value from analysis. AI offers the capability to identify links or trends in data more easily, and even to recommend actions.

Colorado State University Global [4] defines AI as a technology that allows machines and computer applications to mimic human intelligence, learning from experience via iterative processing and algorithmic training. AI can be thought of as being a form of intelligence that is used to solve problems, come up with solutions, answer questions, make predictions, or offer strategic suggestions.

Examples of AI products in the public eye would be the development of self-driving cars and introduction of smart assistants such as Siri and Alexa in the domestic environment. Use in healthcare to assist Doctors in diagnosis is very much a growth area.

AI is a branch of computer science. For the topic of this article, Learning from Lessons, the AI discipline best suited to evaluating data for the construction industry is ML. This is ‘training’ a machine how to make inferences and decisions based on past experience, i.e. to learn, which is done using algorithms and statistical models. This enables it to be used to identify patterns in the data and to analyse against historical data to infer the meaning of the resulting data points (i.e. generate information from the data). It can then reach a possible conclusion, e.g. from the identification of a trend. This is done quickly without having to involve humans in the processing which saves labour time, particularly at scale, and also avoids the possibility of unintended influence in the process.

The algorithms programmed for use in ML can change and evolve according to the interactions they have with the data processed. This facilitates the learning aspect of ML. It becomes possible that, after use, the Programmer will no longer recognise the original algorithm they wrote.

MIT Management Sloan School [5] describes the functions of a machine learning system as:

With information input from lessons, ML can make connections between them, e.g. their frequency and type, which would enable the full extent of problem areas to be identified.

ML may also solve problems more accurately and thereby create opportunities for innovation.

AI software is becoming more affordable for businesses at least, as development has advanced and there is competition amongst major vendors like Amazon, Google, IBM and Microsoft. Their AI products, such as IBM’s Watson [6], can be scalable for different types of organisation and can be deployed on The Cloud.

For the purposes of learning by extracting information from data processing, AI could suit large companies and projects who deal with a significant amount of performance data, especially those around the wider construction industry who manage their own and supplier data.

b) Some points to be borne in mind when using ML for data processing:

1. 'Data Challenge’: The key challenge is turning the experience gained from lessons into data for subsequent input to ML, and then on to data processing to generate output that can be learned from. If it were economics, we would be talking about ‘monetising’ the asset, so, in the case of lessons, it is about converting the experience of a lesson into structured information using ML. It is not sufficient to capture the narrative of the lesson and actions taken, all the data surrounding it must be harvested. It is also the reason a triage procedure is necessary before data input to ML.
2. Governance: Significant governance is required to verify that the information from the lessons is accurate, valid (e.g. up to date), comes from a recognised source and is being used correctly. To ensure data integrity and the effective operation of the machine, a Moderator or ‘Knowledge Agent’ should be appointed to provide leadership, not least to manage change control. Custodians can be nominated for different types of data (to act as advocates or champions), preferably people with experience in the subject area concerned to bring authority to their message. Top management commitment to the use of ML is important for it to be implemented systematically. Such commitment may include creating guidelines for using artificial intelligence tools.
3. Privacy: It can be appropriate to anonymise data before entering it in the machine to avoid a data protection issue, e.g. where the company may be asked by a third party to provide a disclosure under the UK Freedom of Information Act 2000 [7]. For instance, the lesson learned entered in the machine may include the fact that a certain type of pipe did not work in a particular situation, but it should not say that supplier ABC Pipeworks products are not effective. Furthermore, permission is likely to be needed from clients or suppliers to use and share their data. Sharing of data is most beneficial within an organisation where similar types of work/projects are known to follow. However, discretion needs to be applied where similar work is being undertaken with more than one client since they may not wish to share such data with other clients or competitors. Sharing on an industry-wide basis would need the sanction of all parties involved. Similarly, if sharing outside the organisation, protecting the organisation’s Intellectual Property is a consideration.
4. Legal: Also, care needs to be taken where there is a lesson that is supplier-related that could have the potential to impact future contract awards. This can be interpreted as ‘Blacklisting’ which may have legal consequences. When such an issue is identified, a high degree of generalisation can occur which may water-down the lesson to the extent that it becomes meaningless.
5. Security: An AI solution must have in-built security features, e.g. to allow commercially sensitive data to be processed. Management arrangements need to be considered, for instance when there is collaborative working to establish whether the information which may not be shared without executive approval has been defined - Intellectual Property and proprietary (owned) data could be examples.
6. Evaluation: Addressing the risk that machine output may not be dependable (or ‘Human versus Robot’). AI development, including ML, is not at a stage where its output can be relied upon without question. Its results are only as good as the input data and the algorithm itself. Thus there needs to be evaluation (moderation) of the output by a human being, e.g. to reinforce conclusions. It is still normal practice for a workshop to be held, made up of technical experts from the disciplines involved, to analyse a problem using their professional judgement, at which the output of Machine Learning can be from one of several tools used.

5. How the knowledge resulting can be consistently applied at relevant work positions, to future activities and for innovation

Trends from ML can be regarded as knowledge threads which may apply to one or more areas of an organisation’s activity. Getting this knowledge to each area concerned is not only a challenge in communication but also begins the need to retain the information so that it is available to be applied to future work.

a) Apply the learning at relevant work positions

In the construction industry, the output from ML can form an input to Technical Reviews or Design Reviews thereby carrying forward past learning.

Other stages in work procedures where key points from lessons can be surfaced include:

• As criteria on inspection and testing check sheets;
• In audits, through questions incorporated in the audit questionnaires;
• Procurement activity;
• Contact award process.

Therefore, reviews, inspection and testing and audits become types of ‘knowledge renewal opportunity’.

Learning can also be published on electronic devices such as tablets where they are used by people involved in the work.

‘Lessons from learning’ information points can be established at workstations whether they be in the design office or on a construction site. These could take the form of an interactive touch screen electronic display.

Key sections of the Knowledge Plan could also be promulgated there such as how to report a lesson.

The reader may be familiar with ‘Safety Stand-Downs’ often used to brief teams on the causes of an incident and explain the actions necessary to prevent it happening again. In similar vein, ‘Knowledge Stand-downs’ could be held to pass on learning from other types of lesson at the workstation or, in the construction industry, this could form part of a programme of ‘Toolbox Talks’.

b) Improve and innovate

The importance of ML information to the design review process is mentioned in para 5a) above. Nonetheless, this information has equal value as input to design where an organisation wants to innovate to differentiate a product or service from others.

It is a given that organisations must evolve in order to survive in often fast changing operational environments. To evolve it is necessary to improve and innovate, none more so than in the design process whether it be design of temporary or permanent works. It is the ability to generate information valuable to the design process that can lay the foundation to identify areas where improvement and innovation is possible.

6. How this can be applied within complex multi-stage projects or programmes

This article aims be relevant for all sizes of projects or programmes, from those of single organisations to complex, multi-discipline, multi-stage ones.

The big advantage of using AI is its ability to process large volumes of data, something complex projects generate. However, that comes with a challenge to integrate data sets from multiple, loosely connected IT sources, and to use the resulting learning effectively.

a) Data compatibility

The importance of cohesion between companies’ IT platforms (data compatibility) needs to be understood in advance of sharing for there to be coordination between the parties.

‘Clean’ data sets are also necessary to enable sharing across different operations, if reliable information is to be generated.

Ideally, a shared IT knowledge platform is required for efficiency, if the project budget allows.

The system chosen must have adequate storage capability to cope with expansion as the overall project or programme develops.

b) Retention and retrieval

There also needs to be organisation and budget for managing the IT knowledge platform for data retention, particularly where there is an interruption in phases of the project.

Retrieval can initially be facilitated by categorisation of the information generated. This can be by type of work (such as discipline or activity, e.g. ‘Cladding), and by sub-categorisation where there is complexity.

Where there is a break in a phase of a project, it needs to be understood how the information can be restored for subsequent stages at workstations, including new locations.

Thought also needs to be given to incorporating the data of firms new to the project.

c) Training and awareness

Finally, there needs to be sufficient training in the use of the learning information so that people at different locations have a common understanding of its relevance and application.

To facilitate uptake of the learning it needs to be widely understood that the data on which it is based comes from previous successful activities (‘Good Practice’) not just ‘Things gone wrong’. It can also help ensure that what the entity is good at is taken forward for future activities (promoting the organisation’s culture and knowledge).

7. Conclusion

Fundamentally, lessons experienced (both positive and negative) present opportunities to learn. The learning, when harnessed, can then give the capability to act to do things better in future, and even innovate.

There is an important distinction between capturing the information from lessons and making use of it systematically to embed improvement. It is contended that it is only when the process for improvement is embedded that learning is fully taken advantage of. This article offers three steps to learning from lessons experienced.

Table 1 – Three steps to learning from lessons experienced.

For the Capitalise step, this article explains how the new technology of Artificial Intelligence (AI) and Machine Learning (ML) has the potential to augment the learning process in the field of data processing in the construction industry.

To get reliability from the output of ML the most important aspect is the ‘Data Challenge’. This is turning the experience gained from lessons into data for subsequent input to ML, and then on to data processing to generate output that can be learned from. It is not sufficient to capture the narrative of the lesson and actions taken, all the data surrounding it must be harvested. See para 4bi) above.

In the construction industry, the ‘Quick win’ with which to start using ML is for processing inspection and test data, especially where that data is already digitised.

The ambition is to systemise the organisation’s collective memory about its performance such that the knowledge within is available and easily retrievable where and when required.

For the purposes of learning by extracting information from data processing, AI could suit large companies and projects who deal with a significant amount of performance data, especially those around the wider construction industry who manage their own and supplier data.

Reference sources:

• [1] Wikipedia Plan-Do-Check-Act cycle (PDCA)
• [2] Wikipedia – Fourth Industrial Revolution – ‘Industry 4.0’
• [3] The Chartered Quality Institute – ‘Quality 4.0’
• [4] Colorado State University Global defines AI as a technology that allows machines and computer.
• [5] MIT Management Sloan School describes the functions of a machine learning system as:
• [6] AI products, such as IBM’s Watson
• [7] Freedom of Information Act 2000 (UK) - Freedom of Information Act 2000 (legislation.gov.uk)

This article was originally written by Kevin Rogers on behalf of the CQI Construction Special Interest Group, reviewed by Colin Harley and Giorgio Mannelli, members of the Competency Working Group and approved for publication.

--ConSIG CWG

Related articles on Designing Buildings

• Construction quality.
• Commissioning.
• Design freeze: a quality perspective.
• Failure modes and effect analysis (FMEA).
• Lessons learned.
• Quality.
• Quality assurance.
• Quality culture and behaviours.
• Risk assessment.
• Samples.
• Specification.