rfp.html

---
title: The Institute for Ethical AI & Machine Learning
description: The Institute for Ethical AI & Machine Learning is a Europe-based research centre that brings togethers technologists, academics and policy-makers to develop industry frameworks that support the responsible development, design and operation of machine learning systems.
---
<html>
	<head>
		{% include header.html %}


	</head>
	<body>
		<div id="page-wrapper">

			{% include navbar.html %}

			<!-- Main -->
				<div id="main" class="wrapper style1">
					<div class="container">
						<header class="major">
							<h2>AI-RFX Procurement Framework v1.0</h2>
							<p>Online Version for the AI Request for Proposal Template, AI & Machine Learning Solutions.</p>
						</header>


						<!-- Text -->
							<section>
<h2 class="western"><a name="docs-internal-guid-c1391efa-7fff-26d1-d74b-991b5583401a"></a>
0 - Introduction</h2>
<hr>
<h3 class="western">0.1 - Overview</h3>
<p>The “AI Request for Proposal v1.0” is part of the <a href="https://ethical.institue/rfx.html">AI-RFX
Procurement Framework</a>. The objective of this document is to
empower industry practitioners looking to procure AI systems. It
should serve as a guide and a reference when putting together a
tender or when procuring a machine learning solution. This template
should also serve as a guide for suppliers that are looking to put
together a proposal for a machine learning solution. This way
suppliers can ensure they are following best practices in responsible
machine learning development. 
</p>

<p>This AI-RFP template has been built using our “Machine Learning
Maturity Model” as its core, which is <a href="https://ethical.institute/rfx.html#model">available
for download</a> at our website. The “Machine Learning Maturity
Model” goes beyond the machine learning algorithms themselves, and
provides an assessment criteria to evaluate the maturity of the
infrastructure and processes around the algorithms. The concept of
“Maturity” is not just defined as a matter of technical
excellence, scientific rigor, and robust products. It also
essentially involves responsible innovation and development
processes, with sensitivity to the relevant domains of expert
knowledge and consideration of all relevant direct and indirect
stakeholders. 
</p>

<p>All the frameworks under the AI-RFX Procurement Framework,
including the “AI Request for Proposal”, the “Machine Learning
Maturity Model”, and the “AI Tender Competition” templates have
been released as <a href="https://docs.google.com/document/d/1ecJPiCAc0EkAbU_L5f2yxgucAfThynQLc1Qn37jR4Gg/edit#heading=h.5o0sbykhosr6">fully
open source</a>. This allows suppliers and companies to contribute
and update these templates, helping raise the bar for AI safety,
quality and performance. 
</p>
<hr>
<h3 class="western">0.2 - About Us</h3>
<p>The Institute for Ethical AI &amp; Machine Learning is a Europe-based
research centre that carries out world class research into
responsible machine learning systems. We are formed by cross
functional teams of applied STEM researchers, philosophers, industry
experts, data scientists and software engineers.</p>

<p>Our vision is to mitigate risks of AI and unlock its full
potential through frameworks that ensure ethical and conscientious
development of intelligent systems across industrial sectors. We are
building the Bell Labs of the 21st Century by delivering breakthrough
contributions through applied AI research. You can find more
information about us at <a href="https://ethical.institute/">https://ethical.institute</a>.</p>
<hr>
<h3 class="western" >0.3 -
Motivation
<img src="images/MLENG-3.png" name="Image1" width="500" height="500" border="0" style="float:right" align="bottom">
</h3>
<p>There is currently a growing number of companies that are working
towards introducing machine learning systems to automate critical
processes at scale. This has required the “productisation” of
machine learning models, which introduces new complexities. This
complexity revolves around a new set of roles that fall under the
umbrella of “Machine Learning Engineering”. This new set of roles
fall in &nbsp;the intersection between DevOps, data science and
software engineering.</p>

<br style="clear: both">
<img src="images/MLENG-4.png" name="Image2" width="500" height="500" style="float:left" border="0" align="bottom">
<p><br>
</p>
<p>To make things harder, the deployment of machine learning
solutions in industry introduces an even bigger complexity. This
involves the intersection of the new abstract “Machine Learning
Engineering” roles, together with the industry domain experts and
policy makers.</p>

<p>Because of this, there is a strong need to set the AI &amp; ML
standards, so practitioners are empowered to raise the bar for
safety, quality and performance around AI solutions. The AI-RFX
Procurement Framework aims to achieve the first steps towards
this.&nbsp;&nbsp;&nbsp; 
</p>

<br style="clear: both">
<hr>
<h3 class="western">0.4 - How to use this document</h3>
<h4 class="western">0.4.1 - Using as reference</h4>
<p>Many procurement managers may already own internally-approved RFP
templates or RFP portals with standard entries. If that is the case,
this document can be treated as a reference to obtain insights from
key sections specific to machine learning, such as “<a href="https://docs.google.com/document/d/1ecJPiCAc0EkAbU_L5f2yxgucAfThynQLc1Qn37jR4Gg/edit#heading=h.lqzrc25boiog">Chapter
3: Tender Process</a>” and “<a href="https://docs.google.com/document/d/1ecJPiCAc0EkAbU_L5f2yxgucAfThynQLc1Qn37jR4Gg/edit#heading=h.84prvsfhx8jz">Chapter
4: Detailed Scope</a>”. The machine learning specific sections in
this document include:</p>
<ul>
	<li>
<p ><a href="https://docs.google.com/document/d/1ecJPiCAc0EkAbU_L5f2yxgucAfThynQLc1Qn37jR4Gg/edit#heading=h.3498ud5xwm8">2.5
	- Benchmarks &amp; Metrics</a></p>
	</li><li>
<p ><a href="https://docs.google.com/document/d/1ecJPiCAc0EkAbU_L5f2yxgucAfThynQLc1Qn37jR4Gg/edit#heading=h.856p9h8sb15f">3.2
	- Assessment criteria</a></p>
	</li><li>
<p ><a href="https://docs.google.com/document/d/1ecJPiCAc0EkAbU_L5f2yxgucAfThynQLc1Qn37jR4Gg/edit#heading=h.j80rdz2d169d">3.5
	- Project timelines</a></p>
	</li><li>
<p ><a href="https://docs.google.com/document/d/1ecJPiCAc0EkAbU_L5f2yxgucAfThynQLc1Qn37jR4Gg/edit#heading=h.3kr1iqu4gs2f">4.1
	- Competition</a></p>
	</li><li>
<p ><a href="https://docs.google.com/document/d/1ecJPiCAc0EkAbU_L5f2yxgucAfThynQLc1Qn37jR4Gg/edit#heading=h.6bx55ewm3guw">4.4
	- Implementation</a></p>
	</li><li>
<p ><a href="https://docs.google.com/document/d/1ecJPiCAc0EkAbU_L5f2yxgucAfThynQLc1Qn37jR4Gg/edit#heading=h.juw7ka3mgoc">4.7
	- Required service levels</a></p>
</li></ul>
<h4 class="western">0.4.2 - Structure</h4>
<p>The document consists of multiple chapters, each chapter with a
set of sections. The text in the sections is meant to be replaced
completely, and it consist of:</p>
<ul>
	<li>
<p >An explanation of what the
	section should contain</p>
	</li><li>
<p >An example of what it the
	section could look like</p>
</li></ul>
<h4 class="western">0.4.3 - Example</h4>
<p>Each section has an example. The example is consistent throughout
the document, and it is around a fictitious company called
“Manufacturing Group Limited”. The objective is to provide a
broad and generic example that covers multiple useful areas which
companies can relate to. In this specific example, the company is
looking for a machine learning system that performs predictive
maintenance and data analysis automation on a fictitious end-to-end
internal process.</p>
<h4 class="western">0.4.4 - When to use</h4>
<p>This template is relevant primarily for the procurement of machine
learning systems, and hence it is only relevant when looking to
automate a process that involves data analysis that is too complex to
be tackled using simple RPA tools or rule-based systems. 
</p>
<hr>
<h3 class="western">0.5 - Template vs Reality</h3>
<p>This document should serve as a guide. Especially for smaller
projects, the level of detail required may vary significantly, and
some sections can be left out as required. This template attempts to
to provide a high level overview on each chapter (and respective
sections) so the procurement manager and suppliers can provide as
much content as reasonable. 
</p>

<p>Each company putting together a procurement process will have
different set of requirements, and unfortunately this is not a
“one-size-fits-all” document. The main purpose of this document
is to raise the bar both in industry and technology, and provide key
industry stakeholders with a solid basis to start from. Also, it’s
free.</p>
<hr>
<h3 class="western" >0.6 - Open
Source License - Free as in freedom</h3>
<h4 class="western">0.6.1 - Open source License</h4>
<p>This document is open source, which means that it can be updated
by the community. The motivation to release this as open source is so
that it is continuously improved by the community. This will ensure
that the standards for safety, quality and performance of what is
expected in machine learning systems will keep increasing, whilst
being kept in check on a realistic level by both suppliers and
companies.</p>
<h4 class="western">0.6.2 - Contributing.md</h4>
<p>The Institute for Ethical AI &amp; Machine Learning’s AI-RFX
committee is in charge of the contributing community for all of the
templates under the AI-RFX Procurement Framework. &nbsp;Anyone who
would like to contribute, add suggestions, or provide example and
practical uses of this template, please contact us through the
website, or send us an email via <a href="mailto:a@ethical.institute">a@ethical.institute</a>.</p>
<h4 class="western">0.6.3 - License</h4>
<p>This document is registered under <a href="https://github.com/EthicalML/ai-rfx-procurement-framework">this
MIT License</a> (<a href="https://raw.githubusercontent.com/EthicalML/ai-rfx-procurement-framework/master/LICENSE">raw
file</a>), which means that anyone can re-use, modify or enhance this
document as long as credit is given to The Institute for Ethical AI &amp;
Machine Learning. It also includes an “as is” disclaimer. Please
read the license before using this template.</p>

<p ><br>
<br>

</p>
<hr>
<h2>Request for Proposal Template Begins Here:</h2>
<h2>Example company name: MANUFACTURING GROUP LIMITED 
</h2>

<p>Example AI-RFP for Machine Learning Automation System</p>

<p>Issued on: 00/00/2019</p>
<p>Issued by: Person’s Name</p>
<p>Reference Number: #REFNUMBER</p>

<h2 class="western" >1 - Our Company</h2>
<p>This chapter contains a high level overview about the company that
is setting up the tender process.</p>
<hr>
<h3 class="western">1.1 - Company Overview</h3>
<p>This section should provide a brief overview of the company. It
may also be worth adding insights on the company’s commitments and
efforts in the field of AI. 
</p>

<table >
	<colgroup><col width="0*">
	</colgroup><tbody><tr>
		<td  >
			<p>Example Company Overview for “Manufacturing Group Limited”</p>
			
			<p>Manufacturing Group Limited is an OEM manufacturing company
			that was set up in 1992, we are a large manufacturing group with a
			large number of suppliers. We operate machinery across over 10
			countries across the world. The quality of our machines is the
			highest amongst our competitors, and the processes we have created
			around maintenance &amp; operations are the most advanced in the
			industry. We have a commitment of £100m during the next 5 years
			to introduce machine learning into our company to improve the
			quality and efficiency of our processes, and enable our staff to
			focus on more important higher level tasks.</p>
		</td>
	</tr>
</tbody></table>

<hr>
<h3 class="western">1.2 - Key Facts</h3>
<p>This section should contain 5-10 key facts about the company.
Similar to the brief overview, it may be worth adding some key facts
about the current focus as an organisation towards the area of
artificial intelligence and machine learning. 
</p>

<table >
	<colgroup><col width="0*">
	</colgroup><tbody><tr>
		<td  >
			<p>Example Key Facts for “Manufacturing Group Limited”</p>
			
			<ul>
				<li>
<p >We were established in
				1992</p>
				</li><li>
<p >We have 20 offices
				across 10 countries</p>
				</li><li>
<p >We operate over 20,000
				machines world wide</p>
				</li><li>
<p >Our vision is to drive
				the way in machinery through quality and innovation</p>
				</li><li>
<p >Our mission is to
				operate and maintain 50% of machines in the world</p>
				</li><li>
<p >...</p>
			</li></ul>
		</td>
	</tr>
</tbody></table>

<hr>
<h3 class="western" >1.3 - Values</h3>
<p>List your company values as relevant to ensure suppliers are
aware.</p>

<table >
	<colgroup><col width="0*">
	</colgroup><tbody><tr>
		<td  >
			<p>Example Values for “Manufacturing Group Limited”</p>
			
			<ul>
				<li>
<p >Respect 
				</p>
				</li><li>
<p >Transparency</p>
				</li><li>
<p >Responsibility</p>
				</li><li>
<p >Knowledge</p>
				</li><li>
<p >Humbleness</p>
			</li></ul>
			
		</td>
	</tr>
</tbody></table>

<hr>
<h3 class="western">1.4 - Responsibility Commitment</h3>
<p>This section requests suppliers to ensure they follow the 8
Principles for Responsible Machine Learning development. The 8
commitments are implemented in this framework through the Machine
Learning Maturity Model, and they are implicit on each section.</p>

<p>We suggest you use a variation of the default example provided
below.</p>

<table >
	<colgroup><col width="608">
	</colgroup><tbody><tr>
		<td  >
			<p>Example Responsibility Commitment</p>
			
			<p>This request for proposal requires all suppliers to follow the
			8 principles for responsible machine learning defined by The
			Institute for Ethical AI &amp; Machine Learning. The 8 principles
			state best practices in each area, and as it is stated, they don’t
			always have to be implemented in every project, but when this is
			the case it should be an explicit decision not to.</p>
			
			<table >
				<tbody><tr>
					<td  >
						<p>1 - Human Augmentation</p>
					</td>
					<td  >
						<p>2 - Bias Evaluation</p>
					</td>
					<td  >
						<p>3 - Explainability by justification</p>
					</td>
				</tr>
				<tr>
					<td  >
						<p>4 - Reproducible operations</p>
					</td>
					<td  >
						<p>5 - Displacement strategy</p>
					</td>
					<td  >
						<p>6 - Practical accuracy</p>
					</td>
				</tr>
				<tr>
					<td  >
						<p>7 - Trust by privacy</p>
					</td>
					<td  >
						<p>8 - Security risks</p>
					</td>
					<td  >
						
					</td>
				</tr>
			</tbody></table>
			
			<p>You can find the Machine Learning Principles by The Institute
			for Ethical AI &amp; Machine Learning at
			<a href="https://ethical.institute/principles.html">https://ethical.institute/principles.html</a>
						</p>
		</td>
	</tr>
</tbody></table>

<hr>
<h3 class="western">1.5 - External Links</h3>
<p>This section should contain links relevant for suppliers and
solution providers. Some sample links are provided which reference to
some of the points covered in this document, including website
address, reference to background knowledge, and links to domain
specific requirements.</p>

<table >
	<colgroup><col width="0*">
	</colgroup><tbody><tr>
		<td  >
			<p>Example Process Overview for “Manufacturing Group Limited”</p>
			
			<ul>
				<li>
<p >Company website:
				<a href="https://ethical.institute/">https://ethical.institute</a></p>
				</li><li>
<p >The 8 Machine Learning
				Principles: <a href="https://ethical.institute/principles.html">https://ethical.institute/principles.html</a></p>
				</li><li>
<p >...</p>
			</li></ul>
		</td>
	</tr>
</tbody></table>


<h2 class="western" >2 -
Requirements Overview</h2>
<p>The objective of this chapter is not to propose the solution
desired, but to provide details on the end-to-end process and/or
challenge identified within the company. As mentioned in the
introduction, given that this RFP is specific to machine learning
solutions, the ideal way to explain the challenge/process is through
the flow of data across the entire process, including
transformations, stakeholders involved, systems, and data-sources.
This is because machine learning solutions are used to automate the
analysis of data, and hence this would provide suppliers with a good
understanding on all the steps that surround the data itself.</p>

<p>This chapter has been put together using the <a href="https://ethical.institute/rfx.html#model">Machine
Learning Maturity Model</a> assessment criteria. Each section takes
into consideration each of the criteria implicitly. It is recommended
to take into consideration all the 8 assessment criteria when putting
together this section.</p>
<hr>
<h3 class="western">2.1 - Key objectives &amp; outcomes</h3>
<p>This section is one of the most important in this chapter. This
section should provide a high level overview of what are the tangible
outcomes desired as outputs of a successful tender process. The idea
is for this section to provide high level objectives / KPIs. Later
on, <a href="https://docs.google.com/document/d/1ecJPiCAc0EkAbU_L5f2yxgucAfThynQLc1Qn37jR4Gg/edit#heading=h.3ehiifm25uca">Section
2.6 - Target improvements and metrics</a> focuses on lower-level
quantifiable metrics that relate to each of the high level objectives
/ KPIs in this section. 
</p>

<p>Points to include in this section include:</p>
<ul>
	<li>
<p >High level reason why
	process/challenge is being tackled</p>
	</li><li>
<p >Key improvement areas (e.g
	domain expert time freed, reduction of errors, improvement of
	inefficient process, etc)</p>
	</li><li>
<p >Type of high level services
	or qualities that are sought in suppliers being procured</p>
</li></ul>

<table >
	<colgroup><col width="0*">
	</colgroup><tbody><tr>
		<td  >
			<p>Example Aims &amp; Objectives for “Manufacturing Group
			Limited”</p>
			
			<ul>
				<li>
<p >Significantly free time
				from our domain-knowledge experts so they are able to focus on
				high level tasks</p>
				</li><li>
<p >Obtaining a better
				insight on the data from our machines in order to enhance
				existing processes through innovative capabilities such as
				predictive maintenance</p>
				</li><li>
<p >Identify a technology
				provider that can offer a flexible solution that can be used in a
				wider set of automation contexts</p>
				</li><li>
<p >As optional extra, build
				long term relationship with supplier that can help us build data
				science and machine learning capabilities internally 
				</p>
				</li><li>
<p >...</p>
			</li></ul>
		</td>
	</tr>
</tbody></table>

<hr>
<h3 class="western">2.2 - Process/challenge overview</h3>
<p>This section should contain a concise and high level description
of the process / challenge that the organisation has identified which
needs to be automated or improved. As mentioned at the beginning of
the chapter, the overview should focus on the flow of the data across
the steps throughout process. Once again, the reason for this is
because machine learning use-cases in industry are for (full or
partial) automation of data analysis -- whether for business insights
or process automation.</p>

<p>Providing in-depth information on the current flow of data will
allow suppliers to use their expertise to how and to what extent
their solution will be able to automate the process, whether full or
partial automation. 
</p>

<p>The key questions that should be covered in this section include:</p>
<ul>
	<li>
<p >What are the data-sets and
	data-sources involved in the process?</p>
	</li><li>
<p >Who are the stakeholders
	involved?</p>
	</li><li>
<p >What are the
	approximate/high-level volumes processed?</p>
	</li><li>
<p >How common are human/system
	errors?</p>
	</li><li>
<p >What are the inefficiencies
	that were identified?</p>
</li></ul>

<p>Below is an example of how these questions can be answered in a
high level summary.</p>

<table >
	<colgroup><col width="0*">
	</colgroup><tbody><tr>
		<td  >
			<p>Example Process Overview for “Manufacturing Group Limited”</p>
			
			<ul>
				<li>
<p >Machines Group Limited
				currently has identified a bottleneck in an inefficient process
				that is executed on a day to day basis 
				</p>
				</li><li>
<p >This process requires a
				lot of manual actions from various stakeholders, both domain
				experts and staff, to retrieve and analyse data</p>
				</li><li>
<p >The main process
				revolves around evaluating the status of machinery whenever it is
				necessary through image, text and numeric data</p>
				</li><li>
<p >There are multiple
				systems involved in carrying out this process</p>
				</li><li>
<p >At the end of every
				month, it is also necessary to write a report with all the
				results from the previous month</p>
				</li><li>
<p >Often there are errors
				in the data as well, which requires the data to be collected
				again, but identifying those errors is often too expensive</p>
			</li></ul>
		</td>
	</tr>
</tbody></table>


<hr>
<h3 class="western">2.3 - Process Detail</h3>
<p>This section covers the process into detail. It still focuses
mainly on the data-flow, but provides a much more in-depth overview
for the following 4 areas:</p>
<ol>
	<li>
<p >Dataset overview</p>
	</li><li>
<p >Stakeholder overview</p>
	</li><li>
<p >Workstreams overview</p>
	</li><li>
<p >Step by step workstreams
	breakdown</p>
</li></ol>

<h4 class="western">2.3.1 - Dataset Overview</h4>
<p>This subsection should provide information about the datasets
involved across the end-to-end process. There should be a brief
overview of the systems and/or data-sources where the datasets are
stored and/or processed. There should also be a high level overview
of what the datasets consist of. <a href="https://docs.google.com/document/d/1ecJPiCAc0EkAbU_L5f2yxgucAfThynQLc1Qn37jR4Gg/edit#heading=h.84prvsfhx8jz">“Chapter
4: Detailed Scope”</a> covers the systems and the datasets into
much more detail with samples and documentation where relevant.</p>

<p>The table below provides an example of 6 datasets across 3
systems:</p>

<table >
	<colgroup><col width="0*">
	</colgroup><tbody><tr>
		<td  >
			<p>Example Datasets Overview for “Manufacturing Group Limited”</p>
			
			<p>Datasets involved:</p>
			
			<table >
				<colgroup><col width="90">
				<col width="107">
				<col width="72">
				<col width="316">
				</colgroup><tbody><tr>
					<td >
						<p>Name</p>
					</td>
					<td >
						<p>Type</p>
					</td>
					<td >
						<p>System</p>
					</td>
					<td >
						<p>Description</p>
					</td>
				</tr>
				<tr>
					<td >
						<p>Photos of machines</p>
					</td>
					<td >
						<p>Image</p>
					</td>
					<td >
						<p>RoboSys</p>
					</td>
					<td >
						<p>This is a collection of photos that are taken from the
						machines once a day, and stored as a blob format. The image
						resolutions vary. The file types are PNG and JPG.</p>
					</td>
				</tr>
				<tr>
					<td >
						<p>Logs</p>
					</td>
					<td >
						<p>Semi-structured Text</p>
					</td>
					<td >
						<p>LogSys</p>
					</td>
					<td >
						<p>This dataset contains all the logs for the machine which are
						generated whenever any operation takes place. These are
						semi-structured text with a specific format.</p>
					</td>
				</tr>
				<tr>
					<td >
						<p>Building system metrics</p>
					</td>
					<td >
						<p>Numbers</p>
					</td>
					<td >
						<p>RoboSys</p>
					</td>
					<td >
						<p>These are the metrics that are continuously generated by the
						machine. The metrics include the machine temperature, room
						temperature, humidity and revolutions per second.</p>
					</td>
				</tr>
				<tr>
					<td >
						<p>Analysis results</p>
					</td>
					<td >
						<p>Numbers and structured text</p>
					</td>
					<td >
						<p>MyERP</p>
					</td>
					<td >
						<p>These are the results that are put together by the domain
						knowledge expert when analysing the metrics, logs and images.
						This includes the state of the machine, together with the
						results on whether it requires maintenance or review.</p>
					</td>
				</tr>
				<tr>
					<td >
						<p>Report</p>
					</td>
					<td >
						<p>Unstructured text</p>
					</td>
					<td >
						<p>MyERP</p>
					</td>
					<td >
						<p>This is the report that is put together by the domain expert
						every month with an overview of all the individual results
						submitted to the system.</p>
					</td>
				</tr>
				<tr>
					<td >
						<p>Scheduled notifications</p>
					</td>
					<td >
						<p>Structured text</p>
					</td>
					<td >
						<p>MyERP</p>
					</td>
					<td >
						<p>Notifications are sent when a machine requires a planned or
						unplanned evaluation</p>
					</td>
				</tr>
			</tbody></table>
			
		</td>
	</tr>
</tbody></table>

<h4 class="western">2.3.2 - Stakeholder Overview</h4>
<p>This subsection should provide an overview of the stakeholders
that are involved throughout the end-to-end process, including:</p>
<ul>
	<li>
<p >Name of the type of
	stakeholder</p>
	</li><li>
<p >Brief description</p>
	</li><li>
<p >Whether stakeholder is a
	domain expert or staff (more on this below)</p>
</li></ul>

<p>This subsection should outline which stakeholders are “Domain
Experts”. The “Domain Experts” are the stakeholders that use
domain-specific or expert knowledge to analyse data. Often the Domain
Experts need to be available for suppliers to answer questions, as
suppliers may need to obtain insight about how the data is analysed
or how the systems are used, so the right machine learning solution
can be proposed.</p>

<p>The example below covers 4 stakeholders, three of which are domain
experts and perform complex analysis on the datasets.</p>

<table >
	<colgroup><col width="0*">
	</colgroup><tbody><tr>
		<td  >
			<p>Example Stakeholder Overview for “Manufacturing Group
			Limited”</p>
			
			<p>Stakeholders involved in the process:</p>
			
			<table >
				<colgroup><col width="118">
				<col width="86">
				<col width="383">
				</colgroup><tbody><tr>
					<td >
						<p>Name</p>
					</td>
					<td >
						<p>Type</p>
					</td>
					<td >
						<p>Description</p>
					</td>
				</tr>
				<tr>
					<td >
						<p>Machine Evaluation Expert</p>
					</td>
					<td >
						<p>Domain Expert</p>
					</td>
					<td >
						<p>This is a domain expert that has specific knowledge on the
						machines, and has the capability to evaluate the performance of
						the machines</p>
					</td>
				</tr>
				<tr>
					<td >
						<p>Engineering Team</p>
					</td>
					<td >
						<p>Domain Expert</p>
					</td>
					<td >
						<p>Technical individual in charge of the maintenance and
						operation of the systems</p>
					</td>
				</tr>
				<tr>
					<td >
						<p>Delivery manager</p>
					</td>
					<td >
						<p>Domain Expert</p>
					</td>
					<td >
						<p>This person is involved in producing a report once a month</p>
					</td>
				</tr>
				<tr>
					<td >
						<p>Analyst</p>
					</td>
					<td >
						<p>Staff</p>
					</td>
					<td >
						<p>This person is in charge of collecting data</p>
					</td>
				</tr>
			</tbody></table>
			
		</td>
	</tr>
</tbody></table>

<h4 class="western">2.3.3 - Workstreams Overview</h4>
<p>The end-to-end process often may consist of multiple different
independent or dependent workstreams. This step requires an overview
of the workstreams that constitute the process. Each workstream
should contain the following information:</p>
<ul>
	<li>
<p >High-level description of
	each workstream</p>
	</li><li>
<p >Order in which workstreams
	are executed, or alternative whether they are not dependent on each
	other (i.e. parallel or sequence).</p>
</li></ul>

<p>Below is an example with 2 parallel workstreams:</p>

<table >
	<colgroup><col width="0*">
	</colgroup><tbody><tr>
		<td  >
			<p>Example Workstreams Overview for “Manufacturing Group
			Limited”</p>
			
			<p>In this process there are two workstreams identified, which are
			independent of each other and are executed in parallel, but at a
			different frequency.</p>
			
			<table >
				<colgroup><col width="85">
				<col width="86">
				<col width="412">
				</colgroup><tbody><tr>
					<td >
						<p>Order</p>
					</td>
					<td >
						<p>Name</p>
					</td>
					<td >
						<p>Description</p>
					</td>
				</tr>
				<tr>
					<td >
						<p>1 (parallel)</p>
					</td>
					<td >
						<p>Machine evaluation</p>
					</td>
					<td >
						<p>This process involves the machine experts and engineering
						team to review the data from the current machine, and submit
						the conclusion to the MyERP system.</p>
					</td>
				</tr>
				<tr>
					<td >
						<p>1 (parallel)</p>
					</td>
					<td >
						<p>Report creation</p>
					</td>
					<td >
						<p>The report creation requires the Report maker to extract all
						the machine evaluations carried out during the last month
						manually, and produce a report that summarises the results</p>
					</td>
				</tr>
			</tbody></table>
			
		</td>
	</tr>
</tbody></table>

<h4 class="western">2.3.4 - Step-by-Step Workstreams Breakdown</h4>
<p>This subsection should contain every step for each workflow
identified across the end-to-end process. Each step provided should
contain the following where relevant:</p>

<table >
	<tbody><tr>
		<td  >
			<p>#</p>
		</td>
		<td  >
			<p>Name</p>
		</td>
		<td  >
			<p>Description</p>
		</td>
	</tr>
	<tr>
		<td  >
			<p>1</p>
		</td>
		<td >
			<p>Stakeholders</p>
		</td>
		<td >
			<p>All the stakeholders involved in this step</p>
		</td>
	</tr>
	<tr>
		<td  >
			<p>2</p>
		</td>
		<td >
			<p>Datasets</p>
		</td>
		<td >
			<p>All the datasets that are retrieved, transformed or created -
			ideally one action per step</p>
		</td>
	</tr>
	<tr>
		<td  >
			<p>3</p>
		</td>
		<td >
			<p>Systems</p>
		</td>
		<td >
			<p>The systems that are used to retrieve, transform or save data 
			</p>
		</td>
	</tr>
	<tr>
		<td  >
			<p>4</p>
		</td>
		<td >
			<p>Actions</p>
		</td>
		<td >
			<p>Actions performed by stakeholders or systems, generally around
			the data, but also could be different</p>
		</td>
	</tr>
</tbody></table>

<p>(*) Don’t forget</p>
<p>It is important to make sure that there are no steps that are
missed -- especially the following points:</p>
<ul>
	<li>
<p >Detailed steps required to
	extract the data from the systems</p>
	</li><li>
<p >The domain expert
	analysis/transformations that are performed in the data involved</p>
	</li><li>
<p >New creation of datasets
	which could be as small as adding results to a system or writing a
	report</p>
</li></ul>

<p>The example below covers an overview of the first workstream
outlined in the example above.</p>

<table >
	<colgroup><col width="0*">
	</colgroup><tbody><tr>
		<td  >
			<p>Example Step-by-Step Workstreams Breakdown for “Manufacturing
			Group Limited”</p>
			
			<p>The granular step-by-step process for the FIRST workflow,
			machine evaluation is as follows:</p>
			
			<dl>
				<dd>
				<table >
					<colgroup><col width="34">
					<col width="88">
					<col width="445">
					</colgroup><tbody><tr>
						<td >
							<p>#</p>
						</td>
						<td >
							<p>Name</p>
						</td>
						<td >
							<p>Description - including dataset, system, stakeholders +
							actions</p>
						</td>
					</tr>
					<tr>
						<td >
							<p>1</p>
						</td>
						<td >
							<p>Machine requires evaluation</p>
						</td>
						<td >
							<p>When a machine requires a planned or unplanned evaluation,
							the analyst gets an automated notification from the MyERP
							system as an email to begin the collection. The notification
							is sent automatically from the MyERP system to the right email
							address containing the information of the machine number.</p>
						</td>
					</tr>
					<tr>
						<td >
							<p>2</p>
						</td>
						<td >
							<p>Case creation</p>
						</td>
						<td >
							<p>The analyst is then required to manually download all the
							files as follows:</p>
							<ol>
								<li>
<p >Log in to RoboSys</p>
								</li><li>
<p >Go to image portal</p>
								</li><li>
<p >Enter machine ID</p>
								</li><li>
<p >...etc</p>
							</li></ol>
						</td>
					</tr>
					<tr>
						<td >
							
							<p>3</p>
						</td>
						<td >
							<p>Notify evaluator</p>
						</td>
						<td >
							<p>Analyst sends email to machine evaluator with details for
							machine to be evaluated</p>
						</td>
					</tr>
					<tr>
						<td >
							<p>4</p>
						</td>
						<td >
							<p>Access case</p>
						</td>
						<td >
							<p>Machine Evaluator logs into MyERP portal, goes to case
							portal, and enters case ID. 
							</p>
						</td>
					</tr>
					<tr>
						<td >
							<p>5</p>
						</td>
						<td >
							<p>Metrics evaluation</p>
						</td>
						<td >
							<p>Machine evaluator performs initial assessment by looking at
							the metrics and trying to spot any anomalies. If any anomalies
							are found, they are written down.</p>
						</td>
					</tr>
					<tr>
						<td >
							<p>6</p>
						</td>
						<td >
							<p>Logs evaluation</p>
						</td>
						<td >
							<p>Machine evaluator performs initial assessment by looking at
							the logs, and trying to spot anomalies. If any anomalies
							found, they are written down</p>
						</td>
					</tr>
					<tr>
						<td >
							<p>7</p>
						</td>
						<td >
							<p>Image evaluation</p>
						</td>
						<td >
							<p>Machine evaluator performs initial assessment by looking at
							the images and seeing if there are any obvious issues. Based
							on this, machine evaluator decides whether engineering team
							should be deployed to do a brief check on the machines
							physically.</p>
							<p>If machine evaluator believes engineers should check,
							machine evaluator continues with step 8, otherwise that person
							skips to step 10</p>
						</td>
					</tr>
					<tr>
						<td >
							<p>8</p>
						</td>
						<td >
							<p>Sending case to engineers</p>
						</td>
						<td >
							<p>If the machine evaluator thinks there needs to be a
							thorough evaluation, that person sends an email to the
							engineering team with the case number that needs evaluation</p>
						</td>
					</tr>
					<tr>
						<td >
							<p>9</p>
						</td>
						<td >
							<p>Engineers evaluate machine</p>
						</td>
						<td >
							<p>Engineers receive the case, and within 24-48 hours perform
							an assessment to the machine and update the case which sends a
							&nbsp;notification to machine evaluator</p>
						</td>
					</tr>
					<tr>
						<td >
							<p>10</p>
						</td>
						<td >
							<p>Submits results</p>
						</td>
						<td >
							<p>Machine evaluator updates the case in the MyERP system and
							submits the results of the evaluation. 
							</p>
						</td>
					</tr>
				</tbody></table>
			</dd></dl>
			
		</td>
	</tr>
</tbody></table>

<hr>
<h3 class="western">2.4 - Diagram of workstreams</h3>
<p>This section should provide a visual diagram to outline the
current process and more specifically the interactions between its
constituent workstreams. 
</p>

<p>Given the complexity of some of the workstreams, especially where
there is dependency or conditional branches, there is a lot of
clarity that would be gained through a visual diagram. The diagram
should add emphasis on the flow of the data, and provide an insight
of what actions are carried out on each step.</p>

<p>The example below is a simple overview of both of the workstreams
visualised. This is a simplification of the process - in more complex
proposals, it may be important to provide more granular detail, and
potentially different diagrams.</p>

<table >
	<colgroup><col width="0*">
	</colgroup><tbody><tr>
		<td  >
			<p>Example Diagram of Workstreams for “Manufacturing Group
			Limited”</p>
			
			<p>The current workflow for domain-specific analysis on the data
			mentioned is depicted at the end of this section. The diagram
			below provides a high level overview of the data sources involved,
			together with all the data formats that are extracted, created and
			pushed back into systems.</p>
			
			<p>The datasets involved as labelled in the image are as follows:</p>
			<ol>
				<li>
<p >When a new case begins,
				individuals obtain images from system 1 by identifying all the
				images for that particular day</p>
				</li><li>
<p >Individuals obtain</p>
				</li><li>
<p >Financial metrics from
				various different sources that are collected by analysts</p>
				</li><li>
<p >Conclusions explained in
				text written by domain experts</p>
				</li><li>
<p >Report put together from
				domain expert conclusions created that month</p>
			</li></ol>
			<p><img src="images/diagram.png" name="Image3" style="width: 100%; max-width: 800px" border="0" align="bottom">
</p>
			
		</td>
	</tr>
</tbody></table>

<hr>
<h3 class="western">2.5 - Key target metrics 
</h3>
<p>This section provides the benchmarks gathered from the process,
together with the ideal targets, which should be aligned &nbsp;with
the high level objectives provided in <a href="https://docs.google.com/document/d/1ecJPiCAc0EkAbU_L5f2yxgucAfThynQLc1Qn37jR4Gg/edit#heading=h.352ixnz1ijxn">Section
2.1 - Key objectives &amp; outcomes</a>. 
</p>

<p>In practice, it may be hard or near impossible to measure accurate
benchmarks, however it is possible to relax constraints to obtain
higher level metrics that would still be useful for suppliers. The
example in this section provides an overview of some metrics that can
be collected. This section is directly addressed in “Assessment
Criteria #1 - Practical Benchmarks” of the <a href="https://ethical.institute/rfx.html#model">“Machine
Learning Maturity Model”</a>.</p>
<h4 class="western">2.5.1 - Current benchmarks &amp; metrics</h4>
<p>This section should provide benchmark metrics that are gathered
from the process. These will allow suppliers to have a base for
benchmarking their proposed metrics. Some common metrics for
benchmarking are covered below.</p>

<table >
	<tbody><tr>
		<td  >
			<p>Name</p>
		</td>
		<td  >
			<p>Description</p>
		</td>
	</tr>
	<tr>
		<td >
			<p>Time</p>
		</td>
		<td >
			<p>Time taken from start to end of process, broken down by
			workstream where relevant.</p>
		</td>
	</tr>
	<tr>
		<td >
			<p>Volumes</p>
		</td>
		<td >
			<p>The expected number of instances the process will be executed
			per day, month or year (as relevant), with the relevant
			granularity (number of datasets involved in each layer)</p>
		</td>
	</tr>
	<tr>
		<td >
			<p>Accuracy / Error frequency</p>
		</td>
		<td >
			<p>(*) Accuracy will be one of the most important pieces in this
			template. This is covered in more detail below, and it is directly
			aligned with <a href="https://ethical.institute/principles.html">metric
			#6 in the Machine Learning Maturity Model</a>.</p>
		</td>
	</tr>
	<tr>
		<td >
			<p>Resources</p>
		</td>
		<td >
			<p>This should contain information about the resources required to
			have the process operational, such as stakeholders. 
			</p>
		</td>
	</tr>
	<tr>
		<td >
			<p>Costs</p>
		</td>
		<td >
			<p>Costs can include operation costs, but also the costs of an
			error in a specific part of the process.</p>
		</td>
	</tr>
</tbody></table>

<p>(*) Accuracy metrics</p>
<p>As it is mentioned in the Machine Learning Maturity Model, it is
critical to consider what practical accuracy metrics make sense for
each project. Often projects consider only simple accuracy (i.e.
number of correct cases / total number of cases). However it is often
necessary to take into consideration what types of errors are the
most critical in the specific project. Some of these considerations
include:</p>
<ul>
	<li>
<p >A case that should have
	been correct but was missed (Type I Error / Recall)</p>
	</li><li>
<p >A case that was incorrectly
	marked as correct (Type II Error / Precision)</p>
</li></ul>

<p>Below is an example that contains high-level metrics for the first
workstream. For more detail on this section, please refer to the
<a href="https://ethical.institute/rfx.html#model">“Machine Learning
Maturity Model”</a>.</p>

<table >
	<colgroup><col width="0*">
	</colgroup><tbody><tr>
		<td  >
			<p>Example Benchmarking for “Manufacturing Group Limited”</p>
			
			<p>The benchmarking results that we have compiled on the
			workstreams where put together through the following process:</p>
			<ul>
				<li>
<p >A member of the Process
				Operations department was assigned to log and track data over a
				30 day period. 
				</p>
				</li><li>
<p >Additional information
				was provided by accessing the logs of the number of requests.
				Over a 120 day period, a mean was taken to estimate the average.</p>
			</li></ul>
			<p>The resulting benchmarks are as follows:</p>
			
			<table >
				<tbody><tr>
					<td >
						<p>Name</p>
					</td>
					<td >
						<p>Description</p>
					</td>
				</tr>
				<tr>
					<td >
						<p>Time</p>
					</td>
					<td >
						<p>Time to completion per machine is 2-5 days, however the
						total active time invested in this project is 5-10 hours as
						follows:</p>
						<ul>
							<li>
<p >Gathering data - 2
							hours per case</p>
							</li><li>
<p >Machine Evaluation -
							2 hours per machine</p>
							</li><li>
<p >Engineer Evaluation -
							1-6 hours plus travel time</p>
						</li></ul>
						
					</td>
				</tr>
				<tr>
					<td >
						<p>Volumes</p>
					</td>
					<td >
						<ul>
							<li>
<p >We receive 1-300
							requests a day (average 120)</p>
							</li><li>
<p >We receive 1,000 -
							8,000 per month (average 3,600)</p>
						</li></ul>
					</td>
				</tr>
				<tr>
					<td >
						
						<p>Accuracy</p>
					</td>
					<td >
						<ul>
							<li>
<p >An average of 10% of
							machines are actually sent for maintenance during the checks,
							which means that we could avoid large number of these
							notifications</p>
							</li><li>
<p >We currently have an
							accuracy of 85%</p>
							</li><li>
<p >90% recall - our
							experts rarely miss any machines that require maintenance</p>
							</li><li>
<p >80% precision, as our
							experts are quite cautious and sometimes send machines for
							maintenance even though they may not require it</p>
						</li></ul>
					</td>
				</tr>
				<tr>
					<td >
						
						<p>Resources</p>
					</td>
					<td >
						<p>With an average of 3,600 cases per month, we require full
						time 60 experts, 120 full-time analysts, and 120 engineers.</p>
					</td>
				</tr>
				<tr>
					<td >
						<p>Costs</p>
					</td>
					<td >
						<p>The costs of operations are $15,000,000 per year.</p>
						<p>The costs of the maintenance of a machine that didn’t
						require maintenance is $10,000 - $60,000</p>
						<p>The cost of not sending for maintenance a machine that
						requires it is $200,000 - $300,000.</p>
					</td>
				</tr>
			</tbody></table>
			
		</td>
	</tr>
</tbody></table>

<h4 class="western">2.5.2 - Target improvements &amp; metrics</h4>
<p>Each of the targets in this subsection should link to a high level
objective provided in <a href="https://docs.google.com/document/d/1ecJPiCAc0EkAbU_L5f2yxgucAfThynQLc1Qn37jR4Gg/edit#heading=h.352ixnz1ijxn">Section
2.1 - Key objectives &amp; outcomes</a>, and should reference the
metrics in the previous <a href="https://docs.google.com/document/d/1ecJPiCAc0EkAbU_L5f2yxgucAfThynQLc1Qn37jR4Gg/edit#heading=h.jl5mts4jakz0">Subsection
2.5.1 - Current benchmarks &amp; metrics</a>. The points to cover &nbsp;in
this subsection should include the following:</p>

<table >
	<tbody><tr>
		<td  >
			<p>Section referenced</p>
		</td>
		<td  >
			<p>Description</p>
		</td>
	</tr>
	<tr>
		<td >
			<p>Requirements</p>
		</td>
		<td >
			<p>The requirements can be functional or non-functional
			requirements that can be directly related to features that require
			user-friendliness, but also it could be more specific requirements
			such as integrations with user management systems, or the ability
			to extend the platform in case there is an in-house team of
			experts that has the skills.</p>
		</td>
	</tr>
	<tr>
		<td >
			<p>Process</p>
		</td>
		<td >
			<p>The process steps can be referenced to mention what the ideal
			level of automation would be. This could be full automation or
			partial automation. It’s also possible to reference some of the
			workstreams and mention which ones should be automated.</p>
		</td>
	</tr>
	<tr>
		<td >
			<p>Benchmarking</p>
		</td>
		<td >
			<p>Each of the metrics can be referenced with an ideal decrease or
			increase. 
			</p>
		</td>
	</tr>
</tbody></table>
<p><br>
<br>
<br>

</p>
<table >
	<colgroup><col width="0*">
	</colgroup><tbody><tr>
		<td  >
			<p>Example Target improvements and metrics for “Manufacturing
			Group Limited”</p>
			
			<p>The ideal solution would have the following:</p>
			<ul>
				<li>
<p >User interface to
				visualise metrics 
				</p>
				</li><li>
<p >Functionality for domain
				experts to update configuration of the system</p>
				</li><li>
<p >Integration with our
				user database</p>
				</li><li>
<p >Ability to monitor
				errors of the system and action errors</p>
			</li></ul>
			
			<p>The ideal solution would show a quantifiable improvement in
			benchmarks provided as follows:</p>
			<ul>
				<li>
<p >Full automation of the
				second workflow (report generation)</p>
				</li><li>
<p >Partial automation of
				the first workflow - we are willing to accept proposals that
				suggest how the human-machine integration would look like</p>
				<ul>
					<li>
<p >Full automation of data
					collection</p>
					</li><li>
<p >Significant improvement
					in efficiency for initial domain expert evaluation</p>
					</li><li>
<p >Improvement in
					efficiency for engineering team evaluation</p>
				</li></ul>
				</li><li>
<p >Improved accuracy,
				precision and recall across end to end process</p>
			</li></ul>
			
			<p>In order for a solution to be viable, we need to see at least a
			70% in overall cost reduction within the first 3 years.</p>
		</td>
	</tr>
</tbody></table>

<h2 class="western" >3 - Tender
Process</h2>
<p>This chapter introduces suppliers to the end-to-end tender process
that they will have to go through during procurement. This chapter
has been put together using the <a href="https://ethical.institute/rfx.html#model">Machine
Learning Maturity Model</a> assessment criteria. It is recommended to
take into consideration all the 8 assessment criteria when putting
together this section. Each section in this chapter takes into
consideration each of the criteria implicitly, and <a href="https://docs.google.com/document/d/1ecJPiCAc0EkAbU_L5f2yxgucAfThynQLc1Qn37jR4Gg/edit#heading=h.856p9h8sb15f">Section
3.2 - Assessment</a> criteria addresses the Machine Learning Maturity
Model explicitly.</p>
<hr>
<h3 class="western">3.1 - Key point of contact</h3>
<p>This section should have the contact details of the person or
people that are in charge of the day to day communications for this
project. This ideally will be an individual that is well connected
with the relevant stakeholders in the organisation. At the same time,
it has to be someone that will have the time to dedicate to perform
the delivery, communications and internal championing of the
solution. 
</p>

<p>If the key point of contact is not able to engage internally or
externally in an efficient way for any reason (whether not enough
time committed, not enough resources or slow response from other
departments) the success of the project may be affected. It is
critical that the key point of contact and the team are able to work
closely together as necessary for the project.</p>

<table >
	<colgroup><col width="0*">
	</colgroup><tbody><tr>
		<td  >
			<p>Example Key point of contact for “Manufacturing Group
			Limited”</p>
			
			<ul>
				<li>
<p >Alejandro Saucedo will
				be your key contact (and delivery manager) for the RfP Process.
				He can be contacted via <a href="mailto:a@ethical.institute">a@ethical.institute</a>.</p>
			</li></ul>
		</td>
	</tr>
</tbody></table>


<hr>
<h3 class="western">3.2 - Assessment criteria</h3>
<p>This section provides an insight on the criteria that can be used
to evaluate the proposals. The criteria contains multiple areas, but
the core is reflected in the <a href="https://ethical.institue/rfx.html#model">Machine
Learning Maturity Model</a> assessment criteria. 
</p>

<p>The table below contains an overview of common criteria that is
used to assess a proposal:</p>

<table >
	<tbody><tr>
		<td  >
			<p>Criteria</p>
		</td>
		<td  >
			<p>Description</p>
		</td>
	</tr>
	<tr>
		<td >
			<p>Competition results</p>
		</td>
		<td >
			<p>This criteria covers the results obtained by the supplier
			during the optional competition, which is covered in <a href="https://docs.google.com/document/d/1ecJPiCAc0EkAbU_L5f2yxgucAfThynQLc1Qn37jR4Gg/edit#heading=h.3kr1iqu4gs2f">Section
			4.1 - Competition</a>. Often there is a minimum bar set, and if a
			supplier doesn’t pass it is not considered. 
			</p>
		</td>
	</tr>
	<tr>
		<td >
			<p>Machine Learning</p>
			<p>Maturity 
			</p>
			<p>Model</p>
		</td>
		<td >
			<p>The Machine Learning Maturity Model is the assessment criteria
			proposed to evaluate the maturity of a machine learning solution.
			The maturity model is available for <a href="https://ethical.instiute/rfx.html#model">free
			download at our website</a>, and consists of the following 8
			criteria:</p>
			<ul>
				<li>
<p >Practical benchmarks</p>
				</li><li>
<p >Explainability by
				justification</p>
				</li><li>
<p >Infrastructure for
				reproducible operations</p>
				</li><li>
<p >Data and model
				assessment processes</p>
				</li><li>
<p >Privacy enforcing
				infrastructure</p>
				</li><li>
<p >Operational process
				design</p>
				</li><li>
<p >Change management
				capabilities</p>
				</li><li>
<p >Security risk processes</p>
			</li></ul>
		</td>
	</tr>
	<tr>
		<td >
			
			<p>License &amp; service terms</p>
		</td>
		<td >
			<p>This criteria covers the terms that revolve around the legal
			terms around the solution. This includes the terms of the license,
			including the service-level-agreements, security questionnaires,
			maintenance terms, etc.</p>
		</td>
	</tr>
	<tr>
		<td >
			<p>Costs</p>
		</td>
		<td >
			<p>This includes the total costs of the technology, together with
			the proposed improvements in the process itself</p>
		</td>
	</tr>
</tbody></table>

<p>The example below provides a high level overview of assessment
metrics that reference the machine learning maturity evaluation. 
</p>

<table >
	<colgroup><col width="0*">
	</colgroup><tbody><tr>
		<td  >
			<p>Example Assessment Criteria for “Manufacturing Group Limited”</p>
			
			<p>The suppliers will be evaluated using the criteria outlined
			below. Scores will be weighted and then brought together to form
			&nbsp;total qualitative rating for your proposal. This will then
			be reviewed in conjunction with the commercial pricing submission
			to form an overall view. The scoring metrics that will be used for
			each criteria are: 
			</p>
			<ul>
				<li>
<p >Not been met</p>
				</li><li>
<p >Partially met</p>
				</li><li>
<p >Met</p>
				</li><li>
<p >Exceeds requirement</p>
			</li></ul>
			
			<p>The suppliers will be evaluated on the following criteria:</p>
			
			<table >
				<colgroup><col width="122">
				<col width="415">
				<col width="52">
				</colgroup><tbody><tr>
					<td  >
						<p>Criteria</p>
					</td>
					<td  >
						<p>Description</p>
					</td>
					<td  >
						<p>%</p>
					</td>
				</tr>
				<tr>
					<td >
						<p>Competition results</p>
					</td>
					<td >
						<p>Suppliers must pass the competition provided as per Section
						4.1 - Competition.</p>
					</td>
					<td >
						<p>Need pass</p>
					</td>
				</tr>
				<tr>
					<td >
						<p>Machine Learning</p>
						<p>Maturity 
						</p>
						<p>Model</p>
					</td>
					<td >
						<p>Suppliers will be assessed using the Machine Learning
						Maturity Model.</p>
					</td>
					<td >
						<p>40%</p>
					</td>
				</tr>
				<tr>
					<td >
						<p>Extra evaluation criteria</p>
					</td>
					<td >
						<p>Besides the Machine Learning Maturity Model, the suppliers
						will be evaluated using a questionnaire with specific questions
						about this request.</p>
					</td>
					<td >
						<p>20%</p>
					</td>
				</tr>
				<tr>
					<td >
						<p>License &amp; service terms</p>
					</td>
					<td >
						<p>This includes the terms of the license, including the
						service-level-agreements, security questionnaires, availability
						(and backwards compatibility) of upgrades, etc.</p>
					</td>
					<td >
						<p>10%</p>
					</td>
				</tr>
				<tr>
					<td >
						<p>Costs</p>
					</td>
					<td >
						<p>This includes the total costs of the technology, together
						with the proposed improvements in the process itself</p>
					</td>
					<td >
						<p>30%</p>
					</td>
				</tr>
			</tbody></table>
			
		</td>
	</tr>
</tbody></table>

<hr>
<h3 class="western">3.3 - Evaluation team</h3>
<p>This section should have details on the stakeholders that will be
involved in the evaluation process, and the way each of the criteria
will be evaluated. This will provides suppliers with an understanding
of what are the skills and domain specific knowledge in the
evaluation board, and will ensure suppliers add extra (or less)
background knowledge / explanations as relevant.</p>

<p>Below is an example of what the evaluation team could consist of,
and what each of the members will evaluate.</p>

<table >
	<colgroup><col width="0*">
	</colgroup><tbody><tr>
		<td  >
			<p>Example Evaluation Process for “Manufacturing Group Limited”</p>
			
			<ul>
				<li>
<p >Suppliers will be scored
				by the Manufacturing Group Limited team and where necessary, by
				internal stakeholders to review technical aspects of the
				proposals. For example:</p>
				<ul>
					<li>
<p >Regarding IT
					specifications, such as security procedures and technical
					helpdesk capabilities, these will be reviewed by our Chief
					Technology Officer</p>
					</li><li>
<p >Regarding any legal or
					contract amendments - we will obtain advice from our in-house
					legal representatives to examine their impact on your
					proposal/desired solution</p>
				</li></ul>
			</li></ul>
			
		</td>
	</tr>
</tbody></table>

<hr>
<h3 class="western">3.4 - Short-List Negotiations</h3>
<p>Once the supplier has been selected, the framework agreement,
statement of work and relevant contracts should be drafted, and
finalised with the deadline provided. This section should contain
information about what is expected from the chosen supplier.
Commencement dates, as well as timelines should also be drafted.</p>

<table >
	<colgroup><col width="0*">
	</colgroup><tbody><tr>
		<td  >
			<p>Example Short-List Negotiations for “Manufacturing Group
			Limited”</p>
			
			<ul>
				<li>
<p >After short-listing and
				final presentations, Manufacturing Group Limited will be
				selecting their preferred supplier(s) to work with. We will also
				be informing all unsuccessful suppliers why they were overlooked</p>
				</li><li>
<p >We will do our utmost to
				provide all suppliers with constructive feedback covering all 5
				of our key assessment criteria here</p>
			</li></ul>
		</td>
	</tr>
</tbody></table>


<hr>
<h3 class="western">3.5 - Project timeline</h3>
<p>This section should provide a key breakdown for the milestones
expected. Some of the deadlines and timelines would vary depending on
the complexity of the project, but some standard dates that should be
included are provided below. Besides providing these external
deadlines for suppliers, it’s also important to set up internal
deadlines and allocate the right responsibilities to be able to meet
the necessary requirements when interacting with suppliers at each
stage.</p>

<p>Below is a set of the most common points to ensure are covered,
and each of them are linked to a sub-section in this section.</p>

<table >
	<tbody><tr>
		<td  >
			<p>Activity</p>
		</td>
		<td  >
			<p>Description</p>
		</td>
	</tr>
	<tr>
		<td >
			<p>Issue of RFP Documentation</p>
		</td>
		<td >
			<p>This date would require all supporting documents relevant to
			the RFP. If there is a competition being ran, this would also
			include further instructions required for this, together with the
			data correctly labelled as relevant.</p>
		</td>
	</tr>
	<tr>
		<td >
			<p>Suppliers Questions Due</p>
		</td>
		<td >
			<p>Suppliers should be able to submit questions about the
			proposal. This is often done through a portal or sending an excel
			file. Often the answers of suppliers are made public to all other
			suppliers so that all suppliers are able to have the same context
			when proposing the solution.</p>
		</td>
	</tr>
	<tr>
		<td >
			<p>Supplier Proposals Deadline</p>
		</td>
		<td >
			<p>This is the deadline for the proposals. If there is an ongoing
			competition in place, extra deadlines may apply, such as potential
			evaluation day, or requirement to provide access to the solution
			to run an unseen test-case for evaluation.</p>
		</td>
	</tr>
	<tr>
		<td  >
			<p>Short-list announcement</p>
		</td>
		<td  >
			<p>The selected short-list suppliers are notified and invited to
			participate in further negotiations which may require changes in
			the initial proposals. Suppliers that are not selected are also
			notified.</p>
		</td>
	</tr>
	<tr>
		<td >
			<p>Preferred supplier notified</p>
		</td>
		<td >
			<p>Once the preferred supplier is selected, the offer and contract
			are sent, which should provide a the framework agreement together
			with the relevant statements of work required, as well as SLAs,
			license terms, etc.</p>
		</td>
	</tr>
	<tr>
		<td >
			<p>Contract drafted &amp; signed</p>
		</td>
		<td >
			<p>It’s worth also having a deadline for contract signing to
			make sure the process doesn’t drag out.</p>
		</td>
	</tr>
	<tr>
		<td >
			<p>Project kick-off date</p>
		</td>
		<td >
			<p>The project kick-off date should be agreed beforehand, and
			updated depending on changes on scope that may have arisen</p>
		</td>
	</tr>
</tbody></table>

<table >
	<colgroup><col width="0*">
	</colgroup><tbody><tr>
		<td  >
			<p>Example Project Timeline for “Manufacturing Group Limited”</p>
			
			<table >
				<colgroup><col width="311">
				<col width="0*">
				</colgroup><tbody><tr>
					<td >
						<p>Activity</p>
					</td>
					<td >
						<p>Due Date</p>
					</td>
				</tr>
				<tr>
					<td >
						<p>Issue of RFP Documentation</p>
					</td>
					<td >
						<p>1-Jul-2019</p>
					</td>
				</tr>
				<tr>
					<td >
						<p>Supplier Questions Due</p>
					</td>
					<td >
						<p>13-Jul-2019</p>
					</td>
				</tr>
				<tr>
					<td >
						<p>Supplier Proposals Deadline</p>
					</td>
					<td >
						<p>17-Jul-2019</p>
					</td>
				</tr>
				<tr>
					<td >
						<p>Evaluation of Proposals</p>
					</td>
					<td >
						<p>31-Jul-2019</p>
					</td>
				</tr>
				<tr>
					<td >
						<p>Short-list suppliers generated</p>
					</td>
					<td >
						<p>3-Aug-2019</p>
					</td>
				</tr>
				<tr>
					<td >
						<p>Short-Listed Vendor Presentation</p>
					</td>
					<td >
						<p>6/7-Aug-2019</p>
					</td>
				</tr>
				<tr>
					<td >
						<p>Final Negotiations &amp; Discussions</p>
					</td>
					<td >
						<p>14-Aug-2019</p>
					</td>
				</tr>
				<tr>
					<td >
						<p>Preferred Supplier Selected</p>
					</td>
					<td >
						<p>17-Aug-2019</p>
					</td>
				</tr>
				<tr>
					<td >
						<p>Contract Signed</p>
					</td>
					<td >
						<p>31-Aug-2019</p>
					</td>
				</tr>
				<tr>
					<td >
						<p>Implementation &amp; Training</p>
					</td>
					<td >
						<p>30-Nov-2019</p>
					</td>
				</tr>
			</tbody></table>
			
		</td>
	</tr>
</tbody></table>

<hr>
<h3 class="western">3.6 - Issue of RFP Documentation</h3>
<p>This section contains an overview of all the documents that the
suppliers will receive with the documentation. 
</p>

<table >
	<colgroup><col width="0*">
	</colgroup><tbody><tr>
		<td  >
			<p>Example Issue of RFP Documentation for “Manufacturing Group
			Limited”</p>
			
			<p>Manufacturing Group Limited will be sending all suppliers the
			following documentations on the issuing date:</p>
			<ul>
				<li>
<p >“Request for
				Proposal.pdf” - This document</p>
				</li><li>
<p >“Test Competition
				Data.zip” - Zip containing dataset of images, metrics and text</p>
				</li><li>
<p >“Security
				Questionnaire.xsls” - Security sheet</p>
			</li></ul>
		</td>
	</tr>
</tbody></table>


<hr>
<h3 class="western">3.7 - Supplier questions due</h3>
<p>This section should cover instructions of how suppliers should
submit questions. Some of the points that are advised to have
include:</p>
<ul>
	<li>
<p >Suppliers are able to see
	each other’s questions and answers</p>
	</li><li>
<p >Clear deadline that
	reflects the one provided in the previous section</p>
	</li><li>
<p >Instructions for suppliers
	to submit the questions 
	</p>
	</li><li>
<p >Specific points where
	answers cannot be provided for any reason</p>
</li></ul>

<table >
	<colgroup><col width="0*">
	</colgroup><tbody><tr>
		<td  >
			<p>Example Suppliers questions due for “Manufacturing Group
			Limited”</p>
			
			<ul>
				<li>
<p >A deadline has been
				imposed for all questions to be submitted to the delivery manager
				at least 4 working days before supplier submission are due. This
				is to ensure that Manufacturing Group Limited has the appropriate
				time to respond and so that suppliers have all necessary
				information prior to submitting their bids.</p>
				</li><li>
<p >Manufacturing Group
				Limited reserves the right to share any questions and/or answers
				with other bidders prior to the submission date, to ensure
				consistency and transparency in the bidding process.</p>
			</li></ul>
		</td>
	</tr>
</tbody></table>


<hr>
<h3 class="western">3.8 - Supplier proposals deadline</h3>
<p>This section provides details on how the suppliers should submit
the proposals to the company (portal, email, etc). This should also
cover whether extensions can be made for suppliers.</p>

<table >
	<colgroup><col width="0*">
	</colgroup><tbody><tr>
		<td  >
			<p>Example supplier proposals deadline for “Manufacturing Group
			Limited”</p>
			
			<ul>
				<li>
<p >Manufacturing Group
				Limited will be unable to grant any extensions for supplier
				proposal submissions, no matter the circumstances</p>
				</li><li>
<p >The deadline for
				submissions is 19:00 BST on Friday 17th July 2019.</p>
				</li><li>
<p >Manufacturing Group
				Limited is seeking up to 2 proposals from each supplier
				participating in the bidding process. The first proposal must
				meet the specification as defined in section 4 and the second
				must not</p>
				</li><li>
<p >All suppliers are
				required to submit proposals by sending it to the key point of
				contact. 
				</p>
				</li><li>
<p >Any documentation
				received by this post will not be reviewed by Manufacturing Group
				Limited, therefore physical tender submissions are discouraged.
				Please ensure your submissions are sent electronically only.</p>
			</li></ul>
			
		</td>
	</tr>
</tbody></table>

<hr>
<h3 class="western">3.9 - Short-list negotiations</h3>
<p>This section should contain information for suppliers to
understand what they are expected to do during and after the
short-list announcement. The three main points are outlined in the
table below:</p>

<table >
	<tbody><tr>
		<td  >
			<p>Activity</p>
		</td>
		<td  >
			<p>Description</p>
		</td>
	</tr>
	<tr>
		<td >
			<p>Feedback</p>
		</td>
		<td >
			<p>The level of feedback that will be given to successful and
			optional unsuccessful suppliers</p>
		</td>
	</tr>
	<tr>
		<td >
			<p>Negotiations and proposal amendments</p>
		</td>
		<td >
			<p>Short-listed suppliers are often expected to receive feedback
			that will require changes on their initial proposals. This section
			should cover what the expected next steps will be.</p>
		</td>
	</tr>
</tbody></table>

<table >
	<colgroup><col width="0*">
	</colgroup><tbody><tr>
		<td  >
			<p>Example short-list negotiations for “Manufacturing Group
			Limited”</p>
			
			<ul>
				<li>
<p >Those suppliers that are
				successfully short-listed after the evaluation of responses will
				be invited to the Manufacturing Group Limited offices to present
				their proposals in person to discuss their suitability with the
				team</p>
				</li><li>
<p >Suppliers will be
				allocated up to 2 hours on the day; 1 hour to present and up to 1
				hour for sufficient fact-finding &amp; Q&amp;A</p>
				</li><li>
<p >Presentations are
				recommended to not be intradictional lectures. They should be in
				an inclusive and engaging format that fits your proposed
				solution. Posters and strong visuals are recommended to show off
				your work for further review by the team after your presentation</p>
				</li><li>
<p >If in the event that
				suppliers require additional time to conduct a site visit or
				on-site testing, they should advise the Delivery Manager at least
				2 days prior to their scheduled arrival</p>
			</li></ul>
			
		</td>
	</tr>
</tbody></table>

<hr>
<h3 class="western">3.10 - Preferred supplier contract drafting</h3>
<p>This section should contain information for suppliers to
understand what the successful supplier and unsuccessful short-listed
suppliers are supposed to do. The main points covered are provided in
the table below.</p>

<table >
	<tbody><tr>
		<td  >
			<p>Activity</p>
		</td>
		<td  >
			<p>Description</p>
		</td>
	</tr>
	<tr>
		<td >
			<p>Notification channel</p>
		</td>
		<td >
			<p>How suppliers will be notified of the short-list and actions</p>
		</td>
	</tr>
	<tr>
		<td >
			<p>Feedback</p>
		</td>
		<td >
			<p>The level of feedback that will be given to the preferred
			supplier and unsuccessful short-listed suppliers</p>
		</td>
	</tr>
	<tr>
		<td >
			<p>Next steps</p>
		</td>
		<td >
			<p>High-level instructions of what selected supplier should go
			forward - this will mainly consist of drafting and finalising all
			contracts 
			</p>
		</td>
	</tr>
	<tr>
		<td >
			<p>Short-listed supplier terms</p>
		</td>
		<td >
			<p>If there are any terms where it is desired to have other
			suppliers potentially involved at a later date, or if there is the
			potential of having short-listed proposals valid for a longer
			period of time 
			</p>
		</td>
	</tr>
</tbody></table>

<table >
	<colgroup><col width="0*">
	</colgroup><tbody><tr>
		<td  >
			<p>Example preferred supplier contract drafting for “Manufacturing
			Group Limited”</p>
			
			<ul>
				<li>
<p >Manufacturing Group
				Limited fully anticipates there to be minimal and insignificant
				changes required to the draft contract provided as part of this
				RFP</p>
				</li><li>
<p >However in the likely
				event that any amendments do need incorporating, we will be
				allowed a two-week window to address these before both parties
				are due to sign the contract by the 31st August 2019</p>
				</li><li>
<p >To ensure a smooth and
				successful outcome by this date. Manufacturing Group Limited has
				requested a copy of the draft contract be returned with your
				revisions as part of your proposed response. 
				</p>
			</li></ul>
		</td>
	</tr>
</tbody></table>


<h2 class="western" >4 - Detailed
Scope</h2>
<p>This chapter contain an overview of what requirements supplier
proposals are expected to contain. The structure of this chapter was
obtained from the most common stages identified in RFPs for machine
learning solutions, which are outlined in the table below:</p>

<table >
	<tbody><tr>
		<td  >
			<p>Phase</p>
		</td>
		<td  >
			<p>Description</p>
		</td>
	</tr>
	<tr>
		<td >
			<p>Proof of concept competition</p>
		</td>
		<td >
			<p>In the context of machine learning systems, suppliers are often
			required to compete against each other by evaluating the
			performance of their solutions based on the client’s data and
			feature requirements. For smaller procurement processes, this
			could be a more informal evaluation such as trying out multiple
			different systems, but for larger tenders there is a requirement
			for formal processes. This document provides an example of best
			practices when running a competition to evaluate machine learning
			based solutions.</p>
		</td>
	</tr>
	<tr>
		<td >
			<p>Productisation</p>
		</td>
		<td >
			<p>Suppliers should provide detailed insight on the work that
			still needs to be carried out for the solution to be operated
			fully in production. This includes further development on the
			proof of concept, enterprise integration, training and even
			operational transformation required for the system to be in
			production. This is often a one-off service, and should include
			the resources and timelines required to be carried out
			successfully.</p>
		</td>
	</tr>
	<tr>
		<td >
			<p>License</p>
		</td>
		<td >
			<p>The licensing overview provides an insight of contract length,
			service-level-agreement, volumes, limits, training included.</p>
		</td>
	</tr>
	<tr>
		<td >
			<p>Further Services</p>
		</td>
		<td >
			<p>This encompasses an overview of the costs and resources
			required to extend the systems that have been deployed. 
			</p>
		</td>
	</tr>
</tbody></table>

<p>This chapter has been put together using the <a href="https://ethical.institute/rfx.html#model">Machine
Learning Maturity Model</a> assessment criteria. Each section takes
into consideration each of the criteria implicitly. It is recommended
to take into consideration all the 8 assessment criteria when putting
together this section.</p>
<hr>
<h3 class="western">4.1 - Competition</h3>
<p>This section contains the instructions, documentation and datasets
required for the suppliers to be able to carry out the competition.
It’s important that the competition is representative of the
challenge presented, and that it references the right areas in
“Chapter 2 - Our Process”, in order for suppliers to get complete
context.</p>

<p>The details to put together a strong competition for suppliers are
covered in the <a href="https://ethical.institute/rfx.html#competition">“AI
Competition Tender” template</a> which can be downloaded from the
website. However, below there is a high-level process outlined on how
competitions can be set up.</p>

<p>In summary, the generic process required to prepare a competition
for a machine learning system, it requires the following steps:</p>
<ol>
	<li>
<p >For the end-to-end process,
	identify the datasets used (as outlined in chapter 2)</p>
	<ol>
		<li>
<p >For more complex
		challenges, you will require more examples</p>
	</li></ol>
	</li><li>
<p >Identify the final insights
	or decisions that are inferred from these datasets</p>
	<ol>
		<li>
<p >This often requires manual
		work, as it’s important that the datasets come together with the
		answers that are expected to be predicted by the ML models</p>
	</li></ol>
	</li><li>
<p >Split the data into a TEST
	dataset, and an EVALUATION dataset</p>
	<ol>
		<li>
<p >Each contains the datasets
		and answers</p>
	</li></ol>
	</li><li>
<p >Give the TEST dataset to
	the suppliers so they build a machine learning model</p>
	</li><li>
<p >Evaluate the suppliers with
	the evaluation set</p>
</li></ol>

<p>There are key points that need to be taken into consideration,
including:</p>
<ul>
	<li>
<p >Evaluation metrics should
	go beyond accuracy (precision, recall, etc)</p>
	</li><li>
<p >The expertise required to
	operate the system</p>
	</li><li>
<p >The work required to get to
	that level of accuracy</p>
	</li><li>
<p >Access to domain experts as
	relevant for suppliers to obtain understanding on datasets and ways
	of analysing datasets</p>
</li></ul>

<table >
	<colgroup><col width="0*">
	</colgroup><tbody><tr>
		<td  >
			<p>Example Competition for “Manufacturing Group Limited”</p>
			
			<p>Overview</p>
			<ul>
				<li>
<p >Suppliers will be
				evaluated based on their predictive performance on the datasets
				we have provided</p>
				</li><li>
<p >We will provide a TEST
				dataset, containing 400 labelled samples, and we will use an
				EVALUATION dataset that will be provided without labels to
				evaluate performance</p>
				</li><li>
<p >Given that the machine
				data changes as time goes by, we are interested in systems that
				have functionality that allows for programmatic re-training, so
				we will expect the models to be evaluated on a 10-fold
				cross-validation.</p>
				</li><li>
<p >The datasets will
				contain information 40 sets of data, each referencing a machine
				evaluation process as outlined in “Chapter 2 - Our Process”.
				Each instance will contain:</p>
				<ul>
					<li>
<p >30 images of the
					machine</p>
					</li><li>
<p >The logs for the
					machine</p>
					</li><li>
<p >The metrics for the
					machine</p>
					</li><li>
<p >The actual answer: One
					of three flags (OK, CHECK and MAINTENANCE)</p>
				</li></ul>
				</li><li>
<p >You can find further
				information about each dataset in the datasets provided</p>
			</li></ul>
			
			<p>Evaluation Metrics</p>
			<ul>
				<li>
<p >We expect metrics to be
				at least higher than our human benchmarks. This means that the
				accuracy should be no less than 80%</p>
				</li><li>
<p >A critical error in this
				case, is missing a machine that actually requires maintenance, so
				we expect suppliers to use recall as the most important metric</p>
				</li><li>
<p >Marking a machine as
				needing maintenance when it doesn’t is also expensive, so we
				expect suppliers to also give an insight on the precision of
				their model</p>
			</li></ul>
			
			<p>Evaluation Process</p>
			<ul>
				<li>
<p >Suppliers</p>
				</li><li>
<p >Suppliers will be given
				access to the EVALUATION dataset during the morning of the
				submission deadline</p>
				</li><li>
<p >Suppliers are expected
				to submit their predictions as a CSV document in the same order
				as the EVALUATION dataset index.</p>
				</li><li>
<p >The final metrics
				obtained in the results obtained should be submitted in the
				template attached with the datasets.</p>
			</li></ul>
			
		</td>
	</tr>
</tbody></table>

<hr>
<h3 class="western">4.2 - Locations</h3>
<p>This section should contain information about the exact locations
where the system will be installed, as well as the locations where
the system will be used in order to provide an idea &nbsp;of the
support timetable required.</p>

<table >
	<colgroup><col width="0*">
	</colgroup><tbody><tr>
		<td  >
			<p>Example Locations for “Manufacturing Group Limited”</p>
			
			<p>Please find below a table of the locations of the offices
			requiring these goods and services:</p>
			
			<table >
				<colgroup><col width="0*">
				<col width="0*">
				<col width="0*">
				<col width="0*">
				<col width="0*">
				</colgroup><tbody><tr>
					<td >
						<p>Name</p>
					</td>
					<td >
						<p>Type</p>
					</td>
					<td >
						<p>Address</p>
					</td>
					<td >
						<p>City</p>
					</td>
					<td >
						<p>Country</p>
					</td>
				</tr>
				<tr>
					<td >
						<p>Main HQ 
						</p>
					</td>
					<td >
						<p>Installation &amp; Use</p>
					</td>
					<td >
						<p>403, Upper st</p>
					</td>
					<td >
						<p>London</p>
					</td>
					<td >
						<p>UK</p>
					</td>
				</tr>
				<tr>
					<td >
						<p>Europe office</p>
					</td>
					<td >
						<p>Use</p>
					</td>
					<td >
						<p>1001, X Street</p>
					</td>
					<td >
						<p>Berlin</p>
					</td>
					<td >
						<p>Germany</p>
					</td>
				</tr>
				<tr>
					<td >
						<p>Canada office</p>
					</td>
					<td >
						<p>Use</p>
					</td>
					<td >
						<p>12, Y Street</p>
					</td>
					<td >
						<p>Toronto</p>
					</td>
					<td >
						<p>Canada</p>
					</td>
				</tr>
			</tbody></table>
			
		</td>
	</tr>
</tbody></table>

<hr>
<h3 class="western">4.3 - Implementation</h3>
<h4 class="western">4.3.1 - Data-source information 
</h4>
<p>This section contains an overview of the all the data-sources
where the data comes from. The two main approaches for integration of
suppliers is push and/or pull systems. This means that either
supplier systems can pull from the company’s systems, or
alternatively the company can push to the supplier’s APIs. If there
is preference, this should be the section where it should be covered.
If it is expected for suppliers to integrate with the company’s
systems, this section should contain a high level overview of the
datasets in the system.</p>

<table >
	<colgroup><col width="0*">
	</colgroup><tbody><tr>
		<td  >
			<p>Example Data-source information for “Manufacturing Group
			Limited”</p>
			
			<p>The three main data-sources are provided in the table below. We
			expect suppliers to integrate to all of our systems individually
			and poll every hour for new content. We expect suppliers to have a
			way to keep track of the machines that have already been
			processed. We also expect the system to be able to notify the
			relevant stakeholder through our MyERP system.</p>
			
			<table >
				<colgroup><col width="72">
				<col width="504">
				</colgroup><tbody><tr>
					<td >
						<p>System</p>
					</td>
					<td >
						<p>Description</p>
					</td>
				</tr>
				<tr>
					<td >
						<p>RoboSys</p>
					</td>
					<td >
						<p>This system has an XML STOMP feed that continuously updates
						subscribers with new data generated by machines.</p>
					</td>
				</tr>
				<tr>
					<td >
						<p>LogSys</p>
					</td>
					<td >
						<p>Log file system - this system can only be accessed through
						an SMTP, and folder structure contains the information based on
						machine ID.</p>
					</td>
				</tr>
				<tr>
					<td >
						<p>MyERP</p>
					</td>
					<td >
						<p>ERP system that holds system data. This contains an XML API
						that can be interacted programmatically using credentials. 
						</p>
					</td>
				</tr>
			</tbody></table>
			
		</td>
	</tr>
</tbody></table>

<h4 class="western">4.3.2 - Infrastructure Available</h4>
<p>This section should cover the infrastructure that is available for
suppliers to perform the deployment of their platforms. Example
infrastructure available can be:</p>
<ul>
	<li>
<p >Physical servers that can
	be provisioned for the supplier</p>
	</li><li>
<p >Container infrastructure
	(Kubernetes / Openshift Cluster)</p>
	</li><li>
<p >Preferred cloud provider
	(Google cloud, AWS, Azure, etc)</p>
	</li><li>
<p >Hadoop-based infrastructure</p>
</li></ul>

<p>There should also be information about the known limitations and
constraints that suppliers will require, such as maximum RAM or Video
memory. Although not always, machine learning computations may be
quite heavy, so it’s important that this is made explicit by
suppliers. 
</p>

<table >
	<colgroup><col width="0*">
	</colgroup><tbody><tr>
		<td  >
			<p>Example infrastructure available for “Manufacturing Group
			Limited”</p>
			
			<p>We have a simple physical server architecture, but we are
			currently moving towards cloud. Initially suppliers will only have
			access to one server with 60GB RAM, however we expect to migrate
			by mid-next year. Suppliers should include in their proposal a
			migration plan together with complexity and costs.</p>
		</td>
	</tr>
</tbody></table>

<h4 class="western">4.3.3 - Detailed supplier information</h4>
<p>This section should consist of a list of points that suppliers
should make sure they include with their proposals. The example
provided contains a table with a set of commonly required sections
that are expected in supplier proposals.</p>

<table >
	<colgroup><col width="0*">
	</colgroup><tbody><tr>
		<td  >
			<p>Example detailed supplier evaluation for “Manufacturing Group
			Limited”</p>
			
			<p>We expect suppliers to go into detail on their implementation
			requirements, and cover at least the points mentioned in the table
			below.</p>
			
			<table >
				<tbody><tr>
					<td  >
						<p>Area</p>
					</td>
					<td  >
						<p>Description</p>
					</td>
				</tr>
				<tr>
					<td >
						<p>Productisation Roadmap</p>
					</td>
					<td >
						<p>Information about the work required for the system to be
						configured or developed in order for it to be production ready.
												</p>
					</td>
				</tr>
				<tr>
					<td >
						<p>Timeline / schedule</p>
					</td>
					<td >
						<p>Chronological overview of what services will be delivered
						throughout the delivery schedule, together with key
						stakeholders involved (from our company and supplier’s side)</p>
					</td>
				</tr>
				<tr>
					<td >
						<p>Team Involved</p>
					</td>
					<td >
						<p>The team(s) that will be required throughout the
						implementation. This includes:</p>
						<ul>
							<li>
<p >Technical staff
							involved</p>
							</li><li>
<p >Account managers
							involved</p>
							</li><li>
<p >Project managers 
							</p>
							</li><li>
<p >Key stakeholders
							required from our company’s side</p>
						</li></ul>
					</td>
				</tr>
				<tr>
					<td >
						
						<p>Operation in production</p>
					</td>
					<td >
						<p>Overview of the requirements for Manufacturing Group Limited
						to be able to operate the tool in production, and whether it’s
						going to require specific training. 
						</p>
						
					</td>
				</tr>
				<tr>
					<td >
						<p>Machine learning infrastructure maturity</p>
					</td>
					<td >
						<p>There should be an overview around the features within the
						product and/or processes that supplier has for:</p>
						<ul>
							<li>
<p >Operational process
							design</p>
							</li><li>
<p >Data and model
							assessment</p>
							</li><li>
<p >Explainability by
							design</p>
							</li><li>
<p >Reproducibility</p>
							</li><li>
<p >Change management</p>
							</li><li>
<p >Practical accuracy
							metrics 
							</p>
							</li><li>
<p >Data risk security</p>
						</li></ul>
						<p>For more information on each of these, you can refer to the
						framework for responsible machine learning.</p>
					</td>
				</tr>
			</tbody></table>
			
		</td>
	</tr>
</tbody></table>

<hr>
<h3 class="western">4.4 - IP Ownership</h3>
<p>This section should provide an overview of what are the
expectations of IP ownership to be in place. This may vary depending
on how much custom development is being requested by the suppliers,
however a common standard in industry is provided in the example
below.</p>

<table >
	<colgroup><col width="0*">
	</colgroup><tbody><tr>
		<td  >
			<p>Example IP Ownership for “Manufacturing Group Limited”</p>
			
			<p>The high level requirements stated by Manufacturing Group
			Limited in regards to IP ownership are as follows:</p>
			<ul>
				<li>
<p >Supplier has full
				ownership of their product</p>
				</li><li>
<p >Company has full
				ownership of custom code developed outside of product unless
				explicitly specified (e.g. a feature of the product, a bug fix,
				patch, etc)</p>
				</li><li>
<p >Company has full
				ownership of their data</p>
				</li><li>
<p >Company has full
				ownership of models created unless explicitly specified</p>
			</li></ul>
		</td>
	</tr>
</tbody></table>


<hr>
<h3 class="western">4.5 - Key Contract terms</h3>

<p>This section should provide an overview of key term metrics in the
contract/license.</p>

<table >
	<tbody><tr>
		<td  >
			<p>Phase</p>
		</td>
		<td  >
			<p>Description</p>
		</td>
	</tr>
	<tr>
		<td >
			<p>Length</p>
		</td>
		<td >
			<p>Length of the contract 
			</p>
		</td>
	</tr>
	<tr>
		<td >
			<p>Renewal</p>
		</td>
		<td >
			<p>Process for contract to be renewed</p>
		</td>
	</tr>
	<tr>
		<td >
			<p>Cancellation</p>
		</td>
		<td >
			<p>Terms for company or supplier to cancel agreement, together
			with timelines</p>
		</td>
	</tr>
</tbody></table>

<table >
	<colgroup><col width="0*">
	</colgroup><tbody><tr>
		<td  >
			<p>Example Contract Terms (High Level) for “Manufacturing Group
			Limited”</p>
			
			<ul>
				<li>
<p >This contract will be
				for an initial period of 24 months with an option for an
				additional 12 months extension, subject to agreement from both
				parties</p>
				</li><li>
<p >Suppliers should
				perceive this as a 24-month engagement rather than a discrete
				project, followed by reporting and on-going maintenance</p>
				</li><li>
<p >In the event that the
				agreement is not renewed within 30 days of its end date, this
				agreement shall not automatically renew and will cease to remain
				in effect</p>
				</li><li>
<p >At the end of each 12
				month period, Manufacturing Group Limited reserves the right to
				benchmark the prices &amp; service levels of the goods/services
				received from the suppliers</p>
				</li><li>
<p >If in the event that a
				veriation above or equal to 10% is found between the current
				terms and the benchmark obtained by Manufacturing Group Limited ,
				an automating forward-looking price adjustment will be
				implemented by both parties</p>
				</li><li>
<p >Manufacturing Group
				Limited will not however accept any price increase during the
				initial 12 months of any agreement and suppliers must provide
				fixed pricing throughout this period 
				</p>
			</li></ul>
		</td>
	</tr>
</tbody></table>


<hr>
<h3 class="western">4.6 - Required service levels (SLAs)</h3>
<p>This section should contain the service level agreements that the
suppliers should abide by. This would comprise of:</p>
<ul>
	<li>
<p >Definition of issue levels
	(critical, severe, small), together with:</p>
	<ul>
		<li>
<p >Response timeframe</p>
		</li><li>
<p >Bug fix timeframe</p>
		</li><li>
<p >Permanent fix timeframe</p>
	</li></ul>
	</li><li>
<p >Recovery point objective</p>
	</li><li>
<p >Recovery time objective</p>
</li></ul>
<ul>
	<li>
<p >Support locations and
	languages</p>
	</li><li>
<p >Support portal from
	supplier (Helpdesk, portal, email, etc) 
	</p>
</li></ul>

<p>The reason why machine learning accuracy is not added in this
list, is because it is often too ambiguous to define an actual number
as an umbrella. Often the definition of the types of issue cover in
part the point of accuracy indirectly, as the system would become
non-functional if the accuracy decreases significantly. If there are
constraints in the SLA defined for accuracy, it should be approached
with care by both company and suppliers, as if not taken a high-level
approach, it may lead to a more brittle agreement that may affect
negatively the suppliers or the company (or both).</p>
<p><br>
<br>
<br>

</p>
<table >
	<colgroup><col width="0*">
	</colgroup><tbody><tr>
		<td  >
			<p>Example Required Service Levels for “Manufacturing Group
			Limited”</p>
			
			<p>We require our suppliers to provide us with a SLA consisting of
			the following points:</p>
			<ul>
				<li>
<p >Issue levels:</p>
				<ul>
					<li>
<p >Critical - Entire
					service is non-functional</p>
					<ul>
						<li>
<p >Walkaround within 48
						hours</p>
						</li><li>
<p >Bug-fix within 5
						working days</p>
					</li></ul>
					</li><li>
<p >Severe - Key
					functionality not usable, but others still are</p>
					<ul>
						<li>
<p >Walkaround within 72
						hours</p>
						</li><li>
<p >Bug-fix within 10
						working days</p>
					</li></ul>
					</li><li>
<p >Small - Functionality
					slightly affected but reasonable walkaround available</p>
					<ul>
						<li>
<p >Bug-fix within 20
						working days</p>
					</li></ul>
				</li></ul>
				</li><li>
<p >Permanent fix must be
				provided within 30 working days</p>
				</li><li>
<p >Response time should be
				3 hours.</p>
				</li><li>
<p >Language support should
				be English</p>
				</li><li>
<p >Support times are
				9:00-18:00 UTC</p>
				</li><li>
<p >Suppliers should provide
				access to helpdesk portal or email address</p>
			</li></ul>
		</td>
	</tr>
</tbody></table>


<hr>
<h3 class="western">4.7 - Financial and company information</h3>
<p>This section should outline the company information that suppliers
need to provide in their proposals. RFP authors should be aware when
setting the minimum financial requirement timelines, as there may be
less mature startups that would not be able to take part in the
tender process.</p>

<table >
	<colgroup><col width="0*">
	</colgroup><tbody><tr>
		<td  >
			<p>Example Financial and Company information for “Manufacturing
			Group Limited”</p>
			
			<ul>
				<li>
<p >Manufacturing Group
				Limited will require 3 years of audited financial accounts &amp;
				information for submission and evaluation by the team as part of
				each response</p>
				</li><li>
<p >All suppliers must
				ensure that they possess the appropriate tax, business and
				professional license registrations</p>
			</li></ul>
		</td>
	</tr>
</tbody></table>


<hr>
<h3 class="western">4.8 - Sub-contracting</h3>
<p>This section should outline whether subcontracting is expected for
suppliers. Depending on requirements, 
</p>

<table >
	<colgroup><col width="0*">
	</colgroup><tbody><tr>
		<td  >
			<p>Example Sub-contracting for “Manufacturing Group Limited”</p>
			
			<p>The supplier shall not subcontract any aspect of the service
			without prior written consent from Manufacturing Group Limited,
			whose consent shall not be responsibility of the supplier to
			ensure all subcontractors have received the necessary security
			clearance and training, where appropriate.</p>
		</td>
	</tr>
</tbody></table>

<h2 class="western" >5 - Contractual
terms &amp; conditions</h2>
<hr>
<h3 class="western">5.1 - Legal conditions</h3>
<p>This section should contain information about the contract that
will be used. It may be best practice to share a link to a contract,
or provide a template for suppliers to add comments in the first or
second phase of the procurement process.</p>

<table >
	<colgroup><col width="0*">
	</colgroup><tbody><tr>
		<td  >
			<p>Example legal conditions for “Manufacturing Group Limited”</p>
			
			<ul>
				<li>
<p >Successfully selected
				suppliers are expected to submit as part of their proposal, their
				agreement &amp; comments regarding the content of the following
				draft contract: &lt;LINK&gt;</p>
				</li><li>
<p >Suppliers will be scored
				based upon their re-submissions as part of the tender analysis
				and are likely to be disadvantaged if there are considerable
				changes. Manufacturing Group LImited will however favourably
				consider alterations that improve the content and wording of the
				contract.</p>
				</li><li>
<p >Please note that
				suppliers will also be asked as part of their submission in the
				questionnaire to confirm their agreement to various critical
				clauses contained within this draft contract</p>
			</li></ul>
			
		</td>
	</tr>
</tbody></table>

<hr>
<h3 class="western">5.2 - Validity</h3>
<p>This section should contain information on the validity of the
proposals, commercial offers, etc.</p>

<table >
	<colgroup><col width="0*">
	</colgroup><tbody><tr>
		<td  >
			<p>Example validity for “Manufacturing Group Limited”</p>
			
			<p>Suppliers agree that by participating in this process, any
			proposals or commercial offers shall remain valid for a minimum of
			90 days</p>
		</td>
	</tr>
</tbody></table>

<hr>
<h3 class="western">5.3 - Limited liability</h3>

<p>This section should provide information about the liability for
the company and the suppliers.</p>

<table >
	<colgroup><col width="0*">
	</colgroup><tbody><tr>
		<td  >
			<p>Example limited liability for “Manufacturing Group Limited”</p>
			
			<p>Manufacturing Group Limited accepts no liability for any of the
			costs associated with your response to this invitation to tender.
			Suppliers are therefore not entitled to compensation from
			Manufacturing Group Limited for any time, effort or expense
			incurred in preparing bids. You should only respond to this
			document in this basis.</p>
			
			<p>Manufacturing Group Limited reserves the right to:</p>
			<ul>
				<li>
<p >Not award the contract</p>
				</li><li>
<p >Not award the contract
				necessarily to the lowest bidder (but rather to the bidder
				offering the best value)</p>
				</li><li>
<p >To award the contract
				either in full or in part to one or more organisations</p>
				</li><li>
<p >To conduct negotiations
				with one or more bidders in the event that negotiations with the
				preferred bidder prove unsuccessful</p>
				</li><li>
<p >To cancel and/or
				re-issue the RfP</p>
				</li><li>
<p >To extend any deadlines
				and amend the procurement process</p>
			</li></ul>
		</td>
	</tr>
</tbody></table>


<hr>
<h3 class="western">5.4 - Confidentiality</h3>
<p>This section should contain the confidentiality agreement that
suppliers must agree in order to work with the company. The example
below assumes there is an external NDA that was previously signed by
suppliers. This may be required depending the confidential nature of
the project.</p>

<table >
	<colgroup><col width="0*">
	</colgroup><tbody><tr>
		<td  >
			<p>Example Confidentiality for “Manufacturing Group Limited”</p>
			
			<ul>
				<li>
<p >All information within
				this document must be treated with the strictest confidence and
				you are reminded that you must comply with the obligations set
				out in the Non-Disclosure-Agreement (NDA) that you have signed
				prior to receiving this document</p>
				</li><li>
<p >Suppliers as part of
				their response are expected to confirm their willingness to take
				all necessary steps to maintain the confidentiality of
				Manufacturing Group Limited information to which they have access
				and to outline the measure taken to maintain confidentiality of
				that information</p>
				</li><li>
<p >It will be the
				obligation of the supplier to ensure that all staff visiting
				Manufacturing Group Limited carries appropriate identification</p>
				</li><li>
<p >Suppliers should not
				enter into communication or correspondence with Manufacturing
				Group Limited personnel other than those listed on the subject of
				this RFP</p>
				</li><li>
<p >No parties may issue any
				press release or make public any announcement relating to this
				opportunity or subject matter without prior appoval of the other
				party</p>
			</li></ul>
			
		</td>
	</tr>
</tbody></table>
<p><br>
<br>
<br>

</p>

                            </section>
		</div>

		
			{% include footer.html %}


	</body>
</html>