Experiment Design for Innovators

Innovation is often associated with creativity, imagination and breakthrough ideas. While these qualities are important, successful innovation depends just as heavily on learning. The challenge is not simply generating ideas but determining which ideas deserve further investment and development.

Every innovation project contains uncertainty.

Teams make assumptions about users, customers, technologies, markets, behaviours and outcomes. Some assumptions prove correct. Others reveal opportunities, risks or entirely new directions. Experimentation provides a systematic way to explore these uncertainties before significant resources are committed.

Experimentation helps innovators move beyond opinions and predictions. It creates opportunities to generate evidence, validate assumptions and make informed decisions.

Rather than asking:

"Do we think this will work?"

Experimentation asks:

"How can we learn whether this works?"

In the Expand stage of the 5E Design Thinking Framework, experiments transform prototypes into learning opportunities and uncertainty into actionable insight.

Innovation exists within conditions of uncertainty.

When a challenge has a known solution, experimentation may be unnecessary. However, innovation often involves situations where:

• User needs are unclear
• Behaviours are unpredictable
• Technologies are evolving
• Stakeholder expectations differ
• Future outcomes are uncertain

Experiments help teams gather evidence before making major commitments.

They enable innovators to:

• Test assumptions
• Explore possibilities
• Reduce risk
• Validate ideas
• Compare alternatives
• Improve decision-making

Experiments do not eliminate uncertainty.

They help manage uncertainty more effectively.

One of the most important mindset shifts in innovation is recognising the difference between proving and learning.

The Proving Mindset

A proving mindset seeks confirmation.

People often design experiments hoping to demonstrate that their idea is correct.

This can lead to:

• Confirmation bias
• Selective interpretation
• Ignoring contradictory evidence
• Poor decision-making

The Learning Mindset

A learning mindset seeks understanding.

Instead of asking:

"How do we prove this idea works?"

Ask:

"What can we learn about this idea?"

Learning-focused innovators welcome evidence, even when it challenges assumptions.

The objective is not being right.

The objective is becoming better informed.

Every experiment begins with an assumption.

Assumptions are beliefs accepted as true before sufficient evidence exists.

Examples include:

• Students want more personalised learning.
• Customers will pay for convenience.
• Users prefer mobile-first experiences.
• AI can improve decision-making.
• People understand the interface.

Assumptions drive decisions whether they are visible or not.

The first step in experiment design is identifying them.

Desirability Assumptions

Focus on human needs.

Examples:

• Users have a problem.
• The problem matters.
• The solution creates value.

Feasibility Assumptions

Focus on implementation.

Examples:

• The technology works.
• Resources are available.
• The process is achievable.

Viability Assumptions

Focus on sustainability.

Examples:

• Revenue will exceed costs.
• Adoption is sufficient.
• Long-term operation is realistic.

Not all assumptions carry equal risk.

Effective innovators prioritise the assumptions most likely to influence success.

Assumptions become more useful when expressed as hypotheses.

A hypothesis is a testable prediction about what may occur.

Example Assumption

Students want real-time feedback.

Example Hypothesis

If students receive AI-generated feedback immediately after submitting draft work, engagement with revision activities will increase.

The hypothesis creates a clear statement that can be explored experimentally.

Strong hypotheses are:

Specific

Clearly define expected outcomes.

Testable

Can be evaluated through evidence.

Relevant

Address important uncertainties.

Observable

Generate measurable results.

A useful structure is:

We believe that [intervention] will result in [outcome] for [audience].

Example:

We believe that peer review activities will improve project quality for MBA students.

This structure creates clarity and focus.

Experiments often involve relationships between variables.

Independent Variable

The factor being changed or introduced.

Example:

• AI feedback
• New interface
• Revised service process

Dependent Variable

The outcome being observed.

Example:

• Student engagement
• Customer satisfaction
• Task completion rates

Understanding these relationships helps experiments generate clearer insights.

Good experiments require thoughtful design.

A useful process involves six steps.

Step 1: Define the Learning Objective

Ask:

What specifically needs to be learned?

Examples:

• Will people use the solution?
• Does the feature create value?
• Is the experience intuitive?
• Does the intervention influence behaviour?

The learning objective becomes the foundation of the experiment.

Step 2: Identify Critical Assumptions

Determine which assumptions create the greatest uncertainty.

Focus on assumptions that:

• Influence decisions
• Affect project success
• Involve significant risk

Experiments should target meaningful uncertainties rather than minor questions.

Step 3: Formulate a Hypothesis

Create a clear prediction.

Examples:

• Participants will complete tasks more quickly.
• Satisfaction scores will increase.
• More users will register.

The hypothesis provides direction for evidence collection.

Step 4: Choose an Experiment Method

Different questions require different methods.

Common approaches include:

Prototype Testing

Explores user interactions.

Interviews

Investigates perceptions and experiences.

Surveys

Collects structured feedback.

Pilot Programs

Tests solutions in practice.

A/B Testing

Compares alternative versions.

Simulations

Explores future scenarios.

MVP Experiments

Evaluates demand and viability.

Select the simplest approach capable of generating meaningful learning.

Step 5: Define Success Metrics

Metrics help determine whether the experiment generated useful outcomes.

Examples include:

Behaviour Metrics

• Purchases
• Registrations
• Participation rates

Performance Metrics

• Task completion
• Efficiency
• Accuracy

Experience Metrics

• Satisfaction
• Confidence
• Usability

Learning Metrics

• Reflection quality
• Knowledge development
• Skill acquisition

Success metrics should align directly with the hypothesis.

Step 6: Gather Evidence

Experiments become valuable when evidence is collected systematically.

Potential evidence sources include:

• Observation
• Interviews
• Analytics
• Test results
• Feedback forms
• Behavioural data

Multiple evidence sources often produce stronger insights.

Qualitative methods focus on understanding experiences and behaviours.

Examples include:

Interviews

Explore motivations and perspectives.

Observation

Study behaviour in context.

Role Play

Simulate future experiences.

Bodystorming

Explore ideas through physical participation.

Experience Simulations

Test service and customer experiences.

Qualitative methods help answer:

Why is this happening?

Quantitative methods focus on measurable outcomes.

Examples include:

A/B Testing

Compare alternative versions.

Surveys

Collect structured responses.

Analytics

Track behaviour patterns.

Controlled Experiments

Measure changes over time.

Quantitative methods help answer:

How much is this happening?

A/B testing compares two or more alternatives.

Example:

Version A presents one homepage design.

Version B presents another.

Behavioural outcomes are compared.

Possible metrics include:

• Click-through rates
• Registrations
• Purchases
• Engagement levels

A/B testing is useful when optimising existing experiences.

Pilot studies represent small-scale implementations.

They help answer questions such as:

• Will this work in practice?
• Are operational systems effective?
• What challenges emerge?

Pilots often bridge the gap between experimentation and implementation.

Many innovations benefit from rapid experimentation.

These experiments:

• Require minimal resources
• Generate quick learning
• Encourage adaptation
• Support iteration

Examples include:

• Landing pages
• Clickable prototypes
• Mock service experiences
• Simple behavioural tests

Rapid experiments prioritise learning speed.

Lean experimentation focuses on creating maximum learning with minimal investment.

The process often follows:

Build

Create a test.

Measure

Gather evidence.

Learn

Interpret findings.

Repeat

Improve and test again.

This approach encourages continuous adaptation.

Testing Multiple Assumptions Simultaneously

Complex experiments make learning difficult.

Focus on one important question at a time.

Using Vague Success Metrics

Metrics should be specific and meaningful.

Avoid relying solely on opinions.

Seeking Confirmation

Experiments should challenge assumptions, not merely validate them.

Ignoring Context

Behaviour often changes across different environments and situations.

Stopping After One Experiment

Learning accumulates over multiple experiments.

One test rarely provides complete understanding.

Innovation portfolios often contain multiple experiments running simultaneously.

Examples may include:

• User testing
• MVP validation
• Service simulations
• Behavioural experiments
• Pilot studies

Together, these activities create a richer understanding of opportunities and risks.

Artificial intelligence is creating new opportunities for experimentation.

Potential applications include:

• Automated testing
• Data analysis
• Pattern recognition
• Synthetic personas
• Scenario simulations
• Personalised experimentation

However, AI should support learning rather than replace human judgement.

Human-centred interpretation remains essential.

• Which assumptions underpin your project?
• What evidence would reduce uncertainty?
• What experiment could be conducted this week?
• How will success be measured?
• What would challenge your current thinking?

Great innovators are not defined by certainty.

They are defined by learning.

Experiments provide a structured way to explore uncertainty, test assumptions and generate evidence.

The most effective experiments:

• Focus on important assumptions
• Use clear hypotheses
• Gather meaningful evidence
• Support decision-making
• Encourage continuous learning

Innovation is not about predicting the future.

It is about creating opportunities to learn from it.