Undergraduate Course: Robot and Reinforcement Learning (INFR11285)
Course Outline
| School | School of Informatics |
College | College of Science and Engineering |
| Credit level (Normal year taken) | SCQF Level 11 (Year 4 Undergraduate) |
Availability | Available to all students |
| SCQF Credits | 20 |
ECTS Credits | 10 |
| Summary | This course is about enabling agents (e.g., robots, software systems) to plan and act in complex domains. Such agents are increasingly being deployed in applications such as healthcare, self-driving cars, and warehouse automation, but they still need considerable prior knowledge and/or human supervision. This course will explore the challenges and opportunities that arise when modern machine learning methods are leveraged to create adaptive and continually improving agents.
In particular, we will explore reinforcement learning methods that solve sequential decision making problems and are being used in many application domains. We will focus on the development and use of such learning algorithms in the challenging context of problems in robotics and cyber-physical systems that are characterised by continuous state and action spaces, under-actuation, partial observability, data sparsity, and the need for safe behaviour. No prior knowledge of robotics is assumed but students will be expected to engage with the robotics problems used to motivate the design of learning algorithms. |
| Course description |
The course will begin with an introduction to sequential decision making and optimal control as the foundation to planning and acting without learning. We will discuss the fundamentals of reinforcement learning, focusing on dynamic programming and Monte Carlo methods for sequential action selection. We will introduce the development of a model and its importance to (robot) planning and control algorithms. We will cover methods for adaptive control and learning a dynamics model from data, exploring how these models enable model-based reinforcement learning. We will also discuss model-free reinforcement learning methods (e.g., temporal difference methods) and their use when we are unable to model aspects of the environment. In addition, we will discuss how to bootstrap learning via demonstration and how physical simulation may aid exploration.
The field of robotics adapts quickly to the latest developments in AI. By grounding the learning methods in the context of robot decision-making and control, students will be exposed to the latest research, and they will critically analyse, implement, and evaluate modern learning algorithms in a physically-realistic simulation environment.
List of topics may include:
1. Introducing to sequential decision making
2. Fundamentals of reinforcement learning
3. Sequential action selection: Dynamic Programming, Monte Carlo
4. Adaptive control and dynamics model learning
5. Reinforcement learning applied to robotics: model-based, model-free
6. Imitation learning and function approximation
7. Actor-critic and gradient-based optimisation
8. Offline robot learning and the role of simulation
9. Robot foundation models
10. Training and evaluating performance
|
Entry Requirements (not applicable to Visiting Students)
| Pre-requisites |
|
Co-requisites | |
| Prohibited Combinations | Students MUST NOT also be taking
Robot and Reinforcement Learning (UG) (INFR11290)
|
Other requirements | MSc students must register for this course, while Undergraduate students must register for INFR11290 instead.
Students are expected to be proficient in mathematical concepts, particularly:
Probability theory and statistics.
Linear algebra and geometry (e.g., vectors, matrices, rotations, trigonometry).
Calculus (e.g., integration, differentiation)
Students are also expected to be proficient in a programming language, ideally python, and familiar with working in Linux OS systems. |
Information for Visiting Students
| Pre-requisites | MSc students must register for this course, while Undergraduate students must register for INFR11290 instead.
Students are expected to be proficient in mathematical concepts, particularly:
Probability theory and statistics.
Linear algebra and geometry (e.g., vectors, matrices, rotations, trigonometry).
Calculus (e.g., integration, differentiation)
Students are also expected to be proficient in a programming language, ideally python, and familiar with working in Linux OS systems.
|
Course Delivery Information
|
| Academic year 2025/26, Available to all students (SV1)
|
Quota: None |
| Course Start |
Semester 2 |
Timetable |
Timetable |
| Learning and Teaching activities (Further Info) |
Total Hours:
200
(
Lecture Hours 26,
Seminar/Tutorial Hours 2,
Summative Assessment Hours 2,
Revision Session Hours 2,
Programme Level Learning and Teaching Hours 4,
Directed Learning and Independent Learning Hours
164 )
|
| Assessment (Further Info) |
Written Exam
70 %,
Coursework
30 %,
Practical Exam
0 %
|
| Additional Information (Assessment) |
Written Exam __70%
Coursework __30%
|
| Feedback |
Example problem sets and revision lectures will be provided to help with exam preparation. |
| Exam Information |
| Exam Diet |
Paper Name |
Minutes |
|
| Main Exam Diet S2 (April/May) | Robot and Reinforcement Learning (INFR11285) | 120 | |
Learning Outcomes
On completion of this course, the student will be able to:
- explain key concepts of sequential decision making, reinforcement learning, and robot planning and control.
- analyze and evaluate decision-making problems in robot control, and identify those to which learning methods can be applied.
- implement and analyse a subset of established learning algorithms for dynamics learning and stochastic control.
- explain limitations of current learning and optimisation algorithms in the context of robot control problems.
|
Reading List
(i) Reinforcement Learning: An Introduction (second edition). R. Sutton and A. Barto. MIT Press, 2018.
(ii) Algorithms for Reinforcement Learning. C. Szepesvari. Morgan and Claypool Publishers, 2010. |
Additional Information
| Graduate Attributes and Skills |
Problem solving, critical/analytical thinking, handling ambiguity, knowledge integration, planning and organizing, independent learning, creativity, written communication. |
| Keywords | FFL |
Contacts
| Course organiser | Dr Michael Mistry
Tel: (0131 6)50 2937
Email: mmistry@exseed.ed.ac.uk |
Course secretary | Miss Toni Noble
Tel: (0131 6)50 2692
Email: Toni.noble@ed.ac.uk |
|
|
University Homepage