Postgraduate Course: Information Theory (INFR11087)

Course Outline
School	School of Informatics	College	College of Science and Engineering
Course type	Standard	Availability	Available to all students
Credit level (Normal year taken)	SCQF Level 11 (Postgraduate)	Credits	10
Home subject area	Informatics	Other subject area	None
Course website	http://course.inf.ed.ac.uk/it	Taught in Gaelic?	No
Course description	Information theory describes the fundamental limits on our ability to store, process and communicate data, whether in natural or artificial systems. Understanding and approaching these limits is important in a wide variety of topics in informatics. This course covers the theory introduced by Shannon in 1948, which revolutionized how we think about information and communication, and some of the practical techniques for compression and reliable communication that have been developed since.

Entry Requirements (not applicable to Visiting Students)
Pre-requisites		Co-requisites
Prohibited Combinations		Other requirements	This course is open to all Informatics students including those on joint degrees. For external students where this course is not listed in your DPT, please seek special permission from the course organiser. - A solid mathematical background is required. - Essential maths knowledge: Special functions log, exp are fundamental; mathematical notation (such as sums) use throughout; some calculus. - Probability theory will be used extensively: Random variables, expectation, Bernoulli trials, Binomial distribution, joint and conditional probabilities. - A basic level of programming is assumed and not covered in lectures. The assessed assignment will involve programming in a language or your choice.
Additional Costs	None

Information for Visiting Students
Pre-requisites	None
Displayed in Visiting Students Prospectus?	Yes

Course Delivery Information

Delivery period: 2013/14 Semester 1, Available to all students (SV1)			Learn enabled: No	Quota: None
Web Timetable	Web Timetable
Course Start Date	16/09/2013
Breakdown of Learning and Teaching activities (Further Info)	Total Hours: 100 ( Lecture Hours 20, Seminar/Tutorial Hours 6, Summative Assessment Hours 2, Programme Level Learning and Teaching Hours 2, Directed Learning and Independent Learning Hours 70 )
Additional Notes
Breakdown of Assessment Methods (Further Info)	Written Exam 80 %, Coursework 20 %, Practical Exam 0 %
Exam Information
Exam Diet	Paper Name	Hours & Minutes
Main Exam Diet S1 (December)		2:00

Delivery period: 2013/14 Semester 1, Part-year visiting students only (VV1)			Learn enabled: No	Quota: None
Web Timetable	Web Timetable
Course Start Date	16/09/2013
Breakdown of Learning and Teaching activities (Further Info)	Total Hours: 100 ( Lecture Hours 20, Seminar/Tutorial Hours 6, Summative Assessment Hours 2, Programme Level Learning and Teaching Hours 2, Directed Learning and Independent Learning Hours 70 )
Additional Notes
Breakdown of Assessment Methods (Further Info)	Written Exam 80 %, Coursework 20 %, Practical Exam 0 %
Exam Information
Exam Diet	Paper Name	Hours & Minutes
Main Exam Diet S1 (December)		2:00

Summary of Intended Learning Outcomes
By taking this course, students should be able to: - Explain the source coding and noisy channel theorems and describe their implications for applications covered in lectures. - Compute information theoretic quantities, construct bounds and describe+implement algorithms involving high-dimensional probability distributions. - Describe the techniques covered in the course: identify their limitations, discuss their practical merits and design and describe alternatives. - For a novel data source, communication channel or application, identify relevant information theoretic aspects to provide insight or suggest useful methods.

Assessment Information
Written Examination: 80% Assessed Assignments: 20% Oral Presentations: 0% Assessment: The assessed assignments will test that students are keeping up with the material and test an ability to experiment and implement codes while solving problems.

Special Arrangements
None

Additional Information
Academic description	Not entered
Syllabus	-Differential Entropy and information content -Source coding theorem -Symbol codes, Kraft-McMillan inequality, Huffman codes -Stream codes, adaptive models, arithmetic coding -Compression in practice -Relative Entropy, mutual information, related inequalities -Noisy channel coding theorem, channel capacity -Error correcting codes -Codes robust to erasures -Lossy compression -Hash codes
Transferable skills	Not entered
Reading list	ESSENTIAL: "Information Theory, Inference and Learning Algorithms", David MacKay, CUP, 2003. http://www.inference.phy.cam.ac.uk/mackay/itila/book.html BACKGROUND ONLY: Elements of Information Theory, 2nd Edition, Cover and Thomas, Wiley 2006
Study Abroad	Not entered
Study Pattern	Lectures 20 Tutorials 6 Timetabled Laboratories 0 Non-timetabled Assessed Assignments 20 Private Study/Other 54 Total 100
Keywords	Not entered

Contacts
Course organiser	Dr Iain Murray Tel: (0131 6)51 9078 Email: I.Murray@ed.ac.uk	Course secretary	Ms Katey Lee Tel: (0131 6)50 2701 Email: Katey.Lee@ed.ac.uk