diochnos/teaching/CS4033-5033/2025S
CS 4033/5033 – Machine Learning Fundamentals (Spring 2025)
The class is cross-listed as CS 4033 and CS 5033, so that both undergraduate and graduate students can enroll simultaneously. No student may earn credit for both 4033 and 5033.
Table of Contents
Course Description
Topics include decision trees, relational learning, neural networks, Bayesian learning, reinforcement learning, multiple-instance learning, feature selection, learning appropriate representations, clustering, and kernel methods. No student may earn credit for both 4033 and 5033.
[Course Description] [Table of Contents] [Top]
Basic Information
Syllabus
The syllabus is available here.
Time and Location
Mondays, Wednesdays, and Fridays, 11:00am – 11:50am, Sarkeys Energy Ctr N0202.
Contact Information
Please see here.
Teaching Assistants
This class has no teaching assistants.
Office Hours
We will be holding our office hours at the following times.
- Mondays
- 2:00pm – 2:50pm, 230 Devon Energy Hall (Dimitris)
- Wednesdays
- 3:00pm – 3:50pm, 230 Devon Energy Hall (Dimitris)
Exceptions to the Regular Schedule of Office Hours
If you want to meet me outside of my office hours, please send me an email and arrange an appointment.
Wednesday, March 12, 2025: I will be holding office hours between 3:30pm – 4:00pm. That is, only the second half of the hour I will be available to see students as I have a conflict with another appointment during the first half of the hour.
[Basic Information] [Table of Contents] [Top]
Homework Assignments
Assignment 1: Announce on Mon, Jan 20, 2025. Due on Wed, Jan 29, 2025.
Assignment 2: Announce on Mon, Feb 3, 2025. Due on Mon, Feb 17, 2025.
Assignment 3: Announce on Mon, Feb 24, 2025. Due on Fri, Mar 7, 2025.
Assignment 4: Announce on Fri, Mar 7, 2025. Due on Wed, Mar 26, 2025.
Assignment 5: Announce on Wed, Mar 26, 2025. Due on Mon, Apr 7, 2025.
Assignment 6: Announce on Mon, Apr 7, 2025. Due on Sun, Apr 27, 2025.
[Homework Assignments] [Table of Contents] [Top]
Projects
Information related to the projects will show up here.
Ideas for projects
Below are some ideas for your projects.
Reinforcement Learning Ideas
- Gymnasium from Farama Foundation (continuation of the OpenAI Gym):
provides an interface for training your own RL agent to play a computer game.
(Try to select a simple game so that it is easier to deal with it.) - Simple board games for RL.
- Variations of bandit problems.
- A gym environment for the classic game of snake is available here.
- Avoid constraint satisfaction problems (e.g., Sudoku, Worldle, etc.)
Supervised Learning Ideas
Before we discuss any ideas, as a reminder, you cannot use MNIST, because that dataset has been studied with any conceivable algorithm at this point and all the information is available for free online.
- UCI repository: datasets that are available from the repository that is maintained by the University of California, Irvine.
- Make Moons: a synthetic dataset where you can also compare the performance of the models that you will develop from scratch with the equivalent ones from scikit-learn.
- Kaggle: has lots of datasets and you may actually be able to participate in a competition and see how your algorithm and your approach compares against others.
- KDnuggets: datasets from KD nuggets.
- Smiling or not?
- Fashion MNIST
- ImageNet (Though, I would suggest you work with the Tiny ImageNet)
- CIFAR-10 and CIFAR-100
- SVHN (Street-View House Numbers)
- Please note that ImageNet, CIFAR-10, CIFAR-100, and SVHN, are all difficult datasets.
- As a sidenote, if you want to reduce the dimensionality of some datasets (so that you can speed up computations), you can use dimension reduction techniques (e.g., Principal Component Analysis). You are free to use libraries exclusively for this task.
[Projects] [Table of Contents] [Top]
Milestones
Week 2: Homework 1 is announced (beginning of week).
Week 3: Homework 1 is due (mid-week). In-class presentations for the reinforcement learning project (end of week).
Week 4: Homework 2 is announced (beginning of week). Project written proposal is due (beginning of week).
Week 6: Homework 2 is due (beginning of week). Project checkpoint is due (end of week).
Week 7: Homework 3 is announced (beginning of week).
Week 8: Homework 3 is due and homework 4 is announced (end of week).
Week 9 (Spring Break): Reinforcement learning project is due at the end of week.
Week 10: In-class presentations for the supervised learning project (end of week).
Week 11: Homework 4 is due and homework 5 is announced (mid-week). Project written proposal is due (beginning of week).
Week 13: Homework 5 is due and homework 6 is announced (beginning of week). Project checkpoint is due (end of week).
Week 15: Homework 6 is due (end of week).
Week 16: Supervised learning project is due (end of week).
[Milestones] [Table of Contents] [Top]
Machine Learning Resources
Books
The three books that we plan to use for the course are available for free in electronic format in the following links:
- Reinforcement Learning: An Introduction, by Richard S. Sutton and Andrew G. Barto. (1st edition)
- An Introduction to Statistical Learning (with Applications in Python), by Gareth James, Daniela Witten, Trevor Hastie, and Robert Tibshirani.
- Machine Learning, by Tom Mitchell.
Personal Notes
Notes by Others
- Recall, Precision, F1, ROC, AUC, and everything, by Ofir Shalev. (local pdf copy)
- Neural Networks and Deep Learning, by Michael Nielsen.
- Animation comparing ridge and lasso regression.
Papers
- Machine Learning that Matters, by Kiri L. Wagstaff
- A Few Useful Things to Know About Machine Learning, by Pedro Domingos. (alternate version)
- Perceptron-based learning algorithms, by Stephen I. Gallant. (Optional reading; this is the paper for the pocket algorithm.)
[Machine Learning Resources] [Table of Contents] [Top]
Class Log
A log for the class will be held online here.
Week 1
Class 1 (Jan 13, 2025)
About this Course.
Discussion on syllabus and policies.
Class 2 (Jan 15, 2025)
Discussion on projects. Introduction to Machine Learning.
Assigned Reading: Elements of Statistical Learning (ESL), Chapter 1.
Assigned Reading: Sutton & Barto: Chapters 1 and 2.
Assigned today: Think about short and long projects. Think about the topic for your RL project.
Class 3 (Jan 17, 2025)
Introduction to reinforcement learning.
Basic ingredients of RL methods: policy, value function, model.
Discussion on the projects, deadlines, and various expectations. Also, where we can find certain information on Canvas.
Week 2
Monday, January 20, 2025
No class today;Martin Luther King day.
Classes 4-5 (Jan 22-24, 2025)
Assigned (Mon): Homework 1.
Exploration vs Exploitation. The multi-armed bandit problem from the book (Chapter 2).
The prediction problem and the control problem.
Markov Decision Processes (MDPs).
Discussion on the Bellman Expectation Equations. Backup diagrams and solution of the prediction problem using linear algebra. Revisited the recycling robot example and we showed how we can evaluate the policy that picks an action with the same probability at each of the two energy states of the robot.
Bellman optimality equations and the control problem.
Assigned Reading (Mon, 1/30/2023): Sutton & Barto: Chapter 3.
Week 3
Classes 6-7-8 (Jan 27-29-31)
Dynamic programming methods. Policy iteration, value iteration, as well as we had some last remarks on dynamic programming (complexity, asynchronous backups, etc.)
Started our discussion on model-free methods that are used for prediction.
Fri, 1/31/2025: Proposals for reinforcement learning projects; in-class presentations.
Due (Feb 2): Homework 1.
Week 4
Classes 9-10-11 (Feb 3-5-7)
Assigned (Mon, Feb 3): Homework 2.
Due Mon, Feb 3: Written proposal for the reinforcement learning project.
Assigned Reading: Sutton & Barto: Chapter 4.
Overview of Monte-Carlo and Temporal Difference learning.
First-visit and every-visit Monte Carlo methods. Application to Blackjack.
Iterative calculation of empirical average.
Temporal difference learning. Application to the "Return Home" example.
Comparison of Monte Carlo and Temporal Difference learning.
Assigned Reading (Wed): Sutton & Barto: Sections 5.1, 5.2, 6.1, 6.2, 6.3.
n-Step Returns and Eligibility Traces. Forward view, backward view, and equivalence.
Assigned Reading: Sutton & Barto: Sections 7.1, 7.2, 12.1, 12.2.
The control problem. Using $\varepsilon$-greedy policy in order to guarantee enough exploration of the state space so that we are able to calculate accurate optimal values for the value functions.
Solution with an on-policy Monte-Carlo approach.
Assigned Reading (Mon): Sutton & Barto: Sections 5.3, 5.4.
Week 5
Classes 12-13-14 (Feb 10-12-14)
Discussion on information about the class and the second homework.
Continue our discussion on solving the control problem. This time we use the idea of TD learning which leads to Sarsa and we will also discuss the extension to Sarsa($\lambda$).
Assigned Reading (Wed): Sutton & Barto: Sections 6.4, 6.5, 12.7.
Solving the control problem using an off-policy method: Q-Learning.
Introduction to function approximation.
How can we do linear function approximation and solve the prediction and the control problem using our basic methods.
Simple ways to construct features: state aggregation, coarse coding, tile coding. The book has more examples.
Discussion of some examples with function approximation. Among them, Sarsa with linear function approximation on the Mountain Car problem.
Assigned Reading: Sutton & Barto: Sections 9.1 – 9.5, 10.1, 10.2.
Due Sat: Homework 2.
Week 6
Class 15 (Feb 17)
Assigned today: Homework 3.
Review discussion on reinforcement learning.
Class 16 (Feb 19)
Midterm exam on reinforcement learning.
Class 17 (Feb 21)
Introduction to supervised learning.
What is supervised learning? Regression and classification problems. The role of inductive bias.
Definitions on terminology that we will be using throughout the course for supervised learning.
Assigned Reading: ISL 2.1.
Further discussion on the notion of the hypothesis space $\mathcal{H}$. The different algorithms that we will see on supervised learning, largely define how we will perform the seach in this space and come up with a hypothesis $h$ that we will believe will approximate well the ground truth $c$.
Introduction to nearest neighbor learning. Example with flowers based on sepal length and sepal width.
1-Nearest Neighbor classification and connection to the Voronoi Diagram (this is a topic discussed in Computational Geometry classes).
Assigned Reading: Nearest neighbors: Mitchell 8.1, 8.2 – 8.2.3. Alternatively, ISL 2.2.3 (classification) and 3.5 (regression).
[Class Log] [Table of Contents] [Top]