Machine Learning Foundations

By the end of this guide you will be able to take a dataset, build several machine learning models on it, test them honestly, and explain your results to a judge. You don't need to know how to code yet, and the only math you need is algebra. Everything else is taught here.

What you'll be able to do when you finish

• Load, clean, and explore a real dataset in Python.
• Explain how the main families of models work: linear models, nearest neighbors, probabilistic models, trees, ensembles, clustering, and neural networks.
• Train those models with scikit-learn and Keras, and pick between them with cross-validation.
• Choose the right metric for your problem and know what the number actually means.
• Spot the mistakes that sink student projects, like data leakage and overfitting, before a judge does.

How each chapter is built

Every model chapter follows the same order: the idea in plain words, the math behind it, code you can run, the settings that matter, and when to use it or avoid it. Each chapter ends with a short Check yourself list. If you can answer those without looking back, move on.

Running the code

Use Google Colab. It's free, runs in your browser, and already has every library this guide uses. Make a new notebook, paste a code block into a cell, and press Shift + Enter. If you'd rather work on your own computer, install Python and Jupyter through Anaconda.

Type the code out yourself instead of copying it, at least for the first few chapters. It's slower, but you'll remember it.

A suggested pace

Go at whatever pace works for you. Your mentor can help you skip ahead to what your project needs first.

Machine learning is a way of building programs that learn patterns from examples instead of following rules a person wrote by hand. You give the computer data where you already know the answers, it finds a rule that connects the inputs to the answers, and then it uses that rule on new data it has never seen.

Here is the difference. To sort penguins by species the old way, you would write rules yourself: "if the bill is longer than 45 mm and the flipper is longer than 210 mm, it's a Gentoo." With machine learning, you hand the computer measurements from a few hundred penguins whose species you already know, and it works out the rules on its own. Often it finds better rules than you would have.

The vocabulary you'll use every day

Most confusion in ML comes from the same thing having three names. Learn these once.

By convention, X (capital) is the table of input features, with one row per instance and one column per feature. y (lowercase) is the single column of outputs. The number of instances is written n and the number of features is p.

The two big families

Supervised learning means you have the answers for your training data. The model learns from labeled examples. There are two kinds of supervised problems, and which one you have decides almost everything else.

• Regression predicts a number. Examples: a house's rent, a patient's blood insulin level, a wine's quality score.
• Classification predicts a category. Examples: penguin species, spam or not spam, tumor benign or malignant. When there are only two categories it's called binary classification; three or more is multiclass.

A quick test: if the answer could be 3.7, it's regression. If the answer is a name or a yes/no, it's classification.

Unsupervised learning means you have no answers, only inputs. The model looks for structure on its own.

• Clustering finds groups of similar instances, like sorting customers into shopping styles.
• Dimensionality reduction squeezes many features into a few that keep most of the information, which helps with visualization and speed.

Two other families exist that this guide only mentions. Semi-supervised learning uses a few labeled examples plus many unlabeled ones. Reinforcement learning trains an agent through rewards and penalties, the way game-playing AIs learn.

Supervisedyou have the answers (labels)

Regression predict a number

• Linear regression
• Ridge & LASSO
• Polynomial & splines
• k-nearest neighbors
• Decision trees
• Random forests & boosting
• Neural networks

Classification predict a category

• Logistic regression
• Naive Bayes
• Support vector machines
• LDA
• k-nearest neighbors
• Decision trees
• Random forests & boosting
• Neural networks

Unsupervisedinputs only, no answers

Clustering find groups

• k-means
• Hierarchical
• DBSCAN & HDBSCAN
• Gaussian mixtures

Dimensionality reduction fewer features

• PCA
• ICA
• Factor analysis
• MDS & Isomap
• t-SNE & UMAP

Every model in this guide belongs to one of these four boxes. Trees, forests, k-nearest neighbors, and neural networks appear twice because they handle both regression and classification.

Learning versus memorizing

The goal of ML is never to do well on the data you trained on. The goal is to do well on new data. A model that memorizes its training examples, noise and all, will look perfect in training and fail in the real world. This is called overfitting. A model that is too simple to capture the real pattern is underfitting.

Think of studying for a test. Memorizing the answers to last year's practice exam is overfitting: you'll ace that exact exam and fail a new one. Only reading the chapter titles is underfitting. Understanding the ideas is what generalizes. Almost every technique in Chapters 5, 6, and 15 exists to help you find that middle ground and prove you found it.

The machine learning workflow

Every project, from a science fair entry to a model at a tech company, follows roughly these steps.

1. Ask a question that data can answer. "Can tumor cell measurements predict malignancy?" is answerable. "Is cancer bad?" is not.
2. Collect the data, from a public dataset, a lab, a survey, or your own measurements.
3. Explore it. Plot it, count missing values, look for strange entries.
4. Split it into training and test sets before you change anything else (Chapter 5).
5. Prepare it: clean, fill missing values, scale, encode categories (Chapter 4).
6. Train several models and compare them with cross-validation (Chapters 5 and 15).
7. Tune the best candidates' hyperparameters.
8. Test once on the held-out test set to get an honest final score.
9. Interpret and report: what did the model learn, where does it fail, and what does it mean for your question?

Steps 4 and 5 look out of order, and that's on purpose. Splitting first stops information from your test set leaking into training, which is one of the most common ways student projects end up with results that are too good to be true.