Part I · Choosing an approach
How to use this guide
By the end of this guide you will be able to take a research question, pick a machine learning approach that fits your data, evaluate it the way a careful reviewer would, and write it up as a study someone else could check and repeat. This is the last of four guides in the Vector Research machine learning track. Python for Machine Learning taught you the tools. Machine Learning Foundations taught you the models and the basic workflow. Deep Learning in Practice taught you to build and train neural networks in PyTorch. This guide is about judgment: which method, which evaluation, which comparison, and which claim your evidence can support.
What you'll be able to do when you finish
- Frame a question as a machine learning task and decide what kind of contribution your project makes.
- Choose a strong approach for tabular data, time series, text, high-dimensional scientific data, survival times, satellite images, simulations, molecules and proteins, images, and small or partly labeled datasets.
- Design an evaluation that matches how your model would really be used, with nested cross-validation, grouped or time-ordered splits, and external validation where possible.
- Find and remove data leakage using a published checklist.
- Compare models with the right statistical test, and say honestly when a difference is just noise.
- Tune hyperparameters efficiently and report the search.
- Measure calibration and wrap any model's predictions in intervals with a coverage guarantee (conformal prediction).
- Explain what a model learned, check it for fairness across groups, and know when an explanation is or isn't evidence.
- Tell prediction questions from causal questions, and estimate a treatment effect from observational data with its limits stated.
- Read, reproduce, and extend a published ML paper, and report your own study with the checklists reviewers use.
What you need first
You should be comfortable with everything in Machine Learning Foundations: pipelines, cross-validation, the standard metrics, the classic models, tree ensembles, and the basics of tuning. When this guide builds on one of its chapters, it says so ("Machine Learning Foundations, Chapter 5"). You'll also want Deep Learning in Practice if your data is images, audio, or long text, because this guide sends you there for all PyTorch code, CNNs, transformers, fine-tuning, and training recipes. For general statistics (confidence intervals, t-tests, effect sizes, multiple comparisons), the companion guide is Statistics for Research.
What the four guides cover together
The machine learning track is meant to be used as one reference. This map shows where each part of a project is taught.
| Stage of a project | Where it's taught |
|---|---|
| Python, NumPy, pandas, plotting | Python for Machine Learning, Chapters 1 to 11 |
| Framing the question and the task | From Idea to Research Question; this guide, Chapters 1 and 16 |
| Getting data and checking it exists | From Idea to Research Question, Chapter 8; Machine Learning Foundations, glossary ("Where to find datasets"); this guide, Chapter 6 (collecting and labeling your own) |
| Rules and approvals for data from people | From Idea to Research Question, Chapter 10; Writing, Posters and Presenting, Chapter 13; this guide, Chapter 12 |
| Cleaning, encoding, missing values, scaling | Machine Learning Foundations, Chapter 4; this guide, Chapter 2 |
| Splitting without leakage | Machine Learning Foundations, Chapter 5; this guide, Chapters 3, 5, and 7 |
| Classic models, trees, ensembles, clustering, PCA | Machine Learning Foundations, Chapters 7 to 12 |
| Gradient boosting at research grade | This guide, Chapter 2 |
| Neural networks in PyTorch, training, debugging | Deep Learning in Practice, Chapters 2 to 7 |
| Images, detection, segmentation, video | Deep Learning in Practice, Chapters 8 and 11; this guide, Chapter 5 |
| Signals, sensors, and audio | Deep Learning in Practice, Chapter 9; this guide, Chapter 3 |
| Text, transformers, LLMs, retrieval | Deep Learning in Practice, Chapters 10 and 11; this guide, Chapter 4 |
| Graphs, molecules, proteins | Deep Learning in Practice, Chapter 12; this guide, Chapter 5 |
| Survival times, high-dimensional biology, satellites, simulations | This guide, Chapter 5 |
| Generative and self-supervised models | Deep Learning in Practice, Chapter 13 |
| Compute, deployment, and reproducibility | Deep Learning in Practice, Chapters 14 and 15 |
| Metrics, tuning, and model choice | Machine Learning Foundations, Chapters 6 and 15; this guide, Chapters 7 and 9 |
| Statistical comparison and uncertainty | Statistics for Research; this guide, Chapters 8 and 10 |
| Interpretation, fairness, causal questions | This guide, Chapters 11 to 13 |
| Clustering, anomalies, recommenders, reinforcement learning | This guide, Chapter 14 |
| Reading, reproducing, and extending papers | Finding and Reading Research Papers; this guide, Chapter 15 |
| Writing up and presenting | Writing, Posters and Presenting; this guide, Chapter 16 |
To find the chapters for your project, go to Chapter 17. Its six playbooks walk through the most common project types step by step, and its table "Other project types, briefly" covers about 25 more, from survival analysis and audio to recommender systems and model deployment, with the baseline, the evaluation, and the chapters to read for each.
How each chapter is built
Each chapter opens with what it covers and why it matters for a research project. Then come the ideas in plain words, the key formula where there is one, code you can run, and the mistakes that most often sink student projects. Every printed result in this guide came from actually running the code. Each chapter ends with a short Check yourself list. If you can answer those without looking back, move on.
Running the code
Use Google Colab. It's free, runs in your browser, and comes with scikit-learn, NumPy, pandas, and matplotlib already installed. Code blocks within a chapter build on each other, so run them in order in one notebook. Most examples use only scikit-learn and NumPy, and their printed output is shown underneath. A few examples use extra libraries (XGBoost, LightGBM, CatBoost, Optuna, SHAP, sentence-transformers, Captum, Fairlearn, Gymnasium). Those blocks have no output shown, and each one says how to install what it needs with !pip install. This guide was checked against scikit-learn 1.9.1. Check your version with import sklearn; print(sklearn.__version__). If a number differs from the printed output in the last decimal place, a library version is the likely reason. Timings depend on your computer.
How to use the playbooks
Chapter 17 has step-by-step recipes for the most common student project types: tabular prediction, image classification, text classification, time-series forecasting, biomarker discovery from high-dimensional data, and unsupervised subtype discovery. A table at the end of that chapter covers many more types, including survival analysis, audio, object detection, NER, RAG, recommenders, molecules, proteins, satellite imagery, simulation surrogates, multimodal models, and deployment. If you already have a project, read Chapter 1, find your playbook or table row in Chapter 17, and follow its links back to the chapters it needs. Then read the rest when you have time.
A suggested pace
| Weeks | Chapters | Goal |
|---|---|---|
| 1 | 1 to 2 | Frame your project and build strong tabular baselines |
| 2 to 3 | 3 to 6 | Learn the approach for your data type: time series, text, scientific data, or little data |
| 4 to 5 | 7 to 10 | Design a leak-free evaluation, compare models, tune, and quantify uncertainty |
| 6 to 7 | 11 to 14 | Interpret, audit, and go beyond prediction where your question needs it |
| 8 | 15 to 17 | Reproduce a paper, plan and report your own study, and follow your playbook |
Go at whatever pace works for you. If your project is due soon, your mentor can help you jump straight to your playbook and the chapters it points to.
Part I · Choosing an approach · Chapter 1
What counts as an ML research contribution
Training a model is not the same as doing research. This chapter shows the kinds of contributions a machine learning project can make, how to turn a question into a well-defined ML task, and how to match your data and question to an approach.
Five kinds of contribution
A research project has to add something that wasn't known or available before. In machine learning, that addition usually takes one of five forms.
| Contribution | What's new | Student-sized example | What reviewers will check |
|---|---|---|---|
| New application or question | Using ML to answer a question in another field | Predicting which lakes will have harmful algal blooms from public water-quality data | Is the question important, and does the evaluation match real use? |
| New dataset | Data that didn't exist, collected and labeled carefully | 2,000 labeled photos of a local invasive plant, with an annotation guide and agreement statistics | Labeling quality, documentation, licensing, and a baseline result |
| New method | A change to how models are built or trained | A new feature-engineering step for sensor data, tested across several datasets | Fair comparison with strong baselines, ablations, and statistics |
| Evaluation or benchmark study | A careful test of existing methods | Do skin-lesion classifiers perform equally well across skin tones? | Rigor of the test design, not the novelty of the model |
| Replication | Checking whether a published result holds | Re-running a paper's code on new data and finding the reported accuracy was inflated by leakage | Faithful reproduction, clear documentation of differences |
Most strong student projects are new applications, new datasets, or evaluation studies. New methods are the hardest to do convincingly, because you must show the method beats well-tuned alternatives on more than one dataset. Replications are underrated. A clean replication that finds a flaw is a real contribution, and Kapoor and Narayanan (2023) built an influential paper around exactly that kind of work (Chapter 7).
Framing a question as an ML task
A research question becomes a machine learning task when you can name five things precisely. "Can we predict diabetes?" isn't framed yet. This is:
| Element | Question to answer | Example |
|---|---|---|
| Unit of analysis | One row is one what? | One adult patient at a routine checkup |
| Inputs | What is known at the moment of prediction? | Age, BMI, blood pressure, and fasting glucose from that visit |
| Output | What exactly are you predicting, and how was it measured? | A diabetes diagnosis within 5 years, from medical records |
| Population | Who should the model work for? | Adults aged 30 to 70 at primary care clinics |
| Use | What decision would the prediction support? | Flagging patients for an earlier glucose tolerance test |
The use row decides your evaluation. If the model will flag patients for a follow-up test, missing a sick patient is worse than an extra test, so recall at a fixed false-positive rate matters more than accuracy. If the model will be used at a different clinic, you need an external test set (Chapter 7). If it will be used next year, you need a time-based split (Chapter 3). Writing this table before you touch the data prevents most evaluation mistakes.
Some questions aren't prediction questions at all. "Does this supplement lower blood pressure?" is a causal question, and a model that predicts blood pressure well can still give the wrong answer to it (Chapter 13). "Are there subtypes of this disease?" is a discovery question, and its evidence looks different from a test-set score (Chapter 14).
Matching data and question to an approach
This table is the map for the rest of the guide. Find your data type and question, start with the baseline, then climb to the stronger approach only if it beats the baseline under an honest evaluation. For project types not listed here, such as object detection, RAG systems, satellite imagery, or model deployment, see the table at the end of Chapter 17.
| Data | Question | Start with (baseline) | Stronger approach | Chapters |
|---|---|---|---|---|
| Table of features (rows and columns) | Predict a label or number | Logistic or linear regression | Gradient-boosted trees | 2 |
| Measurements over time | Forecast future values | Naive and seasonal-naive forecasts | Lag features + boosted trees or linear models | 3 |
| Text | Classify documents | TF-IDF + logistic regression | Embeddings, fine-tuned transformer, or a validated LLM | 4 |
| Genes, proteins, or other p ≫ n data | Predict an outcome, find markers | Regularized logistic regression | Feature selection inside CV, stability analysis | 5 |
| Images | Classify or detect | Pretrained CNN as a feature extractor | Fine-tuned CNN or vision transformer | 5, 6; Deep Learning in Practice, Ch. 8 and 11 |
| Times until an event, some still unknown | Predict risk or time to event | Kaplan–Meier curves, Cox model | Random survival forest, judged by C-index | 5 |
| Audio or sensor signals | Classify recordings | Summary features + logistic regression | CNN on spectrograms or windows, pretrained audio model | 3; Deep Learning in Practice, Ch. 9 |
| Molecules or proteins | Predict a property | Fingerprints or embeddings + random forest or ridge | Graph neural network or fine-tuned protein model | 5; Deep Learning in Practice, Ch. 12 |
| Users and items | Recommend | Most-popular items | Item similarity, matrix factorization | 14 |
| Few labels, many unlabeled examples | Any of the above | Model trained on labeled data only | Transfer learning, self-training, active learning | 6 |
| Observational data | Does X cause Y? | Naive group difference (to show the bias) | Propensity scores, double machine learning | 13 |
| Unlabeled data | Find groups or anomalies | k-means, isolation forest | Stability-checked clustering, validated anomaly scores | 14 |
| An agent making decisions | Learn a strategy | Random or rule-based policy | Reinforcement learning | 14 |
The baseline habit
Every result in a research project needs a reference point. Use at least two baselines:
- A trivial baseline that uses no real information: predicting the most common class, the training mean, or yesterday's value. It tells you whether the problem is even hard.
- A simple strong baseline: logistic or linear regression on sensible features, a TF-IDF classifier for text, or a seasonal-naive forecast for time series. It tells you whether your fancy model is worth its complexity.
If you can, add a third: the published or current-practice baseline, such as the best result reported on the same dataset, or the rule doctors use today. "Our model reached an AUC of 0.91" means nothing alone. "Our model reached 0.91, compared with 0.88 for logistic regression and 0.84 for the clinical risk score in current use" is a finding.
Scope your project to what you can evaluate
The best student projects are narrow enough to evaluate thoroughly. "AI for cancer" can't be evaluated. "Do classifiers trained on one public chest X-ray dataset keep their accuracy on a second dataset from a different hospital?" can be evaluated in a summer with free data and Colab. When you choose between two ideas, pick the one where you can describe the test set in one sentence.
Unlock the rest of this guide
You've read the first chapter. The other 16 chapters and the glossary are free for every Vector Research applicant. Enter your access code to open them, and every other guide on this site.
What you'll unlock
- §02Tabular data at research grade
- §03Time series done right
- §04Text and large language models in research
- §05Scientific and high-dimensional data
- §06Little data and few labels
- §07Designing an evaluation
- §08Comparing models statistically
- §09Hyperparameter optimization
- §10Uncertainty and calibration
- §11Interpretability
- §12Fairness, robustness, privacy, and documentation
- §13Causal questions
- §14Discovery and decision-making beyond prediction
- §15Reading, reproducing, and extending ML papers
- §16Designing and reporting an ML study
- §17Project playbooks
- AppxGlossary and where to go next