This reference remains available when scripts or browser storage are unavailable.
Concepts
Match the classifier to the target structure
Classification predicts categorical targets or probabilities over categories. Binary tasks have two outcomes, multiclass tasks choose among more than two classes, multilabel tasks can assign several labels to one observation, and ordinal tasks use ordered classes whose gaps are not necessarily equal.
How to interpret it: Identify whether one or several labels can apply and whether label order matters before choosing modeling and evaluation methods. An ordinal target preserves order without turning the levels into an ordinary continuous measurement.
Common mistakes
- Treating a multilabel task as multiclass even though several labels can be correct at once.
- Treating ordered labels as equally spaced numerical values without justification.
Sources: lectures/lecture_05_classification_part_1/lecture_notes.md; lectures/lecture_05_classification_part_1/README.md
KNN depends on meaningful distances
K-Nearest Neighbors stores the training data and predicts from the majority class among the k nearest observations. The distance metric, feature scaling, dimensionality, and choice of k all change which observations count as neighbors.
How to interpret it: Scale numerical features before distance-based comparison when their units differ, and select k with validation. Small k can follow local noise, while large k creates smoother neighborhoods that may blur a real boundary.
Common mistakes
- Letting a large-unit feature dominate distance solely because of its measurement scale.
- Choosing k from training performance alone.
Sources: lectures/lecture_05_classification_part_1/lecture_notes.md; lectures/lecture_05_classification_part_1/lecture_examples/README.md; lectures/lecture_05_classification_part_1/practical_session/README.md
Tree splits seek purer child nodes
A decision tree recursively applies feature-threshold rules. Gini impurity and entropy measure how mixed the classes are in a node, and a useful split reduces impurity or uncertainty in its children.
How to interpret it: Read a split as a local rule that partitions the current observations. Low impurity means one class dominates the node; it does not by itself prove that the rule will generalize to new data.
Common mistakes
- Assuming every impurity-reducing training split improves validation performance.
- Reading Gini impurity as a feature importance score.
Sources: lectures/lecture_05_classification_part_1/lecture_notes.md; lectures/lecture_05_classification_part_1/lecture_examples/README.md
Control tree complexity before it memorizes noise
Deep trees can keep making greedy splits until small training patterns are memorized. Maximum depth, minimum samples per split or leaf, pruning, and cost-complexity control reduce effective tree complexity.
How to interpret it: Compare training and validation behavior while changing a complexity control. A shallow tree may underfit, while an unconstrained tree can be accurate on training data yet fragile on unseen observations.
Common mistakes
- Choosing the deepest tree because it has the highest training accuracy.
- Calling a very deep tree easy to interpret merely because every rule is technically visible.
Sources: lectures/lecture_05_classification_part_1/lecture_notes.md; lectures/lecture_05_classification_part_1/practical_session/README.md
Separate probability estimation from the decision threshold
Logistic regression applies the sigmoid function to a linear score, producing an estimated positive-class probability between zero and one. A threshold, often 0.5 by default, then converts that probability into a class decision.
How to interpret it: Choose the threshold according to the cost of false positives and false negatives. Raising it usually produces fewer positive predictions; lowering it usually catches more positives but can also create more false alarms.
Common mistakes
- Treating 0.5 as the correct threshold for every application.
- Interpreting a positive coefficient as a guaranteed causal effect.
Sources: lectures/lecture_05_classification_part_1/lecture_notes.md; lectures/lecture_05_classification_part_1/lecture_examples/README.md; lectures/lecture_05_classification_part_1/practical_session/README.md
Name each kind of classification outcome
For a binary classifier, the confusion matrix counts true positives, false positives, true negatives, and false negatives. These four counts distinguish correct decisions from the two error types.
How to interpret it: Define the positive class first, then ask which error matters in the application. False-positive rate divides false positives by all actual negatives, while false-negative rate divides false negatives by all actual positives.
Common mistakes
- Swapping false positives and false negatives by forgetting what the actual class was.
- Comparing error counts without considering the number of actual positives or negatives.
Sources: lectures/lecture_05_classification_part_1/lecture_notes.md; lectures/lecture_05_classification_part_1/practical_session/README.md
Choose metrics around the error cost
Precision is TP divided by TP plus FP and asks how often positive predictions are correct. Recall is TP divided by TP plus FN and asks how many actual positives were found. F-beta combines precision and recall, with beta above one emphasizing recall and beta below one emphasizing precision.
How to interpret it: Accuracy can hide failure on a rare class. Prefer precision when false positives are especially costly, recall when missing positives is especially costly, and an F-score when a stated balance between them is useful.
Common mistakes
- Reporting only accuracy for a strongly imbalanced target.
- Assuming F1 reveals whether precision or recall is the weaker component.
Sources: lectures/lecture_05_classification_part_1/lecture_notes.md; lectures/lecture_05_classification_part_1/practical_session/README.md
Evaluate rankings and probabilities, not only hard labels
The ROC curve plots true-positive rate against false-positive rate across thresholds, while ROC-AUC summarizes ranking quality. Log loss evaluates predicted probabilities directly and heavily penalizes confident wrong predictions.
How to interpret it: Use ROC-AUC to examine threshold-spanning ranking behavior and log loss when probability quality matters. A score-distribution view can reveal class overlap that makes threshold selection difficult, even when one hard-label metric looks acceptable.
Common mistakes
- Reading AUC as accuracy at the default threshold.
- Ignoring overconfident probability errors because the final class label happened to be the main reported output.
Sources: lectures/lecture_05_classification_part_1/lecture_notes.md; lectures/lecture_05_classification_part_1/practical_session/README.md
Visualization reference
Decision threshold and confusion matrix
These fixed illustrative scores separate probability estimation from the threshold used to make a positive or negative decision. Move the threshold to see how true positives, false positives, true negatives, false negatives, precision, and recall change together.
Graph fallback: At the 0.50 decision threshold, the fixed illustrative cases give TP 4, FP 2, TN 4, and FN 2. Precision and recall are both about 0.67. Raising the threshold predicts fewer positives; lowering it predicts more positives.
Class-aware decision boundary explorer
The same illustrative Class A and Class B points stay visibly tied to their class labels while you compare candidate linear boundaries. A boundary changes the predicted side of a point; it does not change its true class.
Graph fallback: Class A occupies the lower-left portion of the illustrative feature space and Class B the upper-right. Compare the balanced, conservative-positive, and permissive-positive boundaries while keeping the true A/B class labels fixed.