Beyond the Straight Line: Choosing Between OLS, Interaction Terms, and Tweedie Regression | Towards Data Science
Whether you should stick to a classic Ordinary Least Squares regression, introduce interaction terms, or pivot to a Tweedie distribution depends entirely on how your data handles the messy reality of zeros and extreme outliers.
a machine learning model to predict how much money customers will spend on an e-commerce platform over the next year. The job is data science business as usual: load the data, clean it, understand it, and model the outcome. The big question that comes to our minds:Which algorithm do you pull out of your toolkit?For regression models, I assume that we tend to default to the classics like OLS, or at other times, just jump straight to a complex ensemble method like XGBoost. But by experience, I have been seeing that often a generalized linear framework is exactly what you need for inference, speed, and interpretability. The real challenge lies in choosing the right flavor. Should we stick to a standard Ordinary Least Squares (OLS) regression?Do we need to introduce interaction terms?Or is our data weird enough to warrant aTweedieregression?Choosing the wrong one can lead to models that output impossible predictions, like a customer spending a negative amount of money. Let’s break down these three approaches so you can confidently pick the right tool for your specific data landscape. DatasetThe dataset used for this exercise will be theFrench Motor Third-Party Liability Claims dataset, from R package CASDatasets, under license GPL >2. Christophe Dutang and Arthur Charpentier (2026). CASdatasets: Insurance datasets, R package version 1. 2-1, DOI10. 57745/P0KHAG. # Basics import pandas as pd import numpy as np # Dataviz import matplotlib. pyplot as plt import seaborn as sns sns. set_style('darkgrid') import plotly. express as px # Stats import statsmodels. api as sm import statsmodels. formula. api as smf import scipy. stats as stats # Preprocessing & Modeling from sklearn. linear_model import TweedieRegressor from feature_engine. encoding import OneHotEncoder import warnings from sklearn. metrics import mean_absolute_error import warnings warnings. filterwarnings('ignore')To get the dataset, there’s a code snippet you can use inthis sklearn tutorial [4]. # Load Data df = load_mtpl2(n_samples=250_000)View of the dataset. Image by the author. After exploring the data a little, there are a couple of interesting transformations we can make, such as clipping the data at $500,000 maximum and keep the number of claims under 4, as numbers over that are way too crazy (outliers) and will definetely distort our predictions. # Clipped df = df. query('ClaimAmount 0). astype(int) # Prepare the data for "zero claim" classification df_ohe = ohe. fit_transform(df2) X = df_ohe. drop(columns=['ClaimAmount', 'has_claim']) y = df_ohe['has_claim']Next, there’s a bunch of code. We will split intrainandvalidationsets for LightGBM. Calculates weights to handle class imbalance. Sets tuned LightGBM model parameters. Trains LightGBM model with early stopping. from sklearn. model_selection import train_test_split print("Tuning LightGBM for P(Claim > 0) with updated parameters. . . ") # Split the training data into training and validation sets for early stopping X_train, X_val, y_train, y_val = train_test_split(X, y, test_size=0. 2, random_state=42, stratify=y) train_data_split = lgb. Dataset(X_train, label=y_train) val_data_split = lgb. Dataset(X_val, label=y_val) # Calculate scale_pos_weight for better handling of class imbalance scale_pos_weight_value = (y_train == 0). sum() / (y_train == 1). sum() # Define updated parameters for better control and to mitigate the warning params_tuned = { 'objective': 'binary', 'metric': 'auc', 'learning_rate': 0. 05, # Reduced learning rate 'num_leaves': 31, # Increased num_leaves (default is 31), allowing for more complexity 'max_depth': 7, # Increased tree depth, allowing for more complexity 'min_child_samples': 15, # Slightly decreased minimum data in leaf 'min_gain_to_split': 0. 01, # Decreased minimum gain for a split 'force_col_wise': True, # Keep for dataset compatibility 'scale_pos_weight': scale_pos_weight_value, # Explicitly handle imbalance 'verbose': -1 # Suppress verbose output during training } # Re-train LightGBM model with tuned parameters # Increase num_boost_round given the reduced learning rate and increased complexity model_tuned = lgb. train( params_tuned, train_data_split, num_boost_round=300, # Increased number of boosting rounds valid_sets=[val_data_split], # Provide validation set for early stopping callbacks=[lgb. early_stopping(20, verbose=False)] # Add early stopping and suppress verbose, increased patience )Predicts claim occurrence probabilities. Transforms it to a binary 0 or 1 for classification report calculation. Evaluates model using classification report. Stores predicted probabilities in dataframe. Sets probabilities below 0. 5 to zero. # Predictions and evaluation with the tuned model y_pred = (model_tuned. predict(X) > 0. 5). astype(int) print("\nClassification Report for Tuned LightGBM Model:") print(classification_report(y, y_pred)) # Store the probabilities for the next step of the Zero-Inflated model df2['prob_has_claim'] = model_tuned. predict(X) # If prob_has_claim is under 0. 5, make it 0 df2. loc[df2['prob_has_claim'] 0) using Tweedie Regression # Filter data for positive claims only df_positive = df_ohe[df_ohe['has_claim'] == 1]. copy() X_positive = df_ohe. drop(columns=['ClaimAmount', 'has_claim'], axis=1) y_positive = df_ohe['ClaimAmount'] # Use Exposure as offset for the severity model print("\nFitting Tweedie Model E[ClaimAmount | Claim > 0]. . . ") sklearn_tweedie_model = TweedieRegressor( power=1. 85, link='auto', solver='newton-cholesky', max_iter=500, alpha=1 ) sklearn_tweedie_model. fit(X_positive, y_positive, sample_weight=df['Exposure'])Finally, the prediction of this model will be theprobability estimated by the first modelmultiplied by theTweedie Regressor estimated amount. # Predict conditional severity for all observations (even those with 0 claims) # This assumes the severity model learned from positive claims can generalize. df['pred_severity_if_claim'] = sklearn_tweedie_model. predict(df_ohe. drop(columns=['ClaimAmount', 'has_claim'], axis=1)) # Calculate the final Zero-Inflated Model prediction df['ZeroInflated_pred'] = df2['prob_has_claim'] * df['pred_severity_if_claim'] # Display head of predictions and MAE print("\nZero-Inflated Model Predictions (first 5 rows):") print(df[['ClaimAmount', 'ZeroInflated_pred']]. sample(5)) mae_zero_inflated = mean_absolute_error(df['ClaimAmount'], df['ZeroInflated_pred']) print(f"\nMAE for Zero-Inflated Model: {mae_zero_inflated:. 2f}")This is the result. Fitting Tweedie Model E[ClaimAmount | Claim > 0]. . . Zero-Inflated Model Predictions (first 5 rows): ClaimAmount ZeroInflated_pred IDpol 1179635 0. 0 0. 000000 108738 0. 0 0. 000000 2066923 0. 0 0. 000000 1007186 0. 0 0. 000000 1117109 1128. 0 729. 428288 MAE for Zero-Inflated Model: 87. 79Great, we got another 21% of improvement in MAE. This is amazing!Zero Inflated model results Actual x Predictions. Image by the author. ConclusionData science is rarely about finding a single “perfect” algorithm that solves every problem. Instead, it is about matching the mathematical assumptions of your model to the real-world behavior of your data. Start simple. Look at the distribution of your target variable. If it looks like a bell curve, start withRegular OLS. If you know your features influence one another, introduceInteraction Terms. But if you are staring at a mountain of zeroes and a long tail of large numbers, skip the headaches of OLS entirely and leverage the power of aTweedie Regression. Taking a few minutes to analyze your target’s distribution before modeling can save you days of tuning the wrong architecture later. GitHub Repositoryhttps://github. com/gurezende/Zero-Inflated-Tweedie-RegressionReferences[1. Dataset](https://www. openml. org/search?type=data&sort=runs&id=43593&status=active)[2. Data License and citation](https://dutangc. github. io/CASdatasets/)Christophe Dutang and Arthur Charpentier (2026). CASdatasets: Insurance datasets, R package version 1. 2-1, DOI10. 57745/P0KHAG. [3. Tweedie Regression Documentation](https://scikit-learn. org/stable/modules/generated/sklearn. linear_model. TweedieRegressor. html)[4. Tweedie Regression Tutorial](https://scikit-learn. org/stable/auto_examples/linear_model/plot_tweedie_regression_insurance_claims. html)