polynomial regression torch

losagas

3 min read

·

11-03-2025

27

0

Share

Polynomial regression is a powerful technique used to model non-linear relationships between variables. Unlike linear regression, which assumes a linear relationship, polynomial regression fits a polynomial curve to the data, allowing it to capture more complex patterns. This guide will walk you through implementing polynomial regression using PyTorch, a popular deep learning framework. We'll cover the theory, implementation, and optimization, making it accessible for both beginners and those with some experience in machine learning.

Understanding Polynomial Regression

In simple terms, polynomial regression extends linear regression by adding polynomial terms (e.g., x², x³, etc.) to the model. This allows the model to fit curves instead of just straight lines. The general form of a polynomial regression model is:

y = θ₀ + θ₁x + θ₂x² + θ₃x³ + ... + θₙxⁿ

where:

• y is the dependent variable.
• x is the independent variable.
• θ₀, θ₁, θ₂, ..., θₙ are the model coefficients (parameters to be learned).
• n is the degree of the polynomial.

Choosing the right degree is crucial. A low degree might underfit the data (failing to capture the complexity), while a high degree might overfit (fitting the noise in the data instead of the underlying pattern).

Implementing Polynomial Regression with PyTorch

Let's build a PyTorch model for polynomial regression. We'll use a simple dataset for demonstration purposes.

import torch
import torch.nn as nn
import torch.optim as optim
import numpy as np
import matplotlib.pyplot as plt

# Generate sample data (replace with your own data)
np.random.seed(0)
X = np.linspace(-1, 1, 100)
y = 2*X**2 + 0.5*X + 1 + np.random.normal(0, 0.2, 100)

# Convert to PyTorch tensors
X = torch.tensor(X, dtype=torch.float32).reshape(-1, 1)
y = torch.tensor(y, dtype=torch.float32).reshape(-1, 1)


# Define the model
class PolynomialRegression(nn.Module):
    def __init__(self, degree):
        super(PolynomialRegression, self).__init__()
        self.poly = torch.nn.Linear(degree+1,1) # this will be a linear model trained on transformed X_data
        self.degree = degree
    
    def forward(self, x):
        x = torch.cat([x**i for i in range(self.degree + 1)],dim=1)
        return self.poly(x)


# Initialize the model and optimizer
model = PolynomialRegression(degree=2) # we expect it's a second order polynomial so degree=2
optimizer = optim.SGD(model.parameters(), lr=0.01) # Stochastic Gradient Descent

# Training loop
epochs = 1000
losses = []
for epoch in range(epochs):
    optimizer.zero_grad()
    outputs = model(X)
    loss = nn.MSELoss()(outputs, y)  # Mean Squared Error
    loss.backward()
    optimizer.step()
    losses.append(loss.item())
    if epoch%100==0:
        print(f"Epoch: {epoch}, Loss: {loss.item():.4f}")

# Plot the loss curve
plt.plot(losses)
plt.xlabel("Epoch")
plt.ylabel("Loss")
plt.title("Training Loss Curve")
plt.show()

# Make predictions
predicted = model(X).detach().numpy()

#Plot the results
plt.scatter(X.numpy(), y.numpy(), label='Original data')
plt.plot(X.numpy(), predicted, color='red', label='Fitted Polynomial')
plt.xlabel('X')
plt.ylabel('y')
plt.legend()
plt.title('Polynomial Regression')
plt.show()

This code first generates sample data following a quadratic pattern with added noise. Then, it defines a PolynomialRegression class which takes the polynomial degree as input. The forward pass transforms the input data into a matrix of polynomial features. A linear layer then learns the coefficients for each polynomial term. The code then trains the model using stochastic gradient descent (SGD) and plots the loss curve and the fitted polynomial against the original data. Remember to install the necessary libraries: torch, numpy, and matplotlib.

Choosing the Degree of the Polynomial

The degree of the polynomial is a hyperparameter that needs to be carefully chosen. A low degree might underfit the data, while a high degree might overfit. Techniques like cross-validation can help determine the optimal degree. Regularization techniques (like L1 or L2 regularization) can also help prevent overfitting with higher-degree polynomials.

Advanced Techniques

• Regularization: Adding L1 or L2 regularization to the loss function can prevent overfitting, especially with higher-degree polynomials.
• Different Optimizers: Experiment with different optimizers (Adam, RMSprop) to see if they improve convergence speed and performance.
• Feature Scaling: Scaling the input features can improve the training process.

Polynomial regression with PyTorch offers a flexible and efficient way to model non-linear relationships. By understanding the concepts and implementing the code, you can leverage this technique for various applications involving non-linear data. Remember to carefully choose the degree of the polynomial and consider regularization techniques to avoid overfitting and ensure optimal model performance.