![](https://crypto4nerd.com/wp-content/uploads/2023/07/1rA80EhaJXnDsRvXR3YZ-Xw.png)
We utilize the mathematical formulas mentioned above to code the Logistic Regression model and will evaluate its performance on some benchmark datasets.
Firstly, we initialize the class and parameters.
class LogisticRegression():
def __init__(
self,
learning_rate:float=0.001,
n_iters:int=10000
) -> None:
self.n_iters = n_iters
self.lr = learning_rateself.weights = None
self.bias = None
We require the weights and bias values that will be optimized, so we initialize them here as object attributes. However, we can not set the size here as it depends on the data passed during training. Thus, we set them to None for now. The learning rate and number of iterations are hyperparameters that can be tuned for performance.
Training
def fit(
self,
X : np.ndarray,
y : np.ndarray
):
n_samples, n_features = X.shape# Initialize Weights and Bias with Random Values
# Size of Weights matirx is based on the number of data features
# Bias is a scalar value
self.weights = np.random.rand(n_features)
self.bias = 0
for iteration in range(self.n_iters):
# Get predictions from Model
linear_pred = np.dot(X, self.weights) + self.bias
predictions = sigmoid(linear_pred)
loss = predictions - y
# Gradient Descent based on Loss
dw = (1 / n_samples) * np.dot(X.T, loss)
db = (1 / n_samples) * np.sum(loss)
# Update Model Parameters
self.weights = self.weights - self.lr * dw
self.bias = self.bias - self.lr * db
The training function initializes the weights and bias values. We then iterate over the dataset multiple times, optimizing these values towards convergence, such that the loss minimizes.
Based on the above equations, the sigmoid function is implemented as follows:
def sigmoid(x):
return 1 / (1 + np.exp(-x))
We then use this function to generate predictions using:
linear_pred = np.dot(X, self.weights) + self.bias
predictions = sigmoid(linear_pred)
We calculate loss over these values and optimize our weights:
loss = predictions - y# Gradient Descent based on Loss
dw = (1 / n_samples) * np.dot(X.T, loss)
db = (1 / n_samples) * np.sum(loss)
# Update Model Parameters
self.weights = self.weights - self.lr * dw
self.bias = self.bias - self.lr * db
Inference
def predict(
self,
X : np.ndarray,
threshold:float=0.5
):
linear_pred = np.dot(X, self.weights) + self.bias
predictions = sigmoid(linear_pred)# Convert to 0 or 1 Label
y_pred = [0 if y <= threshold else 1 for y in predictions]
return y_pred
Once we have fit our data during training, we can use the learned weights and bias to generate predictions similarly. The output of the model is in the range [0, 1], as per the sigmoid function. We can then use a threshold value such as 0.5. All values above this probability our tagged as positive labels and all values below this threshold our tagged as negative labels.
Complete Code
import numpy as npdef sigmoid(x):
return 1 / (1 + np.exp(-x))
class LogisticRegression():
def __init__(
self,
learning_rate:float=0.001,
n_iters:int=10000
) -> None:
self.n_iters = n_iters
self.lr = learning_rate
self.weights = None
self.bias = None
def fit(
self,
X : np.ndarray,
y : np.ndarray
):
n_samples, n_features = X.shape
# Initialize Weights and Bias with Random Values
# Size of Weights matirx is based on the number of data features
# Bias is a scalar value
self.weights = np.random.rand(n_features)
self.bias = 0
for iteration in range(self.n_iters):
# Get predictions from Model
linear_pred = np.dot(X, self.weights) + self.bias
predictions = sigmoid(linear_pred)
loss = predictions - y
# Gradient Descent based on Loss
dw = (1 / n_samples) * np.dot(X.T, loss)
db = (1 / n_samples) * np.sum(loss)
# Update Model Parameters
self.weights = self.weights - self.lr * dw
self.bias = self.bias - self.lr * db
def predict(
self,
X : np.ndarray,
threshold:float=0.5
):
linear_pred = np.dot(X, self.weights) + self.bias
predictions = sigmoid(linear_pred)
# Convert to 0 or 1 Label
y_pred = [0 if y <= threshold else 1 for y in predictions]
return y_pred