![](https://crypto4nerd.com/wp-content/uploads/2023/07/0TFVptFyCQtAoeUww.jpeg)
Prasanna Sagar Maddu Uncategorized 23rd Oct 2021 4 Minutes
We all know Naive Bayesian Classifier is widely used and available in various frameworks such as Scikit Learn. In this post I will explain what a generic Bayesian Classifier is and how to build one
You will be able to learn how to build a Bayesian Classifier using Python Scikit-Learn package. This will pave way to build your own machine learning models using Scikit-Learn package
Bayesian models fall under generative models. In general there are two types of models: Generative and Discriminative. To know more about these two you can visit Generative and Discriminative Models. You can learn more about Bayesian statistics here.
As you might have guessed Naive Bayes model is just a special case of Bayesian models where the assumption is that each each feature is independent of one another thereby simplifying the models and increasing computational efficiency at the cost of increased bias.
With generic Bayesian model you have control over how the features in the data are related. For example if we use Gaussian mixture models we can choose different types of covariance such as diagonal or full (refer to Gaussian Mixture Model for more details about types of covariances)
Note: This is not unique way but one of the many ways you can build a Bayesian Classifier.
To build this we will combine Gaussian Mixture Models and Bayes rule. Key idea for implementation here is to use the GaussainMixture class in scikit learn to find the likelihood probabilities i.e. P(w|C) and then using the bayes rule to calculate the a posterior probability of data belonging to a class.
Let’s begin by importing the necessary modules:
from sklearn.mixture import GaussianMixture
from sklearn import base
import numpy
Now we will use the Scikit-Learn model protocol to build the custom model by implementing functions like fit(), transform(), predict(), predict_proba() etc.,
class BayesianGaussianClassifier(base.BaseEstimator, base.ClassifierMixin):
def __init__(self, n_components, covariance_type, random_state, verbose=False):
"""
This is the constructor for this class. We will use numpy data structures instead of
pandas data frames just like other algorithms in scikit learn
"""
self.n_components = n_components
self.covariance_type = covariance_type
self.random_state = random_state
self.verbose = verbose
In the above code we just build a python class called BayesianGaussianClassifier which inherits from generic model prototype classes in Scikit-Learn. This classifier takes as inputs the number of components in the gaussian mixture (basically the number of gaussians), covariance type (diagonal, full etc.,), random state value which can be used to
Now we need to implement the class methods following the Scikit-Learn model protocol.
def fit(self, X, y):
"""
First we will divide the data into one set per class and then fit KernelDensity to each set
"""
self.class_labels_ = numpy.unique(y)
if self.verbose:
print("Class labels: {}".format(self.class_labels_))
self.training_set_per_class = [X[y == class_label] for class_label in self.class_labels_]
if self.verbose:
print(self.training_set_per_class[0])
print(self.training_set_per_class[0].shape)
self.gaussian_mixtures_ = [GaussianMixture(n_components=self.n_components,
covariance_type=self.covariance_type, random_state=self.random_state).fit(class_training_set)
for class_training_set in self.training_set_per_class]
self.logpriors_ = [numpy.log(class_training_set.shape[0]/X.shape[0])
for class_training_set in self.training_set_per_class]
return self
In the above code we implement the fit(…) method which will be used while training the model. Now we can implement the predict_proba(…) method which will be used for inferencing/prediction probabilities.
def predict_proba(self, X):
self.logprobs_ = numpy.array([gaussian_mixture.score_samples(X) for gaussian_mixture in self.gaussian_mixtures_]).T
if self.verbose:
print("P(w|C) probabilities shape: {}".format(self.logprobs_.shape))
posterior_probs = numpy.exp(self.logpriors_ + self.logprobs_)
return posterior_probs
We also need to implement the predict(…) function to predict the class labels rather than just the probabilities of each class label
def predict(self, X):
return self.class_labels_[numpy.argmax(self.predict_proba(X), axis=1)]
The above method will use the class probabilities calculated in previous function and assign the class label with highest probability value for each input data point.
I hope this has helped you building custom Bayesian models. Please feel free to provide suggestions/improvements or corrections. I used this Bayesian classifier with sample from MNIST data and got very good results. I hope you can try this model on your own data and please post any interesting results.
As we now know how to build custom models, we can use this learning to build Bayesian models by using different types of priors. In this post we used a Gaussian(Normal) mixture model as the prior. You can choose any prior which suits your problem at hand.
- https://www.sciencedirect.com/topics/computer-science/bayesian-classifier
- https://www.analyticsvidhya.com/blog/2021/07/deep-understanding-of-discriminative-and-generative-models-in-machine-learning/
- https://scikit-learn.org/stable/modules/generated/sklearn.mixture.GaussianMixture.html
- https://www.analyticsvidhya.com/blog/2016/06/bayesian-statistics-beginners-simple-english/