Bayesian Naive Bayes for Classification with the Dirichlet Distribution

I have a classification task and was reading up on various approaches. In the specific case where all inputs are categorical, one can use “Bayesian Naïve Bayes” using the Dirichlet distribution. 

Poking through the freely available text by Barber, I found a rather detailed discussion in chapters 9 and 10, as well as example matlab code for the book, so took it upon myself to port it to R as a learning exercise.

I was not immediately familiar with the Dirichlet distribution, but in this case it appeals to the intuitive counting approach to discrete event probabilities.

In a nutshell we use the training data to learn the posterior distribution, which turns out to be counts of how often a given event occurs, grouped by class, feature and feature state.

Prediction is a case of counting events in the test vector. The more this count differs from the per-class trained counts, the lower the probability the current candidate class is a match.

Anyway, there are three files. The first is a straightforward port of Barber’s code, but this wasn’t very R-like, and in particular only seemed to handle input features with the same number of states.

I developed my own version that expects everything to be represented as factors. It is all a bit rough and ready but appears to work and there is a test/example script up here. As a bigger test I ran it on a sample  car evaluation data set from here, the confusion matrix is as follows:

testY   acc good unacc vgood

  acc    83    3    29     0

  good   16    5     0     0

  unacc  17    0   346     0

  vgood  13    0     0     6

That’s it for now. Comments/feedback appreciated. You can find me on twitter here

Links to files:

Barber Port

R amenable implementation

Example Usage

Everything in one directory (with data) here

Trading in a low vol world

I wanted to take a look at what works in low vol environments, such as we are currently experiencing. I am open to the idea we have entered a period of structurally low volatility due to increased regulatory burden and flow on effects from the decline of institutional FICC trading. Or it may just be a function of QE, and post-tapering we will see a return to higher levels.

The plan

The main idea is to compare mean reversion (MR) vs. follow through (FT). For simplicity I define mean reversion as an up day being followed by a down day, and a down day being followed by an up day. Conversely, follow through sees an up day followed by another up day, and a down day followed by a down day.

I took a look at the major US equity indices, SPX (GSPC), NDX and RUT.

For each series we calculate daily log returns for the current period and shift the forward to get the return for the next period. Then we calculate realized volatility (RV) and split the data set into "low volatility" and "high volatility", by looking at median realized vol for the whole series.

Then, for each series, we use bootstrapped samples to simulate a number of trajectories/equity curves for each strategy (MR/FT) under the two classes of RV. Finally we take an average of the total return of each trajectory to get a ballpark idea of how they went.

Results

The data is from the start of 1999 to the present, so roughly 15 years. Each run generates 1000 trajectories with a sample size of roughly 950.

For the low vol case, the results are unfortunately ambiguous. Follow through in a low vol environment seemed to do well for NDX and RUT, but the opposite was the case for SPX.

The TR column is the sum of the series over the whole period for the volatility class (i.e. a simple long only strategy), giving an idea of a directional bias that may be present in the sampling.

In the high vol environment, mean reversion was a clear winner, and consistent over the different underlyings.

The results seem relatively stable across trajectory size/sample size.

Outro

I'm not really sure what is going on SPX. My intuition was that FT would do well in low vol environments, but that doesn't seem to be the case, at least not for SPX.

I was actually getting consistent votes for FT in the low vol case, then restarted R to run with a clean environment and started getting the above instead. You can't spell argh without R it seems.

Source is up here. As always you can find me on twitter here. Thanks for stopping by.

Data Mining Tools

Sometimes you're not really sure about what you want do, or you have a data set you just want to throw a bunch of stuff at and see what sticks. I've found the following useful:

Rapid Miner This is a fully functional data mining tool. You create a processing workflow via drag and drop components, typically defining an input source, some processing and an output source. It supports everyone's favourite machine learning/AI techniques like SVMs, Neutral Nets, KNN etc. Thomas Ott at Neural Market Trends has some great rapidminer tutorials that can help you get up to speed.

Eureqa This is a tool which takes a data set and will try and find the function that describes it best. A bit more lightweight in terms of functionality compared to rapid miner, but can also be quite handy. It is windows based and I have not used it extensively, however it did seem to do what it said on the box.