• Markify is a service that searches a large set of databases for registered trademarks that are similar, in sound or in writing, to a given query – like a name you have thought up for your next killer startup. As described on the company’s website, determining similarity is not that clear-cut, so (according to this write-up) they have adopted a data-driven strategy where they train their algorithm on “actual case literature of disputed trademark claims to help it discover trademarks that were similar enough to be contested.” They claim it’s the worl’d most accurate comprehensive trademark search.
• alaTest compiles, analyzes and rates product reviews to help customers select the most suitable product for them.
• Intellus is a business process / business intelligence company. Frankly, these terms and web sites like theirs normally make me fall asleep, but they have an ad for a master’s project out where they propose research to “find and implement an effective way of automating analysis in non-normalized data by applying different approaches of machine learning”, where the “platform for distributed big data analysis is already in place.” They promise a project at “the bleeding edge technology of machine learning and distributed big data analysis.”
• Although I haven’t listed AstraZeneca as a “big data” company (yet), they seem to be jumping the “data science” train as they are now advertising for “data angels” (!) and “predictive science data experts.”

On the US stage, I’m curious about a new company called BigML, which is apparently trying to tackle a problem that many have thought about or tried to solve, but which has proven very difficult, that is, to provide a user-friendly and general solution for building predictive models based on a data source. A machine learning solution for regular people, as it were. This blog post talks about some of the motivations behind it. I’ve applied for an invite and will write up a blog post if I get the chance to try it.

Finally, I’d like to recommend this Top 10 data mining links of 2011 list. I’m not usually very into top-10 lists, but this one contained some interesting stuff that I had missed. Of course, there is the MIC/MINE method which was published in Science, a clever generalization of correlation that works for non-linear relationships (to over-simplify a bit).  As this blog post puts it, “the consequential metric goes far beyond traditional measures of correlation, and rather towards what I would think of as a general pattern recognition algorithm that is sensitive to any type of systematic pattern between two variables (see the examples in Fig. 2 of the paper).”

Then there are of course the free data analysis textbooks, the free online ML and AI courses and IBM’s systems that defeated human Jeopardy champions, all of which I have covered here (I think.) Finally, there are links to two really cool papers. The first of them, Graphical Inference for Infoviz (where one of the authors is R luminary Hadley Wickham), introduces a very interesting method of “visual hypothesis testing” based on generating “decoy plots” that are based on the null hypothesis distribution, and letting a test person pick out the actual observed data among the decoys. The procedure has been implemented in an R package called nullabor. I really liked their analogy between hypothesis testing and a trial (the term “the statistical justice system”!):

Hypothesis testing is perhaps best understood with an analogy to the criminal justice system. The accused (data set) will be judged guilty or innocent based on the results of a trial (statistical test). Each trial has a defense (advocating for the null hypothesis) and a prosecution (advocating for the alternative hypothesis). On the basis of how evidence (the test statistic) compares to a standard (the p-value), the judge makes a decision to convict (reject the null) or acquit (fail to reject the null hypothesis). Unlike the criminal justice system, in the statistical justice system (SJS) evidence is based on the similarity between the accused and known innocents, using a specific metric defined by the test statistic. The population of innocents, called the null distribution, is generated by the combination of null hypothesis and test statistic. To determine the guilt of the accused we compute the proportion of innocents who look more guilty than the accused. This is the p-value, the probability that the accused would look this guilty if they actually were innocent.

The other very cool article is from Gary King’s lab and deals with the question of comparing different clusterings of data, and specifically determining a useful or insightful clustering for the user. They did this by implementing all (!) known clustering methods plus some new ones in a common interface in an R package. They then cluster text documents using all clustering methods and project the clusterings into a space that can be visualized and interactively explored to get a feeling for what the different methods are doing.

As you probably know, a company called Kaggle organizes predictive analytics competitions where data scientists can earn money from their skills and companies can tap into some of the unknown talent of the world. There are other similar companies, like CrowdAnalytix, and more specialized/closed variants on the same idea such as Innocentive, but I think Kaggle has deservedly gotten the most buzz because they have succeeded the best in presenting their business case and vision. For example, here is a presentation that Jeremy Howard from Kaggle gave at the Strata NY 2011 conference, where he outlines how Kaggle wants to become a “meritocractic” platform that allows people who are good at analytics to finally get properly compensated for their skills.

I have known about Kaggle for quite some time and been a fan of their business idea, but with one full-time job, occasional work on the side and two young kids, I figured I’d never have the time to participate fruitfully in the competitions myself. As it happened, I got the chance to chat with Kaggle’s CEO Anthony Goldbloom at a conference (Strata 2011 in Santa Clara) and he persuaded me to give the competitions a try. So I finally jumped in, and found that despite not really having the time to spare, I still enjoyed it and learned a lot. So far I’ve only participated for real in one competition, the Dunnhumby Shopper Challenge, where the task was to predict (based on historical shopping records on thousands of customers) at what date each customer would next visit the store, and how much money (within $10) he or she would spend. This task turned out to be surprisingly non-standard and was definitely not something that you could just throw your favourite algorithm at right out of the box.

Already from this one competition I learned / noticed several things:

- You can sometimes get pretty far just by using common sense and a very simple conceptual model. In fact the winning entry by Alexander d’Yakonov (explained here) used essentially the same basic idea as my model, although he had added a couple of tricks that I hadn’t thought of.

- It’s extremely helpful to learn from your competitors. Kaggle often asks high-scoring contestants to explain how they did it, which is a huge service to the community. For Dunnhumby, there was the winning entry that I linked above, plus this from Neil Schneider, who placed second, and this from William Cukierski, who placed fourth. Similar explanations for other competitions can be found under the “How I did it” tag on Kaggle’s blog.

- A competition can really motivate you to learn new stuff that you wouldn’t have dreamed of touching otherwise. The Dunnhumby competition motivated me to learn survival analysis, although I didn’t end up using that particular statistical framework. (I tried, but couldn’t get it to work well on the problem.) I also started to brush up on time series analysis.

During the past few days, I’ve discovered a couple of really, really good resources about how to get started with prediction contests:

Using R for data mining competitions by Jonathan Lee shows case studies from Kaggle competitions. It’s heavy on the R material (which I like) but go look at it even if you don’t use R, as there is a lot more to the presentation than code.

Getting in shape for the sport of data science by Jeremy Howard of Kaggle (again). This is a really great nuts-and-bolts talk about how to compete in prediction contests, with useful tips on how to “munge” your data into shape using different tools – even Excel! – and set up your models. There are many nice tricks here. Finally he explains the ideas behind the random forest algorithm – “a lot of crappy predictors that are all crap in a slightly different way.” For a while I was tempted to apply the same idea to Kaggle competitors – all are crap in a different way but the occasional competitor stumbles on something good and the rest cancel each other’s errors out but this theory doesn’t hold as many of the top competitors (such as Jeremy himself before he joined Kaggle) are  consistently good.

Soo .. let’s see if I will have time to dive into the next contest in earnest …

P. S. Other interesting resources for learning about prediction / machine learning etc. apart from the stellar presentations mentioned above, and the Kaggle “How I did it” testimonials, include:

Stanford’s free online AI class with Peter Norvig and Sebastian Thrun (of Google’s self-driving car fame)- I’ve been trying to follow this, but predictably enough haven’t had time to keep up

Stanford’s online machine learning class with Andrew Ng – I’ve only watched one lecture but it seems really good

Free lectures from Tom Mitchell’s machine learning class

Gavagai’s blog contains some case studies where online social media have been monitored using the system in order to understand customer loyalty of US mobile network operators, reactions to Rick Perry’s botched interview, the apparently fading interest in Julian Assange, etc.

Numbers from This American Life. This show from 1998 (!) which predates the Quantified Self movement by about 10 years, is as fresh and interesting as anything from the QS movement today. Available for one dollar or so from iTunes.

NPR’s On the Media had a good data-themed show on May 13, 2011, with material on data journalism, personal data, the “decline effect” and (particularly interestingly) two cautionary tales about relying too much on a data driven approach. Freely available on iTunes (etc.)

I’ve also enjoyed the numerous videos from the Sage Commons Conference in April. The main themes are summarized in the web site slogan “genomics, health innovation and open science.” Sage, which I’ve blogged about several times before, is about open data, open science, disease network modelling, Bayesian networks, amongst other things.

Ion Flux (a company that I haven’t heard of until now) works in “Clinical Genomics + High-Performance Computing” and looks for people with the following characteristics:

Do you want to work with an innovative, dynamic team that’s defining the state of the art in clinical diagnostics and healthcare?

Do you see big data problems as an opportunity to produce big results? Are you fascinated by the science, statistics, algorithms, and infrastructure needed to realize the possibilities?

Halcyon Molecular, a DNA sequencing company that “aims to transform biology into an information science”, is looking for people who know statistical learning, in particular Hidden Markov Models and Bayesian statistics. Halcyon’s web page says:

Current gene sequencing methods are too slow, too expensive, and too incomplete to make ‘personalized medicine’ more than a buzzword.  Our nanorobotics and single-atom detection approach will enable megabase single-molecule reads in milliseconds.

In order to make this reality, they need people who are good at image processing and the above-mentioned statistical learning approaches.

The chronic widespread misuse of statistics is usually inadvertent, not intentional. We find cautionary examples in a series of recent papers by Christakis and Fowler that advance statistical arguments for the transmission via social networks of various personal characteristics, including obesity, smoking cessation, happiness, and loneliness. Those papers also assert that such influence extends to three degrees of separation in social networks. We shall show that these conclusions do not follow from Christakis and Fowler’s statistical analyses. In fact, their studies even provide some evidence against the existence of such transmission. The errors that we expose arose, in part, because the assumptions behind the statistical procedures used were insufficiently examined, not only by the authors, but also by the reviewers. Our examples are instructive because the practitioners are highly reputed, their results have received enormous popular attention, and the journals that published their studies are among the most respected in the world. An educational bonus emerges from the difficulty we report in getting our critique published. We discuss the relevance of this episode to understanding statistical literacy and the role of scientific review, as well as to reforming statistics education.

Cosma Shalizi has co-authored another paper (available here) which makes a similar point in a much more, let’s say, polite way. My impression is that Shalizi is both sharp and trustworthy (I’ve learned a lot about statistics from his blog) so I’m inclined to think he is on to something.

Google’s Prediction API is basically a nice way to run your classification and/or prediction tasks through Google’s black-box set of machine learning tools. The way it works is that you upload your training data to Google Storage, which is something like Google’s version of Amazon’s S3: a cloud-based storage system where you store your data in “buckets”. (Google Storage, like S3, uses the term bucket and, also like S3, requires that bucket names only use lower-case letters.) You can activate both Google Storage and the Prediction API from the Google APIs Console. This is also where you will find (click “API access” on the left hand menu) the access key that you will need to run prediction tasks. You’ll have to give credit card details to pay for potential future usage.

The training examples that you put in Storage need to be formatted according to the specification in the Developer’s Guide. Once they have been uploaded, you can train a model on the uploaded data, make predictions about new examples, update existing models and more using one of the client libraries or even simpler, just by copying some of the bash scripts shown on the same page (hidden behind ‘+’ signs which can be expanded.) For these bash scripts to work as written on that page, you need to paste your API key into a file called ‘googlekey’ located in the directory from where you are running the script.

I used this walkthrough example about cancer classification from gene expression data to get up to speed on how Google Prediction API works. Now I’m thinking about what data to throw at it next. Perhaps it would be fun to input some Kaggle contest data sets into it as a kind of “Google baseline” predictor?

However, there are other companies that have taken up the idea. A recent Wired article (Hoops 2.0: Inside the NBA’s Data-Driven Revolution) describes a system called SportsVU, which uses video cameras to track the players and the ball to a remarkable level of detail. The technology grew out of missile-tracking applications and optical recognition algorithms to that. It was first applied to soccer, but it was later decided that basketball would be more lucrative. With the SportsVU system – and, not to forget, the analytics to crunch the raw image data – it’s possible to track some pretty complicated metrics, as illustrated in this example for Kevin Martin.

A different way to quantify basketball player performance is used by Infomotion Sports, as descibed in this Yahoo! Sports article. Infomotion’s 94Fifty technology, instead of using video cameras, actually measures the movements of the ball directly using sensors embedded into the inside of the ball. Through “feeling” the ball’s motion, one can quantify a lot of things about a player, such as dribbling speed, shot angles and spin. A quote from the article:

“A coach can tell me that [a player] needs to work on his left hand,” Kamil said. “[But] we can tell you that his right hand is 14 percent more dominant than his left.”

Of course, a lot of people would be skeptical as to how much you really gain from this kind of data-driven approach compared to the good old gut feeling of a coach. I count myself among the skeptics, but I think the technology is intriguing.

Pete Skomoroch’s tutorial on “geo analytics”. A really good slide deck that manages to introduce [Elastic] MapReduce, Pig, Amazon Mechanical Turk, Python Natural Language Toolkit, GitHub and probably other stuff I don’t remember as seamless components in a project in a way that makes complete sense. In fact I liked this much better than the same presenter’s talk at Strata 2011, maybe because this one goes more into details.

Similarly, I like Recorded Future developer Anders Karlsson’s presentation about how Recorded Future uses Amazon EC2 because it gives a lot of detail about nitty-gritty, day-to-day data analysis/storage challenges. Karlsson also provides some non-obvious thoughts on what works well, and less well, when trying to manage very large data sets on EC2.