Machine learning and AI are hot topics, so there is plenty of material around. We have found 2 really good sources that relate more or less directly to our project: an ongoing course in a university, and a research group tackling the same problem.
Applications of Deep Neural Networks
This is an on-hands course held by Jeff Heaton in Washington University in St. Louis, USA. The course can be taken fully online, and all the material is available in Heaton’s github page. The course is quite a comprehensive one, and I was looking through it to find relevant parts, of which part 11 looked promising. Natural Language Processing – here we come! Conveniently, the course uses same tools as we have chosen – python and TensorFlow.
LogPAI – research group in Hong Kong
“The ultimate goal of LogPAI is to build an open-source AI platform for automated log analysis.”
This group has been on the problem for a while already, so their research might be valuable for this project as well. As we are taking our first steps, their curated collection of publications, sponsors and previous research gives a clue where this project might head. One paper looked liked the ultimate problem of our project has already been solved to some extent: DeepLog: Anomaly Detection and Diagnosis from System Logs through Deep Learning.
Of course this is deep, funded, and dedicated research, and we can only hope we can achieve a working solution of some kind towards these ends.
Detecting log anomalies with machine learning 