Machine Learning

I was reading a paper by Pedro Domingos this evening which had some tips and advice for people using machine learning. I’ve written down some bullet points for my own reference and I hope someone else finds it useful. I know I’ve made some of the mistakes he gives advice about avoiding.

I wrote a few quick bullet points down from the article “8 Proven Ways for improving the “Accuracy” of a Machine Learning Model” for future reference.

Improving Accuracy

• Add more data
• Fix missing values
  • Continuous: impute with median/mean/mode
  • Categorical: treat as separate class
  • Predict missing classes with k-nearest neighbours
• Outliers
  • Delete
  • Bin
  • Impute
  • Treat as separate to the others
• Feature engineering
  • Transform and normalise: scale between 0-1
  • Eliminate skewness (e.g. log) for algorithms that require normal distribution
  • Create features: Date of transactions might not be useful but day of the week may be
• Feature selection
  • Best features to use: identify via visualisation or through domain knowledge
  • Significance: Use p-values and other metrics to identify the right values. Can also use dimensionally reduction while preserving relationships in the data
• Test multiple machine learning algorithms and tune their parameters
• Ensemble methods: combine multiple week predictors (bagging and boosting)

I wrote a few quick bullet points down from the article “8 Tactics to Combat Imbalanced Classes in Your Machine Learning Dataset” for future reference.

Tactics

• Imbalanced datasets occur when you have a class that occurs much more infrequently than the others.
• If a model ignores the class, it can still achieve a high classification accuracy, but that’s not the result we want
• Make sure you use a confusion matrix to ensure that you’re getting acceptable accuracy for all your classes
• One potential solution to this problem is to collect more data
• Resampling your dataset (can be random or non-random – stratified):
  • Add copies of underrepresented classes (oversampling/sampling with replacement). Useful if you don’t have much data – 10s of thousands or less.
  • Delete instances of classes that occur frequently (undersampling). Handy to use if you have a lot of data – 10s-100s of thousands of instances
• Try different algorithms: decision trees can perform well on imbalanced datasets
• Penalised models: Extra penalties for misclassifying minority class. Examples of these algorithms could include penalized-SVM and penalized-LDA.
• There are areas of research dedicated to imbalanced datasets: can look into anomaly detection and change detection.

I wrote a few quick bullet points down from the article “How To Implement Machine Learning Algorithm Performance Metrics From Scratch With Python“.

Metrics

• Classification accuracy
  • Test how well predictions of a model do overall
  • accuracy = correct predictions / total predictions
• Confusion matrix
  • Use to identify how well your predictions did with different classes
  • Very useful if you have an imbalanced dataset
  • I wrote an extremely hacked together confusion matrix for my tag identification software. I had 4 classes (U, C, R, Q) and the confusion matrix shows you what your model predicted against what the real category was.

• Mean absolute error for regression
  • Positive values – the average of how much your predicted value differ from the real value
• Root mean squared error for regression
  • Square root of the mean of squared differences between the actual and predicted value
  • Squaring the values gives you positive numbers and finding the root lets you compare the values to the original units.