We study the covariance between two variables, and its geometrical intuition. Next, we learn about feature standardization, Pearson correlation, and its relationship to linear regression for a single predictor. We also study the phenomenon of regression towards the mean. Correlation does not imply causation: though it is a common fallacy to fall into. We will delve more deeply into this.

We will study linear regression, the concept of least-squares, and gradient-descent to minimize the sum-squared errors. Then we will study the ordinary least squares linear regression, polynomial regression and the Runge phenomenon, nonlinear least-squares, and box-cox transformations. We will also learn about residuals analysis and other model diagnostic techniques, and get introduced to alternate loss functions.

Regularization reduces overfitting and high variance. We look at an additional penalty term to the regression loss function as a regularization hyperparameter. Additionally, we see a geometric interpretation of this term along with Minkowski distance. Lasso (L1), Ridge (L2), and Elastic-Net regularizations are covered in the data-science lab exercises.

We will study classification and the learning of decision-boundaries in the predictor space. In particular, we will study the Logistic Regression Classifier and the Linear and Quadratic Discriminant Analyses classifiers. We will study the goodness of fit diagnostic measures such as the confusion matrix, precision, recall, accuracy, the ROC curve, and the area under the ROC curve.

We study three different approaches to clustering of data in the feature space: K-Means clustering and its close variants; selecting the optimal number of clusters through the scree plots Agglomerative (Hierarchical) clustering, dendrogram and various linkage functions Density-based clustering techniques such as dbscan, optics and denclue

We will study a few techniques for dimensionality reduction. Primarily we will focus on Principal Component Analysis and understand the geometrical interpretation of it. Along with this we will relate it to the Covariance Matrix, and discuss the class of datasets that PCA works best with. We will also cover such simpler approaches as backward and forward selection and Lasso.

The predictive efficacy of the machine-learning algorithms is greatly amplified by feature engineering from the raw input feature-space. A meticulous process of exploratory data analysis (EDA) in search of clues towards meaningful feature extraction often can transform a simple algorithm which was performing very poorly on the original feature-space into an extraordinarily effective one in the new feature space where these extracted features are present.

Approximate & Kernel k-NN is a lazy-learning, non-parametric method that proves remarkably effective in certain situations. We will learn about the interesting notions of distances or similarities that underly the search for nearest neighbors. We will learn about the Curse of Dimensionality, its origin, and the methods to deal with it. We will study how the choice of the "k" provides the bias-variance tradeoffs. Finally, we will learn about a rich collection of distance kernels that mitigate the need for hyperparameter "k" tuning.

Decision Tree provides a versatile means to adapt to non-linearities in the feature space. It is employed for classification as well as regression. The strength of decision trees resides in their very intuitive representation as a tree structure in the way it iteratively partitions the feature space. Thus it is valued for its interpretability, which at the same time being remarkably powerful.

Among the most powerful concepts to have emerged in machine learning is that of ensemble learning. Instead of using only one learner, and making it as powerful as possible so that it comes up with a good hypothesis, the approach here is different. Instead, a crowd of weak learners is taken, each forming its own hypothesis. These hypotheses are methodically synthesized to generate predictions in a manner that there is much greater overall predictive power.

RandomForest is a powerful approach to both classification and regression that employs an ensemble of decision trees and bagging to get a consensus vote or score. Extremely Randomized Trees are a variation of the same approach but exhibit fewer variance errors.

Gradient Boosting is a powerful ensemble method that has proven remarkably effective in recent years. As a result, there is a lot of research and activity in creating continually better performing and more predictive implementations.

Support Vector Machines provide a systematic way to linear the decision boundary by transforming the original feature space into another where the various classes are separated by a linear maximal margin classifier.

The No Free Lunch theorems -- despite their frivolous sounding name -- are a fundamental guideline that no one algorithm is more "powerful" than any other, on the average. Different algorithms make different underlying assumptions of the ground truth and therefore are well suited to different datasets. On average, they all perform equivalently.

A Capstone project where each participant presents an extensive data-science journey over a non-trivial dataset of choice.