Harini Jayaraman

Konigsberg bridge problem and the evolution of Mathematics

harini-jayaraman — Fri, 29 Apr 2022 20:44:00 +0000

Well, In this blog I will discuss one of the problems that led to the development of new branches in mathematics like ‘Graph theory’ and ‘Topology’. Let’s dive into that interesting problem. What’s the Konigsberg problem? Konigsberg, a city you will find a hard time finding on the world map. However, it is one of the most famous cities in the world of Mathematics due to a particular quirk in its geography. The city of Konigsberg was set on both sides of the Pregel River, and included two large islands, which were connected to each other, or to the two mainland portions of the city, by seven bridges. In the 17th century, people were bothered with the question, whether they could walk around all 7 bridges crossing them only once. Let’s try it out! You could very well see other routes in this video and come up with even more routes. No matter how many ever routes you try you have to give up at some point because Euler answered the question in the negative and proved that there could be no route in Konigsberg that used all seven bridges exactly once and laid the foundation for Graph theory. How was it proved? Well, the answer he came up with, did not basically exist then, what he called the ‘Geometry of position’ , now known as the ‘Graph theory’ and which led to the fundamentals of ‘Topology’. Euler realized during his various trials that the geography or route taken to reach the island or riverbanks does not matter. This insight led to the simplifying of map further. What matters is the 4 landmasses and the relationship between them. We could represent each of the four landmasses here named as A,B,C and D respectively as you could see in the picture below, as a single point, which we now call as Nodes in Graph theory. We now have 4 nodes. We draw lines or edges between them to represent the bridges. Formally, we define a graph G = (N, E) as a pair where, The graph can be used to represent binary relations: (i,j) ϵ E if and only if the entity, i ϵ N and the entity j ϵ N interact with each other. The connectivity of nodes is vital for the structure of the graph. From the simplified graph, as you see in the above picture (Forgive my poor editing skills) with just nodes(Pink dots) and edges(Green lines) we can find the degree of the nodes which forms the basis of this solution. The degree of the node is nothing but the number of edges incident to it. So, in our case, it is the number of bridges that connects to each of the landmass. According to our challenge, person entering the landmass via one bridge should exit it through another bridge. Technically, edges leading to and from one node should occur in distinct pairs, meaning that the number of edges touching each node must be even. The only possible exceptions would be the locations of the beginning and the end of the walk. Looking at our graph, we can easily see that all the nodes have odd degree (in yellow in the below figure). Hence, no matter whichever path we choose we will end up crossing a bridge twice. Euler used this as a proof to formulate a general theory that apples to all graphs with two or more nodes. He defined something called Eulerian path that visits each edge only once and is possible only in one of the two scenarios. In the first scenario, all other nodes would be even and the starting point and ending point are different and should be nodes with odd degree. In the second scenario, starting and ending point would be the same. Why does it matter? In the history of mathematics, this is considered to be the first theorem of Graph theory. It basically forms the first proof for Network theory. Network theory in today’s world is a broad area of research. A network is a more general type of data structure that describes relations or interactions between entities. For instance, The Internet network is the physical infrastructure made of internet service providers, servers, routers (entities) and all the physical connections between them. We can never forget the Social networks (Friendships,relationships,work ties, social interactions) in today’s scenario. If time permits, I will discuss more on Social networks in my next blog soon. Current state of Konigsberg After the world war II two of its bridges has been destroyed and Eulerian path has been created. However you can’t find Konigsberg now. It is renamed as Kalingrad and is a part of Russia. I love this particular video explaining this problem in a simple way. You could well play this game based on this theory. Check if you can draw these line by going over each only once. https://www.transum.org/Maths/Activity/without/ Further Reading This lecture notes could quite give you an idea about Eulerian path and this blog could be a great starting point in understanding the world of Graph theory and Networks.

Source

Anomaly detection in Functional Data

harini-jayaraman — Wed, 06 Apr 2022 22:24:00 +0000

As discussed in my previous blog post, Functional data appears in various disciplines and each observation can be considered as real function of time observed at particular time points. Anomaly detection can be more challenging in terms of functional data because outliers are not always easily identified visually. Various methods are being developed to identify outliers in functional data. I’ve defined variety of outliers in functional data in my previous post, of which shape outliers is the most difficult to identify. I will discuss one of the method in identifying outliers in functional data we explored as a team during STOR608 sprints, as it was a great and interesting learning experience for me. Functional depth The anomaly detection techniques in the methods I will discuss depends on something called ‘Functional depth’. It is nothing but a specific ordering of the curves. There are various definitions/ways in literature to order these functional observations , where is closed real interval discretized into points. Modified Band Depth is one of the way. Modified Band Depth (MBD): MBD is defined based on the number of times a given curve is completely contained within the range defined by two other curves from sample. Hence, MBD indicates how central the curve is. Higher the MBD value, more central the curve is. The sample curves are ordered in decreasing value of their MBD implying first curve to be the median curve and last curve as the outlier. This article can give more insight on this topic if you would like to know more. MBD is highly used due to its computational efficiency. Anomaly Detection Methods Functional boxplot Functional box plot can be a best option to identify and classify shape, magnitude and amplitude outliers. However, functional boxplot easily identifies magnitude outliers but sometimes fail to detect shape outliers. By performing some sequential curve transformations we could convert the shape outliers to magnitude outliers and identify them. This idea has been proposed in this paper and could be an interesting read. To construct a functional boxplot, we need to order the curves according to their MBD value. We then find a central region which is analogous to the the inter-quartile range (IQR) for the classical boxplot and is a useful indication of the spread of the central 50% of the curves. The box in a classical boxplot works similar as such with he border of the 50% central region also called as the envelope. A functional boxplot identifies the anomalies by extending the 1.5 times the empiricial outlier criterion used for the classical boxplot. If a curve sits outside of the enclosure formed by inflating the 50 percent central zone by 1.5 times the range of the central region, it is considered an anomaly. Mathematically, a curve is an anomaly if, or Sequential Transformations As mentioned earlier, we transform shape and amplitude outliers into magnitude outliers via some curve transformations. I will discuss just one of the algorithm that has proven to be effective while there are other transformations that exist in reality details of which could be found in this paper. Algorithm Identify anomalies using the functional boxplot constructed with MBD. Anomalies detected at this stage are defined as -anomalies (magnitude anomalies). Apply a vertical alignment transformation to the data Repeat step 1 on the transformed data; anomalies detected at this stage are defined as -anomalies (amplitude anomalies). Apply a normalisation transformation to the data: Repeat step 1 on the transformed data; anomalies detected at this stage are defined as -anomalies (shape anomalies). These figures are from the data we were given during the sprints, as a part of our assignment. The data consists of observations for one variable, recorded at one minute intervals over 50 days. We segment it into periods of length = 1440 for each of the 50 days. You could observe the central region in pink, blue lines being the envelope. After transformation, we could observe four outliers in red dashed lines detected using this method. Further Reading I would like to add few papers that could be of interest and we referenced while working. Dai, W., Mrkvicka, T., Sun, Y., and Genton, M. G. (2020). Functional outlier detection and taxonomy by sequential transformations. Computational Statistics & Data Analysis. Arribas-Gil, A. and Romo, J. (2014). Shape outlier detection and visualization for functional data: the outliergram. Bio-statistics. Sun, Y. and Genton, M. G. (2011). Functional boxplots. Journal of Computational and Graphical Statistics Lopez-Pintado, S. and Romo, J. (2009). On the concept of depth for functional data. Journal of the American statistical Association Ojo, O. T., Fernández Anta, A., Lillo, R. E., Sguera, C. (2021). Detecting and classifying outliers in big functional data. Advances in Data Analysis and Classification I would also like to acknowledge that the works presented here were worked in group as a part of sprints. I hope you enjoyed this blog. For further queries, you could always contact me through mail or message me here.

Source

Anomaly detection

harini-jayaraman — Tue, 29 Mar 2022 10:45:00 +0000

As we had an overview of one of the unsupervised learning method, K-means Clustering, in my previous blog post, this post will introduce you to an another unsupervised learning method called Anomaly Detection. We should also note that Anomaly detection can be done using Supervised and Semi-supervised techniques as well. What is an Anomaly? Anomalies (Outliers) are patterns in data that do not conform to a well defined notion of normal behaviour. An unexpected change within these data patterns, or an event that does not conform to the expected data pattern, is considered an anomaly. In other words, an anomaly is a deviation from the usual happenings. Red Occurences in Figure 1, describes the various anomalies that can occur. In Figure 2, We could observe most of the data occurences are in the region and . The points far off from these normal region, such as and are anomalies. Anomalies can occur for various reasons. Customer acquisition costs, Malicious activity, Web page views, Revenue per click, Credit card frauds and more. These are valuable metrics for any data analyst and is considered advantage in decision making process. Therefore, Anomaly detection is an important process, which if efficient and timely can enable intervention and action to minimize or avoid effects of underlying cause. Even from a statistical context, anomaly detection helps the methods to be more robust and efficient. Types of Anomalies Anomalies are majorly classified into these three main categories which I will discuss below. However, there are wide variety of anomalies. If you are interested in learning various anomaly types and how it is distinguished, this paper could be helpful. Point Anomaly: A datapoint is considered an anomaly if its far outside or extremely away from other data points. It is also called as global outlier. For example , in figure 2, is a point anomaly. Contextual Anomaly: Contextual outliers are data points whose value significantly deviates from other data within some defined context. Also known as conditional outliers, its values are not outside global range but abnormal in comparison to the trend/period or seasonal pattern. For example, on Christmas day. Collective Anomaly: Collective anomalies are defined as sequences of observations that are not anomalous when considered individually, but together form an anomalous pattern. In layman words, having all cars on the freeway move to the left lane simultaneously would be a collective outlier because, even though moving to the left lane is not uncommon, it is unusual for all cars to move at the same time. With different types of data, we come across various types of outliers. If we consider database systems, we will have outliers like insertion, updation or deletion anomalies. In time series data, we have additive outliers, innovational outliers, level shift outliers, seasonal outliers and more. This documentation from IBM gives a general idea of understanding outliers in time-series data if you are interested. Data that appears in a variety of disciplines including biology, medicine, meteorology and engineering, like Functional data (type of data consists of curves varying over a continuum, such as time, frequency, or wavelength), we come across outliers like magnitude, amplitude and shape outliers. Magnitude Outlier: If any or one of the curve is outlying in some parts or across whole design domain. Amplitude Outlier: The curves with unusual oscillation levels Shape/Pattern Outliers: The curves with unusual shape or the ones that significantly differ from pattern exhibited by other curves even after centralizing and normalizing. Anomaly detection is the process of identifying these items or events in data sets that differ from the norm. Anomaly detection is often performed on unlabeled data, which is known as unsupervised anomaly detection. Anomaly detection is based on two assumptions: Anomalies occur very rarely in the data Their features differ significantly from the normal cases. A business system that is considered normal today may not be so in the future. Most business systems change over time as the result of various factors. Best example is the current situation of Covid19, the whole world is experiencing. The unprecedented scenarios everywhere, which often is termed as the “new normal”, it was abnormal or anomalous outcomes earlier. These anomalous outcomes can result in positive or negative impact for any organization and are important to keep track of, for formulating a long term business strategy. Hence, anomaly detection in every domain is important and finding more effective ways to detect it will be most important skill in the future. In my next blog, we will discuss more statistics, a method to detect outliers in functional data. I hope this post gave you an insight into what anomaly is and how anomaly detection matters. Till then, I found this small game interesting, eyeballing and finding outliers. Try it for yourself if you could find. Further Reading: This blog could help you in basic overview of Anomalies. I have shared the links of few papers that can be of interest. https://link.springer.com/article/10.1007/s41060-021-00265-1 https://www.sciencedirect.com/science/article/abs/pii/S0167947320300517?via%3Dihub

Source

K-Means Clustering

harini-jayaraman — Fri, 11 Mar 2022 08:30:00 +0000

In my previous blog, I introduced Supervised and Unsupervised learning methods. We will learn about one of the unsupervised learning methods called Clustering in this blog. Clustering involves dividing a population or set of data points into groups so that the data points in the same group are more similar than those in other groups. The goal is to group similar traits into clusters and assign them to groups. There are several approaches to the task of clustering: Hierarchical clustering (agglomerative, divisive) – Define a set of nested clusters organized as a tree. Model-based clustering (mixture models) – A statistical modelling framework based on probability distributions. Partitional clustering (K-means, K-medoids): Division of the set of data points into non-overlapping subsets (the clusters) such that each data point is in exactly one subset. In this blog post, we will focus on K-means clustering. What is K-Means Clustering? The K-means clustering algorithm depends on two parameters, the number of clusters and Cluster centroids. For a particular choice of the number of clusters K, the objective is to minimize the squared distance of the points within each cluster from its center (Referred to as a centroid). The algorithm is initialized by randomly picking the centers of each cluster and then assigning each data point to the closest centroid. After all the data points have been assigned to clusters, the centroids are updated to be the mean of all the data points for each cluster. Having updated the centroid values, we can assign each data point to its closest updated centroid, and then recompute the centroid values. The process is repeated until the centroid positions stop changing between K clusters. For a more detailed understanding, please check this blog. How to determine what value to use for K? The number of clusters are unknown in most cases. In such cases, measures of internal validation like, Calinski – Harabasz index and Silhouette can be useful to select the number of clusters. Calinski – Harabasz index The K-means algorithm returns the clustering minimizing within the sum of squares (WSS). The WSS measures the variability “within”, that is the variability between the data points assigned to cluster K and the corresponding centroid. The between clusters sum of squares (BSS), measures the variability “between”, that is the variability between the clusters accounting for the center of the data. Because the total variation in the data, the total sum of squares (TSS) is fixed, minimization of WSS corresponds to the maximization of BSS. The Calinski-Harabasz index is given by: with CH = 0 when K = 1 The Calinski-Harabasz index compares the relative magnitudes of BSS and WSS, accounting for K and N.A large value of this index for a given value of K is an indication of a clustering solution with low within variability and large between cluster variability. Silhouette The silhouette is a measure of how close each point in one cluster is to points in the neighboring clusters. Silhouette analysis provides a way to visually assess the number of clusters. The silhouette takes values in the range [−1, 1]. Silhouette coefficients near 1 indicate that the observation is far away from the neighboring clusters. A value of 0 indicates that the observation is on or very close to the decision boundary between two neighboring clusters. Negative values indicate that the observation might have been assigned to the wrong cluster. For each observation in the data compute: , the average dissimilarity (distance) from observation xi to the other members of the same cluster k. The average dissimilarity from observation to the members of the other clusters. Let the minimum of such dissimilarity computed. Hence b(i) can be seen as the average dissimilarity between and the nearest cluster to which it does not belong. The silhouette for observation is: To understand the method better, I have performed a cluster analysis of the audio features data from Spotify using k-means. The dataset contains data about audio features for a collection of songs extracted from the music streaming platform Spotify with 231 observations and 13 variables. The songs are classified into three genres: acoustic, pop, rock. For each song, measurements about audio features and popularity are recorded; more information about the audio features is available here. I have used the Cluster package in R. Storing within the sum of squares and the between the sum of squares, with the output we can calculate and plot the Calinski-Harabasz index. We perform K-means on Spotify data for different values of K. Here we take the maximum value to be K = 10 The “appropriate” K corresponds to the largest value of CH. This shows that an optimal number of clusters is K = 2. The silhouette plot is used to assess cluster coherence. Observations with high silhouette values are much closer to their own cluster members than the members of other clusters. Observations with low (or even negative) silhouettes are close to other cluster members and thus may have ambiguous cluster membership. Using the silhouette function in the cluster package, we find the silhouette width by inputting two arguments, clustering data and distance matrix computed using Euclidean distance. The average Silhouette for K=2 is larger than the average Silhouette for K =3. Hence, Clustering K = 2 is better. By comparing the clustering obtained to the classification into genres, we could observe that the clustering is almost similar to the genre of the given data. We could assess the accuracy by measures of external validations like the RAND index and Adjusted RAND index. However, I have not covered that in this blog. You can have a look at that here. The Rand index has a value between 0 and 1, with 0 indicating that the two data clusterings do not agree on any pair of points and 1 indicating that the data clustering is exactly the same. Here we have a rand index of 0.72 for K = 2 and 0.75 for K = 3. Both these values are almost near 1 hence suggesting K-means clustering using audio features of data is similar to the real clustering of data according to its genres. The Adjusted Rand Index is used to measure the similarity of data points presented in the clusters i.e., how similar the instances that are present in the cluster. So, this measure should be high as possible else we can assume that the data points are randomly assigned in the clusters. Here ARI = 0.46 for K=2 and 0.451 for K=3, which is a relatively good measure. Summary To summarise, the K-means clustering algorithm: aims to group similar data points into clusters requires a choice for the number of clusters iterates between two steps, allocation and updating. Further Reading I found this blog useful for learning about the implementation of the K-means algorithm in R. To read about how to implement the K-means algorithm and other clustering methods in python you can click here. Also, to understand K-means clustering in a simpler way, this video can be helpful.

Source

Supervised Vs. Unsupervised Learning

harini-jayaraman — Sat, 05 Mar 2022 18:54:00 +0000

As the world is getting ‘smarter’ every day, most of the world is moving towards Machine learning and artificial intelligence to keep up with the expectations. Statistical machine learning encompasses automatic learning and data analysis procedures, which learn a task from a series of examples with the goal of identifying patterns and performing predictions under uncertainty. We will explore and understand the differences between the two main machine learning methods in this blog What is Supervised Learning? Supervised learning, as the name indicates, has the presence of a supervisor as a teacher. Basically, Supervised learning is when we teach or train the machine to use the data that is well defined and labelled. Just as a child, how we aren’t able to identify the fruits or vegetables right until we get to see, feel, and learn from the inputs, the same logic is applied here to the machines. In supervised learning, models need to find the mapping function to map the input variable (X) with the output variable (Y). The goal of supervised learning is to train the model so that it can predict the output when it is given new data. For example, let’s assume we have images of different types of fruits. The task of our supervised learning model is to identify the fruits and classify them accordingly. So to identify the image in supervised learning, we will give the input data as well as output for that, which means we will train the model by the shape, size, color, and taste of each fruit. Once the training is completed, we will test the model by giving the new set of fruit. The model will identify the fruit and predict the output using a suitable algorithm. Supervised learning can be separated into two types of problems, classification and regression. Classification problems use an algorithm to accurately assign test data into specific categories, such as separating apples from oranges, or disease or no disease etc. Regression is another type of supervised learning method that uses an algorithm to understand the relationship between dependent and independent variables. For example, study relationship between two variables like height and weight. By measuring and labeling inputs and outputs, the model can learn over time. What is Unsupervised Learning? In unsupervised learning, information that is neither classified nor labeled is used to train a machine, allowing the algorithm to act based on that information without guidance. The objective of the algorithm is to group unsorted data based on similarity, pattern, and difference, without any prior training. The machine does not receive any training, as opposed to supervised learning. Therefore, the machine is limited to finding the hidden structure in unlabeled data on its own. For instance, if we consider the above example as fruits, it will identify the patterns like long, round or curvy, and similarities and categorize them. If we have apples and bananas, we might have two different clusters of bananas and apples after running the algorithm. Unsupervised learning models are used for three main tasks: clustering, association and dimensionality reduction: Clustering is a data mining technique for grouping unlabeled data based on their similarities or differences. For instance, Can we try to find groups of patients with similar symptoms and pain characteristics? Association is another type of unsupervised learning method that uses different rules to find relationships between variables in a given dataset. For example, a person buying a product X also tend to buy product Y. Dimensionality reduction is a learning technique used when the number of features (or dimensions) in a given dataset is too high. Summary Supervised Learning Unsupervised Learning Goals Predict outcome of New data Insight into a large volume of data Complexity Simple Computationally complex Accuracy Better accuracy Less accurate We can choose our methods depending on our input data (labelled or unlabelled) and by defining our goals. Among other things, supervised learning models are ideal for spam detection, sentiment analysis, weather forecasting, and price prediction. Unsupervised learning is better suited for anomaly detection, recommendation engines, customer persona, and medical imaging. Further reading: I found these blogs and youtube videos useful for basic understanding and exploring various methods in these categories: https://www.technologynetworks.com/informatics/articles/supervised-vs-unsupervised-learning-352077 https://towardsdatascience.com/supervised-vs-unsupervised-learning-14f68e32ea8d https://link.springer.com/book/10.1007/978-0-387-84858-7 I hope you found the blog insightful. Will discuss one of the unsupervised method, clustering in my next blog. References : Elements of statistical learning – Trevor Hastie, Robert Tibshirani , Jerome Friedman https://www.ibm.com/cloud/blog/supervised-vs-unsupervised-learning

Source

My First term in STOR-i

harini-jayaraman — Sun, 13 Feb 2022 22:33:00 +0000

Despite all the anticipation, dreaming and planning after getting selected for your dream course, at times the reality hits you hard. My first term experience as a STOR-i MRes student was quite daunting. After all, when the world is struck in pandemic, no amount of thinking or planning can truly prepare you for what it is like being surrounded by uncertainities. I would like to share my experience having completed my first term recently after all the hardships. I am an international student and I am supposed to have a ATAS clearance and VISA to begin my studies. I had my interview in the last week of July and the course was supposed to begin in the first week of October,2021. I had applied for my ATAS clearance as soon as i decided to pursue this course, by the first week of August. ATAS clearance usually happens within 15-20 working days but due to pandemic and back logs, it took 57 working days in my case. I haven’t even applied for VISA then, as it requires ATAS Clearance. My course had already by that time but thankfully ��Ƶ, STOR-i director Jonathan Tawn and Idris Eckley helped me to begin the course without deferring as I didn’t want to waste a year. The first 5 weeks of the term, which was supposed to be a whirlwind of meeting my colleagues in MRes, PhD students and Professors went quiet in front of a computer screen as I tried to work and study on my own. Despite having an experience of an year of distance learning, this wasn’t easy by any means as I was the only one learning from distance in my cohort unlike previous year in my masters and I wasn’t able to attend the lectures. I had even missed the away day, a two-day team building retreat planned to ease us in the STOR-i atmosphere by getting to know various student and staff. However, once my ATAS came, I applied for a priority visa, that arrives in 5 working days and planned to reach Lancaster before the sprint sessions, which is a group learning module. It is a unique element in the course, where a new Statistics or Operations Research (OR) topic will be introduced in a 2 hour lecture at the beginning of the week and we will be researching the topic as a small group for the rest of the week and present our findings. Personally, Since I came to Lancaster only during that week, it was even more challenging to manage academics and settling down in a new place. I managed to find an accommodation before the christmas break and finally settled down. However, the sprint sessions were very rewarding as we explored a lot of new and significant topics. My classmates, professors and staff were very kind and helpful. They never made me feel out of place though I joined them late. I learnt a lot in terms of statistical knowledge, team work and presentation skills. Its been a fascinating experience overall in the first term with some hardships. I look forward to my journey in STOR-i in the next terms which I am sure would be more enriching. See you in my next blog post with some interesting topics in Statistics/OR I have come across in my learnings here.

Source