Statistics – Harini Jayaraman

Konigsberg bridge problem and the evolution of Mathematics

harini-jayaraman — Fri, 29 Apr 2022 20:44:00 +0000

Well, In this blog I will discuss one of the problems that led to the development of new branches in mathematics like ‘Graph theory’ and ‘Topology’. Let’s dive into that interesting problem. What’s the Konigsberg problem? Konigsberg, a city you will find a hard time finding on the world map. However, it is one of the most famous cities in the world of Mathematics due to a particular quirk in its geography. The city of Konigsberg was set on both sides of the Pregel River, and included two large islands, which were connected to each other, or to the two mainland portions of the city, by seven bridges. In the 17th century, people were bothered with the question, whether they could walk around all 7 bridges crossing them only once. Let’s try it out! You could very well see other routes in this video and come up with even more routes. No matter how many ever routes you try you have to give up at some point because Euler answered the question in the negative and proved that there could be no route in Konigsberg that used all seven bridges exactly once and laid the foundation for Graph theory. How was it proved? Well, the answer he came up with, did not basically exist then, what he called the ‘Geometry of position’ , now known as the ‘Graph theory’ and which led to the fundamentals of ‘Topology’. Euler realized during his various trials that the geography or route taken to reach the island or riverbanks does not matter. This insight led to the simplifying of map further. What matters is the 4 landmasses and the relationship between them. We could represent each of the four landmasses here named as A,B,C and D respectively as you could see in the picture below, as a single point, which we now call as Nodes in Graph theory. We now have 4 nodes. We draw lines or edges between them to represent the bridges. Formally, we define a graph G = (N, E) as a pair where, The graph can be used to represent binary relations: (i,j) ϵ E if and only if the entity, i ϵ N and the entity j ϵ N interact with each other. The connectivity of nodes is vital for the structure of the graph. From the simplified graph, as you see in the above picture (Forgive my poor editing skills) with just nodes(Pink dots) and edges(Green lines) we can find the degree of the nodes which forms the basis of this solution. The degree of the node is nothing but the number of edges incident to it. So, in our case, it is the number of bridges that connects to each of the landmass. According to our challenge, person entering the landmass via one bridge should exit it through another bridge. Technically, edges leading to and from one node should occur in distinct pairs, meaning that the number of edges touching each node must be even. The only possible exceptions would be the locations of the beginning and the end of the walk. Looking at our graph, we can easily see that all the nodes have odd degree (in yellow in the below figure). Hence, no matter whichever path we choose we will end up crossing a bridge twice. Euler used this as a proof to formulate a general theory that apples to all graphs with two or more nodes. He defined something called Eulerian path that visits each edge only once and is possible only in one of the two scenarios. In the first scenario, all other nodes would be even and the starting point and ending point are different and should be nodes with odd degree. In the second scenario, starting and ending point would be the same. Why does it matter? In the history of mathematics, this is considered to be the first theorem of Graph theory. It basically forms the first proof for Network theory. Network theory in today’s world is a broad area of research. A network is a more general type of data structure that describes relations or interactions between entities. For instance, The Internet network is the physical infrastructure made of internet service providers, servers, routers (entities) and all the physical connections between them. We can never forget the Social networks (Friendships,relationships,work ties, social interactions) in today’s scenario. If time permits, I will discuss more on Social networks in my next blog soon. Current state of Konigsberg After the world war II two of its bridges has been destroyed and Eulerian path has been created. However you can’t find Konigsberg now. It is renamed as Kalingrad and is a part of Russia. I love this particular video explaining this problem in a simple way. You could well play this game based on this theory. Check if you can draw these line by going over each only once. https://www.transum.org/Maths/Activity/without/ Further Reading This lecture notes could quite give you an idea about Eulerian path and this blog could be a great starting point in understanding the world of Graph theory and Networks.

Source

Anomaly detection in Functional Data

harini-jayaraman — Wed, 06 Apr 2022 22:24:00 +0000

As discussed in my previous blog post, Functional data appears in various disciplines and each observation can be considered as real function of time observed at particular time points. Anomaly detection can be more challenging in terms of functional data because outliers are not always easily identified visually. Various methods are being developed to identify outliers in functional data. I’ve defined variety of outliers in functional data in my previous post, of which shape outliers is the most difficult to identify. I will discuss one of the method in identifying outliers in functional data we explored as a team during STOR608 sprints, as it was a great and interesting learning experience for me. Functional depth The anomaly detection techniques in the methods I will discuss depends on something called ‘Functional depth’. It is nothing but a specific ordering of the curves. There are various definitions/ways in literature to order these functional observations , where is closed real interval discretized into points. Modified Band Depth is one of the way. Modified Band Depth (MBD): MBD is defined based on the number of times a given curve is completely contained within the range defined by two other curves from sample. Hence, MBD indicates how central the curve is. Higher the MBD value, more central the curve is. The sample curves are ordered in decreasing value of their MBD implying first curve to be the median curve and last curve as the outlier. This article can give more insight on this topic if you would like to know more. MBD is highly used due to its computational efficiency. Anomaly Detection Methods Functional boxplot Functional box plot can be a best option to identify and classify shape, magnitude and amplitude outliers. However, functional boxplot easily identifies magnitude outliers but sometimes fail to detect shape outliers. By performing some sequential curve transformations we could convert the shape outliers to magnitude outliers and identify them. This idea has been proposed in this paper and could be an interesting read. To construct a functional boxplot, we need to order the curves according to their MBD value. We then find a central region which is analogous to the the inter-quartile range (IQR) for the classical boxplot and is a useful indication of the spread of the central 50% of the curves. The box in a classical boxplot works similar as such with he border of the 50% central region also called as the envelope. A functional boxplot identifies the anomalies by extending the 1.5 times the empiricial outlier criterion used for the classical boxplot. If a curve sits outside of the enclosure formed by inflating the 50 percent central zone by 1.5 times the range of the central region, it is considered an anomaly. Mathematically, a curve is an anomaly if, or Sequential Transformations As mentioned earlier, we transform shape and amplitude outliers into magnitude outliers via some curve transformations. I will discuss just one of the algorithm that has proven to be effective while there are other transformations that exist in reality details of which could be found in this paper. Algorithm Identify anomalies using the functional boxplot constructed with MBD. Anomalies detected at this stage are defined as -anomalies (magnitude anomalies). Apply a vertical alignment transformation to the data Repeat step 1 on the transformed data; anomalies detected at this stage are defined as -anomalies (amplitude anomalies). Apply a normalisation transformation to the data: Repeat step 1 on the transformed data; anomalies detected at this stage are defined as -anomalies (shape anomalies). These figures are from the data we were given during the sprints, as a part of our assignment. The data consists of observations for one variable, recorded at one minute intervals over 50 days. We segment it into periods of length = 1440 for each of the 50 days. You could observe the central region in pink, blue lines being the envelope. After transformation, we could observe four outliers in red dashed lines detected using this method. Further Reading I would like to add few papers that could be of interest and we referenced while working. Dai, W., Mrkvicka, T., Sun, Y., and Genton, M. G. (2020). Functional outlier detection and taxonomy by sequential transformations. Computational Statistics & Data Analysis. Arribas-Gil, A. and Romo, J. (2014). Shape outlier detection and visualization for functional data: the outliergram. Bio-statistics. Sun, Y. and Genton, M. G. (2011). Functional boxplots. Journal of Computational and Graphical Statistics Lopez-Pintado, S. and Romo, J. (2009). On the concept of depth for functional data. Journal of the American statistical Association Ojo, O. T., Fernández Anta, A., Lillo, R. E., Sguera, C. (2021). Detecting and classifying outliers in big functional data. Advances in Data Analysis and Classification I would also like to acknowledge that the works presented here were worked in group as a part of sprints. I hope you enjoyed this blog. For further queries, you could always contact me through mail or message me here.

Source

Anomaly detection

harini-jayaraman — Tue, 29 Mar 2022 10:45:00 +0000

As we had an overview of one of the unsupervised learning method, K-means Clustering, in my previous blog post, this post will introduce you to an another unsupervised learning method called Anomaly Detection. We should also note that Anomaly detection can be done using Supervised and Semi-supervised techniques as well. What is an Anomaly? Anomalies (Outliers) are patterns in data that do not conform to a well defined notion of normal behaviour. An unexpected change within these data patterns, or an event that does not conform to the expected data pattern, is considered an anomaly. In other words, an anomaly is a deviation from the usual happenings. Red Occurences in Figure 1, describes the various anomalies that can occur. In Figure 2, We could observe most of the data occurences are in the region and . The points far off from these normal region, such as and are anomalies. Anomalies can occur for various reasons. Customer acquisition costs, Malicious activity, Web page views, Revenue per click, Credit card frauds and more. These are valuable metrics for any data analyst and is considered advantage in decision making process. Therefore, Anomaly detection is an important process, which if efficient and timely can enable intervention and action to minimize or avoid effects of underlying cause. Even from a statistical context, anomaly detection helps the methods to be more robust and efficient. Types of Anomalies Anomalies are majorly classified into these three main categories which I will discuss below. However, there are wide variety of anomalies. If you are interested in learning various anomaly types and how it is distinguished, this paper could be helpful. Point Anomaly: A datapoint is considered an anomaly if its far outside or extremely away from other data points. It is also called as global outlier. For example , in figure 2, is a point anomaly. Contextual Anomaly: Contextual outliers are data points whose value significantly deviates from other data within some defined context. Also known as conditional outliers, its values are not outside global range but abnormal in comparison to the trend/period or seasonal pattern. For example, on Christmas day. Collective Anomaly: Collective anomalies are defined as sequences of observations that are not anomalous when considered individually, but together form an anomalous pattern. In layman words, having all cars on the freeway move to the left lane simultaneously would be a collective outlier because, even though moving to the left lane is not uncommon, it is unusual for all cars to move at the same time. With different types of data, we come across various types of outliers. If we consider database systems, we will have outliers like insertion, updation or deletion anomalies. In time series data, we have additive outliers, innovational outliers, level shift outliers, seasonal outliers and more. This documentation from IBM gives a general idea of understanding outliers in time-series data if you are interested. Data that appears in a variety of disciplines including biology, medicine, meteorology and engineering, like Functional data (type of data consists of curves varying over a continuum, such as time, frequency, or wavelength), we come across outliers like magnitude, amplitude and shape outliers. Magnitude Outlier: If any or one of the curve is outlying in some parts or across whole design domain. Amplitude Outlier: The curves with unusual oscillation levels Shape/Pattern Outliers: The curves with unusual shape or the ones that significantly differ from pattern exhibited by other curves even after centralizing and normalizing. Anomaly detection is the process of identifying these items or events in data sets that differ from the norm. Anomaly detection is often performed on unlabeled data, which is known as unsupervised anomaly detection. Anomaly detection is based on two assumptions: Anomalies occur very rarely in the data Their features differ significantly from the normal cases. A business system that is considered normal today may not be so in the future. Most business systems change over time as the result of various factors. Best example is the current situation of Covid19, the whole world is experiencing. The unprecedented scenarios everywhere, which often is termed as the “new normal”, it was abnormal or anomalous outcomes earlier. These anomalous outcomes can result in positive or negative impact for any organization and are important to keep track of, for formulating a long term business strategy. Hence, anomaly detection in every domain is important and finding more effective ways to detect it will be most important skill in the future. In my next blog, we will discuss more statistics, a method to detect outliers in functional data. I hope this post gave you an insight into what anomaly is and how anomaly detection matters. Till then, I found this small game interesting, eyeballing and finding outliers. Try it for yourself if you could find. Further Reading: This blog could help you in basic overview of Anomalies. I have shared the links of few papers that can be of interest. https://link.springer.com/article/10.1007/s41060-021-00265-1 https://www.sciencedirect.com/science/article/abs/pii/S0167947320300517?via%3Dihub

Source