07/08/2014
CorrelationMap / CorrelationScanner
Heat map of correlations and lead-lag relationships between financial data series, combining known and proprietary correlation models and metrics
Weiss & Quast
- Home
- United Kingdom
- London
- Weiss & Quast
Financial Industry consulting since 1995:
Munich-London-NewYork
Front-to-back treasury, trading & he
Munich - London - NewYork
*** Financial technology
Front-to-back system selection, design, development and implementation
*** Risk & portfolio management, trading, hedging
Quantitative consultancy, research, analytics and modeling; financial engineering
Design, review and implementation of risk & pricing models as well as trading, hedging and portfolio strategies for all major sys
tems (Calypso, Murex, Reuters, TradeStation, InteractiveBrokers, InteractiveData / ESignal, NinjaTrader, MultiCharts)
DerivaRisk risk and trading management framework
*** Project management
Project management & organization
Shaping of the project scope, consistent enforcement of project objectives
Resource planning, project controlling & reporting
Risk and quality management
PMO setup and management
Requirements management & business analysis
Structured gathering, analysis and verification of business requirements
Transformation of requirements into use cases, processes, system solution definitions, functional specifications
Feasibility studies, gap and cost-benefit analyses
Managing of complete vendor selection, solution evaluation and POC processes
*** Previous & present customers
Eurostoxx50 companies: MunichRe, UniCredit, EADS/Airbus
Several private hedge funds worldwide
**** CONSULTING AREAS
** Financial markets, risk management, processes & systems
Front-to-back systems & processes
Customization and enhancement of trading systems & infrastructures
System migration and upgrading issues (gap analysis, requirements engineering)
Product evaluation and vendor selection processes
Analysis, design, review and optimization of front-to-back processes, e.g. Pre- and post-trade analysis
Order algorithms
Straight-through processing (STP)
Market & credit risk measurement and management
Portfolio and cash flow management
Settlement, accounting, auditing, compliance
Database modeling, design and setup of tailor-made reporting and BI solutions
** Financial engineering & modeling
Design and review of risk and pricing models for all major systems
Structuring of derivatives (retail and OTC)
Index and ETF composition
Trading, hedging and portfolio strategies; market research
Market research using proprietary smart auto-adaptive technical indicators
Strategy design, review, testing and implementation
Order book / T&S flow analysis and pattern identification algorithms (self-learning neural networks)
Short-term and high-frequency trading setups
Mid-term statistical arbitrage (e.g. spread trading, sector rotation)
Stress testing, back testing, what-if-scenarios
Performance measurement and attribution
Strategy libraries and technical analysis indicators as add-ons for all major systems (Calypso, Murex, Bloomberg, Reuters, TradeStation, InteractiveBrokers, InteractiveData / ESignal, NinjaTrader, MultiCharts, WealthLab)
Implementation of proprietary pattern detection algorithms (based on neural networks and genetic algos) into all major systems
** Risk management & compliance
Servicing the complete risk management process
Identifying risk exposures
Measuring risk exposures
Assessing the effects of exposures
Finding ways to shift or trade risks
Assessing costs and benefits of the different ways
Forming a risk mitigation strategy: Keep, avoid, transfer, mitigate
Evaluating performance
Market risk: VaR, simulations, stress tests, scenarios, EVT
Credit risk: LGD, PD, EAD, EDF, ratings/transition matrices, systems (KMV, Credit Metrics, Credit Risk+)
Operational risk: LDA, severity & frequency, methods (top-down, bottom-up, IMA; BIA, SA, AMA), model risks
Design and implementation of risk management programs and GRC concepts
Assisting in compliance issues: BaFin, MiFid, Basel, Solvency, DerivV, KAG, MaH, MaRisk, NPP, IFRS, IAS, SOX, OpRisk, SEC, FSA, CFTC
VaR (Value-at-risk) approaches: Delta-normal, historicel simulation, Mont-Carlo-simulation; LVAR, CFAR
Regulatory and economic capital, assessment approaches: Standardized, Foundation, Advanced (IRB)
06/08/2014
A stock index or stock market index is a measurement of the value of a section of the stock market. It is computed from the prices of selected stocks (typically a weighted average). It is a tool used by investors and financial managers to describe the market, and to compare the return on specific investments.
An index is a mathematical construct, so it may not be invested in directly. But many mutual funds and exchange-traded funds attempt to "track" an index (see index fund), and those funds that do not may be judged against those that do.
Stock market indices may be classed in many ways. A 'world' or 'global' stock market index includes (typically large) companies without regard for where they are domiciled or traded. Two examples are MSCI World and S&P Global 100.
A 'national' index represents the performance of the stock market of a given nation—and by proxy, reflects investor sentiment on the state of its economy. The most regularly quoted market indices are national indices composed of the stocks of large companies listed on a nation's largest stock exchanges, such as the American S&P 500, the Japanese Nikkei 225, and the British FTSE 100.
Other indices may be regional, such as the FTSE Developed Europe Index or the FTSE Developed Asia Pacific Index.
The concept may be extended well beyond an exchange. The Wilshire 5000 Index, the original total market index, represents the stocks of nearly every publicly traded company in the United States, including all U.S. stocks traded on the New York Stock Exchange (but not ADRs or limited partnerships), NASDAQ and American Stock Exchange. Russell Investment Group added to the family of indices by launching the Russel Global Index.[1]
More specialized indices exist tracking the performance of specific sectors of the market. Some examples include the Wilshire US REIT which tracks more than 80 American real estate investment trusts and the Morgan Stanley Biotech Index which consists of 36 American firms in the biotechnology industry. Other indices may track companies of a certain size, a certain type of management, or even more specialized criteria — one index published by Linux Weekly News tracks stocks of companies that sell products and services based on the Linux operating environment.
Index versions
Some indices, such as the S&P 500, have multiple versions.[2] These versions can differ based on how the index components are weighted and on how dividends are accounted for. For example, there are three versions of the S&P 500 index: price return, which only considers the price of the components, total return, which accounts for dividend reinvestment, and net total return, which accounts for dividend reinvestment after the deduction of a withholding tax.[3] As another example, the Wilshire 4500 and Wilshire 5000 indices have five versions each: full capitalization total return, full capitalization price, float-adjusted total return, float-adjusted price, and equal weight. The difference between the full capitalization, float-adjusted, and equal weight versions is in how index components are weighted.
Weighting
An index may also be classified according to the method used to determine its price. In a price-weighted index such as the Dow Jones Industrial Average, Amex Major Market Index, and the NYSE ARCA Tech 100 Index, the price of each component stock is the only consideration when determining the value of the index. Thus, price movement of even a single security will heavily influence the value of the index even though the dollar shift is less significant in a relatively highly valued issue, and moreover ignoring the relative size of the company as a whole. In contrast, a market-value weighted or capitalization-weighted index such as the Hang Seng Index factors in the size of the company. Thus, a relatively small shift in the price of a large company will heavily influence the value of the index.
Traditionally, capitalization- or share-weighted indices all had a full weighting, i.e. all outstanding shares were included. Recently, many of them have changed to a float-adjusted weighting which helps indexing.
An equal-weighted index is one in which all components are assigned the same value. For example, the Barron's 400 Index assigns an equal value of 0.25% to each of the 400 stocks included in the index, which together add up to the 100% whole.[6]
A modified capitalization-weighted index is a hybrid between capitalization weighting and equal weighting. It is similar to a capitalization weighting with one main difference: the largest stocks are capped to a percent of the weight of the total stock index and the excess weight will be redistributed equally amongst the stocks under that cap. Moreover, in 2005, Standard & Poor's introduced the S&P Pure Growth Style Index and S&P Pure Value Style Index which was attribute-weighted. That is, a stock's weight in the index is decided by the score it gets relative to the value attributes that define the criteria of a specific index, the same measure used to select the stocks in the first place. For these two stocks, a score is calculated for every stock, be it their growth score or the value score (a stock cannot be both) and accordingly they are weighted for the index.[7]
Criticism of capitalization-weighting
The use of capitalization-weighted indices is often justified by the central conclusion of modern portfolio theory that the optimal investment strategy for any investor is to hold the market portfolio, the capitalization-weighted portfolio of all assets. However, empirical tests conclude that market indices are not efficient.[citation needed] This can be explained by the fact that these indices do not include all assets or by the fact that the theory does not hold. The practical conclusion is that using capitalization-weighted portfolios is not necessarily the optimal method.
As a consequence, capitalization-weighting has been subject to severe criticism (see e.g. Haugen and Baker 1991, Amenc, Goltz, and Le Sourd 2006, or Hsu 2006), pointing out that the mechanics of capitalization-weighting lead to trend-following strategies that provide an inefficient risk-return trade-off.
Also, while capitalization-weighting is the standard in equity index construction, different weighting schemes exist. First, while most indices use capitalization-weighting, additional criteria are often taken into account, such as sales/revenue and net income (see the “Guide to the Dow Jones Global Titan 50 Index”, January 2006). Second, as an answer to the critiques of capitalization-weighting, equity indices with different weighting schemes have emerged, such as "wealth"-weighted (Morris, 1996), “fundamental”-weighted (Arnott, Hsu and Moore 2005), “diversity”-weighted (Fernholz, Garvy, and Hannon 1998) or equal-weighted indices.[8]
Indices and passive investment management
There has been an accelerating trend in recent decades to create passively managed mutual funds that are based on market indices, known as index funds. Advocates claim that index funds routinely beat a large majority of actively managed mutual funds; one study[citation needed] claimed that over time, the average actively managed fund has returned 1.8% less than the S&P 500 index - a result nearly equal to the average expense ratio of mutual funds (fund expenses are a drag on the funds' return by exactly that ratio). Since index funds attempt to replicate the holdings of an index, they obviate the need for — and thus many costs of — the research entailed in active management, and have a lower churn rate (the turnover of securities which lose fund managers' favor and are sold, with the attendant cost of commissions and capital gains taxes).
Indices are also a common basis for a related type of investment, the exchange-traded fund or ETF. Unlike an index fund, which is priced daily, an ETF is priced continuously, is optionable, and can be sold short.
Ethical stock market indices
A notable specialised index type is those for ethical investing indices that include only those companies satisfying ecological or social criteria, e.g. those of The Calvert Group, KLD, FTSE4Good Index, Dow Jones Sustainability Index, Standard Ethics Italian Index and Wilderhill Clean Energy Index.
In 2010, the OIC announced the initiation of a stock index that complies with Islamic law's ban on alcohol, to***co and gambling. Other such equities, such as the Dow Jones Islamic Market World Index, already exist.[9]
Another important trend is strict mechanical criteria for inclusion and exclusion to prevent market manipulation, e.g. in Canada when Nortel was permitted to rise to over 30% of the TSE 300 index value. Ethical indices have a particular interest in mechanical criteria, seeking to avoid accusations of ideological bias in selection, and have pioneered techniques for inclusion and exclusion of stocks based on complex criteria. Another means of mechanical selection is mark-to-future methods that exploit scenarios produced by multiple analysts weighted according to probability, to determine which stocks have become too risky to hold in the index of concern.
Critics of such initiatives argue that many firms satisfy mechanical "ethical criteria", e.g. regarding board composition or hiring practices, but fail to perform ethically with respect to shareholders, e.g. Enron. Indeed, the seeming "seal of approval" of an ethical index may put investors more at ease, enabling scams. One response to these criticisms is that trust in the corporate management, index criteria, fund or index manager, and securities regulator, can never be replaced by mechanical means, so "market transparency" and "disclosure" are the only long-term-effective paths to fair markets. From a financial perspective, it is not obvious whether ethical indices or ethical funds will out-perform their more conventional counterparts. Theory might suggest that returns would be lower since the investible universe is artificially reduced and with it portfolio efficiency. On the other hand, companies with good social performances might be better run, have more committed workers and customers, and be less likely to suffer reputational damage from incidents (oil spillages, industrial tribunals, etc.) and this might result in lower share price volatility.[10] The empirical evidence on the performance of ethical funds and of ethical firms versus their mainstream comparators is very mixed for both stock [11][12] and debt markets.[13]
06/08/2014
Correlation and dependence
This article is about correlation and dependence in statistical data.
In statistics, dependence is any statistical relationship between two random variables or two sets of data. Correlation refers to any of a broad class of statistical relationships involving dependence.
Familiar examples of dependent phenomena include the correlation between the physical statures of parents and their offspring, and the correlation between the demand for a product and its price. Correlations are useful because they can indicate a predictive relationship that can be exploited in practice. For example, an electrical utility may produce less power on a mild day based on the correlation between electricity demand and weather. In this example there is a causal relationship, because extreme weather causes people to use more electricity for heating or cooling; however, statistical dependence is not sufficient to demonstrate the presence of such a causal relationship (i.e., correlation does not imply causation).
Formally, dependence refers to any situation in which random variables do not satisfy a mathematical condition of probabilistic independence. In loose usage, correlation can refer to any departure of two or more random variables from independence, but technically it refers to any of several more specialized types of relationship between mean values. There are several correlation coefficients, often denoted ρ or r, measuring the degree of correlation. The most common of these is the Pearson correlation coefficient, which is sensitive only to a linear relationship between two variables (which may exist even if one is a nonlinear function of the other). Other correlation coefficients have been developed to be more robust than the Pearson correlation – that is, more sensitive to nonlinear relationships.[1][2][3] Mutual information can also be applied to measure dependence between two variables.
Several sets of (x, y) points, with the Pearson correlation coefficient of x and y for each set. Note that the correlation reflects the noisiness and direction of a linear relationship (top row), but not the slope of that relationship (middle), nor many aspects of nonlinear relationships (bottom). N.B.: the figure in the center has a slope of 0 but in that case the correlation coefficient is undefined because the variance of Y is zero.
Contents
[hide]
1 Pearson's product-moment coefficient
2 Rank correlation coefficients
3 Other measures of dependence among random variables
4 Sensitivity to the data distribution
5 Correlation matrices
6 Common misconceptions
6.1 Correlation and causality
6.2 Correlation and linearity
7 Bivariate normal distribution
8 Partial correlation
9 See also
10 References
11 Further reading
12 External links
Pearson's product-moment coefficient[edit]
Main article: Pearson product-moment correlation coefficient
The most familiar measure of dependence between two quantities is the Pearson product-moment correlation coefficient, or "Pearson's correlation coefficient", commonly called simply "the correlation coefficient". It is obtained by dividing the covariance of the two variables by the product of their standard deviations. Karl Pearson developed the coefficient from a similar but slightly different idea by Francis Galton.[4]
The population correlation coefficient ρX,Y between two random variables X and Y with expected values μX and μY and standard deviations σX and σY is defined as:
ho_{X,Y}=mathrm{corr}(X,Y)={mathrm{cov}(X,Y) over sigma_X sigma_Y} ={E[(X-mu_X)(Y-mu_Y)] over sigma_Xsigma_Y},
where E is the expected value operator, cov means covariance, and, corr a widely used alternative notation for the correlation coefficient.
The Pearson correlation is defined only if both of the standard deviations are finite and nonzero. It is a corollary of the Cauchy–Schwarz inequality that the correlation cannot exceed 1 in absolute value. The correlation coefficient is symmetric: corr(X,Y) = corr(Y,X).
The Pearson correlation is +1 in the case of a perfect direct (increasing) linear relationship (correlation), −1 in the case of a perfect decreasing (inverse) linear relationship (anticorrelation),[5] and some value between −1 and 1 in all other cases, indicating the degree of linear dependence between the variables. As it approaches zero there is less of a relationship (closer to uncorrelated). The closer the coefficient is to either −1 or 1, the stronger the correlation between the variables.
If the variables are independent, Pearson's correlation coefficient is 0, but the converse is not true because the correlation coefficient detects only linear dependencies between two variables. For example, suppose the random variable X is symmetrically distributed about zero, and Y = X2. Then Y is completely determined by X, so that X and Y are perfectly dependent, but their correlation is zero; they are uncorrelated. However, in the special case when X and Y are jointly normal, uncorrelatedness is equivalent to independence.
If we have a series of n measurements of X and Y written as xi and yi where i = 1, 2, ..., n, then the sample correlation coefficient can be used to estimate the population Pearson correlation r between X and Y. The sample correlation coefficient is written
r_{xy}=frac{sumlimits_{i=1}^n (x_i-ar{x})(y_i-ar{y})}{(n-1) s_x s_y} =frac{sumlimits_{i=1}^n (x_i-ar{x})(y_i-ar{y})} {sqrt{sumlimits_{i=1}^n (x_i-ar{x})^2 sumlimits_{i=1}^n (y_i-ar{y})^2}},
where x and y are the sample means of X and Y, and sx and sy are the sample standard deviations of X and Y.
This can also be written as:
r_{xy}=frac{sum x_iy_i-n ar{x} ar{y}}{(n-1) s_x s_y}=frac{nsum x_iy_i-sum x_isum y_i} {sqrt{nsum x_i^2-(sum x_i)^2}~sqrt{nsum y_i^2-(sum y_i)^2}}.
If x and y are results of measurements that contain measurement error, the realistic limits on the correlation coefficient are not −1 to +1 but a smaller range.[6]
For the case of a linear model with a single independent variable, the coefficient of determination (R squared) is the square of r, Pearson's product-moment coefficient .
Rank correlation coefficients[edit]
Main articles: Spearman's rank correlation coefficient and Kendall tau rank correlation coefficient
Rank correlation coefficients, such as Spearman's rank correlation coefficient and Kendall's rank correlation coefficient (τ) measure the extent to which, as one variable increases, the other variable tends to increase, without requiring that increase to be represented by a linear relationship. If, as the one variable increases, the other decreases, the rank correlation coefficients will be negative. It is common to regard these rank correlation coefficients as alternatives to Pearson's coefficient, used either to reduce the amount of calculation or to make the coefficient less sensitive to non-normality in distributions. However, this view has little mathematical basis, as rank correlation coefficients measure a different type of relationship than the Pearson product-moment correlation coefficient, and are best seen as measures of a different type of association, rather than as alternative measure of the population correlation coefficient.[7][8]
To illustrate the nature of rank correlation, and its difference from linear correlation, consider the following four pairs of numbers (x, y):
(0, 1), (10, 100), (101, 500), (102, 2000).
As we go from each pair to the next pair x increases, and so does y. This relationship is perfect, in the sense that an increase in x is always accompanied by an increase in y. This means that we have a perfect rank correlation, and both Spearman's and Kendall's correlation coefficients are 1, whereas in this example Pearson product-moment correlation coefficient is 0.7544, indicating that the points are far from lying on a straight line. In the same way if y always decreases when x increases, the rank correlation coefficients will be −1, while the Pearson product-moment correlation coefficient may or may not be close to −1, depending on how close the points are to a straight line. Although in the extreme cases of perfect rank correlation the two coefficients are both equal (being both +1 or both −1) this is not in general so, and values of the two coefficients cannot meaningfully be compared.[7] For example, for the three pairs (1, 1) (2, 3) (3, 2) Spearman's coefficient is 1/2, while Kendall's coefficient is 1/3.
Other measures of dependence among random variables[edit]
The information given by a correlation coefficient is not enough to define the dependence structure between random variables.[9] The correlation coefficient completely defines the dependence structure only in very particular cases, for example when the distribution is a multivariate normal distribution. (See diagram above.) In the case of elliptical distributions it characterizes the (hyper-)ellipses of equal density, however, it does not completely characterize the dependence structure (for example, a multivariate t-distribution's degrees of freedom determine the level of tail dependence).
Distance correlation and Brownian covariance / Brownian correlation [10][11] were introduced to address the deficiency of Pearson's correlation that it can be zero for dependent random variables; zero distance correlation and zero Brownian correlation imply independence.
The Randomized Dependence Coefficient [12] is a computationally efficient, copula-based measure of dependence between multivariate random variables. RDC is invariant with respect to non-linear scalings of random variables, is capable of discovering a wide range of functional association patterns and takes value zero at independence.
The correlation ratio is able to detect almost any functional dependency,[citation needed][clarification needed] and the entropy-based mutual information, total correlation and dual total correlation are capable of detecting even more general dependencies. These are sometimes referred to as multi-moment correlation measures,[citation needed] in comparison to those that consider only second moment (pairwise or quadratic) dependence.
The polychoric correlation is another correlation applied to ordinal data that aims to estimate the correlation between theorised latent variables.
One way to capture a more complete view of dependence structure is to consider a copula between them.
The coefficient of determination generalizes the correlation coefficient for relationships beyond simple linear regression.
Sensitivity to the data distribution[edit]
The degree of dependence between variables X and Y does not depend on the scale on which the variables are expressed. That is, if we are analyzing the relationship between X and Y, most correlation measures are unaffected by transforming X to a + bX and Y to c + dY, where a, b, c, and d are constants. This is true of some correlation statistics as well as their population analogues. Some correlation statistics, such as the rank correlation coefficient, are also invariant to monotone transformations of the marginal distributions of X and/or Y.
Pearson/Spearman correlation coefficients between X and Y are shown when the two variables' ranges are unrestricted, and when the range of X is restricted to the interval (0,1).
Most correlation measures are sensitive to the manner in which X and Y are sampled. Dependencies tend to be stronger if viewed over a wider range of values. Thus, if we consider the correlation coefficient between the heights of fathers and their sons over all adult males, and compare it to the same correlation coefficient calculated when the fathers are selected to be between 165 cm and 170 cm in height, the correlation will be weaker in the latter case. Several techniques have been developed that attempt to correct for range restriction in one or both variables, and are commonly used in meta-analysis; the most common are Thorndike's case II and case III equations.[13]
Various correlation measures in use may be undefined for certain joint distributions of X and Y. For example, the Pearson correlation coefficient is defined in terms of moments, and hence will be undefined if the moments are undefined. Measures of dependence based on quantiles are always defined. Sample-based statistics intended to estimate population measures of dependence may or may not have desirable statistical properties such as being unbiased, or asymptotically consistent, based on the spatial structure of the population from which the data were sampled.
Sensitivity to the data distribution can be used to an advantage. For example, scaled correlation is designed to use the sensitivity to the range in order to pick out correlations between fast components of time series.[14] By reducing the range of values in a controlled manner, the correlations on long time scale are filtered out and only the correlations on short time scales are revealed.
Correlation matrices[edit]
The correlation matrix of n random variables X1, ..., Xn is the n × n matrix whose i,j entry is corr(Xi, Xj). If the measures of correlation used are product-moment coefficients, the correlation matrix is the same as the covariance matrix of the standardized random variables Xi / σ (Xi) for i = 1, ..., n. This applies to both the matrix of population correlations (in which case "σ" is the population standard deviation), and to the matrix of sample correlations (in which case "σ" denotes the sample standard deviation). Consequently, each is necessarily a positive-semidefinite matrix.
The correlation matrix is symmetric because the correlation between Xi and Xj is the same as the correlation between Xj and Xi.
Common misconceptions[edit]
Correlation and causality[edit]
Main article: Correlation does not imply causation
See also: Normally distributed and uncorrelated does not imply independent
The conventional dictum that "correlation does not imply causation" means that correlation cannot be used to infer a causal relationship between the variables.[15] This dictum should not be taken to mean that correlations cannot indicate the potential existence of causal relations. However, the causes underlying the correlation, if any, may be indirect and unknown, and high correlations also overlap with identity relations (tautologies), where no causal process exists. Consequently, establishing a correlation between two variables is not a sufficient condition to establish a causal relationship (in either direction).
A correlation between age and height in children is fairly causally transparent, but a correlation between mood and health in people is less so. Does improved mood lead to improved health, or does good health lead to good mood, or both? Or does some other factor underlie both? In other words, a correlation can be taken as evidence for a possible causal relationship, but cannot indicate what the causal relationship, if any, might be.
Correlation and linearity[edit]
Four sets of data with the same correlation of 0.816
The Pearson correlation coefficient indicates the strength of a linear relationship between two variables, but its value generally does not completely characterize their relationship[16] . In particular, if the conditional mean of Y given X, denoted E(Y|X), is not linear in X, the correlation coefficient will not fully determine the form of E(Y|X).
The image on the right shows scatterplots of Anscombe's quartet, a set of four different pairs of variables created by Francis Anscombe.[17] The four y variables have the same mean (7.5), variance (4.12), correlation (0.816) and regression line (y = 3 + 0.5x). However, as can be seen on the plots, the distribution of the variables is very different. The first one (top left) seems to be distributed normally, and corresponds to what one would expect when considering two variables correlated and following the assumption of normality. The second one (top right) is not distributed normally; while an obvious relationship between the two variables can be observed, it is not linear. In this case the Pearson correlation coefficient does not indicate that there is an exact functional relationship: only the extent to which that relationship can be approximated by a linear relationship. In the third case (bottom left), the linear relationship is perfect, except for one outlier which exerts enough influence to lower the correlation coefficient from 1 to 0.816. Finally, the fourth example (bottom right) shows another example when one outlier is enough to produce a high correlation coefficient, even though the relationship between the two variables is not linear.
These examples indicate that the correlation coefficient, as a summary statistic, cannot replace visual examination of the data. Note that the examples are sometimes said to demonstrate that the Pearson correlation assumes that the data follow a normal distribution, but this is not correct.[4]
Bivariate normal distribution[edit]
If a pair (X, Y) of random variables follows a bivariate normal distribution, the conditional mean E(X|Y) is a linear function of Y, and the conditional mean E(Y|X) is a linear function of X. The correlation coefficient r between X and Y, along with the marginal means and variances of X and Y, determines this linear relationship:
E(Ymid X) = E(Y) + rsigma_yfrac{X-E(X)}{sigma_x},
where E(X) and E(Y) are the expected values of X and Y, respectively, and σx and σy are the standard deviations of X and Y, respectively.
Partial correlation[edit]
Main article: Partial correlation
If a population or data-set is characterized by more than two variables, a partial correlation coefficient measures the strength of dependence between a pair of variables that is not accounted for by the way in which they both change in response to variations in a selected subset of the other variables.
See also[edit]
06/08/2014
Correlation does not imply causation.
Correlation does not imply causation is a phrase in science and statistics that emphasizes that a correlation between two variables does not necessarily imply that one causes the other.[1][2] Many statistical tests calculate correlation between variables. A few go further and calculate the likelihood of a true causal relationship; examples are the Granger causality test and convergent cross mapping.
The counter assumption, that correlation proves causation, is considered a questionable cause logical fallacy in that two events occurring together are taken to have a cause-and-effect relationship. This fallacy is also known as cm hoc ergo propter hoc, Latin for "with this, therefore because of this", and "false cause". A similar fallacy, that an event that follows another was necessarily a consequence of the first event, is sometimes described as post hoc ergo propter hoc (Latin for "after this, therefore because of this").
In a widely studied example, numerous epidemiological studies showed that women who were taking combined hormone replacement therapy (HRT) also had a lower-than-average incidence of coronary heart disease (CHD), leading doctors to propose that HRT was protective against CHD. But randomized controlled trials showed that HRT caused a small but statistically significant increase in risk of CHD. Re-analysis of the data from the epidemiological studies showed that women undertaking HRT were more likely to be from higher socio-economic groups (ABC1), with better-than-average diet and exercise regimens. The use of HRT and decreased incidence of coronary heart disease were coincident effects of a common cause (i.e. the benefits associated with a higher socioeconomic status), rather than cause and effect, as had been supposed.[3]
As with any logical fallacy, identifying that the reasoning behind an argument is flawed does not imply that the resulting conclusion is false. In the instance above, if the trials had found that hormone replacement therapy caused a decrease in coronary heart disease, but not to the degree suggested by the epidemiological studies, the assumption of causality would have been correct, although the logic behind the assumption would still have been flawed.
In logic, the technical use of the word "implies" means "to be a sufficient circumstance". This is the meaning intended by statisticians when they say causation is not certain. Indeed, p implies q has the technical meaning of the material conditional: if p then q symbolized as p → q. That is "if circumstance p is true, then q follows." In this sense, it is always correct to say "Correlation does not imply causation."
However, in casual use, the word "imply" loosely means suggests rather than requires. The idea that correlation and causation are connected is certainly true; where there is causation, there is a likely correlation. Indeed, correlation is used when inferring causation; the important point is that such inferences are made after correlations are confirmed as real and all causational relationship are systematically explored using large enough data sets.
Edward Tufte, in a criticism of the brevity of "correlation does not imply causation", deprecates the use of "is" to relate correlation and causation (as in "Correlation is not causation"), citing its inaccuracy as incomplete.[1] While it is not the case that correlation is causation, simply stating their nonequivalence omits information about their relationship. Tufte suggests that the shortest true statement that can be made about causality and correlation is one of the following:[4]
"Empirically observed covariation is a necessary but not sufficient condition for causality."
"Correlation is not causation but it sure is a hint."
General pattern
For any two correlated events, A and B, the following relationships are possible:
A causes B;
B causes A;
A and B are consequences of a common cause, but do not cause each other;
There is no connection between A and B; the correlation is coincidental.
Less clear-cut correlations are also possible. For example, causality is not necessarily one-way; in a predator-prey relationship, predator numbers affect prey, but prey numbers, i.e. food supply, also affect predators.
The cm hoc ergo propter hoc logical fallacy can be expressed as follows:
A occurs in correlation with B.
Therefore, A causes B.
In this type of logical fallacy, one makes a premature conclusion about causality after observing only a correlation between two or more factors. Generally, if one factor (A) is observed to only be correlated with another factor (B), it is sometimes taken for granted that A is causing B, even when no evidence supports it. This is a logical fallacy because there are at least five possibilities:
A may be the cause of B.
B may be the cause of A.
some unknown third factor C may actually be the cause of both A and B.
there may be a combination of the above three relationships. For example, B may be the cause of A at the same time as A is the cause of B (contradicting that the only relationship between A and B is that A causes B). This describes a self-reinforcing system.
the "relationship" is a coincidence or so complex or indirect that it is more effectively called a coincidence (i.e. two events occurring at the same time that have no direct relationship to each other besides the fact that they are occurring at the same time). A larger sample size helps to reduce the chance of a coincidence, unless there is a systematic error in the experiment.
In other words, there can be no conclusion made regarding the existence or the direction of a cause-and-effect relationship only from the fact that A and B are correlated. Determining whether there is an actual cause-and-effect relationship requires further investigation, even when the relationship between A and B is statistically significant, a large effect size is observed, or a large part of the variance is explained.
The point of view that correlation implies causation may be regarded as a theory of causality, which is somewhat inherent to the field of statistics. Within academia as a whole, the nature of causality is systematically investigated from several academic disciplines, including philosophy and physics.
In academia, there is a significant number of theories on causality; The Oxford Handbook of Causation (Beebee et al. 2009) encompasses 770 pages. Among the more influential theories within philosophy are Aristotle's Four causes and Al-Ghazali's occasionalism.[13] David Hume argued that causality is based on experience, and experience similarly based on the assumption that the future models the past, which in turn can only be based on experience – leading to circular logic. In conclusion, he asserted that causality is not based on actual reasoning: only correlation can actually be perceived.[14] Immanuel Kant, according to Beebee et al., held that "a causal principle according to which every event has a cause, or follows according to a causal law, cannot be established through induction as a purely empirical claim, since it would then lack strict universality, or necessity".[15]
Outside the field of philosophy, theories of causation can be identified in classical mechanics, statistical mechanics, quantum mechanics, spacetime theories, biology, social sciences, and law.[15] To establish a correlation as causal within physics, it is normally understood that the cause and the effect must connect through a local mechanism (cf. for instance the concept of impact) or a nonlocal mechanism (cf. the concept of field), in accordance with known laws of nature.
From the point of view of thermodynamics, universal properties of causes as compared to effects have been identified through the Second law of thermodynamics, confirming the ancient, medieval and Cartesian[16] view that "the cause is greater than the effect" for the particular case of thermodynamic free energy. This, in turn, is challenged by popular interpretations of the concepts of nonlinear systems and the butterfly effect, in which small events cause large effects due to, respectively, unpredictability and an unlikely triggering of large amounts of potential energy.
Causality construed from counterfactual states
See also: Verificationism
Intuitively, causation seems to require not just a correlation, but a counterfactual dependence. Suppose that a student performed poorly on a test and guesses that the cause was his not studying. To prove this, one thinks of the counterfactual – the same student writing the same test under the same circumstances but having studied the night before. If one could rewind history, and change only one small thing (making the student study for the exam), then causation could be observed (by comparing version 1 to version 2). Because one cannot rewind history and replay events after making small controlled changes, causation can only be inferred, never exactly known. This is referred to as the Fundamental Problem of Causal Inference – it is impossible to directly observe causal effects.[17]
A major goal of scientific experiments and statistical methods is to approximate as best possible the counterfactual state of the world.[18] For example, one could run an experiment on identical twins who were known to consistently get the same grades on their tests. One twin is sent to study for six hours while the other is sent to the amusement park. If their test scores suddenly diverged by a large degree, this would be strong evidence that studying (or going to the amusement park) had a causal effect on test scores. In this case, correlation between studying and test scores would almost certainly imply causation.
Well-designed experimental studies replace equality of individuals as in the previous example by equality of groups. The objective is to construct two groups that are similar except for the treatment that the groups receive. This is achieved by selecting subjects from a single population and randomly assigning them to two or more groups. The likelihood of the groups behaving similarly to one another (on average) rises with the number of subjects in each group. If the groups are essentially equivalent except for the treatment they receive, and a difference in the outcome for the groups is observed, then this constitutes evidence that the treatment is responsible for the outcome, or in other words the treatment causes the observed effect. However, an observed effect could also be caused "by chance", for example as a result of random perturbations in the population. Statistical tests exist to quantify the likelihood of erroneously concluding that an observed difference exists when in fact it does not (for example see P-value).
Causality predicted by an extrapolation of trends
See also: Inertia
When experimental studies are impossible and only pre-existing data are available, as is usually the case for example in economics, regression analysis can be used. Factors other than the potential causative variable of interest are controlled for by including them as regressors in addition to the regressor representing the variable of interest. False inferences of causation due to reverse causation (or wrong estimates of the magnitude of causation due the presence of bidirectional causation) can be avoided by using explanators (regressors) that are necessarily exogenous, such as physical explanators like rainfall amount (as a determinant of, say, futures prices), lagged variables whose values were determined before the dependent variable's value was determined, instrumental variables for the explanators (chosen based on their known exogeneity), etc. See Causality and Economics. Spurious correlation due to mutual influence from a third, common, causative variable, is harder to avoid: the model must be specified such that there is a theoretical reason to believe that no such underlying causative variable has been omitted from the model. In particular, underlying time trends of both the dependent variable and the independent (potentially causative) variable must be controlled for by including time as another independent variable.[citation needed]
Use of correlation as scientific evidence
Much of scientific evidence is based upon a correlation of variables[19] – they are observed to occur together. Scientists are careful to point out that correlation does not necessarily mean causation. The assumption that A causes B simply because A correlates with B is often not accepted as a legitimate form of argument.
However, sometimes people commit the opposite fallacy – dismissing correlation entirely, as if it does not suggest causation at all. This would dismiss a large swath of important scientific evidence.[19] Since it may be difficult or ethically impossible to run controlled double-blind studies, correlational evidence from several different angles may be the strongest causal evidence available.[20] The combination of limited available methodologies with the dismissing correlation fallacy has on occasion been used to counter a scientific finding. For example, the to***co industry has historically relied on a dismissal of correlational evidence to reject a link between to***co and lung cancer.[21]
Correlation is a valuable type of scientific evidence in fields such as medicine, psychology, and sociology. But first correlations must be confirmed as real, and then every possible causative relationship must be systematically explored. In the end correlation can be used as powerful evidence for a cause-and-effect relationship between a treatment and benefit, a risk factor and a disease, or a social or economic factor and various outcomes. But it is also one of the most abused types of evidence, because it is easy and even tempting to come to premature conclusions based upon the preliminary appearance of a correlation.
Correlations are used in Bell's theorem to disprove local causality.
Address
207 Regent Street, 3rd Floor
London
W1B3HH
Telephone
Website
Alerts
Be the first to know and let us send you an email when Weiss & Quast posts news and promotions. Your email address will not be used for any other purpose, and you can unsubscribe at any time.
Contact The Business
Send a message to Weiss & Quast:
