In my capstone project I explored the use of Natural Language Processing applied to Twitter in attempt to gain insight into stock price movement. Ultimately I found that there is use in this technique but like most things, it needs more exploration before it can be confidently deployed. I had success using both the TextBlob and Vader algorithms but when examining a series of individual Tweets and the scores that the algorithms gave them, I found significant room for improvement. Vader in particular is very good at determining sentiment on normal types of tweets because it is specifically tuned for social media. However when applied specifically to stock market related Tweets, it was quite lacking. The fact that I was able to get positive results given how wrong Vader often was is somewhat of a miracle and makes me even more inclined to dig deeper into this topic. Vader’s shortcomings fall into two main categories. The first is that there a large amount of words and phrases in finance which have no meaning in normal language that when applied to markets have very strong meanings. The second is that traders as a population are remarkably cynical, snarky and rely heavily on what can best be described as inside jokes.
The large amount of normally neutral words and phrases that have strong and specific market meaning makes sentiment analysis challenging. The words bull and bear are the most well known examples of this. They represent the animal spirits in the market that are constantly fighting with one another; bull being reference to increasing price and bear to decreasing price. Words like “easing” and “increased liquidity” are strong bullish words and their bearish counterparts “tightening” and “decreased liquidity” are referencing broad monetary conditions that affect all stocks. “Dove” and “hawk” are bullish and bearish references to the inclinations of policy makers and are used mostly in reference to central bankers but also to the federal budget. “Rotation” implies that something is being bought and something else is being sold; for example “tech rotating to banks” is bearish on technology stocks and bullish on banks. There is also language surrounding chart patterns such as “dip”, “rip” and “fade” that are given meaning by the words around them. “Long the dip” and “fade the rally” are bullish and bearish respectively. References to indexes and derivatives can have the exact opposite meaning; “VIX getting crushed” sounds very bearish but the VIX is an index that moves opposite the stock market and thus it is a bullish statement on stocks. To a normal algorithm “Apple P/C +10” likely means nothing or slightly positive because of the plus sign, however this is a very bearish reference to people making leveraged bets that Apple’s stock price will fall. Then there are references to interest rates that are challenging even for experienced traders such as “yield curve inversion” being bearish for stocks. These are just a few of the myriad of examples that make NLP very challenging to apply to market related tweets.
The other major challenge is the general personality type that comprises a large portion of market related social media. In order to make money in markets, you have to do something different than everyone else; which naturally draws people who are skeptical, suspicious and contrarian. Perusing a trading message board can often be like reading conspiracy theories in a foreign language to someone unfamiliar with markets. For example the phrases “Fed put” and “PPT” (plunge protection team) describes the suspected implicit promise of the US Central Bank to print a bunch of money and buy enormous amounts of stock in the event the market drops. “Waiting on the fed put” and “ppt is out” are both bullish but “fed put expired” and “ppt asleep” are both bearish. “QE4ever” is a non-sense phrase born out of a play on words that is extremely bullish on stock prices. There is also strong suspicion among tweeting traders that large institutions and central banks coordinate programmed trading algorithms to move the markets once they have placed large bets. This is often referred to as “machines”, “algos” and “bots” being at work. “Turn on the machines” and “algos gone wild” are bullish while “bots asleep” is bearish.
Reference to financial media outlets and personalities often have the opposite meaning of what they might seem. “CNBC at it again” and “Cramer brought his pom poms” refers to the corporate news network’s famous relentless cheerleading of stock prices no matter the situation and it is likely that the tweeter is bearish on the market. “Trump trying to get re-elected” is a bullish comment that the president will go to greater than normal lengths to ensure that stock prices stay high ahead of the 2020 election. Even more challenging are the coded references to a company’s executives can be difficult to parse even for experienced humans. “Cook is flustered” (Apple CEO Tim Cook) is negative sentiment on Apple but “diamond looking shiny” (JP Morgan Chase CEO Jamie Dimon) is positive sentiment on the bank.
The common thread among all of these example is that they have no obvious meaning to the non-financial person and thus almost certainly have no meaning to a language processing algorithm programmed by a non-financial person. After thinking about all of this I became more impressed and surprised that I saw positive results using out of the box sentiment analyzers, and also more convinced that a properly tuned NLP could yield significant results.