In the high stakes world of financial investing, superior information drives superior returns. And increasingly, traditional sources like financial statements, analyst reports and economic indicators just don‘t cut it anymore. The most sophisticated hedge funds and asset managers now rely on alternative data to consistently beat the market.
Alternative data refers to data derived from non-traditional sources outside of conventional financial metrics. This includes data scraped from the internet, satellite imagery, geolocation data, credit card transactions, social media sentiment and countless other creative data types.
According to a Greenwich Associates survey, 85% of financial institutions now utilize alternative data in their investment processes. It‘s become an indispensable tool to gain an informational trading edge in an industry where even milliseconds matter.
The Intensifying Data Arms Race in Finance
"Data and analytics are the lifeblood of our business," explained the CIO of a $50 billion hedge fund to Greenwich Associates. "It enables everything, it‘s the oxygen, it‘s critical."
This sentiment exemplifies just how high the stakes around data have become in finance. In Deloitte‘s 2022 Alternative Data Survey, 97% of investment management firms cited improved alpha generation as a key driver of alternative data adoption.
With tens of billions in annual profits on the line, financial institutions are locked in a constant data arms race. A 2022 report by Opimas LLC found top hedge funds planned to increase their data budgets by more than 30% annually.
To put some concrete numbers around the scale of these investments, back in 2020 the top 126 hedge funds globally were projected to spend over $2.5 billion on alternative data. This represents a nearly 20 fold increase from 2012.
According to Barry Hurewitz, Chief Data Officer at Morgan Stanley, "The ability to efficiently manage data is absolutely critical, it requires investment." Financial firms are pouring tens of millions into data science teams, analytics tools, data infrastructure and alternative data itself.
Those who fail to adapt risk losing their competitive edge. Matt Ober, an executive at FourSpring Capital said "I fear firms that rest on tradition and are unwilling to challenge their assumptions." The promise of alternative data is now too powerful for any firm to ignore.
The Evolution of Data in Finance
To understand today‘s alternative data gold rush, it helps to understand finance‘s historical data evolution:
Traditional Financial Data – For decades, investors relied almost entirely on financial statements, earnings reports, analyst forecasts, economic indicators and news. Very slow and limited data.
The Quant Revolution – In the 1980s and 90s, hedge funds began applying advanced analytics to traditional data to model markets algorithmically. This gave quants an edge.
Low Latency Data – Around 2010, funds began utilizing low latency feeds of market data and news for high speed algorithmic trading. Milliseconds provided an advantage.
Alternative Data Explosion – Now investors are tapping into immense alternative data sources like web scraped content, satellites, social media, location data and more for superior insights.
This evolution shows how competitive pressures cause financial firms to constantly seek out new data edges. What was once considered alternative quickly becomes tablestakes. Firms know that data mastery means profits.
The Myriad Uses and Benefits of Alternative Data
Alternative data opens up invaluable new insights across a diverse range of applications for financial institutions:
Investment Research – Hedge funds scrape internet data on a vast scale to mine early signals from industry sites, job postings, shipping data, news, forums and more to predict stock performance.
Business Forecasting – Granular analysis of smartphone GPS pings, satellite imagery and credit card spend provides much more accurate economic indicators to fine tune forecasts.
Fraud Detection – Behavioral data like web traffic patterns, browser fingerprints and network analytics provides vital signals to detect financial crimes early.
Credit Risk – Social media, web traffic, satellite imagery and other data on businesses provides additional signals on company health and credit risk beyond financials alone.
Sentiment Analysis – Scraping social media platforms like Twitter, StockTwits and Reddit provides invaluable data on consumer sentiment, brand perception and emerging trends.
Compliance – Web scraped data helps detect undisclosed connections, leaked documents and other signals to avoid regulatory fines for compliance failures.
Algorithmic Trading – Scraped alternative data gives algos an edge by quickly identifying trading opportunities from news, social media, weather and a myriad other sources.
The applications are constantly expanding as new alternative data sources are uncovered. According to Greenwich Associates, the top benefit cited by financial institutions is improved insights and decision making, selected by 85% of firms surveyed.
But turning all this raw data into actionable insights is no simple feat. "The big challenge is making alternative data usable for investment decisions," explained the CTO of a $60 billion investment firm. Let‘s explore some of the top alternative data types and collection techniques financial firms are leveraging to maximize value.
Leading Alternative Data Types Powering Smarter Investing
While the universe of alternative data grows more diverse each day, some key categories stand out as most valuable for financial institutions:
Web Scraped Content – The web is scraped at massive scale for early signals from sites like news, forums, reviews, jobs, shipping data, government databases and company sites before official reports come out.
Satellite Imagery – Leveraging satellites, drones and aerial footage provides invaluable economic indicators based on activity levels at factories, mines, ports, malls, parking lots and other business sites.
Geolocation Data – Analysis of anonymized and aggregated smartphone GPS pings reveals detailed foot traffic trends at retail locations, airports, entertainment venues and more to gauge business performance.
Credit Card Data – Credit transaction data offers unprecedented insight into consumer spending patterns and business revenue trends across sectors, geographies and demographics.
Social Media – Scraping platforms like Twitter, Reddit and StockTwits provides the pulse of consumer sentiment, brand perception, product trends and breaking news.
Web Traffic & Behavior – Granular web traffic and search data reveals clues about company prospects and economic activity through metrics like site visitors, time on site and scrolling behavior.
Survey & Market Research – Purchased access to aggregated market research and survey panel data provides additional lenses into consumer sentiment across industries.
Mobile App Data – Insights derived from mobile app usage patterns, downloads, engagement and transactions signals shifts in consumer behavior and business activity.
While most alternative data types can only be collected through purchase or licensing arrangements, the open web presents a vast opportunity for self-service scraping of public data. Let‘s explore the potential of web scraping in finance…
Web Scraping: An Essential Alternative Data Tool
Among alternative data collection techniques, web scraping stands out for its immense potential scale and scope of data extraction. Through web scraping, financial institutions can unlock a truly staggering breadth of valuable public data from across the internet.
Rather than wait for delayed and limited official reports, firms can scrape blogs, forums, news sites, reviews, job listings, shipping sites, social media, government databases and countless other sources to spot emerging trends in real-time.
According to Greenwich Associates, "Web scraping allows you to grab exponential amounts of data – it‘s a real advantage." It‘s no surprise their survey found 87% of North American financial institutions already utilize web scraping, a figure that continues to grow.
With an enterprise-grade web scraping solution, firms can build customized scrapers to efficiently collect millions of relevant data points on everything from warehouse job postings to shipping container volumes to mall foot traffic.
Of course, web scraping does entail challenges around compliance, bot detection, successfully parsing complex sites and managing large datasets. Navigating these hurdles requires thoughtful protocols and a robust, scalable solution.
The legal considerations around web scraping are also complex, making it critical that financial institutions follow proper guidelines like avoiding password protected sites and minimizing scrape volume to avoid overloading sites.
Additionally, sourcing data from reputable vendors with strict compliance frameworks provides an added layer of quality and risk mitigation. When executed properly, web scraping delivers immense alternative data value.
To provide a glimpse into the power of web scraping, let‘s explore a case study…
Scraping Supply Chain Data to Predict Retail Earnings
A hedge fund manager wanted to improve predictions around upcoming retail sector earnings reports. They hypothesized mining supply chain data could provide an advanced indicator of results.
Using a sophisticated web scraping platform, the fund built a customized scraper to collect weekly data on shipping volumes, fuel prices, import records, warehouse job postings, freight rates, traffic at ports and rail yards across thousands of sites.
By combining this alternative supply chain data with traditional retail sales figures, the fund identified correlating trends that allowed them to accurately predict earnings beats and misses days before official reports. This allowed them to take profitable positions ahead of stock price movements once the results went public.
The fund ultimately generated a 32% annualized return over two years using this strategy powered by scraped supply chain data. Web scraping opened up an invaluable alternative data source for superior predictive insights.
This example highlights in real terms how web scraping can drive better risk-adjusted returns – the holy grail for hedge funds.
Surging Ahead: The Future of Alternative Data in Finance
Given the clear informational and competitive advantages alternative data confers, financial institutions plan to continue aggressively adopting new data sources and enhancing their analytics capabilities.
According to Deloitte‘s 2022 Alternative Data Survey, a whopping 94% of capital markets firms plan to increase their alternative data budgets over the next 3 years. And Opimas projects total spending on alternative data by financial institutions to soar from $7 billion in 2024 to over $17 billion by 2030.
"The importance of alternative data is only going up," explained Victoria Chernova, Global Head of Alternative Data at Citi. "We expect exponential growth."
Looking ahead, financial services firms aim to combine traditional data with a growing universe of alternative data for unprecedented insights through sophisticated analytics.
Many also expect machine learning to help uncover novel alternative data sets and derive value from exponential data scale. "The big unlock is artificial intelligence," predicts Matthew Granade, co-founder of Domino Data Labs.
No firm can afford to fall behind in the intensifying alternative data arms race. As Kevin Koy, Head of Data Science at Balyasny Asset Management declared, "It‘s an absolute race 100% of the time." The firms that master alternative data will continue to reap the investment returns.
