Build a dataset for your master thesis
A thesis lives or dies by its data. This guide shows where to get reliable finance & economics data — Kaggle, the ECB, FRED, the World Bank and more — how to pull each into Python, and how to assemble it into one clean, citable, reproducible dataset.
Before you download anything
Fit the question, not the hype
Pick the smallest dataset that can actually answer your hypothesis. Frequency (daily/monthly), span, and universe should follow your research design — not the other way round.
Mind point-in-time & look-ahead bias
Macro figures get revised and prices get adjusted. Use the data as it was known at decision time, and never let a future value leak into a past prediction.
Beware survivorship bias
Series that only contain firms/funds still alive today overstate returns. Prefer sources with delisted entities, or acknowledge the limitation explicitly.
Document provenance
Record the source, exact series ID, access date, and licence for every variable. Your supervisor (and reproducibility) will thank you.
Where to get the data
Free daily/intraday prices for equities, ETFs, FX, indices and crypto. The fastest way to get a price panel — but it is an unofficial feed, so document your pull date.
pip install yfinance · no API key
import yfinance as yf
px = yf.download(["AAPL", "MSFT", "^GSPC"],
start="2015-01-01", end="2024-12-31")["Close"]
returns = px.pct_change().dropna()800,000+ US & international macro series: rates, CPI, GDP, unemployment, spreads. The gold standard for macro controls in a thesis.
pip install fredapi · free API key
from fredapi import Fred
fred = Fred(api_key="YOUR_FREE_KEY")
cpi = fred.get_series("CPIAUCSL") # US CPI
fedfunds = fred.get_series("FEDFUNDS")
t10y = fred.get_series("DGS10") # 10Y Treasury yieldEuro-area official data: policy & market rates, HICP inflation, yield curves, reference exchange rates. SDMX REST API, no key — pull straight to a DataFrame as CSV.
requests / pandasdmx · no API key
import pandas as pd, requests, io
# Daily USD/EUR reference rate (series key EXR/D.USD.EUR.SP00.A)
url = "https://data-api.ecb.europa.eu/service/data/EXR/D.USD.EUR.SP00.A"
r = requests.get(url, params={"format": "csvdata"})
fx = pd.read_csv(io.StringIO(r.text), parse_dates=["TIME_PERIOD"])Thousands of ready-made finance / crypto / alternative datasets. Great for a quick start — but vet provenance & licence before citing one in a thesis.
pip install kaggle · token (kaggle.json) in ~/.kaggle/
# CLI (after placing kaggle.json in ~/.kaggle/)
kaggle datasets download -d camnugent/sandp500 --unzip
# or in Python, no manual unzip:
import kagglehub
path = kagglehub.dataset_download("camnugent/sandp500")Cross-country development & macro indicators (GDP, inflation, population, trade) for 200+ economies — ideal for panel / cross-sectional designs.
pip install wbgapi · no API key
import wbgapi as wb
# GDP growth (annual %) for a country panel
gdp = wb.data.DataFrame("NY.GDP.MKTP.KD.ZG",
["FRA", "DEU", "GBR"], time=range(2000, 2024))Official EU statistics: HICP inflation, unemployment, national accounts, trade. Harmonised across member states — clean for EU-focused theses.
pip install eurostat · no API key
import eurostat
# HICP — annual rate of change (inflation)
hicp = eurostat.get_data_df("prc_hicp_manr")US company filings & standardized XBRL financials (revenue, assets, EPS) straight from the regulator. Free — but you MUST send a descriptive User-Agent header.
requests · no key (User-Agent required)
import requests
headers = {"User-Agent": "Your Name your.email@uni.edu"}
# Apple (CIK 320193) revenue concept
url = ("https://data.sec.gov/api/xbrl/companyconcept/"
"CIK0000320193/us-gaap/Revenues.json")
facts = requests.get(url, headers=headers).json()Free crypto prices, market caps & volume across thousands of coins and fiat currencies — no key for the public API.
requests · free public API
import requests, pandas as pd
r = requests.get("https://api.coingecko.com/api/v3/coins/bitcoin/market_chart",
params={"vs_currency": "eur", "days": "365"})
prices = pd.DataFrame(r.json()["prices"], columns=["ts", "price"])
prices["date"] = pd.to_datetime(prices["ts"], unit="ms")Also worth knowing: Nasdaq Data Link (formerly Quandl), OECD, IMF and BIS for more macro; Stooq for free EOD prices; and Refinitiv/Bloomberg terminals — often available through your university library — for premium data you usually cannot redistribute.
Assemble one clean dataset
The hard part is rarely getting the data — it is merging sources of different frequencies into one tidy table without introducing bias. The pattern: collect → align frequency → as-of merge → clean → save.
import pandas as pd, yfinance as yf
from fredapi import Fred
# 1) Collect — a price series and a macro control
px = yf.download("^GSPC", start="2010-01-01")["Close"].rename("sp500")
cpi = Fred(api_key=KEY).get_series("CPIAUCSL").rename("cpi")
# 2) Align frequency — month-end, then returns
m = px.resample("ME").last().to_frame()
m["ret"] = m["sp500"].pct_change()
# 3) As-of merge — uses the LAST macro value known at each date
# (backward direction => no look-ahead bias)
df = pd.merge_asof(m.sort_index(),
cpi.to_frame().sort_index(),
left_index=True, right_index=True,
direction="backward")
# 4) Clean & persist (keep raw and processed separate)
df = df.dropna()
df.to_parquet("data/processed/thesis_panel.parquet")Make it reproducible & citable
Never overwrite raw data. Keep data/raw/ (exactly as downloaded) and data/processed/ (your cleaned output) separate, and regenerate processed from a script.
Pin your environment. Freeze versions with requirements.txt or environment.yml so results reproduce a year from now.
Write a data dictionary. One row per variable: name, source, exact series ID, units, frequency, transform, and the date you accessed it.
Cite every source & check the licence. FRED, ECB, World Bank and Eurostat are free to use with attribution; many Kaggle datasets carry their own licence; terminal data (Bloomberg/Refinitiv) usually cannot be shared. State each in your appendix.
Cite or link this page
Lecturers and students: this page is free to link from syllabi, course notes and theses — the URL is stable.
APA
Forler, S. M. (2026). Build a Thesis Dataset — where to get finance & economics data and how. EcoFinLearn. https://ecofinlearning.com/thesis-data
BibTeX
@misc{ecofinlearn_thesis_data_2026,
author = {Forler, Sitraka M.},
title = {Build a Thesis Dataset — where to get finance & economics data and how},
year = {2026},
publisher = {EcoFinLearn},
howpublished = {\url{https://ecofinlearning.com/thesis-data}},
note = {Free interactive learning platform}
}Pull your data with the Python & pandas skills from the tracks, then explore it in the SQL lessons.
Back to your courses