Build a dataset for your master thesis
A thesis lives or dies by its data. This guide shows where to get reliable finance & economics data - Kaggle, the ECB, FRED, the World Bank and more - how to pull each into Python, and how to assemble it into one clean, citable, reproducible dataset.
Before you download anything
Fit the question, not the hype
Pick the smallest dataset that can actually answer your hypothesis. Frequency (daily/monthly), span, and universe should follow your research design - not the other way round.
Mind point-in-time & look-ahead bias
Macro figures get revised and prices get adjusted. Use the data as it was known at decision time, and never let a future value leak into a past prediction.
Beware survivorship bias
Series that only contain firms/funds still alive today overstate returns. Prefer sources with delisted entities, or acknowledge the limitation explicitly.
Document provenance
Record the source, exact series ID, access date, and licence for every variable. Your supervisor (and reproducibility) will thank you.
Where to get the data
Free daily/intraday prices for equities, ETFs, FX, indices and crypto. The fastest way to get a price panel - but it is an unofficial feed, so document your pull date.
pip install yfinance · no API key
import yfinance as yf
px = yf.download(["AAPL", "MSFT", "^GSPC"],
start="2015-01-01", end="2024-12-31")["Close"]
returns = px.pct_change().dropna()800,000+ US & international macro series: rates, CPI, GDP, unemployment, spreads. The gold standard for macro controls in a thesis.
pip install fredapi · free API key
from fredapi import Fred
fred = Fred(api_key="YOUR_FREE_KEY")
cpi = fred.get_series("CPIAUCSL") # US CPI
fedfunds = fred.get_series("FEDFUNDS")
t10y = fred.get_series("DGS10") # 10Y Treasury yieldEuro-area official data: policy & market rates, HICP inflation, yield curves, reference exchange rates. SDMX REST API, no key - pull straight to a DataFrame as CSV.
requests / pandasdmx · no API key
import pandas as pd, requests, io
# Daily USD/EUR reference rate (series key EXR/D.USD.EUR.SP00.A)
url = "https://data-api.ecb.europa.eu/service/data/EXR/D.USD.EUR.SP00.A"
r = requests.get(url, params={"format": "csvdata"})
fx = pd.read_csv(io.StringIO(r.text), parse_dates=["TIME_PERIOD"])Thousands of ready-made finance / crypto / alternative datasets. Great for a quick start - but vet provenance & licence before citing one in a thesis.
pip install kaggle · token (kaggle.json) in ~/.kaggle/
# CLI (after placing kaggle.json in ~/.kaggle/)
kaggle datasets download -d camnugent/sandp500 --unzip
# or in Python, no manual unzip:
import kagglehub
path = kagglehub.dataset_download("camnugent/sandp500")Cross-country development & macro indicators (GDP, inflation, population, trade) for 200+ economies - ideal for panel / cross-sectional designs.
pip install wbgapi · no API key
import wbgapi as wb
# GDP growth (annual %) for a country panel
gdp = wb.data.DataFrame("NY.GDP.MKTP.KD.ZG",
["FRA", "DEU", "GBR"], time=range(2000, 2024))Official EU statistics: HICP inflation, unemployment, national accounts, trade. Harmonised across member states - clean for EU-focused theses. Dataset codes change when Eurostat reclassifies (HICP moved to prc_hicp_minr in 2026), so check the latest period before relying on a series.
pip install eurostat · no API key
import eurostat
# HICP annual inflation, ECOICOP ver.2 (from January 2026 data).
# The older monthly HICP tables are archived and stop at 2025-12.
hicp = eurostat.get_data_df("prc_hicp_minr", filter_pars={
"unit": "RCH_A", "coicop18": "TOTAL",
"geo": ["EA21", "FR"], "startPeriod": "2015-01"})
# Wide table: one column per month.
# EA21 = the 21-country euro area; "EA" follows changing membership.US company filings & standardized XBRL financials (revenue, assets, EPS) straight from the regulator. Free - but you MUST send a descriptive User-Agent header. Firms change XBRL tags over time: data.sec.gov/api/xbrl/companyfacts/CIK##########.json lists every concept a firm files, so check it before picking one.
requests + pandas · no key (User-Agent required)
import requests, pandas as pd
headers = {"User-Agent": "Your Name your.email@uni.edu"}
# Apple (CIK 320193) revenue. Apple's "Revenues" tag stops at FY2018;
# later filings use this concept (companyfacts lists every tag).
url = ("https://data.sec.gov/api/xbrl/companyconcept/CIK0000320193/"
"us-gaap/RevenueFromContractWithCustomerExcludingAssessedTax.json")
rev = pd.DataFrame(requests.get(url, headers=headers).json()["units"]["USD"])
# frame CY2025 = the fiscal year closest to calendar 2025 (Sep FY end)
annual = rev[rev["frame"].str.fullmatch(r"CY\d{4}", na=False)]Free crypto prices, market caps & volume across thousands of coins and fiat currencies - no key for the public API.
requests · free public API
import requests, pandas as pd
r = requests.get("https://api.coingecko.com/api/v3/coins/bitcoin/market_chart",
params={"vs_currency": "eur", "days": "365"})
prices = pd.DataFrame(r.json()["prices"], columns=["ts", "price"])
prices["date"] = pd.to_datetime(prices["ts"], unit="ms")Also worth knowing: Nasdaq Data Link (formerly Quandl), OECD, IMF and BIS for more macro; Stooq for free EOD prices; and Refinitiv/Bloomberg terminals - often available through your university library - for premium data you usually cannot redistribute.
Assemble one clean dataset
The hard part is rarely getting the data - it is merging sources of different frequencies into one tidy table without introducing bias. The pattern: collect → align frequency → as-of merge → clean → save.
import pandas as pd, yfinance as yf
from fredapi import Fred
# 1) Collect - a price series and a macro control
px = yf.download("^GSPC", start="2010-01-01")["Close"].rename("sp500")
cpi = Fred(api_key=KEY).get_series("CPIAUCSL").rename("cpi")
# 2) Align frequency - month-end, then returns
m = px.resample("ME").last().to_frame()
m["ret"] = m["sp500"].pct_change()
# 3) As-of merge - uses the LAST macro value known at each date
# (backward direction => no look-ahead bias)
df = pd.merge_asof(m.sort_index(),
cpi.to_frame().sort_index(),
left_index=True, right_index=True,
direction="backward")
# 4) Clean & persist (keep raw and processed separate)
df = df.dropna()
df.to_parquet("data/processed/thesis_panel.parquet")Make it reproducible & citable
Never overwrite raw data. Keep data/raw/ (exactly as downloaded) and data/processed/ (your cleaned output) separate, and regenerate processed from a script.
Pin your environment. Freeze versions with requirements.txt or environment.yml so results reproduce a year from now.
Write a data dictionary. One row per variable: name, source, exact series ID, units, frequency, transform, and the date you accessed it.
Cite every source & check the licence. FRED, ECB, World Bank and Eurostat are free to use with attribution; many Kaggle datasets carry their own licence; terminal data (Bloomberg/Refinitiv) usually cannot be shared. State each in your appendix.
Cite or link this page
Lecturers and students: this page is free to link from syllabi, course notes and theses - the URL is stable.
APA
Forler, S. M. (2026). Build a Thesis Dataset - where to get finance & economics data and how. EcoFinLearn. https://ecofinlearning.com/thesis-data
BibTeX
@misc{ecofinlearn_thesis_data_2026,
author = {Forler, Sitraka M.},
title = {Build a Thesis Dataset - where to get finance & economics data and how},
year = {2026},
publisher = {EcoFinLearn},
howpublished = {\url{https://ecofinlearning.com/thesis-data}},
note = {Free interactive learning platform}
}Learn the skills on this page
Each step of the workflow above has a free lesson, with code that runs in your browser.
- Align frequencyResample Daily Prices to Monthly ReturnsData Manipulation
- Merge sourcesMerging Price & Fundamentals DataData Manipulation
- Avoid look-ahead biasPoint-in-Time Macro JoinAdvanced Finance SQL
- CleanCleaning Messy Financial DataData Manipulation
- Query it in SQLSQL Date Filtering & Monthly ReturnsAdvanced Finance SQL