Suggestions for Usage

Determining Data Availability

Unfortunately, many of the indicators listed as available in the lists of input codes returned by imfp.imf_parameters() are not actually available. This is a deficiency of the API rather than the library; someone at the IMF presumably intended to provide these indicators at some point, but never got around to it.

The only way to be certain whether an indicator is available is to make a request to the API and see if it succeeds. If not, you will receive an error message indicating that no data was found for your parameters. In general, if you see this message, you should try making a less restrictive version of your request. For instance, if your request returns no data for an indicator for a given country and time period, you can omit the country or time period parameter and try again. If you still get no data, that indicator is not actually available through the API.

While it is not fully predictable which indicators will be available, as a general rule you can expect to get unadjusted series but not adjusted ones. For instance, real and per capita GDP are not available (although they are listed) through the API, but nominal GDP is. The API does, however, make available all the adjustment variables you would need to adjust the data yourself. See the Common Data Transformations section below for examples of how to make adjustments.

Working with Large Data Frames

Inspecting Data

imfp outputs data in pandas DataFrames, so you will want to use the pandas package for its functions for viewing and manipulating this object type.

For large datasets, you can use the pandas library’s info() method to get a quick summary of the data frame, including the number of rows and columns, the count of non-missing values, the column names, and the data types.

import imfp
import pandas as pd

df: pd.DataFrame = imfp.imf_dataset(
    database_id="PCPS",
    commodity=["PCOAL"],
    unit_measure=["IX"],
    start_year=2000, end_year=2001
)

# Quick summary of DataFrame
print(
    df.info()
)

<class 'pandas.core.frame.DataFrame'>
Index: 34 entries, 0 to 23
Data columns (total 8 columns):
 #   Column        Non-Null Count  Dtype 
---  ------        --------------  ----- 
 0   freq          34 non-null     object
 1   ref_area      34 non-null     object
 2   commodity     34 non-null     object
 3   unit_measure  34 non-null     object
 4   unit_mult     34 non-null     object
 5   time_format   34 non-null     object
 6   time_period   34 non-null     object
 7   obs_value     34 non-null     object
dtypes: object(8)
memory usage: 2.4+ KB
None

Alternatively, you can use the head() method to view the first 5 rows of the data frame.

# View first 5 rows of DataFrame
print(
    df.head()
)

  freq ref_area commodity unit_measure unit_mult time_format time_period  \
0    A      W00     PCOAL           IX         0         P1Y        2000   
1    A      W00     PCOAL           IX         0         P1Y        2001   
0    Q      W00     PCOAL           IX         0         P3M     2000-Q1   
1    Q      W00     PCOAL           IX         0         P3M     2000-Q2   
2    Q      W00     PCOAL           IX         0         P3M     2000-Q3   

          obs_value  
0  39.3510230293202  
1  49.3378587284039  
0  36.2253202528364  
1  38.3233649120962  
2  39.5896503864216

Cleaning Data

Numeric Conversion

All data is returned from the IMF API as a text (object) data type, so you will want to cast numeric columns to numeric.

# Numeric columns
numeric_cols = ["unit_mult", "obs_value"]

# Cast numeric columns
df[numeric_cols] = df[numeric_cols].apply(pd.to_numeric)

Categorical Conversion

You can also convert string columns to categorical types for better memory usage.

# Convert categorical columns like ref_area and indicator to category type
categorical_cols = [
  "freq",
  "ref_area",
  "commodity",
  "unit_measure",
  "time_format"
]

df[categorical_cols] = df[categorical_cols].astype("category")

NA Removal

After conversion, you may want to drop any rows with missing values.

# Drop rows with missing values
df = df.dropna()

Time Period Conversion

The time_period column can be more difficult to work with, because it may be differently formatted depending on the frequency of the data.

Annual data will be formatted as a four-digit year, such as “2000”, which can be trivially converted to numeric.

However, quarterly data will be formatted as “2000-Q1”, and monthly data will be formatted like “2000-01”.

You can use the pandas library’s to_datetime() method with the format="mixed" argument to convert this column to a datetime object in a format-agnostic way:

# Convert time_period to datetime
df["datetime"] = pd.to_datetime(df["time_period"], format="mixed")
print(df[["freq", "datetime"]].head())

  freq   datetime
0    A 2000-01-01
1    A 2001-01-01
0    Q 2000-01-01
1    Q 2000-04-01
2    Q 2000-07-01

Alternatively, you can split the time_period column into separate columns for year, quarter, and month, and then convert each to a numeric value:

# Split time_period into separate columns
df["year"] = df["time_period"].str.extract(r"(\d{4})")[0]
df["quarter"] = df["time_period"].str.extract(r"Q(\d{1})")[0]
df["month"] = df["time_period"].str.extract(r"-(\d{2})")[0]

# Convert year, quarter, and month to numeric
df["year"] = pd.to_numeric(df["year"])
df["quarter"] = pd.to_numeric(df["quarter"])
df["month"] = pd.to_numeric(df["month"])

print(df[["time_period", "year", "quarter", "month"]].head(5))

  time_period  year  quarter  month
0        2000  2000      NaN    NaN
1        2001  2001      NaN    NaN
0     2000-Q1  2000      1.0    NaN
1     2000-Q2  2000      2.0    NaN
2     2000-Q3  2000      3.0    NaN

Summarizing Data

After converting columns to numeric, you can use the describe() function to get a quick summary of the statistical properties of these, including the count of rows, the mean, the standard deviation, the minimum and maximum values, and the quartiles.

# Statistical summary
df.describe()

	unit_mult	obs_value	datetime	year	quarter	month
count	34.0	34.000000	34	34.000000	8.000000	24.000000
mean	0.0	44.344441	2000-11-28 21:52:56.470588288	2000.500000	2.500000	6.500000
min	0.0	35.588859	2000-01-01 00:00:00	2000.000000	1.000000	1.000000
25%	0.0	39.221034	2000-06-08 12:00:00	2000.000000	1.750000	3.750000
50%	0.0	44.345342	2000-12-16 12:00:00	2000.500000	2.500000	6.500000
75%	0.0	50.108524	2001-05-24 06:00:00	2001.000000	3.250000	9.250000
max	0.0	51.478057	2001-12-01 00:00:00	2001.000000	4.000000	12.000000
std	0.0	5.757895	NaN	0.507519	1.195229	3.526299

Viewing Data

For large data frames, it can be useful to view the data in a browser window. To facilitate this, you can define a View() function as follows. This function will save the data frame to a temporary HTML file and open it in your default web browser.

import tempfile
import webbrowser

# Define a simple function to view data frame in a browser window
def View(df: pd.DataFrame):
    html = df.to_html()
    with tempfile.NamedTemporaryFile('w', 
    delete=False, suffix='.html') as f:
        url = 'file://' + f.name
        f.write(html)
    webbrowser.open(url)

# Call the function
View(df)

Common Data Transformations

The International Financial Statistics (IFS) database provides key macroeconomic aggregates that are frequently needed when working with other IMF datasets. Here, we’ll demonstrate how to use three fundamental indicators—GDP, price deflators, and population statistics—to transform your data.

These transformations are essential for:

Converting nominal to real dollar values
Calculating per capita metrics
Harmonizing data across different frequencies
Adjusting for different unit scales

For a complete, end-to-end example of these transformations in a real analysis workflow, see Jenny Xu’s superb demo notebook.

Fetching IFS Adjusters

First, let’s retrieve the key adjustment variables from the IFS database:

# Fetch GDP Deflator (Index, Annual)
deflator = imfp.imf_dataset(
    database_id="IFS",
    indicator="NGDP_D_SA_IX",  # GDP deflator index
    freq="Q",
    start_year=2010
)

# Fetch Population Estimates (Annual)
population = imfp.imf_dataset(
    database_id="IFS",
    indicator="LP_PE_NUM",
    freq="A",
    start_year=2010
)

We’ll also retireve a nominal GDP series to be adjusted:

# Fetch Nominal GDP (Domestic currency, annual)
nominal_gdp = imfp.imf_dataset(
    database_id="IFS", 
    indicator="NGDP_XDC",
    freq="A",
    start_year=2010
)

Key IFS Indicators:

NGDP_D_SA_IX: GDP deflator index (seasonally adjusted)
LP_PE_NUM: Population estimates
NGDP_XDC: Nominal GDP in domestic currency

Unit Multiplier Adjustment

IMF data often includes a unit_mult column that indicates the scale of the values (e.g., millions, billions). We can write a helper function to apply these scaling factors:

def apply_unit_multiplier(df):
    """Convert to numeric, adjust values using IMF's scaling factors, and drop
    missing values"""
    df['obs_value'] = pd.to_numeric(df['obs_value'])
    df['unit_mult'] = pd.to_numeric(df['unit_mult'])
    df['adjusted_value'] = df['obs_value'] * 10 ** df['unit_mult']
    df = df.dropna()
    return df

# Apply to each dataset
deflator = apply_unit_multiplier(deflator)
population = apply_unit_multiplier(population)
nominal_gdp = apply_unit_multiplier(nominal_gdp)

Harmonizing Frequencies

When working with data of different frequencies, you’ll often need to harmonize them. For example, GDP data is typically reported quarterly, while some other indicators may be annual. There are two common approaches:

Using Q4 values: This approach is often used for stock variables (measurements taken at a point in time) and when you want to align with end-of-year values:

# Keep only Q4 observations for annual comparisons
deflator = deflator[deflator['time_period'].str.contains("Q4")]

# Extract just the year from the time period for Q4 data
deflator['time_period'] = deflator['time_period'].str[:4]

Calculating annual averages: This approach is more appropriate for flow variables (measurements over a period) and when you want to smooth out seasonal variations:

# Alternative: Calculate annual averages
deflator = deflator.groupby(
    ['ref_area', deflator['time_period']], 
    as_index=False
).agg({
    'obs_value': 'mean'
})

Choose the appropriate method based on your specific analysis needs and the economic meaning of your variables.

Merging Datasets

After harmonizing unit scales, we can combine the datasets using pd.DataFrame.merge() with ref_area and time_period as keys:

merged = (
    nominal_gdp.merge(
        deflator,
        on=['ref_area', 'time_period'],
        suffixes=('_gdp', '_deflator')
    )
    .merge(
        population,
        on=['ref_area', 'time_period']
    )
)

Calculating Real Values

With the merged dataset, we can now calculate real GDP and per capita values:

# Convert nominal to real GDP
merged['real_gdp'] = (
    (merged['obs_value_gdp'] / merged['obs_value_deflator']) * 100
)

# Calculate per capita values (using population obs_value)
merged['real_gdp_per_capita'] = merged['real_gdp'] / merged['obs_value']

# Display the first 5 rows of the transformed data
print(
    merged[['time_period', 'real_gdp', 'real_gdp_per_capita']].head(5)
)

  time_period       real_gdp  real_gdp_per_capita
0        2010  186427.433557            34.743745
1        2011  191151.403936            35.463845
2        2012  188340.077895            34.782662
3        2013  186666.274400            34.320063
4        2014  186231.666379            34.099558