Determining Data Availability
The only way to be certain whether data is available for a given set of parameters is to make a request to the API and see if it succeeds. If you get an empty data frame, try a less restrictive version of your request.
Working with Large Data Frames
Inspecting Data
imfp outputs data in pandas DataFrames, so you will want to use the pandas package for its functions for viewing and manipulating this object type.
For large datasets, you can use the pandas library’s info() method to get a quick summary of the data frame, including the number of rows and columns, the count of non-missing values, the column names, and the data types.
import imfp
import pandas as pd
# Set float format to 2 decimal places for pandas display output
pd.set_option('display.float_format', lambda x: '%.2f' % x)
df: pd.DataFrame = imfp.imf_dataset(
database_id="PCPS",
indicator=["PCOAL"],
data_transformation=["IX"]
)
# Quick summary of DataFrame
df.info()
/home/runner/work/imfp/imfp/imfp/data.py:257: UserWarning: ['IX'] not valid value(s) for data_transformation and will be ignored. Use imf_parameters('PCPS') to get valid parameters.
warn(
<class 'pandas.core.frame.DataFrame'>
RangeIndex: 1758 entries, 0 to 1757
Data columns (total 6 columns):
# Column Non-Null Count Dtype
--- ------ -------------- -----
0 country 1758 non-null object
1 indicator 1758 non-null object
2 data_transformation 1758 non-null object
3 frequency 1758 non-null object
4 time_period 1758 non-null object
5 obs_value 1758 non-null float64
dtypes: float64(1), object(5)
memory usage: 82.5+ KB
Alternatively, you can use the head() method to view the first 5 rows of the data frame.
# View first 5 rows of DataFrame
df.head()
|
country |
indicator |
data_transformation |
frequency |
time_period |
obs_value |
| 0 |
G001 |
PCOAL |
INDEX |
A |
1992 |
49.89 |
| 1 |
G001 |
PCOAL |
INDEX |
A |
1993 |
43.28 |
| 2 |
G001 |
PCOAL |
INDEX |
A |
1994 |
45.21 |
| 3 |
G001 |
PCOAL |
INDEX |
A |
1995 |
55.43 |
| 4 |
G001 |
PCOAL |
INDEX |
A |
1996 |
53.18 |
Cleaning Data
Numeric Conversion
All data is returned from the IMF API as a text (object) data type, so you will want to cast numeric columns to numeric.
# Numeric columns
numeric_cols = ["obs_value"]
# Cast numeric columns
df[numeric_cols] = df[numeric_cols].apply(pd.to_numeric)
Categorical Conversion
You can also convert string columns to categorical types for better memory usage.
# Convert categorical columns like ref_area and indicator to category type
categorical_cols = [
"frequency",
"country",
"indicator"
]
df[categorical_cols] = df[categorical_cols].astype("category")
NA Removal
After conversion, you may want to drop any rows with missing values.
# Drop rows with missing values
df = df.dropna()
Time Period Conversion
The time_period column can be more difficult to work with, because it may be differently formatted depending on the frequency of the data.
Annual data will be formatted as a four-digit year, such as “2000”, which can be trivially converted to numeric.
However, quarterly data will be formatted as “2000-Q1”, monthly data will be formatted like “2000-M01”, etc.
You can use the pandas library’s to_datetime() method with the format="mixed" argument to convert this column to a datetime object in a format-agnostic way:
# Convert time_period to datetime
df["datetime"] = pd.to_datetime(df["time_period"], format="mixed")
df[["frequency", "datetime"]].head()
|
frequency |
datetime |
| 0 |
A |
1992-01-01 |
| 1 |
A |
1993-01-01 |
| 2 |
A |
1994-01-01 |
| 3 |
A |
1995-01-01 |
| 4 |
A |
1996-01-01 |
Alternatively, you can split the time_period column into separate columns for year, quarter, and month, and then convert each to a numeric value:
# Split time_period into separate columns
df["year"] = df["time_period"].str.extract(r"(\d{4})")[0]
df["quarter"] = df["time_period"].str.extract(r"[Q](\d{1})")[0]
df["month"] = df["time_period"].str.extract(r"[M](\d{2})")[0]
# Convert year, quarter, and month to numeric
df["year"] = pd.to_numeric(df["year"])
df["quarter"] = pd.to_numeric(df["quarter"])
df["month"] = pd.to_numeric(df["month"])
# Return head for non-na months
df[["time_period", "year", "quarter", "month"]].dropna(subset=["month"]).head()
|
time_period |
year |
quarter |
month |
| 34 |
1992-M01 |
1992 |
NaN |
1.00 |
| 35 |
1992-M02 |
1992 |
NaN |
2.00 |
| 36 |
1992-M03 |
1992 |
NaN |
3.00 |
| 37 |
1992-M04 |
1992 |
NaN |
4.00 |
| 38 |
1992-M05 |
1992 |
NaN |
5.00 |
Summarizing Data
After converting columns to numeric, you can use the describe() function to get a quick summary of the statistical properties of these, including the count of rows, the mean, the standard deviation, the minimum and maximum values, and the quartiles.
# Statistical summary
df.describe()
|
obs_value |
datetime |
year |
quarter |
month |
| count |
1758.00 |
1758 |
1758.00 |
414.00 |
1242.00 |
| mean |
42.12 |
2008-10-29 00:36:56.165529088 |
2008.74 |
2.49 |
6.46 |
| min |
-69.21 |
1992-01-01 00:00:00 |
1992.00 |
1.00 |
1.00 |
| 25% |
-3.92 |
2000-01-01 00:00:09 |
2000.00 |
1.00 |
3.00 |
| 50% |
9.24 |
2009-01-01 00:00:02.500000 |
2009.00 |
2.00 |
6.00 |
| 75% |
66.03 |
2017-04-01 00:00:00 |
2017.00 |
3.00 |
9.00 |
| max |
577.58 |
2026-04-01 00:00:00 |
2026.00 |
4.00 |
12.00 |
| std |
77.29 |
NaN |
9.96 |
1.12 |
3.45 |
Viewing Data
For large data frames, it can be useful to view the data in a browser window. To facilitate this, you can define a View() function as follows. This function will save the data frame to a temporary HTML file and open it in your default web browser.
import tempfile
import webbrowser
# Define a simple function to view data frame in a browser window
def View(df: pd.DataFrame):
html = df.to_html()
with tempfile.NamedTemporaryFile('w',
delete=False, suffix='.html') as f:
url = 'file://' + f.name
f.write(html)
webbrowser.open(url)
# Call the function
View(df)