Pandas DataFrame shift() method – Explained with examples

by

Pandas is a powerful and versatile library in Python for data manipulation and analysis. One of its many useful functions is the shift() method. In this blog, we will explore the shift() method in detail, providing a comprehensive explanation and various examples to illustrate its usage.

What is shift() Method?

The shift() method is used to shift the values in a DataFrame or Series by a specified number of periods. This can be particularly useful for creating lagged or leading datasets, which are common in time series analysis and other applications.

Syntax:

Python
DataFrame.shift(periods=1, freq=None, axis=0, fill_value=None)

Parameters:

  • periods: Number of periods to shift. Default is 1.
  • freq: DateOffset, timedelta, or time rule string (e.g., ‘EOM’). Only applicable to time series data.
  • axis: Shift direction (0 for rows, 1 for columns). Default is 0.
  • fill_value: Value to use for missing data after the shift. Default is NaN.
Basic Usage

Let’s start with a simple example to understand the basic usage of shift().

Python
import pandas as pd

# Creating a sample DataFrame
data = {'A': [1, 2, 3, 4, 5], 'B': [10, 20, 30, 40, 50]}
df = pd.DataFrame(data)

print("Original DataFrame:")
print(df)

# Shifting the DataFrame by 1 period
shifted_df = df.shift()

print("\nShifted DataFrame:")
print(shifted_df)

Output:

Markdown
Original DataFrame:
   A   B
0  1  10
1  2  20
2  3  30
3  4  40
4  5  50

Shifted DataFrame:
     A     B
0  NaN   NaN
1  1.0  10.0
2  2.0  20.0
3  3.0  30.0
4  4.0  40.0

In this example, each value is shifted down by one position, and the new missing values are filled with NaN.

Shifting Data Along Different Axes

You can also shift data along different axes. By default, shift() operates along the rows (axis=0), but you can shift along the columns (axis=1) as well.

Python
# Shifting the DataFrame along columns
shifted_df_axis1 = df.shift(axis=1)

print("\nShifted DataFrame along columns:")
print(shifted_df_axis1)

Output:

Markdown
Shifted DataFrame along columns:
   A   B
0 NaN  1
1 NaN  2
2 NaN  3
3 NaN  4
4 NaN  5
Using shift() with Different Periods

The periods parameter allows you to specify the number of periods to shift. You can shift forward or backward by setting a positive or negative value, respectively.

Python
# Shifting the DataFrame by 2 periods
shifted_df_2 = df.shift(periods=2)

print("\nShifted DataFrame by 2 periods:")
print(shifted_df_2)

# Shifting the DataFrame backward by 1 period
shifted_df_minus1 = df.shift(periods=-1)

print("\nShifted DataFrame backward by 1 period:")
print(shifted_df_minus1)

Output:

Markdown
Shifted DataFrame by 2 periods:
     A     B
0  NaN   NaN
1  NaN   NaN
2  1.0  10.0
3  2.0  20.0
4  3.0  30.0

Shifted DataFrame backward by 1 period:
   A   B
0  2  20
1  3  30
2  4  40
3  5  50
4 NaN NaN
Handling Missing Data

By default, shift() fills the new missing values with NaN. You can change this behavior using the fill_value parameter.

Python
# Shifting the DataFrame with a fill value
shifted_df_fill = df.shift(fill_value=0)

print("\nShifted DataFrame with fill value 0:")
print(shifted_df_fill)

Output:

Markdown
Shifted DataFrame with fill value 0:
   A   B
0  0   0
1  1  10
2  2  20
3  3  30
4  4  40

Real-World Examples

Example 1: Calculating Percentage Change

One common use of shift() is to calculate the percentage change between the current and previous values in a time series. This can be done by dividing the current value by the previous value (obtained using shift()) and then subtracting one.

Python
# Calculating percentage change using shift()
df['Previous'] = df['A'].shift(1)
df['Pct_Change'] = (df['A'] / df['Previous']) - 1

print("\nDataFrame with Percentage Change:")
print(df)

Output:

Markdown
DataFrame with Percentage Change:
   A   B  Previous  Pct_Change
0  1  10       NaN         NaN
1  2  20       1.0    1.000000
2  3  30       2.0    0.500000
3  4  40       3.0    0.333333
4  5  50       4.0    0.250000

In this example, the shift() method is used to create a new column Previous that contains the previous value of column A. The percentage change is then calculated as (current_value / previous_value) - 1.

Example 2: Creating Lagged Features

In time series forecasting, you might want to create lagged features to use as predictors.

Python
# Creating lagged features
df['Lag_1'] = df['A'].shift(1)
df['Lag_2'] = df['A'].shift(2)

print("\nDataFrame with Lagged Features:")
print(df)

Output:

Markdown
DataFrame with Lagged Features:
   A   B  Lag_1  Lag_2
0  1  10    NaN    NaN
1  2  20    1.0    NaN
2  3  30    2.0    1.0
3  4  40    3.0    2.0
4  5  50    4.0    3.0

In this example, we demonstrated how to create lagged features by shifting the values in a column by one and two periods, resulting in new columns Lag_1 and Lag_2. These lagged features can be used as predictors in time series forecasting models to help capture temporal dependencies.

Conclusion

The shift() method in Pandas is a versatile tool for shifting data within a DataFrame or Series. It is particularly useful for time series analysis, creating lagged features, and calculating changes over time. By understanding and utilizing the various parameters of shift(), you can effectively manipulate your data to suit your analysis needs.

By exploring the examples provided, you should now have a solid understanding of how to use the shift() method in Pandas. Whether you are working on time series data or need to create lagged datasets for predictive modeling, shift() is a method worth adding to your data manipulation toolkit.

Also Explore:

Leave a Comment