Pandas to_datetime() function – Explained with examples

Pandas is a powerful library in Python used for data manipulation and analysis. One of the essential tasks in data analysis is handling dates and times. Pandas provides a versatile function, to_datetime(), to convert various date and time formats into datetime objects, which are easier to manipulate and analyze. This blog post will explore the pandas.to_datetime() function, its usage, parameters, and practical examples.

Introduction to Pandas.to_datetime()

The pandas.to_datetime() function converts an argument to a datetime object. This argument can be a string, a list of strings, a Series, or even a DataFrame. By converting these formats to datetime objects, you can perform a variety of operations such as filtering data by date, calculating time differences, and resampling time series data.

Basic Usage

The basic syntax of the to_datetime() function is as follows:

import pandas as pd

pd.to_datetime(arg, errors='raise', format=None, exact=True, unit=None, infer_datetime_format=False, origin='unix', cache=True)

Let’s break down the parameters:

• arg: The data to be converted. It can be a string, list, Series, or DataFrame.
• errors: Specifies how to handle errors. It can take three values:
• 'raise' (default): Raises an exception if the conversion fails.
• 'coerce': Converts invalid parsing to NaT (Not a Time).
• 'ignore': Returns the input without raising an error.
• format: The strftime format to parse time. If not specified, the function will attempt to infer the format.
• exact: If True, only exact format matches are considered.
• unit: Specifies the unit of the argument if it is an integer. It can be 'D', 's', 'ms', 'us', or 'ns'.
• infer_datetime_format: If True, the function will attempt to infer the format of the datetime strings.
• origin: Defines the origin for the dates. It can be 'unix', 'julian', or a specific date.
• cache: If True, the function will use a cache of unique, converted dates to speed up parsing.

Examples

1. Converting a Single Date String

date_string = "2023-07-01"
date = pd.to_datetime(date_string)
print(date)

Output:

2023-07-01 00:00:00

The pd.to_datetime() function converts a single date string "2023-07-01" into a datetime object date, representing July 1, 2023 at midnight.

2. Converting a List of Date Strings

date_list = ["2023-07-01", "2023-07-02", "2023-07-03"]
dates = pd.to_datetime(date_list)
print(dates)

Output:

DatetimeIndex(['2023-07-01', '2023-07-02', '2023-07-03'], dtype='datetime64[ns]', freq=None)

Here, pd.to_datetime() converts a list ["2023-07-01", "2023-07-02", "2023-07-03"] into a DatetimeIndex dates, which is a specialized index structure in Pandas for datetime data.

3. Handling Different Date Formats

date_string = "01/07/2023"
date = pd.to_datetime(date_string, format="%d/%m/%Y")
print(date)

Output:

2023-07-01 00:00:00

By specifying the format="%d/%m/%Y", the function correctly interprets the date string "01/07/2023" as July 1, 2023, even though it’s in a different format.

4. Handling Errors

invalid_date = "not a date"
date = pd.to_datetime(invalid_date, errors='coerce')
print(date)

Output:

NaT

Setting errors='coerce' makes pd.to_datetime() convert an invalid date string "not a date" into NaT (Not a Time), a special Pandas datetime value indicating a missing or undefined date.

5. Working with Unix Timestamps

timestamp = 1688227200
date = pd.to_datetime(timestamp, unit='s')
print(date)

Output:

2023-07-01 00:00:00

When pd.to_datetime() is given a Unix timestamp 1688227200 and unit='s', it converts it into a datetime object date representing July 1, 2023 at midnight.

The parameter unit='s' in the pd.to_datetime() function specifies that the input is in seconds. When converting a numeric timestamp to a datetime object, this tells Pandas that the numbers represent Unix timestamps in seconds, ensuring accurate conversion to a datetime object. For example, 1688227200 with unit='s' converts to 2023-07-01 00:00:00.

6. Inferring Datetime Formats

date_strings = ["2023-07-01", "01/07/2023", "Jul 1, 2023"]
dates = pd.to_datetime(date_strings, infer_datetime_format=True, format='mixed')
print(dates)

Output:

DatetimeIndex(['2023-07-01', '2023-01-07', '2023-07-01'], dtype='datetime64[ns]', freq=None)

<ipython-input-27-23fb96798802>:2: UserWarning: The argument 'infer_datetime_format' is deprecated and will be removed in a future version. A strict version of it is now the default, see https://pandas.pydata.org/pdeps/0004-consistent-to-datetime-parsing.html. You can safely remove this argument.
  dates = pd.to_datetime(date_strings, infer_datetime_format=True, format='mixed')

By passing a list of date strings ["2023-07-01", "01/07/2023", "Jul 1, 2023"] with infer_datetime_format=True, Pandas infers the common datetime format and converts them into a DatetimeIndex dates with uniform datetime objects. The format='mixed' parameter allows Pandas to handle date strings in various formats.

As seen in the warning message, the infer_datetime_format argument is deprecated and will be removed in a future version of Pandas. According to the Pandas documentation, the strict version of datetime parsing is now the default. Therefore, you can safely remove the infer_datetime_format argument from your code.

Now the code can be corrected as,

date_strings = ["2023-07-01", "01/07/2023", "Jul 1, 2023"]
dates = pd.to_datetime(date_strings, format='mixed)
print(dates)

Output:

DatetimeIndex(['2023-07-01', '2023-01-07', '2023-07-01'], dtype='datetime64[ns]', freq=None)

The format='mixed' parameter in the pd.to_datetime function is used to indicate that the date strings in the input list might be in various formats, and Pandas should attempt to infer the format for each element individually. This is useful when dealing with a list of date strings that do not follow a consistent format.

Practical Applications

1. Filtering Data by Date

data = {'date': ['2023-07-01', '2023-07-02', '2023-07-03'], 'value': [10, 20, 30]}
df = pd.DataFrame(data)
df['date'] = pd.to_datetime(df['date'])
filtered_df = df[df['date'] > '2023-07-01']
print(filtered_df)

Output:

        date  value
1 2023-07-02     20
2 2023-07-03     30

In this example, a DataFrame df with a datetime column date is filtered to include only rows where the date is after July 1, 2023. This is achieved by using df[df['date'] > '2023-07-01'].

2. Calculating Time Differences

start_date = pd.to_datetime("2023-07-01")
end_date = pd.to_datetime("2023-07-10")
diff = end_date - start_date
print(diff)

Output:

9 days 00:00:00

By subtracting start_date from end_date, the code calculates the difference between two datetime objects, resulting in a timedelta object diff representing 9 days.

3. Resampling Time Series Data

date_rng = pd.date_range(start='2023-07-01', end='2023-07-10', freq='D')
df = pd.DataFrame(date_rng, columns=['date'])
df['data'] = np.random.randint(0, 100, size=(len(date_rng)))
df.set_index('date', inplace=True)
resampled_df = df.resample('2D').sum()
print(resampled_df)

Output:

            data
date            
2023-07-01   125
2023-07-03   160
2023-07-05   106
2023-07-07    88
2023-07-09   113

This snippet demonstrates how to generate a time series DataFrame df with random data, resample it to sum values over a 2-day frequency, and create a new resampled DataFrame resampled_df with aggregated data points.

Conclusion

The pandas.to_datetime() function is a powerful and flexible tool for handling date and time data in Pandas. Whether you need to convert a single date string, a list of dates, or handle different formats and errors, this function provides the necessary functionality. By mastering to_datetime(), you can efficiently manage and analyze temporal data, unlocking a wide range of possibilities in data analysis.

Feel free to experiment with the examples provided and explore the Pandas documentation for more advanced usage and tips!

Also Explore:

• Timestamp.now in Pandas – Explained with examples
• Pandas DataFrame iloc Property – Explained with Examples