python - Converting string into datetime

Question

I've got a huge list of date-times like this as strings:

Jun 1 2005  1:33PM
Aug 28 1999 12:00AM

I'm going to be shoving these back into proper datetime fields in a database so I need to magic them into real datetime objects.

This is going through Django's ORM so I can't use SQL to do the conversion on insert.

Answer 1

datetime.strptime is the main routine for parsing strings into datetimes. It can handle all sorts of formats, with the format determined by a format string you give it:

from datetime import datetime

datetime_object = datetime.strptime('Jun 1 2005  1:33PM', '%b %d %Y %I:%M%p')

The resulting datetime object is timezone-naive.

Links:

• Python documentation for strptime: Python 2, Python 3

• Python documentation for strptime/strftime format strings: Python 2, Python 3

• strftime.org is also a really nice reference for strftime

Notes:

• strptime = "string parse time"
• strftime = "string format time"
• Pronounce it out loud today & you won't have to search for it again in 6 months.

Answer 2

Use the third party dateutil library:

from dateutil import parser
parser.parse("Aug 28 1999 12:00AM")  # datetime.datetime(1999, 8, 28, 0, 0)

It can handle most date formats, including the one you need to parse. It's more convenient than strptime as it can guess the correct format most of the time.

It's very useful for writing tests, where readability is more important than performance.

You can install it with:

pip install python-dateutil

Answer 3

Check out strptime in the time module. It is the inverse of strftime.

$ python
>>> import time
>>> my_time = time.strptime('Jun 1 2005  1:33PM', '%b %d %Y %I:%M%p')
time.struct_time(tm_year=2005, tm_mon=6, tm_mday=1,
                 tm_hour=13, tm_min=33, tm_sec=0,
                 tm_wday=2, tm_yday=152, tm_isdst=-1)

timestamp = time.mktime(my_time)
# convert time object to datetime
from datetime import datetime
my_datetime = datetime.fromtimestamp(timestamp)
# convert time object to date
from datetime import date
my_date = date.fromtimestamp(timestamp)

Answer 4

I have put together a project that can convert some really neat expressions. Check out timestring.

Here are some examples below:

pip install timestring

>>> import timestring
>>> timestring.Date('monday, aug 15th 2015 at 8:40 pm')
<timestring.Date 2015-08-15 20:40:00 4491909392>
>>> timestring.Date('monday, aug 15th 2015 at 8:40 pm').date
datetime.datetime(2015, 8, 15, 20, 40)
>>> timestring.Range('next week')
<timestring.Range From 03/10/14 00:00:00 to 03/03/14 00:00:00 4496004880>
>>> (timestring.Range('next week').start.date, timestring.Range('next week').end.date)
(datetime.datetime(2014, 3, 10, 0, 0), datetime.datetime(2014, 3, 14, 0, 0))

Answer 5

python >= 3.7

to convert YYYY-MM-DD string to datetime object, datetime.fromisoformat could be used.

from datetime import datetime

date_string = "2012-12-12 10:10:10"
print (datetime.fromisoformat(date_string))
2012-12-12 10:10:10

Answer 6

Remember this and you didn't need to get confused in datetime conversion again.

String to datetime object = strptime

datetime object to other formats = strftime

Jun 1 2005 1:33PM

is equals to

%b %d %Y %I:%M%p

%b Month as locale’s abbreviated name(Jun)

%d Day of the month as a zero-padded decimal number(1)

%Y Year with century as a decimal number(2015)

%I Hour (12-hour clock) as a zero-padded decimal number(01)

%M Minute as a zero-padded decimal number(33)

%p Locale’s equivalent of either AM or PM(PM)

so you need strptime i-e converting string to

>>> dates = []
>>> dates.append('Jun 1 2005  1:33PM')
>>> dates.append('Aug 28 1999 12:00AM')
>>> from datetime import datetime
>>> for d in dates:
...     date = datetime.strptime(d, '%b %d %Y %I:%M%p')
...     print type(date)
...     print date
... 

Output

<type 'datetime.datetime'>
2005-06-01 13:33:00
<type 'datetime.datetime'>
1999-08-28 00:00:00

What if you have different format of dates you can use panda or dateutil.parse

>>> import dateutil
>>> dates = []
>>> dates.append('12 1 2017')
>>> dates.append('1 1 2017')
>>> dates.append('1 12 2017')
>>> dates.append('June 1 2017 1:30:00AM')
>>> [parser.parse(x) for x in dates]

OutPut

[datetime.datetime(2017, 12, 1, 0, 0), datetime.datetime(2017, 1, 1, 0, 0), datetime.datetime(2017, 1, 12, 0, 0), datetime.datetime(2017, 6, 1, 1, 30)]

Answer 7

Many timestamps have an implied timezone. To ensure that your code will work in every timezone, you should use UTC internally and attach a timezone each time a foreign object enters the system.

Python 3.2+:

>>> datetime.datetime.strptime(
...     "March 5, 2014, 20:13:50", "%B %d, %Y, %H:%M:%S"
... ).replace(tzinfo=datetime.timezone(datetime.timedelta(hours=-3)))

Answer 8

Here are two solutions using Pandas to convert dates formatted as strings into datetime.date objects.

import pandas as pd

dates = ['2015-12-25', '2015-12-26']

# 1) Use a list comprehension.
>>> [d.date() for d in pd.to_datetime(dates)]
[datetime.date(2015, 12, 25), datetime.date(2015, 12, 26)]

# 2) Convert the dates to a DatetimeIndex and extract the python dates.
>>> pd.DatetimeIndex(dates).date.tolist()
[datetime.date(2015, 12, 25), datetime.date(2015, 12, 26)]

Timings

dates = pd.DatetimeIndex(start='2000-1-1', end='2010-1-1', freq='d').date.tolist()

>>> %timeit [d.date() for d in pd.to_datetime(dates)]
# 100 loops, best of 3: 3.11 ms per loop

>>> %timeit pd.DatetimeIndex(dates).date.tolist()
# 100 loops, best of 3: 6.85 ms per loop

And here is how to convert the OP's original date-time examples:

datetimes = ['Jun 1 2005  1:33PM', 'Aug 28 1999 12:00AM']

>>> pd.to_datetime(datetimes).to_pydatetime().tolist()
[datetime.datetime(2005, 6, 1, 13, 33), 
 datetime.datetime(1999, 8, 28, 0, 0)]

There are many options for converting from the strings to Pandas Timestamps using to_datetime, so check the docs if you need anything special.

Likewise, Timestamps have many properties and methods that can be accessed in addition to .date

Answer 9

I personally like the solution using the parser module, which is the second Answer to this question and is beautiful, as you don't have to construct any string literals to get it working. BUT, one downside is that it is 90% slower than the accepted answer with strptime.

from dateutil import parser
from datetime import datetime
import timeit

def dt():
    dt = parser.parse("Jun 1 2005  1:33PM")
def strptime():
    datetime_object = datetime.strptime('Jun 1 2005  1:33PM', '%b %d %Y %I:%M%p')

print(timeit.timeit(stmt=dt, number=10**5))
print(timeit.timeit(stmt=strptime, number=10**5))
>10.70296801342902
>1.3627995655316933

As long as you are not doing this a million times over and over again, I still think the parser method is more convenient and will handle most of the time formats automatically.

Answer 10

Something that isn't mentioned here and is useful: adding a suffix to the day. I decoupled the suffix logic so you can use it for any number you like, not just dates.

import time

def num_suffix(n):
    '''
    Returns the suffix for any given int
    '''
    suf = ('th','st', 'nd', 'rd')
    n = abs(n) # wise guy
    tens = int(str(n)[-2:])
    units = n % 10
    if tens > 10 and tens < 20:
        return suf[0] # teens with 'th'
    elif units <= 3:
        return suf[units]
    else:
        return suf[0] # 'th'

def day_suffix(t):
    '''
    Returns the suffix of the given struct_time day
    '''
    return num_suffix(t.tm_mday)

# Examples
print num_suffix(123)
print num_suffix(3431)
print num_suffix(1234)
print ''
print day_suffix(time.strptime("1 Dec 00", "%d %b %y"))
print day_suffix(time.strptime("2 Nov 01", "%d %b %y"))
print day_suffix(time.strptime("3 Oct 02", "%d %b %y"))
print day_suffix(time.strptime("4 Sep 03", "%d %b %y"))
print day_suffix(time.strptime("13 Nov 90", "%d %b %y"))
print day_suffix(time.strptime("14 Oct 10", "%d %b %y"))​​​​​​​

Answer 11

In [34]: import datetime

In [35]: _now = datetime.datetime.now()

In [36]: _now
Out[36]: datetime.datetime(2016, 1, 19, 9, 47, 0, 432000)

In [37]: print _now
2016-01-19 09:47:00.432000

In [38]: _parsed = datetime.datetime.strptime(str(_now),"%Y-%m-%d %H:%M:%S.%f")

In [39]: _parsed
Out[39]: datetime.datetime(2016, 1, 19, 9, 47, 0, 432000)

In [40]: assert _now == _parsed

Answer 12

Django Timezone aware datetime object example.

import datetime
from django.utils.timezone import get_current_timezone
tz = get_current_timezone()

format = '%b %d %Y %I:%M%p'
date_object = datetime.datetime.strptime('Jun 1 2005  1:33PM', format)
date_obj = tz.localize(date_object)

This conversion is very important for Django and Python when you have USE_TZ = True:

RuntimeWarning: DateTimeField MyModel.created received a naive datetime (2016-03-04 00:00:00) while time zone support is active.

Answer 13

It would do the helpful for converting string to datetime and also with time zone

def convert_string_to_time(date_string, timezone):
    from datetime import datetime
    import pytz
    date_time_obj = datetime.strptime(date_string[:26], '%Y-%m-%d %H:%M:%S.%f')
    date_time_obj_timezone = pytz.timezone(timezone).localize(date_time_obj)

    return date_time_obj_timezone

date = '2018-08-14 13:09:24.543953+00:00'
TIME_ZONE = 'UTC'
date_time_obj_timezone = convert_string_to_time(date, TIME_ZONE)

Answer 14

Create a small utility function like:

def date(datestr="", format="%Y-%m-%d"):
    from datetime import datetime
    if not datestr:
        return datetime.today().date()
    return datetime.strptime(datestr, format).date()

This is versatile enough:

• If you don't pass any arguments it will return today's date.
• There's a date format as default that you can override.
• You can easily modify it to return a datetime.

Answer 15

arrow offers many useful functions for dates and times. This bit of code provides an answer to the question and shows that arrow is also capable of formatting dates easily and displaying information for other locales.

>>> import arrow
>>> dateStrings = [ 'Jun 1  2005 1:33PM', 'Aug 28 1999 12:00AM' ]
>>> for dateString in dateStrings:
...     dateString
...     arrow.get(dateString.replace('  ',' '), 'MMM D YYYY H:mmA').datetime
...     arrow.get(dateString.replace('  ',' '), 'MMM D YYYY H:mmA').format('ddd, Do MMM YYYY HH:mm')
...     arrow.get(dateString.replace('  ',' '), 'MMM D YYYY H:mmA').humanize(locale='de')
...
'Jun 1  2005 1:33PM'
datetime.datetime(2005, 6, 1, 13, 33, tzinfo=tzutc())
'Wed, 1st Jun 2005 13:33'
'vor 11 Jahren'
'Aug 28 1999 12:00AM'
datetime.datetime(1999, 8, 28, 0, 0, tzinfo=tzutc())
'Sat, 28th Aug 1999 00:00'
'vor 17 Jahren'

See http://arrow.readthedocs.io/en/latest/ for more.

Answer 16

If your string is in ISO8601 format and you have Python 3.7+ you can use the following simple code:

import datetime

aDate = datetime.date.fromisoformat('2020-10-04')

for dates and

import datetime

aDateTime = datetime.datetime.fromisoformat('2020-10-04 22:47:00')

for strings containing date and time. If timestamps are included the function datetime.datetime.isoformat() supports the following format

YYYY-MM-DD[*HH[:MM[:SS[.fff[fff]]]][+HH:MM[:SS[.ffffff]]]]

where * matches any single character. See also here and here

Answer 17

You can use easy_date to make it easy:

import date_converter
converted_date = date_converter.string_to_datetime('Jun 1 2005  1:33PM', '%b %d %Y %I:%M%p')

Answer 18

You can also check out dateparser

dateparser provides modules to easily parse localized dates in almost any string formats commonly found on web pages.

Install:

$ pip install dateparser

This is, I think, the easiest way you can parse dates.

The most straightforward way is to use the dateparser.parse function, that wraps around most of the functionality in the module.

Sample Code:

import dateparser

t1 = 'Jun 1 2005  1:33PM'
t2 = 'Aug 28 1999 12:00AM'

dt1 = dateparser.parse(t1)
dt2 = dateparser.parse(t2)

print(dt1)
print(dt2)

Output:

2005-06-01 13:33:00
1999-08-28 00:00:00

Answer 19

If you want only date format then you can manually convert it by passing your individual fields like:

>>> import datetime
>>> date = datetime.date(int('2017'),int('12'),int('21'))
>>> date
datetime.date(2017, 12, 21)
>>> type(date)
<type 'datetime.date'>

You can pass your split string values to convert it into date type like:

selected_month_rec = '2017-09-01'
date_formate = datetime.date(int(selected_month_rec.split('-')[0]),int(selected_month_rec.split('-')[1]),int(selected_month_rec.split('-')[2]))

You will get the resulting value in date format.

Answer 20

See my answer.

In real-world data this is a real problem: multiple, mismatched, incomplete, inconsistent and multilanguage/region date formats, often mixed freely in one dataset. It's not ok for production code to fail, let alone go exception-happy like a fox.

We need to try...catch multiple datetime formats fmt1,fmt2,...,fmtn and suppress/handle the exceptions (from strptime()) for all those that mismatch (and in particular, avoid needing a yukky n-deep indented ladder of try..catch clauses). From my solution

def try_strptime(s, fmts=['%d-%b-%y','%m/%d/%Y']):
    for fmt in fmts:
        try:
            return datetime.strptime(s, fmt)
        except:
            continue

    return None # or reraise the ValueError if no format matched, if you prefer

Answer 21

emp = pd.read_csv("C:\\py\\programs\\pandas_2\\pandas\\employees.csv")
emp.info()

it shows "Start Date Time" Column and "Last Login Time" both are "object = strings" in data-frame

<class 'pandas.core.frame.DataFrame'>
RangeIndex: 1000 entries, 0 to 999
Data columns (total 8 columns):
First Name           933 non-null object
Gender               855 non-null object

Start Date           1000 non-null object

Last Login Time      1000 non-null object

Salary               1000 non-null int64
Bonus %              1000 non-null float64
Senior Management    933 non-null object
Team                 957 non-null object
dtypes: float64(1), int64(1), object(6)
memory usage: 62.6+ KB

By using parse_dates option in read_csv mention you can convert your string datetime into pandas datetime format.

emp = pd.read_csv("C:\\py\\programs\\pandas_2\\pandas\\employees.csv", parse_dates=["Start Date", "Last Login Time"])
emp.info()


<class 'pandas.core.frame.DataFrame'>
RangeIndex: 1000 entries, 0 to 999
Data columns (total 8 columns):
First Name           933 non-null object
Gender               855 non-null object

Start Date           1000 non-null datetime64[ns]
Last Login Time      1000 non-null datetime64[ns]

Salary               1000 non-null int64
Bonus %              1000 non-null float64
Senior Management    933 non-null object
Team                 957 non-null object
dtypes: datetime64[ns](2), float64(1), int64(1), object(4)
memory usage: 62.6+ KB

Answer 22

It seems using pandas Timestamp is the fastest

import pandas as pd 

N = 1000

l = ['Jun 1 2005  1:33PM'] * N

list(pd.to_datetime(l, format=format))

%timeit _ = list(pd.to_datetime(l, format=format))
1.58 ms ± 21.6 µs per loop (mean ± std. dev. of 7 runs, 1000 loops each)

Other solutions

from datetime import datetime
%timeit _ = list(map(lambda x: datetime.strptime(x, format), l))
9.41 ms ± 95.7 µs per loop (mean ± std. dev. of 7 runs, 100 loops each)

from dateutil.parser import parse
%timeit _ = list(map(lambda x: parse(x), l))
73.8 ms ± 1.14 ms per loop (mean ± std. dev. of 7 runs, 10 loops each)

If the string is ISO8601 string please use csio8601

import ciso8601

l = ['2014-01-09'] * N

%timeit _ = list(map(lambda x: ciso8601.parse_datetime(x), l))
186 µs ± 4.13 µs per loop (mean ± std. dev. of 7 runs, 10000 loops each)

Answer 23

A short sample mapping a yyyy-mm-dd date string to a datetime.date object:

from datetime import date
date_from_yyyy_mm_dd = lambda δ : date(*[int(_) for _ in δ.split('-')])
date_object = date_from_yyyy_mm_dd('2021-02-15')