AWS Glue: How to filter out data from DynamicFrame when date format is wrong or bad data

Question

In Aws Glue after extracting data in DynamicFrame I'm converting date time format to UTC, But if in case date format is wrong for eg Invalid value for date, It will break entire glue flow. So I want to Filter out these bad data from DynamicFrame before processing it further.

I'm using Filter.apply for filtering data and my date is present in "Date": "2022-01-01T12:11:27.251Z" this format.

Answer 1

You can parse the Date field to check if it has the expected format. Example:

from datetime import datetime

date_str = "2022-01-01T12:11:27.251Z"

try:
    datetime_obj = datetime.strptime(date_str, "%Y-%m-%dT%H:%M:%S.%fZ")
    # date_str has the correct format, continue processing row
except ValueError:
    # date_str does not have the correct format, do something...

You can include this logic in the implementation of Filter.apply(). For example, if the Date field has an invalid format, the row can be filtered out.