Read all rows with pd.read_csv

Question

I'm using Python to read a file of 5,000,000 rows but currently it only reads 1,000,000 rows. The file is around 125mb.

I'm using the pd.read_csv function but this only leads to reading 1,000,000 rows of my file.

Answer 1

For reading long files in Python, I recommend using Dask:

import dask.dataframe as dd
df = dd.read_csv('yourfile.csv')

For more information, visit the Dask documentation page:

https://docs.dask.org/en/stable/generated/dask.dataframe.read_csv.html