Creating a Huge CSV File with Real Content in Python

Introduction

I’ll show how to create a huge CSV file with real content in Python. Note that this approach takes a fair amount of time to run. There’s probably a better way, but please bear with me.

If you just want to create mock data without meaningful content, the method I introduced in this article is recommended.

Macでお手軽にモックファイルを作る - Qiita

はじめに フォームのバリデーション等を確認する際に、100MBのファイルや1GBのファイルをアップロードする必要がある場合があります。そのような場合、実際のファイルを用意するのは手間がかかります。そこで、Macで簡単にモックファイルを作成する方法を紹介します。 作成方法...

https://qiita.com/EveSquare/items/f80e3dd2f3f9e0c720c5

How to create it

Run pip install pandas beforehand to install pandas.

import csv

import numpy as np

column_names = ["id", "name", "price", "category"]
data_types = [np.int64, np.object_, np.int64, np.object_]
num_rows = 40000000

data = []
for _ in range(num_rows):
    row = []
    for col, dtype in zip(column_names, data_types):
        if dtype == np.int64:
            row.append(np.random.randint(low=1, high=10000))
        elif dtype == np.object_:
            row.append(np.random.choice(["A usage", "B usage", "C usage"]))
        else:
            raise ValueError(f"{dtype} is not defined")
    data.append(row)

with open("mock_data.csv", "w", newline="") as f:
    writer = csv.writer(f)
    writer.writerow(column_names)
    writer.writerows(data)

This creates 40,000,000 rows of data. Creating 40,000,000 rows took about 10 minutes. 40,000,000 rows results in a file of about 1.3 GB. I created the column data arbitrarily, so change it as needed.

Conclusion

I’m not sure when you’ll need this script, but I’ve shown how to create a huge CSV file. I hope it helps someone.