Various Ways to Open a Huge CSV File

Introduction

Last time I showed how to create a huge CSV file in Python.

Pythonで中身のある巨大なCSVファイルを作成する - Qiita

はじめに Pythonで中身のある巨大なCSVファイルを作成する方法を紹介します。 ただし、この方法は作成するのに割と時間かかります。 もっといい方法があると思いますが、ご了承ください。 単純に中身のないモックデータを作成したいのであれば、こちらの記事で紹介した方法がおす...

https://qiita.com/EveSquare/items/67552dbccb3a7c326703

This time, I’ll show how to read a huge CSV file. Finder’s Quick Look and VSCode’s preview feature can’t open a huge CSV file. In that case, try the following methods.

Reading only the first few lines

1. Use the head command

The head command displays the beginning of a file. It’s handy when you only want to see a few lines.

head -n 10 <CSV file name>.csv

This command displays the first 10 lines.

2. Use the less command

The less command lets you browse a file’s contents page by page. You can browse even large files comfortably.

less <CSV file name>.csv

Press the q key to exit less.

3. Use Python

You can also open a CSV file using Python. Here’s an example Python script.

import pandas as pd

# Read the first few rows of the CSV file
df = pd.read_csv('<CSV file name>.csv', nrows=10)
print(df)

This script reads and displays the first 10 rows of <CSV file name>.csv. To run the Python script, use the following command in the terminal.

python main.py

Reading specific lines

Let me explain how to view specific lines. Below are a few ways to display specific lines.

1. Use the sed command

You can use the sed command to display specific lines. For example, to display line 10, do the following.

sed -n '10p' <CSV file name>.csv

To display multiple specific lines, specify a range with a comma. For example, to display lines 10 through 20, do the following.

sed -n '10,20p' <CSV file name>.csv

2. Use the awk command

You can also use the awk command to display specific lines. For example, to display line 10, do the following.

awk 'NR==10' <CSV file name>.csv

To display lines 10 through 20, do the following.

awk 'NR>=10 && NR<=20' <CSV file name>.csv

4. Use Python

You can also use Python to display specific lines. Here’s an example.

import pandas as pd

line_number = 10
df = pd.read_csv('<CSV file name>.csv', skiprows=line_number - 1, nrows=1)
print(df)

This script reads and displays line 10.

Conclusion

Use whichever fits your purpose.