How to Split large CSV file into smaller files

To split a large CSV file into smaller parts, you can use the split command

tail -n +2 access_log.csv | split -l 50000 - --filter='sh -c "{ head -n1 access_log.csv; cat; } > $FILE"'

1	tail -n +2 access_log.csv \| split -l 50000 - --filter='sh -c "{ head -n1 access_log.csv; cat; } > $FILE"'

The above command will split file access_log.csv into smaller parts, each part will have the header from the CSV file.

split -l 50000 means we will have 50,000 lines per file. If you need a different number of lines per file, you can change the number.

To rename all generated log files numerically, use the following command

counter=0; for file in x*; do let "counter+=1"; echo "mv '$file' 'log-${counter}.log'" ; done | bash

1	counter=0; for file in x*; do let "counter+=1"; echo "mv '$file' 'log-${counter}.log'" ; done \| bash

Back to split