How to Split large CSV file into smaller files
To split a large CSV file into smaller parts, you can use the split command
tail -n +2 access_log.csv | split -l 50000 - --filter='sh -c "{ head -n1 access_log.csv; cat; } > $FILE"'
The above command will split file access_log.csv into smaller parts, each part will have the header from the CSV file.
split -l 50000 means we will have 50,000 lines per file. If you need a different number of lines per file, you can change the number.
To rename all generated log files numerically, use the following command
counter=0; for file in x*; do let "counter+=1"; echo "mv '$file' 'log-${counter}.log'" ; done | bash
Back to split