awk

unique values of column

	awk -F';' '$5 ~ /PATTERN/ {seen[$7]++} END {for (val in seen) print val}' fichier.csv

Concatenate CSV files with awk: Why and How

When merging CSV files with headers, using cat can cause problems — it blindly appends all files, including their headers, resulting in duplicates like:

datetime;message key;...
data1
data2
datetime;message key;...
data3

A cleaner way? Use ###awk to keep only the first header:

awk 'NR==1 || FNR>1' file1.csv file2.csv file3.csv > merged.csv

Or

awk 'NR==1 || FNR>1' files*.csv > merged.csv

Explanation:

• FNR: Line number within the current file
• NR: Line number across all files
• NR==1: Keep the first line of the first file (the header)
• FNR>1: Skip the first line of all other files