AmirWatad.com

Removing Control Characters From a Text File in Linux

A few weeks ago, I had a text file generated in widows, and it had the “^M” control character at the end of each line. I had to compare it with a similar file generated on my Linux machine (using diff), and because of these control characters it failed diff.

The first step was to check why diff failed. Normally, you cannot see these control characters, and one might be surprised that the comparison fails thought the files are exactly the same. In order to see these characters, one should use “-A” flag with the cat command:

cat -A textfile

Next step is to removc these characters. This wasn’t easy, and I asked a friend who referred me to the program “dos2unix” which converts text files generated in dos/widows into their Linux counterparts (and vice versa).

This worked!

Recently, I found another way to do this, using “sed”. The trick was to type the control character “^M” in the terminal.

This is how we’d want the sed command to look like:

1
sed -e 's/^M//g' widosTextFile > newTextFile

This should replace the character “^M” by nothing (The ‘g’ is for global: i.e. replace every instance of ^M in each line, not only the first instance in each line), which is exactly what we want.

But, just typing ^M won’t work, since it will be interpretted as two characters, ^ and M, which is not the behavior we want. To type the control character ^M one should use this:  [ctrl]+[v] and then [ctrl]+[m]. This will be displayed as ^M in the terminal, but it will have a completely different meaning.

Hope it helps somebody.

Related posts:

1. Re-Use A Bash Command With Different Parameters

Tags: Bash, Command Line Interface, Linux