Linux Bash文本操作之sed篇其二

上一篇总结了sed的基础应用(Linux Bash文本操作之sed篇其一)，内容实在有够多，这里再对稍微高级一些的用法做一个整理，以方便使用时查阅。

查看文本内容

示例1表示在第一到第四行匹配到的行后面添加一行空行。

示例2带行号显示文本，行号与内容间隔两个空格，也可以是使用  \t  换成制表符。

示例3可以用来实现类似于 grep 的操作。使用正则表达式显示文中匹配到  sed  的行。

示例4中使用正则表达式配合  d  指令实现了与3相同的结果。

示例5是  grep  命令得到的结果，与前两个示例所不同的是  grep  显示出来的是带有颜色信息的，清晰地指明了匹配发生的位置。

利用标注的标签进行跳转，相当于分支，用于改变流处理顺序。

示例6显式匹配到字符串的下一行的内容，使用  n  来读取下一行内容到模式空间，替换当前行内容。

示例7使文本显示为左对齐。

: label     Label for b and t commands.
b           label Branch to label; if label is omitted, branch to end of script.
t label     If a s/// has done a successful substitution since the last input line was read and
            since the last t or T command, then branch to label; if label is omitted, branch to
            end of script.

cv@cv:~/myfiles$ sed -n '1,4{/stream/G;p}' test.txt    #example-
NAME
       sed - stream editor for filtering and transforming text

SYNOPSIS
       sed [OPTION]... {script-only-if-no-other-script} [input-file]...
cv@cv:~/myfiles$ sed = test.txt | sed 'N;s/\n/  /'    #example-
  NAME
         sed - stream editor for filtering and transforming text
  SYNOPSIS
         sed [OPTION]... {script-only-if-no-other-script} [input-file]...
  DESCRIPTION
         Sed  is  a  stream  editor.  A stream editor is used to perform basic text transformations on an input stream (a file or input from a pipeline).  While in some ways similar to an
         editor which permits scripted edits (such as ed), sed works by making only one pass over the input(s), and is consequently more efficient.  But it is sed's ability to filter text
         in a pipeline which particularly distinguishes it from other types of editors.

         -n, --quiet, --silent
                suppress automatic printing of pattern space
         -e script, --expression=script
                add the script to the commands to be executed
         -f script-file, --file=script-file
                add the contents of script-file to the commands to be executed
         --follow-symlinks
                follow symlinks when processing in place
         -i[SUFFIX], --in-place[=SUFFIX]
cv@cv:~/myfiles$ sed -n '/sed/p' test.txt    #example-
       sed - stream editor for filtering and transforming text
       sed [OPTION]... {script-only-if-no-other-script} [input-file]...
       Sed  is  a  stream  editor.  A stream editor is used to perform basic text transformations on an input stream (a file or input from a pipeline).  While in some ways similar to an
       editor which permits scripted edits (such as ed), sed works by making only one pass over the input(s), and is consequently more efficient.  But it is sed's ability to filter text

cv@cv:~/myfiles$ sed '/sed/!d' test.txt    #example-
       sed - stream editor for filtering and transforming text
       sed [OPTION]... {script-only-if-no-other-script} [input-file]...
       Sed  is  a  stream  editor.  A stream editor is used to perform basic text transformations on an input stream (a file or input from a pipeline).  While in some ways similar to an
       editor which permits scripted edits (such as ed), sed works by making only one pass over the input(s), and is consequently more efficient.  But it is sed's ability to filter text

cv@cv:~/myfiles$ grep 'sed' test.txt    #example-
       sed - stream editor for filtering and transforming text
       sed [OPTION]... {script-only-if-no-other-script} [input-file]...
       Sed  is  a  stream  editor.  A stream editor is used to perform basic text transformations on an input stream (a file or input from a pipeline).  While in some ways similar to an
       editor which permits scripted edits (such as ed), sed works by making only one pass over the input(s), and is consequently more efficient.  But it is sed's ability to filter text

cv@cv:~/myfiles$ sed -n '/OPTION/{n;p}' test.txt         #example-
DESCRIPTION

cv@cv:~/myfiles$ sed ':a;s/^.\{1,70\}$/ &/;ta;s/\( *\)//' test.txt     #example-
NAME
sed - stream editor for filtering and transforming text
SYNOPSIS
sed [OPTION]... {script-only-if-no-other-script} [input-file]...
DESCRIPTION
Sed  is  a  stream  editor.  A stream editor is used to perform basic text transformations on an input stream (a file or input from a pipeline).  While in some ways similar to an
editor which permits scripted edits (such as ed), sed works by making only one pass over the input(s), and is consequently more efficient.  But it is sed's ability to filter text
in a pipeline which particularly distinguishes it from other types of editors.

-n, --quiet, --silent
suppress automatic printing of pattern space
-e script, --expression=script
add the script to the commands to be executed
-f script-file, --file=script-file
add the contents of script-file to the commands to be executed
--follow-symlinks
follow symlinks when processing in place
-i[SUFFIX], --in-place[=SUFFIX]

我们还可以结合临时空间与模式空间之间的特性实现将文本前后各行顺序颠倒的效果，也就是第一行变成最后一行，第二行变成倒数第二行，第三行变成倒数第三行……最后一行变成第一行，类似于  tac  的操作。

下面有一张图清晰的表明了整个转换过程，图的来源是参考文献[3]给出的链接。

cv@cv:~/myfiles$ sed '1!G;h;$!d;' test.txt
       -i[SUFFIX], --in-place[=SUFFIX]
              follow symlinks when processing in place
       --follow-symlinks
              add the contents of script-file to the commands to be executed
       -f script-file, --file=script-file
              add the script to the commands to be executed
       -e script, --expression=script
              suppress automatic printing of pattern space
       -n, --quiet, --silent

       in a pipeline which particularly distinguishes it from other types of editors.
       editor which permits scripted edits (such as ed), sed works by making only one pass over the input(s), and is consequently more efficient.  But it is sed's ability to filter text
       Sed  is  a  stream  editor.  A stream editor is used to perform basic text transformations on an input stream (a file or input from a pipeline).  While in some ways similar to an
DESCRIPTION
       sed [OPTION]... {script-only-if-no-other-script} [input-file]...
SYNOPSIS
       sed - stream editor for filtering and transforming text
NAME

还可以实现类似于tail的效果。

如示例1相当于 tail -n  ，除最后两行之外合并，以  \n  相连接，然后再从模式空间删除。

D表示删除模式空间中从第一个字符到第一个换行符的内容，并且跳转到命令开头重新执行。并且当模式空间仍有内容时，不读入新的输入行，类似形成一个循环。

P命令仅打印模式空间中从第一个字符到第一个换行符的内容，重新在模式空间的内容上执行编辑命令，类似形成一个循环。

示例2打印最后  m-  行，通过  N  和  D  命令的循环来实现。如果到最后一行，终止退出，否则的话读入下一行追加到当前模式空间。

示例3用于打印匹配行的下一行内容。

示例4用于打印匹配行的上一行与下一行。

D   If pattern space contains no newline, start a normal new cycle as if the d command was issued.
    Otherwise, delete text in the pattern space up to the first newline, and restart cycle with the
    resultant pattern space, without reading a new line of input.

cv@cv:~/myfiles$ sed '$!N;$!D' test.txt             #example-

cv@cv:~/myfiles$ sed ':a;$q;N;3,$D;ba;' test.txt     #example-
              follow symlinks when processing in place
       -i[SUFFIX], --in-place[=SUFFIX]

cv@cv:~/myfiles$ sed -n '/sed/{g;1!p;};h' test.txt    #example-
NAME
SYNOPSIS
DESCRIPTION
DESCRIPTION

cv@cv:~/myfiles$ sed -n '/sed/{x;1!p;=;g;$!N;p;D;};h' test.txt    #example-
NAME

       sed - stream editor for filtering and transforming text
SYNOPSIS
SYNOPSIS

       sed [OPTION]... {script-only-if-no-other-script} [input-file]...
DESCRIPTION
DESCRIPTION

       Sed  is  a  stream  editor.  A stream editor is used to perform basic text transformations on an input stream (a file or input from a pipeline).  While in some ways similar to an
       editor which permits scripted edits (such as ed), sed works by making only one pass over the input(s), and is consequently more efficient.  But it is sed's ability to filter text
       Sed  is  a  stream  editor.  A stream editor is used to perform basic text transformations on an input stream (a file or input from a pipeline).  While in some ways similar to an

       editor which permits scripted edits (such as ed), sed works by making only one pass over the input(s), and is consequently more efficient.  But it is sed's ability to filter text
       in a pipeline which particularly distinguishes it from other types of editors.

删除文本内容

还有一种  addr1  为   的地址范围表示法，不过这时  addr2  只能使用正则表达式来表示。其结果与前面给出的两个地址的结果相同。

示例1删除每一行开头所有空格。

示例2表示删除每一行末尾所有空格。

示例3是1和2的综合作用，删除每一行开头和末尾的所有空格。

示例4表示只删除匹配到的第一行内容。

,addr2    Start out in "matched first address" state, until addr2 is found. This is similar to ,addr2, except that if addr2 matches the
           very first line of input the  ,addr2  form will be at the end of its range, whereas the ,addr2 form will still be at the beginning
　　　　　　 of its range. This works only when addr2 is a regular expression.

cv@cv:~/myfiles$ sed -n 's/^[ ^t]*//p' test.txt    #example-

cv@cv:~/myfiles$ sed -n 's/[ ^t]*$//p' test.txt    #example-

cv@cv:~/myfiles$ sed -n -e 's/^[ ^t]*//;s/[ ^t]*$//p' test.txt    #example-
cv@cv:~/myfiles$ sed '0,/sed/{//d;}' test.txt    #example-
NAME
SYNOPSIS
       sed [OPTION]... {script-only-if-no-other-script} [input-file]...
DESCRIPTION
       Sed  is  a  stream  editor.  A stream editor is used to perform basic text transformations on an input stream (a file or input from a pipeline).  While in some ways similar to a
       editor which permits scripted edits (such as ed), sed works by making only one pass over the input(s), and is consequently more efficient.  But it is sed's ability to filter tex
       in a pipeline which particularly distinguishes it from other types of editors.

       -n, --quiet, --silent
              suppress automatic printing of pattern space
       -e script, --expression=script
              add the script to the commands to be executed
       -f script-file, --file=script-file
              add the contents of script-file to the commands to be executed
       --follow-symlinks
              follow symlinks when processing in place
       -i[SUFFIX], --in-place[=SUFFIX]

对于空行的删除我们还可以有很多不同的方法。可以在我们的测试文件里添加一些空行试试效果。

示例1表示删除文本中正文第一行之前的若干空行。

示例2删除文中连续的空行，只保留一行，也可以理解为将中间连续的空行合并成一行。

示例3将多于一行的连续空行删除掉，或者说合并成一行，与上面的命令效果相同。

示例4将多于两行的连续空行删减成两行，或者说只保留两行。

示例5删除最尾部的空白行，找到最后一行则删除，不是最后一行用N命令追加到其下一行。

cv@cv:~/myfiles$ sed '/./,$!d' test.txt              #example-

cv@cv:~/myfiles$ sed '/./,/^$/!d' test.txt           #example-

cv@cv:~/myfiles$ sed '/^$/N;/\n$/D' file.txt         #example-

cv@cv:~/myfiles$ sed '/^$/N;/\n$/N;//D' test.txt     #example-

cv@cv:~/myfiles$ sed ':a;/^\n*$/{$d;N;ba}' test.txt  #example-

当我们想要删除文本最后一行时非常容易，直接 sed -i '$d' test.txt 即可，而当我们想要删除最后几行时又该如何操作呢？

诚然我们可以写一个脚本循环删除最后一行，如示例1所示。或者根据文本总行数进行行范围删除操作，如示例2。

我们还可以结合  N P D  来操作，如示例3。推荐使用这种方式。

如果要删除的行数比较少，如最后两行，可以使用示例4或者示例5。

示例5大括号内的N有几个就表示删除几加一行。

#!/bin/bash
#myscript for example-1

num=
count=
while [ $count -le $num ];do
    sed -i '$d' test.txt
    let count=count+
done
echo "Done!"

#!/bin/bash
#myscript for example-2
n=
file=cptest.txt
A=$(sed -n '$=' $file)
sed $(($A-$n+)),${A}d $file

cv@cv:~/myfiles$ bash ./myscript.sh    #example-
cv@cv:~/myfiles$ bash ./myscript.sh    #example-

cv@cv:~/myfiles$ n=;sed '1{:a;N;'$n'!ba};$d;N;P;D' test.txt    #example-

cv@cv:~/myfiles$ sed 'N;$!P;$!D;$d' test.txt    #example-

cv@cv:~/myfiles$ sed -n '1{N;N;};:a;N;P;D;ta' test.txt    #example-

增加文件内容

下面的指令用于交换模式空间和临时空间的内容。

示例1实现在每一个匹配行的前面添加一行空行。读到文本的一行后，模式空间内为该行内容，临时空间暂时为空，此时如果交换两者内容，模式空间变为空，打印出来就是空行；

然后再将两者交换回来，模式空间又变成了刚才读到的一行内容，这是再打印就是文本内容了。最终结果就是在文本之前多了一行空行。

示例2实现在每一个匹配行的前面和后面各添加一行空行。

第二次交换后模式空间中为原来读到的一行内容，临时空间为空，这时将临时空间的内容追加到模式空间之后，就相当于在行后又添加了一行空行。

x   Exchange the contents of the hold and pattern spaces.

cv@cv:~/myfiles$ sed -n '/sed/{x;p;x;p}' test.txt    #example-

       sed - stream editor for filtering and transforming text

       sed [OPTION]... {script-only-if-no-other-script} [input-file]...

       Sed  is  a  stream  editor.  A stream editor is used to perform basic text transformations on an input stream (a file or input from a pipeline).  While in some ways similar to an

       editor which permits scripted edits (such as ed), sed works by making only one pass over the input(s), and is consequently more efficient.  But it is sed's ability to filter text

cv@cv:~/myfiles$ sed -n '/sed/{x;p;x;G;p}' test.txt | sed '$d'    #example-

       sed - stream editor for filtering and transforming text

       sed [OPTION]... {script-only-if-no-other-script} [input-file]...

       Sed  is  a  stream  editor.  A stream editor is used to perform basic text transformations on an input stream (a file or input from a pipeline).  While in some ways similar to an

       editor which permits scripted edits (such as ed), sed works by making only one pass over the input(s), and is consequently more efficient.  But it is sed's ability to filter text

替换/转换文本内容

之前我们使用的都是基本正则表达式，如果在环境选择时加上-r的选项，则可以使用扩展正则表达式。

在基本正则中，几个字符(  ? + {} () |  )默认被解释为普通字符，也即字面意思，跟  a b c  这样的字符一样表示 ASCII 字符，如果想要使用其对应的特殊含义，如使用  ( )  表示选项列表，  |  表示选项等，则需要加反斜杠  \  转义。

而在扩展正则中，默认情况下上述字符  ? + {} () |  被解释为特殊含义，如果想要使用其所对应的普通含义，作为一个普通字符来使用，需要加反斜杠  \  转义。

-r, --regexp-extended    use extended regular expressions in the script.

示例1表示将所有以s或S开头的单词转换成全部大写的形式。  \<  词首锚定，用于匹配单词词首。

示例2是使用之前所说的单词边界匹配符，可以得到同样的结果，注意这里的  () +  在普通正则中使用时需要转义成特殊含义。

示例3表示在匹配到字符串的第一行，将其中的sed替换成目标字符串。

cv@cv:~/myfiles$ sed -n -r 's/\<(s|S)[a-z]+/\U&/gp' test.txt    #example-
       SED - STREAM editor for filtering and transforming text
       SED [OPTION]... {SCRIPT-only-if-no-other-SCRIPT} [input-file]...
       SED  is  a  STREAM  editor.  A STREAM editor is used to perform basic text transformations on an input STREAM (a file or input from a pipeline).  While in SOME ways SIMILAR to an
       editor which permits SCRIPTED edits (SUCH as ed), SED works by making only one pass over the input(s), and is consequently more efficient.  But it is SED's ability to filter text
       -n, --quiet, --SILENT
              SUPPRESS automatic printing of pattern SPACE
       -e SCRIPT, --expression=SCRIPT
              add the SCRIPT to the commands to be executed
       -f SCRIPT-file, --file=SCRIPT-file
              add the contents of SCRIPT-file to the commands to be executed
       --follow-SYMLINKS
              follow SYMLINKS when processing in place
cv@cv:~/myfiles$ sed -n -e 's/\b\(s\|S\)[a-z]\+/\U&/gp' test.txt    #example-2
       SED - STREAM editor for filtering and transforming text
       SED [OPTION]... {SCRIPT-only-if-no-other-SCRIPT} [input-file]...
       SED  is  a  STREAM  editor.  A STREAM editor is used to perform basic text transformations on an input STREAM (a file or input from a pipeline).  While in SOME ways SIMILAR to an
       editor which permits SCRIPTED edits (SUCH as ed), SED works by making only one pass over the input(s), and is consequently more efficient.  But it is SED's ability to filter text
       -n, --quiet, --SILENT
              SUPPRESS automatic printing of pattern SPACE
       -e SCRIPT, --expression=SCRIPT
              add the SCRIPT to the commands to be executed
       -f SCRIPT-file, --file=SCRIPT-file
              add the contents of SCRIPT-file to the commands to be executed
       --follow-SYMLINKS
              follow SYMLINKS when processing in place
cv@cv:~/myfiles$ sed -n '0,/sed/s//to_that/p' test.txt    #example-
       to_that - stream editor for filtering and transforming text

在根据匹配进行字符串等的替换时，我们可以指定替换第几个匹配。

如示例1与示例2，在s//num中num指的是第几个匹配位置，比如1表示该行第一个匹配，2表示第二个。

示例1就表示将每行第一个sed替换成目标字符串。

示例2就表示将每行第二个替换掉。

示例3表示替换倒数第二个匹配。

示例4表示替换倒数第一个，也就是最后一个匹配。

cv@cv:~/myfiles$ sed -n 's/sed/XXX/1p' test.txt    #example-
       XXX - stream editor for filtering and transforming text
       XXX [OPTION]... {script-only-if-no-other-script} [input-file]...
       Sed  is  a  stream  editor.  A stream editor is uXXX to perform basic text transformations on an input stream (a file or input from a pipeline).  While in some ways similar to an
       editor which permits scripted edits (such as ed), XXX works by making only one pass over the input(s), and is consequently more efficient.  But it is sed's ability to filter text

cv@cv:~/myfiles$ sed -n 's/sed/XXX/2p' test.txt    #example-
       editor which permits scripted edits (such as ed), sed works by making only one pass over the input(s), and is consequently more efficient.  But it is XXX's ability to filter text

cv@cv:~/myfiles$ sed -n 's/\(.*\)sed\(.*sed\)/\1XXX\2/p' test.txt    #example-
       editor which permits scripted edits (such as ed), XXX works by making only one pass over the input(s), and is consequently more efficient.  But it is sed's ability to filter text

cv@cv:~/myfiles$ sed -n 's/\(.*\)sed/\1XXX/p' test.txt    #example-
       XXX - stream editor for filtering and transforming text
       XXX [OPTION]... {script-only-if-no-other-script} [input-file]...
       Sed  is  a  stream  editor.  A stream editor is uXXX to perform basic text transformations on an input stream (a file or input from a pipeline).  While in some ways similar to an
       editor which permits scripted edits (such as ed), sed works by making only one pass over the input(s), and is consequently more efficient.  But it is XXX's ability to filter text

参考文献

[1] Learning Linux Commands: sed

[2] Shell正则表达式

[3] 【系统工程师的自我修养】sed篇