There are a myriad of ways to create files the most common of which are manual editing (with vi, vim, nano, etc) and standard output redirection…
# echo “Hello World!” >> /tmp/hello.txt
The echo command is a bit of a relic today and the use of the far more flexible printf is encouraged:
# printf “Integer: %d
Integer: 5 float: 123.45 string: long string
The cat command is pretty straight-forward and it can be used not only to view files, but also to concatenate them:
# cat file1 file2 file3 >> file123
There are a few useful tricks we can use with cat:
# cat /tmp/test1
line1
line2
.
.
.
line3
.
# cat -s /tmp/test1 -> the -s flag suppresses duplicate empty lines
line1
line2
.
line3
.
# cat -vET test2 -> shows non-printable characters (-v), tabs (-T) and end-of-line (-E) with the $ sign
line1$
line2$
^I$
$
M-BM-6$
line3$
.
# cat -A test2 -> the -A flag does the same as -vET
line1$
line2$
^I$
$
M-BM-6$
line3$
.
# cat -n test3 -> enumerate all lines
1 line1
2 line2
3
4
5 ¶
6 line3
.
# cat -b test3 -> enumerate non-blank lines
1 line1
2 line2
3 ¶
4 line3
.
If we need to reverse the order of lines in a file we can do that provided we have a sorting key:
root:/tmp> cat normal-order.txt | sort
1 aaa
2 bbb
3 ccc
4 ddd
.
root:/tmp> cat normal-order.txt | sort -r
4 ddd
3 ccc
2 bbb
1 aaa
But what can we do if there is no sorting column? Then we can go the script route… or use tac:
root:/tmp> tac normal-order.txt | tee reverse-order.txt
4 ddd
3 ccc
2 bbb
1 aaa
The tac command does exactly the same as cat but in reverse order, starting from the last line of a file and working its way up to the 1st.
The more command is also used a lot…
marc:~> more .bashrc
# .bashrc
.
# Source global definitions
if [ -f /etc/bashrc ]; then
. /etc/bashrc
fi
.
# Uncomment the following line if you don’t like systemctl’s auto-paging feature:
# export SYSTEMD_PAGER=
.
# User specific aliases and functions
source ~/.bash_profile
The most commonly used keys when more-ing a file are:
– RETURN: moves forward one line
– SPACE: moves forward one screen
– b: moves backward one screen
– q: quit
But there are a few more than can be pretty handy:
v startup editor in current line (usually vi but can be changed with EDITOR variable)
= show current line
:f show current file name and line number
:n move on to next file
:p move back to previous file
/pattern searches the given regexp
n finds the next regexp match
!cmd execute command in subshell
. repeat previous command
And we can include the pattern search and line number when invoking more:
marc:~> more +35 .viminfo -> jump straight to line 35
marc:~> more +/AUTH_KEY .viminfo -> jump to first occurrence of AUTH_KEY
The less command is a bit more convenient than more as it offers more navigation and search options and does not need to load the whole target file in order to view it. That is especially useful for very large files!
The exact same navigational options that can be used with more (see above) can also be used with less. Additionally though we can also use:
– UP-ARROW: move up one line
– DOWN-ARROW: move down one line
– LEFT-ARROW: move up half a screen
– RIGHT-ARROW: move down half a screen
– g: go to 1st line
– G: go to last line
There are a few more navigational options available (i.e. bracket & parenthesis matching and screen resizing) but we’ll skip them for now as they’re rarely used.
The same pattern match searches that work with more also work with less, but we have a few more tricks available to us:
/!pattern will find lines NOT matching the given pattern
/*pattern will find lines matching the pattern in the current file and starting in the current position and, if none are found, it will continue with the rest of target files
/@pattern will find lines matching the pattern starting in the 1st line of the 1st file
?pattern will find lines matching the pattern upwards or backwards
&pattern will find lines matching the pattern and hide all the rest from view
n find the next match
N find the previous match
Another two viewing commands that are very useful are head and tail.
root:/var/log> head dnf.librepo.log
22:16:19 Current date: 2016-12-27T22:16:19+0100
22:16:19 lr_download: Target: file:///etc/dnf/dnf.conf (-)
22:16:19 select_next_target: Selecting mirror for: file:///etc/dnf/dnf.conf
22:16:19 prepare_next_transfer: URL: file:///etc/dnf/dnf.conf
22:16:19 lr_download: Downloading started
22:16:19 check_transfer_statuses: Transfer finished: file:///etc/dnf/dnf.conf (Effective url: file:///etc/dnf/dnf.conf)
22:16:19 lr_download: Target: file:///etc/yum.repos.d/fedora-cisco-openh264.repo (-)
22:16:19 select_next_target: Selecting mirror for: file:///etc/yum.repos.d/fedora-cisco-openh264.repo
22:16:19 prepare_next_transfer: URL: file:///etc/yum.repos.d/fedora-cisco-openh264.repo
22:16:19 lr_download: Downloading started
Without any arguments, head will show the first 10 lines of a given file. But we can show the number of lines we want or limit the output by bytes rather than lines:
root:/var/log> head -n 5 dnf.log -> show only the 1st 5 lines
Dec 27 22:16:19 INFO — logging initialized —
Dec 27 22:16:19 DDEBUG timer: config: 3 ms
Dec 27 22:16:19 DEBUG cachedir: /var/cache/dnf
Dec 27 22:16:19 DEBUG Loaded plugins: copr, protected_packages, Query, debuginfo-install, needs-restarting, download, playground, config-manager, builddep, noroot, reposync, generate_completion_cache
Dec 27 22:16:19 DEBUG DNF version: 1.1.10
.
root:/var/log> head -5 dnf.log -> same as previous command
Dec 27 22:16:19 INFO — logging initialized —
Dec 27 22:16:19 DDEBUG timer: config: 3 ms
Dec 27 22:16:19 DEBUG cachedir: /var/cache/dnf
Dec 27 22:16:19 DEBUG Loaded plugins: copr, protected_packages, Query, debuginfo-install, needs-restarting, download, playground, config-manager, builddep, noroot, reposync, generate_completion_cache
Dec 27 22:16:19 DEBUG DNF version: 1.1.10
.
root:/var/log> head -c 99 dnf.log -> show only the 1st 99 characters
Dec 27 22:16:19 INFO — logging initialized —
Dec 27 22:16:19 DDEBUG timer: config: 3 ms
.
root:/var/log> head -5 dnf.log dnf.librepo.log -> show first 5 lines of multiple files
==> dnf.log <==
Dec 27 22:16:19 INFO — logging initialized —
Dec 27 22:16:19 DDEBUG timer: config: 3 ms
Dec 27 22:16:19 DEBUG cachedir: /var/cache/dnf
Dec 27 22:16:19 DEBUG Loaded plugins: copr, protected_packages, Query, debuginfo-install, needs-restarting, download, playground, config-manager, builddep, noroot, reposync, generate_completion_cache
Dec 27 22:16:19 DEBUG DNF version: 1.1.10
.
==> dnf.librepo.log <==
22:16:19 Librepo version: 1.7.18 with CURL_GLOBAL_ACK_EINTR support (libcurl/7.51.0 NSS/3.27 zlib/1.2.8 libidn2/0.11 libpsl/0.14.0 (+libidn2/0.10) libssh2/1.8.0 nghttp2/1.13.0)
22:16:19 Current date: 2016-12-27T22:16:19+0100
22:16:19 lr_download: Target: file:///etc/dnf/dnf.conf (-)
22:16:19 select_next_target: Selecting mirror for: file:///etc/dnf/dnf.conf
22:16:19 prepare_next_transfer: URL: file:///etc/dnf/dnf.conf
.
root:/var/log> head -5 -q dnf.log dnf.librepo.log -> “-q” removes the file headers
Dec 27 22:16:19 INFO — logging initialized —
Dec 27 22:16:19 DDEBUG timer: config: 3 ms
Dec 27 22:16:19 DEBUG cachedir: /var/cache/dnf
Dec 27 22:16:19 DEBUG Loaded plugins: copr, protected_packages, Query, debuginfo-install, needs-restarting, download, playground, config-manager, builddep, noroot, reposync, generate_completion_cache
Dec 27 22:16:19 DEBUG DNF version: 1.1.10
22:16:19 Librepo version: 1.7.18 with CURL_GLOBAL_ACK_EINTR support (libcurl/7.51.0 NSS/3.27 zlib/1.2.8 libidn2/0.11 libpsl/0.14.0 (+libidn2/0.10) libssh2/1.8.0 nghttp2/1.13.0)
22:16:19 Current date: 2016-12-27T22:16:19+0100
22:16:19 lr_download: Target: file:///etc/dnf/dnf.conf (-)
22:16:19 select_next_target: Selecting mirror for: file:///etc/dnf/dnf.conf
22:16:19 prepare_next_transfer: URL: file:///etc/dnf/dnf.conf
.
root:/var/log> tail -n 3 dnf.log -> show last 3 lines
Dec 30 13:23:11 DEBUG Making cache files for all metadata files.
Dec 30 13:23:11 INFO Metadata cache refreshed recently.
Dec 30 13:23:11 DDEBUG Cleaning up.
.
root:/var/log> tail -3 dnf.log -> same as above
Dec 30 13:23:11 DEBUG Making cache files for all metadata files.
Dec 30 13:23:11 INFO Metadata cache refreshed recently.
Dec 30 13:23:11 DDEBUG Cleaning up.
.
root:/var/log> tail -c 157 dnf.log -> show last 157 characters
Dec 30 13:23:11 DEBUG Making cache files for all metadata files.
Dec 30 13:23:11 INFO Metadata cache refreshed recently.
Dec 30 13:23:11 DDEBUG Cleaning up.
We can tail the last n lines of a file and keep checking every 1 second for any changes with the -f flag (“f” for “follow”):
root:/var/log> tail -3f dnf.log -> show last 3 lines and any added afterwards
Dec 30 13:23:11 DEBUG Making cache files for all metadata files.
Dec 30 13:23:11 INFO Metadata cache refreshed recently.
Dec 30 13:23:11 DDEBUG Cleaning up.
.
root:/var/log> tail -n 3 -f –sleep-interval=2 dnf.log -> change interval check to 2 secs
Dec 30 13:23:11 DEBUG Making cache files for all metadata files.
Dec 30 13:23:11 INFO Metadata cache refreshed recently.
Dec 30 13:23:11 DDEBUG Cleaning up.
.
If the file we are tailing is renamed, as the file name descriptor should remain the same, the tailing should go on indefinitely. If we do not want that…
root:/var/log> tail -n 3 –follow=name –sleep-interval=2 dnf.log -> follow the filename and not the descriptor
Dec 30 13:23:11 DEBUG Making cache files for all metadata files.
Dec 30 13:23:11 INFO Metadata cache refreshed recently.
Dec 30 13:23:11 DDEBUG Cleaning up.
.
We can set the tail command to NOT fail even if the file cannot be read with the “–retry” option. This is useful when the file to tail has not been yet created or is not available for some interim reason. Furthermore, we can set the tail to end when some process is terminated with the “–pid” option.
root:/var/log> tail -n 3 –follow=name –sleep-interval=2 –retry –pid=28421 dnf.log
Another file creation command is touch. This command is most often used to create empty regular files or to update its access & modification times. But it can do a few more things:
root:/tmp> touch /tmp/test123
root:/tmp> ls -l /tmp/test123
-rw-r–r–. 1 root root 0 Oct 9 11:34 /tmp/test123
.
root:/tmp> touch -a /tmp/test123 → change only access time
root:/tmp> ls -l /tmp/test123
-rw–r–r–. 1 root root 0 Oct 9 11:34 /tmp/test123
.
root:/tmp> touch -m /tmp/test123 → change only modification time
root:/tmp> ls -l /tmp/test123
-rw-r–r–. 1 root root 0 Oct 9 11:37 /tmp/test123
.
root:/tmp> touch -m --date=”20040227 14:19:13″ /tmp/test123 → set explicit mtime
root:/tmp> ls -l /tmp/test123
-rw-r–r–. 1 root root 0 Feb 27 2004 /tmp/test123
.
root:/tmp> touch -a --date=”20040227 14:19:13″ /tmp/test123 → set explicit atime
root:/tmp> ls -l /tmp/test123
-rw-r–r–. 1 root root 0 Feb 27 2004 /tmp/test123
.
root:/tmp> touch --date=”20040227 14:19:13″ /tmp/test123 → set explicit atime & mtime
root:/tmp> ls -l /tmp/test123
-rw-r–r–. 1 root root 0 Feb 27 2004 /tmp/test123
.
root:/tmp> rm -f /tmp/test456
root:/tmp> touch -c /tmp/test456 → change atime & mtime if file exists or exit
root:/tmp> ls -l /tmp/test456
ls: cannot access /tmp/test456: No such file or directory
As is the case with touch, we can create empty files with the utility xfs_mkfile. But we can also give those new files any size in bytes, mbytes, gbytes or OS blocks.
# xfs_mkfile /tmp/testfile → let’s create a new file
# ls -l /tmp/testfile → didn’t work because we did not specify its size!
ls: cannot access ‘/tmp/testfile’: No such file or directory
.
# xfs_mkfile 0b testfile0 → let’s do it again with size 0 blocks
# xfs_mkfile 512 testfile1 → 512 bytes (minimum)
# xfs_mkfile 2b testfile2 → 2 OS blocks (4096 bytes)
# xfs_mkfile 2m testfile3 → 2 mbytes
# xfs_mkfile 1g testfile4 → 1 gigabyte
# ls -l testfile*
-rw——-. 1 root root 0 Apr 9 20:00 testfile0
-rw——-. 1 root root 512 Apr 9 20:09 testfile1
-rw——-. 1 root root 8192 Apr 9 20:09 testfile2
-rw——-. 1 root root 2097152 Apr 9 20:09 testfile3
-rw——-. 1 root root 1073741824 Apr 9 20:10 testfile4
The default behaviour of xfs_mkfile is to zero-out the whole file and that might take some time for large files. If we want to save time and do not care about initialising all the blocks, we can use the “-n” flag:
# xfs_mkfile -n -256m testfile4
With the “-n” we are skipping all the uninitialised blocks and just write one block at the end of the file.
Another very useful and used command is ln which is used to create hard & soft links between files and directories. Just as a reminder, a hard link is a pointer to an inode in disk with the actual data. Every file has at least one hard link or it becomes an orphan inode (placed in the lost+found directory). By creating an extra hard link we are creating another pointer to the file’s inode an incrementing the link count by 1.
root:/tmp> ls -l test.1
-rw-r–r–. 1 marc marc 7 Oct 8 10:59 test.1
.
root:/tmp> ln test.1 tmp1/test.2
root:/tmp> ls -l test.1
-rw-r–r–. 2 marc marc 7 Oct 8 10:59 test.1
.
root:/tmp> ln test.1 tmp2/test.3
root:/tmp> ls -l test.1
-rw-r–r–. 3 marc marc 7 Oct 8 10:59 test.1
Logically, we will only delete a file for good when we delete all the hard links to it!
A symbolic link on the other hand is nothing more than a pointer to the original pointer to the inode. So by deleting the original hard link we will be deleting the actual file and breaking the symbolic links. The main advantages of symbolic links is that they can span different filesystems and they can point to hard links that do not yet exist.
To create a symbolic link we have to add the “-s” flag:
root:/tmp/tmp1> ls -l
total 568
-rw-r–r–. 2 root root 55 Oct 9 13:02 2resolv.txt
-rwxr-xr-x. 2 root root 624 Oct 9 13:02 bin.lst
-rwxr-xr-x. 2 root root 288 Oct 9 13:02 monitoring.lst
-rw-r–r–. 2 root root 319721 Oct 9 13:02 ssh.pdf
-rw-r–r–. 2 root root 25007 Oct 9 13:02 strace.log
.
root:/tmp/tmp1> cd ../tmp2
root:/tmp/tmp2> ls -l
total 0
.
root:/tmp/tmp2> ln -s ../tmp1/* .
root:/tmp/tmp2> ls -l
total 0
lrwxrwxrwx. 1 root root 19 Oct 9 13:07 2resolv.txt > ../tmp1/2resolv.txt
lrwxrwxrwx. 1 root root 15 Oct 9 13:07 bin.lst > ../tmp1/bin.lst
lrwxrwxrwx. 1 root root 22 Oct 9 13:07 monitoring.lst > ../tmp1/monitoring.lst
lrwxrwxrwx. 1 root root 15 Oct 9 13:07 ssh.pdf > ../tmp1/ssh.pdf
lrwxrwxrwx. 1 root root 18 Oct 9 13:07 strace.log > ../tmp1/strace.log
The last 3 file creation commands are used far less often but can come in very handy in specific situations.
The first one of these is mkfifo which is used to create FIFO pipes for inter-process communication. To see how it works let’s show a simple example:
# create pipe with explicit permissions
root:/home/marc> mkfifo -m 0700 /var/tmp/fifo1
.
# check file type and permissions
root:/home/marc> ls -l /var/tmp/fifo1
prwx——. 1 root root 0 Oct 9 10:35 /var/tmp/fifo1
.
# from another process we send input to the pipe
root:/home/marc> echo “Input from another process sent through FIFO pipe!” >> /var/tmp/fifo1
.
# and we can read it from the other side of it
root:/home/marc> tail -f /var/tmp/fifo1
Input from another process sent through FIFO pipe!
We can have one or more processes using the pipe as their standard output and one reading from it. We can also have 2 processes communicating with each other reading and writing to it. As its name implies a pipe is nothing more than a file that 2 or more processes can use as stdin/stdout to communicate in a synchronous manner.
We can also use the command mknod to create FIFO pipes…
root:/tmp> mknod -m 0700 /var/tmp/fifo2 p
root:/tmp> ls -l /var/tmp/fifo*
prwx——. 1 root root 0 Oct 9 10:36 /var/tmp/fifo1
prwx——. 1 root root 0 Oct 9 10:51 /var/tmp/fifo2
… but also block and character devices. In normal circumstances we would not need to manually create block/character devices but sometimes (i.e. with Oracle RAC we might want to create device aliases with certain names linked to existing devices) there is no option.
# create character device /dev/ora_db1_raw1 pointing to the existing device
# with major number 43 and minor number 202
root:/tmp> mknod /dev/ora_db1_raw1 c 43 202
.
# create block device /dev/xvda pointing to device 67:109
root:/tmp> mknod /dev/xvda b 67 109
At times we do need a temporary file or directory whose name is irrelevant and that’s where mktemp comes into the picture.
root:/tmp> mktemp
/tmp/tmp.O7sV4FI8Ms
Without any arguments the mktemp command creates a temporary file in the $TMPDIR directory (if set) or /tmp , with u+rw permissions minus umask restrictions and a name like tmp.XXXXXXXXXX that is shown upon creation. We can change the naming pattern, the destination directory and whether it is a file or a directory that is created:
root:/tmp> export TMPDIR=/var/tmp
root:/tmp> mktemp
/var/tmp/tmp.dIKZQCHGiV
.
# create temp file with the pattern app9.XXXXXXXXXX.tmp in /var/run.
root:/tmp> mktemp -p /var/run app9.XXXXXXXXXX.tmp
/var/run/app9.fPzdUpogZR.tmp
.
# create temp directory with the pattern app9.XXXXXXXXXX.tmp in /var/run
root:/tmp> mktemp -d -p /var/run app9.XXXXXXXXXX.dir
/var/run/app9.zIp6xkoRxH.dir
Finally, to create directories we use mkdir:
root:/tmp> mkdir logs -> create directory logs with default mask in current directory
root:/tmp> mkdir /tmp/archive -> create directory archive given full path
root:/tmp> mkdir -p /var/opt/oracle -> create full path in one go if required
root:/tmp> mkdir -m 700 old-logs -> create directory with 700 permissions