edgecase
The unplanned organism is a question asked by Nature and answered by death.
~ Morpheus, Deus Ex
Author: StJohn Piano
Published: 2018-01-15
Datafeed Article 30
This article has been digitally signed by Edgecase Datafeed.
5629 words - 2765 lines - 70 pages



GOAL:



List the tools that are available on Centos 6.9 Minimal.



CONTENTS:



- Goal
- Contents
- Downloadable Assets
- Project Log



DOWNLOADABLE ASSETS:



available_tools_on_centos_69_minimal.txt

display_bytes.py

available_tools_on_centos_69_minimal.txt is a list of the available tools (commands, aliases, builtins, keywords) in the bash shell on Centos 6.9 Minimal. It can be searched using e.g.
grep
.

display_bytes.txt is a Python 2.7 program that reads a file (specified in the code in the variable
file_path
) and displays the contents as ASCII characters with some substitutions that facilitate reading whitespace and unprintable character bytes.



PROJECT LOG:



I recently succeeded in installing Centos 6.9 Minimal on Kalkin.


I would now like to know what tools are built into the bash shell on the Centos 6.9 Minimal operating system.


Press the power button to boot Kalkin.


Screen:

Booting from local disk...

Intel(R) Boot Agent GE v1.3.63
Copyright (C) 1997-2010, Intel Corporation

Intel(R) Boot Agent PXE Base Code (PXE-2.1 build 089)
Copyright (C) 1997-2010, Intel Corporation

Initializing and establishing link...



Screen:

Booting from local disk...

Intel(R) Boot Agent GE v1.3.63
Copyright (C) 1997-2010, Intel Corporation

Intel(R) Boot Agent PXE Base Code (PXE-2.1 build 089)
Copyright (C) 1997-2010, Intel Corporation

PXE-E61: Media test failure, check cable
PXE-M0F: Exiting Intel Boot Agent.



Screen:

Non-System disk or disk error
replace and strike any key when ready



Hm.

Press the power button to turn Kalkin off.

Press the power button to boot Kalkin.

Centos boots.


Reboot five more times.

Centos boots each time.

Odd.



Press and hold the power button to turn Kalkin off.

Press the power button to boot Kalkin.


Screen:

Press any key to enter the menu

Booting CentOS 6 (2.6.32-696.e16.x86_64) in 3 seconds...



[A booting screen is displayed]


Screen:

CentOS release 6.9 (Final)
Kernel 2.6.32-696.el6.x86_64 on an x86_64

kalkin login: root
Password:
Last login: Thu Jan 4 00:43:02 on tty1
[root@kalkin ~]#


[root@kalkin ~]# pwd

/root

[root@kalkin ~]# ls -1

anaconda-ks.cfg
install.log
install.log.syslog



I'd like to record bash commands and their output into a text file. I could then mount a USB memory stick and copy this data directly into this project log, instead of transcribing it.


Google "record terminal output".


Fourth result:
askubuntu.com/questions/161935/how-do-i-log-all-input-and-output-in-a-terminal-session


Excerpt:

Q:

How do I keep logs of all the work done via terminal? For example, I want a record of all the work done on database via terminal.

edited Jul 10 '12 at 12:31
Eliah Kagan

asked Jul 10 '12 at 10:58
Ankit


A:

When you are ready to start recording a log file, type:

script screen.log


Now, until you stop the script, all input and output in the Terminal will be stored in screen.log. When you are done, just type:

exit


Your screen.log file will stored in the local directory. If you want to redirect it, use an absolute pathname such as ~/screen.log. This will do exactly what you are looking for.

Source: Ubuntu Guide - How To Log Terminal Commands [link]

edited Nov 1 '16 at 23:34
Alan Thompson

answered Jul 10 '12 at 11:21
xlukasx



There's no guarantee that something from Ubuntu will work on Centos, but in principle the idea of the script utility is simple and useful enough that it's a good bet that it will exist on many versions of Linux. It probably predates Ubuntu.


[root@kalkin ~]# which script

/usr/bin/script

[root@kalkin ~]# script --version

script (util-linux-ng 2.17.2)

[root@kalkin ~]# script screen.log

Script started, file is screen.log

[root@kalkin ~]# cat screen.log


[root@kalkin ~]# exit

exit
Script done, file is screen.log

[root@kalkin ~]# cat screen.log

Script started on Fri 12 Jan 2018 01:16:45 PM GMT
[root@kalkin ~]# cat screen.log
[root@kalkin ~]# exit
exit

Script done on Fri 12 Jan 2018 01:17:10 PM GMT



Excellent.

I'll mount a memory stick and transfer this data.


Insert memory stick (Sandisk, 8 GB) into Kalkin.


[root@kalkin ~]# ls /mnt


[root@kalkin ~]#
usb 2-1.6: device descriptor read/64, error -110
usb 2-1.6: device descriptor read/64, error -110
usb 2-1.6: device descriptor read/8, error -110
usb 2-1.6: device descriptor read/8, error -110
usb 2-1.6: device descriptor read/8, error -110
usb 2-1.6: device descriptor read/8, error -110
hub 2-1:1.0: unable to enumerate USB device on port 6


Output hangs. Press enter. New prompt appears.


Hm.

Remove memory stick.

Insert memory stick into Aineko (2008 Macbook).

Open Disk Utility.

Disk Utility reports that the memory stick's partition map scheme is Master Boot Record and its format is MS-DOS (FAT32).

Eject and remove memory stick from Aineko.


Hm.

Let's try with a different memory stick (Kingston, 32 GB).

Insert memory stick into Aineko (2008 Macbook).

Open Disk Utility.

Disk Utility reports that the Kingston memory stick's partition map scheme is Master Boot Record and its format is MS-DOS (FAT32).

Eject and remove memory stick from Aineko.


Insert this memory stick into Kalkin.

[root@kalkin ~]#
sd 9:0:0:0: [sdg] Assuming drive cache: write through
sd 9:0:0:0: [sdg] Assuming drive cache: write through
sd 9:0:0:0: [sdg] Assuming drive cache: write through


Output hangs. Press enter. New prompt appears.


[root@kalkin ~]# fdisk -l


Disk /dev/sda: 2000.4 GB, 2000398934016 bytes
255 heads, 63 sectors/track, 243201 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x000605ad

Device Boot Start End Blocks Id System /dev/sda1 * 1 64 512000 83 Linux Partition 1 does not end on cylinder boundary. /dev/sda2 64 243202 1953001472 8e Linux LVM


Disk /dev/mapper/vg_kalkin-lv_root: 53.7 GB, 53687091200 bytes
255 heads, 63 sectors/track, 6257 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x00000000


Disk /dev/mapper/vg_kalkin-lv_swap: 8413 MB, 8413773824 bytes
255 heads, 63 sectors/track, 1022 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x00000000


Disk /dev/mapper/vg_kalkin-lv_home: 1937.8 GB, 1937768448000 bytes
255 heads, 63 sectors/track, 235586 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x00000000


Disk /dev/sdg: 31.1 GB, 31051513856 bytes
255 heads, 63 sectors/track, 3775 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0xaac1950c

Device Boot Start End Blocks Id System /dev/sdg1 * 1 3776 30323712 c W95 FAT32 (LBA)



Disk /dev/sdg is the expected size (~32 GB) for the memory stick.

Mount the memory stick.

[root@kalkin ~]# mkdir /mnt/test1


[root@kalkin ~]# mount /dev/sdg1 /mnt/test1


[root@kalkin ~]# ls -1 /mnt

test1

[root@kalkin ~]# ls -1 /mnt/test1

CentOS-6.9-x86_64-minimal.iso


The file CentOS-6.9-x86_64-minimal.iso is left over from a previous project.

[root@kalkin ~]# rm /mnt/test1/CentOS-6.9-x86_64-minimal.iso

rm: remove regular file `/mnt/test1/CentOS-6.9-x86_64-minimal.iso'? y

[root@kalkin ~]# cp screen.log /mnt/test1/


[root@kalkin ~]# ls -1 /mnt/test1

screen.log

[root@kalkin ~]# umount /dev/sdg1



Remove the memory stick.

Insert the stick into Aineko.

It mounts automatically and an icon appears on the Desktop.

Open screen.log.

It's empty.


Hm.


Eject and remove the memory stick from Aineko.

Insert the memory stick into Kalkin.

[root@kalkin ~]#
sd 10:0:0:0: [sdg] Assuming drive cache: write through
sd 10:0:0:0: [sdg] Assuming drive cache: write through
sd 10:0:0:0: [sdg] Assuming drive cache: write through


Output hangs. Press enter. New prompt appears.


[root@kalkin ~]# mount /dev/sdg1 /mnt/test1


[root@kalkin ~]# ls -1 /mnt/test1

screen.log

[root@kalkin ~]# cat /mnt/test1/screen.log



Why has the contents of screen.log not been preserved?

I'll try copying to a specific filename instead of targeting a directory.


[root@kalkin ~]# cp screen.log /mnt/test1/foo.txt


[root@kalkin ~]# cat /mnt/test1/foo.txt

Script started on Fri 12 Jan 2018 01:16:45 PM GMT
[root@kalkin ~]# cat screen.log
[root@kalkin ~]# exit
exit

Script done on Fri 12 Jan 2018 01:17:10 PM GMT

[root@kalkin ~]# umount /dev/sdg1


[root@kalkin ~]# mount /dev/sdg1 /mnt/test1


[root@kalkin ~]# cat /mnt/test1/foo.txt

Script started on Fri 12 Jan 2018 01:16:45 PM GMT
[root@kalkin ~]# cat screen.log
[root@kalkin ~]# exit
exit

Script done on Fri 12 Jan 2018 01:17:10 PM GMT



Hm. It looks as though the data has been successfully copied onto the disk.


[root@kalkin ~]# umount /dev/sdg1



Remove the memory stick from Kalkin.

Insert the memory stick into Aineko.

It mounts automatically and an icon appears on the Desktop.


Open foo.txt.

Contents (as shown in TextWrangler):

Script started on Fri 12 Jan 2018 01:16:45 PM GMT
[root@kalkin ~]# cat screen.log

[root@kalkin ~]# exit

exit


Script done on Fri 12 Jan 2018 01:17:10 PM GMT


I don't why there are extra newlines.


Let's look at the raw bytes in foo.txt.


I'll write some Python code to do this.


Display Bytes in Python

python 2.7.13

[download this code]
#!/opt/local/bin/python

file_path = "foo.txt"

f = open(file_path)
data = f.read()
f.close()

display_string=""

for byte in data:
	
	# get the decimal integer value of the byte
	n = ord(byte)
	
	# these are the byte values for the printable characters of ASCII plus the newline, the tab, and the space.
	allowed_values = [9,10,32] + range(33,127)
	
	if n not in allowed_values:
		# non-printable ASCII bytes
		display_string += "[ord_%d]" % n
		
	elif n in [9,10,32]:
		# whitespace bytes
		if n == 9: 
			# tab byte
			display_string += "[tab]\t"
		elif n == 10:
			# new line byte
			display_string += "[newline]\n"
		elif n == 32:
			display_string += " "

	else:
		# printable ASCII characters
		char = byte
		display_string += char

print display_string



Move foo.txt to Downloads directory.

Save the Python code in the Downloads directory as display_bytes.py.


aineko:~ stjohnpiano$ which python

/opt/local/bin/python

aineko:~ stjohnpiano$ python --version

Python 2.7.13

aineko:~ stjohnpiano$ cd Downloads


aineko:Downloads stjohnpiano$ chmod 700 display_bytes.py


aineko:Downloads stjohnpiano$ ./display_bytes.py

Script started on Fri 12 Jan 2018 01:16:45 PM GMT[newline]
[root@kalkin ~]# cat screen.log[ord_13][newline]
[root@kalkin ~]# exit[ord_13][newline]
exit[ord_13][newline]
[newline]
Script done on Fri 12 Jan 2018 01:17:10 PM GMT[newline]



There are [ord_13] bytes in all the locations where I saw an extra newline in TextWrangler's rendering of foo.txt.

Looking at my ASCII reference table, I find that byte 13 is CR (carriage return). Note: The carriage return is often typed in e.g. Python strings as "\r".

Evidently TextWrangler displays carriage returns while
cat
ignores them.

I'm slightly surprised that
script
outputs carriage return bytes.


Delete foo.txt and screen.log from the memory stick.

Delete foo.txt from the Downloads directory on Aineko.

Eject and remove the memory stick from Aineko.


I'll stop here for today.

Shut down Kalkin.

[root@kalkin ~]# halt



Another day, another crack at the problem.


Press the power button to boot Kalkin.


Screen:

Booting from local disk...

Intel(R) Boot Agent GE v1.3.63
Copyright (C) 1997-2010, Intel Corporation

Intel(R) Boot Agent PXE Base Code (PXE-2.1 build 089)
Copyright (C) 1997-2010, Intel Corporation

Initializing and establishing link...



Screen:

Booting from local disk...

Intel(R) Boot Agent GE v1.3.63
Copyright (C) 1997-2010, Intel Corporation

Intel(R) Boot Agent PXE Base Code (PXE-2.1 build 089)
Copyright (C) 1997-2010, Intel Corporation

PXE-E61: Media test failure, check cable
PXE-M0F: Exiting Intel Boot Agent.



Screen:

Non-System disk or disk error
replace and strike any key when ready



Hm.

Press Enter.

Screen:

Non-System disk or disk error
replace and strike any key when ready



Press the power button to turn Kalkin off.


Press the power button to boot Kalkin.

Centos boots.

Press and hold the power button to turn Kalkin off.


Reboot four more times.

Centos boots each time.


It's as if the first boot changes a setting somewhere, which is stored in a capacitor, the charge in which then fades when the computer is off for a reasonably long period of time.


Press and hold the power button to turn Kalkin off.

Press the power button to boot Kalkin.


Screen:

Press any key to enter the menu

Booting CentOS 6 (2.6.32-696.e16.x86_64) in 3 seconds...



[A booting screen is displayed]


I want to look at this problem again:
- Copying a file to a target directory (instead of a target filename) on a mounted memory stick results in an empty 0-byte file.


Screen:

CentOS release 6.9 (Final)
Kernel 2.6.32-696.el6.x86_64 on an x86_64

kalkin login: root
Password:
Last login: Thu Jan 4 00:43:02 on tty1
[root@kalkin ~]#


[root@kalkin ~]# ls -1

anaconda-ks.cfg
install.log
install.log.syslog
screen.log

[root@kalkin ~]# rm screen.log

rm: remove regular file `screen.log'? y

[root@kalkin ~]# echo "hello world" > bar.txt


[root@kalkin ~]# cat bar.txt

hello world


Insert memory stick (Kingston, 32 GB) into Kalkin.


[root@kalkin ~]#
sd 9:0:0:0: [sdg] Assuming drive cache: write through
sd 9:0:0:0: [sdg] Assuming drive cache: write through
sd 9:0:0:0: [sdg] Assuming drive cache: write through


Output hangs. Press enter. New prompt appears.


I'd like to find the device identifier of the inserted memory stick via reading something more concise than the output of
fdisk -l
.


While installing Centos 6.9 Minimal on Kalkin, I discovered the
blkid
command.

[root@kalkin ~]# blkid

/dev/sda1: UUID="e0b18dd5-f392-42ca-bbf1-89a1a0229e38" TYPE="ext4"
/dev/sda2: UUID="rWbUY4-wwiQ-J2Vp-n0F5-bngF-G8yb-HWQyEv" TYPE="LVM2_member"
/dev/mapper/vg_kalkin-lv_root: UUID="508692c4-54da-4a93-b2e2-6f1584e74afe" TYPE="ext4"
/dev/mapper/vg_kalkin-lv_swap: UUID="707ee542-6bfe-453d-b43c-bcfc58c59b6f" TYPE="swap"
/dev/mapper/vg_kalkin-lv_home: UUID="6134733c-158e-4d62-a74d-e7d00e2ed8fd" TYPE="ext4"
/dev/sdg1: LABEL="KINGSTON" UUID="7195-020C" TYPE="vfat"


The Kingston stick has the identifier
/dev/sdg1
.

[root@kalkin ~]# mount /dev/sdg1 /mnt/test1


[root@kalkin ~]# ls -1 /mnt/test1


[root@kalkin ~]# cp bar.txt /mnt/test1


[root@kalkin ~]# cat /mnt/test1/bar.txt

hello world



Hm. Ah, maybe the trailing slash on the target directory makes a difference. I used one before when the
cp
command produced a 0-byte file.


[root@kalkin ~]# cp bar.txt bar2.txt


[root@kalkin ~]# cp bar2.txt /mnt/test1/


[root@kalkin ~]# cat /mnt/test1/bar2.txt

hello world



Hm.

[root@kalkin ~]# umount /dev/sdg1



Remove the memory stick.

Insert it into Aineko.

It mounts automatically and an icon appears on the Desktop.


aineko:~ stjohnpiano$ ls -1 /Volumes

KINGSTON
Macintosh HD
Untitled

aineko:~ stjohnpiano$ cd /Volumes/KINGSTON


aineko:KINGSTON stjohnpiano$ ls


aineko:KINGSTON stjohnpiano$ ls -1a

.
..
.Spotlight-V100
.Trashes
._.Trashes
.fseventsd



Hm. Neither file is present.

Eject and remove the memory stick from Aineko.


Insert the memory stick into Kalkin.

[root@kalkin ~]#
sd 10:0:0:0: [sdg] Assuming drive cache: write through
sd 10:0:0:0: [sdg] Assuming drive cache: write through
sd 10:0:0:0: [sdg] Assuming drive cache: write through


Output hangs. Press enter. New prompt appears.


Running
blkid
shows me that the identifier is again
/dev/sdg1
.


[root@kalkin ~]# mount /dev/sdg1 /mnt/test1


[root@kalkin ~]# ls /mnt/test1



Neither file is present.

Hm. Let's try again, but this time I'll dismount it and remount it without removing it from Kalkin.

Hm. [some reading occurs here]. Wait a second, I've been using
umount
incorrectly. One should target the mountpoint directory, not the device identifier.

[root@kalkin ~]# cat bar.txt

hello world

[root@kalkin ~]# cp bar.txt /mnt/test1


[root@kalkin ~]# cat /mnt/test1/bar.txt

hello world

[root@kalkin ~]# umount /mnt/test1


[root@kalkin ~]# mount /dev/sdg1 /mnt/test1


[root@kalkin ~]# ls /mnt/test1

bar.txt

[root@kalkin ~]# cat /mnt/test1/bar.txt

hello world



It looks like the problem was caused by incorrect use of
umount
.


[root@kalkin ~]# umount /mnt/test1


Remove the memory stick from Kalkin.

Insert the memory stick into Aineko.

It mounts automatically and an icon appears on the Desktop.


aineko:KINGSTON stjohnpiano$ ls


aineko:KINGSTON stjohnpiano$ ls

bar.txt

aineko:KINGSTON stjohnpiano$ cat bar.txt

hello world



Excellent.


Eject and remove memory stick from Aineko.

Insert memory stick into Kalkin.


[root@kalkin ~]#
sd 11:0:0:0: [sdg] Assuming drive cache: write through
sd 11:0:0:0: [sdg] Assuming drive cache: write through
sd 11:0:0:0: [sdg] Assuming drive cache: write through


Output hangs. Press enter. New prompt appears.


[root@kalkin ~]# mount /dev/sdg1 /mnt/test1


[root@kalkin ~]# ls /mnt/test1

bar.txt

[root@kalkin ~]# cat /mnt/test1/bar.txt

hello world



When I was checking Kalkin's hardware details using a Centos 6.9 bootable USB stick, I discovered the
compgen
command, a bash builtin, which generates possible completion matches for words in the bash shell. If no words are provided, and it is given at least one option, it will list all possible words (given that option).


I'm going to run
compgen
with various options in order to gather lists of the available commands on Centos 6.9 Minimal.


I will use the memory stick to transfer the
compgen
results, recorded via
script
, to Aineko for inclusion in this log.


[root@kalkin ~]# rm bar.txt bar2.txt

rm: remove regular file `bar.txt'? y
rm: remove regular file `bar2.txt'? y

[root@kalkin ~]# cd /home


[root@kalkin home]# mkdir work


[root@kalkin home]# cd work


[root@kalkin work]# compgen



[a shutdown occurs here. I've rerun the appropriate commands and arrived back at this point.]


[root@kalkin work]# script work.log



[various commands are run here.
compgen
output will be pasted into this log later.]


[root@kalkin work]# rm /mnt/test1/bar.txt

rm: remove regular file `bar.txt'? y

[root@kalkin work]# cp work.log /mnt/test1


[root@kalkin work]# ls /mnt/test1

work.log

[root@kalkin work]# umount /mnt/test1



Remove the memory stick from Kalkin.

Insert the stick into Aineko.

It mounts automatically and an icon appears on the Desktop.


The file work.log is present but it's 0 KB i.e. empty.

Hm.

Eject and remove memory stick from Aineko.

Insert the stick into Kalkin.


[root@kalkin work]#
sd 10:0:0:0: [sdg] Assuming drive cache: write through
sd 10:0:0:0: [sdg] Assuming drive cache: write through
sd 10:0:0:0: [sdg] Assuming drive cache: write through


Output hangs. Press enter. New prompt appears.


I'll copy to a filename rather than to a directory.

[root@kalkin work]# cp work.log /mnt/test1/work.log

cp: overwrite `/mnt/test1/work.log'? y

[root@kalkin work]# head -1 /mnt/test1/work.log

Script started on Mon 15 2018 02:10:00 AM GMT

[root@kalkin work]# cat /mnt/test1/work.log | wc -l

9072

[root@kalkin work]# umount /mnt/test1



Remove the memory stick from Kalkin.

Insert the stick into Aineko.

It mounts automatically and an icon appears on the Desktop.


work.log is again 0 KB.


Looking back at this log, it appears that when I followed this sequence:
- unmount memory stick on Kalkin
- mount memory stick on Kalkin
- unmount memory stick on Kalkin
and only then removed the memory stick and inserted it into Aineko, the data was preserved. Odd.

Let's try that again.


Eject and remove memory stick from Aineko.

Insert the stick into Kalkin.


[root@kalkin work]#
sd 11:0:0:0: [sdg] Assuming drive cache: write through
sd 11:0:0:0: [sdg] Assuming drive cache: write through
sd 11:0:0:0: [sdg] Assuming drive cache: write through


Output hangs. Press enter. New prompt appears.


[root@kalkin work]# mount /dev/sdg1 /mnt/test1


[root@kalkin work]# ls /mnt/test1

work.log

[root@kalkin work]# cat /mnt/test1/work.log


[root@kalkin work]# cp work.log /mnt/test1/work.log

cp: overwrite `/mnt/test1/work.log'? y

[root@kalkin work]# cat /mnt/test1/work.log | wc -l

9072

[root@kalkin work]# umount /mnt/test1


[root@kalkin work]# mount /dev/sdg1 /mnt/test1


[root@kalkin work]# ls /mnt/test1

work.log

[root@kalkin work]# cat /mnt/test1/work.log | wc -l

9072

[root@kalkin work]# umount /mnt/test1



Remove the memory stick from Kalkin.

Insert the stick into Aineko.

It mounts automatically and an icon appears on the Desktop.


work.log is present and its size is 131 KB.

The unmount and remount step seems to do something that causes the data to be preserved on the memory stick.



aineko:KINGSTON stjohnpiano$ ls

work.log



On opening the file in TextWrangler, I see the extra newlines caused by the inclusion of carriage return bytes.


I'd like to remove the carriage return bytes from the output.


Google "count occurrences of particular byte".

Second result:
stackoverflow.com/questions/1603566/count-occurrences-of-a-char-in-plain-text-file


Excerpt:

Q:

Is there any way under linux/terminal to count, how many times the char f occurs in a plain text file?

edited Oct 21 '09 at 21:24
Rob Hruska

asked Oct 21 '09 at 20:56
cupakob


A:

[...]

tr -cd f < file | wc -c


[...]

edited Jan 29 '13 at 10:05
erik

answered Jan 17 '13 at 0:33
user1985553


Comment:

To count several characters, e.g.
a
,
b
and
c
:
tr -cd abc < file | wc -l
.

Skippy le Grand Gourou Apr 3 '17 at 13:26




Well, I know from running
cat work.log | wc -l
that work.log is 9072 lines long and therefore contains 9072 newline bytes. This is a data point.

aineko:KINGSTON stjohnpiano$ tr -cd \n < work.log | wc -c

3116

aineko:KINGSTON stjohnpiano$ tr -cd \ < work.log | wc -c

482

aineko:KINGSTON stjohnpiano$ tr -cd n < work.log | wc -c

3116

aineko:KINGSTON stjohnpiano$ tr -cd "\n" < work.log | wc -c

9072


Cool. Evidently there's some rules about quoting and backslash characters.


aineko:KINGSTON stjohnpiano$ tr -cd a < work.log | wc -c

4032

aineko:KINGSTON stjohnpiano$ tr -cd b < work.log | wc -c

1431

aineko:KINGSTON stjohnpiano$ tr -cd ab < work.log | wc -c

5463


4032 + 1431 = 5463

I'm not sure why this addition didn't hold for \, n, and \n, (not quoted, so treated as separate bytes?) in the earlier commands. Probably backslash is special.


aineko:KINGSTON stjohnpiano$ tr -cd "\r" < work.log | wc -c

9069



9069 = 9072 - 3

This carriage return byte count matches the first script recording, in which the first line and the last two lines each had only a newline, instead of carriage return + newline.


aineko:KINGSTON stjohnpiano$ man tr



Excerpt from man page:

NAME

tr -- translate characters

SYNOPSIS

tr [-Ccsu] string1 string2
tr [-Ccu] -d string1
tr [-Ccu] -s string1
tr [-Ccu] -ds string1 string2

DESCRIPTION

The tr utility copies the standard input to the standard output with substitution or deletion of selected characters.

The following options are available:

-C
Complement the set of characters in string1, that is ``-C ab'' includes every character except for `a' and `b'.

-c
Same as -C but complement the set of values in string1.

-d
Delete characters in string1 from the input.

[...]

In the second synopsis form, the characters in string1 are deleted from the input.

[...]

The following conventions can be used in string1 and string2 to specify sets of characters:

character
Any character not described by one of the following conventions represents itself.

\character
A backslash followed by certain special characters maps to
special values.
\a <alert character>
\b <backspace>
\f <form-feed>
\n <newline>
\r <carriage return>
\t <tab>
\v <vertical tab>
A backslash followed by any other character maps to that character.

[...]

EXAMPLES

The following examples are shown as given to the shell:

Create a list of the words in file1, one per line, where a word is taken to be a maximal string of letters.

tr -cs "[:alpha:]" "\n" < file1

[...]

Strip out non-printable characters from file1.
tr -cd "[:print:]" < file1

[...]

BSD July 23, 2004




Hm. From the man page descriptions, -C and -c appear to be the same option. I suspect that the description of -C should actually be "include only characters in the set of values in string1".

In the examples, I note the use of double quotation marks around string1 and string2. Evidently they can be omitted if the input string consists of non-backspace-escaped characters.

No mention of how to target the backslash character when using
tr
.

Hm. "A backslash followed by any other character maps to that character." Perhaps in the command
tr -cd \ < work.log | wc -c
, the target string was actually "\ ", i.e. the backslash and also the space following it, which would then be treated as a space.


aineko:KINGSTON stjohnpiano$ tr -cd \ < work.log | wc -c

482

aineko:KINGSTON stjohnpiano$ tr -cd " " < work.log | wc -c

482


Yup. It counted spaces.

Further thoughts on "A backslash followed by any other character maps to that character":
- targeting \n without quotation marks actually counted the occurrences of n by itself, which is why the results for \n and n were the same.
- to target a backslash, I should probably use "\\".


In the command:
tr -cd "\r" < work.log

tr
proceeds through each line in the input (which in this case is the file work.log), copying the result of evaluating that line to the standard output.
-c
selects every byte that is not "\r".
-d
deletes every selected byte.

Adding
| wc -c
to this command uses the
wc
(which stands for "word count") utility to count each character that
tr
sent to the standard output.

So, in this result:

aineko:KINGSTON stjohnpiano$ tr -cd "\r" < work.log | wc -c

9069


I can see that the carriage return byte "\r" occurs 9069 times in work.log.


If I can correctly count a byte type, this means I can select it, which means I can delete it.

aineko:KINGSTON stjohnpiano$ wc -c work.log

124870 work.log


There are 124870 total in work.log.


I'll leave out the
-c
option, so instead
tr
should delete every instance of "\r" and copy all other bytes to the standard output.

aineko:KINGSTON stjohnpiano$ tr -d "\r" < work.log | wc -c

115801


124870 - 9069 = 115801

115801 is the number of bytes expected after removing all instances of "\r".


Finally, I'll redirect the output into a file. This file will be the same as work.log, except that it will not contain any carriage return bytes.

aineko:KINGSTON stjohnpiano$ tr -d "\r" < work.log > work2.log


aineko:KINGSTON stjohnpiano$ wc -c work2.log

115801 work2.log



Excellent.


Here are the relevant excerpts from work2.log, reordered and edited a bit.


There are a few strange renderings of the command inputs e.g.
compgen -a [Kc
and
\compgen -c | sort
. I'm not sure why these have occurred. They are in work.log as well as work2.log. I'll remove them manually.

I've rerun the
compgen
commands on Kalkin, piping the output into
less
, and checked that the output matches that shown here. I didn't find that any of the output was rendered differently, but it's good to be sure.


Descriptions of the
compgen
commands:

compgen -c
will list all the commands you could run.
compgen -a
will list all the aliases you could run.
compgen -b
will list all the builtins you could run.
compgen -k
will list all the keywords you could run.
compgen -A function
will list all the functions you could run.
compgen -A function -abck
will list all the above in one go.


Script started on Mon 15 Jan 2018 02:10:00 AM GMT


[root@kalkin work]# compgen --version

bash: compgen: --: invalid option
compgen: usage: compgen [-abcdefgjksuv] [-o option] [-A action] [-G globpat] [-W wordlist] [-F function] [-C command] [-X filterpat] [-P prefix] [-S suffix] [word]


[root@kalkin work]# compgen -a

cp
l.
ll
ls
mv
rm
which


[root@kalkin work]# compgen -b

.
:
[
alias
bg
bind
break
builtin
caller
cd
command
compgen
complete
compopt
continue
declare
dirs
disown
echo
enable
eval
exec
exit
export
false
fc
fg
getopts
hash
help
history
jobs
kill
let
local
logout
mapfile
popd
printf
pushd
pwd
read
readarray
readonly
return
set
shift
shopt
source
suspend
test
times
trap
true
type
typeset
ulimit
umask
unalias
unset
wait


[root@kalkin work]# compgen -k

if
then
else
elif
fi
case
esac
for
select
while
until
do
done
in
function
time
{
}
!
[[
]]
coproc


[root@kalkin work]# compgen -A function



[root@kalkin work]# compgen -c | sort

:
!
.
[
[
[[
]]
{
}
addftinfo
addgnupghome
addpart
addr2line
adduser
agetty
alias
alternatives
anacron
applygnupgdefaults
ar
arch
arp
arpd
arping
arping
as
attr
audispd
auditctl
auditd
augenrules
aulast
aulastlog
aureport
ausearch
ausyscall
authconfig
authconfig-tui
autrace
auvirt
avcstat
awk
awk
badblocks
base64
basename
bash
bashbug-64
berkeley_db_svc
bg
bind
blkdeactivate
blkdiscard
blkid
blockdev
brcm_iscsiuio
brctl
break
bridge
build-locale-archive
builtin
bunzip2
bzcat
bzcmp
bzdiff
bzgrep
bzip2
bzip2recover
bzless
bzmore
cacertdir_rehash
cache_check
cache_check
cache_dump
cache_dump
cache_metadata_size
cache_metadata_size
cache_repair
cache_repair
cache_restore
cache_restore
cal
ca-legacy
caller
capsh
captoinfo
case
cat
catchsegv
cbq
cciss_id
cd
certutil
cfdisk
c++filt
chacl
chage
chattr
chcon
chcpu
checkmodule
checkpolicy
chfn
chgrp
chkconfig
chmod
chown
chpasswd
chroot
chrt
chsh
chvt
cksum
clear
clock
clockdiff
cmp
cmsutil
col
colcrt
colrm
column
comm
command
compgen
complete
compopt
consoletype
continue
coproc
cp
cp
cpio
cracklib-check
cracklib-format
cracklib-packer
cracklib-unpacker
create-cracklib-dict
crlutil
crond
crontab
cryptsetup
csplit
ctrlaltdel
curl
cut
cut
dash
date
db_archive
db_checkpoint
db_codegen
db_deadlock
db_dump
db_dump185
db_hotbackup
db_load
db_printlog
db_recover
db_stat
db_upgrade
dbus-binding-tool
db_verify
dd
ddate
deallocvt
debugfs
declare
delpart
depmod
df
dhclient
dhclient-script
diff
diff3
dir
dircolors
dirname
dirs
disown
dmesg
dmeventd
dmsetup
dmstats
dnsdomainname
do
domainname
done
dracut
du
dumpe2fs
dumpkeys
e2freefrag
e2fsck
e2image
e2label
e2undo
echo
echo
efibootmgr
egrep
elif
else
enable
env
env
eqn
eqn2graph
era_check
era_check
era_dump
era_dump
era_invalidate
era_invalidate
era_restore
era_restore
esac
ether-wake
ethtool
ethtool
eval
ex
exec
exit
expand
export
expr
factor
faillock
fallocate
false
false
fc
fdformat
fdisk
fg
fgconsole
fgrep
fi
file
filefrag
find
find
findfs
findmnt
fipscheck
fipshmac
fixfiles
flock
floppy
fmt
fold
for
free
fsadm
fsck
fsck.cramfs
fsck.ext2
fsck.ext3
fsck.ext4
fsck.ext4dev
fsck.xfs
fsfreeze
fstab-decode
fstrim
function
fuser
fusermount
fusermount
gawk
gawk
gdbus
gencat
genhomedircon
genhostid
geqn
getcap
getconf
getenforce
getent
getfacl
getfattr
getkey
getkeycodes
getopt
getopts
getpcaps
getsebool
gindxbib
gio-querymodules-64
glib-compile-schemas
glibc_post_upgrade.x86_64
glookbib
gmake
gneqn
gnroff
gpasswd
gpg
gpg2
gpg-agent
gpgconf
gpg-connect-agent
gpg-error
gpgkey2ssh
gpgparsemail
gpgsplit
gpgv
gpgv2
gpg-zip
gpic
gprof
grefer
grep
grn
grodvi
groff
groffer
grog
grolbp
grolj4
grops
grotty
groupadd
groupdel
groupmems
groupmod
groups
grpck
grpconv
grpunconv
grub
grubby
grub-crypt
grub-install
grub-md5-crypt
grub-terminfo
gsoelim
gtar
gtbl
gtroff
gunzip
gunzip
gzexe
gzip
gzip
halt
hash
head
help
hexdump
history
hostid
hostname
hpftodit
hwclock
hwclock
i386
iconv
iconvconfig
iconvconfig.x86_64
id
idn
if
ifcfg
ifconfig
ifdown
ifenslave
ifup
igawk
in
indxbib
info
infocmp
infokey
infotocap
init
initctl
insmod
insmod.static
install
install-info
installkernel
ionice
ip
ip6tables
ip6tables-1.4.7
ip6tables-multi
ip6tables-multi-1.4.7
ip6tables-restore
ip6tables-restore-1.4.7
ip6tables-save
ip6tables-save-1.4.7
ipcalc
ipcmk
ipcrm
ipcs
ipmaddr
iptables
iptables-1.4.7
iptables-multi
iptables-multi-1.4.7
iptables-restore
iptables-restore-1.4.7
iptables-save
iptables-save-1.4.7
iptables-xml
iptables-xml-1.4.7
iptunnel
iscsiadm
iscsid
iscsi-iname
iscsistart
iscsiuio
isosize
jobs
join
kbd_mode
kbdrate
kill
kill
kill
killall
killall5
kpartx
l.
last
lastb
lastlog
lchage
lchfn
lchsh
ld
ldattach
ldconfig
ldd
less
lessecho
lesskey
lesspipe.sh
let
lgroupadd
lgroupdel
lgroupmod
lid
link
linux32
linux64
lkbib
ll
ln
lnewusers
lnstat
loadkeys
load_policy
load_policy
loadunimap
local
locale
localedef
logger
logger
login
logname
logout
logrotate
logsave
lokkit
look
lookbib
losetup
lpasswd
ls
ls
lsattr
lsblk
lscpu
lsinitrd
lslogins
lsmod
lua
luac
luseradd
luserdel
lusermod
lvchange
lvconvert
lvcreate
lvdisplay
lvextend
lvm
lvmchange
lvmconf
lvmconfig
lvmdiskscan
lvmdump
lvmetad
lvmsadc
lvmsar
lvreduce
lvremove
lvrename
lvresize
lvs
lvscan
m4
mailq
mailq.postfix
make
MAKEDEV
mapfile
mapscrn
matchpathcon
matchpathcon
mbchk
mcookie
md5sum
mdadm
mdmon
mesg
mii-diag
mii-tool
mingetty
mkdict
mkdir
mke2fs
mkfifo
mkfs
mkfs.cramfs
mkfs.ext2
mkfs.ext3
mkfs.ext4
mkfs.ext4dev
mkfs.xfs
mkhomedir_helper
mkinitrd
mklost+found
mknod
mksock
mkswap
mktemp
modinfo
modprobe
modutil
more
mount
mount.fuse
mountpoint
mount.tmpfs
mpathconf
mpathpersist
multipath
multipathd
mv
mv
namei
nameif
neqn
netreport
netstat
newaliases
newaliases.postfix
newgrp
new-kernel-pkg
newusers
nice
nisdomainname
nl
nm
nohup
nologin
nproc
nroff
nsenter
nstat
objcopy
objdump
od
oldfind
open
open_init_pty
openssl
openvt
p11-kit
packer
pam_console_apply
pam_tally2
pam_timestamp_check
partx
passwd
paste
pathchk
pcregrep
pcretest
pdata_tools
pdata_tools
peekfd
pfbtops
pgawk
pgrep
pic
pic2graph
pidof
pinentry
pinentry-curses
ping
ping6
ping6
pinky
pivot_root
pk12util
pkg-config
pkill
plipconfig
pluginviewer
plymouth
plymouth
plymouthd
plymouth-set-default-theme
pmap
popd
postalias
postcat
postconf
postdrop
postfix
post-grohtml
postkick
postlock
postlog
postmap
postmulti
postqueue
postsuper
poweroff
ppp-watch
pr
pre-grohtml
printenv
printf
printf
ps
psfaddtable
psfgettable
psfstriptable
psfxtable
pstree
pstree.x11
ptx
pushd
pvchange
pvck
pvcreate
pvdisplay
pvmove
pvremove
pvresize
pvs
pvscan
pwck
pwconv
pwd
pwd
pwdx
pwunconv
pydoc
python
python2
python2.6
raid-check
ranlib
raw
rdisc
read
readarray
readelf
readlink
readlink
readonly
readprofile
reboot
refer
reload
rename
renice
reset
resize2fs
resizecons
restart
restorecon
restorecond
return
rev
rhgb-client
rm
rm
rmail
rmail.postfix
rmdir
rmmod
route
rpcgen
rpm
rpm2cpio
rpmdb
rpmquery
rpmsign
rpmverify
rsyslogd
rtacct
rtcwake
rtmon
runcon
run_init
runlevel
run-parts
runuser
rvi
rview
saslauthd
sasldblistusers2
saslpasswd2
scp
script
scriptreplay
scsi_id
sdiff
secon
securetty
sed
sedismod
sedispol
select
selinuxconlist
selinuxdefcon
selinuxenabled
semodule
semodule_deps
semodule_expand
semodule_link
semodule_package
sendmail
sendmail.postfix
seq
service
sestatus
set
setarch
setcap
setenforce
setfacl
setfattr
setfiles
setfont
setkeycodes
setleds
setmetamode
setsebool
setsid
setsysfont
setterm
setup-nsssysinit.sh
sfdisk
sftp
sg
sh
sha1sum
sha224sum
sha256sum
sha384sum
sha512sum
shift
shopt
showconsolefont
showkey
shred
shuf
shutdown
signtool
signver
size
skill
slabtop
slattach
sleep
sln
slogin
smtp-sink
smtp-source
snice
soelim
sort
source
split
sprof
sqlite3
ss
ssh
ssh-add
ssh-agent
ssh-copy-id
sshd
ssh-keygen
ssh-keyscan
ssltap
start
start_udev
stat
status
stdbuf
stop
strings
strip
stty
su
sudo
sudoedit
sudoreplay
sulogin
sum
sushell
suspend
swapoff
swapon
switch_root
sync
sysctl
sys-unconfig
tabs
tac
tail
tailf
tar
taskset
tbl
tc
tee
telinit
test
test
testsaslauthd
tfmtodit
then
thin_check
thin_check
thin_delta
thin_delta
thin_dump
thin_dump
thin_ls
thin_ls
thin_metadata_size
thin_metadata_size
thin_repair
thin_repair
thin_restore
thin_restore
thin_rmap
thin_rmap
thin_trim
thin_trim
tic
time
timeout
times
tload
toe
togglesebool
top
touch
tput
tr
tracepath
tracepath
tracepath6
tracepath6
trap
troff
true
true
truncate
tset
tsort
tty
tune2fs
tunelp
type
typeset
tzdata-update
tzselect
udevadm
udevd
ul
ulimit
ulockmgr_server
ulockmgr_server
umask
umount
unalias
uname
unexpand
unicode_start
unicode_stop
uniq
unix_chkpwd
unix_update
unlink
unset
unshare
until
update-alternatives
update-ca-trust
update-mime-database
uptime
urlgrabber
useradd
userdel
usermod
usernetctl
users
usleep
utmpdump
uuidgen
vdir
vgcfgbackup
vgcfgrestore
vgchange
vgck
vgconvert
vgcreate
vgdisplay
vgexport
vgextend
vgimport
vgimportclone
vgmerge
vgmknodes
vgreduce
vgremove
vgrename
vgs
vgscan
vgsplit
vi
view
vigr
vipw
visudo
vmstat
w
wait
wall
watch
watchgnupg
wc
weak-modules
whereis
which
which
while
whiptail
who
whoami
wipefs
write
x86_64
xargs
xfs_admin
xfs_bmap
xfs_check
xfs_copy
xfs_db
xfs_estimate
xfs_freeze
xfs_fsr
xfs_growfs
xfs_info
xfs_io
xfs_logprint
xfs_mdrestore
xfs_metadump
xfs_mkfile
xfs_ncheck
xfs_quota
xfs_repair
xfs_rtcp
xmlcatalog
xmllint
xmlwf
yes
ypdomainname
yum
zcat
zcmp
zdiff
zdump
zegrep
zfgrep
zforce
zgrep
zic
zless
zmore
znew
zsoelim


Script done on Mon 15 Jan 2018 02:19:42 AM GMT



After listing the available commands with
compgen
, I then did some exploration of the filesystem, using e.g.
ls /usr/bin
, etc.

Unfortunately the output produced by
script
was rather strange, so I haven't included any here, except for the excerpt below that demonstrates the issue:

Excerpt:

[m[root@kalkin work]# ls -1 /bin [K [K [K

[0m [01;34mbin [0m
[01;34mboot [0m
[01;34mdev [0m
[01;34metc [0m
[01;34mhome [0m
[01;34mlib [0m
[01;34mlib64 [0m
[01;34mlost+found [0m
[01;34mmedia [0m
[01;34mmnt [0m
[01;34mopt [0m
[01;34mproc [0m
[01;34mroot [0m
[01;34msbin [0m
[01;34mselinux [0m
[01;34msrv [0m
[01;34msys [0m
[30;42mtmp [0m
[01;34musr [0m
[01;34mvar [0m
[m[root@kalkin work]# ls -1 /boot

config-2.6.32-696.el6.x86_64
[0m [01;34mefi [0m
[01;34mgrub [0m
initramfs-2.6.32-696.el6.x86_64.img
[01;34mlost+found [0m
[01;31msymvers-2.6.32-696.el6.x86_64.gz [0m
System.map-2.6.32-696.el6.x86_64
[01;32mvmlinuz-2.6.32-696.el6.x86_64 [0m




Last thing: Create a searchable list of the available tools (commands, aliases, builtins, keywords) in the bash shell on Centos 6.9 Minimal.

Manually copy the outputs of:
-
compgen -a

-
compgen -b

-
compgen -k

-
compgen -c

into a new file. Remove input lines and empty lines. Save it as tmp.txt in Downloads directory on Aineko.

I'll sort tmp.txt and remove duplicate entries.

aineko:KINGSTON stjohnpiano$ cd ~/Downloads


aineko:Downloads stjohnpiano$ ls -1

display_bytes.py
tmp.txt

aineko:Downloads stjohnpiano$ sort -u < tmp.txt > tmp2.txt


aineko:Downloads stjohnpiano$ mv tmp2.txt available_tools_on_centos_69_minimal.txt


aineko:Downloads stjohnpiano$ rm -f tmp.txt



I'll add available_tools_on_centos_69_minimal.txt and display_bytes.py as downloadable assets at the beginning of this article.


Tidy up:
- On the Kingston memory stick, delete work.log and work2.log.






[start of notes]



Changes from the original text:

- I have not always preserved the format of any excerpts from webpages on other sites (e.g. not preserving the original bold/italic styles, changing the list structures, not preserving hyperlinks).

- I have not always preserved the format of any computer output (e.g. from running bash commands). Examples: Setting input lines in bold text, adding/removing newlines in order to make a sequence of commands easier to read, using hyphens for lists and sublists instead of indentation, breaking wide tables into consecutive sections.


[end of notes]