Meeting Minutes - sh - Posix Shell Session - Day 3 - July 08, 2020

Todays session started on time. As usual Shrini provided free shell account to users to remote access to one of his temporary aws server for practice. Here are the topics we covered today

  • what is mean by default file descriptors and what are they
  • how these default file descriptors are mapped for each command
  • using ls -l /proc/<pid>/fd to view file descriptors mapping
  • the /proc special directory
  • the /proc/self special directory
  • using ‘echo $$’ to get the pid of current shell
  • what is mean by pid
  • how the default file descriptors ‘0’, ‘1’ and ‘2’ all mapped to /dev/pts/1
  • what is mean by /dev/pts/1
  • how every terminal is basically a character device file resides in /dev/pts directory
  • how you access the default file descriptors as STDIN, STDOUT and STDERR in ‘c’ programming
  • how to do redirection by changing the mapping of these default file descriptors
  • introducing cat command
  • how cat commands reads standard input and writes it to standard output
  • redirecting cat command output to a file instead of using standard output
  • redirecting cat command input from a file instead of using standard input
  • how cat reports errors through STDERR
  • how to redirect standard error to /dev/null to silence cat command
  • what is mean by pipe
  • how pipes differ from redirection
  • how to join standard output of one command to standard input of another command using |
  • how to find process id of particular command using ps axf | grep <command>
  • introducing grep command and how it searches using standard input
  • Explaining famous “do one thing and do it well” unix philosophy
  • how this unix philosophy shaped unix by having lot of tools instead of having big monolithic GUI applications
  • introducing background jobs
  • how to start background job using &
  • how to view background jobs using jobs command
  • how to bring back an background executing job to foreground using fg command
  • introducing shell variables
  • introducing environment variables
  • how normal shell variables differ from environment variables
  • how to view environment variables using env command
  • how to export a normal variable to environment variable using export command
  • why a child shell only gets environment variables, not normal variables
  • introducing less command, how to navigate output through page by page
  • Q & A

IRC Logs

2020-07-08 16:58:48 +mohan43u-we good evening to all
2020-07-08 16:58:58 +mohan43u-we welcome to day 3 of 'sh' Posix Shell session
2020-07-08 16:59:27 +mohan43u-we before we begin, if you are using 'webchat.freenode.net' you can hide join/part messages
2020-07-08 16:59:41 +mohan43u-we the instruction is provided in the 'training.ilugc.in' page
2020-07-08 16:59:58 +mohan43u-we also we have a new practice server. IP address got changed
2020-07-08 17:00:17 +mohan43u-we so make sure you login to the correct server to practice
2020-07-08 17:00:29 +mohan43u-we without further delay, lets start
2020-07-08 17:01:18 +mohan43u-we yesterday we learnt about man command, manual pages and howto execute more than one command with ';', '&&' and '||'combination
2020-07-08 17:01:28 +mohan43u-we today, we will start with redirection
2020-07-08 17:02:02 +mohan43u-we as I explained earlier, a command have "--options", "[args]' arguments, as well as it give returncode when it exits
2020-07-08 17:02:37 +mohan43u-we apart from that. a command have input and output and error-output
2020-07-08 17:03:30 +mohan43u-we these three are called default file descriptors for a command
2020-07-08 17:04:08 +mohan43u-we as you see in presenter terminal, I executed a command 'ls -l /proc/self/fd'
2020-07-08 17:04:23 +mohan43u-we the 'ls' is the command
2020-07-08 17:04:41 +mohan43u-we I asked to show me the directory listing of /proc/self/fd directory
2020-07-08 17:04:49 +mohan43u-we with detailed listing '-l'
2020-07-08 17:04:52 +mohan43u-we so it showed
2020-07-08 17:05:15 +mohan43u-we what special about this is, /proc is a special directory in unix filesystem
2020-07-08 17:05:51 +mohan43u-we hi
2020-07-08 17:06:28 +mohan43u-we this /proc directory is the way we see details about every process in the unix system
2020-07-08 17:07:06 +mohan43u-we the directory /proc/self basically points to the directory which kernel maintains about a particular process
2020-07-08 17:07:43 +mohan43u-we in our case 'ls' is that particular process
2020-07-08 17:08:19 +mohan43u-we this may be too deep, but just assume /proc/self/fd directory show all the file descriptors opened by the command
2020-07-08 17:09:03 +mohan43u-we I just now switched to 'sh' the posix shell :)
2020-07-08 17:09:12 +mohan43u-we previously I was operating in 'bash'
2020-07-08 17:09:25 +mohan43u-we coming to the talk, so as you see, I just typed 'echo $$'
2020-07-08 17:09:53 +mohan43u-we $$ is a special variable which shows the pid of the current shell
2020-07-08 17:10:05 +mohan43u-we the current shell pid is 710059
2020-07-08 17:10:31 +mohan43u-we pid means process id, every process in unix have process-id
2020-07-08 17:11:06 +mohan43u-we now, I'm going to use this pid to show you the file descriptors this shell is currently having
2020-07-08 17:11:54 +mohan43u-we see, the listing show that "0 -> /dev/pts/1", "1 -> /dev/pts/1" and "2 -> /dev/pts/1"
2020-07-08 17:12:12 +mohan43u-we just ignore "10 -> /dev/tty" for simplicity
2020-07-08 17:12:44 +mohan43u-we so it tell that the shell is using /dev/pts/1 file as its standard input (number 0)
2020-07-08 17:13:12 +mohan43u-we it also tells that the same file is used as standard output as well as standard error output (number 1, and 2)
2020-07-08 17:13:55 +mohan43u-we so it means /dev/pts/1 is the STDIN, STDOUT and STDERR for my current shell
2020-07-08 17:14:22 +mohan43u-we see, my current terminal is actually /dev/pts/1
2020-07-08 17:14:41 +mohan43u-we every terminal in unix is a file which resides in /dev/pts directory
2020-07-08 17:15:01 +mohan43u-we earlier I said right 'everything in unix is a file', this is one example
2020-07-08 17:15:11 +mohan43u-we /dev/pts/1 is a special file
2020-07-08 17:15:39 +mohan43u-we its a character special file
2020-07-08 17:16:00 +mohan43u-we it is represented by 'c' in 'crw--w----' permission string
2020-07-08 17:16:32 +mohan43u-we so basically, the shell is using the special terminal file called /dev/pts/1 for its STDIN, STDOUT and STDERR
2020-07-08 17:16:52 +mohan43u-we so, redirection means, changing these default file descriptors to something else
2020-07-08 17:17:46 +mohan43u-we before that, I'm going to tell about 'cat' command
2020-07-08 17:18:04 +mohan43u-we this is a very simple command, which reads input from STDIN and shows output to STDOUT
2020-07-08 17:18:31 +mohan43u-we as we all aware now, every command in unix will get default STDIN, STDOUT and STDERR mapped to the current terminal
2020-07-08 17:18:53 +mohan43u-we in that way, 'cat' will also get the current terminal as STDIN, STDOUT and STDERR
2020-07-08 17:19:38 +mohan43u-we I just typed some lines
2020-07-08 17:19:48 +mohan43u-we 'cat' just repeated that line
2020-07-08 17:20:10 +mohan43u-we because it is a dump command which does nothing but reading from STDIN and throwing out to STDOUT
2020-07-08 17:20:25 +mohan43u-we so when I typed "hi" and press enter, it just write back the "hi"
2020-07-08 17:21:20 +mohan43u-we the pid of cat command we currently executing is 712280
2020-07-08 17:21:28 +mohan43u-we I just listed its file descriptors
2020-07-08 17:21:41 +mohan43u-we all points to /dev/pts/1 the same terminal where we run 'sh' shell
2020-07-08 17:22:35 +mohan43u-we see this time, I redirected STDOUT of the cat command to output.txt
2020-07-08 17:22:51 +mohan43u-we now we see what fd the current cat command have
2020-07-08 17:23:40 +mohan43u-we see, the STDOUT (file descriptor 1) is now mapped to /home/trainer/output.txt instead of the default mapping of /dev/pts/1
2020-07-08 17:23:58 +mohan43u-we thats why the previous echo didn't happen this time
2020-07-08 17:24:47 +mohan43u-we to end the input to 'cat' command, we need to press 'ctrl-d' which means EOF (or end of file
2020-07-08 17:24:58 +mohan43u-we now we got the shell prompt back
2020-07-08 17:25:30 +mohan43u-we just like how we redirected STDOUT, we are now going to redirect STDIN
2020-07-08 17:26:25 +mohan43u-we see, this time, cat didn't waited for our input, becuase we did a STDIN redirect
2020-07-08 17:26:46 +mohan43u-we so 'cat' command instead of waiting for our input it just straignt to read output.txt file
2020-07-08 17:27:16 +mohan43u-we thus, we did STDIN and STDOUT redirection
2020-07-08 17:27:40 +mohan43u-we there is STDERR, lets do the redirection of that also
2020-07-08 17:28:20 +mohan43u-we we told 'cat' to read a file which is not available in the filesystem
2020-07-08 17:28:41 +mohan43u-we because that file is not available, 'cat' command throwed error message like 'No such file"
2020-07-08 17:29:25 +mohan43u-we cat command uses STDOUT for its output and STDERR to throw error messages
2020-07-08 17:29:40 +mohan43u-we now lets redirect the stderr
2020-07-08 17:31:00 +mohan43u-we cat can also read file given as first argument instead of using STDIN, lets use that method
2020-07-08 17:31:32 +mohan43u-we ok, now cat is throwing error message instead of 'sh' shell throwing error message
2020-07-08 17:32:11 +mohan43u-we see, that error message disappared, because of '2>/dev/null'
2020-07-08 17:32:21 +mohan43u-we '2' represents the stderr file descriptor
2020-07-08 17:33:32 +mohan43u-we instead of just doing '>' we can also use '1>' to do stdout redirection, '2>' to do stderr redirection
2020-07-08 17:33:48 +mohan43u-we so this is how redirection works
2020-07-08 17:34:12 +mohan43u-we ok, move on to the next topic
2020-07-08 17:34:15 +mohan43u-we before going
2020-07-08 17:34:31 +mohan43u-we I would like to say people who is using webchat.freenode.net
2020-07-08 17:34:51 +mohan43u-we you can disable join/part messages in the web irc client by
2020-07-08 17:35:28 +mohan43u-we clicking 'settings' icon on the top right corner (near to x), then uncheck "Show when People Join" option
2020-07-08 17:35:43 +mohan43u-we https://pasteboard.co/JgGXsrf.gif
2020-07-08 17:35:46 +mohan43u-we here is a gif
2020-07-08 17:36:29 +mohan43u-we ok, now move to the next topic
2020-07-08 17:36:47 +mohan43u-we pipes
2020-07-08 17:37:33 +mohan43u-we in redirection, we used a file in the filesystem as endpoint, like '>/to/some/file'
2020-07-08 17:37:48 +mohan43u-we but in pipes, instead of using a file, we use a process as its endpoint
2020-07-08 17:38:58 +mohan43u-we as a example, I executed 'top' command in the 'sh' shell
2020-07-08 17:39:16 +mohan43u-we I switched to another shell and typed 'ps axf'
2020-07-08 17:39:34 +mohan43u-we 'ps' command is used to list processes in unix system
2020-07-08 17:39:46 +mohan43u-we 'ps axf' will show every running process in unix system
2020-07-08 17:41:11 +mohan43u-we ps command will list all process details in STDOUT, instead of doing 'ps axf >myprocesses.txt' to store the process details, I used 'grep' command to filter a like containing 'top' word
2020-07-08 17:41:37 +mohan43u-we so this is one of the way you can search for a process id in unix
2020-07-08 17:41:52 +mohan43u-we the first like is the details about the top process
2020-07-08 17:42:03 +mohan43u-we *like*line*
2020-07-08 17:42:35 +mohan43u-we the second line provides details about the grep command we used to search the top process id
2020-07-08 17:43:10 +mohan43u-we ignore the second like for simplicity, so there are two matches from the 'ps axf' output
2020-07-08 17:43:24 +mohan43u-we we need the actual top commands details, so we take the first like
2020-07-08 17:43:31 +mohan43u-we *like*line*
2020-07-08 17:43:59 +mohan43u-we the first field in the first line is the process id of top command which we are running in the 'sh' shell
2020-07-08 17:44:17 +mohan43u-we so this is how pipe works
2020-07-08 17:44:38 +mohan43u-we we redirect one command's standard output to another commands standard input
2020-07-08 17:45:11 +mohan43u-we grep by default read everything from standard input and look for <regular-expression> provided as first argument to grep
2020-07-08 17:46:27 +mohan43u-we we redirected standard input of the grep command to use output.txt file as STDIN, and asked grep to search for lines containing word 'hi'
2020-07-08 17:46:38 +mohan43u-we so it throwed only one line
2020-07-08 17:47:31 +mohan43u-we see, now I used 'how' word, it throwed three lines containing how word
2020-07-08 17:48:01 +mohan43u-we with this functionality of grep and the functionality of 'ps', I'm combining both to find out the pid of the top command
2020-07-08 17:48:15 +mohan43u-we simple but yet powerful
2020-07-08 17:48:25 +mohan43u-we this is how unix works
2020-07-08 17:49:17 +mohan43u-we unix dont have big big kitchen sink GUI applications to do work, but small small tools like 'ps', 'grep', 'cat' etc
2020-07-08 17:49:30 +mohan43u-we these are like tools, we are goldsmiths
2020-07-08 17:49:48 +mohan43u-we we use these tools to combine them to create golds called 'shell scripts'
2020-07-08 17:51:00 +mohan43u-we one of the famous and important ideology of unix is 'do one thing and do it well'
2020-07-08 17:52:01 +mohan43u-we so the unix commands are all tools and we combine them to achieve out goals
2020-07-08 17:53:29 +mohan43u-we the command which I currently executed shows that there are 161 people watching the live sesion through 'Presenter Terminal'
2020-07-08 17:54:07 +mohan43u-we now, move on to the next topic
2020-07-08 17:54:12 +mohan43u-we background execution
2020-07-08 17:54:40 +mohan43u-we till now we executed commands and waited for output
2020-07-08 17:55:43 +mohan43u-we ok guys, I think today we have lot to cover, I'm going to take another half hour to complete today's session
2020-07-08 17:56:00 +mohan43u-we take 5 min break and comback at 18.00 PM to continue
2020-07-08 17:56:04 +mohan43u-we we have lot to cover
2020-07-08 17:56:16 * mohan43u-we taking 5 min break
2020-07-08 18:00:38 * mohan43u-we back from 5 min break
2020-07-08 18:01:07 +mohan43u-we so, lets continue our session
2020-07-08 18:01:48 +mohan43u-we for anyone joined new, and struggling to see the chat because of join/part messages, see this https://pasteboard.co/JgGXsrf.gif link to disable it
2020-07-08 18:02:33 +mohan43u-we so lets continue
2020-07-08 18:03:10 +mohan43u-we background execution means, we dont have to wait for a command to complete to get a shell prompt
2020-07-08 18:04:06 +mohan43u-we see, I just executed one for look
2020-07-08 18:04:29 +mohan43u-we that for look basically does nothing but slep for 100 seconds
2020-07-08 18:04:44 +mohan43u-we I dont have patient to wait for 100 seconds to get back the command prompt
2020-07-08 18:05:01 +mohan43u-we so I just used '&' symbol at the end of the command line
2020-07-08 18:05:20 +mohan43u-we thus I get 'command prompt' immediately to type the next command
2020-07-08 18:05:26 +mohan43u-we this is called background execution
2020-07-08 18:05:40 +mohan43u-we to see whether the command still running, use bg command
2020-07-08 18:06:03 +mohan43u-we see that background for look completed
2020-07-08 18:06:32 +mohan43u-we now I increased the time
2020-07-08 18:07:51 +mohan43u-we ok, 'sh' dont provide range expansion, for simplicity I'm switching myself to bash, which provides range expansion
2020-07-08 18:08:30 +mohan43u-we also, the command to check the background commands is called 'jobs' not 'bg'
2020-07-08 18:08:50 +mohan43u-we so I typed the for look and put it in background
2020-07-08 18:09:09 +mohan43u-we now I'm checking whether it is running or not
2020-07-08 18:09:33 +mohan43u-we the first line of 'job' command output shows that the for loop is still running
2020-07-08 18:10:17 +mohan43u-we to bring back the background executed command to front, we use 'fg' command
2020-07-08 18:10:37 +mohan43u-we see, '%1' here represents the job number
2020-07-08 18:11:04 +mohan43u-we 'fg %1' means, big %1 background job to front and wait till that command completes
2020-07-08 18:11:41 +mohan43u-we I just canceled that for look in the mid way using 'ctrl+c' keystroke
2020-07-08 18:11:48 +mohan43u-we and I got command prompt immediately
2020-07-08 18:12:10 +mohan43u-we then I run 'jobs', it show there are no background jobs currently executing
2020-07-08 18:12:16 +mohan43u-we that all about background execution
2020-07-08 18:12:54 +mohan43u-we I'll tell about subshell execution after we talk about variables
2020-07-08 18:13:01 +mohan43u-we because that will make sense
2020-07-08 18:13:15 +mohan43u-we now coming to the 'variables' topic
2020-07-08 18:13:29 +mohan43u-we till now we saw commands, and how we can combine them
2020-07-08 18:13:46 +mohan43u-we shell also provides way to define variable
2020-07-08 18:14:54 +mohan43u-we to create a variable, assign something to it and start using
2020-07-08 18:15:12 +mohan43u-we now I'm in 'sh' posix shell
2020-07-08 18:15:34 +mohan43u-we I typed 'var1=val1' thats it, var1 variable got created
2020-07-08 18:15:48 +mohan43u-we to get back the value of 'var', use echo
2020-07-08 18:16:09 +mohan43u-we the $ symbol plays important part here
2020-07-08 18:16:58 +mohan43u-we it means instead of showing the word 'var1' search for a variable called 'var1' in the current shell environment and give the value of that variable
2020-07-08 18:17:21 +mohan43u-we thats how you assign and retrive
2020-07-08 18:17:47 +mohan43u-we you can also use the same way to assign the value of var1 to var2
2020-07-08 18:17:53 +mohan43u-we now both contains val1
2020-07-08 18:18:39 +mohan43u-we these simple variables are called shell variables which you create and use
2020-07-08 18:18:51 +mohan43u-we there are some predefined variables called ENVIRONMENT variables
2020-07-08 18:19:16 +mohan43u-we these variables are inheried from parent process to process
2020-07-08 18:19:27 +mohan43u-we to see those variables, we use env
2020-07-08 18:19:31 +mohan43u-we env command
2020-07-08 18:19:52 +mohan43u-we env command will provide all the predefined environment variables
2020-07-08 18:20:24 +mohan43u-we I just used pipe method to see the 'env' command output
2020-07-08 18:20:46 +mohan43u-we 'less' command is a pager, its main funcitonality is to show standard input page by page
2020-07-08 18:21:14 +mohan43u-we so I combined 'env' and 'less' achieve this current look in presenter terminal
2020-07-08 18:21:46 +mohan43u-we in 'less', it will show one full page of output and it will wait for your response
2020-07-08 18:21:58 +mohan43u-we to see the next full page, you have to press 'space' key
2020-07-08 18:22:27 +mohan43u-we now I'm at the end of the output, I can go back using 'b' key
2020-07-08 18:22:41 +mohan43u-we see, now I'm top of the output
2020-07-08 18:23:10 +mohan43u-we coming back to Environment variables, you can see that there are lot of environment variables 'sh' inherited from its parent process
2020-07-08 18:23:18 +mohan43u-we one such variable is PATH
2020-07-08 18:23:36 +mohan43u-we another is HOME
2020-07-08 18:23:45 +mohan43u-we another is USER
2020-07-08 18:24:12 +mohan43u-we another is PWD which is now shown currently but you can look at the output
2020-07-08 18:24:40 +mohan43u-we so the 'env' command provides all the environment variables
2020-07-08 18:25:02 +mohan43u-we you may ask that
2020-07-08 18:25:21 +mohan43u-we where is var1 and var2 which we defined here
2020-07-08 18:25:28 +mohan43u-we in the output of env
2020-07-08 18:25:43 +mohan43u-we there is big difference between local variable and exported environment variable
2020-07-08 18:26:11 +mohan43u-we normal variables like 'var1' or 'var2' dont get exported to the child shells
2020-07-08 18:26:21 +mohan43u-we only exported variabled go to child shells
2020-07-08 18:27:18 +mohan43u-we see, I just exported var1 which is a normal variable to make it as environment variable
2020-07-08 18:27:40 +mohan43u-we now when I typed 'env' command, it now shows 'var1' which has value 'val1'
2020-07-08 18:27:53 +mohan43u-we now this is the right time to say about subshell
2020-07-08 18:29:38 +mohan43u-we there are currently two variables var1 and var2
2020-07-08 18:29:48 +mohan43u-we I exported var1 but didn' texport var2
2020-07-08 18:30:06 +mohan43u-we I start another 'sh' shell
2020-07-08 18:30:23 +mohan43u-we now I type 'echo $var1' it will show you the value of var1
2020-07-08 18:30:38 +mohan43u-we but when I type 'echo $var2' it will not show
2020-07-08 18:31:03 +mohan43u-we see, this is what I mean by subshell
2020-07-08 18:31:30 +mohan43u-we so, when you program shell scripts, you need to be very careful about normal variables and environment variables
2020-07-08 18:31:56 +mohan43u-we only environment variables pass through subshells, or child commands, or child processes
2020-07-08 18:32:09 +mohan43u-we normal variables are not accessable from subshells
2020-07-08 18:32:49 +mohan43u-we that all from today. we will continue the rest in the next session
2020-07-08 18:33:06 -- Mode #ilugc [-m] by mohan43u
2020-07-08 18:33:19 +mohan43u-we the channel is open for questions
2020-07-08 18:33:30 +mohan43u-we you can ask any doubts from todays session
2020-07-08 18:34:29 -- Mode #ilugc [-v mohan43u-we] by mohan43u
2020-07-08 18:34:31 SATHISHKUMARPVR Thank you
2020-07-08 18:34:39 -- Mode #ilugc [-v shrini] by mohan43u
2020-07-08 18:35:20 Nero007 Was a subshell created when you typed sh
2020-07-08 18:35:26 @mohan43u Nero007: yes
2020-07-08 18:36:09 Nero007 Had it the same priority as the previous shell or was it inferior and thats why the name subshell
2020-07-08 18:37:02 linuxbaskar how many subshells can be created, and how the user know whether the shell is subshell or not
2020-07-08 18:37:04 @mohan43u Nero007: you can use the word subshell in various context, I used it to show the relashionship between parent process and child process
2020-07-08 18:37:36 @mohan43u Nero007: and also to explain the importance of using 'export' to export environment variables
2020-07-08 18:37:49 hellboy212 mohan43u-we: can you plz tell once again the command to export a normal variable as an environment variable in the chat? i had missed it
2020-07-08 18:38:06 @mohan43u hellboy212: 'export var=value'
2020-07-08 18:38:21 hellboy212 mohan43u: thank you
2020-07-08 18:38:31 @mohan43u hellboy212: instead of just assigning value to variable, type 'export' before so that variable become environment variable
2020-07-08 18:38:55 hellboy212 ohh okay
2020-07-08 18:38:59 Nero007 mohan43u: Thanks
2020-07-08 18:41:48 hellboy212 can we delete an environment variable from shell?
2020-07-08 18:42:09 @mohan43u linuxbaskar: 'ulimit' command can provide you various limits, one of the limit is maximum processes, you can use 'ulimit -p' to see the maximum number of child processes you can create
2020-07-08 18:43:59 linuxbaskar I have invoked multiple subshell. how can i find the pid of the current subshell
2020-07-08 18:46:55 linuxbaskar echo $$ displays the pid for the current subshell
2020-07-08 18:47:14 @mohan43u hellboy212: unset <variable-name>
2020-07-08 18:48:39 @mohan43u hellboy212: 'unset' command unsets variable (means deletes variable from current environment), if the variable is a environment variable, then it also removes the environment variable from the environment
2020-07-08 18:49:39 Nero007 mohan43u:It's a request to please update the logs
2020-07-08 18:50:31 @mohan43u Nero007: will do it mate. dont worry.
2020-07-08 18:53:31 shrini we got 92 users logged in the remote server
2020-07-08 19:05:07 humachine mohan43u: The part on file descriptors was very good. Like the way you really getting behind the scenes in explaining the concept.:)
2020-07-08 19:06:21 mohan43u humachine: thanks. After nobody asked questions, I thought I screwed somewhat and nobody understand. you gave me hope :)
2020-07-08 19:07:05 humachine mohan43u: :)
view raw meeting-minutes-sh-posix-sell-session-day-3-july-2020 hosted with ❤ by GitHub