In the previous lecture, we looked at the output of the find command. When searching for all.com files, it was too long to fit into one screen. Now imagine you run the search command that found 100,000 files, there is no way you can find anything useful in the output when it is printed to the terminal. In this lecture, we will talk about Linux pipelines. A pipeline in Linux is a very powerful concept of combining or chaining several commands into one pipe to pass the output of the first command to the second, the output of the second command to the third, and so on. In our example, we'll define output we can paginate it using a viewer like less.

These applications are also known as pagers, because they used for exactly this reason to paginate the output will now pass the output of the find command to a page or less, find slash etc. Minus name, search for corner files. Now I will add an additional pipeline option writer and pas the output to the last. Now we can navigate the search results easier. We could search using slash and do other operations as we learned in the tutorial for less applications. The same thing we can do with text files.

In the resources attached to this lecture, you find the CSV file. It is a comma separated value file has comes from the CSV extension. Let's look inside it. It contains a list of people in the neighborhood visiting different stores on days of the past month. It really doesn't matter what's inside this file. What is important is that it contains an unsorted list of entries in three columns.

These are text fields name and store and a number between one and 31. Let's first print the content and page it using less. We'll see the same thing as calling less Hello But you'll see later why we do it in steps. Let's now sort the output we can do in using source command. Again, the output doesn't fit into one screen so we can apply the same trick and paginated. Now we can navigate in the file, the output is now sorted by a string sorting function.

In this case, the line with 19 will come before the new line tool. Let's try to see unique people whoever went to any store. For this we need to select just one first column of the output. Now we can do this using cut command. I pipe the output to cut. I'll use the argument minus d for delimiter.

I will specify comma and I want to print the first field specified by the minus f one argument. I can also print the same line as in the original file by attaching fields One, two and three this way, and it will print basically the same content as the original file. I can also specify the output delimited for the cut command by appending, an additional argument minus minus output dash delimiter. And for instance, I want to use a space. For now I'm only interested in the first column. I can now sort the output.

And now I can print unique lines by applying additional pipeline operation unique. When you use the unique command, make sure that you sort the output beforehand. for performance reasons it removes duplicates only in sequential lines. If you omit the sort function, we don't get what we expect to get. Let me show the output once again. Now we can print the last 20 lines of this output using the tail command that we talked about before, minus 20.

Or I can do the same and print the first 20 lines using head, head on us and 20. We can also count the lines in the file using wc command with additional argument minus L, or we can count the characters in the file with minus c argument for the wc command. In this simple case, with one piping operator, we could of course use the simpler form with the wc command like this. Let's look at filtering the output. Let me try to find the lines in this file that are relevant for person Bob. We can do it using the grep command.

Adding this to the second pipe after printing the output grab Bob This way it will find all the lines having tags Bob in it, we can enhance our search by adding a more complex search pattern. I will now add the argument for grabs in quotation marks. And you'll see in a second why I will now add the or symbol that looked exactly as the pipe symbol. Now this time I wanted to be in the printer for the grep argument and not the complete line. Hence I prepended with the backslash n eight an additional text that I want to search for Jane. I close the argument with the quotation marks.

And how to find the lines that have either Bob or Jane in the line or maybe both. We can all combine to grep commands in the pipeline and seeing which days Bob visited only Aldi store. And with one more additional argument to the grep command. I can see in which day is Bob visited all stores. But LD with a minus vi argument grep is a powerful command also in the sense that we can use regular expressions to match any occurrence of the text. I now want to find text that matches my pattern, which has been any character dot star, and B again.

This will find mostly Bob Of course, we can also ignore cases by adding the minus AI argument to the grep command. Oh try to search this way be any character see. There's no matched or lines that have B and C and any characters in between them ignore in the case. grep alone can also be applied to single files, and it has a few more additional options. I can also print the line at which the currents was found using minus an argument I can also print the line that matched my text with a so called context. I can also print the context of the line before.

I can also print the context after the matching line with minus a. As you saw now grep can do useful searches and filtering the output. There are arguments for searches in the pipe output, as well as files and there are various options for pattern matching. Some of the common arguments are listed here. It takes time to learn grep usage, but once you must see grep and regular expressions, you can make very complex patterns for searching the file. This is a Swiss knife in piping operations.

And you will definitely have fun learning it. As you saw, we applied multiple operators in the pipeline commands. You can combine as many commands as you like applying grab sort cards and other commands. The key point when using pipelines is that you build your pipeline step by step. This way, you can see the output of each step, and it can even debug your pipelines. Sometimes it is convenient to make a small shell script with the pipeline you are creating this way can execute the script instead of typing all the commands again.

In this lecture, we looked at pipelines of commands to make the output more useful for our consumption. All this time we've printed the output to the terminal. In the next lecture, we will look at how we can print the output of commands and pipelines to a file and make output redirects