Hello, everyone. For now we have a way to load images from our disk, which is by using our image loader function. Let's make use of it in this video by implementing a date set pre processing function. This will load all images and the respective labels at once. As we discussed in the previous video, I've already prepared the head of the function. So now we will discuss the arguments for it and start our implementation.

This function takes four arguments. The first one is data set path, which is the path to the train or the test folder. If we take a look at the data set directory, it has two sub folders, train and test and all images are located inside. We have one more file inside the data set directory called labels. If we take a look at our data set pre processing function, the second argument is the path to this file, then we have our image size argument, which is the width and height of a single image, same as we had previously in the image loader function. The last part of this function that we're implementing is to save all paths of every single image of our training or testing data set.

And this will help us later on when we need to locate the most similar images and load them. Because of this, we have the fourth argument, image paths people, simply put, this argument is the name of a pickle file, where all paths will be stored. Okay, now that we are done with our arguments, let's implement the function. First thing that we have to do is to get all classes that we have in our data set. For that we have to open our labels file, right with open and provide a path to the labels file and open it in the reading mode. Now, we have to create Least of all classes in this file, open it with the read method and splitting on each newline character.

This will create a list where each element is a line of its own. or in our case, each line is a class name. Let's bring it out and check the results. As you can see, all classes are here, but we got one bonus class an empty one. To fix this, we'll put column minus one here, which will take all elements of our list except the last one. And now we have all our classes.

The next thing that we will do is to define empty lists, in which will attend images, labels and image packs. images will hold all the images, labels will hold all the respective classes for each Image and Image paths will hold the path to each other And this list will be stored in a pickle file. Let's fill out the return statement of the function by returning empty array of images, an empty array of labels. The reason that we are using empty array instead of regular lists, because we will need to use some NumPy functions for fast array manipulation, and we wouldn't be able to do that with regular Python lists. Right before the return statement, however, we will assert that Len of images is equal to land on labels. If you want to make sure that one part of the code is correct or some checker in this case, to make sure that we have the same amount of elements in images as well in labels, at least, we use the assert.

If this statement is not correct, the rest of the code won't be executed. Okay, back to the body of the function will open a for loop and go for all images in our data set folder. We'll make a try catch statement here, and just simply pass if we encounter an error in the try block, we will append images to our list of images. And we will utilize our image loader function for this purpose. It takes image path and image size as arguments. Image path can be built using oise dot path dot join, where we join the data set path and image name.

We pass image path as an argument to image loader as well as image size, which is also an argument of the function that we are building right now. Next up, we append the path of the current image through our image path lists. Now we need to go through all of our classes and check if the class can be found in the image name. And as you can see, this is how the data set is billed in our case In our case, each image has a respective class name in its name. Now we need to go through all classes and check if a class name appears in the image name. We open a for loop and go through all classes.

And we check if any of the class names is found in image name. We'll write an example here, zero underscore frog dot png isn't just an example of an image found in our data set. If there is a match, we append the index of the current label, since we only need indices. The next thing we should do is to save all of these paths to our people here will open a new file called image path pickle and append dot pickle to it and open it in the binary writing. will use pickle dot dump which takes an object in our case a list of paths file which is opened, execute the cell. And let's store the results of the function inside images and labels, execute the function and it's going to take a bit to run.

So I'll get back to you when it finishes. Okay, the function is done on my end. So let's change the shape of our training set images. There we go. As we discussed, there are 50,000 images here. Each image is 32 by 32, with free color channels.

And that's it for this tutorial. If you have any questions or comments, leave them please in the comment section. Otherwise, assume the next tutorial