Welcome to lab 11 web job and background processing. In lab 10, we discussed about app model app services model and we discussed about cloud services model. And then we said that in lab 11, we will concentrate on background processing that means on web job and worker role, but when I analyze the situation, I saw that the web job topics are large in number. So what we will do is in lab 11, we will just do web jobs because this lab itself will take me approximately 40 minutes at least to complete. So if I also add worker role in this, it will almost take like one and a half hour so what we will do is we'll do first 40 minutes on web job and then we will look into worker role. And the second most important point which I want to stress here is that that jobs should be given more priority than worker role because worker role and web role are the old deployment models that is a cloud services And I feel that Microsoft is not pushing too much now for cloud services model, and they're giving more emphasis on the app services model.

So I would suggest, if you're, if you're starting a new project, and if you want to do background processing, then web jobs should be a priority, right. So with all that in mind, let us cover web job in this lab. So in this 40 minutes of lab, we will be first starting with why web jobs, the different types of web jobs like triggered continuous and so on. We will also see that when web jobs run at the background, how to go and see the logs of the web jobs. And also we will see something called as cron expression, you know, which helps us to run web jobs every five minutes, 10 minutes every Sunday, or every, you know, on every specific month, and so on. Also, we'll talk about what is the importance of always on in continuous web jobs.

We will also see different storage accounts which are needed for web jobs. And finally, the most important is that we will also see how to publish web jobs using Visual Studio So, let us concentrate on in this lab only on web jobs and later we will do worker role in some other lab right. So, let us start. So, as we have said in the previous part of the video, web jobs are nothing but you know, they are background tasks or background activities, which run continuously. Now, these background tasks and these background activities, you know can be a simple eggsy they can be some kind of a bad file, they can be a PowerShell program, they can be anything you know, which is an executable, so you can take up any kind of an executable, which as your understands, and that executed putable, you can go and run it as a web job.

So, what I'm going to do is I'm going to go and take the simple console application here you can see that I've created a simple console application. And in this console application, I'm just saying console dot write line Hello. So let us go ahead and try to host this console application as our job Also please note when we say web job, we say it is an app service. When we say worker role, then we are talking about cloud service. So in the previous part of the video, I have already talked about the differences. So let us go ahead and try to host this application, the simple e xe as a web job.

So you can see that I have built a program it is all successful. So now let us go to our portal here. And let us first go and create a service, a web service, right, a web app service, right. So I'm going to go and create I want to create a App Service. App services. So I'll say create an app service.

You can see the there's a menu down below I will say create an app service. And I'll say web app. Now remember that even though we are creating a web app service, we can always go and host in this web app A web role, right? So when you say you're going to go and create an app service, right, it means that a create an web service, that means that you can also go and host web jobs into it. So I'm gonna go and click on Create here. So let us give this app service name as my app service.

Test. All these names are booked on my app service. ship, okay. So I will say pay as you go. I want to host it on a Windows platform. And here let us select an app service plan.

Let's take a new plan. I just want to take a basic ones. What I'm gonna do is I'm gonna go and create a simple equity cover Let me take up the simplest one. Let me not let me take up the f1. Free thing, right? app service plan my free plan.

Let's see. Okay. So first thing is I'm creating an app service here. And in this app service, I will go and I will host this E xe, this console application e xe as a web role, right. So there it is, you can see the app service is getting created. So let us give it some time.

So there you can see that the service is created. Now one very important point to note here is that we do not have different flavors of app services for web job, and website. So when you create an app service, Right when you create an app service like this, in this same app service, you can go and you can deploy a website. So you can go to the deployment credentials, you can get the FTP, and you can host a website. Remember in the previous part of the video we have, we had shown that how you can host using FTP, how you can go and host using Visual Studio. So in this you can go and host and website, as well as you can go and host on web jobs.

So somewhere down below, you will find that there is a web job web job in case you are not able to find it. You can always go and search here. So there it is. So this web job menu over here helps you to host he sees bad files, and so on right so let me go here. Now my e xe my executable at this moment is in the bin folder. So in this bin folder, this console app seven is my BSc right.

So let me go here. Right, let me say add. So let's, let's give a name here saying this is web job console. Okay? And the file to upload, so I'm going to go and give this file to upload. Now, a web job can be triggered in various ways, it can be either be manually triggered, so it can be either be triggered manually, or it can either be continuous, right?

So what I'll do is I'll say that this is triggered manually. So in other words, somebody is going to go and trigger this, it is not a continuous web job. So remember, there are two types of web jobs which we can host one is continuous, which keeps running at the background. And the second one is triggered triggered triggered means that some event goes and starts that web job right. And also, other important thing is that do it Do you want it to be manual? So, manual means that basically that somebody will go and run this web job again and again or do we want it to be sheduled shedule means that it will run every five minutes or every 10 minutes right.

So, for now, let us make it simple let us say manual. So now there you can see, you know the web job has been created. So, it says that this is a manually triggered web job, the name of the web job is web job console and it is ready. Now, at this moment, his web job is not running because it is manually triggered. So, let us select this and let us click on this run button out here by which this web job will run. Now, you can see it seeing that web job is now running and it says over here completed just now, okay.

Also I can go and run it multiple times, you know, I wish. Now, if you see at this moment, normally web jobs will run continuously at the background. They will do activities like pulling data from one database, pushing it to other ones. They would be doing some kind of massaging on files, you know, or doing archiving and so on. So they would be doing activities, which are not visible, normally to the end user, right. But as a developer, what we do is we put a lot of logging messages, we say that this activity has finished.

If there is an error, we go and log things, right. So if you see in our application at this moment, I have said here, console dot write line. Hello. So I would like to go and see that. Has he really ran this web job? And is this Hello logged somewhere, right?

So in order to see the logs of a web job, you can click on the web job, and you can click on logs out here. So you can go and click on the logs here. Right? So you can see here that this web job has ran two times. You know, this was one minute ago, and this was 56 seconds ago. So I can zoom here.

So if I go and click on this thing, it goes here and it shows you that Okay, so this web job, actually ran on this time. And you can see the Hello message out here, right? So in case you want to go and see log messages and console messages, you can always go and click on this logs out here. And you can see things in more detail. So now that we have seen one example where we were manually triggering the web job, but in a lot of scenarios, you know, you'd like to have this as an auto trigger auto trigger means that we would like to trigger the web job every five minutes. So what I've done is I've deleted the old web job, I selected that and I hit delete, and it got deleted.

So I'm going to go and add a new one here and say this is web job. Continuous, right. So this is web job continuous. So again, I'll go and select the same application. And I will say yes, this is triggered, right this is triggered, but this is sheduled means that we are going to go and give some kind of We are going to go and specify saying that it has to run every five minutes, or it has to run only on Sundays and so on right? Now in order to go and specify a schedule, we need to go and define it in a format something called as cron expression.

Okay. So what are cron expression cron expression you can think about is of something this kind of cron expression has these elements out here. So it has 123456. So there are six elements in cron expression. And so it first starts when I say that six elements in a cron expression means this is the format, okay? So you can specify the seconds, you can specify the minutes, you can specify the hours, you can specify the day, month or day of the week, when you want the web job to run.

So for now, if you see my cron expression would be something like this. We will not go in depth into cron expression at this moment was Be I will have a separate video for this. For now our concentration is on web job. So I'll I'll try to quickly introduce you to cron expressions at this moment. But later on, we can have a full blown video for it right. So for now, a cron expression starts with a second followed by a minute, then by an hour, day, month and the weekday, right?

So I'll say, let's say that we want to run this cron expression. We want to run this job every five minutes, right? So I'll say that zero seconds. So that means I'm not worried about seconds, right? It should run every five minutes. Let us let us make it two minutes ago so that we can see the output fast.

I'm not worried about hours. I'm not worried about days, so any day, any month, and any week. So you can see now this is a simple cron expression which says that you want to run it every two minutes. And four, you want to run it every two minutes. And you are not worried about what is our what is the day, what is the month and so on. Right?

So I'm going to go and put this cron expression. Let me quickly check this cron expression with. Okay, so, so I'm going to go and put this cron expression right out here. Right? It is showing some kind of an error, not a valid cron expression. Let me check my cron expression.

Oh, we need spaces, I think in between, is it? So let's put the space sorry for it. Yeah, there it is. So you can see. So basically, this was a proper font expression, but you need to put spaces after each element. So I've done that.

And I can see a green sign out here, right? So let me click on OK. And let's see that will this web job run every day? Two minutes. So we are specified that this web job will run every two minutes right. We can see that there is a refresh button out here you can always click on the refresh button. You can see at this moment it is saying ready status.

In case it runs it will say I have executed right. So, let us click refresh and let us see what happens. So now you can see it is running, can you see that it is running. And if I refresh again it has completed. So, if I now go to the logs here, so, if I click here and if I go to the logs I should see the same kind of output so that it is loading the console, loading the console messages. Slow that it is so you can see now it has ran only for first time, right.

So, I'll just go here and I can see Hello Yeah, I remember that. This Ping will run every two minutes. So if I if I refresh this, I should see that it should be running every two minutes. So, after two minutes, I should see one more entry out here still two minutes has not passed by. But in case when two minutes passes by, I should see one more entry out here. So that you can see now the web job again ran after two minutes, right.

So basically, in order to run a web job, in an auto trigger mode, we need to go and specify the cron expression and the cron expression is in the format of second followed by the minutes, hour, day, month and week, right. And I've shown a very simple cron expression which I've written out here that we are not worried about seconds. They should run every two minutes. And it can be any hour any day, any month. than any week, and also very important is that there should be spaces in between each one of these elements. So, we have created two kinds of web jobs till now, one web job, which will be triggered manually, and the other web job, which will be auto triggered.

But now there is one more scenario, you know where we want the web job to run continuously. So the time you go and deploy the web job, it will start running continuously, right. So let us see that how we can go and create a web job which runs continuously. So we'll take the same console application. So you can see that I've clicked on add here and we will say, this is job continuously, right. Let's select the same ESC.

So I would go and select the same console app seven. And this time I will say this is not triggered, but this is continuous. And when I select continuous, you can see that there is something called as in scale here, we'll talk about this Later on okay for now, let us concentrate on creating a continuous web job right. I will talk about the scale later on. So I will just say okay. So now the time I say okay, it will try to create a web job which will run continuously.

So continuously the console application will run. You can see at this moment, you know there is a small failure out here. So, if you see it, it's saying that failed to add the web job. So, let us try to understand what failure has occurred. So, I'm going to go and refresh you can see that it is saying web job is here, but if you see here the status it says failure, so, there is some problem out there, right. So, over here it is saying failure.

So, let us go and check the logs. So, I'm going to go and check the logs out here and let us see that, why has the web job failed to deploy, right. So, I've clicked on the logs. So, let us read the logs in a detailed manner. So, there it is. So, you can see here when Go and see the logs, it says that always on is not enabled for this web app.

So in other words, to ensure that you can create a continuous web job, you need to enable the always on thing and for always on is only available in basic standard and premium modes. If you remember, for this service for this app service, I had selected the free one. So, if you select the free one, then you cannot go and create a web job which runs continuously. So, let us go ahead and first change that free thing into something like basic standard or premium and then we will go and enable always on and then this will keep running continuously. So let us go ahead. So let me first go and delete this web job.

So let us go to our app service and let us go and change the plan. So I've clicked on clicked on the search your plan, I can see a lot of things. But I need to go and scale up. So I'll just go and scale up this and I will ask you I said that this free thing will not work, it has either to be basic, it has either to be shared standard, or it has either to be premium, right. So I'm going to go and take this basic one, which is have the lowest cost, so this is the lowest cost. So let me select this.

So you can see that it is now updating the app service plan. So once this is updated, I can go and then enable the always on and then we can also go and run our web job continuously. So let us wait and first update this app service plan. So there you can see that the plan has been updated successfully. So now let us go to web job and let us try to create a continuous web job. So let us go here and just try to add a web job.

Slow Okay, so let's add a web job saying web job, continuous, semi Xel Select All these are standard now. Continuous, you can see now, I can see here that I can go and say multi instance. Or I can say a single instance you know. So for now we'll just say continuous. And as a single instance, I'm not interested in multi instance, I will explain this property later on about the single instance and multi instance right. And I will say okay, so that you can see that the web job has been created.

So if I click on this web job, and if I say start, let us say if I say start, you can see that it's web job has started. So if I refresh this so if I say start here, and if I refresh this, you can see that it starts and it says here and look at the right hand side it says, successfully started the web job. And then it goes in a pending restart mode, right? If I go to the logs very quickly to just see what's happening in the logs. So, let us see what's happening in the log. So, in the logs, I can see that he is he has ran Hello, it then went into a pending restart, then when I again clicked on start it again ran Hello, and it again went into a pending restart.

So, what is happening out here, why isn't it running continuously? Now remember, when we say that it has to run continuously that has to be done by the program by the console application. So in other words, in your console application out here, we need to have a kind of an infinite loop or a continuous loop like you know, while one equal to one, something like this, you know, so with this, you know, the web job will become continuous. So the web job does not become continuous. By default as such, your code should be such that it's running continuously, right? So that's why at this moment, what is happening is, if I go and click on Start, it actually starts, it runs our web job.

But then again, it goes in a pending restart mode. So in order that it does not go in a pending restart mode, we need to have an infinite loop or we need to have an continuous loop. Right. So for that, you know, the best way is that rather than creating an web job by using a simple console application, I would suggest to use that template so you can go and say, create a new project and use the web job templates. So that is the as your web job template, you can use this, this has the necessary basic code, you know, which helps you to run things continuously. So you don't have to really write that basic logic of running the things continuously.

So you can see that I've selected the as your web job template, and in that web job template, that continuous loop is there by default, there is a built in code for that. So if you see out here, you can see that it is creates a job post and this whole says run and block. So, this running block means that this code out here will keep running continuously right. So, this this host this host actually goes and runs this function right and this function actually goes and runs continuously. So, basically if you want to really go and you know have one continuous web job as such, what you can do is you can select that template and write your code whatever is necessary in this function and what this run and block does it it actually goes and continuously calls this process cue message right. So, this process cue message is where you can go and you can write logic.

So, now, let us do a simple exercise let us go ahead and host this web job in a continuous mode. So, let us go ahead and host this web job. And while we are hosting this web job, what we will do is we will use this publish as a As your web job wizard to do the publish, so if you see in the previous part of the video I was going in uploading the XC, but, you know, you know, what I'll do is you know for this demo, I will upload it by using the Publish as as your web job is that right? So, if you see first thing this boilerplate code what we have this sample code you know, what is given by Microsoft in this, what they have done is you know, they have created a created method here called as process queue message and this process queue message gets called continuously because of this method your run and block you can see this small comment here which says the following code ensures that the web job will run continuously right.

So, and in this process given message, what it is doing is, it is actually going and reading from a queue called sq. So, the name of the queue is q. So, whenever there is an entity into this queue, right? If there is any kind of queue message, it will actually read that message and then process it right. So let us give this cue as name my Q, because Q is a very generic kind of a name, right. So, we'll go and we'll create a queue called as my queue.

And whenever anything is entered into the my queue, this will read automatically and send that message over here. And what I will do out here is I will go and I will write that message into the console so that we can see in the log, right, so I'm going to go and write this message in the console, so that we can see it in the log, okay. And I'll build a solution. So let us go ahead and first create this my queue queue right now. We need to go and specify two storage accounts, you know, when we actually publish a web job. So if you see here This app config file, it is asking for two storage account here, it's you can see that one storage or one storage account which what it wants is, is the as your web jobs dashboard.

And the other storage account is as your web job storage. Now, what is this as your web job storage and dashboard, this dashboard is for logging activity. So whenever like, you know, when you went to your web job runs, when it has started, when it has stopped when it has terminated all those log messages actually go into this storage account, right. And this as your web job storage is for, for example, if you see in our example, we are using this mic queue, right. So whatever business related data, your actual data is stored into this account as your web job storage. So there are two accounts we need to create one is as your web job dashboard, and the other one is as your as your web job storage.

So let us go ahead and create two storage accounts first thing, so I'm going to go here and say that I want to go and create a storage account. Now remember that in the previous part of the video, I've already explained how to create a blob, how to create a container, right? How to create queues. So we'll say that this is as your web job dashboard shift. Okay? And I'll just make it simple storage.

I'll just make it a locally redundant storage cannot contain Okay, so it cannot contain capitals so as your or I'll say, lock shift. And also lock shift means this is where the log messages of the web job will be logged in. So I've just selected the basic one. Locally redundant storage I've selected general purpose v one. So these are all standard one we have discussed in the previous part of the video, I'll just go and create it. So one is I am saying lock shift and the other one ultimate as data shift.

So in lock shave all the logging information of the web job will be done. While in data shift our main data, our queue, our tables will be there, right? So let me go ahead here and create one more account here. Data shave, right. And again, I'll keep this a standard use existing resource create. So I've created two storage accounts.

Here at this moment, one storage account is a lock shift and the other storage account is the data shift. So let us refresh let us create these two accounts. So we'll take the connection string right and put those connections string in these in this config file out here right. So, there it is, both the storage accounts have been created. So one lock shift has been created. So let us go to lock shift.

Let's go to access keys and let us take the connection string. So this will go into the web job. All right. So this is the lock Civ, right? And the other one is the storage. So that is the storage, the data shift.

So again, I'll go and take the access keys and copy this. And this goes into the second one. So both the connection strings have have been provided. So now that we have created the web job, it is time to now go and publish this web job. So we are going to go and publish this web job inside this inside my app service ship, so inside the app service, we want to go and publish this web job right. So I'm gonna go and say here publish as as your web job right.

It will ask for a lot of informations like you know, just let them go previous out here. So you can see in the previous it will ask you like A lot of information out here like your username, your password, and so on, it will say that you need to go and if you see out here, so when you actually run the wizard, it will say that Okay, tell me what is your subscription? Tell me your resource group, right? So I'll say my resource group. So it's a resource group. And in, in this my app, ship, I want to go and publish this web job, right.

So I'll do a next, everything next. And I'll say publish. So there it is, it is publishing. And once this web job gets deployed, right, it should be visible. So it should be actually be visible in the web job section. So if the web job is deployed properly, it should actually get reflected right out here, right.

So let's see. It's still deploying still. It's happening. So there it is. The web app has published successfully. Great.

So let us do a refresh. And I can see that the web job is running in a continuous mode. And it is great, right? So let me go and quickly check the logs just to check out that you know if there are any errors if there are any issues and so on. So that means I've clicked on the logs out here, and I can see that I don't see any error as such, I've toggled output and things are great, right. So now, what should happen is, if I go and put something in the queue, so if you remember the code, what we have written out here, the code, what is your says that if you go and put something in my queue, this web job will go and process that data, right, it will process the data and write it in the console.

So let us go ahead and add some data in my queue. So I'm going to go to my data shave, right and in that we have created, oh, we have not created the queue still right. So let us go ahead and create this my queue, right. So let us create the my queue. So there it is. So in this my queue, I'm going to go and add a message here, this is coming from queue.

Right? And I will go and add. Now as soon as I add here, it should actually read this data and process it. So, you know, I'm not sure what time it will take to read this data. But it should actually go and read this data and process this data of the queue. Right?

So let me go and quickly check in my web job what is happening, so I will go to my web job. And let's quickly check. That is it trying to read that thing, so I'm going to go and click on logs out here, and let's toggle output. So you can see that there are some errors out here. It says that the storage lock shift is incorrect. You can see that there are some errors, which I see right out here.

Just check what it is saying it is saying that storage lock shift is incorrect. Why is it incorrect? storage lock ship is incorrect. I think I created this my queue late round, right? So I'm not sure really, if that is a problem. So I'm going to go and delete this web job for now and let me publish it.

If you see what has happened is basically the my queue was not existing. So I must have published a web job and it then said, Oh, like my queue is not existing and the program crashed. Right. So what I'm gonna do now is I will just go ahead and republish it, you know, and I hope that this time I should not have problems. So let's Go and say publish as as your web job, I'll just say publish. Let us see.

So I have figured out the problem, the problem was not really off the queue name. If you see we had two kinds of storage accounts we have specified. One is for the logs. And the other one is where our actual data storage is right. Now, this log storage, that is the as your web job dashboard, right, you also need to go and provide in the application settings. So in the application setting of your service, your app, your app service, you need to go to connection string.

So let me go and delete this. Actually, I had added one connection string, which was an old one. So let me go and delete this. So we need to go and add a new connection string here. And we will say this is custom. And you will go and you will put this whole connection string right out here.

And the name of this connection string will be as your web jobs dashboard. Now, this is something which I did not like because I feel that I have already specified the as your web jobs dashboard in my app config file, why do I need to really go and provide it again, in the application setting, but that's how it is for now. Right? So I'm gonna go and save this. So, this was the problem. So, now, let us go ahead and let us again go to web jobs.

So this time it is stopped, okay. So let me go and start this, right. So there you can see the web job is now running. So let's go and see the logs. And the expectation here is that from the log, the queue message should be processed right. So you can see here that it has it is running the job continuously.

And you can see that he has called the process queue message. And you can see that, that it is saying that I've invoked this function, and if I click on this function, you can see that in the message parameter, the value is this is coming from the queue. So if I again go and put something in the in the data shift. So, let me go back here and let me put something in the queues, queues. Right. So let me go here and put something.

So I'll say this is this is coming from the queue again right. Let's see okay. And let us go back to our app services and let us see what happens. Let us go to our web job. And the expectation here is that web job should have ran again and invoke the process q message again right. So let us see that so you can see let us try to refresh this.

So let us refresh this you can see now there are two invocation of the function the first one, this one was 34 minutes back and this one is now so if I go and say, again, it is processed So basically here now, as soon as the message comes in the queue, the web job runs, takes the message from the queue and processes it. Now, if you remember, I had said that we do not have different different app service as such, you can just create one app service. And in that app service, you can go and host an web application. And also you can go and host an web job. So there can be scenarios, you know, where you can have multiple projects. So you can have one project, for example, this web job for which is actually doing the part of web job.

And also you can have one more project, you know, which is for web application like ASP. NET MVC application, right? So what you'd like to do is, you'd like to go and publish both of these projects in one go into the same app service, right. So for that, what you can do is you can see that now I have two projects here. One is web application two and one is web job for and I would like that both of these projects should get installed Or get allocated to this my app service ship, right? So for that, what you need to do is you can go and say here, add.

And you can say, existing. First thing, if you want to create a new as your web job, you can do it. But at this moment, we already have an existing project, right? So I'm going to say existing project, I want to go and add it into this project. So now what happens with this is if you see there is a small JSON file here called is web jobless Jason. And this web jobless JSON actually says that, when I go and say publish this project, when I go ahead and save publish this web project, you will also go and publish this web job.

So for example, if I now go and delete my web job, for example, it may go and delete my web job and then let us try to publish it via this web application. So you can see that it is deleting my web job for so if I now go and say bye If I go and save, publish, right? And let me publish, let me publish it as an as your website. So I will go and say okay, publish, okay publish what happened? Okay, publish and I want to go and publish the app to Azhar. So start so it's an app service existing one, right publish.

And already I have given my account name here you can see at the left hand side I've given this question interview question Gmail, right. So I want to go and publish the website as well as the web job into this I'm going to say okay, and that it is it is publishing bill started it. So let us give it some time and expedition here is that in world Go both the web application as well as the web job should get deployed. So that it is it is adding the necessary DLLs. So that you can see adding file, adding DLLs, adding the XML files, you know, so all the necessary files, what are needed are getting added both for the web application as well as for the web job. So once this is done over here, what should happen is in this web job, I should see my web job getting published.

You can see my web job food is published right out here. And also, I should see my website. So if I go and see so this is my website, so even my website of my MVC application should be seen. So okay, my app service is up and running. Right? Okay.

I hope the website it is Getting Okay, it's published okay. So, now, my website will get published here and my web job you can see has also got published. So, both of them have got published in one go. still loading. Right. So, if you want to go and add if you want to go and publish both web application as well as web job in one go, do not forget this.

In case you have an existing one, you can go and say add an existing project as as your web job. So whatever is a web job will get added to the web application. Or if you say, a new one, a new as your web job project will start right. So there you can see my MVC homepage is there and as well as now that job is there, right. So that brings us to the end of this session. And, you know, in this session, I was thinking to cover both web job and worker role.

But then later on, I saw that web job needs more attention and you If you look at the Microsoft roadmap, you know, they are more pushing developers, you know, to use the app service deployment model as compared to the cloud service deployment model, right? And a web job topic, you know, when analyzed, you know, it's going to take almost 40 to 50 minutes you can see that almost This video is now 40 minutes and about right. So I said that let us go part by part. So in this video, I have completed web job which is an important topic. In the next video, we will look into worker role, right? So I hope that you enjoyed this video.

This video was all about web jobs. The next video is worker role. Thank you very much.