CMS in Azure App Service and weird behavior
Posted on: March 4, 2024I was troubleshooting incident over a year ago with Azure App Service. Customer was running a Content Management System (CMS) on the App Service, and it started randomly doing something weird. Before this weird behavior, it has been running fine for years.
When troubleshooting, I made all the usual questions:
- Has something changed recently?
- Have you deployed a new version of the application?
- Have you changed some configurations or settings?
Nothing had really changed so we had to dig deeper.
But as many times before, when applications have been running for a long time, you might end up having more and more data in your system. This can happen with databases, and they start to become slower over time. You might be able to fix the situation by adding missing indexes but sometimes you need to study quite a lot more why performance decreases over time.
In this case the problem wasn’t the database. App Service ended up doing restarts and we didn’t quite understand why. At first glance platform metrics or logs weren’t indicating anything special to us.
In the discussions with the customer, we found out that they had background processing logic in the application. It was doing some kind of caching of image etc. files. It has been running again for ages so it should not cause any issues.
Suddenly, we found something interesting:
HostingEnvironment:
The hosting environment shut down the application domain.
This typically happens when the app is restarted either by user or platform
And then additional details:
Directory rename change notification for ‘D:\home\site\wwwroot’.
Overwhelming Change Notification in wwwroot
After seeing that error message, things started to become more clear. That specific word “overwhelming” already told me, that this might not be actually App Service specific issue but instead something underneath. App Service is a Platform as a Service (PaaS) offering and it does build on top of other existing products and capabilities. In this specific setup it means building on top of Windows, IIS and ASP.NET.
If you search for “overwhelming change notification” you will find millions of search results. That’s a lot! And many of these blogs are older than 10 years e.g., ASP.NET File Change Notifications, exactly which files and directories are monitored from year 2007. So, quite quickly you realize that this it’s not that uncommon issue and most likely you can find solution from these different blogs. Here are two example blogs with tons of good information azure-web-restarting-automatically-due-to-overwhelming-change-notification and all-about-aspnet-file-change-notification-fcn.
It’s quite much easier to solve the problem if you understand the problem.
Soon after the above finding, we understood the root cause. Indeed, that background processing logic which created files and folders was causing the issues in the application.
Before going to the solution, let’s first understand what happened.
I’ll create App Service with ASP.NET V4.8 as the runtime stack:
Then I’ll deploy my test application from here:
JanneMattila/webapp-and-folders
Key piece of code is taken from this repository (from the above blog posts):
It displays detailed debugging information about FCN configuration in the simple view.
After deploying the application and refreshing the FCN view, it shows me this:
That is pretty big list of things it is monitoring.
Let’s add some spices to the mix by adding two WebJobs to the application.
- WebJob that creates folders and files under
wwwrootfolder - WebJob that fetches those pages
Here is a simple PowerShell script to create 50000 folders and files:
$filePath = "C:\home\site\wwwroot"
Set-Location $filePath
mkdir a -ErrorAction SilentlyContinue
Set-Location a
for ($i = 0; $i -lt 50000; $i++) {
mkdir $i -ErrorAction SilentlyContinue
"<html><body><h1>Test $(Get-Date)</h1></body></html>" | Out-File "$i\index.html"
}
And here is the WebJob that fetches those pages:
for ($i = 1; $i -lt 50001; $i++) {
$i
iwr -UseBasicParsing "https://myfilesystemtestingapp.azurewebsites.net/a/$i" | Out-Null
}
We can now deploy those above scripts as WebJobs to the App Service and start them:
If we now refresh the FCN view, we can see that it monitors even more things:
So, indeed it’s expanding that list of things it is monitoring. This is the default behavior.
FCN can be configured using fcnMode.
If you run the above test using Single as the FCN mode, then
this would be visible in the debugging view:
The above list does not expand anymore. But if there are “overwhelming” number of changes, it might restart the application.
Please read the above mentioned blogs for more details about the behavior and especially before you plan to do any changes to your application.
In our case, we took the approach to disable FCN monitoring completely.
This can be done by adding fcnMode="Disabled" configuration to the web.config file:
<?xml version="1.0" encoding="utf-8"?>
<configuration>
<system.web>
<httpRuntime targetFramework="4.8" fcnMode="Disabled" />
After deploying this change and restarting the application, we can see that FCN monitoring is disabled:
But that does come with the price. Now we need to restart the application every time we deploy a new version of the application. This is because FCN is not monitoring changes in the application folder anymore.
Disabling FCN hurts your local development experience as well because you need to restart the application every time you make changes to the application. Therefore, you most likely want to enable FCN in your local development setup.
Few learnings from these kinds of cases:
- Learn to use Diagnose and solve problems feature in the App Service:
- Use diagnostic settings to push logs to Log Analytics, so that you can easily query them:
Originally, I wrote about this topic in my notes:
I hope you find this useful!