So in my inbox I find a report of an issue where the backup is failing on one of our servers.
The lack of detail error message from the backup software was simply:
[Critical] From: Agent@ServerName Time: 10/16/12 20:31:16 Cannot back up files with a pathname longer than 1024 characters!
No additional info.
This made me scratch my head, as the file system on the server is NTFS, so someone saving a file to a path longer than 260 including the filename would be unlikely. However I have previously seen backup software create this scenario, when backing up MacBooks to a Windows file server.
So first order of business, find the path with the problem.
Using psexec to run powershell as NT AUTHORITY\SYSTEM (to avoid access denied errors):
c:\>psexec -is powershell
and then running the following powershell command:
PS C:\> get-childitem -Recurse -ErrorAction STOP
it would only be a short time before the path was found.
But no joy. The command finished without errors.
So some more head scratching…
Using the backup application, I eventually found that the problem was in a users profile folder, more specifically once the backup application hit the “c:\users\theuser\AppData\Local\Application Data” folder it found another “Application Data” folder and then another underneath and another and another … until it hit the 1024 maximum and threw the error seen above
First the basics, from NT 6.0 (Vista and newer) this Application Data folder is actually a junction pointing to “c:\Users\theuser\AppData\Local”
C:\Users\theuser\AppData\Local>dir /ah Volume in drive C has no label. Volume Serial Number is F46B-1743 Directory of C:\Users\theUser\AppData\Local 02/20/2012 02:16 PM Application Data [C:\Users\theUser\AppData\Local] 02/20/2012 02:16 PM History [C:\Users\theUser\AppData\Local\Microsoft\Windows\History] 02/20/2012 07:23 PM 775,794 IconCache.db 02/20/2012 02:16 PM Temporary Internet Files [C:\Users\theUser\AppData\Local\Microsoft\Windows\Temporary Internet Files] 1 File(s) 775,794 bytes 3 Dir(s) 12,979,027,968 bytes free C:\Users\theUser\AppData\Local>
This has been put in place for legacy application support.
Under normal circumstances, this folder cannot be opened in Windows Explorer.
You can access the folder from an elevated command prompt, but that will just land you back in the “c:\Users\theuser\AppData\Local” folder
So why did the backup application decide that there was a newer ending nested group of “Application Data” folders?
Well the nested folders where actually there. But why…
The default permissions for the “Application Data” junction:
C:\Users\theUser\AppData\Local>icacls "Application Data" Application Data Everyone:(DENY)(S,RD) NT AUTHORITY\SYSTEM:(I)(OI)(CI)(F) BUILTIN\Administrators:(I)(OI)(CI)(F)
Notice the explicit DENY set for Everyone.
If this permission is removed, for instance by an admin who does not know that it is supposed to be there, then a browse into the folder with Windows Explorer will actually create a new folder, so that we now have “c:\Users\theUser\Appdata\local\Application Data\Application Data”
Do this enough times and we hit the error with the backup application. The above nested folder creation can also happen when using Robocopy
But why did the simple powershell test not catch this. Well, that is because I am a dork, and forgot about the fact of hidden folders… the command should have been:
PS C:\> get-childitem -Recurse -Force -ErrorAction STOP
Well now that this is cleared up, a cleanup task is required to remove the broken application data folder, but how to do that. Using Windows explorer – error. Appears that the mischievous admin broke some more permissions. That is easy to fix. First take ownership:
C:\Users\theUser\AppData\Local> takeown /F * /A /R /D Y
then, reset permissions:
C:\Users\theUser\AppData\Local> icacls * /reset /T /Q
But that errors out as well.
So I need a program that ignores permissions and can delete these long paths, and any file which may reside in the folders. Robocopy can do that:
Create a empty folder, say c:\dummy, then:
C:\Users\theUser\AppData\Local> Robocopy c:\dummy "C:\Users\theUser\AppData\Local\Application Data" /B /Purge
The /B instructs Robocopy to run in backup mode, which disregards file permissions. /Purge removes anything in the destination which is not in the source.
Final step is to fix the permissions on the Application data folder.
Note of course that the above does not consider the impact on the users profile, so use at own risk.