Random Clients fail to download content (0x80070003)



******Update July 27 2017:  I have confirmed that the below issue is addressed via: KB4035759.  Thanks to all who helped! ******



This posts outlines an issue that I am seeing in Configuration Manager starting in 1702. I currently have a call open with MS and will update the post once the issue is resolved. I am currently aware of 2 environments with the issue and am posting this in case this is also an issue for others. If you have this issue in your environment, please shoot me a message with any cases you have open so Microsoft can see the commonalities in our environments.

Steps to Reproduce:
1. Issue occurs on newly built computers (this may be for the simple fact as they are installing the most software). We have reproduced the issue on computers built with our current gold image, last December’s gold image as well as using a new task sequence and VLSC ISO. Failure rates are around 20% so a minimum of 5 computers should be built. For our tests we built 15 computers total all in the same subnet.
2. Mass add computers to software deployment collections

Continue reading

Advertisements

Silent Scripted PNP Driver Installation

Occasionally, you may find the need to push a new driver to computers.  Perhaps a driver is causing BSOD issues or whatever the reason.  Since DotNet does not have a direct way to do this, you are usually left with depending on the driver publisher to include an silent installation method.  In reality this rarely happens.  You definitely don’t want to run around and manually install the drivers, and tools like Configuration Manager don’t have support for post OS deployment of drivers.

Continue reading

Azure Pack Automation (SMA) Get-SMAJobOutput fails

I recently ran into an issue where regardless of the method used, I was unable to get any job output from Service Management Automation.

Get-SMAJobOutput -JobID '5c773933-5a9b-4021-a793-768b2efd0165' -WebServiceEndPoint $WebServiceEndpoint -Stream Output

simply returned:

Get-SmaJobOutput : The job 5c773933-5a9b-4021-a793-768b2efd0165 cannot be found: System.Data.Services.Client.DataServiceClientException: <?xml version="1.0" 
encoding="utf-8" standalone="yes"?><error xmlns="http://schemas.microsoft.com/ado/2007/08/dataservices/metadata"><code></code><message xml:lang="en-US">An error 
occurred while executing the command definition. See the inner exception for details.</message></error>
   at System.Data.Services.Client.BaseAsyncResult.EndExecute[T](Object source, String method, IAsyncResult asyncResult)
   at System.Data.Services.Client.QueryResult.EndExecuteQuery[TElement](Object source, String method, IAsyncResult asyncResult)
At line:1 char:1
+ Get-SmaJobOutput -Id $id -WebServiceEndpoint $webserviceendpoint
+ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
    + CategoryInfo          : NotSpecified: (:) [Get-SmaJobOutput], InvalidOperationException
    + FullyQualifiedErrorId : System.InvalidOperationException,Microsoft.SystemCenter.ServiceManagementAutomation.GetSmaJobOutput

Note that this is not the issue where the output fails with the same error if using the wrong -Stream case.

Additionally, getting the job output from the Azure Pack admin site, simply returned a red exclamation mark.

Next, I adapted a REST based function to get the output from the REST api (originally posted here: http://www.laurierhodes.info/?q=node/105)

# Original script from http://www.laurierhodes.info/?q=node/105
# Modified by Christopher Keyaert (christopher.keyaert@inovativ.be)
 

 Function Get-SMARESTJobOutput{
    Param(
        [Parameter(Mandatory=$true)]
        [string]$ID,
        [Parameter(Mandatory=$true)]
        [validateset('Any','Progress','Output','Warning','Error','Debug','Verbose')]
        [String]$stream,
        [string]$webServiceEndPoint,
        [System.Management.Automation.PSCredential]$Credential
    )
# Ignore SSL Self-Signed certificate  
Try{
    [System.Net.ServicePointManager]::CertificatePolicy = new-object IgnoreSelfSignedCertificate
}Catch{
add-type @"
    using System.Net;
    using System.Security.Cryptography.X509Certificates;
 
        public class IgnoreSelfSignedCertificate : ICertificatePolicy {
        public IgnoreSelfSignedCertificate() {}
        public bool CheckValidationResult(
            ServicePoint sPoint, X509Certificate cert,
            WebRequest wRequest, int certProb) {
            return true;
        }
    }
"@ -ErrorAction SilentlyContinue
    [System.Net.ServicePointManager]::CertificatePolicy = new-object IgnoreSelfSignedCertificate
} 
    $SMAServer = "${WebServiceEndpoint}:9090"
    $VerbosePreference = "Continue"
    #$VerbosePreference = "SilentlyContinue"
 
    Write-Verbose ""

    $URI =  "$SMAServer/00000000-0000-0000-0000-000000000000/Jobs(guid'" + $ID  + "')"
    $Response = if($Credentials){Invoke-RestMethod -Uri $URI  -Method Get -Credential $credentials}else{Invoke-RestMethod -Uri $URI  -Method Get} 
    $JobStatus = $Response.entry.properties.JobStatus 
    
    Write-Verbose "Job Status = $JobStatus"
   
    $URI =   "$SMAServer/00000000-0000-0000-0000-000000000000/JobStreams/GetStreamItems?jobId='" + $ID +"'&streamType='$Stream' "
    Try{
        $Result = if($Credentials){Invoke-RestMethod -Uri $URI  -Method Get -Credential $credentials}else{Invoke-RestMethod -Uri $URI  -Method Get}
    }Catch [exception]{
        $Exception = $_
    }
  
    Write-verbose "StreamText = $($Result.content.properties.StreamText.InnerText)"
 
    $outputname = "Stream${Stream}"
    New-object -TypeName PSCustomObject -ArgumentList @{'ID'=$ID;'Status'=$JobStatus;$outputname=$Result.content.properties.StreamText.InnerText}
}

This returned the same error as Get-SMAJobOutput (An error
occurred while executing the command definition. See the inner exception for details)

The more I researched this, the more it smelled like a performance issue querying the database.
After doing a bit of digging, I found that my [SMA].[Stream].JobStreams table was ~17,000,000 rows. Using the following command, I dropped the job history down to 30 days (we had previously set it to 90):

Set-SmaAdminConfiguration -PurgeJobsOlderThanCountDays 30 -WebServiceEndpoint $webserviceendpoint

This took 8 hours to run and dropped the table down to 9,000,000 rows. The issue still remained, so I dropped it further to 25 days and also disabled debugging and process logging on all runbooks. This dropped the table down to 7,500,000 rows. Now the issue occurred ~75% of the time….Progress!

Since I had now verified that it was an SQL performance issue, I turned to the Activity Monitor of SQL Studio.
Sure enough, the JobStreams table did not have an index usable to query the data required by the Get-SMAJobOutput powershell command. Adding the following index dropped the query time to instantaneous:

USE [SMA]
GO
CREATE NONCLUSTERED INDEX [NC_JobIDTenantID]
ON [Stream].[JobStreams] ([TenantId],[JobId])
INCLUDE([StreamTime],[StreamTypeName],[Stream],[StreamText],[RunbookVersionId])
GO

The issue was now resolved! Please note that this fix may not be supported by Microsoft. Please use at your own discretion.

ConfigMgr Client Fails to Install: Unable to Compile UpdatesAgent.mof

We’ve had a couple of computers in the past being unable to re-install the Configuration Manager client due to the error:
“Unable to compile UpdatesAgent.mof”

This error can have a couple of different causes.

As such, here are a couple of steps you can try:

1. Reinstall the Windows Update agent. https://support.microsoft.com/en-ca/kb/949104
2. Uninstall any existing ConfigMgr client, stop the ccmsetup service and delete c:\windows\ccm, c:\windows\ccmsetup and c:\windows\ccmcache folders
3. Run the following commands to delete the ConfigMgr namespaces completely from WMI:

Gwmi –query “Select * from __Namespace Where Name=’sms’” –NameSpace “root\cimv2” | Remove-WmiObject
Gwmi –query “Select * from __Namespace Where Name=’ccm’” –NameSpace “root” | Remove-WmiObject
Gwmi –query “Select * from __Namespace Where Name=’smsdm’” –NameSpace “root” | Remove-WmiObject

Since #3 is quite drastic, you will want to try steps 1 and 2 first before 3. However if attempting step 3, you will want to complete both steps 2 and 3 together. After this, the ConfigMgr client should successfully install.

Hopefully this helps!

Content Library Explorer – The Legacy Package Does Not Have Exactly One Content

I recently ran into an issue, where my Primary site server was running low on disk space. This turned into a general spring cleaning of the ConfigMgr environment. As part of the cleanup process, I wanted to check the distribution points for old or stale packages.

Microsoft has provided a toolkit for cleanup operations such as this:
https://www.microsoft.com/en-us/download/details.aspx?id=50012

Part of this toolkit is the Content Library Explorer. However, after aiming this at my distribution point, I was confronted with the following error:

ContentLibExpl-LegacyPackage1Content

Not exactly an insightful message. I did however find a useful thread regarding this issue:
https://social.technet.microsoft.com/Forums/en-US/c7757792-3e5c-41c3-bb74-e57b0fe7258f/content-library-explorer-says-the-legacy-package-does-not-have-exactly-one-content?forum=configmanagergeneral

Using the provided script, I happily identified 3 packages that were causing issues. I simply removed the extra old folders and redistributed these. The extra folders were now gone, but the error message remained.

After doing some more digging with procmon, I identified the verification steps the content explorer appears to make as well as 3 possible different problems which could lead to the above error message.

1. More than 1 data folder exists for a given package in the datalib subfolder of the SCCMContentLib folder. (This is addressed by the script in the link above)
2. There exists an ini file in the pkglib subfolder of the SCCMContentLib folder, but the associated ini file in the datalib folder is missing.
3. There are multiple content versions listed in the ini file located in the pkglib folder.

I have written the following function to test the SCCMContentLib folder for problems. Problems that are found by this script are fixable by removing any extra folders for the given package from the DataLib folder, removing the distribution point from the package, waiting for the files to disappear and redistributing the package to the distribution point.

Function Test-DPLegacyContent{
    Param($DPFolderPath)

    #Calculate child folders
    $pkgdir = join-path -path $DPFolderPath -ChildPath 'pkglib'
    $datadir = join-path -path $DPFolderPath -childpath 'datalib'
    $childdatafolders = Get-ChildItem -Directory $datadir

    ForEach($file in (get-childitem -file $pkgdir)){
        $filecontent = Get-content $file.FullName
        $expectedcontent = $filecontent.split('`n')[1].replace('=','')
        if($expectedcontent -match $file.basename){
            #legacy package
            $packageID = $file.basename
            #Check for missing INI files
            if(!(test-path (join-path $datadir -ChildPath "${expectedcontent}.ini"))){
                [pscustomobject]@{'PackageID'=$PackageID; 'Error'="Ini file missing in datalib for $packageID"}
            }

            #Check for mismatch in folder count
            [array]$matchingFolders = [array]($childdatafolders | Where{$_.Name -match $packageID})
            $foldercount = $matchingfolders.count
            if($foldercount -ne 1){
                [pscustomobject]@{'PackageID'=$PackageID; 'Error'="$foldercount folders found"}
            }
        }
    
        #Check for multiple content versions in pkg ini
        if(($filecontent.split('`n')[2].replace('=','')) -match $file.basename){
            [pscustomobject]@{'PackageID'=$PackageID; 'Error'="Multiple package versions found in pkglib ini"}
        }
    }
}

Test-DPLegacyContent -dpfolderpath '\\DPServer\d$\SCCMContentLib\'

To use this script, simply change the dpfolderpath parameter to the path of your SCCMContentLib.

SysJam Powershell RightClick Tool – Part 6 – Getting Running ConfigMgr Jobs with Powershell

One of the key functionality improvements I wanted to include in the Sysjam Powershell RightClick tool was realtime running job monitoring. There are few tools that don’t require you to click some type of refresh button to see the running jobs.

Part 1 of providing this functionality is the using powershell jobs combined with a timer object to polling any data on a refresh cycle and updating the form. I have covered this previously here

Part 2 of this is querying WMI for the status of each running jobs…and translating this to english.

This post will concentrate on Part 2

Continue reading

SysJam Powershell RightClick Tool – Part 5 – By-Passing User Logon Requirement for a Program

Most of the time when deploying software I’ll set the program to run only with the user logged off. This is to avoid situations when the user may have an older version of the application open when they receive the advertisement. For testing however, this can be a pain….which is why I included the “ByPass User Logon Requirement (Temporary)” button in the Sysjam Powershell RightClick tool. This button sets the requirement to “None” temporarily within WMI. The next time the system does a Machine Policy refresh this setting gets overwritten.
Continue reading

Installing DotNet 3.5 on Server 2012 R2 using an offline provisioned image

Consider the following situation:

You have a custom captured wim of Windows Server 2012 R2 that was built before August 2014.
This wim has served you well, but the amount of updates required on servers built by this image means you want to update this image. In this case, you use the offline servicing SCCM feature to update your image.

You also have a task sequence which installs DotNet 3.5 offline, and you are using this task sequence with standalone media.
You use the following command to do this:
Dism /online /enable-feature /featurename:netfx3 /norestart /all /source:D:\Source\sxs
Continue reading

Starting and waiting for an SMA runbook from remote computers

If you are going to manually trigger a SMA runbook from a single computer you’ll want to install the Powershell module from the Orchestrator setup splash screen by clicking on PowerShell administration under Service Management.

However in order to deploy this to a team of administrators we’ll want to automate this. According to the technet library we should be able to use the command:

msiexec.exe /i PowershellModuleInstaller.msi /qn

We can get this PowershellModuleInstaller.msi file from the System Center 2012 R2 Orchestrator 2012 R2 iso in the SMA subfolder.

Continue reading

WSUS/SCCM – Tracking Down Updates with Failed Eula Downloads

By running the report “Troubleshooting 1 – Scan errors” you can identify some errors which may be preventing computers in your environment from properly reporting compliance for Windows Updates.

One of the errors you may see in here is -2145124301 which translates to “License terms could not be downloaded”. If there are large numbers of computers reporting this error, you may have an issue with a specific update. Continue reading

SysJam Powershell RightClick Tool – Part 4 – Getting Users Logged onto a Remote Computer

This post is a combination of the Get-LoggedOnUser script by Jaap Brasser available on the technet gallery here and my post on powershell jobs available here. I am taking both of these concepts and wrapping it all inside my powershell right click tool for SCCM available at https://github.com/mdedeboer/SysJamRightClickToolForConfigMgr.

As you may know, there is no nice way to check a user’s session information with any out-of-the-box powershell cmdlet. Additionally, you could try query this from the SCCM client via wmi, however you’ll be hard pressed to get this to include disconnected terminal sessions as well as console sessions. At least however there is a command available for this. It is called qwinsta or if you prefer its alias quser or another alias “Query User”. (Note there is also a rwinsta command with its alias “Reset Session”).

If you were to type quser into your powershell console however it just spits out text. No nice objects. So if we were to try something like “quser /server:127.0.0.1 | Select-Object -Property Username” you’d get a whole lot of nothing. We’ll need to parse the output ourselves. Thankfully Jaap Brasser has done much of the work for us! This script block will be run by a separate powershell job, so in order to make sure I get all the data I have added each custom object holding the user session information into a hash table that I can pass back to my powershell job. I could just have easily used an array, so no specific reason here.

Lets take a look:

$sbGetLoggedOnUsers = {
	Param ($CompName)     #CompName predefined
	$HashTableCollection = @{ }  #Hashtable to keep all the user sessions in
	$i = 0
	quser /server:$CompName 2> "" | Select-Object -Skip 1 | ForEach-Object { #Get the user information (ignore any errors), skip the title line, for each line after the first line.....
		$CurrentLine = $_.Trim() -Replace '\s+', ' ' -split '\s'  #get rid of repeating whitespaces and split on whitespaces (gives an array of strings)
		$HashProps = @{      #Pre-define a hashtable of properties
			UserName = $CurrentLine[0] #first string in $Currentline
			ComputerName = $CompName
			SessionName = $null
			ID = $null
			State = $null
			IdleTime = $null
			LogonTime = $null
		}
		If ($CurrentLine[2] -eq 'Disc')  #If the session is disconnected quser gives a different layout than otherwise
		{
			$HashProps.SessionName = $null #since the session is disconnected it doesn't have a name
			$HashProps.ID = $CurrentLine[1] #second string in $Current line
			$HashProps.State = $CurrentLine[2] #third string in $Current line
			$HashProps.IdleTime = $CurrentLine[3] #forth string in $Current line
			$HashProps.LogonTime = $CurrentLine[4..6] -join ' ' #fifth to seventh strings in $current line
		}
		else
		{
			$HashProps.SessionName = $CurrentLine[1] #second string in $Current line
			$HashProps.ID = $CurrentLine[2] #third string in $Current line
			$HashProps.State = $CurrentLine[3] #forth string in $Current line
			$HashProps.IdleTime = $CurrentLine[4] #fifth string in $Current line
			$HashProps.LogonTime = $CurrentLine[5..7] -join ' ' #sixth to eighth strings in $current line
		}
		$UserObject = New-Object -TypeName PSCustomObject -Property $HashProps | Select-Object -Property UserName, ComputerName, SessionName, ID, State, IdleTime, LogonTime #Create a new powershell object holding all the properties defined in the hashtable
		if (!($HashTableCollection.ContainsKey($i)))
		{
			$HashTableCollection.Add($i, $UserObject) #Add the custom object to another hashtable
		}
		$i++ #Next user
	}
	return $HashTableCollection #Return hashtable of all user sessions
}

If you are following along at home, you’ll notice a few differences with my version than Jaap’s (besides the script block and extra hash table).
1. quser /server:$CompName 2> “” | Select-Object -Skip 1 | ForEach-Object {
The 2> “” silently discards any error data. This allows us to be sure of what we are parsing.
2. $HashProps = @{
UserName = $CurrentLine[0]
ComputerName = $CompName
SessionName = $null
ID = $null
State = $null
IdleTime = $null
LogonTime = $null
This pre-defines the hashtable’s keys before it has values which provides more consistency.

After the script block is complete you essentially have something similar to this returned (where PSCustomObject is the custom object holding each user’s session information):

@{
   1=PSCustomObject
   2=PSCustomObject
}

The next function basically triggers the scriptblock and when it completes updates a pre-created datagridview with the following columns: Username, ComputerName, SessionName, ID, State, IdleTime, LogonTime. These columns could be named anything, just so long as they are in this order.

If you want the code for the Add-JobTracker function, see my post on this here

function fnLoggedOnUsers
{
	$statusBar1.Text = "Getting Logged On Users"
	if ($dGUsers.Rows.Count -ne 0)
	{
		$dGUsers.Rows.Clear()
	}
	$strComputer = $tbCompName.Text

	if ($strComputer -and $strComputer -ne "")
	{
		Add-JobTracker -Name (Get-Random) `
					   -JobScript $sbGetLoggedOnUsers `
					   -CompletedScript {
			Param ($Job)
			$HashTableUsers = @{ }
			$HashTableUsers = Receive-Job -Job $Job
			ForEach ($key in $HashTableUsers.keys)
			{
				$dGUsers.Rows.Add($HashTableUsers.Item($key).UserName, $HashTableUsers.Item($key).ComputerName, $HashTableUsers.Item($key).SessionName, $HashTableUsers.Item($key).ID, $HashTableUsers.Item($key).State, $HashTableUsers.Item($key).IdleTime, $HashTableUsers.Item($key).LogonTime)
			}
			$statusBar1.Text = ""
		}`
					   -UpdateScript {
		}`
					   -ArgumentList $strComputer
	}
}

So what is happening here? This is all covered in my previous post, but here is a high-level overview:
1. Update a label with the value: “Getting Logged On Users”
2. Clear the datagridview
3. Get the computer name from a text box
4. If the computer name is something, spawn a new powershell job that runs the script block we created above with the computername passed as a parameter.
5. A timer (defined in blog post part 2) checks each powershell job to see if it is finished.
6. When the job is finished the -CompletedScript {} script block is executed in the original thread. This script block uses the output of the first script block as it’s input. For this reason we can be confident that we are getting a hashtable of custom powershell objects as input. We then add each item from the hashtable to the datagridview.

SysJam Powershell Right Click Tool – Part 3 Doing more than 1 thing at once more – Powershell Runspaces and WMI Eventing

I’ll be the first to admit…this is an advanced concept post.  If you think Powershell is cool and you want to learn some pretty crazy things, keep reading.  This post is based loosely on the Process monitoring tab of the SysJam Right Click tool for Configuration manager available at https://github.com/mdedeboer/SysJamRightClickToolForConfigMgr.  I say loosely because I wasn’t satisfied with how much wmi querying it took to get all the information from the remote computer all the way back to you.  I hope to get the code used here into the right click tool sometime soonish…I’m just not sure when yet. Continue reading

SysJam Powershell Right Click Tool – Part 2 Doing more than 1 thing at once – Powershell Jobs

When you build powershell forms for the first time you will notice that whenever you execute code while your form is open everything appears to freeze. What is happening is that you are utilizing the same powershell thread is used to update your form (including when a user tries to move it/scroll etc…) and to also run your code.

To get around this we have two options. The most straight forward one of the two is a little slower running but much easier to code. This is using Powershell jobs. The second option is much quicker to execute and more flexible….it will also require you learn gymnastics within Powershell. This is using synchronized objects within Powershell runspaces.
Continue reading

SMA – Time to Execution Performance

The following TechNet blog post from the Orchestrator team can be used for measuring stress and load performance of your SMA environment: Configure SMA for Optimum Performance

This got me wondering what other ways I could improve performance. I noticed one key area in my scripts was the time while the script was in the “starting” phase. We know that SMA compiles the workflows at run time…so that was my target area.
Continue reading