Article ID: 125784, created on Jun 8, 2015, last review on Jun 8, 2015

  • Applies to:
  • Operations Automation 6.0
  • Operations Automation 5.5

Symptoms

After update of Windows Azure Pack environment to UR6 all WAP-related tasks fail with errors like:

Task ID 37887991
Queue name      aps_application_instance54406 subscription_id1
Task name       Provisioning "AddonInstallationTask" for APS application Windows Azure Pack APS Package
Task description Provisioning resource 254ee2b1-553d-47d2-90fd-75e4a41df2ca of type 'addon-task' (http://www.parallels.com/infrastructure/internal/addon-task/1.0) for APS application 'Windows Azure Pack APS Package-1.4-297'
Subscription ID 1
Queue status    Failed
Start not earlier than  Jun-08-2015 15:35
Method name     taskExecuteWorkflow on SCREF:SaaSManagerTasks:0
Last execution output   

APS Application Error:
406 Not Acceptable
[ApplicationUnknownError] Provisioning: resource 254ee2b1-553d-47d2-90fd-75e4a41df2ca of type 'addon-task' (http://www.parallels.com/infrastructure/internal/addon-task/1.0) for APS application 'Windows Azure Pack APS Package-1.4-297': P100001: Operation failed; details: [Failed to synchronization subscription , 1082631].

Body: {
"error": "ApplicationUnknownError",
"message": "P100001: Operation failed; details: [Failed to synchronization subscription , 1082631]",
"http_request": "POST /addon-task to '0.0.0.0:4485'"
}

The following error apper in /var/log/WAP/wap.log:

2015-06-08 16:25:07,810 INFO  LoggingFilter [apsc(5)] - 70 * [WAP->AZURE] Client out-bound request
70 > POST https://wapadminapi.ccccloud.com/subscriptions/a7056607-9fee-4bdc-9a0a-6900e37079d2/addons
70 > Authorization: <hidden>
70 > X-ReqId: 548
70 > Accept: text/plain,application/json
70 > Content-Type: application/json
{
  "AddOnId" : "GBAzuhskh0ild"
}

2015-06-08 16:25:16,201 INFO  LoggingFilter [apsc(5)] - 70 * [WAP<-AZURE] Client in-bound response
70 < 500
70 < X-AspNet-Version: 4.0.30319
70 < Date: Mon, 08 Jun 2015 13:25:16 GMT
70 < Content-Length: 206
70 < Expires: -1
70 < Content-Type: application/json; charset=utf-8
70 < X-Powered-By: ASP.NET
70 < Server: Microsoft-IIS/8.5
70 < Pragma: no-cache
70 < Cache-Control: no-cache
70 <
{"Code":"ErrorFromUnderlyingNotificationSubscribers","Message":"One or more errors occurred while contacting the underlying notification subscribers. The operation may be partially completed.","Details":[]}

The following error apper in Event Viewer on the WAP node in MgmtSvc-Usage\Operational log:

Publishing notification to an end point failed with the exception: System.Net.Http.HttpRequestException: An error occurred while sending the request. ---> System.Net.WebException: Unable to connect to the remote server ---> System.Net.Sockets.SocketException: No connection could be made because the target machine actively refused it 172.16.32.41:4484
   at System.Net.Sockets.Socket.EndConnect(IAsyncResult asyncResult)
   at System.Net.ServicePoint.ConnectSocketInternal(Boolean connectFailure, Socket s4, Socket s6, Socket& socket, IPAddress& address, ConnectSocketState state, IAsyncResult asyncResult, Exception& exception)
   --- End of inner exception stack trace ---
   at System.Net.HttpWebRequest.EndGetRequestStream(IAsyncResult asyncResult, TransportContext& context)
   at System.Net.Http.HttpClientHandler.GetRequestStreamCallback(IAsyncResult ar)
   --- End of inner exception stack trace ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.WindowsAzure.Server.Usage.Service.NotificationRequestSender.<PostAsync>d__0`1.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.WindowsAzure.Server.Usage.Service.NotificationRequestHandler.<PublishNotificationsAsync>d__1`1.MoveNext()

Cause

Rollup installation changed settings on WAP node, it could be checked with command below:

PS C:\Windows\system32> get-MgmtSvcNotificationSubscriber

Name                   : Billing
Enabled                : True
SubscriberType         : BillingService
Endpoint               : https://wap-aps:4484/wap/billing
AuthenticationMode     : Basic
AuthenticationUsername : wap
AuthenticationPassword : pass_word

Port was changed to 4484 instead of 4486 and trailing slash was removed

Resolution

Apply proper configuration with:

PS C:\Windows\system32> Set-MgmtSvcNotificationSubscriber -Name Billing -SubscriberType BillingService -Enabled $true -Endpoint 'https://wap-aps:4486/wap/billing/' -AuthenticationMode Basic -AuthenticationPassword 'pass_word' -AuthenticationUsername 'wap' -force

And restart IIS after:

PS C:\Windows\system32> iisreset

Attempting stop...
Internet services successfully stopped
Attempting start...
Internet services successfully restarted

Search Words

task failed - Provisioning "AddonInstallationTask" for APS application Windows Azure Pack APS Package

Unable to connect to the remote server

406 Not Acceptable

WAP

AddonInstallationTask

5356b422f65bdad1c3e9edca5d74a1ae caea8340e2d186a540518d08602aa065 e12cea1d47a3125d335d68e6d4e15e07 956c448bddc7e1f3585373687602379f 6f1456866eed87488c0f02b298a741c0 5b048d9bddf8048a00aba7e0bdadef37 2554725ed606193dd9bbce21365bed4e

Email subscription for changes to this article
Save as PDF