Search Engine: Elastic

Article ID: 129758, created on Oct 31, 2016, last review on Feb 23, 2017

  • Applies to:
  • Operations Automation 6.0
  • Business Automation 6.0

Symptoms

A Billing Upgrade Order fails with the message after 10 minutes:

"Change Service Plan failed: Operations Automation is not available. Connection to 172.16.3.12:8440 has failed: Connection timeout. Please check network settings.".

Cause

This case is related to hard-coded timeout of 10 minutes on the Billing side. Corresponding pem.upgradeSubscription call to OA is still runing, despite of the fact, that order has been already failed. This case has been submitted to the Maintenance Team as internal issue with id APS-37640: "Impossible to migrate subscription from LSH to NG in case it contains many domains".

Resolution

Option 1:

  1. Resubmit order for provisioning;

  2. Find the beginning of upgradeSubscription transaction in poa.log:

    INF [openapi:XX.XX.XX>XX:30140 1:21869:ae7f4b70 OpenAPI ]: ==>>> Executing OpenAPI method 'pem.upgradeSubscription' with parameters: [{ 'custom_resource_limits' => [ { 'limit' => '0', 'resource_type_id' => 1000010, }, ... 'subscription_id' => XXXXXXX,'txn_id' => '123456', } ]
    

    Check that you have found correct request by parameter subscription_id. Note txn_id field value;

  3. Wait for result of pem.upgradeSubscription request in poa.log (request can be identified by openapi prefix (For example openapi:172.16.3.10:42153)

    Like this:

    INF [openapi:172.16.3.10:42153 1:22531:ae7f6b70 OpenAPI ]: <<<== Method 'pem.upgradeSubscription' is executed, with status:
    ERR [openapi-firewall:ad8e7b70 1:22531:ad8e7b70 OpenAPI ]: [ iqxmlrpc-server] iqxmlrpc::Http_server_connection: Socket::send: Broken pipe
    

    Message "iqxmlrpc::Http_server_connection: Socket::send: Broken pipe" after pem.upgradeSubscription, means that this method was executed successfully, but you need to commit transaction.

  4. Perform manual commit for transaction:

    # curl -k -d@/tmp/txn.Commit.xml http://XX.XX.XX.XX:8440
    
    <?xml version="1.0" encoding="UTF-8"?>
    <methodResponse><...><name>status</name><value><i4>0</i4><...></methodResponse>
    

    Content of "txn.Commit.xml" (use the number from step 2):

    <?xml version="1.0"?>
    <methodCall>
    <methodName>txn.Commit</methodName>
    <params>
        <param>
            <value>
                <struct>
                    <member>
                        <name>txn_id</name>
                        <value><string>123456</string></value>
                    </member>
                </struct>
            </value>
        </param>
    </params>
    </methodCall>
    

    You can find additional information about this API request in OSA API documentation.

  5. Resubmit billing order again. If the order fails again, but OA completed necessary actions (customer's subscription's ST has been changed), then complete order manually.

Option 2: (in rare cases option 1 may not work)

  1. Get the whole set of parameters of pem.upgradeSubscription call from poa.log and run it manually, example:

    [root@core ~]# cat upgradeSubscription.py
    #!/usr/bin/env python
    from poaupdater.openapi import OpenAPI
    
    api = OpenAPI()
    
    api.pem.upgradeSubscription(custom_resource_limits = [ { 'limit' : '1', 'resource_type_id' : 1000010, }, { 'limit' : '10485760', 'resource_type_id' : 1000150, }, { 'limit' : '209715200', 'resource_type_id' : 1000152, }, { 'limit' : '30', 'resource_type_id' : 1000168, }, { 'limit' : '30', 'resource_type_id' : 1000169, }, { 'limit' : '5', 'resource_type_id' : 1000176, }, { 'limit' : '3145728', 'resource_type_id' : 1000177, }, { 'limit' : '30', 'resource_type_id' : 1000186, }, { 'limit' : '30', 'resource_type_id' : 1000374, }, { 'limit' : '30', 'resource_type_id' : 1000376, }, { 'limit' : '50', 'resource_type_id' : 1000417, }, { 'limit' : '5', 'resource_type_id' : 1000419, }, { 'limit' : '200', 'resource_type_id' : 1000420, }, { 'limit' : '30', 'resource_type_id' : 1000437, }, { 'limit' : '30', 'resource_type_id' : 1000438, }, { 'limit' : '30', 'resource_type_id' : 1000439, }, { 'limit' : '30', 'resource_type_id' : 1000440, }, { 'limit' : '30', 'resource_type_id' : 1000591, }, { 'limit' : '30', 'resource_type_id' : 1001803, }, { 'limit' : '30', 'resource_type_id' : 1002626, }, { 'limit' : '0', 'resource_type_id' : 1002647, }, { 'limit' : '30', 'resource_type_id' : 1002670, }, { 'limit' : '50', 'resource_type_id' : 1002700, }, { 'limit' : '25000', 'resource_type_id' : 1002706, }, { 'limit' : '20', 'resource_type_id' : 1002718, }, { 'limit' : '100', 'resource_type_id' : 1002724, }, { 'limit' : '50', 'resource_type_id' : 1002730, }, { 'limit' : '1556480', 'resource_type_id' : 1002736, }, { 'limit' : '1556480', 'resource_type_id' : 1002742, }, { 'limit' : '10485760', 'resource_type_id' : 1002748, }, { 'limit' : '10485760', 'resource_type_id' : 1002754, }, { 'limit' : '1048576', 'resource_type_id' : 1002760, }, { 'limit' : '100', 'resource_type_id' : 1002776, }, { 'limit' : '0', 'resource_type_id' : 1003167, }, { 'limit' : '30', 'resource_type_id' : 1003200, }, { 'limit' : '1', 'resource_type_id' : 1003560, }, { 'limit' : '-1', 'resource_type_id' : 1003561, }, { 'limit' : '50', 'resource_type_id' : 1003562, }, { 'limit' : '1048576', 'resource_type_id' : 1003580, }, { 'limit' : '0', 'resource_type_id' : 1003587, }, { 'limit' : '0', 'resource_type_id' : 1003592, }, { 'limit' : '0', 'resource_type_id' : 1003595, }, { 'limit' : '0','resource_type_id' : 1003598, }, { 'limit' : '0', 'resource_type_id' : 1003601, }, { 'limit' : '0', 'resource_type_id' : 1003604, }, { 'limit' : '0', 'resource_type_id' : 1003637, }, { 'limit' : '0', 'resource_type_id' : 1003640, }, { 'limit' : '50', 'resource_type_id' : 1003750, }, { 'limit' : '0', 'resource_type_id' : 1003753, }, { 'limit' : '0', 'resource_type_id' : 1003756, }, { 'limit' : '0', 'resource_type_id' : 1004002, } ], service_template_id = 121, subscription_id = 1003020)
    
    [root@core ~]# python upgradeSubscription.py
    
  2. Once the API call is completed successfully, re-submit the order.

198398b282069eaf2d94a6af87dcb3ff caea8340e2d186a540518d08602aa065 e12cea1d47a3125d335d68e6d4e15e07 3627d36199b8ff577605df76e2fa222b bb7e9177fb03488961a3ea554120f328 5356b422f65bdad1c3e9edca5d74a1ae 956c448bddc7e1f3585373687602379f 6f1456866eed87488c0f02b298a741c0

Email subscription for changes to this article
Save as PDF