How to set cancellation (penalty) fee for resource downgrade in customers' subscription?
No such functionality in the current release of PBA, it is going to be implemented in the future versions of the product in scope of the request #PBA-50330.
As an alternative 'No refund' value for the 'After Refund Period' parameter can be set on subscription period in a Service Plan. In this case prepaid amount for the resource is not being refunded in case of downgrade.
In the default PBA installation all downgrade orders are being created in the 'New' status and must be approved by a Provider staff member before processing. So, Provider staff members have full control on the downgrade orders processing in PBA and may add adjustment to a downgrade order manually.
Also, the possibility to prohibit placing downgrade orders in the PBA Customer Control Panel is going to be implemented in the future versions of the product in scope of the request #PBA-36895.