Is there a way to predictively detect inconsistent trusted storage?

Yes

Summary

Question

A publisher wants to add functionality that bridges the time if storage breaks so that their customers can use the system for another 5 days until it finally breaks (kind of a grace period) in order to not turn off critical client systems just because of license infrastructure problems.

Is there a way to programmatically (via api) check the integrity of the trusted storage (is it broken, is it untrusted, ...)?
What states can such a trusted storage (as a whole) have? or is the state always represented by the content?
How can we force these states programmatically within a unit test (eg force a broken trusted storage, force untrusted content, ...)?

They can provide an ASR to each customer and have their software automatically switch over to it under these use cases, but are there any methods available to gracefully detect the switchover? Or are they limited to "buffering" a failure and programmatically deciding what to do with it prior to alerting the user to a valid license checkout? Are there other options?

Answer

To detect inconsistent trusted storage:

The API to check whether trusted storage (TS) is trusted is flxActCommonProdLicSpcTrustFlagsGet
The state is either trusted or untrusted
One way to cause trusted storage to become untrusted is to restore an old version. For example:

? load an ASR
? back up TS
? generate an offline request (increments anchor sequence number)
? restore backed up TS. TS should now incur a RESTORE (anchor) break

Also see the attached Trusted Storage recovery best practices white paper (some info is a bit dated, but most is still relevant).

Additional Information

"trust" is a property of individual fulfillment records, not trusted storage as a whole. This is because FNP allows fulfillment records (FRs) to be created in different trusted sections with different binding and anchoring specifications. Typically, however, customers place FRs in one trusted section and in this case all will lose HOST and/or RESTORE trust together. TIME trust has an added factor - only expiring fulfillments lose TIME trust, not permanent ones.

The activation API functions flxActCommonLicSpcPopulateAllFromTS() and flxActCommonProdLicSpcTrustFlagsGet() can be used to determine the trust of each individual fullment record. If an attempt is made to check out a feature and the checkout can only be satisfied from a fulfillment record that has lost trust, the checkout will fail LM_REPAIR_NEEDED. In general it is not possible to determine which FR that was, so best practice in your case would be to load the grace period ASR then in due course repair (or re-install) all FRs that have lost trust.

Note that if the grace period ASR uses a trial anchor it can only be used once - a new one will need to be sent to the customer after the repair. If it does not use a trial anchor then exploits exist; some defence can be added by checking whether the grace FR already exists but backup/restore exploits that allow repeated use of the grace FR cannot be defended.