HIP-367 : Remove Token Association Limit

HIP/hip-367.md

@@ -0,0 +1,236 @@
+---
+hip:367
+title: HIP Unlimited token associations per Account
+author: Anirudh Ghanta <anirudh.ghanta@hedera.com>
+working-group: Richard Bair<richard.bair@hedera.com>, Jasper Potts<jasper.potts@hedera.com>, Michael Tinker<michael.tinker@hedera.com>, Anirudh Ghanta<anirudh.ghanta@hedera.com>
+type: Standards Track
+category: Service
+needs-council-approval: Yes
+status: Draft
+created: 2022-02-17
+discussions-to: [Discussion thread](https://github.com/hashgraph/hedera-improvement-proposal/discussions/380)
+updated: 2022-03-01, 2202-03-04
+---
+
+## Abstract
+Permits every account to hold an unlimited number of token associations. Retains the current pricing for each token association. Modifies the number of token associations returned in the getAccountInfo and getAccountBalance query.
+
+## Motivation
+The current Services API permits the user to have at most 1,000 token associations per account.
+A user can work around this limitation by creating multiple accounts, each of which is limited to
+1,000 token associations. For some use cases, this 1,000 token association limit is frustrating.
+Due to the current implementation, it makes state proofs large and expensive. In addition, the
+current implementation imposes excessive costs as the number of token associations per account
+grows.
+
+Finally, returning large numbers of token associations in the free queries can lead to excessively long query response times.
+
+## User stories
+As a user I should be able to associate to any number of tokens without any limitations under one account instead of dispersing them across multiple accounts.
+
+- As a user I should be able to pay for as many token associations upfront as I want to, using the automaticAssociationSlots 
+- As a primary creator of an NFT collection, I should be able to mint a large number of NFT serials in a single collection
+- As a secondary NFT marketplace dApp, I should be able to create many thousands of collections (created by multiple primary creators of the collections), such that each collection can hold a large number of NFTs
+- As an avid collector of NFT, I want to manage my entire collection within the same account.
+
+## Specification
+ Currently, we have a limit of 1000 token associations per Account put in place which is dictated by the bootstrap property `tokens.maxPerAccount`.
+This is enforced everytime we associate token/tokens to an account. 
+
+When using the HederaTokenStore :
+```java
+    public ResponseCodeEnum associate(AccountID aId, List<TokenID> tokens, boolean automaticAssociation) {
+      ...
+          if ((accountTokens.numAssociations() + tokenIds.size()) > properties.maxTokensPerAccount()) {
+              validity = TOKENS_PER_ACCOUNT_LIMIT_EXCEEDED;
+          }
+      ...
+    }
+```
+
+When using the Account model:
+```java
+    public void associateWith(List<Token> tokens, int maxAllowed, boolean automaticAssociation){
+        final var alreadyAssociated=associatedTokens.size();
+        final var proposedNewAssociations=tokens.size()+alreadyAssociated;
+        validateTrue(proposedNewAssociations<=maxAllowed,TOKENS_PER_ACCOUNT_LIMIT_EXCEEDED);
+        ...
+    }
+```
+
+This limit enforcement needs to be removed. This limit is also enforced when creating and updating an account with maxAutomaticAssociations described by this [HIP-23](https://github.com/anighanta/hedera-improvement-proposal/blob/master/HIP/hip-23.md)
+That enforcement also has to be removed.
+
+Currently, the list of tokens associated to an account are saved on the state using a MerkleLeaf node `MerkleAccountTokens` which holds the array of Ids of the token as `CopyOnWriteIds` and each MerkleAccount will have this as a child. 
+The new way of efficiently doing this is by using a `MapValueLinkedList` data structure which will be represented by Account-Token relationship Map of <EntityNumPair, MerkleTokenRelStatus> in which `MerkleTokenRelStatus` serves as a node of a doubly linkedList. 
+The `nextKey` and `prevkey` in this `MerkleTokenRelStatus`will point to the next and previous token relationships of the account. And only the latest account-token association key is saved in MerkleAccountState.
+And the `TokenAssociationMetadata` encapsulates three properties of the MerkleAccount
+1. The total number of tokens that this account is associated with `numAssociations`
+2. The total number among these associations which have zero balance `numZeroBalances`
+3. And finally the latest association represented as an `EntityNumPair` of MerkleAccount's id and token's id `lastAssociation` 
+
+![img.png](../assets/hip-1000/updates.png)
+
+These fields are tracked / updated at :
+1. `numAssociations` -- whenever a token associates and dissociates [explicit and auto associations] and during migration
+2. `numZeroBalances` --
+   1. whenever a token associates or dissociates with an account.
+   2. tokenWipe operation is performed with the account's token balance becoming zero.
+   3. tokenBurn operation is performed with treasury's balance becoming zero.
+   4. tokenMint operation is performed with treasury moving away from zero balance.
+   5. cryptoTransfer leaving the sender with zero balance or receiver moving away from zero balance.
+
+#### Migration
+When loading older states, use the deprecated `MerkleAccountTokens` to deserialize the associated tokens of an account 
+from the state during migration. Then use the Map<EntityNum, MerkleAccount> accounts 
+and Map<EntityNumPair, MerkleTokenRelStatus> tokenAssociations maps which will not have any of the new fields populated to fill out the newly added fields.
+All of this has to be done during the `migrate()` call in `ServicesState`
+
+```
+Algorithm:
+1. for each account in the accounts map fetch the list of tokenIds from `MerkleAccountTokens`.
+2. for each token on this list, use the account id and token id to build `EntityNumPair` and fetch the MerkleTokenRelStatus from the tokenAssociations map.
+3. update the nextKey and prevKey for each of these associations and persist the changes.
+4. increment the `numAssociations` on the account's tokenAssociationMetadata by 1.
+5. if the balance on the association is zero, then increment the `numZeroBalances`  on the account's tokenAssociationMetadata by 1.
+6. set the account's `lastAssociatedToken` to the last key that is built in the above loop.
+7. Finally remove the 3rd child on each MerkleAccount.
+```
+
+### Token Association
+As the account's `lastAssociation` from its `tokenAssociationMetadata` will always have the latest association key which suggests that we always insert at the head of the linkedList.
+This should cover the Association logic in `Account.associateWith()` which handles explicit token associations and `HederaTokenStore.associate()` which handles automatic associations. 
+
+```
+Algorithm:
+1. for each token that is to be associated to the Account, build `EntityNumPair` from the tokenId and accountId.
+2. create a new `MerkleTokenRelStatus` object and set the nextKey as the current `lastAssociation` from its `tokenAssociationMetadata`.
+3. update the account's `lastAssociation` to the new `EntityNumPair` that is built in step 1.
+4. increment the `numAssociations` on the account by 1 for each associating token.
+5. increment the `numZeroBalances` on the account by 1 for each associating token.
+6. save this `MerkleTokenRelStatus` object in prevRel tracker to update the prevkey in the next iteration.
+7. persist all of the newly added `MerkleTokenRelStatus` objects and the original account's `tokenAssociationMetadata` which will have the updated `numAssociations`, `numZeroBalances` and `lastAssociation`.
+```
+
+### Token Dissociation
+When dissociating a token, get the next and prev relationship and update the keys such that they point to each other rather than the dissociating token relationship.
+
+```
+Algorithm:
+1. if the dissociating token relationship is the  account's `lastAssociation` from its `tokenAssociationMetadata`, check if there are any other associations
+    a. if not then update the `lastAssociation` field to the default [0]
+    b. else get the next `MerkleTokenRelStatus` and update the prevKey to the default [0] and set the `lastAssociation` to the next relation's key 
+2. else get the next and prev `MerkleTokenRelStatus` association objects and update the next and prevKeys to point to each other respectively.
+3. if the nextKey in step 2 is default [0] then we are dissocaiting the last tokenAssocation in the list and we dont have to process the next relation and just set the prev relation's next key as default [0]
+4. decrement the  `numAssociations` on the account for each dissociating token.
+5. decrement the  `numZeroBalances` on the account for each dissociating token if the dissociating relation's balance is 0.
+6. persist all of the updated `MerkleTokenRelStatus` objects.
+```
+
+### GetAccountInfo and GetAccountBalance queries
+When fetching all the tokens that are associated to the account, we have to traverse the linkedlist and get the data of each token association. We have to limit the number of tokenIds we return which will be dictated by the dynamic property `tokens.maxPerAccount`.
+
+```
+Algorithm:
+1. get the latest token association from the account's `lastAssociation` from its `tokenAssociationMetadata`
+2. fetch the `MerkleTokenRelStatus` using this key from the tokenAssociations map
+3. add the tokenId to the list for the info
+4. get the nextKey from this `MerkleTokenRelStatus` object and repeat step 2 and 3 until the nextKey equals default [0] which means we reached the end of the linkedList or until we reach the limit specified by `tokens.maxPerAccount`.
+```
+
+### CryptoTransfer
+when transferring any token units [Fungible or Non-Fungible] we have to track the tokenBalances on sender and receiver to update the `numZeroBalances` on each of those accounts respectively.
+
+```
+Algorithm:
+On every `tryAdjustment` and `updateLedgers` calls when adjusting token units
+1. If the sender is left with no more token units, then increment the `numZeroBalances` by 1
+2. If the receiver's initial balance for this token type is 0, then decrement the `numZeroBalances` by 1. 
+```
+
+### TokenWipe
+When a TokenWipe operation is performed on an account, we have to track the tokenBalances on the account for this token and update the `numZeroBalances` accordingly.
+
+```
+Algorithm:
+When wiping either Fungible token units or NFT units on an account
+1. If the remaining units on that token for that account is 0, then increase the `numZeroBalances` by 1. 
+```
+
+### TokenBurn
+When a TokenBurn operation is performed, we burn the asked amount off the treasury account and update the total supply. Update the `numZeroBalances` of the treasury if the burn left the treasury with zero balance on that token.
+
+```
+Algorithm
+When burning either Fungible token units or NFT units of a token
+1. If the treasury has no more units left, then increase the `numZeroBalances` by 1. 
+```
+
+### TokenMint
+When a TokenMint operation is performed, we mint asked amount of token units, add to the treasury account's token balance and update the total supply of that token. Update the `numZeroBalances` of the treasury if the original balance for treasury on that token is 0.
+
+```
+Algorithm
+When minting either Fungible token units or NFT units of a token
+1. If the treasury has 0 units of that token before minting, then decrease the `numZeroBalances` by 1.
+```
+
+### Crypto Account Deletion
+Currently, we return `TRANSACTION_REQUIRES_ZERO_TOKEN_BALANCES` when a `cryptoDelete` transaction is submitted on an account with non-zero balances on the tokens [ not `deleted`] that it is associated with.
+
+```java
+    if (!ledger.allTokenBalancesVanish(id)) {
+        txnCtx.setStatus(TRANSACTION_REQUIRES_ZERO_TOKEN_BALANCES);
+        return;
+    }
+```
+
+Checking if the associated token is not `deleted` and then validating the balance on that association gets very costly when the association limit is removed.
+So we will include the deleted tokens as well when checking if the account has any non-zero token balances and use the fields `numZeroBalances` and `numAssociations` to match if all the associations have zero balances so that we can avoid traversing the list of unlimited token associations.
+
+> An account must dissociate from deleted tokens if it has any token balances left pertaining to that token to be eligible for deletion
+
+### Balance exporter
+Balance exporter would need all the token associations without the limitation enforced by `tokens.maxPerAccount`
+
+### AutoRenew
+AutoRenew fee calculations would require the `numAssociations` from `tokenAssociationMetadata` on the autoRenewing account.
+
+## Backwards Compatibility
+
+When the Account has more than 1000 tokens associated to it, the getAccountInfo and getAccountBalance query will not support fetching all of those associations.
+Rather the number of token associations that will be fetched from the queries will be dictated by the dynamic property `tokens.maxPerAccount`.
+And the token associations returned by these queries will have the 1000 [if `tokens.maxPerAccount = 1000`] **latest** token associations rather than the **first** 1000.
+
+Also Accounts can longer be marked as `deleted` with persisting token units of a `deleted` token.
+
+> Eventual goal is to limit this number significantly and encourage users to query the mirror nodes for this data.
+
+## Security Implications
+CryptoDelete could potentially cause an attack vector for DOS attack if we don't skip the check on the non-zero token balances.
+Traversing a million or more associations to check for deleted tokens and zero balance token associations cannot be avoided if we don't skip this check and remove the token association limit.
+
+## How to Teach This
+
+N/A
+
+## Reference Implementation
+
+## Rejected Ideas
+1. Users can create multiple accounts and hold 1000 tokens in each
+2. Users can create a smart contract and manage around the current 1000 token limit by creating multiple accounts each with 1000 tokens and use smart contract logic to manage the mapping
+3. Users can opt to create tokens on the EVM layer, which is not subjected to the 1000 token limit but this doesn't use Hedera’s native tokenization
+4. Create an `exchange-account` type of account where we charge more to create, but they have a higher association limit
+
+## Open Issues
+[`MapValueLinkedList`](https://github.com/hashgraph/hedera-services/issues/2842)
+[Increase token association limit](https://github.com/hashgraph/roadmap/issues/81)
+[HIP - 1000 : Remove limit on the number of tokens that can be associated to an account](https://github.com/hashgraph/hedera-services/issues/2917)
+[Services-PR](https://github.com/hashgraph/hedera-services/pull/2934)
+## References
+
+Add garbage collection hip here.
+
+## Copyright/license
+
+This document is licensed under the Apache License, Version 2.0 -- see [LICENSE](../LICENSE) or (https://www.apache.org/licenses/LICENSE-2.0)