[FEATURE] Support on-demand triggered incremental refresh

[dai-chen]

Is your feature request related to a problem?

When the Flint index is created with auto_refresh=true, it triggers a long-running Spark streaming job that continuously refreshes the index data. While this setup is ideal for continuous data ingestion scenarios, it becomes inefficient for use cases where data ingestion is infrequent, leading to unnecessary resource utilization, operational cost and user expenses.

What solution would you like?

Leveraging the existing AvailableNow trigger in Spark structured streaming, the streaming job can autonomously conclude after processing all currently available new data. Later, users can trigger the job manually to fetch the most recent data when needed.

To implement this approach effectively, the following modifications may be necessary:

1. Ensure the Flint index state transits to 'ACTIVE' to prevent potential misinterpretation as a refresh failure
2. Provide an API, either 'RECOVER INDEX JOB,' or a more user-friendly alternative like [FEATURE] Support incremental refresh in refresh index statement #100, to facilitate user-initiated job restarts

What alternatives have you considered?

Stick with current default trigger and user controls by CANCEL/RECOVER statement in #190.