get_spark_job_list() from azure.synapse.monitoring.operations does not return full livy request set

[bpkroth]

• Package Name: azure.synapse.monitoring.operations
• Package Version: 0.2.0
• Operating System: Linux
• Python Version: 3.8

Describe the bug
When calling get_spark_job_list() a limited number of requests are returned (e.g. > 100)

To Reproduce
Steps to reproduce the behavior:

1. Create a number of Livy requests (e.g. > 100)
2. Call the get_spark_job_list() API
3. Note how a reduced set is returned.

Expected behavior
The API should provide a way to paginate and skip around in the the list of Spark jobs.
For instance:
https://github.com/Azure/azure-sdk-for-python/blob/main/sdk/synapse/azure-synapse-spark/azure/synapse/spark/operations/_spark_batch_operations.py#L46

    def get_spark_batch_jobs(
        self,
        from_parameter=None,  # type: Optional[int]
        size=None,  # type: Optional[int]
        detailed=None,  # type: Optional[bool]
        **kwargs  # type: Any
    ):
...

Uses the from_parameter and size to skip around in the full set of results returned. Unfortunately, it does not contain livy_id details in it's results returned (which I need for my application).

vs.

https://github.com/Azure/azure-sdk-for-python/blob/main/sdk/synapse/azure-synapse-monitoring/azure/synapse/monitoring/operations/_monitoring_operations.py#L46

    def get_spark_job_list(
        self,
        x_ms_client_request_id=None,  # type: Optional[str]
        **kwargs  # type: Any
    ):
...

Which does not allow appending the skip=N parameter to the query string generated (identified from tracing network requests generated by the web version of the workspace's monitoring view).

[ghost]

Thanks for the feedback! We are routing this to the appropriate team for follow-up. cc @aim-for-better, @idear1203.

Issue Details

---

• Package Name: azure.synapse.monitoring.operations
• Package Version: 0.2.0
• Operating System: Linux
• Python Version: 3.8

Describe the bug
When calling get_spark_job_list() a limited number of requests are returned (e.g. > 100)

To Reproduce
Steps to reproduce the behavior:

1. Create a number of Livy requests (e.g. > 100)
2. Call the get_spark_job_list() API
3. Note how a reduced set is returned.

Expected behavior
The API should provide a way to paginate and skip around in the the list of Spark jobs.
For instance:
https://github.com/Azure/azure-sdk-for-python/blob/main/sdk/synapse/azure-synapse-spark/azure/synapse/spark/operations/_spark_batch_operations.py#L46

    def get_spark_batch_jobs(
        self,
        from_parameter=None,  # type: Optional[int]
        size=None,  # type: Optional[int]
        detailed=None,  # type: Optional[bool]
        **kwargs  # type: Any
    ):
...

Uses the from_parameter and size to skip around in the full set of results returned. Unfortunately, it does not contain livy_id details in it's results returned (which I need for my application).

vs.

https://github.com/Azure/azure-sdk-for-python/blob/main/sdk/synapse/azure-synapse-monitoring/azure/synapse/monitoring/operations/_monitoring_operations.py#L46

    def get_spark_job_list(
        self,
        x_ms_client_request_id=None,  # type: Optional[str]
        **kwargs  # type: Any
    ):
...

Which does not allow appending the skip=N parameter to the query string generated (identified from tracing network requests generated by the web version of the workspace's monitoring view).

Author:	bpkroth
Assignees:	idear1203
Labels:	Client, Service Attention, Synapse, feature-request
Milestone:	-

[xiangyan99]

@bpkroth Thanks for your feedback. Unfortunately, /monitoring/workloadTypes/spark/Applications does not support "skip" parameter so I am afraid adding this parameter in the client side SDK will not help.

@idear1203 Please take a look if we want to have this support in restful API.

[bpkroth]

@bpkroth Thanks for your feedback. Unfortunately, /monitoring/workloadTypes/spark/Applications does not support "skip" parameter so I am afraid adding this parameter in the client side SDK will not help.

@idear1203 Please take a look if we want to have this support in restful API.

I've tested this, and it absolutely returns more data when I do this to assemble multiple pages instead of just a single get_spark_job_list() call:

(snippet from my wrapper library)

    def get_spark_jobs(self) -> List[SparkJob]:
        """
        Gets the set of Spark Jobs (e.g. from the Monitoring pane of the Workspace).
        Note: this wrapper handles things like paging, but not filtering (yet).
        """
        # FIXME: The output of this gets limited/paged if there are many many jobs.
        # See Also: <https://github.com/Azure/azure-sdk-for-python/issues/19855>
        MAX_SPARK_JOBS = 10000
        SPARK_JOBS_STEP = 100 # FIXME: This is not currently configurable as far as I know
        spark_jobs: Dict[int, SparkJob] = dict()
        for i in range(0, MAX_SPARK_JOBS, SPARK_JOBS_STEP):
            spark_jobs_subset = self.monitoring_client.monitoring.get_spark_job_list(skip=i).spark_jobs
            added_jobs = 0
            if spark_jobs_subset != None and len(spark_jobs_subset) > 0:
                # Handle the case that the local module doesn't have our unsupported skip API workaround.
                for spark_job in spark_jobs_subset:
                    livy_id = int(spark_job.livy_id)
                    if livy_id not in spark_jobs:
                        spark_jobs[livy_id] = spark_job
                        added_jobs += 1
            if added_jobs == 0:
                break
        return list(spark_jobs.values())

As I said above, I pulled that out of watching the HTTP network requests and their query strings my browser was generating when I click around in the Monitoring pane of the spark workspace's web interface.

I don't know if there's a size parameter for the REST API though. It seemed to be defaulting to 100 in the network traces I looked at.

[xiangyan99]

This is interesting.

@idear1203 could you check if this is by design? (the un-documented parameter)

[idear1203]

@bpkroth thanks for reporting. The SDK is generated based on Swagger spec here. We cannot fix the Python SDK code directly because the code change will lose during the next SDK generation. Instead, we will fix the issue on Swagger spec. We will try fixing the issue in the September release.

cc @wonner

[github-actions]

Hi @bpkroth, we deeply appreciate your input into this project. Regrettably, this issue has remained inactive for over 2 years, leading us to the decision to close it. We've implemented this policy to maintain the relevance of our issue queue and facilitate easier navigation for new contributors. If you still believe this topic requires attention, please feel free to create a new issue, referencing this one. Thank you for your understanding and ongoing support.