Skip to content

Zendesk Incremental Calls stuck in loop #629

@justin-fundrise

Description

@justin-fundrise

dlt version

1.12.1

Source name

Zendesk

Describe the problem

I am running the dlt verified source for zendesk in Dagster.
I have a dagster asset setup that is calling the zendesk_talk endpoint, specifically looking to load data via the incremental calls

I am running into an issue where the get_url end up getting stuck on the last page and continually requests it.

It seems like there may be a missing condition where the response_json["next_page"] is the same as the currently requested url and it gets stuck in a loop.

Expected behavior

I would expect that there is a condition that will check to ensure the response_json["next_page"] does not equal the last/current page. if it does, it should break the loop.
One potential way to fix could be changing this line to >1 since the last page keeps returning one record, the same record.

Steps to reproduce

Using the zendesk verified source dlt init zendesk snowflake
I created an asset

@dlt_assets(
    dlt_source=zendesk_talk(start_date="2025-05-24T12:00:00Z"),
    dlt_pipeline=dlt.pipeline(
        pipeline_name="zendesk",
        dataset_name="zendesk_dlt",
        destination="snowflake",
        progress="log",
    ),
    name="zendesk",
    group_name="zendesk",
)
def dagster_zendesk_assets(context: AssetExecutionContext, dlt: DagsterDltResource):
    yield from dlt.run(context=context)

When materializing this asset, it continually runs requesting the same url over and over again even though there are no new records(added logging for url)
Next page URL: https://subdomain.zendesk.com/api/v2/channels/voice/stats/incremental/calls.json?start_time=1749493139
Next page URL: https://subdomain.zendesk.com/api/v2/channels/voice/stats/incremental/calls.json?start_time=1750695087
Next page URL: https://subdomain.zendesk.com/api/v2/channels/voice/stats/incremental/calls.json?start_time=1750775021
----------------------------- Extract zendesk_talk -----------------------------
Resources: 0/1 (0.0%) | Time: 1.73s | Rate: 0.00/s
calls_incremental: 2094 | Time: 1.17s | Rate: 1790.63/s
Memory usage: 313.19 MB (77.10%) | CPU usage: 0.00%

Next page URL: https://subdomain.zendesk.com/api/v2/channels/voice/stats/incremental/calls.json?start_time=1750775021
Next page URL: https://subdomain.zendesk.com/api/v2/channels/voice/stats/incremental/calls.json?start_time=1750775021
Next page URL: https://subdomain.zendesk.com/api/v2/channels/voice/stats/incremental/calls.json?start_time=1750775021
Next page URL: https://subdomain.zendesk.com/api/v2/channels/voice/stats/incremental/calls.json?start_time=1750775021
----------------------------- Extract zendesk_talk -----------------------------
Resources: 0/1 (0.0%) | Time: 2.75s | Rate: 0.00/s
calls_incremental: 2098 | Time: 2.19s | Rate: 959.10/s
Memory usage: 313.27 MB (77.20%) | CPU usage: 0.00%

Next page URL: https://subdomain.zendesk.com/api/v2/channels/voice/stats/incremental/calls.json?start_time=1750775021
Next page URL: https://subdomain.zendesk.com/api/v2/channels/voice/stats/incremental/calls.json?start_time=1750775021
Next page URL: https://subdomain.zendesk.com/api/v2/channels/voice/stats/incremental/calls.json?start_time=1750775021
Next page URL: https://subdomain.zendesk.com/api/v2/channels/voice/stats/incremental/calls.json?start_time=1750775021
Next page URL: https://subdomain.zendesk.com/api/v2/channels/voice/stats/incremental/calls.json?start_time=1750775021

How you are using the source?

I run this source in production.

Operating system

macOS

Runtime environment

Local

Python version

3.12.7

dlt destination

snowflake

Additional information

No response

Metadata

Metadata

Assignees

Labels

No labels
No labels

Type

No type

Projects

Status

Planned

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions