-
Notifications
You must be signed in to change notification settings - Fork 68
Description
dlt version
1.12.1
Source name
Zendesk
Describe the problem
I am running the dlt verified source for zendesk in Dagster.
I have a dagster asset setup that is calling the zendesk_talk endpoint, specifically looking to load data via the incremental calls
I am running into an issue where the get_url end up getting stuck on the last page and continually requests it.
It seems like there may be a missing condition where the response_json["next_page"]
is the same as the currently requested url and it gets stuck in a loop.
Expected behavior
I would expect that there is a condition that will check to ensure the response_json["next_page"] does not equal the last/current page. if it does, it should break the loop.
One potential way to fix could be changing this line to >1
since the last page keeps returning one record, the same record.
Steps to reproduce
Using the zendesk verified source dlt init zendesk snowflake
I created an asset
@dlt_assets(
dlt_source=zendesk_talk(start_date="2025-05-24T12:00:00Z"),
dlt_pipeline=dlt.pipeline(
pipeline_name="zendesk",
dataset_name="zendesk_dlt",
destination="snowflake",
progress="log",
),
name="zendesk",
group_name="zendesk",
)
def dagster_zendesk_assets(context: AssetExecutionContext, dlt: DagsterDltResource):
yield from dlt.run(context=context)
When materializing this asset, it continually runs requesting the same url over and over again even though there are no new records(added logging for url)
Next page URL: https://subdomain.zendesk.com/api/v2/channels/voice/stats/incremental/calls.json?start_time=1749493139
Next page URL: https://subdomain.zendesk.com/api/v2/channels/voice/stats/incremental/calls.json?start_time=1750695087
Next page URL: https://subdomain.zendesk.com/api/v2/channels/voice/stats/incremental/calls.json?start_time=1750775021
----------------------------- Extract zendesk_talk -----------------------------
Resources: 0/1 (0.0%) | Time: 1.73s | Rate: 0.00/s
calls_incremental: 2094 | Time: 1.17s | Rate: 1790.63/s
Memory usage: 313.19 MB (77.10%) | CPU usage: 0.00%
Next page URL: https://subdomain.zendesk.com/api/v2/channels/voice/stats/incremental/calls.json?start_time=1750775021
Next page URL: https://subdomain.zendesk.com/api/v2/channels/voice/stats/incremental/calls.json?start_time=1750775021
Next page URL: https://subdomain.zendesk.com/api/v2/channels/voice/stats/incremental/calls.json?start_time=1750775021
Next page URL: https://subdomain.zendesk.com/api/v2/channels/voice/stats/incremental/calls.json?start_time=1750775021
----------------------------- Extract zendesk_talk -----------------------------
Resources: 0/1 (0.0%) | Time: 2.75s | Rate: 0.00/s
calls_incremental: 2098 | Time: 2.19s | Rate: 959.10/s
Memory usage: 313.27 MB (77.20%) | CPU usage: 0.00%
Next page URL: https://subdomain.zendesk.com/api/v2/channels/voice/stats/incremental/calls.json?start_time=1750775021
Next page URL: https://subdomain.zendesk.com/api/v2/channels/voice/stats/incremental/calls.json?start_time=1750775021
Next page URL: https://subdomain.zendesk.com/api/v2/channels/voice/stats/incremental/calls.json?start_time=1750775021
Next page URL: https://subdomain.zendesk.com/api/v2/channels/voice/stats/incremental/calls.json?start_time=1750775021
Next page URL: https://subdomain.zendesk.com/api/v2/channels/voice/stats/incremental/calls.json?start_time=1750775021
How you are using the source?
I run this source in production.
Operating system
macOS
Runtime environment
Local
Python version
3.12.7
dlt destination
snowflake
Additional information
No response
Metadata
Metadata
Assignees
Labels
Type
Projects
Status