`WaitJobAsync` returning too much data

[robertmclaws]

Describe the bug

I'm watching the SDK crawl the website in my previous bug report, and due to the size of the website, it's sending an AWFUL lot of data back down. There are 54 pages, and refreshing every second is causing an explosion in memory used.

It feels like there needs to be a backoff algorithm based on the number of items the response says is being crawled, and/or I need to be able to pass in a different wait interval so I can better control the timing.

Steps to reproduce the bug

Use the .NET sample code to crawl https://burnrate.io.
Use Fiddler to watch how many times the Firecrawl API is hit and how much data comes back.

Expected behavior

I would expect to either have a measure of control over the refresh time, and/or have a backoff algorithm based on the number of items we're ultimately waiting for.

Screenshots

No response

NuGet package version

Latest

Additional context

No response

[robertmclaws]

I fixed this myself via an extension method:

        public static async Task<CrawlStatusResponseObj> WaitJobAsync(this CrawlingClient client,
            string jobId,
            int delayMilliseconds = 1000,
            CancellationToken cancellationToken = default)
        {
            int delay = delayMilliseconds;
            while (true)
            {
                cancellationToken.ThrowIfCancellationRequested();

                await Task.Delay(delay, cancellationToken).ConfigureAwait(false);

                var statusResponse = await client.GetCrawlStatusAsync(
                    id: jobId,
                    cancellationToken: cancellationToken).ConfigureAwait(false);
                if (statusResponse.Status is "completed" or "failed")
                {
                    return statusResponse;
                }
                else
                {
                    delay = ((statusResponse.Total ?? 1) - (statusResponse.Completed ?? 1)) * delayMilliseconds;
                }
            }

        }

[HavenDV]

not sure i understand it

Question:

• Why are there memory issues here - these queries don't return anything massive?
• Firecrawl internal logic can be different and results can be obtained in parallel and may not depend on the number of elements

[robertmclaws]

It's bandwidth and serialization. Checking it as frequently as you are means a big crawl can return a LOT of data.