Display cell run history

[dmadisetti]

Description

I'm working with a large dataset and occasionally running into OOM errors. When the process is killed, there no way on the frontend to determine the last run cell (or cell that was running before OOM)

Suggested solution

• Clicking the broken link scrolls to the last cell
• Some styling on the broken cell (just the red error might be fine)

Alternative

No response

Additional context

Happy to put this in myself, just leaving a bookmark. Small QOL improvement

[mscolnick]

I've wanted to add another helper panel that just lists out a history of all the cell executions and their times and status (success/failure) in a list. That might help solve this as well?

[dmadisetti]

That absolutely works. LMK if there's an existing issue so I can merge it

[mscolnick]

I haven't seen one, but this relates: #974.

I am unsure if its a stack/history of runs or a flame graph (or both).

[Light2Dark]

hey, happy to take a stab at this. It's kinda complex imo, so understand if you'd like to take it up (& I have no guarantees on my progress haha)

I drew 2 designs but I really like a log-style approach (top one), since it can be quite extensible (imagine expanding each cell run to view attributes in json/yaml format or adding a timeline view like I drew out).

One question I've been struggling with is how to determine the cells that are part of a unique run. Cmiiw, but the frontend currently doesn't know. So maybe the frontend would have some clever logic to aggregate cells that have run, or we could add something like a 'run_id' to the notebook data.

oh, and where the cell run history data should live, but I think ephemeral state is fine for now.

[mscolnick]

@Light2Dark i agree the top one is better. i could imagine each run being closed by default (except maybe the top one) and you can expand to see the cell that were run, times, and result (just like you have designed).

for the chart, I am not sure what libraries out there exist/would be good for this, but we do already use vega-lite internally. you could maybe create a chart using the vega spec. here is a sample chart
the keywords to look are likely: gantt chart or timeline chart.

in future, we could add zooming on the chart and panning to filter the list of cells.

[Light2Dark]

The lib is good! Although the chart has to be built separately and then placed next to the trace rows (so alignment is fiddly). Can't really build it row by row like a react component afaik.

[mscolnick]

wow that looks great! i wonder if its would be better to stack them. it wouldn't be aligned then, but you would get:

• you can pan and zoom the full graph to filter the individual cells below it.
• scales way better, e.g. past 20 cells
• more horizontal realstate for the chart itself. (we are rich on vertical realestate

You can also add higlight-on-hover synced when hovered over a bar or the cell summary

[Light2Dark]

hmm, do you have any sample / inspo on the stacked view? Is that a timeline view?

another thing, I'm fixing the chart width and the library is smart enough to readjust each bar's width proportionally. We could put the chart underneath the trace rows to save horizontal space (and maybe make this option a config).

higlight-on-hover synced

yeah that's ideal, that will require some deep integration with vega-lite that I'm not sure is possible. But I'll explore it.

[mscolnick]

I was thinking the browser network tab is good inspo, which does both above and to the right. The above allows for pan/zoom. But i find the right-side is easier to parse (as you've done)

So maybe we do both, eventually, but start with one for now.