How do you track progress on CPU/GPU heavy loads?

Question

I&rsquo;ve been researching creative ways to track program progress and give myself a notification when things are done (or about to be done). For example, training a ML model or processing a bunch of photos takes an indeterminate amount of time.I&rsquo;ve seen some interesting ideas around CPU/GPU monitoring/tensor board/checking in with TeamViewer/etc but want to get an idea for the full variety of solutions out there for measuring (i) progress on your own code&rsquo;s execution and (ii) progress on other programs you might want progress on (like an installer or a long render). What works for you?

WJW · Accepted Answer

For things that take a known amount of time and/or provide updates on progress ("X out of Y items processed"), I'll often include a progress bar with some gem or npm module.If you don't know how long something is going to take and it emits no intermediate steps, a progress bar won't be useful since it'll go from zero to 100% in one go. One trick I had success with is running the activity like `./myscript; send_push_message "task complete!". The send_push_message in my case pushes to a custom app I wrote some time ago but it could be as simple as sending an email.