I am working in WordPress VIP environment. This is an enterprise solution and I've had to make some modifications to the plugin to make it work appropriately in that environment, mainly just rewriting the Filesystem class, it seems to work well when I test locally where I'm mimicking the WP VIP environment, however, I am only running on a single container when I test locally. As soon as I transfer to WP VIP's servers, I've found that the SSE/EventSource stream fails after about 150-200 records, I'm thinking this is as a result of the requests being sent to different containers by the load balancer.
The expected response to that would be to re-create the connection once failure occurs, but it doesn't seem to be doing that. I can't look at the JS as it's already compiled and no sourcemaps were provided, but it seems like you're using jQuery AJAX to execute the "run" endpoint, but I don't see the EventSource event handlers (ping/message/etc) anywhere. I could be missing them though IDK.
More importantly, I need to have this run in a scheduled manner. I have not reviewed the scheduler/cron system you have in place, but it seems to be running into the same issue where it gets stuck after just a couple hundred in production. Again, locally, I've had no issues, so that's really the only thing I can think that is different is that WPVIP is running a kubernetes cluster and I'm just running a single docker container. Any ideas? I'm not really getting much in the way of logging either FYI and WP VIP doesn't give me direct server access so I'll have to try and figure out the best way to get some server side debug info. It would be helpful to know where to start though. Thank you!