About three weeks ago we had our research system vendor deploy a patch that for all purposes looked to work perfectly. We should have known better.
Historically patch deployment to the server has always been hit and miss, sometimes we are lucky and a patch has no flow-on effects. Then, other times, despite our best efforts at testing a patch it will only reveal its hidden gems after deployed to production. Sometimes his has been linked to slight differences between the PROD and QA environments, but other times it has been linked to the patch affecting areas that were not associated to begin with.
Our most recent patch involved a patch to our batch communications functionality, we use this to generate documents that are then inserted into a record’s attachments. The latest patch was relatively minor, just correcting functionality that prevented our showing effort to 3 decimal places. The patch worked as expected, first time too, thus we happily requested it deployed. Oh how I wish we hadn’t as this had a flow-on effect to our schedule alerts functionality, these simply stopped running the next day.
Since then we’ve had our vendor and IT Service department working to try resolve the issue, they’ve argued between one another as to the cause, but still we’ve not resolved it. Thus, I am forced daily to run reports manually, sometimes taking up to several hours depending on the number to be run.
The start of July kicked in and I opened Outlook, checked the folder to which we have a rule redirect the scheduled alerts reports, my heart skipped a beat, the reports had run on the 1st. I did notice though that only 1 report had run on the 2nd which was unusual. It was now that my suggestion to our vendor for a daily digest of reports statuses would have been oh so useful, I had suggested an enhancement around 6 months ago whereby a daily digest would show the list of reports run, the number of results returned and whether a report had been generated; currently you are left guessing whether the report ran and whether there were any results.
As I later discovered my glee would be short lived, indeed the alerts had run, though their execution ceased at 10pm on 1 July as reports scheduled for 11pm onwards did not execute. My supervisor did try to console me by letting me know when he’d had to look after this task one time he’d had to manually run the reports for 8 months before the vendor and IT Services managed to resolve the issue, I didn’t feel terribly consoled.