DOS Stopped Dosing Overnight, No Clear Reason Why?

[jdhoover]

tl;dr:
Something happened to my Apex Controller at 1:03:56 am last night, and one byproduct of this is that my alkalinity dosing didn't resume after the event. How do I confirm what happened at 1:03:56, and why didn't alkalinity dosing continue after the event?

Background
My DOS unit doses Alkalinity out of the first pump, and Calcium out of the second. It has been working flawlessly for months. This morning I noticed the pH was a little lower than usual. Overnight, the DOS stopped dosing alkalinity about mid-way through the dosing profile, and I don't know why. I can still dose manually, and I have FeedC set to dose a little extra when I want to - and it works fine, too. I have not changed my dosing profiles, there was no power outage (I'm also on battery backup). I did upgrade the AOS to 5.02_5A17 about a week ago - but it has been working fine since then. The SWRev is version 7 (shows OK in the module setup, so assume no firmware for the DOS is needed).

Here's the DOS Config for Pump 1 (Alkalinity)
Set OFFIf FeedC 000 Then Alk3mL_1
If Time 18:00 to 08:35 Then AlkDose
If pH > 8.35 Then OFF

Here's the dosing profile:
ID: 3
Name: AlkDose
Type: Dose
Dose Rate: 7 mL/min
Pump Direction: Forward
Per Dose Amount: 1.6 mL
Dose Intervals: sec
Dose Count: 33

The only thing that's odd is that 2 minutes before it was supposed to give its next dose of Alkalinity, all of my systems seem to show up in the outlet log with the same timestamp. See the log here, where all of my outlets seem to show up in the log starting at line 6 (all with the same timestamp). This seems to have set AlkDose again, so even if the unit rebooted due to an error, I would expect it to continue to deliver Alkalinity solution since the time is still within range of the configuration (ie between 1800 and 0835).

DOS Pump Failure - Outlet Log.PNG

[jdhoover]

I also forget where I grab syslog and status from via the web UI or telnet, which might tell me what happened at that time?

[jdhoover]

Found the syslog, and it doesn't show any events last night - just the restarts back 12 days ago from the firmware update.

05/14/2017 14:02:00 initfs: Push secure stream closed
05/14/2017 14:02:39 initfs: Push secure stream opened
05/19/2017 14:41:32 initfs: System Restart-0_SS04G,3,5344,2
05/19/2017 14:43:09 initfs: Push secure stream opened
05/19/2017 14:45:10 initfs: System Restart-0_SS04G,3,5344,2
05/19/2017 14:45:43 initfs: Push secure stream opened
05/25/2017 01:43:51 initfs: Push secure stream closed
05/25/2017 01:52:33 initfs: Push secure stream opened
05/28/2017 18:45:13 initfs: Push secure stream closed
05/28/2017 18:45:53 initfs: Push secure stream opened

[jdhoover]

I compared this event to the system restart events from my firmware upgrade (even though this isn't looking like a system restart last night). What's interesting is that the restarts were during my calcium dosing schedule. It started calcium dosing back up again successfully, though. From the log in my original post, it looks like it starts up AlkDose correctly as well, but I wonder if it didn't dispense anything because my alkalinity config passes over a day boundary. That might be a bug I'm guessing, if that's the case. The only other thing that looks different from the logs is the setting of SndAlarm_I6 to OFF last night as the first event. I don't see this event during the last system restarts, and I don't actually have that configured - it's simply Set OFF in the configuration.

So, yeah, not sure what happened overnight, and also wonder if there's a bug with the way dosing is handled after a recovery from an event, when the profile config spans a day boundary.