Page 1 of 2 12 LastLast
Results 1 to 25 of 47

Thread: Network issues/reboots with AB17 firmware

  1. #1
    Regular Vistor
    Join Date
    Sep 2017
    Location
    Dayton, OH
    Posts
    33

    Network issues/reboots with AB17 firmware

    It seems some people are starting to report network stability and rebooting issues with the new firmware load in another thread:
    https://forum.neptunesystems.com/sho...will-be-ending

    I thought I'd start this thread to consolidate people's issues to provide more visibility.

    For myself, I had problems with wireless a month or so ago, but since hard wiring to my router, it's been quite solid, with no reboots and just an occasional heartbeat miss. Since updating firmware, I've had 2 reboots and have been averaging 15-20 heartbeat misses per day. The only thing that was changed was the new firmware.

    Since I didn't hear others observing this on here, I wrote it off to network issues, but now that I hear some others are experiencing this, I'm not so sure.

    I'll say it again, I have absolutely no idea why network issues should cause a reboot, since the Apex shouldn't even need a network to operate at all, only to monitor and change parameters. It's an issue for me since my return is tied to the variable speed control on the Apex unit, so will not take a fallback value, so it stops altogether, gurgles and then restarts...very irritating.

    If you're having issues, please report this to Neptune so we can get some traction on them addressing this for those who do experience issues.

  2. #2
    NSI Member Krazie4Acans's Avatar
    Join Date
    Nov 2013
    Location
    US, Mountain Time
    Posts
    447
    I reported the reboot issue in the NSI forum, but on my Classic, before this version was released and got no replies before it was archived. My scenario is very similar to yours except that I am using WiFi (which never disconnected or gave heartbeat errors before this update). Now I can actually cause my Apex to reboot by hammering my WiFi with lots of streaming or large downloads.

    My personal read on this is that the new network updates in this firmware are checking for network connectivity (if there is network Link)and if it's not detecting network communication it reboots to try and establish connection again.

  3. #3
    Regular Vistor
    Join Date
    May 2014
    Location
    WI
    Posts
    24
    Same here on classic with the AB17 software.
    Hardwired LAN
    Static IP
    Assigned port 5000
    Has been function with same network hardware for 2 years, updated to AB17 software 3 days ago and disconnects 8-10 times per day. I also have the variable speed pump which cannot fallback.
    Sometimes the Kessil also resets the ramp. Occasionally can catch the whole thing on the camera so would have to say the network in the house is not going down to support a live camera.

  4. #4
    Apex User zombie's Avatar
    Join Date
    Dec 2013
    Location
    Denver, CO
    Posts
    7,067
    This is slightly beside the point of the OP, but regarding your ramp resets, if you use the wizard instead of profiles, you won't have to worry about ramps resetting ever. For the variable speed part, Fallback percentages are now supported because of this firmware update. I dont believe it works for the base variable ports but does work for a VDM. I forgot to test the base ports to see if that functionality was also added.

    You might be an engineer if...You have no life and can prove it mathematically.

  5. #5
    Regular Vistor
    Join Date
    Sep 2017
    Location
    Dayton, OH
    Posts
    33
    Quote Originally Posted by zombie View Post
    This is slightly beside the point of the OP, but regarding your ramp resets, if you use the wizard instead of profiles, you won't have to worry about ramps resetting ever. For the variable speed part, Fallback percentages are now supported because of this firmware update. I dont believe it works for the base variable ports but does work for a VDM. I forgot to test the base ports to see if that functionality was also added.
    Just tested the variable ports on the base with the latest firmware by rebooting the Apex via fusion with a % in the fallback (using the new hardcoded % feature which is nice!). It still resets the return pump OFF unfortunately.

    Support asked me to reset my router and Apex to see if that helps. Will check to see if that helps overnight.

  6. #6
    Master Control Freak RussM's Avatar
    Join Date
    Dec 2012
    Location
    California - US Pacific
    Posts
    15,886
    Quote Originally Posted by rjkrejci View Post
    Just tested the variable ports on the base with the latest firmware by rebooting the Apex via fusion with a % in the fallback (using the new hardcoded % feature which is nice!).
    Quote Originally Posted by zombie View Post
    For the variable speed part, Fallback percentages are now supported because of this firmware update. I dont believe it works for the base variable ports but does work for a VDM. I forgot to test the base ports to see if that functionality was also added.
    Fallback using % is not available for VDM variable outputs, as is listed in the All About FALLBACK sticky and in the table shown in the Programming Enhancements in AOS 5.03 and 4.53 sticky. If you specify a % value - any value 0-100 - for Fallback of a VDM variable output, the result is the same as Fallback ON. Only Fallback ON or Fallback OFF may be used for VDM variable outputs.

    Fallback in any form is not available for base variable outputs (or alarm outputs). Fallback is a special programming command which, if used in an output's program, tells an output in an AquaBus module what to do if that module loses communications with the Apex/Apex Classic/Apex Lite base unit or the Apex Jr. Think about this... the base unit is not an Aquabus module; it cannot "lose communications" with itself; the base unit is either up & running and base variables will be doing what they are programmed to be doing, or the Apex base is powered down or otherwise not functioning, and in that case, neither will be the embedded outputs. Any Fallback statement used in a base variable will have no effect whatsoever.
    I'm not a Neptune support rep. Please do not send me PMs with technical questions or requesting assistance - use the forums for Apex help. PM me ONLY if the matter is of a private or personal nature. Thanks.

  7. #7
    Regular Vistor
    Join Date
    Sep 2017
    Location
    Dayton, OH
    Posts
    33
    Seems like a really bad idea for VDM Fallback to default to ON for ANY % value. So many in the hobby buy over-sized equipment (pumps, lights...) and defaulting to full blast could result in disaster, whereas defaulting to OFF should not since we test our tanks to be in a safe state during a power outage.

    The best thing would be warning the user with a programming error if you try to use a value that's not appropriate for a given input so the behavior equals what the user intended. But If I merrily put a 0% fallback in a VDM variable output and receive no complaints from the Apex interface when saving, I'd certainly be surprised when it goes to 100% on a reboot or other disconnect, potentially overflowing the tank.

  8. #8
    Regular Vistor
    Join Date
    May 2014
    Location
    WI
    Posts
    24
    Quote Originally Posted by zombie View Post
    This is slightly beside the point of the OP, but regarding your ramp resets, if you use the wizard instead of profiles, you won't have to worry about ramps resetting ever. For the variable speed part, Fallback percentages are now supported because of this firmware update. I dont believe it works for the base variable ports but does work for a VDM. I forgot to test the base ports to see if that functionality was also added.

    You might be an engineer if...You have no life and can prove it mathematically.
    Doh....Ive been caught. Was originally set up this way before the wizard and in my"not broke don't fix it" mind haven't changed it.

    Back on point, no reset overnight, started again today @ ~7:30 CDT and reset 5 times in 10 minutes.

  9. #9
    Regular Vistor
    Join Date
    Sep 2017
    Location
    Dayton, OH
    Posts
    33
    To follow up...I reset all my network equipment and the apex and things are back to being fairly stable (only 2 heartbeat misses in 24 hrs and no reboots). While the instability started with loading the new firmware, it appears that maybe it was the act of loading the firmware that caused the instability rather than inherent in the firmware itself. Or I can't discount the possibility of random network issues at home causing the disconnects, although other IOT devices I have had no issues.

    Doesn't explain why network issues should ever cause reboots which I still view as unacceptable. Understand not being able to connect remotely if your network is unstable, but should NEVER cause a reboot since networking isn't a safety critical feature

  10. #10
    Regular Vistor
    Join Date
    May 2014
    Location
    WI
    Posts
    24
    To update from my side....

    Reset all network equipment with no change. At about 7:30 am is when this starts the reboot cycle and usually stops before I get home @ 6pm. If the network cable or router, modem is left off then no reboots.
    Tomorrow I will go direct to modem with Apex reset to port 80 and see what happens.

    Neptune Support indicates it is my port forwarding and network. If this is then some "timing" or packet size changed in the software for the apex when contacting the server.

  11. #11
    Regular Vistor
    Join Date
    Sep 2017
    Location
    Dayton, OH
    Posts
    33
    A few things that help my 2016 Apex stability a little were to turn off DHCP on the Apex and hard-code the local IP address to an address outside of the routers DHCP pool (and hardcode it on the router). Also, setting the DNS servers to google's 8.8.8.8 and 8.8.4.4. Obviously still had some issues, but those things seemed to help.

    The problem is that it's so random and sensitive that you really have no idea if a given incident or change is the cause or remedy or did the network issues spontaneously resolve themselves? I don't do the port forwarding since Fusion allows the remote access without it and could open it for a DOS attack.

    Again, network functions need to be isolated in firmware from critical functions, so the box just acts like the network is unplugged momentarily rather than reboot.

  12. #12
    Regular Vistor
    Join Date
    May 2014
    Location
    WI
    Posts
    24
    Have been running with static IP on the Apex for 2 years. But within the DHCP pool so that would be something I also can try. I have the MAC address reserved to the IP, since I use the classic dashboard occasional remotely is why the port forwarding is in place. I am also lucky (or not lucky) that my ISP provides a static IP to my house.

    Haven't changed the DNS yes, can try that too.

    Quote Originally Posted by rjkrejci View Post
    Again, network functions need to be isolated in firmware from critical functions, so the box just acts like the network is unplugged momentarily rather than reboot.
    What do you mean with the above statement as I don't completely understand. Not only do I get the heartbeat alarms when this occurs also get power failed/restored and the biggest issue is the return pump shutsdown, skimmer floods (fallback doesn't kick in) and pump restarts and ramps to profile speed.

  13. #13
    Regular Vistor
    Join Date
    Sep 2017
    Location
    Dayton, OH
    Posts
    33
    Quote Originally Posted by sltwtrfsh View Post
    What do you mean with the above statement as I don't completely understand. Not only do I get the heartbeat alarms when this occurs also get power failed/restored and the biggest issue is the return pump shutsdown, skimmer floods (fallback doesn't kick in) and pump restarts and ramps to profile speed.
    Sorry, That statement was directed at Neptune...they need to partition the Critical system functions from the networking functions such that networking can't cause a reboot. I work on critical avionics SW and if we coupled the flight controls to the entertainment system we wouldn't be in business and people would die. This situation isn't as dire obviously, but proper modularity and design can easily prevent this.

    Agreed, the heartbeat alarms don't bother me. But the reboots are unacceptable. I've resorted to unplugging the variable speed from my Varios return and just control it at the power outlet for feeding and maintenance. My skimmer is AC and just controlled via power outlet, so fallback of OFF is honored on a reboot, although not necessary anymore because of unplugging the variable speed return

  14. #14
    Apex User zombie's Avatar
    Join Date
    Dec 2013
    Location
    Denver, CO
    Posts
    7,067
    Quote Originally Posted by rjkrejci View Post
    Sorry, That statement was directed at Neptune...they need to partition the Critical system functions from the networking functions such that networking can't cause a reboot. I work on critical avionics SW and if we coupled the flight controls to the entertainment system we wouldn't be in business and people would die. This situation isn't as dire obviously, but proper modularity and design can easily prevent this.

    Agreed, the heartbeat alarms don't bother me. But the reboots are unacceptable. I've resorted to unplugging the variable speed from my Varios return and just control it at the power outlet for feeding and maintenance. My skimmer is AC and just controlled via power outlet, so fallback of OFF is honored on a reboot, although not necessary anymore because of unplugging the variable speed return
    I doubt that they intentionally couple the two. What is far more likely is that some bug that is being triggered by network issues on your system is throwing the processor into some cycle like an infinite for loop and crashing it, which is a situation that a reboot is better than the alternative. They have had issues like this before that have been successfully patched, so I'm sure once they can debug the problem, a fix will be out in no time. It's definately not affecting my system on my classic or 2016 so it's likely uncommon and I would suggest reporting the issue to support to help them track down the cause for those affected.

    You might be an engineer if...You have no life and can prove it mathematically.

  15. #15
    Regular Vistor
    Join Date
    May 2014
    Location
    WI
    Posts
    24
    All



    To let you know the status at this point. Friday night I moved the firewall and router from my home network and went directly to the ISP modem that I reset to factory and rebooted.



    Saturday morning I had no reboots, was unsure if the reboots only occurred during the week as they started Monday morning consistently.



    Then re-installed router that was also factory reset. As I suspect there is no communication with Apex (or anything) in the house as this router will not let any traffic unless specified, even port 80.



    I removed the router and used the ISP modem, engaged the firewall as the NAT table was crazy with the hits from the open LAN. Once the firewall was even on the lowest setting the Apex didn't reboot but would not link with fusion. Long story short and many hours later I caught the IP of the Apex attempting to access 54.208.22.117 port 1199 from my gateway IP though port 54068, then 54541 and a couple other 54xxx ports.



    Once I opened all outbound misc ports (not 443 and controlled ports etc) then it would link to fusion.



    System ran all weekend with no issues so far, then reinstalled port forwarding so I can access classic dashboard externally along with the camera and external ISP modem control. So far system is working and has not lost link to fusion and or re-booted. Will monitor for a couple days, but so far so good and since it has not re-booted as of yet (after the typical time it started) we may be out of the woods.



    Somewhere in the software change the ports or something changed, I have never had these ports open to outbound traffic and without knowing this change suspect is why random people are having the issue. The average person plugs their router in and leaves at factory settings or doesn't have pro- routers which stop all /outbound traffic.



    Ill update in a day or so unless the re-booting starts again.

  16. #16
    Regular Vistor
    Join Date
    May 2014
    Location
    WI
    Posts
    24
    Spoke to soon, re-booting has restarted again.

  17. #17
    Frequent Visitor
    Join Date
    Aug 2015
    Location
    Georgia
    Posts
    79
    So....dont do any more firmware updates..

    ....got it

    Sent from my SM-G950U1 using Tapatalk

  18. #18
    Regular Vistor inh's Avatar
    Join Date
    Sep 2016
    Location
    Arizona
    Posts
    24
    Mine has been rebooting too on the latest firmware, and I leave town in two days


    Sent from my iPhone using Tapatalk

  19. #19
    Frequent Visitor
    Join Date
    Aug 2015
    Location
    Georgia
    Posts
    79
    Does the uploader allow you to load an older firmware version?

    I have never tried it.

    Sent from my SM-G950U1 using Tapatalk

  20. #20
    Apex User zombie's Avatar
    Join Date
    Dec 2013
    Location
    Denver, CO
    Posts
    7,067
    Quote Originally Posted by Allentown View Post
    Does the uploader allow you to load an older firmware version?

    I have never tried it.

    Sent from my SM-G950U1 using Tapatalk
    Not on the 2016. Only apex classic IF you saved an earlier version.

    You might be an engineer if...You have no life and can prove it mathematically.

  21. #21
    Frequent Visitor
    Join Date
    Feb 2016
    Location
    Toronto
    Posts
    54
    Quote Originally Posted by zombie View Post
    Not on the 2016. Only apex classic IF you saved an earlier version.

    You might be an engineer if...You have no life and can prove it mathematically.
    If you have a classic the old firmware is still available in the classic area of this fourm.

  22. #22
    Regular Vistor
    Join Date
    May 2014
    Location
    WI
    Posts
    24
    Quote Originally Posted by inh View Post
    Mine has been rebooting too on the latest firmware, and I leave town in two days


    Sent from my iPhone using Tapatalk
    Same here, leaving for vacation in 4 days.

    Update: Currently Apex is wired directly to the ISP modem (also reset to factory) with no firewall active, Apex is reset to port 80 and DHCP is on.

    Will update when I get home as I cannot access syslog (or if skimmer is full) to see if any restarts/re-boots occurred without Fusion sending warnings.

  23. #23
    Regular Vistor inh's Avatar
    Join Date
    Sep 2016
    Location
    Arizona
    Posts
    24
    Quote Originally Posted by sltwtrfsh View Post
    Same here, leaving for vacation in 4 days.

    Update: Currently Apex is wired directly to the ISP modem (also reset to factory) with no firewall active, Apex is reset to port 80 and DHCP is on.

    Will update when I get home as I cannot access syslog (or if skimmer is full) to see if any restarts/re-boots occurred without Fusion sending warnings.

    Do you have heartbeat enabled?

    I was thinking about disabling this. I did notice as soon as my apex rebooted last night my ISP also dropped my internet connection. There seems to be a direct coloration when my ISP drops sync, and my Apex reboots. Heartbeat is the only think I could think of programmatically that would cause that.

  24. #24
    Regular Vistor
    Join Date
    May 2014
    Location
    WI
    Posts
    24
    Yes I do have heartbeat enabled.

  25. #25
    Regular Vistor inh's Avatar
    Join Date
    Sep 2016
    Location
    Arizona
    Posts
    24
    Have you disabled it at all in your trouble shooting?


    I just disabled it. Its been rebooting every night. Hopefully will see if it happens tonight. I also opened a case.

Bookmarks

Bookmarks

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •