Firmware 3.0.4 stops responding over ethernet after 24-48hrs

[dave4444]

Upgraded to 3.0.4, and hydroflash stopped responding to network requests (http, mobileapp, ping, etc...) after running for 24-48 hours.

The LCD on the module was still active and still showed it's IP address etc.. but the device was not responding to http, mobileapp, ping, etc...

No unusual logs were present except a lone 'Failed to locate file on METAR download attempt'. Not sure if its related

[Support]

What was the state of the Amber LED on the WiFi module? On solid? Blinking?
Are you using a Fixed IP or DHCP?
How long was it in this state?
What did you do to restore it?
What version did you upgrade from?
Has this ever happened before?

Thanks,

[dave4444]

Wired version, not wifi. Ethernet link was up.
DHCP for v4, static address and autoconf address for v6 (no DHCPv6).
The LCD was showing correct IPv4 Information, and cyclcing through it's screens.
I noticed it was not responding about 47 hours after firmware upgrade, it was working for at least the first 24 hours.
I power cycled it to recover.
Was running 2.0.2 prior.
Nope, first time. I'm going to keep an eye on it to see if it happens again.

[Support]

We discovered an issue with the Wired Version 3.00.04 that should have had a compile time directive in place that was missing and this could perhaps result in the behavior you saw (depending on network traffic).

Additionally there was a bug fix for both Wired and WiFi versions related to the water EVERYXDAYS scene processing that was fixed.

Please update your device to the latest version by clicking on the 'Update from Cloud Server' button on the Upload Firmware page. Your HydroFLASH device will download the latest version and reboot. The latest Wired Version is 3.00.06. We will post this information to the Firmware Updates topic shortly.

Let us know if the issue is resolved.

Thanks,

[dave4444]

hung again with 3.0.8. this time it was responding to pings, but http and iphone app connection attempts were rejected

[dave4444]

looks like a memory leak when trying to connect to the cloud service. every 5 seconds, mallocstat shows a 80 byte memory increase.

[Support]

Hi,

We are taking a look at repeating this here. Are you consistently seeing this increase every 5 seconds?

The assumption is that in this state your mobile app in not connected to HydroFLASH via cloud mode and that HydroFLASH is polling the cloud for connections. It this the state you are in and seeing the 80 byte increases?

[dave4444]

mobile app is direct connect.
my firewall blocks the connections from hydroflash to AWS so it's getting a TCP RST each time it tries to connect (every 5 seconds)

[DennisD]

With 3.00.07 - System Up Time 7 days 18 hours 46 minutes 5 seconds, I haven't seen the problem, but I using "--reject-with icmp-host-unreachable" instead of sending a TCP RST.

[dave4444]

yes, interesting observation.
TCP RST to the connection request causes the memory leak, but ICMP host-unreach does not. See 'total allocated space' below every 5 seconds.

=========

08:15:32.766188 IP 10.10.24.50.57171 > 54.243.136.9.11010: S 2782601936:2782601936(0) win 8688 <mss 1460,nop,wscale 0,nop,nop,sackOK,nop,nop,timestamp 112410 0>
08:15:32.766254 IP 54.243.136.9.11010 > 10.10.24.50.57171: R 0:0(0) ack 2782601937 win 0

08:15:37.766114 IP 10.10.24.50.64986 > 54.243.136.9.11010: S 2783863907:2783863907(0) win 8688 <mss 1460,nop,wscale 0,nop,nop,sackOK,nop,nop,timestamp 117410 0>
08:15:37.766176 IP 54.243.136.9.11010 > 10.10.24.50.64986: R 0:0(0) ack 2783863908 win 0

08:15:42.766214 IP 10.10.24.50.42377 > 54.243.136.9.11010: S 2785032643:2785032643(0) win 8688 <mss 1460,nop,wscale 0,nop,nop,sackOK,nop,nop,timestamp 122410 0>
08:15:42.766281 IP 54.243.136.9.11010 > 10.10.24.50.42377: R 0:0(0) ack 2785032644 win 0



$ telnet hydroflash
Trying 10.10.24.50...
Connected to hydroflash
Escape character is '^]'.
Welcome NET+OS 7.5.2.2 Command Line Interface
login: root
Password: ********

#> mallocstat
heap info structure:
system space available for heap 5238143
space allocated from system for heap 1584512
number of non-inuse chunks 7
number of mmapped regions 0
total space in mmapped regions 0
total allocated space 1539608
total non-inuse space 44904
top-most, releasable space 17848

#> mallocstat
heap info structure:
system space available for heap 5238143
space allocated from system for heap 1584512
number of non-inuse chunks 7
number of mmapped regions 0
total space in mmapped regions 0
total allocated space 1539688
total non-inuse space 44824
top-most, releasable space 17848

#> mallocstat
heap info structure:
system space available for heap 5238143
space allocated from system for heap 1584512
number of non-inuse chunks 7
number of mmapped regions 0
total space in mmapped regions 0
total allocated space 1539768
total non-inuse space 44744
top-most, releasable space 17848

#>


=========


08:23:27.762718 IP 10.10.24.50.57730 > 54.243.136.9.11010: S 2902767104:2902767104(0) win 8688 <mss 1460,nop,wscale 0,nop,nop,sackOK,nop,nop,timestamp 587410 0>
08:23:27.762789 IP 10.10.24.1 > 10.10.24.50: ICMP host 54.243.136.9 unreachable, length 72

08:23:30.762195 IP 10.10.24.50.57730 > 54.243.136.9.11010: S 2902767104:2902767104(0) win 8688 <mss 1460,nop,wscale 0,nop,nop,sackOK,nop,nop,timestamp 590410 0>
08:23:30.762271 IP 10.10.24.1 > 10.10.24.50: ICMP host 54.243.136.9 unreachable, length 72

08:23:36.762111 IP 10.10.24.50.57730 > 54.243.136.9.11010: S 2902767104:2902767104(0) win 8688 <mss 1460,nop,wscale 0,nop,nop,sackOK,nop,nop,timestamp 596410 0>
08:23:36.762171 IP 10.10.24.1 > 10.10.24.50: ICMP host 54.243.136.9 unreachable, length 72


#> mallocstat
heap info structure:
system space available for heap 5238143
space allocated from system for heap 1600896
number of non-inuse chunks 9
number of mmapped regions 0
total space in mmapped regions 0
total allocated space 1553624
total non-inuse space 47272
top-most, releasable space 15096

#> mallocstat
heap info structure:
system space available for heap 5238143
space allocated from system for heap 1600896
number of non-inuse chunks 9
number of mmapped regions 0
total space in mmapped regions 0
total allocated space 1553624
total non-inuse space 47272
top-most, releasable space 15096

#> mallocstat
heap info structure:
system space available for heap 5238143
space allocated from system for heap 1600896
number of non-inuse chunks 9
number of mmapped regions 0
total space in mmapped regions 0
total allocated space 1553624
total non-inuse space 47272
top-most, releasable space 15096

#>