Learn how to Go Out of Enterprise
One of many quickest methods to exit of enterprise is to ship product that requires a technician to return to the house and exchange it – aka a “Truck Roll”. One in every of these notorious failures is the Wi-Fi door lock maker Lockstate. In 2017 Lockstate put out a firmware replace that prompted a proportion of their locks to “brick” and require the lock be eliminated and returned to the producer. Lockstate didn’t exit of enterprise, however they did should rebrand as RemoteLock on account of their repute harm. Luckily for them solely a small proportion of the locks bricked so the corporate did survive, barely. The small proportion of failures means the replace had handed regular High quality Assurance (QA) testing. There may be by no means sufficient testing!
Each IoT machine has bugs. In the present day’s units have many 1000’s of traces of code and too many {hardware} options that means there are many bugs hiding in each machine. Bugs are in your product. I do know it, and you recognize it. I suggest that one of the best resolution for these bugs is to “harden” your firmware to make it extra resilient and carry on truckin’ if one thing unhealthy occurs. Beneath are my Ideas and Methods for hardening Z-Wave firmware to outlive a failure. These concepts are for Silicon Labs SDK however comparable methods apply to Trident IoT with clearly some totally different particulars.
I’m solely discussing resilient firmware and never speaking about making a product safer which can be referred to as “hardening”. Safety is a vital matter which should be deliberate earlier than coding begins. Essentially the most primary safety measure is to set the Debug Lock on all manufacturing items. Debug Lock disables the debugger port of the chip making it troublesome to learn out the flash picture. There are a number of extra layers of safety within the Z-Wave chips that may be enabled however at a naked minimal make sure to set Debug Lock.
Seven Tricks to Harden Z-Wave Firmware
- Assume All the pieces is Damaged
- Use FOR As an alternative of WHILE
- Substitute Default_Handler
- Allow the Different Watchdog
- Reboot if no Comms in a Day
- Allow Stack Overflow Checking in FreeRTOS
- Run Static Evaluation Instruments
1. All the pieces is Damaged
This can be a philosophical concept it’s worthwhile to preserve behind your thoughts with each line of code you write. Assume the whole lot is damaged on a regular basis – {hardware} by no means goes “prepared”, a queue is all the time full, a mutex by no means switches, an not possible state happens and comparable types of failures. The commonest code approach is to all the time examine for error circumstances of any operate that returns a worth. At all times examine inputs for validity.
Essentially the most insidious failures are stack overflows. This could occur the place components of RAM are overwritten and I’ve discovered it superb that the code can preserve operating even after trashing probably lots of of reminiscence areas. The problem is there isn’t any solution to predict what would possibly occur. All types of issues that “can by no means occur” completely will occur when the stack overflows. Because of the restricted RAM on Z-Wave chips, it’s straightforward to overflow the stack.
One other widespread not possible situation is when the ability provide sags simply sufficient to flip bits in methods which might be technically not possible. Robust magnetic fields from close by motors and even cosmic radiation can flip bits in not possible methods. There may be really nothing that “can’t occur”. Thus, all the time code with the thought that the not possible can occur as a result of finally it will.
Wi-fi IoT units utilizing Z-Wave are sometimes wired on to mains-power. They can’t be simply rebooted such as you do along with your laptop when it freezes. If a tool bricks, it’s useless probably for months and even years earlier than an influence failure brings it again on-line. As soon as a tool is wired in, it’s often there for a few years and with Z-Wave it may be many years. Guaranteeing the firmware is resilient when (not if!) the not possible happens will preserve prospects completely happy since they by no means knew the machine rebooted – it saved on truckin’.
2. Use FOR as a substitute of WHILE
The Silicon Labs SDK together with the bootloader has many whereas(hardware_busy) loops that may wait ceaselessly and might trigger the machine to brick. For instance, in Silicon Labs if you happen to allow the LFXO (32KHz crystal oscillator) however don’t have the crystal wired up, the startup code waits for the LFXO to be “prepared” with some time loop. Whereas that is apparent when debugging firmware this may trigger a tool within the area to brick if for some cause the crystal stops working. The straightforward resolution to that is so as to add a FOR loop with a timeout enabling the code to proceed.
Instance in em_cmu.c:
Substitute: whereas ((LFXO->STATUS & _LFXO_STATUS_ENS_MASK) != 0U) { }
With: for (int i=0; (iSTATUS & _LFXO_STATUS_ENS_MASK) != 0U); i++) {__NOP()}
Word the __NOP() is critical to forestall the compiler from optimizing the loop and eradicating it. The timeout worth (1000 on this case) should be chosen primarily based on testing. I often set it to 10X the standard worth. Following the FOR needs to be an assert to examine that the timeout didn’t happen. Word the usage of
3. Default_Handler
Segger has an awesome article on debugging the numerous “fault handlers” within the Cortex-M processors. The article gives code for a lot of totally different handlers to assist debug the fault and make the code extra resilient.
Default_Handler is in startup_
At a minimal, put the Segger beneficial code in for at the very least a few of the exception handlers to make debug simpler. Usually, it’s a good suggestion to gentle an LED (ideally pink) or another exterior indicator to assist throughout debug. Different concepts are to log the tackle and situation that prompted the exception and retailer it within the Consumer Knowledge Web page/NVM that may be learn out from manufacturing items that had been returned from the sphere by indignant prospects. Then carry out forensic evaluation to establish the trigger and launch a firmware replace that solves the issue.
4. Watchdogs
Watchdog timers are essential for dependable 24x7x365 operation of an IoT machine. A watchdog timer is a timer that slowly counts down. From time to time, the firmware “feeds” the watchdog by resetting the counter to a excessive worth. If the counter reaches zero, a full reset of the chip is triggered which reboots the chip and hopefully resolves the error situation. The watchdog timer usually takes a few seconds of being starved earlier than the reset to make sure it doesn’t falsely reset. The trick to a resilient watchdog is deciding when to feed it, and extra importantly, when to not. I wrote a blog post on watchdog timer greatest practices again within the 500 sequence days which nonetheless applies.
There could also be alternatives to additional align the present 700/800 sequence watchdog timer implementation within the Z-Wave SDK with established greatest practices. The watchdog is fed each time the FreeRTOS idle activity is executed. The one two methods the watchdog will fireplace is that if FreeRTOS crashes and stops servicing the idle activity or if a activity sits in a good loop lengthy sufficient. There are numerous different failure mechanisms that may happen that brick the machine however proceed to execute the idle activity. The commonest failure is a queue is full and the code skips over writing the queue and waits till later however the activity draining the queue has crashed. Even when the watchdog is rebooting the chip, if the failure isn’t resolved by a reboot, then the machine continues to be bricked because it’s caught in a reset loop.
The idle activity is often compiled right into a library which makes altering it not possible. Trident has your entire supply code for the SDK out there, making it attainable to enhance the code to comply with one of the best practices. The primary effort in bettering the watchdog is figuring out the whole lot that may trigger the code to lock up. Verify that every one queues, mutexes, state machines and even perhaps peripherals are idle earlier than feeding the watchdog. The opposite secret’s to disable the watchdog throughout growth. Watchdog resets are notoriously good at hiding faults because the chip appears to briefly cease however you ship the command once more the whole lot is working once more as a result of the chip rebooted and is hiding the bug! The Z-Wave chips have a second watchdog timer so you possibly can create your personal strong watchdog following one of the best practices.
5. Reboot When no Communication for a Day
Hi there, is anybody listening? The idea right here is principally a long-duration watchdog timer. If the controller hasn’t despatched a body and/or hasn’t acknowledged the receipt of a body in twenty-four hours, possibly a reboot will clear issues up. I’ve seen this within the 500 sequence the place on uncommon events studying the HomeID from the exterior NVM would fail. Consequently, the machine would neglect the HomeID and assume some random quantity. This random quantity would then be caught within the machine for days or even weeks and even years till the machine rebooted for some cause. This was a basic Not possible Situation that appeared to occur on a reasonably common foundation when a number of tens of 1000’s of Z-Wave units have been working for a couple of months. The one resolution for the end-customer was to tear the bricked machine out of the wall! Or extra generally manufacturing unit reset it and rejoin the community which additionally resulted in one-star opinions.
The answer is straightforward, setup a 24-hour software program timer and examine the RX/TX statistics and if nothing has made it via, reboot! This fixes uncommon not possible circumstances in a manner that almost all prospects wouldn’t even discover. This examine is just wanted for always-on or FLiRs (LSEN) units as deep sleeping units reboot each time they get up. Beneath is an implementation you possibly can drop into your utility.
| / Often that is in app.c or your personal utility information
#embrace #outline ONCE_PER_DAY (1000*60*60*24UL) static SSwTimer CheckTxStatsTimer; void CheckTxStatsCallback(SSwTimer *pTimer){ pStats = ZAF_getNetworkStatistics(); if ((pStats->tx_frames == 0) && (pStats->rx_frames == 0)) { whereas(true); // reboot if no frames have been transmitted or obtained within the final 24hrs } zpal_radio_clear_network_stats(); // zero the stats. } // in ApplicationTask simply earlier than the FOR loop AppTimerInit(EAPPLICATIONEVENT_TIMER, xTaskGetCurrentTaskHandle()); AppTimerRegister(&CheckTxStatsTimer, true, CheckTxStatsCallback); // auto reloads TimerStart(&CheckTxStatsTimer,ONCE_PER_DAY); |
6. Stack Overflow Checking
An actual-time-operating-system provides complexity, however FreeRTOS has a function to examine for a stack overflow which is enabled by default. The variable configCHECK_FOR_STACK_OVERFLOW is about to 2 by default in FreeRTOSConfig.h. This permits some checking and fills the stack area with 0xA5s which is then checked with every activity change. Inspecting RAM after operating the code for a while within the debugger can present insights as to how near overflowing the stack has occurred to this point. The examine calls vApplicationStackOverflowHook if there’s a failure however there may be solely an assert within the weak operate. My advice is so as to add a breakpoint right here throughout testing and think about rebooting within the launched code. Stack overflow checking is just beneficial throughout growth and testing because of the extra overhead.
The insidious downside with stack overflows is that they usually require a number of issues to go fallacious on the similar time – a activity change, an interrupt, the radio sending or receiving information and having to discover a new mesh route, code allocating sizable short-term buffers and possibly much more code that makes use of up the restricted stack area. As talked about above, I’ve noticed the stack overflowing, trashing many dozens of reminiscence areas and the code retains operating however finally there may be an not possible situation or extra usually a hardfault exception. Consequently, the failure is commonly neglected because it solely occurred that “one time” however in actuality, it occurs rather a lot. Getting the failure to occur repeatedly in a managed surroundings is commonly very troublesome. I’ve had dozens of items arrange testing a particular failure case which might take all weekend to lastly set off. Then not having sufficient information on the unit that failed makes it much more exasperating.
7. Static Code Evaluation
Use Claude, CodeX or different static code analysis instruments to assessment all firmware. AI continues to enhance at an exponential fee to grade code high quality. Typically AI can suggest modifications to repair the code, however I might fastidiously examine over the recommendations. AI can hallucinate or just begin making issues up out of nowhere. The GCC compiler has a -fanalyzer possibility that may discover a couple of fascinating issues. Coverity is an trade chief on this area however is expensive. What instruments have you ever used?
Assets
I’m not the one one to make the siren name to make your code resilient. Jack Ganssle wrote the Embedded Muse publication for 27 years on embedded programming. He includes a failure each difficulty with loads of fascinating tales and plenty of tips about resilient embedded C coding. Micheal Barrs’ C Coding Standard needs to be on each firmware engineer’s bookshelf or bookmarked. The e book is crammed with sensible recommendation on coding for resilience and maintainability. Possibly I ought to write a e book on embedded coding primarily based on this Journey?
Subsequent Steps
Half 6 of the Z-Wave Developer’s Journey discusses {hardware} greatest practices. I current my suggestions and methods for making low-cost, straightforward to debug and manufacture Z-Wave merchandise from my 25 plus years of Z-Wave expertise. As we proceed alongside the Z-Wave Developer’s Journey, I welcome your feedback and questions. Please be happy to succeed in out to me instantly through e mail.
Concerning the Writer
Eric Ryherd has been on the forefront of Z-Wave innovation since 2003, starting as a marketing consultant and later serving as a Area Utility Engineer at Silicon Labs. Over the course of his profession, he has contributed to the design and growth of a variety of Z-Wave merchandise, together with sensors, distant controls, motorized window shades, and in-wall dimmers, a lot of that are available on the market right this moment.
Though he “retired” in 2022, Eric stays deeply engaged in embedded programs and Z-Wave growth via his weblog, DrZWave.weblog, and ongoing IoT consulting tasks. He’s additionally a well-known face at Z-Wave Alliance Unplug Fests, the place he continuously serves because the lead coordinator, supporting interoperability and developer collaboration.
