Skip to main content

Console Post Mortem (10/27 Connectivity Issues)

· 2 min read
Dal
Dal
Marc Nijdam
Marc Nijdam

On 10/27 a fix based on a community-raised issue was deployed along with a number of other updates in the 2021.10.27.0 Hotspot firmware release.

The firmware update included related PR miner#1136 which caused timing issues that resulted in perpetual join accept and requests, and impacted downlinks.

More details:

This issue only affected devices attempting joins or downlinks. Devices sending uplinks were not affected.

An emergency update 2021.10.29.0 was pushed on 10/29 afternoon to miners/hotspots to fix the issue. Read blog for more information.

Next Steps​

To increase the robustness of the network and prevent a similar issue from reoccurring the team agreed upon the following next steps.

More focused releases​

  • Focus on targeted releases which address related items vs just opportunistically adding unrelated items.

More scrutiny for issues raised​

Request clarity from community for relevant issues by asking members up front to include additional context to answer the following:

  • What’s the problem that needs fixed or the opportunity for this feature?
  • Define the success criteria to test in the field to make sure it’s fixed or working.

More canaries​

  • Broaden beta Hotspot test environment to include more canaries (hardware devices connected to beta Hotspot test groups).
  • Add automated network side black-box testing to verify devices continue to operate as expected after beta Hotspots updated.