Understanding the Notecard Penalty Boxes

If you have a Notecard that is no longer communicating with Notehub or is repeatedly unable to connect to a cellular network, the Notecard may have been placed in a figurative "penalty box".

note

If lack of a reliable cellular connection is the reason your Notecard is not functional, you can get help troubleshooting your connection in this guide on Diagnosing Cellular Connectivity.

What is a Penalty Box?

A penalty box is a state in which the Notecard is isolated and not allowed to even attempt a connection to a network (or, in the case of the GPS penalty box, a new GPS seek). Repeating a network connection failure multiple times can unnecessarily drain an attached battery. Therefore, the penalty box is set up to preserve battery life while also occasionally "opening up" and allowing the Notecard to attempt a new connection at certain intervals.

If you have enabled the capture of trace logs on a Notecard, you may see log events that refer to the penalty box, such as:

{"text":"penalty: entering penalty box: connect delayed (89 min remaining): can't open session to notehub: socket open PPP: ppp connection timeout? {service}{extended-service-failure}"}

{"text":"penalty: removed from penalty box: penalty period expired"}

In this guide we will look at the reasons why a Notecard may be placed in a penalty box and under which conditions a Notecard may be removed from a penalty box.

note

The failure of a Notecard to sync with Notehub does not necessarily mean it is in a penalty box. It could also be due to the use of an invalid ProductUID or incorrect Notehub host name. These can be diagnosed in the response to a hub.sync.status command. For example:

COPY

{"req":"hub.sync.status"}

COPY

{"alert":true,"status":"connect delayed (60 min remaining): can't open session to discovery service: socket: cannot find host: blah.notefile.net {service}{host-unreachable} {service}{extended-service-failure}","time":1638397092,"completed":24}

Types of Penalty Boxes

There are four types of penalty boxes that a Notecard may enter after repeated failures:

1. Insufficient Power Supply
2. Network Registration Failure
3. Inability to Connect to Notehub
4. Inability to Acquire GPS Satellite Fix

Each penalty box is explained below including commands on how to manually remove a Notecard from a penalty box (when available).

Insufficient Power Supply

If the Notecard reboots due to insufficient current supply while the modem is trying to register the Notecard on a cellular network, it will enter a penalty box. If the reboot is due to a card.restart, card.restore, or a DFU-induced restart, the Notecard will not enter a penalty box.

The length of time the Notecard is in the penalty box is 1/2 of the outbound sync interval or 120 minutes (whichever is greater). For each subsequent failure during modem registration, the penalty box interval grows by 120 minutes, up to a maximum of three days.

The Notecard will be released from the penalty box upon a successful registration on a cellular network, or if the host issues any of the following:

1. A hub.sync request with the allow:true parameter.
2. A card.wireless.penalty request with the reset:true parameter.
3. A hub.log request with the sync:true parameter.
4. Any change sent in a hub.set request.

Network Registration Failure

A network registration attempt is considered a failure if the cellular modem on the Notecard cannot register with a network within the applicable registration timeout:

A single failure does not put a Notecard in the penalty box. Registration failures must occur four consecutive times before the penalty box takes effect, or five consecutive times when the Notecard is switching between an internal and external SIM in auto-SIM mode.

The length of time the Notecard is in the penalty box is extended by each subsequent network registration failure, by a minimum of 15 minutes. For each failure the previous delay is multiplied by 1.25 + 15 minutes, up to a maximum of 3 days. For example, the first set of retry intervals (in minutes) are: 15, 34, 57, 86, 123, 169, 226, and 298.

The Notecard will be released from the penalty box upon a successful registration on a cellular network, or if the host issues any of the following:

1. A hub.sync request with the allow:true parameter.
2. A card.wireless.penalty request with the reset:true parameter.
3. A hub.log request with the sync:true parameter.
4. A hub.set request that changes a synchronization parameter (for example mode, outbound, inbound, host, or product).

Overriding Network Registration Penalty Box Settings

warning

The misuse of this feature may result in the cellular carrier preventing the Notecard from future connections because it's effectively "spamming" the network. The cellular carrier may blacklist devices that it thinks are attempting to connect too frequently.

The default settings that determine the duration of the Network Registration Penalty Box are calculated based on the number of consecutive registration failures and can be overridden at runtime by using the card.wireless.penalty API:

{
   "req":  "card.wireless.penalty",
   "set":  true,
   "rate": <PenaltyMultiplierRate(float)>,
   "add":  <PenaltyAddMins>,
   "max":  <PenaltyMaxMins>,
   "min":  <PenaltyMinMins>
}

note

If overriding the default settings with this API request, all of the above fields MUST be specified.

When using this API request, the first retry interval is PenaltyMinMins. For each successive failure, the prior delay is multiplied by PenaltyMultiplierRate plus another PenaltyAddMins is added, up to a maximum of PenaltyMaxMins.

For example, by sending the following request, the retry intervals would be 5, 20, 50, 110, 230, 470, and 720 minutes:

{"req":"card.wireless.penalty","set":true,"rate":2.0,"add":10,"max":720,"min":5}

Likewise, the default settings for a Network Registration Penalty Box can also be overridden with the environment variable _wireless_penalty:

_wireless_penalty = <PenaltyMultiplierRate(float)>,<PenaltyAddMins>,<PenaltyMinMins>,<PenaltyMaxMins>

Inability to Connect to Notehub

If the cellular modem can successfully register with the network, but is unable to connect to Notehub, the Notecard may be placed in a "Notehub connection failure" penalty box.

The Notecard will enter this penalty box if connection failures to Notehub happen 3 consecutive times (if the Notecard is battery-powered) or 5 consecutive times (if the Notecard is USB/line-powered). Note that this penalty box will only come into play if there are no other network registration or power-related errors.

How long the Notecard remains in the penalty box depends on how it is powered:

• Battery-powered: 1/2 of the configured outbound sync interval or 60 minutes, whichever is greater.
• USB or line-powered: the fifth failure and the two after it each result in a short randomized delay of 3-8 minutes, giving a transient Notehub or network problem a chance to clear. The eighth consecutive failure is the first to incur the full 60 minute penalty.

The Notecard will be released from the penalty box upon a successful connection to Notehub, or if the host issues any of the following:

1. A hub.sync request with the allow:true parameter.
2. A card.wireless.penalty request with the reset:true parameter.
3. A hub.log request with the sync:true parameter.
4. A hub.set request that changes a synchronization parameter (for example mode, outbound, inbound, host, or product).

Inability to Acquire GPS Satellite Fix

If the Notecard is configured to acquire its location via GNSS (GPS) but is unable to make a position fix, it may be placed in a penalty box (provided all of the following conditions are met):

1. The device is in motion (as determined by the on-board accelerometer).
2. Enough GPS satellites are visible for a fix to be plausible (at least three simultaneously tracked satellites).
3. A GPS position fix is NOT made before the seek expires, which by default is after approximately 17 minutes (1,000 seconds) of searching.

note

If the GPS is not able to track at least three satellites at any point during a 90 second window, the seek is abandoned early and the Notecard is not placed in a penalty box (or it will be removed from the penalty box if it was already in one). This is deliberate: a device parked deep inside a building, where no satellites are visible at all, should not accumulate an ever-growing penalty.

The length of time the Notecard is in the penalty box is 30 minutes multiplied by the number of consecutive GPS position fix failures (up to a maximum of 12 hours). Reaching the 12 hour maximum requires 24 consecutive expired seeks, which takes approximately 6 days of continuous GPS position fix failures on a device that keeps moving—after three extended cancels, motion alone no longer triggers a new seek.

The Notecard will be released from the penalty box if either:

1. The GPS makes a valid position fix.
2. The GPS is unable to track enough satellites and the seek is abandoned early.

note

Unlike the network penalty boxes, the GPS penalty box has no manual reset request—card.wireless.penalty with reset:true does not clear it.

Checking Penalty Box Status

Rather than inferring a penalty box from a Notecard's silence, you can ask for it directly.

On the Device

Sending card.wireless.penalty with no arguments reads the current penalty state instead of changing it:

{"req":"card.wireless.penalty"}

{
  "seconds": 3324,
  "minutes": 69,
  "status": "network: can't connect (55.4 min remaining) {registration-failure}{network}{extended-network-failure}",
  "count": 6
}

• seconds is the time remaining in the penalty box.
• status is the penalty box error, using the tokens described in Notecard Error and Status Codes.
• count is the number of consecutive network registration failures.
• minutes is the time elapsed since the first of those failures.

When no penalty is pending, the seconds and status fields are absent. Note that minutes and count are reported for cellular transports only.

The same information is decorated onto hub.sync.status. For example:

{"req":"hub.sync.status"}

{"alert":true,"status":"network: can't connect (55.4 min remaining) {registration-failure}{network}{extended-network-failure}","seconds":3324}

The GPS penalty box is reported separately, by card.location. When a GPS penalty is in effect, the response's status field includes a {gps-penalty} token followed by the number of seconds remaining, and count reports the number of consecutive GPS cancellations:

{"req":"card.location"}

{"status":"GPS inactive {gps-inactive} {gps-penalty} (3421 s)","mode":"periodic","count":4}

From Notehub

A Notecard reports how long it spent in a penalty box before it managed to connect, so you can spot penalized devices across a fleet without physical access. Each _session.qo event carries:

• penalty_secs: the number of seconds the Notecard was in a penalty box during the prior session.
• failed_connects: the number of connection failures during the prior session.

Because these are reported at connection time, they are retrospective. They tell you a device was penalized, not that it is penalized right now.

Additional Resources

• Capture Trace Logs
• card.wireless.penalty API
• Notecard Error and Status Codes