Had a NR instance do something weird, any idea what this might be?

General
1

Had a NR instance do something weird, any idea what this might be? Had nodered service crash on a Raspberry Pi. When I restarted the nodered service got the following in an endless loop via output from node-red-log...

Error: Cannot find module 'merge-descriptors'
Require stack:
- /usr/lib/node_modules/node-red/node_modules/express/lib/express.js
- /usr/lib/node_modules/node-red/node_modules/express/index.js
- /usr/lib/node_modules/node-red/red.js
    at Function.Module._resolveFilename (internal/modules/cjs/loader.js:957:15)
    at Function.Module._load (internal/modules/cjs/loader.js:840:27)
    at Module.require (internal/modules/cjs/loader.js:1019:19)
    at require (internal/modules/cjs/helpers.js:77:18)
    at Object.<anonymous> (/usr/lib/node_modules/node-red/node_modules/express/lib/express.js:17:13)
    at Module._compile (internal/modules/cjs/loader.js:1133:30)
    at Object.Module._extensions..js (internal/modules/cjs/loader.js:1153:10)
    at Module.load (internal/modules/cjs/loader.js:977:32)
    at Function.Module._load (internal/modules/cjs/loader.js:877:14)
    at Module.require (internal/modules/cjs/loader.js:1019:19) {
  code: 'MODULE_NOT_FOUND',
  requireStack: [
    '/usr/lib/node_modules/node-red/node_modules/express/lib/express.js',
    '/usr/lib/node_modules/node-red/node_modules/express/index.js',
    '/usr/lib/node_modules/node-red/red.js'
  ]
}
nodered.service: Main process exited, code=exited, status=1/FAILURE
nodered.service: Failed with result 'exit-code'.
nodered.service: Service RestartSec=100ms expired, scheduling restart.
nodered.service: Scheduled restart job, restart counter is at 223.

I have not surface tested the SD card yet, it is likely failing. But just wondered if anyone had an idea of what the above implied. Checking dmesg definitely have a few fsck warnings.

Rebooted and the reboot DID NOT run a media check on the SD card. Which is kinda spooking, because I know there were several fsck warnings logged. For example...

[662165.029222] EXT4-fs error (device mmcblk0p2): htree_dirblock_to_tree:1023: inode #28976: block 10157: comm mandb: bad entry in directory: inode out of bounds - offset=9296, inode=1079491, rec_len=20, name_len=10, size=4096
[750053.200235] EXT4-fs (mmcblk0p2): error count since last fsck: 1
[750053.200258] EXT4-fs (mmcblk0p2): initial error at time 1601424001: htree_dirblock_to_tree:1023: inode 28976: block 10157
[750053.200276] EXT4-fs (mmcblk0p2): last error at time 1601424001: htree_dirblock_to_tree:1023: inode 28976: block 10157
[836561.641964] EXT4-fs (mmcblk0p2): error count since last fsck: 1
[836561.641985] EXT4-fs (mmcblk0p2): initial error at time 1601424001: htree_dirblock_to_tree:1023: inode 28976: block 10157
[836561.642002] EXT4-fs (mmcblk0p2): last error at time 1601424001: htree_dirblock_to_tree:1023: inode 28976: block 10157
[921368.246810] EXT4-fs error (device mmcblk0p2): htree_dirblock_to_tree:1023: inode #28976: block 10157: comm mandb: bad entry in directory: inode out of bounds - offset=9296, inode=1079491, rec_len=20, name_len=10, size=4096
[921372.302998] EXT4-fs error (device mmcblk0p2): htree_dirblock_to_tree:1023: inode #28989: block 10111: comm mandb: bad entry in directory: rec_len is smaller than minimal - offset=140, inode=0, rec_len=0, name_len=0, size=4096
[921372.853522] EXT4-fs error (device mmcblk0p2): htree_dirblock_to_tree:1023: inode #28989: block 10111: comm mandb: bad entry in directory: rec_len is smaller than minimal - offset=140, inode=0, rec_len=0, name_len=0, size=4096
[923072.265187] EXT4-fs (mmcblk0p2): error count since last fsck: 4
[923072.265210] EXT4-fs (mmcblk0p2): initial error at time 1601424001: htree_dirblock_to_tree:1023: inode 28976: block 10157
[923072.265226] EXT4-fs (mmcblk0p2): last error at time 1601683208: htree_dirblock_to_tree:1023: inode 28989: block 10111
[1009582.942741] EXT4-fs (mmcblk0p2): error count since last fsck: 4
[1009582.942773] EXT4-fs (mmcblk0p2): initial error at time 1601424001: htree_dirblock_to_tree:1023: inode 28976: block 10157
[1009582.942824] EXT4-fs (mmcblk0p2): last error at time 1601683208: htree_dirblock_to_tree:1023: inode 28989: block 10111
[1096093.184938] EXT4-fs (mmcblk0p2): error count since last fsck: 4
[1096093.184992] EXT4-fs (mmcblk0p2): initial error at time 1601424001: htree_dirblock_to_tree:1023: inode 28976: block 10157
[1096093.185017] EXT4-fs (mmcblk0p2): last error at time 1601683208: htree_dirblock_to_tree:1023: inode 28989: block 10111
[1182603.904236] EXT4-fs (mmcblk0p2): error count since last fsck: 4
[1182603.904265] EXT4-fs (mmcblk0p2): initial error at time 1601424001: htree_dirblock_to_tree:1023: inode 28976: block 10157
[1182603.904289] EXT4-fs (mmcblk0p2): last error at time 1601683208: htree_dirblock_to_tree:1023: inode 28989: block 10111
[1269114.175752] EXT4-fs (mmcblk0p2): error count since last fsck: 4
[1269114.175782] EXT4-fs (mmcblk0p2): initial error at time 1601424001: htree_dirblock_to_tree:1023: inode 28976: block 10157
[1269114.175809] EXT4-fs (mmcblk0p2): last error at time 1601683208: htree_dirblock_to_tree:1023: inode 28989: block 10111
[1355624.429519] EXT4-fs (mmcblk0p2): error count since last fsck: 4
[1355624.429549] EXT4-fs (mmcblk0p2): initial error at time 1601424001: htree_dirblock_to_tree:1023: inode 28976: block 10157
[1355624.429600] EXT4-fs (mmcblk0p2): last error at time 1601683208: htree_dirblock_to_tree:1023: inode 28989: block 10111
[1442134.269767] EXT4-fs (mmcblk0p2): error count since last fsck: 4
[1442134.269796] EXT4-fs (mmcblk0p2): initial error at time 1601424001: htree_dirblock_to_tree:1023: inode 28976: block 10157
[1442134.269823] EXT4-fs (mmcblk0p2): last error at time 1601683208: htree_dirblock_to_tree:1023: inode 28989: block 10111
[1528644.954008] EXT4-fs (mmcblk0p2): error count since last fsck: 4
[1528644.954039] EXT4-fs (mmcblk0p2): initial error at time 1601424001: htree_dirblock_to_tree:1023: inode 28976: block 10157
[1528644.954063] EXT4-fs (mmcblk0p2): last error at time 1601683208: htree_dirblock_to_tree:1023: inode 28989: block 10111

Like I said this is kinda interesting and spooking... TO EARLY FOR ALL HALLOWS EVE? RIGHT?

At the moment NR instance is up and running, after reboot no fsck warnings as yet. But the editor is still trashed. So it is a headless NR instance at the moment. But not sure I trust the OS at this point. Never mind the SD media below it.

2

It is either a dying card or corrupted (by power fail possibly). If you value time over money then bin the card and start again. Otherwise reinstall the OS and it may be ok, or it may fail again.

3

Right, just as test of the SD, I reimaged it, because when I tried to selectively reinstall NR, the install script throw errors that something in the file system tree under ./node-red was corrupted. I forced fsck to run via cmdline.txt on next reboot, and it passed. So watching the new image running just to see what it does.

Side note, talking to Kingston about a suspected fake SD card issue, apparently the fake SD cards are getting harder to spot, and now are appearing on more sites for sale than just eBay. I got a batch of SD cards from Amazon a while ago that looked and where qualified as valid, but they failed faster than expected, and sure enough, they are suspect.

Moreover, there has been evidently several production quality issues, where legit SD cards are failing sooner than desired. For example, I have had some known legit SD cards die just after the 90 day warranty window closes definitely in less than 6 months.

Fail to format due to the reported size issue is significant in my experience. The number of Google hits on this has significantly increased per my perception, and mirrors my experience recently. I will not name the producer, but it is one of the big names, previously known for quality SD card production.

4

This topic was automatically closed 60 days after the last reply. New replies are no longer allowed.