Print

Please login or register.
1 Hour 1 Day 1 Week 1 Month Forever
Login with username, password and session length

[bkenobi]

As a primer, I'm not using AHP anymore (turned it off today) and am now using a Raspberry Pi with HomeGenie as my HA software.  The Raspi is a very compact Linux based board that can do some pretty amazing things for the price.  It uses SD as its storage rather than a hard disk, so it is very low power.  This flash media does pose a concern for me...

I've read that flash storage has a limited number of writes before the device will fail.  I am using my Raspi as a home automation server and have it set up to log activity.  At the current rate, it seems to be writing something to the SD several times per minute (on the order of 10).  I'm concerned that this will prematurely kill the device, so I'm looking at alternative configurations.  The options I've considered so far are:

• Don't write anything - Not valid option for HA since I need to know what has happened to verify everything is working correctly
• Write to SD and keep a backup for when the original dies - Currently this is my best option.  MicroSD is cheap and if it only dies once a year, that wouldn't be tragic.  I would prefer never touching it, but this method seems like the current front runner.
• Install a flash drive in the second USB for logging - A flash device is a flash device, so this will still die just like the SD.  The benefit here is that if I kill of a flash drive, I can always install a new one without reconfiguring anything.  I like this option also since I can pull the device to check logs on a different machine if necessary.  Only down side is I have to figure out how to change where logs are written by the HA software.
• Write logs to a network device (either existing file server or new NAS dedicated to HA server) - Same as flash drive option, but this shouldn't die due to too many writes.  Requires more power if I add a new device, but if I use existing server, no new hardware.  Only question is whether network issues could screw up the software during a write if the drive was not found.
• Buy/install a USB hard drive - Possibly simpler than a NAS option.  Local device keeps everything together.
• Install a microdrive in a CF->USB adapter - Similar to local USB HDD, this would just reduce power requirements.  These seem cheap (<$10 shipped on ebay for 6GB) and would provide plenty of storage.

The intent is to have this be completely autonomous and require essentially no maintenance.  Logging is for use as a maintenance tool and should not create maintenance.  The most important thing is reducing down time to effectively zero.  Any comments/suggestions?

[dhouston]

I had composed a reply shortly after you posted but when I clicked Send, nothing happened and then the forum disappeared down some blackhole.

I doubt you will have a problem although it depends on the exact storage method you choose. Here's a quote from an article dated to about six years back

• http://www.notebookreview.com/news/sdhc-cards-vs-hard-drive-vs-ssd/

For example, the 16GB A-DATA SDHC card has an estimated endurance or lifetime of 1,000,000 write cycles. What does that mean is "real world" terms? You would have to constantly write, erase and re-write data non-stop for several years before you need to be concerned about failure.

If your SDHC is 4GB with a formatted capacity of 3900MB, and you do nothing but write to it as fast as you can – at, say, 30MB/s – you’ll still only be able to replace its entire contents every 130 seconds. At that rate, it’ll take you 1,500 days (4.1 years) to hit 1,000,000 cycles.

In short, by the time you need to worry about SDHC failure we’ll probably have 320GB SDHC cards or the computer industry will start using another type of storage medium. That said, every electronic device ever created can fail. We’ve had brand new hard drives and brand new SSDs fail in our office after less than a week of use. Bottom line, in most cases we don’t believe using an SDHC card is any less safe than any other storage methods.

The crucial parameter is erase/write cycles. This is per cell so each bit can be erased/written a certain number of times. A 32GB microSDHC card that can withstand 1,000,000 erase/write cycles will likely outlast you. You may need to devise a routine to overwrite the oldest entries once the card is full (if ever) but you can rewrite the whole shebang a million times. Of course, not all microSDHC cards are created equal so you need to verify the one you choose is rated for 1,000,000 write cycles.

[Brian H]

The X10 Forums and Wiki. Seemed to be off line this weekend. As I could not access either one of them.
Maybe you got caught just as you hit send it went down.

[bkenobi]

The data that is of concern is log and system statistics data written several times per minute.  The write is small, but I don't know if debian writes to the same location on disk or a new location and moves the file.  Either way, it's probably a non-issue.  I say that even though there is at least 1 report that a HomeGenie user has seen a SD fail.

[dhouston]

I would organize the writes by month, starting a new file each month. Then, as you run out of space, you can erase the oldest files or merely replace the microSDHC card.

[bkenobi]

I'm trying to convince the author of the software to modify how writes are taken care of.  I think ideally he would keep the statistics in a ram disk and only write the data to the SD once a day or so.  The logs that I am generating have a configurable write location or could be disabled.  At this time, there is no way to make the main system read only unless the software is installed on the second media as well.  But, that would require setting things up again if the second flash failed, too.  I suppose this is a slightly better option than losing the whole system, but only slightly since HG has a raspberry pi image that can be written to SD in which everything works out of the box.  There are a few things that need tweaking after writing, but it's pretty close to ready.

[JeffVolp]

I had been looking at replacing the boot drive in my new computer with a solid-state drive, but am also concerned about the durability of those drives after reading reviews on many of them on Amazon.  While most people seem to be very happy with them, there are reports of SSDs failing in a short time, making it impossible to recover anything from the drive.

I think it depends on how the controller stores data on the drive.  There should be no problem if it is smart enough to move frequently changed data to different memory cells.  But if that data is maintained at a fixed location, those cells may fail in a relatively short time.  Something changed once a second would be 86,400 writes per day.  At that rate it doesn't take long to get to 1,000,000.

Jeff

[dhouston]

I think it depends on how the controller stores data on the drive.  There should be no problem if it is smart enough to move frequently changed data to different memory cells.  But if that data is maintained at a fixed location, those cells may fail in a relatively short time.  Something changed once a second would be 86,400 writes per day.  At that rate it doesn't take long to get to 1,000,000.

Most SSDs and some SDHCs employ wear leveling which ensures even usage of the entire disc. I suspect most SDHCs rated for 1,000,000 erase/write cycles employ it.

[dhouston]

I'm trying to convince the author of the software to modify how writes are taken care of. 

Not having low-level control complicates the issue. My Zarduino^TM design had a slot for a FAT based microSDHC card so that influenced my train of thought.

[bkenobi]

The software is open source, so I suppose I could tweak things if necessary.  I'd rather not since any updates would then require a custom compile.

[dhouston]

Probably, your best option is to communicate your concerns to the HomeGenie folks. Perhaps they can address your issues.

[bkenobi]

Yup, working that path too.  Just trying to make sure all bases are covered.  Thanks for the suggestions!

[dhouston]

I had to refresh my own memory on this as it has been 18+ months since I explored it. My understanding is that SD, SDHC, microSDHC, etc. cards have a built-in microcontroller that automatically and transparently handles wear-leveling and that FAT is compatible with this. I doubt you need to worry about it.

EDIT: It has been awhile - I was unaware that there are now SDXC and microSDXC cards with 128GB. While pricey ($40+) I doubt you would see it fail in your lifetime.

[bkenobi]

Good to know.  I'm not so worried about the cost as I am the potential for down time that is inconvenient (read embarrassing or a reduction to WAF).  If all I have to do is swap a card once every few years and move on, that wouldn't be a problem.  Sounds like it SHOULDN'T be an issue especially if I can convince the author to reduce the number of writes to a few times a day rather than many times a minute.

[bkenobi]

From what I have read, the write endurance of SD is actually much less than 1M.  I find it's closer to 100K.  If so, 1 write/second would kill the block in 1.15 days if the same block were written each time.  I'm not sure if the block becomes unusable or if the whole card is dead.  Since there's a microprocessor that controls wear leveling, I would hope that it knows the number of writes and the health of the block.  If so, then I would imagine that over time, the card's capacity simply shrinks.