X10 Community Forum

🔌General Home Automation => Automating Your House => Topic started by: bkenobi on June 21, 2014, 01:23:32 PM

Title: Non-flash storage for HA server
Post by: bkenobi on June 21, 2014, 01:23:32 PM
As a primer, I'm not using AHP anymore (turned it off today) and am now using a Raspberry Pi with HomeGenie as my HA software.  The Raspi is a very compact Linux based board that can do some pretty amazing things for the price.  It uses SD as its storage rather than a hard disk, so it is very low power.  This flash media does pose a concern for me...

I've read that flash storage has a limited number of writes before the device will fail.  I am using my Raspi as a home automation server and have it set up to log activity.  At the current rate, it seems to be writing something to the SD several times per minute (on the order of 10).  I'm concerned that this will prematurely kill the device, so I'm looking at alternative configurations.  The options I've considered so far are:



The intent is to have this be completely autonomous and require essentially no maintenance.  Logging is for use as a maintenance tool and should not create maintenance.  The most important thing is reducing down time to effectively zero.  Any comments/suggestions?
Title: Re: Non-flash storage for HA server
Post by: dhouston on June 23, 2014, 09:25:41 AM
I had composed a reply shortly after you posted but when I clicked Send, nothing happened and then the forum disappeared down some blackhole.

I doubt you will have a problem although it depends on the exact storage method you choose. Here's a quote from an article dated to about six years back

Quote
For example, the 16GB A-DATA SDHC card has an estimated endurance or lifetime of 1,000,000 write cycles. What does that mean is "real world" terms? You would have to constantly write, erase and re-write data non-stop for several years before you need to be concerned about failure.

If your SDHC is 4GB with a formatted capacity of 3900MB, and you do nothing but write to it as fast as you can – at, say, 30MB/s – you’ll still only be able to replace its entire contents every 130 seconds. At that rate, it’ll take you 1,500 days (4.1 years) to hit 1,000,000 cycles.

In short, by the time you need to worry about SDHC failure we’ll probably have 320GB SDHC cards or the computer industry will start using another type of storage medium. That said, every electronic device ever created can fail. We’ve had brand new hard drives and brand new SSDs fail in our office after less than a week of use. Bottom line, in most cases we don’t believe using an SDHC card is any less safe than any other storage methods.

The crucial parameter is erase/write cycles. This is per cell so each bit can be erased/written a certain number of times. A 32GB microSDHC card that can withstand 1,000,000 erase/write cycles will likely outlast you. You may need to devise a routine to overwrite the oldest entries once the card is full (if ever) but you can rewrite the whole shebang a million times. Of course, not all microSDHC cards are created equal so you need to verify the one you choose is rated for 1,000,000 write cycles.
Title: Re: Non-flash storage for HA server
Post by: Brian H on June 23, 2014, 09:43:16 AM
The X10 Forums and Wiki. Seemed to be off line this weekend. As I could not access either one of them.
Maybe you got caught just as you hit send it went down.
Title: Re: Non-flash storage for HA server
Post by: bkenobi on June 23, 2014, 09:49:21 AM
The data that is of concern is log and system statistics data written several times per minute.  The write is small, but I don't know if debian writes to the same location on disk or a new location and moves the file.  Either way, it's probably a non-issue.  I say that even though there is at least 1 report that a HomeGenie user has seen a SD fail.
Title: Re: Non-flash storage for HA server
Post by: dhouston on June 23, 2014, 09:55:19 AM
I would organize the writes by month, starting a new file each month. Then, as you run out of space, you can erase the oldest files or merely replace the microSDHC card.
Title: Re: Non-flash storage for HA server
Post by: bkenobi on June 23, 2014, 10:36:05 AM
I'm trying to convince the author of the software to modify how writes are taken care of.  I think ideally he would keep the statistics in a ram disk and only write the data to the SD once a day or so.  The logs that I am generating have a configurable write location or could be disabled.  At this time, there is no way to make the main system read only unless the software is installed on the second media as well.  But, that would require setting things up again if the second flash failed, too.  I suppose this is a slightly better option than losing the whole system, but only slightly since HG has a raspberry pi image that can be written to SD in which everything works out of the box.  There are a few things that need tweaking after writing, but it's pretty close to ready.
Title: Re: Non-flash storage for HA server
Post by: JeffVolp on June 23, 2014, 10:40:04 AM
I had been looking at replacing the boot drive in my new computer with a solid-state drive, but am also concerned about the durability of those drives after reading reviews on many of them on Amazon.  While most people seem to be very happy with them, there are reports of SSDs failing in a short time, making it impossible to recover anything from the drive.

I think it depends on how the controller stores data on the drive.  There should be no problem if it is smart enough to move frequently changed data to different memory cells.  But if that data is maintained at a fixed location, those cells may fail in a relatively short time.  Something changed once a second would be 86,400 writes per day.  At that rate it doesn't take long to get to 1,000,000.

Jeff
Title: Re: Non-flash storage for HA server
Post by: dhouston on June 23, 2014, 11:01:47 AM
I think it depends on how the controller stores data on the drive.  There should be no problem if it is smart enough to move frequently changed data to different memory cells.  But if that data is maintained at a fixed location, those cells may fail in a relatively short time.  Something changed once a second would be 86,400 writes per day.  At that rate it doesn't take long to get to 1,000,000.

Most SSDs and some SDHCs employ wear leveling which ensures even usage of the entire disc. I suspect most SDHCs rated for 1,000,000 erase/write cycles employ it.
Title: Re: Non-flash storage for HA server
Post by: dhouston on June 23, 2014, 11:18:56 AM
I'm trying to convince the author of the software to modify how writes are taken care of. 

Not having low-level control complicates the issue. My ZarduinoTM design had a slot for a FAT based microSDHC card so that influenced my train of thought.
Title: Re: Non-flash storage for HA server
Post by: bkenobi on June 23, 2014, 12:15:41 PM
The software is open source, so I suppose I could tweak things if necessary.  I'd rather not since any updates would then require a custom compile.
Title: Re: Non-flash storage for HA server
Post by: dhouston on June 23, 2014, 12:36:44 PM
Probably, your best option is to communicate your concerns to the HomeGenie folks. Perhaps they can address your issues.
Title: Re: Non-flash storage for HA server
Post by: bkenobi on June 23, 2014, 12:59:29 PM
Yup, working that path too.  Just trying to make sure all bases are covered.  Thanks for the suggestions!
Title: Re: Non-flash storage for HA server
Post by: dhouston on June 23, 2014, 03:20:43 PM
I had to refresh my own memory on this as it has been 18+ months since I explored it. My understanding is that SD, SDHC, microSDHC, etc. cards have a built-in microcontroller that automatically and transparently handles wear-leveling and that FAT is compatible with this. I doubt you need to worry about it.

EDIT: It has been awhile - I was unaware that there are now SDXC and microSDXC cards with 128GB. While pricey ($40+) I doubt you would see it fail in your lifetime.
Title: Re: Non-flash storage for HA server
Post by: bkenobi on June 23, 2014, 05:02:09 PM
Good to know.  I'm not so worried about the cost as I am the potential for down time that is inconvenient (read embarrassing or a reduction to WAF).  If all I have to do is swap a card once every few years and move on, that wouldn't be a problem.  Sounds like it SHOULDN'T be an issue especially if I can convince the author to reduce the number of writes to a few times a day rather than many times a minute.
Title: Re: Non-flash storage for HA server
Post by: bkenobi on June 23, 2014, 05:50:49 PM
From what I have read, the write endurance of SD is actually much less than 1M.  I find it's closer to 100K.  If so, 1 write/second would kill the block in 1.15 days if the same block were written each time.  I'm not sure if the block becomes unusable or if the whole card is dead.  Since there's a microprocessor that controls wear leveling, I would hope that it knows the number of writes and the health of the block.  If so, then I would imagine that over time, the card's capacity simply shrinks.
Title: Re: Non-flash storage for HA server
Post by: dhouston on June 23, 2014, 06:14:22 PM
From what I have read, the write endurance of SD is actually much less than 1M.  I find it's closer to 100K.

That seems to depend on the brand and is why I said to make sure of the specs for the card you choose. Wear-leveling spreads the wear over all of the card so storage capacity remains constant but expected life is reduced as berase/write cycles increase.
Title: Re: Non-flash storage for HA server
Post by: bkenobi on June 23, 2014, 07:10:23 PM
I just finished reading a lengthy thread on the Raspberry Pi forum:

http://www.raspberrypi.org/forums/viewtopic.php?f=63&t=19554

It has a lot of information on the actual implementation of different commands between Linux and the SD card or SSD.  It's good information, but I'm left a little confused as to what is actually implemented in the standard Raspi Debian release and how to properly set things up.  The consensus is that either SD is very robust, or it is very prone to corruption.  Very little in-between.  I think the best approach is to limit the number of needless write cycles where ever possible.  So, I'll continue to press the author to see what can be done.

With as many "experts" as were posting in that thread, I guess I'm unclear on why there isn't a consensus!   B:(
Title: Re: Non-flash storage for HA server
Post by: dhouston on June 24, 2014, 02:32:50 PM
I'll look at the thread you posted later. I've been trying to find a definitive rating for the expected life and have found everything from 10,000 to 1,000,000 erase/write cycles but have yet to find it clearly stated on any of the cards I looked at they tend to opt for years of use which likely came from marketing rather than engineering. AFAIK this is the same type of memory as used by Microchip & Atmel in their microcontrollers and, most if not all, of these have offered 1,000,000 erase/write cycles for several years.

That may soon be moot as the latest SDXC standard supports up to 2TB capacity. So far, I've only found 128GB readily available. And, they use exFAT which may not currently work with the Pi.