The Linux Adventure Part 5: Everything is Broken

Ye gods.

You know all the stuff I wrote in the last five blog posts or so about the NAS unit? Well, it turns out that while it worked, side effects broke some the unit's functionality. First, the unit's network light would flicker and then it would cease to go to sleep, a critical function for energy savings on a 24/7 appliance. Then I also noticed the Diskstation would not shut down or restart when given the manual command to do so in the software.

After much gnashing of teeth and many face palms, I learned that bootstrapping the unit caused the issues. Synology generally is quite liberal about bootstrapping, even including information on how to do it in its official support resources. However, the operating system software, Disk Station Manager (DSM), gets updated regularly. Usually, this is a good sign that shows a company hasn't abandoned its product and is actively supporting and improving it. However, it also means that the environment running the station is a moving target. And by also allowing bootstrapping, Synology has led me into the very bear traps I see corporate IT shops sticking their balls into every day.

Young Profession, Old Debate

In the corporate IT world, there's an established debate about "build vs buy". Do you build the software you need from scratch or do you buy an off-the-shelf package? The debate usually has these points:

Build

• PRO: Software is customized to your needs and way of doing business
• PRO: You have complete control of the code
• PRO and CON: You have complete responsibility for the code
• PRO: Proprietary business knowledge is institutionalized in the code
• PRO: Enterprise processes are enforced by the software globally (barring proper implementation)
• CON: In-house development is expensive
• CON: In-house development often requires non-software shops to have proficiency in software development (bigger con: in reality most IT shops have at best mediocre competency).
• PRO: Properly implemented, a custom software team can more rapidly change the software than a major vendor can or will

Buy

• PRO: Buying someone else's software is less expensive because you don't have to have in-house development resources and licensing is cheaper than development. That's the theory anyway...in reality I'm not convinced it's less expensive, but I agree that writing a check monthly is probably less work than having to account for a staff of developers and all the extra HR overhead.
• PRO: Buying means you just use the software and the vendor handles development and support. Those of you that know better can laugh now.
• CON: Your business now does business the way the vendor's software wants you to, not necessarily the way your business people want to.
• CON: There may be limited capacity for customization of the vendor software to implement proprietary strategic processes
• CON: Reality shows that for core business (not commodity items like word processing or spreadsheets) buying is still expensive and implementation missteps often negate any cost advantage
• PRO: Certain industry standard processes, if implemented well, can be a part of the vendor software
• CON: It is rare that all companies do business exactly the same way
• PRO: Vendors may be able to capitalize on integration with common third party packages for things like accounting
• CON: You may think that a vendor is more dedicated to the principles of good software design and best practices and therefore will deliver stable, efficient and intuitive high-quality software just like the kind you find on Apple computers. The truth is that some shops may be very good, but most are made up the same knuckleheads that soured you on the "build" approach.

We Want to Have our Cake and Eat it Too

In the 80's and 90's it was common for companies to take a "build" approach. Software was new and exciting and there weren't many vendor packages available, so the typical IT guy said, "Oh, it's easy, all we have to do is..." and much spaghetti code was born.

Later companies started realizing what a mess it was to maintain crappy systems. Prodded along by drinks on the golf course and rides on corporate yachts, they decided to buy instead. But they made a critical mistake. They still wanted to do business "their" way and that required changing the software they bought. They wanted the best of both worlds but ended up with the worst of both worlds by buying and then heavily customizing.  Oops.

Now they had vendors that wouldn't or couldn't respond quickly to changing the software to fix or add desired functionality. Their in-house customizations did address some of the core software's gaps but built by inexperienced developers, proved cantankerous and bug-ridden and the users hated using them. And there was another issue: when the vendor had a new version, upgrades were that much harder because they could break the customizations. Many would suffer nervous breakdowns during this time, but consulting companies would happily offer help in exchange for a chunk of the company's life savings.

Holy Crap I did it Too!

That brings me back to the NAS unit. I got it initially so I could store my photos, documents and music in one location instead of having them dispersed on four different PCs and dozens of other flash cards and portable drives and memory sticks. I could have rolled my own by putting together a simple Linux server in a low profile case, but I wanted the off-the-shelf solution that would let me plug-and-play. I wanted the benefits of commodity.

But like the corporate idiots before me, I got greedy and wanted this wonderful Linux server to do more. So I bootstrapped. I customized. And I encountered an incompatibility with the latest version of the NAS software that doesn't work when you also have the NAS change the default shell to bash from ash. It caused other commands in the script to fail, so several services such as the sleep function and the audio server and photo server ceased to work. The unit would not even heed the manual shutdown or restart commands. Ugh.

I suppose I'm not quite as stupid as the so called "leadership" which commits to decisions that really create hassles for thousands of people and ultimately cost billions and billions of dollars. My suffering is confined to just me; honest men wouldn't have it any other way. But I do feel some embarrassment at having made a similar mistake.

Sometimes It Takes Two to Mess Things Up

I had help though. For the first year I owned the Synology unit I was pretty happy with it. I still am, when it comes to the basic functionality of the unit. However, when I spend money on something I expect it to work, not to be brittle like the rest of the software out there. The support forums for the Diskstations are filled with people that have similar problems as mine, and some even from folks that did not bootstrap. It appears that the regular updates to the DSM software can be risky, which shouldn't surprise me, but the level of dramatic errors and functionality loss that can occur do. This is Synology's hardware, not mine, so they ought to be able to release a beta that doesn't crush functionality. This excellent thread [serverfault.com] shows though that apparently Synology's developers took several short cuts and made some sloppy moves in building their software and products.

In my case, I had to remove the lines I added to profile config to launch the bash shell and allow it to remain in the default ash shell. Now the Diskstation again responds to the manual shutdown and restart commands. After reindexing, Audiostation is back to working status. However, the sleep behavior is still broken, and now the next thing for me to try is downgrading the DSM software (currently in version 4.3 beta) back to perhaps version 4.2. The DSM front-end software warns before doing DSM updates that the DSM software cannot be rolled back, but thanks to the enterprising user community, there are ways to do it.

I'll be working on downgrading the software. I will probably lose the GUI-based task scheduler since that was part of the 4.3 beta, but I may still be able to access crontab in ash, and create the scheduled job that way. And having the Diskstation off for a bit isn't so bad; it'll be nice not to worry about those Internet pinheads that keep probing the machine. In the meantime I'll continue to use my laptop for some of my Linux needs.