Craftsmanship is Dead

Sorry in advance for the negativity but I just encountered something that has really pissed me off. Also, apologies in advance for salty language, but what I'm about to show you doesn't deserve professionalism. Again: WARNING, FOUL LANGUAGE FOLLOWS.

I complain about software developers misunderstanding their job as "easy" mostly because they're lazy bums that don't want to do a complete job. But developers aren't the only half-assing bastards out there.

I'm still living in my first home and I am regularly appalled at the shortcuts I've seen the builders take in making this house. There are places where the most fundamental of building constructs, the 90 DEGREE RIGHT ANGLE, was built incorrectly. Good grief, how stupid do you have to be to mess up a right angle?

When I look in the attic, it's just a head-smacking collection of hacks and patches. And I paid money for this?

Yesterday I replaced a garbage disposal unit that went bad after 14 years. This Whirlaway 191 1/3 horsepower unit is not what pissed me off. It was dirty but 14 years in a day when everything is disposable is not bad, and really it was just one of the blades that went bad, the rest of the unit probably could have soldiered on a couple more years.

But the part that got me was when I went to remove the existing power cable so I could transition it to a replacement Whirlaway 291. The power cable's ground wire had not been fastened to the ground screw on the old unit. Wow. Really? The guy that installed this was that lazy? He just cut the ground wire so it wouldn't be in the way, connected the other two wires, and plugged it in. Not a care in the world about any possible power surge or electrocution. Fucking asshole!

And it gets better! When I took a look at the plug to verify the larger prong so I could trace it back to the exposed wire and connect it to the proper wire PER INSTRUCTIONS, this is what I find:

Yep. Take a close look at those prongs. Just look at them. Yes, that's right, the lazy fucker filed them down so he wouldn't have to worry about whether he had the right wires connected. FUCK ME. I thought this kind of bullshit was supposed to exist only in the realm of Tim Allen jokes.

Yeah, I know, there are dozens of professional electricians out there that will say, "Oh, you're making too much of a big deal about it. This is low voltage bullshit that won't hurt anyone."

Except...THIS ISN'T ABOUT VOLTAGE. This is about doing a COMPLETE, THOROUGH and CORRECT job! This is about the simple task of following instructions handed down by the professionals that made the garbage disposal unit. Was it really that hard that the contractors couldn't do it right? Would it really have taken that much longer to do it to specifications? To be, oh, I don't know...SAFE?

Here's the kicker...the guy that did this is not just lazy, he's stupid too. Because he didn't file down the ground prong, filing down the other prongs doesn't make a difference...the ground prong forces you to put the plug into the socket correctly. So I'm left to think this guy filed down the prongs out of habit.

It turns out the wires were connected correctly, but the evidence points to this being a stroke of luck rather than the product of professionalism and preparedness.

Holy shit. I paid for this shit. This entire thing is just fucking embarrassing. It brings me such comfort to know this house was at least partially built by zombies.

You know, in the past I've withheld names to protect the not-so-innocent. But I can't take it anymore. The only thing that will make me feel better is the truth. I bought this house from PULTE, owned by BILL PULTE, a so-called MASTER BUILDER. The fact is that Bill Pulte never touched my house, it was one of the thousands of contractors he's got working for him that he's never even met (decision-consequence gap, bitches!). So there you go. When you're looking at houses, remember what I told you. But I wouldn't be surprised if other builders were cutting corners too.

CRAFTSMANSHIP IS DEAD.

The Linux Adventure Part 5: Everything is Broken

Ye gods.

You know all the stuff I wrote in the last five blog posts or so about the NAS unit? Well, it turns out that while it worked, side effects broke some the unit's functionality. First, the unit's network light would flicker and then it would cease to go to sleep, a critical function for energy savings on a 24/7 appliance. Then I also noticed the Diskstation would not shut down or restart when given the manual command to do so in the software.

After much gnashing of teeth and many face palms, I learned that bootstrapping the unit caused the issues. Synology generally is quite liberal about bootstrapping, even including information on how to do it in its official support resources. However, the operating system software, Disk Station Manager (DSM), gets updated regularly. Usually, this is a good sign that shows a company hasn't abandoned its product and is actively supporting and improving it. However, it also means that the environment running the station is a moving target. And by also allowing bootstrapping, Synology has led me into the very bear traps I see corporate IT shops sticking their balls into every day.

Young Profession, Old Debate

In the corporate IT world, there's an established debate about "build vs buy". Do you build the software you need from scratch or do you buy an off-the-shelf package? The debate usually has these points:

Build

• PRO: Software is customized to your needs and way of doing business
• PRO: You have complete control of the code
• PRO and CON: You have complete responsibility for the code
• PRO: Proprietary business knowledge is institutionalized in the code
• PRO: Enterprise processes are enforced by the software globally (barring proper implementation)
• CON: In-house development is expensive
• CON: In-house development often requires non-software shops to have proficiency in software development (bigger con: in reality most IT shops have at best mediocre competency).
• PRO: Properly implemented, a custom software team can more rapidly change the software than a major vendor can or will

Buy

• PRO: Buying someone else's software is less expensive because you don't have to have in-house development resources and licensing is cheaper than development. That's the theory anyway...in reality I'm not convinced it's less expensive, but I agree that writing a check monthly is probably less work than having to account for a staff of developers and all the extra HR overhead.
• PRO: Buying means you just use the software and the vendor handles development and support. Those of you that know better can laugh now.
• CON: Your business now does business the way the vendor's software wants you to, not necessarily the way your business people want to.
• CON: There may be limited capacity for customization of the vendor software to implement proprietary strategic processes
• CON: Reality shows that for core business (not commodity items like word processing or spreadsheets) buying is still expensive and implementation missteps often negate any cost advantage
• PRO: Certain industry standard processes, if implemented well, can be a part of the vendor software
• CON: It is rare that all companies do business exactly the same way
• PRO: Vendors may be able to capitalize on integration with common third party packages for things like accounting
• CON: You may think that a vendor is more dedicated to the principles of good software design and best practices and therefore will deliver stable, efficient and intuitive high-quality software just like the kind you find on Apple computers. The truth is that some shops may be very good, but most are made up the same knuckleheads that soured you on the "build" approach.

We Want to Have our Cake and Eat it Too

In the 80's and 90's it was common for companies to take a "build" approach. Software was new and exciting and there weren't many vendor packages available, so the typical IT guy said, "Oh, it's easy, all we have to do is..." and much spaghetti code was born.

Later companies started realizing what a mess it was to maintain crappy systems. Prodded along by drinks on the golf course and rides on corporate yachts, they decided to buy instead. But they made a critical mistake. They still wanted to do business "their" way and that required changing the software they bought. They wanted the best of both worlds but ended up with the worst of both worlds by buying and then heavily customizing.  Oops.

Now they had vendors that wouldn't or couldn't respond quickly to changing the software to fix or add desired functionality. Their in-house customizations did address some of the core software's gaps but built by inexperienced developers, proved cantankerous and bug-ridden and the users hated using them. And there was another issue: when the vendor had a new version, upgrades were that much harder because they could break the customizations. Many would suffer nervous breakdowns during this time, but consulting companies would happily offer help in exchange for a chunk of the company's life savings.

Holy Crap I did it Too!

That brings me back to the NAS unit. I got it initially so I could store my photos, documents and music in one location instead of having them dispersed on four different PCs and dozens of other flash cards and portable drives and memory sticks. I could have rolled my own by putting together a simple Linux server in a low profile case, but I wanted the off-the-shelf solution that would let me plug-and-play. I wanted the benefits of commodity.

But like the corporate idiots before me, I got greedy and wanted this wonderful Linux server to do more. So I bootstrapped. I customized. And I encountered an incompatibility with the latest version of the NAS software that doesn't work when you also have the NAS change the default shell to bash from ash. It caused other commands in the script to fail, so several services such as the sleep function and the audio server and photo server ceased to work. The unit would not even heed the manual shutdown or restart commands. Ugh.

I suppose I'm not quite as stupid as the so called "leadership" which commits to decisions that really create hassles for thousands of people and ultimately cost billions and billions of dollars. My suffering is confined to just me; honest men wouldn't have it any other way. But I do feel some embarrassment at having made a similar mistake.

Sometimes It Takes Two to Mess Things Up

I had help though. For the first year I owned the Synology unit I was pretty happy with it. I still am, when it comes to the basic functionality of the unit. However, when I spend money on something I expect it to work, not to be brittle like the rest of the software out there. The support forums for the Diskstations are filled with people that have similar problems as mine, and some even from folks that did not bootstrap. It appears that the regular updates to the DSM software can be risky, which shouldn't surprise me, but the level of dramatic errors and functionality loss that can occur do. This is Synology's hardware, not mine, so they ought to be able to release a beta that doesn't crush functionality. This excellent thread [serverfault.com] shows though that apparently Synology's developers took several short cuts and made some sloppy moves in building their software and products.

In my case, I had to remove the lines I added to profile config to launch the bash shell and allow it to remain in the default ash shell. Now the Diskstation again responds to the manual shutdown and restart commands. After reindexing, Audiostation is back to working status. However, the sleep behavior is still broken, and now the next thing for me to try is downgrading the DSM software (currently in version 4.3 beta) back to perhaps version 4.2. The DSM front-end software warns before doing DSM updates that the DSM software cannot be rolled back, but thanks to the enterprising user community, there are ways to do it.

I'll be working on downgrading the software. I will probably lose the GUI-based task scheduler since that was part of the 4.3 beta, but I may still be able to access crontab in ash, and create the scheduled job that way. And having the Diskstation off for a bit isn't so bad; it'll be nice not to worry about those Internet pinheads that keep probing the machine. In the meantime I'll continue to use my laptop for some of my Linux needs.

The Linux Adventure Part 4: Putting the machine to work

All right. Moving the vpnc command to the main script has indeed worked, though I do need to quickly get onto obfuscating the password so it doesn't sit visible in the config file.

Once I did that, I confirmed it ran ok, connecting to the VPN, running the database copy, and then disconnecting from the VPN. All ran well.

To automate, I utilized a new feature of the Synology Diskstation's DSM software, version 4.2. There's now an integrated Task Scheduler. Interestingly, it doesn't appear to be a simple GUI interface to crontab. Instead it has some proprietary commands and probably data structures. It's also poorly documented. The help file explains what the various parts of the Task Scheduler screen are but doesn't have Synology's usually good tutorials on operating the feature. DSM 4.2 is in beta I believe, so perhaps the documentation will improve when the final product is out.

In any event, it's fairly self-explanatory to set the custom user job up. You give it a name and enter the command exactly as you would enter it at the command line interface, and then set a frequency. You can also run on demand at any time from the GUI.

I don't like that there's no record of tasks in the logs when you run them. Perhaps that will also be improved in the final release. But as it stands, when you run a task it doesn't look like anything is happening and none of the task's output is displayed anywhere.

You have to run this line from the terminal session and it will give you information on the scheduled tasks and what their last run status was:

/tmp/synoschedtask --get

Aside from the /tmp directory being a weird place to put a main feature, this line will return a list of the scheduled tasks and their configurations, along with a last run time and status.

At this point I have a boat load of error handling and feedback features to add but can now start taking advantage of automation to have this thing run automatically every day.

Synology Diskstation Security Tips

If you have a Synology Diskstation and you're only using it for local hosting of files on your local lan or wifi network, this may not be as critical.

IP Auto Block

But if you've opened up the unit to accept connectivity from the outside world via the Internet you would be well advised to regularly log in as the admin and review the system logs. I didn't notice much activity at first when I allowed remote access to the Diskstation, but recently I started seeing regular attempts to login from unknown sources. Several each day like these:

It's really unnerving to see that. I'm not naïve but my server is literally a tiny speck of nothing in the universe and has nothing of value to anyone but me. Yet the universe must be filled with pinheads who have nothing better to do than try and hack random IP addresses. Actually, it's highly likely the login attempts aren't being done by a human but by a bot that's already found its way onto other servers and is just probing.

The Diskstations have a feature called IP Auto Block. You can find it in the Control Panel of the Synology server admin tool. Turning it on will make the Diskstation automatically block any IP communication once it has witnessed five failed login attempts. I highly recommend you turn this on. Since turning it on my Diskstation has regularly been blocking several addresses each week. Ye gods, is there no honor left in this world?

Antivirus Essential

Also, Synology offers a utility called Antivirus Essential that you'll find in the Package Center under Security. It's a free anti-virus tool. So far I haven't had issues with viruses getting on the Diskstation but I'd also recommend installing this package to help ward off some potential problems.

(Update Jan 2015) Maintain System Updates

People complain all the time about Microsoft's regular patching. Unfortunately, given the propensity for people with too much time on their hands to be constantly hacking systems, I think Windows users must keep abreast of updates (even if some of the updates are of questionable quality!). The same is true for the Synology gear. I recently posted about issues [burningends.blogspot.com] I had with the Diskstation updates, and it turned out my device might have been compromised by some bitcoin hack. So, although updates have sometimes wreaked havoc with my Diskstation's sleep function, the reality is that part of my security responsibility is to keep up with updates.

The Linux Adventure Part 3: A Path to Success

Installing MySQL on the Diskstation

Continuing from the last installment: there are several items I listed on the to-do list that actually were already done. I didn't need to install MySQL on the Diskstation as it is already on there and enabled either by default or when I first configured it at setup.

Installing Bash on Ash

From the last installment, I had already installed the ipkg package management software. The command ipkg install bash will pull down the bash shell and install it on the Diskstation.

Test Manual Run of Existing Script

In preparation for the script test I first ran /opt/sbin/vpnc . This opened a connection to my client's VPN.

I then copied my existing bash script over to the Diskstation. But ran into several issues when trying to execute it. I had to relearn simple things like making sure the path was a part of calling a script unless it was already in the working directory (pwd shows current working directory).

I also had to learn new confusing things about Linux since a few confusing things weren't enough.

I made bash the initial shell by adding a few lines to the root profile to launch the bash shell. You can find these on the Synology Google hits for "Synology bash" but even after now getting the bash prompt when signing in (bash-3.2#)I still had to use special syntax to run the bash scripts through the bash shell.

In other words, I installed the bash shell, got it to be the default when logging in, but still the scripts will try to run under the Diskstation's native ash shell [Wikipedia.org]. Got that? Thank you so much Linux. So upon initial attempts to run dbbackup.sh I would get beautifully intuitive syntax error messages like this:

Line 15: syntax error: unexpected "("

Using parenthesis when defining variables indicates you are using an array to store the value. I don't know many languages that don't support parenthesis or arrays, but because ash is designed to be super lightweight it doesn't quite have all the same functionality as bash. Here is a thread [busybox.net] explaining that ash doesn't support arrays. You can take a look at the documentation for ash [in-ulm.de]. It's pretty amazing considering its tiny footprint.

But rather than look for ways to reinterpret the script in ways compatible for ash, I found some other people had similar problems. Ipkg installs bash under /opt/bin and you have to execute the bash script almost like a parameter to the bash command. Like so (where the working directory is the location of the script):

/opt/bin/bash ./dbbackup.sh

This now made the Diskstation try to execute the script, but I was not home free yet. There were some calls in the script to commands such as mysqldump and mysql, and both of these also needed to be prefaced with their directory locations. So I updated the script by adding /usr/syno/mysql/bin to the front of those command calls.

Now the script started to run, but I got one last error before it would complete the backup. The Diskstation told me it couldn't find the destination database on MySQL localhost where I would be copying the source database to. So I launched phpMyAdmin (a Diskstation natively supported package) and connected to the localhost server, then simply created an empty database for the backup.

It Works! I've Finally done Something that Works!  

    - Doc Brown (Back to the Future)

Now I ran the script and it worked just as it had on my laptop, reaching out via the VPN to the client database, grabbing the tables it needed, and copying them to the Diskstation. Awesome. Additional tips:

• The Diskstation's MySQL installation is found at /usr/syno/MySQL/bin
• The Diskstation's MySQL localhost server stores databases at /volume1/@database/mysql 

Now I have opened up some options and benefits for this task. I have:

• Multiple ways to back up my client's database
• mobile notebook
• desktop at home
• remotely connecting to the Diskstation
• Increased my knowledge of the Diskstation and my love for Linux's idiosyncrasies
• Gained a backup of the backup (the Diskstation utilizes mirroring)

There are a few things left to do before the process is fully automated. I need to add the vpnc commands to the script if possible so I don't have to do those manually. Also, add error handling to that if something goes wrong the script will correctly tidy up after itself, closing the vpnc connection and exiting the script.

The Linux Adventure Part 2: How to get VPNC working on a Synology DiskStation 212

This is more of a personal note post to help me remember how the hell I navigated all this arcane Linux shit before I forget it.

In the future, all storage will be a personal NAS device

Have you used a Network Attached Storage device yet? It's basically a personal server in the internet cloud. Except that it sits on your desk at home and you can totally control the whole thing. You can buy ones that will do disk mirroring so you always have a backup, and these days you can do all kinds of great shit from them like stream music and movies and photos. People also run security systems off them and host web sites. If you're willing to tinker a bit with whatever OS is running the thing you can do some really cool shit.

What I'm doing right now is trying to get my Synology Diskstation 212 to automate a task I've been doing by hand for a while. I have a Linux laptop that has MySQL and vpnc installed. I connect to a client's VPN and then run MySQL commands to do some database backups for them. I've got the scripts down to where they're pretty reliable and I just have to handle the connecting and the launching of the script, which then runs through a list of the tables it wants and takes care of copying the data. It only takes me a few minutes but having to boot the laptop daily gets old. You Linux guys can laugh at me, but sometimes fiddling with Linux gets old too.

You want to know why people still use Windows? This is why. All this crap below should not have taken me the time it did. The arcane error messages made me think I was still using early versions of Oracle. Then there were the labyrinthine paths to find various executables and packages and different kinds of packages and worries about whether commands and directories in my version of Linux are the same as the one in the documents I found...yes, this is why Windows with all its faults is still useful. Being an explorer is fun when I'm doing something leisurely, not when I'm just trying to get some damn work done. And yes, I do appreciate that Linux doesn't cost anything, so all the "fun" I had was for free!

Anyway, I found out that the Synology Diskstation runs a lightweight version of Linux but that it should be able to handle what I'm needing. It took me a while to finally get vpnc to work, but here's what I had to do:

1. Install a tool that lets you communicate with the Diskstation.
1. I was able to navigate the Diskstation's files with the great free tool WinSCP [winscp.net]. Its excellent GUI made moving around easy for Windows users. However, when it comes to the command line, WinSCP's built-in terminal doesn't handle interactive sessions, and is mainly just for submitting single commands.
2. To augment WinSCP you can install Putty and have WinSCP point to it. I happened to have another great free tool called Tunnelier from Bitvise [bitvise.com] installed so I used that.
2. Reference this guide [Synology] on enabling modifications and the command line interface on a Synology station.
3. In the same guide linked above you can also find information on bootstrapping the Diskstation.
1. You will have to do some legwork to figure out what processor your Diskstation is using and install some software.
2. Bootstrapping allows you to run a package management system [wikipedia] called ipkg [wikipedia].
3. Packages are software modules ipkg loads on the Diskstation. There is one for vpnc, which is what I ended up using.
4. This link [dd-wrt.com] and this one [codrspace.com] are to some quick ipkg tutorials that helped me update the ipkg list of packages and find and install vpnc.
1. Originally when I tried to install vpnc via ipkg install vpnc, I received an error "Cannot satisfy the following dependencies for vpnc: kernel-module-tun".
2. To resolve the above I used options on the ipkg command to bypass dependencies and force the install: ipkg -force-depends install vpnc
5. I confirmed vpnc installed by running ipkg list_installed
6. I tried running the command vpnc and got another error: vpnc: can't open /dev/net/tun, check that it is either device char 10 200 or (with DevFS) a symlink to ../misc/net/tun (not misc/net/tun): No such device
1. To resolve the above, I found that I had to install a kernel module called tun using this command: insmod /lib/modules/tun.ko. See also this.
2. Verify the tunneling module is installed with this command: lsmod | grep tun
3. Ok, so you might ask, why didn't I install tun first, then put vpnc on? I did try that many times on the journey but either I didn't do it right or Linux just has a sick sense of humor. I'm betting on the latter.
7. Finally, I was able to then go to /opt/etc/vpnc and add a .conf file with my vpn settings.
8. Now when I tried vpnc it found the vpn and I was able to enter credentials and connect.
9. Then I yelled "Fuck yeah Linux is great after all!" (Sorry, I have been chasing this issue for months and was happy to finally get it).

Here's what I am going to do next:

• I have to get MySQL up and running
• Get the bash shell going on the Diskstation
• Test a manual run of my existing scripts
• Further improve the scripts to automate the vpn connection
• Set up some kind of chron job to schedule the script
• See if I can get the Diskstation to email from the script
• Further enhance the script by having the daily job email a status

Corporate America You're Doing it Wrong: Part 3

Employees

Next up, let's talk a little about your employees. You know those people right? They're the ones killing you with payroll costs that you're trying to replace with robots.

If you believe Drucker's correction that there is only one profit center (the clients) then you and your employees are all about delivering your product or service to the client.

We're not Standing at an Assembly Line Anymore

A lot of managers think, rather selfishly, that all that matters is that the manager make the deadline and that the employees are automatons that simply are there for the paycheck and really don't have human aspirations to learn or advance. The manager is really only worried about one thing and that is to uphold the metrics that his bosses judge him by. This is usually, get X task done by Y date. So everything is done to make a date even things like cutting corners or knowingly producing less than a quality product.

I understand this is the way the world works. But if management and executive management is not going to care about anything else but the lowest common denominator in the world of work, then you don't get to whine when I call it out as selfish, unprofessional and inefficient. 

It's already been documented elsewhere that a happy employee is a productive one. It's also true that oppressive tactics can yield productivity gains, although I don't know many people that would want to work for Nazis cracking whips, and thus those managers will always have to deal with the overhead of turnover (as long as there is a competitive free market that employees can navigate).

Your Job is More than Looking at a Gantt Chart

But how do you fix the employees in the product-client-employee triangle? You start by looking in the mirror instead of pointing a finger. You're management. Employees will look to you for direction. Management should be about more than dates. It should also be about leadership, but the bulk of those in management roles are falling short here.

Not that I don't appreciate the challenges of management. There is a huge one as we deal with the integrating global economy. Outsourcing, cost-cutting and frantic desperation aren't totally inconceivable when faced with trying to be competitive with the Indians and the Chinese. But I can't help but feel that I'm better off now as an IT consultant rather than an IT employee. I'm doing the same work but actually treated better. That's a horrible thing. Your employees should feel proud to be a part of your company and feel that there's always something more they can do for the customer and are excited to do so. That's not the case, especially in IT, as I've discussed in posts past.

I guess there's no easy solution to the employee puzzle. But I believe that if management took a more encompassing approach to its job, it would go far beyond dates. Treat employees like, well, human beings. They will take direction and they will do your work, but if you want more than just a robot you have to develop it. Some employees are really good about this, and they will demand various conditions and sometimes when they're proven as effective workers you meet some of their demands. There's a whole contingent of them that are also not quite as vocal but still appreciate the concept of continued skills development and growth opportunities. And this is the humble group that will often go the extra mile when you need it and not complain too much, at first. Ultimately though they would like to be included in the plans, to have some sort of career path.

What Employees Want: Money is Nice, but so is a Plan 

But wait, how do we make it so that everyone can advance if there's ultimately only one CEO spot available at time? Well, I think people are smart enough to recognize everyone can't all be in the limited spots, but there's a myriad of roles people can play through different layers and columns of the organization. Be open to the idea of cross-training and gaining the benefits of creating a workforce that is less narrow; where someone can understand HR, accounting, operations, and IT. There are already proven benefits when a developer has a good grasp of the business; think of other synergies that could be gained if they also understood data pathways of HR and operations and perhaps streamline some of that data and utilize it in multiple ways. At some point you'd have a heck of an analyst/programmer/consultant that would do great things for the company, ultimately to more efficiently serve the customer. The point is that continuous intellectual stimulation is a huge part of motivating people, especially knowledge workers. Promotion is nice, when it's warranted (and I'm not sure it's always warranted when it happens) but if you can't give a promotion, at least you can fight stagnation.

Quality is the Bulwark Against Production Support

The other thing you can do is something I've touched on before. Push for quality, not just dates. In IT if you have lots of production support, you might think it's because the users are stupid or computers are unreliable. Both of those might be true, but the heavy production support is actually an indicator that you're paying for the sins of the past. All those corners you cut and sloppy crap you pushed into production, all those times you gave the business analysis phase the short shrift, all those times you cut training, or barked at developers to "do it fast rather than right" have come to haunt you now with systems that are counter-intuitive, brittle and cantankerous. The sad thing is that not only you are paying for the sins of the past, but it's likely these were someone else's sins. And we'll continue to make these same mistakes again and again in a problem endemic to the IT industry because of the decision-consequence gap. It's a gap that can only be closed by a managerial intervention that's probably never going to happen.

Stop Thinking in Binary

We also have this damaging philosophy in IT that there are two kinds of people, project workers and support workers, and that they are to be separated by a steel wall. That is a terribly narrow-minded and broken view that only leads to more production support. It contributes to the decision-consequence gap because the gap isn't just about management; developers too tend to view themselves in the projects vs support light. The "project" side tends to get more respect because those projects involve creating new code. But having been on both sides [programmers.stackexchange.com, at least until someone deletes the thread], I believe you become a better developer when you are forced to see how users are receiving your product. That means rotating among both the project and support sides. One flaw of only being on the project side is that you don't learn from your mistakes because, thanks to the decision-consequence gap, you don't have to see them. That flaw leads to another: you make the same mistakes repeatedly, and you're constantly looking to try new technologies rather than perfect every day techniques [burningends.blogspot.com]. Again, developers left to their own devices will continue to game the system and stay on the irresponsible "I can have sex and never worry about raising the child" path.

Management that is smarter than this will find ways to change this; and not necessarily with a whip. Find ways to incentivize creation of more rounded developer and a fairer distribution of work types.

You're Doing It Wrong

Of course I'm not holding my breath here waiting for a great manager to fall from the sky like Thor and fix problems with his magical PMBOK hammer. But if the industry won't change, then I'll stay a consultant instead; playing the game instead of letting it play me. I'll help cover for your past sins, but it's going to cost you. You can outsource, but that might only make it worse; you'd better be gone before your boss finds out it's not really cheaper to do it that way. I'm hope I'm not being unreasonable here and to be fair, employees have to come halfway too and accept that work is sometimes about doing stuff that isn't always fun. Everyone should share in the work.

At the same time, I have a foolish hope that someday I'll find a company that gets IT and how to manage people. I'm more than happy to come back to the fold if management can fix the employee situation.

The Burning Ends Annual Memorial Day Post of 2013

It's become sort of a tradition that I post on Memorial Day. So here we are in 2013 and the cynical bastard is at it again.

American Sniper

I've just finished reading American Sniper [Amazon.com] by Chris Kyle. It's Kyle's memoir about his life as a US Navy SEAL. As SEAL biographies go, it about par for the course with plenty of action and anecdotes about the grueling training program and missions in Iraq. It's told with the help of professional writers and they do a good job of capturing Kyle's personality. Contrary to what some grunts probably believe about any of the special forces teams and their PR engines, Kyle is very reverent to other services, paying respects and recognizing they were truly a bigger part of the conflict than a few sniper teams. He is repeatedly humble about his achievements always felt like he was there to help others.

Kyle would have probably been proud to refer to himself as a Texas redneck, but there is something admirable about the no-nonsense common sense and honesty in this country boy's words. Kyle isn't shy about sharing his opinions and in this day of political correctness, I appreciate that even if I didn't agree with him on everything. Every day, and on Memorial Day especially, we honor our veterans by respecting free speech and free thought.

As I read the book, I found it echoed many of the themes I cover here at The Burning Ends. Kyle ranted about how administrative overhead could interfere with getting the actual job done. He dealt with a SOX-like regulation of his own where he had to document each shooting and have witnesses, and how sometimes those rules of engagement meant he had to refrain from sniping a questionable contact that might have gone on to harm other people. But Kyle wasn't stupid, he understood why those rules were there and even conceded that the recordkeeping had a benefit of helping him track his work.

The Decision-Consequence Gap goes to War

The book also reminded me often of my last post, The Decision-Consequence Gap. Careerist officer types more concerned with looking good on paper would withhold Kyle and his teams from working the dangerous zones where they could be more effective at helping troops. By being able to report low or no casualties the officer would look good. This is analogous to IT leaders that do not empower their teams to understand, master, and improve processes, hardware and software that will ultimately improve the business. In sports, that approach is often referred to as "playing not to lose" rather than "playing to win". I'll give the officers in question a bit more slack than Kyle does though, as human life is a harder chip to play than user comfort and efficiency or a little more profit.

The Cost

But there is always a cost for managerial complacency and selfishness. In Iraq, was the cost that more line troops lost lives without the overwatch duties Kyle's sniper teams could have provided?

That's an important point about cost. I wrote earlier about free speech and free thought and I do believe that how we carry ourselves and cherish the benefits of the American way is a measure of respect for veterans (recall Saving Private Ryan's [Amazon.com] admonition at the end of the film: people died for you to have this life. Make it worth it.). But Memorial Day is really about the cost, not the benefit.

So to every veteran, thank you. Again. We can't say it enough.

The Difference

And for you idiots that think all things military are bad, well, too bad. I know, there are Iraqi and Afghan citizens that call US soldiers the terrorists, and if someone lost a child to collateral damage from a drone-fired Hellfire missile, I sympathize for a pain that I wish no one would ever have to endure. But that goes for the children of 911 too, and I don't take kindly to being called a terrorist because there is a difference between us even with the mistakes we've made.

Kyle mentions Mike Monsoor, a SEAL that dove on a grenade to save his fellow teammates. It's not a one-time thing. There's also the story of Specialist Ross McGinnis who did a similar thing in Iraq to save others. THAT is the difference. For all our faults, Americans at large revere life. You don't hear stories about insurgents doing such things, they're too busy blowing up civilians. America reveres life enough to spend significantly more on soldier training and equipment. To spend significantly more on search and rescue operations. That is the difference; the enemy our military has been fighting for the last decade would do no such things even if they could. Kyle made no bones about calling the enemy plain evil, and as crude a description as it is, it is also the truth.

Chris Kyle, Rest in Peace

It is on a somber note that I close this post. I purchased the American Sniper as an eBook in early 2013. I later found that recently Kyle passed away [ABC News]. It was not from war, but from a tragic incident in which Kyle was trying to help another veteran. It's a terribly sad thing to happen after what Kyle endured in dealing with war and coping with the challenges of returning to civilian life and focusing on his family. But he tried to help others because he cared about life. And that's the difference that we should take from it.

The Decision-Consequence Gap

Last time, I posted about an empowerment ratio, which can be an indicator of how much a company trusts its people to effect improvements in an organization's products and processes. Another concept I see that leads to difficulties for people is the decision-consequence gap. This term refers to the distance between a decision-maker and the consequences of his or her decisions.

Narrow Gap

When the gap is narrow, the decision-maker is cognizant of how his or her strategy is working; he knows whether unforeseen factors have made executing the plan more challenging and is aware of how subordinates are carrying out the tactics in meeting a plan's goals.

Wide Gap

When the gap is wide, the decision-maker lives in a vacuum and tends to take credit for success and pass blame for failures often without understanding how results are achieved or missed.

Gap Effects

This gap is a big factor in leadership, and most leaders seem to misunderstand or not care about it. It's not just in IT either, and dysfunction is visible within this concept across top leadership in all industries. And wide decision-consequence gaps often lead to poor empowerment ratios.

It's a harsh recipe especially in software development too, when functional owners and project managers are given strategic and tactical control over non-functional elements. Although not all will make mistakes here, under the duress of time constraints there is often a push to favor speed over quality. Non-functional concerns will be given little priority despite the advice of the developers presiding over the construction.

If you've ever supported poorly designed and poorly implemented software, then you are quite familiar with this already. The people that decide a team will cut corners to make a date are not the same people who have to support it and work the production support load.

The project manager's common refrain here is that if developers are left to their own devices, the product will never ship. I agree that developers who care and are perfectionists may draw the development cycle longer than is comfortable for the business. And it is often true too that developers, in the interest of pleasing management, will comply in promoting a shoddy release to make a date.

Cheap Fix for a Cheap Methodology

But what I hope for in a better and more mature IT world is not necessarily perfection. It is that we collectively strive for something better than the C-minus level software most corporations deal with now. Can we maybe compromise and get to a B grade? The difference between a C-minus and a B would be perhaps that:

• Some routines feature hard-coding, but that routine is properly isolated in a method that can be easily tested and refactored.
• We put more eyeballs on our database structure so that common pitfalls don't make it into production and compromise future enhancement and refactoring efforts.
• We not short the concepts of testing and training, so that fewer "diaper change" type of requests get turned into weekly flurries of productivity-and-morale-sapping help requests.
• User interfaces aren't so awful as to make our users cringe when they have to work with our systems, or so confounding and unintuitive that it leads to more support calls or worse, bad data getting into the system, which leads to more data fixes.

None of the above suggestions are terribly expensive and a lot of it is good preventative maintenance and best practice. Yes, there will be a time cost somewhere. But you can't read a magazine article about agile and then decree your teams will start using it and expect immediate benefits if you haven't empowered them to write software that is properly objectified and can be tested or maintained without massive effort. And you won't understand any of these concerns if you suffer from a sizable decision-consequence gap.

Are You Managing a Project for You or for Your Company?

Let me ask you managers out there a question. Would you tell a developer, "Oh, just do it sloppy so we can make the date?" Actually, don't answer that question, I already have experience showing the answer is Yes. Let me ask a different question instead. Would you allow the contractor building your home to cut corners to finish faster? I'm betting the answer is No, and do you know why? Because YOU have to live in that house. YOU will be the one dealing with the leaks or creaks or breakage. When it comes to YOUR home, you pay heed to the decision-consequence gap. I guess I missed the part in the PMBOK calling selfishness a PM trait.

The Empowerment Ratio

It's really hard to be a grunt in IT. It's even hard to be a manager or leader in IT. At least, it is if you care and really want to do more for your users and clients and your teams. As this industry advances you'd think we'd collectively improve based on lessons learned applied institutionally. It doesn't really happen like that though.

Yes, we do see some improvements. Since I started working in IT, several things are better than they've ever been: user interfaces, databases, source control, languages, tools, and some processes. And of course the hardware is amazing now and just getting faster and smaller and sometimes practically invisible: virtualization is a very useful tool. But even with all those trappings of the software development space getting better, the people parts of the industry continue to exhibit tremendous and appalling flaws contrary to the ideals of such intellectual pursuits.

One of the things that has most frustrated me, and I'm betting it also dogs many of my industry brethren, is a concept I call the power-responsibility ratio, or the empowerment ratio. I mention it in this post because it's a concept that will be used in a future post. It's a problem not just in IT, but in all industries. The power-responsibility ratio is the rating of power a person has to effect change relative to the amount of responsibility they have over a particular process or domain.

Example: the typical IT support team has responsibility for an application's operation. This includes being the first point of contact for anything that can go wrong: network infrastructure, hardware, software defects, user error, process error, data error, interfaces, cross-system discrepancies and reconciliation, bug fixes, new features, project management, data fixes, source control, documentation, and probably a dozen other things I've forgotten to mention.

Now that's a lot of responsibility. But if any one of those factors in the system ecology isn't efficient, it can affect the total productivity of the team. So if there's a problem, the team just changes it right? Not exactly. IT grunts are increasingly beholden to productivity sapping cruft and dense processes invented by people that don't have to suffer the consequences of these process deficiencies:

• Want to change to faster hardware? Get permission from someone else.
• Want to move to a more efficient document system? Get permission from someone else.
• Is there a particular user that's proving intransigent about using the system correctly and doesn't want to learn, but continues to create problems? Good luck replacing him.
• Someone on your own team proving unwilling to follow the best practices everyone else is? Didn't you hear corporations don't fire people that are incompetent anymore, they only fire people that are politically incorrect.
• Want to upgrade the software to the latest version of a framework? No can do, the project manager thinks that effort is wasted because it's not part of the business needs.
• Think you found a way to reduce the overhead in those data fixes that are killing the team's capacity for more strategic work? After filling out forms in triplicate, you find out it was denied because the powers that be just don't feel it's that important; powers of course who don't have to suffer with processing the repeated data fixes.

I'm being a little facetious here because not all management says "no" to everything, just most. But the point I'm making here is not about the having to ask, though that's part of the inefficiency of it all. It's more that IT workers may be the least empowered people in modern organizations. Accounting has a lot of power because it manages finances and in the case of some controllers, accounting actually has to take managerial responsibility for some regions and therefore must understand the business. IT is similar in that it learns about the business and the data that flows through a company, but do IT people have the power of accountants? Not really. R&D, they're usually getting to do things that help the company and as they're seen as a part of the company's core business and its future, so they are deservedly respected and have some influence. HR certainly seems to have garnered a big part in organizations now. Sales and marketing? Please. But IT is expected to be on-call 24x7 and can't get approval for even simple changes.

That's an incredibly frustrating position to be in: You hold significant responsibility over a domain but have very little power to improve it.

What's really sad about this is that the solution has long been documented. In the military, effective leaders know they have to trust the boots on the ground to be able to manage their unique situations, and those boots need to be able to make changes efficiently, or at least as efficiently as possible in the behemoth military. [Note that I'm not saying the military is all good here, it's certainly had its share of gaffes, key leaders are often clueless, and it is hardly a universal exemplar of a properly applied empowerment ratio. But selected groups, like the Special Forces, have figured it out.] And in many management books the concept of empowerment has been a popular one. It's not like this is some secret that can only be implemented with excruciating pain or arcane spell casting. The solution is a simple thing, a single word: trust.

A lot of organizations have a long way to go before they will be able to take advantage of truly superior IT.

Corporate America, You're Doing it Wrong: Part 2

So how do you fix it? Last post, there were three things that have gotten pooched: products, employees, and clients.

Let's fix the most important part first.

Clients

A lot of Corporate American heads like to say, "I'm a profit center along with my vaunted staff of silver-tongued gilded warrior salesman. Even Peter Drucker said so. The rest of you are shit, er, I mean, cost centers."

According to a Google search [full disclosure: blogspot is owned by Google], Drucker indeed coined the phrases "cost center" and "profit center." And yes, he used the terms to split a company's resources into two camps; the revenue generators and...the rest. And his book was probably on the corner of a lot of coffee tables in a lot of executive suites, so of course everyone believed it. It wasn't totally ridiculous and Drucker presented it professionally, so there you have it.

But years after that, Drucker himself relented from this divisive concept, calling it the worst mistake he'd made. He offered a correction and identified that there is really only one profit center, that being the paying customer. Everyone else, from the CEO down, is a cost center. Painful though it may be for a lot of fat heads to accept, this new line of thought from Drucker makes more sense and is much more truthful about the whole business equation. As I said in an earlier post, if an executive thinks IT workers can easily be replaced by remote resources in India, or better yet, robots, then the other side of that is true too. An Indian PhD in business ought to be a lot less expensive than the typical American executive (and possibly a lot smarter).

Every bit of effort in a company by all its resources should be about ultimately serving the customer. Not the board. Not the executives. Not the rest of the employees.

Make great product; not an ok one, not a good one, make a great one. Then stand behind it and fix it, but make a product good enough that your customer support and production support costs are low because it works right the first time. Charge a fair price, adjusting for competition. I know, the competition includes companies from China that don't respect intellectual property so they copy your product and use cheap labor to build clones for cheap...and those damned customers will buy them. First, fix that attitude because the customers are doing exactly what you'd do if someone gave you two comparable products and had a lower price on one of them. Second, compete the best way you can: with innovation. They can copy your product but they can't quickly copy your initiative, your customer knowledge, and your customer relationship. Keep innovating and they will always be playing catch-up. How to innovate? More on that when we fix your employees.

But putting the customer first isn't how it is. Maybe in a few companies. Maybe in the past it was more like that. But not in the contemporary world. Stockholders only care about returns, and that causes boards to put pressure on getting results faster and faster. This in turn leaves company leaders to resort to putting the most obvious metric at the front of all decisions, project evaluations, and employee evaluations: the date. Get it done by a date. If it's imperfect, thats ok, we'll fix it later. And thus is endorsed a philosophy of not accepting technical debt as a necessary evil but as a strategic option. It comes from the top and it works its way into all ranks and pretty soon delivering broken shit or half-assed shit is ok.

Of course, not all companies have the same leeway here. Compromised safety standards will injure people and ultimately sales. Established companies like Toyota and BP would therefore never do such corner-cutting for the sake of short-term sales.

Looking at these pressures, you have to wonder if perhaps they're at least part of the reason Dell has chosen to revert from being a public company back to a private one. Dell offers another example here; at one time the company tried to outsource all its technical support to India, but backlash from customers caused Dell to retain US technicians for corporate customers. In the latter case, Dell recognized it had to keep its largest revenue generators satisfied. Of course the rest of the little customers got screwed, but at least some of this corroborates the Drucker correction. But it remains to be seen what Dell will do with this recovered initiative.

"The customer comes first" is an old tenet of business. For a lot of companies it's also a forgotten one.

You're doing it wrong.

Corporate America, You're Doing it Wrong

I know, you're looking at the title of the post and rolling your eyes at me being a cynical bastard again. Well, you can tell yourself that if you want. But you are doing it wrong...it's why I became a mercenary. I figured out the game and now I play it instead of letting it play me.

The More Things Change the More they Stay the Same
Well, not really. I'm still being exploited for someone else's profit, but it's better than it used to be. I've given up dreams of institutional achievement and only two constituents really matter anymore: my family and my clients. The rest of you are just infrastructure, trying to sell me on the golden rung at the top of a ladder that I know is made of pyrite.

Your Job is to take Care of Products, Employees, and Clients
Oh there's some gold there all right, but you have to sell your soul to get it, not by making a deal with the devil, but by complacently doing the same c-level executive horseshit that most of them do. You might not even be doing it intentionally but you're destroying dreams by the thousands by succumbing to the typical corporate insensibilities. Your constituent is just one entity: you. You will blame the stockholders and board of directors for making you do stupid things like purposely putting your product quality at risk, your employees in the crapper, and your clients last in the food chain. That excuse doesn't wash; if you are going to blame the stockholders and the board and then take profits, you don't understand leadership. And that's your primary job. You don't build widgets or beat the pavement anymore; you're supposed to be a strategic thinker that builds the company's future with keen decisions, reliable relationships, a vision for the future, and respect for the past. But nope. It's all about you, and it's gutless.

You're doing it wrong.