A Job to Do

Memorial Day (Observed) - May 29, 2006

I'm going to take a quick break and just say a 'Thank you to our veterans.

I continue to be impressed with members of The Greatest Generation. Whenever I talk to them at airshows or special events, they are always humble about the role they played in WWII. In particular, they avoid being pretentious about their achievements and are modest about them. What a contrast from the younger generations. The WWII vets defeat Germany and Japan and, after one well-earned homecoming party nicknamed the Baby Boom, go quietly about their regular jobs. The younger generations watch their favorite professional sports team win a championship on TV and then spend the next year spouting trash talk.

But there's something that just sums up the WWII generation so well. It's a phrase I've heard several of them say, almost as if they'd collectively rehearsed that this would be their response to questions about their hardships.

"We had a job to do."

When visiting the USS Hornet museum in San Francisco, the veteran leading our tour spoke of the working conditions in the bowels of the aircraft carrier. Temperatures in the engine rooms could reach above 100 Fahrenheit but still the young sailors maintained their posts. "We had a job to do," was all he said in response to our amazement.

At another function I chatted with bomber crews. The bombers, despite their size and defensive armaments, where easy pickings for enemy fighters. Many were lost. Still the crews manned their stations and kept flying. "We had a job to do."

-----

Here at The Burning Ends, my inspiration comes largely from spending time in the often inoptimal working conditions of most corporate American IT shops. I named the blog after the concept of harsh overtime but none of what modern workers deal with compares to the challenges met by our veterans. Those guys on the aircraft carriers worked 12-hour shifts every day, and here I am whining about having to pull a 50-hour week.

I am justified too, because my complaints aren't directed at the the concept of overtime itself, which I often give anyway when I'm happily engrossed in a project. They're directed at inadequate or unethical leadership. And that's something worth bitching about. On that note, what a satisfying verdict the Enron trial gave us a few days ago. Now several of the rotten leaders of the 1990's boom are formally criminals. This latest news reminds that humanity isn't a lost cause, no matter how hard we try to prove otherwise. Perhaps we have not wasted the efforts of our veterans.

Each generation has plenty of problems, usually more than it can solve. But as long as humans keep pushing out a few solutions too, then we have a chance to stay in that perpetual uphill battle.

Tonight, I'll raise a glass to the veterans, and to a job well done.

The Five Things Developers Forget about in Heaven

Yes, this entry's title is a riff on Mitch Albom's book The Five People You Meet in Heaven. Largely inspired by the previous blog entry, here's a post about things typical developers ignore because they're either inexperienced, lazy, or plain stupid.

These are the kinds of things that many developers don't want to think about; they just want to do the 'fun' part, the quick hacking together of a GUI and some half-baked business rules. But that's like only doing the fun part of making a baby. Failing to uphold responsibilities as a parent and partner can yield a very imbalanced child. People wanting to only do the fun parts should get out of enterprise IT and become developers only after they get to the big workstation in the sky (I couldn't write 'cubicle in the sky' because I'm not sure there are cubicles in heaven).

And no, no one person is perfect and has the time to be a complete expert in all the below items. But someone on the project team better be thinking about these, and preferably at design time so they don't become emergency projects later in a system's life. If making software right is hard, making it right after it's already in production is harder. The alternative? Get ready to burn that candle at both ends.

Archiving
If a system is going to last a while, it's going to collect data. Queries, I/O operations, and general system performance all get slower as a system has to hash through more data. But an archiving approach for older data always seems to be an afterthought to most application developers. This is also becoming more important in an age of legal compliance (read: Sarbanes Oxley).

Concurrency and Scalability
Thanks to years like 1999, the developer ranks are filled with Visual Basic* and Excel* macro people that think applications have only one user. So they don't design to protect against dirty reads and writes. What's sad is that RDBMSs and some programming language tools have support for managing concurrency and protecting data, but these features are either ignored or worse, coded around!

* Note that I'm not necessarily against Excel or VB, both of which can be outstanding tools, but more the mentality of people that spend a few months with them and then, with no exposure to advanced computing issues, assume they can work on enterprise systems (and at fat contract rates!).

Data Modeling
This is arguably the most important item on the list. A database structure that satisfies the business needs and is technically sound is paramount to a system's business value, agility, performance, and maintainability. Too many developers think programming is all about building a GUI. But a cruddy GUI on a solid database can always be replaced. It is more difficult and expensive to repair a broken database foundation under a beautiful GUI.

As an adjunct point, SQL, and cousins like PL/SQL and T-SQL, are not to be scoffed at and seen as nuisances. They are tools like any other and when used properly can simplify the work of the client and middle tiers. They are valuable allies in addressing the challenge of application partitioning and code reuse.

Documentation
All companies suffer turnover. Sometimes it's because the company stinks. Sometimes it's because the company is fine but the market is good and someone just gets a great opportunity. Sometimes people decide for a career change or want to have kids. In the end, someone other than the original developer(s) will have to look at the code. For the love of God, document. Document in design papers. Document in code comments. Document in technical papers and user manuals. Others will benefit from it and so will you.

Yes, it's possible to document too much (see Scott Ambler's articles on agile documentation). Overbearing loads of documentation can put people to sleep and can themselves become a maintenance hassle to keep current. But a lack of documentation or the presence of poor documentation are much more common concerns.

Security
It's the age of the Internet, viruses, email fraud, wireless networks, and cheap multiple-gigabyte portable storage. But no matter what age it is, there are always humans around. Humans are evil. It's better to plan for protection at design time rather than a year after going live.

How do these guys find jobs?

I'm working right now on something that should be fairly simple in most applications: adding two columns of data to a screen. I have to update the logic that retrieves and saves the data, and also modify the GUI to accept user input and display the new columns.

The Horror
But this task is taking me two or three times the normal amount of time because the original developer didn't understand the programming tool's strengths and probably wasn't too hot on data modeling either. In exploring the table that stores this data, I found that the table has no primary key and nary an index. There's no relational integrity established through keys or triggers, and columns used to define uniqueness at a row level can have mutating values if the user wants to change the data. *Shudder*

Moreover, in the GUI the original developer didn't cater to the strengths of the programming language's construct for managing data (it's the PowerBuilder datawindow, one of the best object-relational mapping devices out there, I don't care what anyone else thinks). The constuct makes database interactions very easy if you work with it rather than against it. The original developer instead bypasses the tool and puts responsibility for updates and deletes for key columns in list box controls outside of the datawindow, manually controlling everything. This is like clubbing yourself in the knees with a sledgehammer.

He loads the list boxes with the possible values for the columns. You pick what you want and save it to the database. But when scrolling through the datawindow, he is also reloading the list box values whenever a different row gets focus, and then populating the list boxes with the values in the current row, so that as you navigate rows, the list boxes mirror the values. He could have simply kept all this in the main datawindow, because the user can already see what the values are there, but that would have been too easy.

And of course, the coup de grace is that for everything (the updates, the loads of the list box values, and deletes), instead of simply configuring the datawindow correctly and issuing a single PowerBuilder update() command, this fellow offloads everything to embedded SQL, which is a mistake in several ways. It spreads data access across several points increasing the complexity of coupling. It also creates a context switch from programming language to SQL, which can cause performance problems in some languages. It gets worse; you know the part where he is keeping the list box values in sync with the current datawindow row? He is triggering a full repopulation of the list box values with each row change, and the list boxes are populated with embedded SQL cursors, a totally needless construct in the PowerBuilder world. This means that even though you have everything you need in the main datawindow, this app is generating several SQL calls to the database with every click on a different row. This creates both network traffic and database server loads that are completely unnecessary.

Sepukku
This means that instead of having to make small modifications to the table, a stored procedure, and the GUI screen, I have to create new list boxes, plus tons of new code to manage them within the same "framework" the original developer created. What should have been about half a day of work becomes almost two days given the extra work and testing that needs to be done.

Why didn't I just rebuild the whole thing? This is where the Ghost of Project Manager Past is supposed to come in and take the blame, but it doesn't happen and even if it did, the truth remains: I am on a deadline to get this work done, and it will be actually less expensive to do it inefficiently than to fix it and make it right. I groan with every line I write, despising myself for it, but I get it done and sneak in an improvement or two. If I was the Ghost of Project Manager Present, I would change the project scope and maybe rebuild the app. But Boss of Project Manager Present would probably not approve.

Lesson Learned?
How do guys like the original developer get hired in the first place? And why aren't they held accountable for their work? Please see my earlier entry, Mind Your Contractors.