Why Defined Coding/Operational Standards Matter in Modern Software Engineering Environments

After a brief stint in the hellish 1990's world of programming in Perl for a half-billion dollar company for a few months before the realisation that it was indeed a living code base designed and maintained (I used the word loosely) as if it were still two decades prior, I accepted a new client in the position of Lead Software Engineer & Architect at a local company in Montgomery County, PA, USA.

Much akin to what I experienced at the aforementioned company, I found myself in a microcosmic variant of my former employer save for the fact that the system I am to replace is written in ASP Classic on an older IIS windows centric system.

These situations are not at all uncommon in the real-world, and hopefully they are diminishing at an increasing rate.  This is not me condoning the carte blanche manner of always re-writing an existing system from scratch whether than includes changing the existing infrastructure (for better or worse).  Each instance/environment must be assessed individually along with company goals, available budgeting and forecast to ascertain whether or not such an approach is viable.

Bringing this back to the core reason for this post is that a well defined coding standard didn't exist for either environment, not to mention a revolving door of people assigned and/or employed which had a hand in touch either system.  Both environments differed in that one was a fairly modern agile approach with daily scrums, open communal work spaces and solid department leads for projects and that the other was very much waterfall design based, separated (officed) individuals with little communication and a lack of leadership in terms of working environment understanding (though a solid understanding of the business logic and customer relations carried this disconnect for without it, all would have faltered).   Neither environment possessed nor utilized any semblance of coding standards.

What kind of standards are we talking about here.  The whole gamut is my best response.  How are function names derived, how are attributes accessed, how are constants, classes, variables, methods, etc. named (verbs for methods, ambiguous/nonsensically, etc), typed (cAmElCaSe, alllowercasewithoutbreaks, properly_divided_with_underscores, ALLUPERCASE, MixTUREoFMeans, wthtvwls, etc.).  No set rules for braces or brackets (K&R vs same line), indentations (2, 4, tabs, spaces?).  Lengths of lines, length of method calls before being broken down.  Use of native data constructs.  Rampant string concatenation and massive use of type casting instead of a well kept control of expected content.  Complete lack of logging and little if any use of exception handling (where available).  Poor hiring practices which allowed for a single point of decision to be made either based on a inconsequential question about the language du jour at a given company or because the hiring manager lacks any breadth regarding software engineering both in practice and theory and does not do group interviews with potentials as a precautionary measure.  Lack of enforcement (or implementation at all) of code versioning systems/controls, proper QA practices, etc.

All of those situations were (and still are) prevalent in the existing systems, as well as many more which I will not get into now (solely because there isn't enough brain bleach to forget that they still exist.)   What I can do is point out that because of the lack of all of the above, both companies (while still in existence):

- Find themselves over the years running into longer lead times for changes which would otherwise be quick and simple.
- Experience bug fixes which require lead times of weeks due to lack of documentation, version control, communication amongst peers and of course proper reviews up front before production pushes.  - Struggle to keep current due to lack of faith in the infrastructure of existing systems that any new code would cause massive breakage and production down time.

This is never the way to run a company and after roughly two decades in the field professionally, it is disheartening to have to endure.  There is a light at the end of the tunnel.  The former company mentioned has a few good remaining people that provide a glimmer of hope for change in the right direction if they can convince the powers that be (ego and all) through all the crimson coloured tape that  these changes are necessary which could lead to the arduous process of modernizing.  The latter company referenced has already made that investment both with full ownership backing down the entire hierarchy of the org chart, and in that case, influenced heavily by yours truly.  I see much greater things for that latter company because it has done the realistic self assessment and see that much is lacking, ultimately choosing to do something aggressive to resolve the situation with a great outlook towards future business.

Hopefully we can see more and more existing companies learn from these examples before they find their up and coming competitors who embrace a method to their madness soaring by and taking their bread and butter in the process.

Dealing with Horrible Legacy Code

I know, I know.. it has indeed been a while.  What can I say, I've been busy with work.  Even though the US is deep in a recession so bad that Microsoft and IBM are executing mass layoffs (Microsoft's ever), there are those of us; especially on the Unix side of things (I gather Linux users as well) who are swamped with new projects.  

As many readers would know, I have for the past year and a half been offering my software engineering services to an international publishing firm based out of New York with branches in Pennsylvania and Japan.  Yesterday I was in Manhattan for a meeting of introductions to the individuals with whom I would be coordinating on not one, nor two nor even three projects, but four new magazine entities.   This now brings my overall responsibility in terms of publications I  handle to a clean half dozen.  

I'm glad I'm not in the Microsoft world (for multiple reasons not to mention primarily because I refuse to work with junk and this includes any MS OS), but for the reason that with all of these layoffs, a mass amount will be MCSE's and VB/.NET programmers, architects and engineers.  This does however segue into my first point (trust me, there is a relation).  

In my most recent project, a sister publication of it actually needed some work done to their rss feeds.  Apparently this was first and foremost to fix a broken rss subscription page (where one could select which 'feeds' to follow).  This problem had been present for over a year from what I've told, and it was so bad that the page itself was throwing a PHP debug page when accessed.  This is wrong on many levels, most immediately that the debug mode was still on in the server configuration (the other errors being that they were using PHP, and allowed something to go on for over a year, broken.)  

Not wanted to reinvent the wheel as well as having a sense of right and wrong I contacted the original developers of the software and let them know that since they were the only individuals with write access to the entire group of applications involved (not to mention this was custom built by them), that it was their fault 100%, and their responsibility to fix it, and quickly at that.  As a side note, I had been informed by my client that they had made repeated requests to said developers about this problem for over the past year and were informed that it would take 'a lot of time', and that they would be billed accordingly.  Long story short, I got in the developers faces about professionalism, the fact that had they any procedures in place for regression testing when changes were made and a proper set of document and qa tests, this problem would have been squashed the moment it was introduced.  Three hours after sending my well crafted letter, I received an email back from said development house that the problem was rectified, all is working and that there was no charge.  No charge indeed, I wouldn't accept one if they tried as it was entirely on them.  

This brings me on to point number two as I could rant about the prior issue if left to my own devices.  Such heavy use of PHP in this specific application has raised some concern.  I know it was a great idea that filled a niche when it was created, but in its current iteration, it has grown to be something potentially problematic: the sucessor to visual basic.  I say this because as with VB, the purpose was to make creating software in said language possible, even 'easy' for the 'non-programmer'.  This is a simple point to remember so I won't waste additional time stating it.  Non-programmers shouldn't be programming, period.  You either are a programmer, or you aren't.  If you aren't, you have no business behind the keyboard writing code for anything that ever goes into use in a company by anyone, including yourself.  Engineering isn't a child's game and it requires discipline and continual study and exercising of one's skills.  We won't pretend to do your job, you don't pretend to do ours as you are only making things worse. I will however point out that PHP has been making non-stop strides towards a feature rich set of functionalities of its peer languages, though the battle has been uphill since it entered the ring having only the web server integration has a core plus.

This came about because while looking at the code to find the original rss problem (which was fixed without having to write any code in that monstrosity),  I became disgusted at how hackish the whole application was.  It was guilty of all of the following and more:

- ambiguous variable names

- includes of code snippets where objects should have been 

- nested structures 4+ levels deep

- very few if any comments in the code

- very little meaning in those few comments that actually pass as informative. 

- non-sensical file hierarchy for modules, includes, etc.

- hardcoding of parameters in both equality tests and branch based if statements. 

& much, much more.

It is enough to make one partake in an involuntary protein spill reflexively.  The good news of it all is that this old code base (I use the term code VERY loosely) will be redesigned and re-implemented by me and I can guarantee (because I DO guarantee my work) that it will be faster, cleaner, easier-to-use, easier-to-upgrade (including maintenance) than the existing system and will be constructed in a considerably shorter period of time. 

It is times like these where I'm glad that these other places exist.  It is hard to not look good when everyone else is so hideously bad.  I just feel bad for the poor companies and individuals who utilise these companies' services. 

recent job, nasty code built hackish bit by bit, quickly

no comments

In defence of the lone coder archetype

All too often these days we see companies and methodology evangelists touting the system du jour whether it be Extreme Programming (XP/Pair Programming), or Agile as being the "One True Path" to coding enlightenment. All other be damned is roughly what said mantra translates to in common speak. We're told to beware the engineer who works alone in a dark back room for weeks on end. We're told to such a thing is heresy against the gods of modern application development.

We're being lied to by the methodology evangelists. I'm not here to down speak any of the other methodologies currently in use for software development. I am here, however to point out the benefits of the 'lone engineer' (not to be confused with the lone gunman). Lets set a few things straight though.

A coder with little to no professional experience in the field is not the optimal individual for this kind of setup. This is more for a seasoned professional who has at least been a team lead on several projects from conception through maintenance phases. I'm not saying it isn't possible to pull it off without significant experience, but I wouldn't place bets on such an individual succeeding at the whole task at hand in said company.

This kind of environment requires a disciplined engineer. We're talking about someone who knows the questions to ask, and how to dig deep to find the true needs of the client(s). This also means that the engineer needs to stand his/her ground when it comes to setting a solid schedule for phases of the project(s). There is room for variability in terms of time frames for each phase, but that the plan must be laid out in a linear fashion. This is engineering after all, and one doesn't plan to build the rooms before the foundation is laid.

Agile methodologies are more akin (though not parallel to) modular housing construction. Individual sections or units can be build simultaneous and changed (to a certain extent) by separate sub-teams. Whereas the lone engineer mentality is more akin to traditional construction in which the design is finalised, the foundation is laid and construction occurs in layers from the bottom up with customisations being generally last.

This isn't to imply that software built in this manner cannot change or adapt in a timely manner, far from it. Seasoned engineers/developers understand that the only true constant is change. We constantly work on new ways in which to design flexibility and mutability into our systems regardless of the specifics. This however just goes back to one of my earlier points in that this requires someone who isn't wet behind their ears. You can only know and plan for the unexpected via flexible designs after having cut your teeth in real world projects, and by faltering. Everyone makes mistakes, and it is through these mistakes that we grow and better ourselves. I'm not implying that senior level engineers don't make mistakes, I'm just pointing out that their success to failure ratio is pretty strong in favour of the success side.

One of the key benefits of the lone engineer methodology is that focus can remain solid with no deviation due to intra-coder/intra-team conflicts. And while this can at teams be a negative, overall it proves itself beneficial towards the ultimate goal, a finished software package/eco-system designed from the ground up to benefit the requisitioner(s)/client(s).

The issues of coding standards still must be kept, however there is definite consistency when only one individual is involved. This is also a liability as said engineer must have strict standards, commentary and documentation from beginning to end so that in case of any situation rending the engineer on the project incapacitated in any form, another could come in and continue with minimal delay. These are issues which affect other methodologies, some more than others.

There are downsides as well though. There are no direct peers upon which one can depend in times of need or collaboration. There are no peers to assist in code review sessions, meaning that the QA of any project must be proficient at testing, and enforce strict recursion testing when any changes are made to a product in production as QA becomes that final line of defence against human error. The reality is, no usable (as in serves a real purpose) piece of software is bug free and to think otherwise is the sign of a fool. What is realistic is having a system robust to handle errors when they occur, even if that is infinitesimally infrequent.

I'm getting off track here. My point is that in this newer, more extreme era in which coding fundamentalists believe they possess the one-true-way of coding and all others be damned, we must step back and realise the grave miscalculation they are making so boldly. After all, those methodologies weren't what were utilised to get us where we are today, nor were they even thought of when some of the staples of our art were invented. This doesn't mean that the next great piece of software or language won't be produced in such a manner, as it/they may. It means that we as professionals shouldn't be so narrow minded with tunnel vision when approaching different methodologies in an attempt to find which works best for the project/company/industry at hand.

Updating Existing Perl to 'use strict' Standards

     Now that a little over a week and a half has passed at my new place of employment, I find that I understand my new environment enough to make some observations in a not-too-specific manner out of respect for my new employer.

    First let me state that all of the existing software is a product of its environment and that it all functions as it was intended.  That being said I’m able to say with a clean conscience (after that little preamble) state that the code was ... lacking.  

    I am thankful for this to some degree.  For one thing it provided an opportunity for employment at a place I enjoy with some very intelligent individuals who all seem to have their own special abilities and areas of expertise.  

    More importantly though, I’m thankful because it places me in a situation I find most mentally stimulating.  It makes me re-think an entire existing architecture and being that I have held the role of Software Architect (amongst others) for much of my professional life, it is all the more appropriate. 

    It is one thing to walk into an new environment with a clean slate in which one may design to their heart’s content, yet another wholly different situation when the software exists in a production environment of one form or another.  There are so many more facets with which to deal when the database structure and all of the depending software is tightly build upon that aforementioned code base.  

    The whole point of this rambling is that my first week and a half has passed and while I have done much more with the database redesign, class design, object handlers, etc., I have finally been able to enjoy that great feeling which comes when turning previously un ‘strict’able perl code into a fully compliant piece of code.  I might also add that I ensured the code conformed within the guidelines of Damnian Conway’s “Perl Best Practices” book, which while a little different than my own manner of laying out perl code, is wonderful none the less.

    Now that this honeymoon is over, I can move onward and upward to greater code causes to champion, and based upon the intents of the owner of the company I don’t doubt that there will be a wonderful logic requiring plethora of future projects for which I am to contend.  I only hope that others out there are as lucky in their endeavours.

My Adventures in Software Engineering Consulting

  Starting in August of 2006 I ventured into the world of independent Software Engineering contracting.  Having just completed my previous salaried tenure at a financial firm, and a small side stint for a Fashion magazine as a favour to the owner(s)., I came upon a new offering.  I was hesitant at first because it was a 1099 contract job, a realm I’d never ventured into before in my professional career, and as such the time had apparently come. 

    The position was simple, the company in question was a contractor to a large retail clothier whose name rhymes with ‘Hurlington Boat Factory’ and is located in a town in New Jersey possessing a similar sounding name.  The CEO of the contracting firm requested specifically someone with considerable experience in Python who could build a real-time return transactions processing system which would utilise both XML and Oracle 10g.  While I have a strong dislike (along with many other software engineers) for XML, the reality of getting paid decent money for coding 100% in Python was all that I needed to hear.

    I have to say that I was excited not only because of the prospect of a Python pure coding environment but that I would be working initially and then periodically out of a satellite office in Cherry Hill.  This was exciting because for the first time in my decade plus career, I was finally in a scenario where I was writing code for a company whose primary purpose was producing JAVA software applications for sale.  The office in Cherry Hill consisted primarily of the CEO, a lead developer and a secondary developer.  There were several other individuals who would at times utilise the office including the owners, project managers and other associates of various roles, but ultimately it was an office with other competent coders one of those being quite the master of Cold Fusion technologies.  

    After my first few weeks on site in the Cherry Hill office it was time to start working on-site at Hurlington’s headquarters along with the only other consultant coding in Python.  This was the beginning of a very enjoyable period in the contract due to the excitement of the project and the joy and experience of worthing with another Pythonista.   

    The details of the project are much like any other project you might encounter when modelling a new project off of a customer version produced for a specific client.  It got ugly for a while due to the multiple aspects of the project including client systems which needed to be integrated for proper functioning along with an existing array of registers distributed nationwide and a database number records approach a half a billion entries.  The normal in fighting and finger pointing existed but it was all worked out in the end with a finished product delivered and a contract satisfied 100 percent. 

    The only issues encountered which left a bad taste in my mouth were those pertaining to a clueless project manager who regardless of his claim of years of experience was apparently lacking considerably outside of his realm of Oracle, which caused unnecessary attitude due to his ignorance of coding and APIs.  There were other issues dealing primarily with suits, but I doubt that this specific issues varies much anywhere.  Suits and Engineers rarely if ever mix, let alone get along except on a faux-cordial level.   The ultimate end of the contract was due to the project portion I was responsible for coming to fruition, even though I was already working on leaving regardless.   It wouldn’t have mattered much anyway given that the main contracting company is located up north in the centre of New England and are in the process of relocating everyone to that locale, an act which I am unwilling to do.

    I do like certain aspects of software consulting, but given that there is a considerable amount of extra taxes which must be held aside, some unholy hours which must be adhered to for the purposes of meeting quite strict time lines, and the reality of being a second class citizen in the work environment (or 3rd class in my case as a sub-contractor).  It all comes to an end in two days and I go back to a normal, preferred work environment in five days when I start up in a salaried position again working for a web host writing a multitude of applications and revisions of existing software, in a nice relaxed geek environment, also in New Jersey, but this time around, I’m excited to not be a consultant.   At least for the time being.