Enforcing the Software Engineer Archetype & Related Principles

It is hard to believe I've been engineering software for the past 14 years, mainly because that isn't reality. I started off like most others, as a programmer/developer and grew personally and professionally over time. As time progresses in our studies and experiences in design, development and deployment we change. This change is more oft than not, for the best.

When I was growing up, I looked towards getting a CS degree so as to become the aptly named "Computer Scientist". What I've found over the years via practical and situational based circumstances is that Computer Science isn't my primary interest. While it is true that there are pieces of the CS realm which have always garnered my attention, such as Artificial Intelligence, it doesn't ring true as a whole.

Today reminded me just how much I would like to think I have changed both in my focus, overall goals and discipline pertaining to the whole process of building software systems. Now the example I'm going to loosely reference is centered around phase one of a much larger long term project. This first phase was somewhat of a rush job as per the client due to botched (i.e. prematurely advertised) promotions for said application's 'live-date'.

I knew that taking this on such short notice would require a very strict set of time guidelines and clearly defined checkpoints and milestones if all was going to be implemented in a proper manner, one supporting a proper holistic software lifecycle approach. This of course required the initial overview of the phase being clarified, the requirements gathering phase, the initial layout with timeline estimates and expectations being put forth and finally said estimates being agreed upon with a little 'wiggle' room so as to allow for human error in the previous steps.

Well, here's what happened today which precipitated this posting. This project has a launch date of 15-Jun-09 and today being the last business day prior to that date, one could say that the end was almost upon me/us at the time. Weekends are out because as an adult and a family person with children, I value my personal and family time very highly, much higher than that of my professional work. That being said, I am indeed an experience professional and know how to accurately plan work into a give time frame clearly stating what can and cannot be realistically expected within a given time schema.

Today approximately one hour before the weekend officially arrived thus signaling the end of this phase of the project, both the designer and the overall project coordinator (not on the Software Engineer side mind you) started throwing out 'new' items for this existing phase almost completed. It is at moments such as these where the undisciplined and junior level individuals panic and ultimately sacrifice their own time for the sake of someone else's lack of professionalism by agreeing to make the changes.

I did the opposite. I made the correct move by stated clearly to all parties that we had agreed upon timelines and that they have been kept. The idea of introducing new components not part of the original design at such a late stage of the phase was outright idiocy. I pride myself in my completeness of the whole process and will not let a failure to plan on someone else's behalf negatively affect the quality work I strive so hard to ensure. Any changes need to be reviewed to ascertain what side-effects might be caused by their inclusion (especially into a more mature codebase at this point) and not to mention the quality/testing cycle which obviously wouldn't be possible due to time constraints.

What I am trying to get at is that I learned that for the sake of professionalism, quality and ethics, one must have the ability to say "No" when others fail in their planning/design. After all, the requirements gathering phase is when a competent Software Engineer brings to light questions that would hopefully coax such ideas from the requirements 'givers' if you will. It is our duty and creed to help our clients both internal and external to bring clarity to their actual needs as many times they are unsure of the specifics until discussed with others.

So I say unto aspiring Software Engineers of the future: People look to us for accountability, and part of that equation is keeping the other variables in the equation (team members, requirement providers, planners, designers and what not) accountable to the process, even if it means telling someone 'No'. Provide your reasons, and hold steadfast as these software engineering processes exist for the benefit of our projects' quality and overall success, not for the sake of being friendly or 'helping out' someone who failed to do their part in the overall planning and execution of a project and/phase thereof.

New Projects, and how to handle the juggling act that ensues.

Today marks yet another year in which I've been aboard this mortal coil as it circles our solar system's centre point. So as I sit here watching to see if the remake of 'The Day the Earth Stood Still" is a much of a car wreck as has been stated and re-stated for the past few months, or if it will end up being one of those guilty pleasures. More so on to the point of this most recent update:

We in the United States of America are currently experiencing a rather nasty economic downturn/recession due to unregulated mismanagement of the country over the past many years and as such are having (as a country) to deal with roughly one in every nine persons of working age being unemployed. Yet in all of this unfortunate turmoil as brought about by the economic calamity, I find myself inundated with more simultaneous clients and projects than ever.

While I have managed multiple projects before, it was usually with the benefit of a staff. In this instance I find myself as the sole Engineer on the job. I enjoy being the centre of a given project, especially focused projects with clear requirements as they encompass the most fluidity in terms of project and process flow beginning to end. In this case however, due to a recent spate of decisions by several key individuals in charge of my various client entities, there has been an influx of new projects, primarily new ventures and re-launched (and recently acquired) web entities.

These are all primarily fashion and lifestyle media groups and magazines and while they all have a similar bent to them, the amount of design behind the scenes differs greatly from one entity to another. I find that the sites in which there is a solid plan are to most enjoyable on which to work because this is a start, middle and end whereas projects lacking any real direction simply waste a considerable amount of time, effort and never seem to measure up to properly designed sites.

There are several major concerns here when juggling this many projects, but thankfully there are just as many solutions:

Problem #1: Keeping focused on a specific code base.

Solution #1: Thanks to the beauty of multi-windowed environments, comments and code versioning systems such as Mercurial (Hg), Subversion or Git, we can save our place, with comments and safely return to them at a later time with notes on where we left off. A bigger part of the solution here is a proper code editor that focuses on all elements of a project in a shelf or sub window. By utilising an editor of this type (such as TextMate for OS X), we can keep one window open for each contracted project, each with its own attached drawer and as such simply minimising a given window completely puts a specific project out of sight and out of mind.

Problem #2: Estimations and management of many projects for multiple clients.

Solution #2: Give only rough estimates and keep in mind any potential work which may or may not come into the fray. There is also an amazing tool which only requires a writing surface and the appropriate complimentary writing implement (paper & pencil, whiteboard and a dry erase marker, etc.) The infamous GANTT chart, which allows for a wonderful representation of project portions/phases and time phases. One should not be afraid to over-estimate their time frames for a given project and/or portion of a project. One bit of wisdom which was learned after having it repeatedly played out by both myself and others is that men (not as much as on the women's side of the equation) generally underestimate by a factor of 3. If it is assume that a guy honestly believes that a project with take x minutes, in reality the time frame would be closer to x³ minutes. Gauge oneself over time and projects and adjust the factor accordingly, however start with the aforementioned suggestion as it has proven accurate in my experiences and that of others whom I know personally and professionally.

Problem #3: How should one handle priority requests and/or 'must dos' such as time sensitive changes or additions necessary to client business function regardless of assumed actual worth/priority.

Solution #3: This is much simpler than one would expect. When discussing the issue with the client (wether internal (if a salaried employee and dealing with internal 'customers') or external) point out that this will be shifting the entire project timeline by the time required to complete this unscheduled emergency. Now this isn't always practical or appropriate such as in situations in which a previously scheduled change was turned into a 'must do'. In situations such as these that portion of the scheduled project can be removed from ones GANTT (or other scheduling) chart(s) it their entirety.

There is one caveat with this approach; when removing a project portion due to unexpected/unplanned rushed completion, always add in additional time when shifting the remaining pieces of the project(s). The primary reason for this is simply because additional time should be made available regression testing and/or additional testing due to the reduced time frame and unscheduled manner in which said changes were made, one in which a great possibility for error introduction was more likely. The other major reason for doing as such is simply to protect oneself when another one of these situations occur. It would be foolish for anyone to think that if this happens once, that it is unlikely to occur again. Generally there are those who have little emergencies all the time, and those who 'suffer' from such events rarely if ever. If it happens once, be sure to assume it will occur again as it is usually the result of bad planning or communication somewhere between the engineer and the end customer though more oft than not it is a middle-manager or a sales person making promises which had they been honest and/or considerate of others, they wouldn't have made in the first place.

Handling multiple projects can be easy if one enforces certain rules (albeit with a willingness to bend as long as attention is paid to making adjustments so as to not allow oneself to be run into the ground by continually pushing more amounts of work into a time frame never intended for said work as such.). The importance of communication is key in this instance as expressing realistic time frames in the first place would resolve many of the ugly situation which sadly arise in real world environments on an ongoing basis.

New Projects & Version Control Systems

It has been over a month and a half since my last entry and I apologise for the delay. I have been working tirelessly to finish up my primary design, development, implementation and maintenance functions on what has been my primary project for the past thirteen months. I've been doing this so that I may jump head first into my next project, which while not being a 'huge' undertaking is big enough. By big enough I mean that doing well at the project will land me the role of redesigning a major site from the ground up into the Django framework.

For the past 13 months I've been working heavily in the world of Django and find myself absolutely enamoured with it. I have designed a considerable amount of applications for Django, though none of them are released to the world because they are the product of a contract. My field of expertise was never in that of online applications and/or frameworks so it is stil a new world for me (if we exclude the nine years running a twelve line bulletin board system on a heavily modified Amiga).

I've always been happiest working in middleware and engineering backend systems, though I have to say that I'm finding a certain level of pleasure from the results of my front-end work. When i wrote backend systems, the only others who could appreciate said system(s) other than myself, were developers privy to the project. On frontend systems, there are actual users who I would like to think 'benefit' from my designs and implementations. It is nice to know from actual users that your work is appreciated. Call it ego stroking, or call it what you will but I find that it does inspire one to keep pressing on regardless. I happy to be doing these projects and as soon as it is launched, I'll be happy to post about it here.

This brings me onto the second purpose of this entry, my new favourite version control system. In the beginning of my experience using version control systems back in the 1990's, I simply used CVS like the rest of the world (sans those lucky enough to use Perforce). As time and technologies advanced to newer and better systems, I followed along and moved up to Subversion. Subversion was a well done upgrade for CVS users and easy enough for the uninitiated to learn. I used this most recently when I did work for Curlington Boat Factory (name changed to project myself) writing their point-of-sale returns authorisation/queue system in Python for four hundred some odd cash registers. This worked our quite well despite the fact that the software house for whom my partner (at the time) and I were contracting weren't able to provide the subversion server on their machines until we were ⁷/₈^ths of the way through said project. We ended up (very early on mind you) taking matters into our own hands and setting up on my partner's server at his apartment. This wasn't looked upon highly by said software house, but any CVS on any server regardless of not being in control of the software house was far better than no source control at all.

Next we come to my brief stint working for a OCD kid who started his business at the right time with mom and dad's money, but due to poor decisions on his behalf watched it dwindle in popularity, further plagued by poor management and attempts to treat a small business as if it were a conglomerate thereby sealing its fate with a tender kiss on its cheek. I thankfully left the company before the owner went psycho and let go of all the talent in a horribly vicious manner. Before all of that ugliness transpired, I went forward with trying the newest and brightest trend in distributed version control. I am speaking of the one and only 'Git', championed by a certain arrogant (albeit brilliant) Finnish kernel programmer. There was an immediate joy in using Git and I have to say that most striking feature is the blazing speed when dealing with a large quantity of files. I established all of the existing software base for said company (over ten years worth) into its first repository, utilising Git. Yes, for over ten years, there was no VCS in place. Git really did shine in this role, though I am pretty sure that due to myself and the system administrator not longer being there, cob webs must be forming in the places where Git once speedily did its work.

Let us fast forward to my newest discovery (if you can actually call it one as such). On this newest project which I just started today full time, I have been doing administrative setup work for the past few weeks on and off in small gaps of time as they availed themselves to me. The normal deal of establishing a dedicated server on a FreeBSD host (as Linux just will not cut it for me even though I've been using it since kernel v0.99d), along with all the necessary bits. Once the development languages, utilities, database and web servers were in place the issue of VCS came into play. For this time around I decided that Git, while great for many projects doesn't seem as pythonic a package as I'd like. Using Python long enough really makes one desire beauty, power, simplicity and consistency in all of their tools. This brings us to the most pythonic vcs of them all, Mercurial (known simply by its periodic table element, Hg). Mercurial, like Git is also a distributed version (revision) control system. Easy to setup, easy to use and highly recommended. I'm not going to wax poetic about the differences between these systems as they all serve a purpose. What I will do is state very clearly that Mercurial feels the most natural, works rather well and is quick at what it does. Your mileage may vary.

Until next time...

Django and Gantt Charts

It has now been almost a full week since I started the complete inkedmag.com site and infrastructure redesign in Python using the Django web publishing framework on FreeBSD, and I am happy to report that it is awesome. Mind you we're talking version .96 of the product, yet it truly is a dream with which to work.

I only recently started working with (and writing about) Cheetah, a wonderful python template engine, and had to very quickly learn yet another (Django's own template system). I must admit that Cheetah is easier to ready and learn quickly, but Django's system is considerably more agile in terms of conditionals and modifiers inside the template itself. There even happens to be a simple mechanism for cycling through a list continually changing on each iteration of the loop within which the cycle conditional resides. Simply put, it is wonderful for automatically changing the background colour of a row in a list.

For those unfamiliar with Django, it is simply one of the better web frameworks for content publishing on the web these days. While learning curve can be a little steep for some pieces of the framework, as a whole the speed at which once can produce working pages and applications is staggering. The ease and elegance of the system truly makes one enjoying creating new applications within the framework.

The first application I chose to migrate from native python into the framework was a simple store locator. The new version not only is considerably less lines of code, the database management was done for me at application creation/initialisation. I then simply exported the data from my existing application and imported it into the new table(s) Django created.

I could go on waxing poetic about every little bell and whistle, but I'd just be paraphrasing what many others have already pointed out online and otherwise. Don't think of it as Ruby on Rails because it isn't, though that isn't to be taken as an insult to Ruby. It is much more focused, cleaner and far simpler to setup and get running, including all of its own admin interfaces for the applications you create, as well as its own standalone development web server. Check it out, you won't be disappointed. This is going to save me a considerable amount of time.

Which brings me to my second point; Gantt charts. They are simply not something I find myself utilising on any regular basis, though I think that is going to change. I'm my own boss and have found that gantt charts produce the easiest visual way to show people the various pieces necessary for a project, when each portion can be expected to start and finish, all in parallel with the other projects for which I'm responsible (and/or coordinating).

I feel that the use of this tool more than others really gives a great method by which to see which projects will take the bulk of the time, and what projects overlap, etc. We have a system rewrite to produce and a whole server to replace, not to mention migrating certain custom software into the framework all before the new year. This is doable, but only because we've clearly set realistic (though tight nonetheless) goals and time frames. Consider using a gantt chart if you have more than one project or component of a project which needs to be done in a given time frame. Use one if you need to share with one or more people your schedule and need them to understand as quickly and clearly as possible that with which you are juggling or dealing. You find yourself quickly addicted to its usability.