The Allure of Small...

     When I first started out professionally in the ever growing realm of software development, I'd taken a position with a mid-sized company of roughly three hundred and fifty employees.  Given that it was my first grown-up gig, I had the common misconception that all companies functioned in a similar manner whether minute or enterprise just at a micro or macro level respectively.  I stayed with that company for close to eight years making a multitude of personal and professional relationships along the way.  It was only when I'd learned the inevitable fact regarding first jobs, especially applicable to software development in that the pay scale will never equal the amount of information, skills and experience attained even if one were to be at said company for two decades.  This being said, I made my exit when I was offered a position a small (less than 10 employees) financial startup in the suburbs of Philadelphia, Pennsylvania.

     The first thing I noticed was that the pace was definitely quicker.  This wasn't as glitzy a startup as another at which I'd interviewed (a company called Health Market Science which at the time was in Conshohocken, Pennsylvania) which was far more geek couture.  I was brought in to replace the existing PHP infrastructure with perl.cgi at the time (as well as the primary back-end processing scripts critical to meeting banking and fed. regulations in a timely (read daily) manner.  The first task with which I was charged was an ETL (Extract, Transform, Load) process from client content to our proprietary Goldleaf accounting software for ACH transactions.  This was a good first test and I passed with flying colours.  No sooner had I completed that task (in days) did I have a new project for one of the largest online Canadian gambling sites (solely a client, before the days of the prohibition against online gambling in the United States) to write an API for handling client verification via Lexus Nexus (which at the time had a different name which i forget) in XML.

     This had been the first time I'd been required to do anything with XML outside of a parse here or there, once or twice at my former employer for a one off project.  This is where I noticed one of the biggest differences between larger companies and leaner, smaller companies.  Had this project been part of my tasks at my former employer there would've been dozens of hours of meetings and the project would be measured in a timeline of many months, unlike the three or so weeks it ultimately took from conception through testing to production.  It was in this manner that subsequent projects continued for the next four or so years, even with the company being sold and my transition to CTO somewhere in the middle of all the work, and the building of a solid team of mostly college graduates though I was glad to have a slightly more experienced embedded systems developer which while not part of our required skill set definitely had his own approach to issues thanks to his more diverse experiences prior.

     Ultimately, the position ended when the company ran afoul of the US Government due to several of its clients causing issues for the US Postal Inspector and was ultimately shut down.  I was quickly snatched up to work on a contract with a Massachusetts anti-fraud retail software house called Retail Expert Inc. (eventually consumed by Tyco's Fire & Safety division.)  I was again in a very small, agile environment working with a single other developer building Python (non-web) real-time returns authentication systems for Burlington Coat Factory nationally.  This of course was in the context of being part of a larger company ecosystem but the project of allowing for tendered currency refunds on returns as opposed to store credit was really down to a small team outside of both my cohort and myself.  This was a contract and it was successfully implemented in time for Boxing Day (December 26th in the US) as expected with great results.

     Having enjoyed working with the smaller companies and their associated dynamic I moved on to another contract with an internet hosting provider with a total of three empoyees (not including myself) in New Jersey working on developing an aging, poorly written Perl codebase.  I established best practices, implementing Git version control and a system to push updates to all two hundred and twenty eight servers (at the time) in Python, long before tools like puppet and chef existed.

     I'm not going to continue with my past history at this point because this is about the allure of small, not my entire work history.  I would eventually delve back into the world of larger and even enterprise companies (as well as being a co-founder and lead engineer of other startups respectively) as I was testing the waters of what it would be like.  To stress my point there were quite a few observations which I have now solidified in my mind, based on my experience.

     Small teams, and small companies while lacking the raw capital (and operating costs) are far more capable of accomplishing tasks at hand.  This of course requires a more thoroughly vetted group of seasoned generalists (unlike the specialists more common in enterprise environments) as the importance and value on each individual in these smaller, more tightly knit ventures is much higher.  The one bus rule is very much applicable, more so than when the number of employees increases.

     Larger companies have considerably more red tape and seemingly superfluous meetings.  I say seemingly solely because it does vary from group to group, company to company.  When lack of understanding exists whether of ability, functionality and/or scope, the number of meetings will naturally increase.  This is further complicated by logistics of individuals involved from various departments and teams to the point that it can be debilitating.  The use of carved-in-stone technologies and the lack of ability to move outside of that realm seems to diminish as the size of an organization grows, primarily due to client contract requirements established early on and the maintenance nightmare which ensues.  However this does mean that while a company is firmly built using flat head screwdrivers and hammers, it is very difficult to implement the use of philips head screwdrivers and reciprocating saws when the tasks at hand are much better suited to said technologies.

     It was based on these earliest exposures to what smaller groups were capable of accomplishing, the speed at which they were able to adapt and the focus possible in direct contrast with my personal experiences at larger organizations, coupled with latter experiences whilst dipping my experiential toes into the bigger ponds of enterprise scale that I'd ultimately decided to only approach and work with the former type of companies barring few exceptions.

     The good news is that thanks to the ever growing market and need for the next-big thing, the proliferation of startups and the relatively low cost of entry into emerging markets for crack teams of agile generalists, the future looks pretty bright for myself and my brethren and sistren (yes, that is indeed the correct though fallen-out-of-use word) in the field of software development.

Enforcing the Software Engineer Archetype & Related Principles

It is hard to believe I've been engineering software for the past 14 years, mainly because that isn't reality. I started off like most others, as a programmer/developer and grew personally and professionally over time. As time progresses in our studies and experiences in design, development and deployment we change. This change is more oft than not, for the best.

When I was growing up, I looked towards getting a CS degree so as to become the aptly named "Computer Scientist". What I've found over the years via practical and situational based circumstances is that Computer Science isn't my primary interest. While it is true that there are pieces of the CS realm which have always garnered my attention, such as Artificial Intelligence, it doesn't ring true as a whole.

Today reminded me just how much I would like to think I have changed both in my focus, overall goals and discipline pertaining to the whole process of building software systems. Now the example I'm going to loosely reference is centered around phase one of a much larger long term project. This first phase was somewhat of a rush job as per the client due to botched (i.e. prematurely advertised) promotions for said application's 'live-date'.

I knew that taking this on such short notice would require a very strict set of time guidelines and clearly defined checkpoints and milestones if all was going to be implemented in a proper manner, one supporting a proper holistic software lifecycle approach. This of course required the initial overview of the phase being clarified, the requirements gathering phase, the initial layout with timeline estimates and expectations being put forth and finally said estimates being agreed upon with a little 'wiggle' room so as to allow for human error in the previous steps.

Well, here's what happened today which precipitated this posting. This project has a launch date of 15-Jun-09 and today being the last business day prior to that date, one could say that the end was almost upon me/us at the time. Weekends are out because as an adult and a family person with children, I value my personal and family time very highly, much higher than that of my professional work. That being said, I am indeed an experience professional and know how to accurately plan work into a give time frame clearly stating what can and cannot be realistically expected within a given time schema.

Today approximately one hour before the weekend officially arrived thus signaling the end of this phase of the project, both the designer and the overall project coordinator (not on the Software Engineer side mind you) started throwing out 'new' items for this existing phase almost completed. It is at moments such as these where the undisciplined and junior level individuals panic and ultimately sacrifice their own time for the sake of someone else's lack of professionalism by agreeing to make the changes.

I did the opposite. I made the correct move by stated clearly to all parties that we had agreed upon timelines and that they have been kept. The idea of introducing new components not part of the original design at such a late stage of the phase was outright idiocy. I pride myself in my completeness of the whole process and will not let a failure to plan on someone else's behalf negatively affect the quality work I strive so hard to ensure. Any changes need to be reviewed to ascertain what side-effects might be caused by their inclusion (especially into a more mature codebase at this point) and not to mention the quality/testing cycle which obviously wouldn't be possible due to time constraints.

What I am trying to get at is that I learned that for the sake of professionalism, quality and ethics, one must have the ability to say "No" when others fail in their planning/design. After all, the requirements gathering phase is when a competent Software Engineer brings to light questions that would hopefully coax such ideas from the requirements 'givers' if you will. It is our duty and creed to help our clients both internal and external to bring clarity to their actual needs as many times they are unsure of the specifics until discussed with others.

So I say unto aspiring Software Engineers of the future: People look to us for accountability, and part of that equation is keeping the other variables in the equation (team members, requirement providers, planners, designers and what not) accountable to the process, even if it means telling someone 'No'. Provide your reasons, and hold steadfast as these software engineering processes exist for the benefit of our projects' quality and overall success, not for the sake of being friendly or 'helping out' someone who failed to do their part in the overall planning and execution of a project and/phase thereof.