Tod means Fox

Supporting decisions through sound data management

Posts Tagged Productivity

The 80/20 Principle and Software Development

Sep 3

Posted by Tod McKenna in Software Development

In the book “The 80/20 Principle: The Secret to Success by Achieving More with Less The 80/20 Principle and Software Development“, author Richard Koch describes how a minority of “causes, inputs, or efforts” will lead to the majority of “results, outputs, or rewards”.

A great demonstration of how the 80/20 Principle (aka Pareto’s Law) works can be seen through examining source code. Clearly, a small minority of code in an application produces the vast majority of business benefit to the user. Usually, this code is well defined, tested, and performs great — after all, it is the core of what the application does. Once in production, this code is low maintenance. Any changes or enhancements are usually well scrutinized and will be tested thoroughly. Your applications are bought based on this core minority of code.

Then there’s the other code.

The other code plays more of a supporting or supplemental role: It’s the code that handles your ultra-cool menu system; It’s that extra group of reports that seldom get run; It’s that cool Calendar control that seemed like a good idea at the time; It’s all the extra features that you’ve tacked on over the years. This vast majority of code contributes least to the business needs of the application, costs you the most money to maintain, and is likely to contain the most bugs.

So what’s a developer to do?

Surely bells and whistles, gold plating, and other extras help sell the product. So I am not an advocate of stripping software down so that it is only functional. Software users expect to have a somewhat enjoyable experience behind the keyboard. Boring, functional applications would look a lot like a dos window and be the subject of many scornful conversations at the water cooler.

Instead, get the most bang for your buck. Identify the 20% and expand it. Identify the 80% and depreciate it. Revisit and repeat at each release cycle. Easier said than done? I’ll give you an example:

80/20 In Action

The volume of user reports in a mature application can get out of hand. I’ve worked on Applications in the past that have had more than 100 user reports. One day I wondered (out loud to my coworkers) which reports were actually being used. No one truly knew the answer, but we all had a hunch that the answer was “not many”. So, I added a secret logging script in my report class that logged report usage to a flat text file stored in the root directory of the application. I stored the report name, the user, and the timestamp. We retrieved the file when dialed in for support services. I found that out of 130 custom reports, only 6 were being used on a regular basis at about 100 installed locations, and a whole 50% of the reports were never run at all over the test period (11 months).

Whoa. Consider now that we spent time regression testing these reports during our latest release, that we regularly trained our users on how to run them, and that they are all included and dutifully updated in our documentation. Bad management? Poor software design? Scope bloat? Or just a perfect demonstration of how the 80/20 Principle works in software development: you decide.

The solution in this example was simple: We stopped supporting the unused reports (we continued to monitor the usage logs) and would not add any new reports without careful consideration. It felt good to trim the fat and in the end, we saved ourselves a considerable amount of work.

80/20 Analysis

During the different release periods of the software life cycle, an 80/20 Analysis should be done to both identify the core minority and the excess majority. Start with actual features (User Reports, Ad hoc Reports, Custom Query Engine, the Internet Backup Utility, Etc.) and then drill down into each feature and look for low hanging fruit. Once identified, monetize its impact on your engineering, quality, and client services teams. Also, place a value (using real sales dollars if available) on the feature itself.

This kind of analysis should reveal some interesting things about the software. If there are gray areas, for example if you are unsure of what is being utilized, consider doing something like I did to track report usage. Coverage profiling engines will allow you to identify seldom run or never run lines of code. These areas might give you clues as to what code will buy your house or pull you under. You can create a simple but professional survey on your website and ask your users what features they find most helpful and which features they don’t use. Not only would this survey tell you what is not being used, but it could also help you identify areas where you can increase the 20%.

For more information on the 80/20 Principle, I invite you to pick up Richard Koch’s book at Amazon. Have a read, and share with me your thoughts!

Buy The 80/20 Principle: The Secret to Success by Achieving More with Less The 80/20 Principle and Software Development now at Amazon.

Tags: , ,

2 Comments

Ezelsbruggen

Aug 14

Posted by Tod McKenna in Personal

Or as I like to say, my “easel’s broken”, because it helps me remember how to pronounce it. Which is funny because ezelsbruggen, in dutch (and German too, I think), translates into “donkey bridges” (according to the Internet and my wife). A donkey bridge is a memory technique for creating visual bridges between disparate words. In English, the translation is “memory aid”, or “mnemonic”.

Whether you have a rat in separate, your princiPAL is your PAL, or your committee does nothing but mutter mutter, talk talk, and eat eat, donkey bridges are quite handy. They help you remember little bits and pieces of information, like the form instantiation order in FoxPro (the lovely Lisa G.). Oftentimes donkey bridges are illogical, arbitrary, and downright silly — but they always seem to work. Who is this Lisa G. anyway?

So, here is a list of some of the mnemonics — Ezelsbruggen — that I have learned over the years. If you have any fun or interesting ones, please share!

  • LISA G.
    Load, Init, Show, Activate, GotFocus
  • Lucky Cows Drink Milk
    The order of Roman numerals (ascending): LCDM
  • Please Excuse My Dear Aunt Sally
    Parentheses, Exponents, Multiplication, Division, Addition, Subtraction.
  • My Very Educated Mother Just Served Us Nachos
    There are only eight planets now! Sad… but true.
  • Better Be Right Or Your Great Big Van Goes West
    Ohms value for the color bands of resistors (Black, brown, red, orange, yellow, green, blue, violet, gray, white).
  • Silly People Drive Fast
    Spectroscopic notation, after F, the rest is alphabetic

So here I am in a canoe in the middle of the Atlantic Ocean, drinking the salt water out of a coffee filter. Smoke from my hibachi is getting in my eyes as my veggie burgers begin to burn. Although I’m ankle deep in plums, I manage to toss Ben and Jerry overboard so they can fix the motor, which is nothing more than a dust buster turned from suck to blow.

That, my friends, was yesterday’s shopping list. Poland Springs, coffee filters, veggie burgers, plums, ice cream, and a new hand vacuum. If I don’t create an absolutely absurd story, I’m liable to forget something without a list.

Method of Loki

Another great way to remember sequences and lists is employing the Method of Loki. In its most basic form, you simply find a location you are familiar with in your head (your bedroom, office, kitchen, etc), and begin placing things inside the room. It’s a method many use for giving paperless speeches or for remembering meeting agendas. For example, in my office, I may see the head of the research department typing away at my computer, a spinning globe just off to her right, and a bowing shelf of paper documents within arm’s reach. This will remind me to discuss the new research project, segue into our International strategy, and finally end by discussing a much needed content management solution.

These and other such techniques for remembering items, spellings, sequences, etc, have been a big help for me. In the health care industry, I’m often required to remember many acronyms, processes, and procedures that are quite foreign (I have no medical background whatsoever — unless you count years of watching ER). The same might be true for you as well.

Now if you’ll excuse me, I need to see the quill from behind the tall man who’s sitting on a beanbag chair fiddling with a rubix cube.

Tags:

4 Comments

5 Ways to Automate Development in FoxPro

Jul 30

Posted by Tod McKenna in Software Development

I enjoyed putting together my last top seven list so much, that I decided to make another, more targeted list. This one, lists five ways that you can automate software development in Visual FoxPro. To clarify: this article is not about FoxPro automation! But rather, five tips on how you can get the development environment to work for you.

It surprises me how many FoxPro developers I know do not automate, even though the Fox team at Microsoft has given us a lot of tools to do so. As I mentioned in my last article, automation is one of the keys to better productivity. The more tasks you can do automatically, without thought and with little effort, the greater productivity you will enjoy! On top of that, by automation, we can eliminate a lot of thinking, planning, repetition, and needless toiling over the most mundane tasks.

Not every version of FoxPro features every item in my list (although, VFP9 gets them all). The point of this article is not to be exhaustive or to painstakingly document every feature’s introduction into the product; but rather, to get you thinking about automation to improve your production.

Without further ado:

1.) Use Macros

No, no. Not macro substitution! Macros from the Tools menu! FoxPro Macros are powerful little scripts that you can initiate with a mere key combination (like ALT+CTRL+A). You can write a macro to do almost anything. Their primary purpose in life is to automate keystrokes. For example, I have a whole set of macros that open various projects, sets system defaults, and closes/opens databases as necessary. A nice thing about FoxPro Macros is that it will record your keystrokes for you. To start, simply go to Tools / Macros; Click Record; enter the keystroke combination and macro name; and start typing in the command window. When done, click back on Tools / Macros and click OK to stop recording. Next time you want to run the sequence you just created, simply do the keystroke you defined!

Please note: chances are good that you’ll need to tweak the generated code, but don’t fret. In my experience, this consists of adding {ENTER} at the end of a command, or perhaps cleaning up some automatically-inserted values from IntelliSense. No big deal.

2.) Utilize and Customize IntelliSense

Speaking of IntelliSense! As most of you are aware, IntelliSense is a form of automated auto-completion of keywords, class names and methods, parameter definitions, _VFP and _SCREEN system variables, ActiveX controls, COM servers, and the like. But you can also add your own records in the IntelliSense database! You might want to define common enumerated values (like a long list of DEFINES you have tucked away in an include file), custom class definitions, and type libraries. You can also add IntelliSense support for registered type libraries, user-defined types, members, and code elements, enumerated values, and custom classes. IntelliSense is a powerful and easy to manipulate tool that can save you tremendous amounts of time during development.

There is already a great repository of custom IntelliSense scripts for you to browse through and incorporate into your FoxPro programs (hey, did you already forget how selfless the Fox community is?): find them over at the FoxPro Wiki. Andy Kramek also wrote (part 1, part 2) a very informative blog entry about the subject some time ago.

3.) Use Project Hooks

Project hooking allows you to manipulate your project and contained files programmatically. I find this incredibly helpful during builds (tapping into BeforeBuild and AfterBuild, for example), but there are dozens of great uses for hooking into your project. One of my favorite things to do is to log my project builds in a build table. This allows me to monitor and audit the build process (and produce some interesting metrics such as build time, number of builds, etc.).

If you’re at all interested in pursuing this any further, I highly recommend you look into White Light Computing, Inc’s Project Builder and ProjectHook. On that page, there is a very good Whitepaper detailing the tool (and a screenshot of the UI). Best part is, it’s free and developed by one of the community’s best.

The bottom line is that you can automate a ton of activities using these hooks. You can perform backups, make copies, test integrity, check for updates, copy builds to remote locations, etc. The limit is your imagination (for the most part!).

4.) Customize Your Toolbox

I love the Toolbox in VFP9 — especially because I’m used to developing in C#.Net and SSIS. The Toolbox (available form the Tools menu) to me is an intuitive piece of the IDE, and fits nicely into an automated work environment. It is divided into sections which you can customize. These sections are full of collections of various tools. These tools are typically classes, text scraps, Active X controls, and can even hold tables, images, reports, labels, and forms. You can drag and drop items from the toolbox into the command window, a program file, or onto a form. For automation, you can create lots of very cool text scraps (like comment headers) to save you some type and formatting time while developing.

To dig into the toolbox, and customize it fully, right click anywhere on the control and select ‘Customize Toolbox’. You can adjust various behaviors and add and remove items from the different categories. Note: to dock in VFP9, first set the ‘always on top’ property to false. Then, you can dock the toolbox anywhere your heart desires!

5.) Automated Testing

Although I have not had any success with commercial automated testing products (the last one I tried was Borland’s SilkTest, which did not work well with VFP9), I have had mild success using the Automated Test Harness that ships with VFP. The harness gives you the ability to create and run various scripts that will play back mouse and keyboard events — essentially running various parts of your application for you. I admit that even this tool isn’t the best, but you can automate enough tasks and perform enough tests automatically to save you time and effort on regression. The harness taps into Microsoft Active Accessibility (MSAA) technology. To run, type DO (HOME() + “toolstestaatest”). It takes a bit getting used to, but in an afternoon, you can have the tool up and running, testing your application automatically. As an added bonus, the harness allows you turn coverage profiling on and off.

…

Well, that’s that! I hope you find these five ways to automate development in FoxPro useful. Assuredly, there are others. Feel free to comment and post articles, tips, or other feedback to round my list out!

Tags: , , ,

8 Comments

7 Productivity Tips for Better Software Development

Jul 24

Posted by Tod McKenna in Software Development

I’ve always wanted to make a ‘Top 7′ list.

Most of our time is wasted away throughout the day on trivial and unimportant tasks. Some of this is busy work that we create, while other things creep in unexpectedly. Here is a list of seven things that I feel will increase your productivity during a typical workday. Let me know if you think I’m missing or over-emphasizing something. I’m always looking for ways to improve my effectiveness and efficiency; to do more with less.

1.) Find your most productive time
Some developers prefer to work through the night. Most CEOs start their day before 6am. Find your sweet spot and plan your day around this block of time. From what I’ve read, this block of time varies from 4 to 7 good hours. Mine happens to be from about 5am through to 11am. After this, my concentration wanes and my patience thins (until I get home from work, at which time I’m re-energized).

2.) Identify critical tasks
List your most critical tasks for the day, and do them first. Critical does not mean most difficult. Critical tasks may be the easiest and quickest tasks of your day. Perhaps it’s as simple as calling back an important client, or a more difficult task such as finally fixing that nasty bug that’s hard to reproduce. I find that making this list the day before helps me to get started the next day much quicker.

3.) Avoid distractions
Distractions can really lead you astray and spell disaster for your carefully planned schedule. Avoid these things, especially when in your most productive block of time, and certainly while you are completing your critical tasks. Because I spend a large part of my workday in a cubical, and cubes are distraction magnets to begin with, I’m often inundated with visitors who have come by for various reasons (few of them have any relevance to my current tasks). To help combat this, I will often put headphones on to deter all but the most important guest.

I consider mail (e-, voice-, and snail-) a distraction. Very few actual mail items require my immediate attention throughout the week. I suspect you’re in the same boat. So, Manage your mail wisely. Set aside specific time for reading and responding to mail in all its forms. This is especially good advice for email. Never look at email during your most productive time. And please, turn off the automatic notification and get rid of that envelop icon that Outlook puts in your tray. Every email that hits your Inbox is a potential detour. To combat this, I try to tackle email first thing in the morning (5am), with 2 additional checks throughout the day (10am, 3pm-ish). I will only open Outlook during these times.

4.) Automate as much as possible
Automation is the key to more productivity. The more you can automate, the more time you can spend on important tasks. For years I had been manually making additional ‘sanity’ backups of my source code (above and beyond VSS and normal backups). Finally, I woke up and realized that this (and many other things) can be automated. You can be fancy and schedule a backup through some backup software, or be simple and create some .bat files to do the heavy lifting. Sounds simple enough. This is just one example. There are potentially hundreds of tasks you can automate.

5.) Apply Pareto’s Law
Otherwise known as the 80/20 principal (or rule), this law states that 80% of the effects comes from 20% of the causes. The vital many verse the trivial few. Learning how this works can be like hiring a new employee (a better you) and firing that resource hog (the old you). Find out what busy work you’re doing and eliminate it. Identify the work you do that produces the best results, and capitalize.

6.) No more multitasking
Despite popular belief, multitasking breaks concentration, causes distraction, and inevitably costs you time and money. This is especially true of technical work, like software development. Shutdown your Outlook, your Accounting software, your Internet Browser, your News Aggregators (you can leave mine on, though icon wink 7 Productivity Tips for Better Software Development ), etc. Focus on the task at hand, complete it, and move on.

7.) Ergonomics are your friend
Get comfortable and put a greater focus on your work environment. It should be healthy, comfortable, and safe. Stand up once in a while to stretch and relax your hands, wrists, and neck. You can loosen the kung fu grip on your mouse, flatten your feet on the floor, adjust the top of the monitor to be at eye level, and do many other things to improve your condition. I’m terrible at all of this: I seldom get up, I sit Indian-style, and my dual monitors are angled and a little high. But I have seen others recover from this and start on the path of a more ergonomic-enriched existence. Being comfortable helps you to be productive.

Do you have productivity advice? Is there anything missing form my list?

Tags: , ,

6 Comments

Broken Promises of Drag-and-Drop Programming in SSIS

Jun 21

Posted by Tod McKenna in Software Development

I’ll come right out and say it: I am no fan of drag-and-drop programming. Perhaps I’ve been emotionally scarred by FrontPage 2000 or something? This doesn’t mean that I don’t appreciate the ability to drop a command button on a form, or create an HTML table with a pencil. But it does mean that I don’t like (or appreciate) companies telling me that their drag-and-drop environments will ‘jump start’ me or ‘do all the dirty work’, because inevitably, I find myself in a position where I have to either (a) debug the generated code, (b) work around an undesired behavior, or (c) write the code myself anyway.

Take SSIS for example.

I am in the process of a large ETL project involving VFP and SQL Server 2005. We are taking data from SQL Server and populating VFP free tables (for re-distribution with our installed apps). Yes, you read right: SQL Server back to FoxPro.

So the company, long ago (and maybe in a galaxy far, far away), decided to use SSIS as the ETL tool of choice. Now, everything is in SSIS and a lot of time and effort has been put into the endeavor. With scores of packages extracting data from a variety of sources, SSIS has proved reasonably useful from a management and execution point-of-view. But I have to tell you, writing complex SSIS packages is no breeze — and the whole ‘drag-and-drop’ approach is seriously getting in the (read: my) way.

I took an inventory of all the work I needed to do for this project. Here’s the skinny:

Variables

The extracted data can have one or multiple sources (spanning multiple SQL Server databases) and destinations (each with a different FoxPro data schema and each potentially on different computers on the Network). I also need to be able to call packages from other packages. As a result, a lot of time was spent on setting up variables (and configuring the child packages to receive them from their parent). These variables are used extensively throughout the solution (in the connection manager and in various expressions for example — and don’t get me started on the expression builder!).

Sources

Here’s were drag-and-drop programming really starts to fall short. My sources, almost every one of them (there are dozens of sources in the solution) use SQL statements — instead of actual Tables (or views) — to gather the data (using the OLE provider, not the native driver, by the way). This really isn’t a big deal and in fact gives me a ton of flexibility, and I’m sure this approach is rather common.

For example, the FoxPro datasets do not support NULLs in any column (don’t ask me why, and it’s too late to fix it!). In the drag-and-drop world, one might drag a Derived Column tool, fight with the expression field, and ‘replace’ the NULLed column using an expression with some non-null value one at a time. Ugh. I tried this once and let me tell you: just build NULL elimination into the SQL statement using coalesce instead. Save yourself some aggravation and a headache. Also, while you’re at it, do all your converts, trims, calculations in there as well. In my tests, performance was not affected by this approach (using a database with several million records). As I’ve mentioned many times before though, I’m not an expert on benchmarking. But you can write SQL much quicker than using that darned expression dialog box — not to mention everything is right out in front of you in an SQL statement, and if you need to debug or experiment with the SQL, just copy and paste it into SSMS and off you go (try doing that with the Derived Column tool).

Transformations

Why is it that, with the entire toolbox of transformations provided by SSIS, I am always forced into using the Script transformation, the Union All as a terminator (so my lookups don’t complain about error output), and non-standard tools (such as the Lookup) to manipulate and filter data? I’ve already mentioned that Derived Columns and similar tools can be accomplished using good old fashioned SQL.

Debugging

Sorry folks, I’m not going here. Forget it. I might be likely to say something I’ll regret.

Sigh

When looking at this integration project as a whole, I came to realize two important facts:

  • The core functionality of the package is hand-written code (the SQL, transformations, even setting up the variables);
  • And, that most of the time to put this together (about 60 hours of programming in total) was spent on getting SSIS to behave and ‘flow’.

It turns out that drag-and-drop programming isn’t really the answer IMO. Sure, it might get you off the ground, but I find that tools like SSIS just get in the way more than anything else.

So I have to ask: Why bother? If I had hand-coded this in VFP (or any other data-centric language), it would have been done last week, and it would have been just as easy to execute, maintain, and turn over. Especially if I had a framework to start off with. Hmmn…

Tags: , , ,

4 Comments

Debug Environment

Nov 29

Posted by Tod McKenna in Software Development

Not to take too much time away from my series on testing applications and bullet-proofing source code, but I felt that a little time dedicated to setting up a proper debugging environment would be helpful. A developer should have a different set of tools available than the end user. These tools will give the developer the ability to stop program execution, run some utilities, start a coverage, track events, etc.

Below are the main features of a debugging environment that I find important. Because the implementation can be done in several ways, I won’t get bogged down into the details at this time. Perhaps I will address one or two of these in a future article:

1.) A global debug flag. This flag would be part of a custom application object that can be used to determine whether or not the debug features are present or absent in the current instance of the application. Setting the flag can be done in a config file, a setup table, or even via a parameter during execution. The flag should be smart enough to know if the application started from the VFP command window (or the program/do menu) or via a desktop shortcut. We want to avoid ‘feature not available’ errors; we would also like to remind the developer that he or she is trying to run debug mode from a shortcut.

2.) A custom error handler. It is highly recommended to include custom error handling at the class level (via a form, container, textbox, etc). But there should also be a global handler that can address any issues that arise otherwise. These error handlers must be smart enough to recognize when a developer is in debug mode. These methods should give the developer a means to suspend execution (perhaps by spitting out the default VFP error notification) and to save environment settings for further research.

3.) A debug menu. When in debug mode, a special menu called ‘Debug’ should be created; pop it in right after the help menu. The menu should contain some useful features, such as a SUSPEND command; window activation (Trace, Command, Data); Coverage and Event logging; ability to open log files and tables; perhaps the ability to output information about the current environment, call stack, and objects; and various custome functions and programs not available to the end user. In addition, it should be chock-full of utilities and functions that can be called when needed.

4.) There should be a built-in restore mechanism that will automatically restore key labels, menus, and the VFP environment at anytime. Coupled with this would be a mechanism to cancel the program in a clean way for the developer.

This is the bulk of what is required for a proper debugging environment. These additions could save hours during debugging sessions. Exploring them further is well worth the effort.

Tags: , ,

No Comments