Tod means Fox

Day two was a great day. I find that the size of the conference is perfect for the technologies being discussed. A lot of attention is given to the decline of conference sizes over the past 5 or 6 years. This is true for VFP and other conferences as well. But clearly, the number of “heads” do not diminish the worth of the presentations or the work that the speakers have put into them. I like the intimate atmosphere and will be looking forward to attending more conferences like it.

Fox Trails

Brian’s Fox Trails presentation was great. I didn’t know what to expect (even though I had heard of Fox Trails, and knew what it was supposed to do). I’m not a Ruby guy, and I’ve honestly never evaluated the technology. But his Ruby-borrowed techniques for accessing data and functions in FoxPro was eyeopening and insightful.

One of Ruby’s tenets is: “Don’t repeat yourself”. Brian was quickly to repeat himself… and he almost did. A hearty laughter ensued… Another Ruby tenet is “Convention over Configuration”, making applications more “self aware”. Using directory naming conventions, for example, allows you to write code that can assume certain things about its environment. I’ve always been a big advocate of “Reconfigure over Recode”. I should change mine to “Convention over Reconfigure over Recode”!

He introduced us to “MVC Architecture”, which stands for Model (data and access), View (presentation layer), Controller (business logic). This architecture parallels 3-Tier architecture which many VFP developers are used to. Those of you who have evaluated or are using Ruby will get Fox Trails right away. Those of you who do not, should take a moment to have a look.

Some additional information here and here. Check it out!

VFP Service Architecture

I’m really bummed I didn’t think of this before. I’m really bummed that I didn’t hear Stewart’s session 10 years ago. Come to think of it, ‘bummed’ is not really the word.

Stewart showed us how we can use service architecture, stored on the server as a process, to pass various tasks to the server for processing. As he mentioned, there are literally hundreds of services that you can create on the server. He demonstrated the Mailslot Messaging and PDF report generation. This means that you don’t have to rely on user’s workstation settings to send mail. It also means that they don’t need a PDF writer installed to print to PDF! You configure the service on the server, hit the service from VFP, and send mail; or print a PDF using GhostScript; or zip a file; or run some encryption; or send data via FTP; you get the idea. This architecture can dramatically reduce the amount of workstation configuration needed.

Later in the day, Stewart and I sat through Stephen Bodnar’s MapPoint presentation. I think his bulb went off when mine did. We asked Stephen, “could MapPoint be used as a service”? Because of cost and installation issues, it seemed like a perfect fit for service architecture. We’ll need to read the EULA agreement though. Stephen was skeptical. (More on MapPoint in a few sections.)

I also saw some old Fox commands that I don’t think I’ve ever had an occasion to use: “Save To” and “Restore From”. “Save to” allows us to store current variables and arrays to a variable file or memo field and “Restore from” allows us to retrieve variables and variable arrays saved in a variable file or a memo field, placing them into memory (according to the VFP9 documentation). Come to think of it, I’ve probably used them before, but just can’t remember. They seem too useful!

Developing in n-tier fashion

Michael Babcock showed us how to write non-monolithic applications in multiple tiers. He went through the basics of tier development ,from 1-tier up to n-tier and talked about Logical (same process) verses physical n-tier design (when you can’t see through the tiers, like when using a com server or the Internet).

Mike used the phrase “pass the buck” often to remind us that each tier has a job, and that all tasks (business calls, UI, data access, etc) should be handled in the appropriate tier. In other words, object responsibility. This discipline is forward-thinking and allows you to scale and upgrade applications easily. If you have ever wondered what it would take to build a web UI using your business and data access in VFP, n-tier (and perhaps with some Fox Trails) is the way to go. The same holds true for .NET front ends, mobile applications, and the like.

His business object demonstrations were great, showing the group how to correctly use the business object tier. He also went through the proper methods for using the user interface and data tiers of the design. He walked through real-world scenarios of how you might use an Oracle back end verses a CSV file using the same UI and Business Object. Mike was engaging, well prepared, and his presentation was spot on.

Map Your Data

Using MapPoint, automating it, and utilizing it through ActiveX on a VFP form, Stephen Bodnar brought us through the basics of using the tool. MapPoint with VFP is very cool. He talked about his trip back from an Advisor conference in San Diego back to Detroit (they were grounded on 9/11); then showed us the map of their travels (we hear that Whil Hentzen is a speed demon, averaging something like 80mph on the trip back).

The idea with MapPoint is to use maps to show geographic information in a better context. Just like Google and Yahoo.

I’ve been waiting to see this presentation all day because I have about a million and one needs for a solution like this. Whether showing a map of property sales, or inspection routes, this tool can certainly be used in many different contexts. I’ve been preparing myself for diving into GIS applications (probably ESRI’s tool) for this very purpose. Now I can evaluate MapPoint as an alternative, feeling confident that it will work with VFP and that it will give me the routes and maps I need for my apps.

MapPoint has its problems, though. It is not updated frequently, and all address data needs to be exact for the tool to work correctly. You need to spend about $200 per license on every machine that uses MapPoint.

I also learned a new trick. In VFP (starting in VFP7) the Object Browser can be used to create a set of #DEFINE statements which lists all the constants in a type library by using drag and drop from editor window. The only problem with this approach is that VFP truncates decimals if contained in the constant. Rick Strahl’s tool, “Get Constants” circumnavigates this known VFP bug from the object browser. His tool is here.

…

On to day 3, the final day of FoxForward 2007!

In the book “The 80/20 Principle: The Secret to Success by Achieving More with Less“, author Richard Koch describes how a minority of “causes, inputs, or efforts” will lead to the majority of “results, outputs, or rewards”.

A great demonstration of how the 80/20 Principle (aka Pareto’s Law) works can be seen through examining source code. Clearly, a small minority of code in an application produces the vast majority of business benefit to the user. Usually, this code is well defined, tested, and performs great — after all, it is the core of what the application does. Once in production, this code is low maintenance. Any changes or enhancements are usually well scrutinized and will be tested thoroughly. Your applications are bought based on this core minority of code.

Then there’s the other code.

The other code plays more of a supporting or supplemental role: It’s the code that handles your ultra-cool menu system; It’s that extra group of reports that seldom get run; It’s that cool Calendar control that seemed like a good idea at the time; It’s all the extra features that you’ve tacked on over the years. This vast majority of code contributes least to the business needs of the application, costs you the most money to maintain, and is likely to contain the most bugs.

So what’s a developer to do?

Surely bells and whistles, gold plating, and other extras help sell the product. So I am not an advocate of stripping software down so that it is only functional. Software users expect to have a somewhat enjoyable experience behind the keyboard. Boring, functional applications would look a lot like a dos window and be the subject of many scornful conversations at the water cooler.

Instead, get the most bang for your buck. Identify the 20% and expand it. Identify the 80% and depreciate it. Revisit and repeat at each release cycle. Easier said than done? I’ll give you an example:

80/20 In Action

The volume of user reports in a mature application can get out of hand. I’ve worked on Applications in the past that have had more than 100 user reports. One day I wondered (out loud to my coworkers) which reports were actually being used. No one truly knew the answer, but we all had a hunch that the answer was “not many”. So, I added a secret logging script in my report class that logged report usage to a flat text file stored in the root directory of the application. I stored the report name, the user, and the timestamp. We retrieved the file when dialed in for support services. I found that out of 130 custom reports, only 6 were being used on a regular basis at about 100 installed locations, and a whole 50% of the reports were never run at all over the test period (11 months).

Whoa. Consider now that we spent time regression testing these reports during our latest release, that we regularly trained our users on how to run them, and that they are all included and dutifully updated in our documentation. Bad management? Poor software design? Scope bloat? Or just a perfect demonstration of how the 80/20 Principle works in software development: you decide.

The solution in this example was simple: We stopped supporting the unused reports (we continued to monitor the usage logs) and would not add any new reports without careful consideration. It felt good to trim the fat and in the end, we saved ourselves a considerable amount of work.

80/20 Analysis

During the different release periods of the software life cycle, an 80/20 Analysis should be done to both identify the core minority and the excess majority. Start with actual features (User Reports, Ad hoc Reports, Custom Query Engine, the Internet Backup Utility, Etc.) and then drill down into each feature and look for low hanging fruit. Once identified, monetize its impact on your engineering, quality, and client services teams. Also, place a value (using real sales dollars if available) on the feature itself.

This kind of analysis should reveal some interesting things about the software. If there are gray areas, for example if you are unsure of what is being utilized, consider doing something like I did to track report usage. Coverage profiling engines will allow you to identify seldom run or never run lines of code. These areas might give you clues as to what code will buy your house or pull you under. You can create a simple but professional survey on your website and ask your users what features they find most helpful and which features they don’t use. Not only would this survey tell you what is not being used, but it could also help you identify areas where you can increase the 20%.

For more information on the 80/20 Principle, I invite you to pick up Richard Koch’s book at Amazon. Have a read, and share with me your thoughts!

Buy The 80/20 Principle: The Secret to Success by Achieving More with Less now at Amazon.

It’s been said that if you can’t do something 3 different ways in FoxPro, then it can’t be done. I’m not sure where this phrase originated, but it’s stuck with me over the years. With options though, come responsibility. Yes, you can often solve a problem using several methods, but your job as a developer is to pick the most reliable and best performing of the options available. This could mean making a choice to use the REPLACE statement instead of SQL-UPDATE, MEMLINES/MLINE over ALINES, or TRANSFORM instead of STR.

Trimmed conversion of number n

For example, run this code from a program file a few times over. They are functionally equivalent, producing a trimmed string conversion of a number n:

clear
nSec = SECONDS()
FOR n = 1 TO 100000
  lc = ALLTRIM(STR(n))
NEXT
? "Run 1 with STR(): "
?? SECONDS() - nSec
 
nSec = SECONDS()
FOR n = 1 TO 100000
  lc = TRANSFORM(n)
NEXT
? "Run 2 with TRANSFORM(): "
?? SECONDS() - nSec
 
nSec = SECONDS()
FOR n = 1 TO 100000
  lc = CAST(n as Varchar(10))
NEXT
? "Run 3 with CAST: "
?? SECONDS() - nSec

You will notice right away that using CAST is by far the fastest (on average 2.5 times faster than ALLTRIM(STR())). Transform is a little faster than ALLTRIM(STR()) as well (on average 1.3 times faster using 100000 iterations as a test). If you reduce the number of iterations, this ratio reduce until finally ALLTRIM(STR()) begins to outperform TRANSFORM().

CAST is new to VFP9, so it is likely that it hasn’t made its way into a lot of your code — but perhaps some refactoring is in order! Especially in loops with many iterations or in SQL statements.

nLength conversion of number n

In the previous example, I returned a trimmed string of a number n. Consider that you want your numeric transformation to be of a particular size. What’s faster now?:

clear
nSec = SECONDS()
FOR n = 1 TO 1000000
  lc = STR(n,10,2)
NEXT
? "Run 1 with STR(): "
?? SECONDS() - nSec
 
nSec = SECONDS()
FOR n = 1 TO 1000000
  lc  = TRANSFORM(n,"9999999.99")
NEXT
? "Run 2 with TRANSFORM(): "
?? SECONDS() - nSec
 
nSec = SECONDS()
FOR n = 1 TO 1000000
  lc = CAST(n as Character(10))
NEXT
? "Run 3 with CAST: "
?? SECONDS() - nSec

Again, CAST is faster than both STR and TRANSFORM (on average 2.7 times faster). Transform (which surprised me) still outperforms STR() (at 100,000 iterations, the ratio on average was still about 1.3). But take heed! The last example here isn’t functionally equivalent: CAST will left-align the results, while STR and TRANSFORM right-align the results. STR and TRANSFORM will round decimals to the specified amount, CAST will not.

Obviously, there are many things you can do with TRANSFORM that you cannot do with either STR or CAST. In most cases, however, CAST seems to be the way to go. Using ‘Varchar’ in CAST trims your string, using ‘Character’, does not. TRANSFORM automatically produces a TRIMed string as well, unless you specify a format code.

I like the new CAST function in VFP9. I got used to using it while writing code to interact with SQL Server. But I soon found CAST sneaking into my code in various other places — often replacing ALLT(STR()) and TRANSFORM.

Andy Kramek put together a fantastic blog entry earlier today, titled “Creating Classes with a Factory“. This type of design pattern belongs to the Creational family of patterns, and is used mainly to, um, create things!

What I really like about Andy’s entry is his emphasis on metadata to accompany the pattern. He writes: “[metadata] is what makes this implementation so useful in my opinion, and why VFP is such an ideal tool in which to implement this particular pattern”. And of course, he’s absolutely right!

But more importantly, Andy has uncovered a very useful function of metadata that can be applied to many other elements of software development. That is, the ability to use metadata to store code snippets that can be updated or tested by developers while the executable is running. This isn’t only a handy time-saver, but it’s also a great way to compartmentalize code, deliver custom functions to clients, and build robust and scalable solutions. By including code in memo fields, you can customize applications without changing the core engine. And, as Andy pointed out, you can test various implementations while running the executable.

I’ve used this approach for many years; for as long as EXECSCRIPT has been around. I’ve used to it generate SQL statements on the fly, run conversion routines (some clients like data stored using inches, while others prefer millimeters for example), and separate complex and specialized algorithms that have different implementations depending on the client. I rely on it heavily for all my data integration tasks. In fact, logical data maps (which are more than just “logical” in my world) are used solely to drive all data transformations — not a single piece of business logic is included in the application. The alternative to the metadata approach is to embed these things into the source code itself — but in my experience, this just leads to greater maintenance, regression headaches, and clients that need to wait a little longer.

Using metadata allows you to separate functionality from implementation. In the case of Andy’s Factory pattern, this is one of his important bullet points. In Visual FoxPro, we have the luxury of an in-house, native database that is fast and easy to use. Perfect for metadata (even if you’re using some other database for the backend).

I enjoyed putting together my last top seven list so much, that I decided to make another, more targeted list. This one, lists five ways that you can automate software development in Visual FoxPro. To clarify: this article is not about FoxPro automation! But rather, five tips on how you can get the development environment to work for you.

It surprises me how many FoxPro developers I know do not automate, even though the Fox team at Microsoft has given us a lot of tools to do so. As I mentioned in my last article, automation is one of the keys to better productivity. The more tasks you can do automatically, without thought and with little effort, the greater productivity you will enjoy! On top of that, by automation, we can eliminate a lot of thinking, planning, repetition, and needless toiling over the most mundane tasks.

Not every version of FoxPro features every item in my list (although, VFP9 gets them all). The point of this article is not to be exhaustive or to painstakingly document every feature’s introduction into the product; but rather, to get you thinking about automation to improve your production.

Without further ado:

1.) Use Macros

No, no. Not macro substitution! Macros from the Tools menu! FoxPro Macros are powerful little scripts that you can initiate with a mere key combination (like ALT+CTRL+A). You can write a macro to do almost anything. Their primary purpose in life is to automate keystrokes. For example, I have a whole set of macros that open various projects, sets system defaults, and closes/opens databases as necessary. A nice thing about FoxPro Macros is that it will record your keystrokes for you. To start, simply go to Tools / Macros; Click Record; enter the keystroke combination and macro name; and start typing in the command window. When done, click back on Tools / Macros and click OK to stop recording. Next time you want to run the sequence you just created, simply do the keystroke you defined!

Please note: chances are good that you’ll need to tweak the generated code, but don’t fret. In my experience, this consists of adding {ENTER} at the end of a command, or perhaps cleaning up some automatically-inserted values from IntelliSense. No big deal.

2.) Utilize and Customize IntelliSense

Speaking of IntelliSense! As most of you are aware, IntelliSense is a form of automated auto-completion of keywords, class names and methods, parameter definitions, _VFP and _SCREEN system variables, ActiveX controls, COM servers, and the like. But you can also add your own records in the IntelliSense database! You might want to define common enumerated values (like a long list of DEFINES you have tucked away in an include file), custom class definitions, and type libraries. You can also add IntelliSense support for registered type libraries, user-defined types, members, and code elements, enumerated values, and custom classes. IntelliSense is a powerful and easy to manipulate tool that can save you tremendous amounts of time during development.

There is already a great repository of custom IntelliSense scripts for you to browse through and incorporate into your FoxPro programs (hey, did you already forget how selfless the Fox community is?): find them over at the FoxPro Wiki. Andy Kramek also wrote (part 1, part 2) a very informative blog entry about the subject some time ago.

3.) Use Project Hooks

Project hooking allows you to manipulate your project and contained files programmatically. I find this incredibly helpful during builds (tapping into BeforeBuild and AfterBuild, for example), but there are dozens of great uses for hooking into your project. One of my favorite things to do is to log my project builds in a build table. This allows me to monitor and audit the build process (and produce some interesting metrics such as build time, number of builds, etc.).

If you’re at all interested in pursuing this any further, I highly recommend you look into White Light Computing, Inc’s Project Builder and ProjectHook. On that page, there is a very good Whitepaper detailing the tool (and a screenshot of the UI). Best part is, it’s free and developed by one of the community’s best.

The bottom line is that you can automate a ton of activities using these hooks. You can perform backups, make copies, test integrity, check for updates, copy builds to remote locations, etc. The limit is your imagination (for the most part!).

4.) Customize Your Toolbox

I love the Toolbox in VFP9 — especially because I’m used to developing in C#.Net and SSIS. The Toolbox (available form the Tools menu) to me is an intuitive piece of the IDE, and fits nicely into an automated work environment. It is divided into sections which you can customize. These sections are full of collections of various tools. These tools are typically classes, text scraps, Active X controls, and can even hold tables, images, reports, labels, and forms. You can drag and drop items from the toolbox into the command window, a program file, or onto a form. For automation, you can create lots of very cool text scraps (like comment headers) to save you some type and formatting time while developing.

To dig into the toolbox, and customize it fully, right click anywhere on the control and select ‘Customize Toolbox’. You can adjust various behaviors and add and remove items from the different categories. Note: to dock in VFP9, first set the ‘always on top’ property to false. Then, you can dock the toolbox anywhere your heart desires!

5.) Automated Testing

Although I have not had any success with commercial automated testing products (the last one I tried was Borland’s SilkTest, which did not work well with VFP9), I have had mild success using the Automated Test Harness that ships with VFP. The harness gives you the ability to create and run various scripts that will play back mouse and keyboard events — essentially running various parts of your application for you. I admit that even this tool isn’t the best, but you can automate enough tasks and perform enough tests automatically to save you time and effort on regression. The harness taps into Microsoft Active Accessibility (MSAA) technology. To run, type DO (HOME() + “tools\test\aatest”). It takes a bit getting used to, but in an afternoon, you can have the tool up and running, testing your application automatically. As an added bonus, the harness allows you turn coverage profiling on and off.

…

Well, that’s that! I hope you find these five ways to automate development in FoxPro useful. Assuredly, there are others. Feel free to comment and post articles, tips, or other feedback to round my list out!

There are many ways to parse or tokenize data in VFP. I like using STREXTRACT and GETWORDNUM for example. But there are cases in which the actual parsing needs to be a bit smarter than what these common methods offer. Take a concatenated name field for example. Common to many legacy (and unfortunately, some modern) systems, is a ‘name’ or ‘address1′ field that stores a person or company’s full name. This method is not looked upon favorably by data integrators or people who like atomic, normalized data (like me!). These types of fields don’t work well with indexes either.

Of course, the design solution is to create 5 or more fields that represent a person’s name: prefix / first / middle / last / suffix. These atomic values can then be arranged as needed (”last, first middle”; “first last”, initials only, etc.) in reports and for display. But oftentimes, making this decision now is too late.

An alternative is to write a smart parser that can place each name element into its appropriate box. In the following code example, I provide a solution that can handle this tasks. My only assumption for this code is that the name is stored in prefix / first / middle / last / suffix order. By data profiling, you can determine how true and consistent this assumption is on your data. You may discover that 95% of the file is stored in lastname / firstname order instead. Whatever the case may be, you can use the method below as a starting point.

The following snippet gets us started, and in the end, loops through the names and prints the parsed results to screen:

#DEFINE PREFIX  1
#DEFINE FIRSTNAME  2
#DEFINE MIDDLENAME  3
#DEFINE SURNAME  4
#DEFINE SUFFIX  5
 
CLEAR
 
*-- some names to demonstrate
DIMENSION aMyNames[13]
aMyNames[1] = "Andrew MacNeill"
aMyNames[2] = "Andy Kramek"
aMyNames[3] = "Bernard Bout"
aMyNames[4] = "Calvin Hsia"
aMyNames[5] = "Cesar Chalom"
aMyNames[6] = "Craig Baily"
aMyNames[7] = "Craig Berntson"
aMyNames[8] = "Emerson Santon Reed"
aMyNames[9] = "Eric den Doop"
aMyNames[10] = "gonzomaximus"
aMyNames[11] = "Tod J. McKenna II"
aMyNames[12] = "Dr. Ralph Kimball"
aMyNames[13] = "Miss Lucy Jones"
 
*-- placeholder for tokenized name, passed by reference into split_name
DIMENSION aParts[1]
 
FOR x = 1 TO 13
  cLastFirst = split_name(aMyNames[x], @aParts)
  ? cLastFirst
  ? "firstname: " + aParts[FIRSTNAME] + CHR(9)
  ?? "lastname: " + aParts[SURNAME]
NEXT

Here’s the code in function “split_name”

FUNCTION split_name
LPARAMETERS tcName , taParts
 
*-- variables and arrays I will use
LOCAL cPart AS Character
LOCAL x , nTotParts AS Integer
LOCAL lIsSuffix , lIsPrefix AS Logical
LOCAL ARRAY aTemp[1]
EXTERNAL ARRAY taParts
 
*-- get the data from tcName into an array for easy looping
tcName = STRTRAN(UPPER(ALLT(tcName))," ",CHR(13))
DIMENSION taParts[5]
STORE "" TO taParts
DIMENSION aTemp[1]
FOR x = 1 TO MEMLINES(tcName)
  cPart = ALLTRIM(MLINE(tcName,x))
  IF !EMPTY(cPart)
    aTemp[ALEN(aTemp,1)] = cPart
    IF x < MEMLINES(tcName)
      DIMENSION aTemp[ALEN(aTemp,1)+1]
    ENDIF
  ENDIF
NEXT
 
*-- now loop through temp array and put each name segment into a smart little box
nTotParts = ALEN(aTemp,1)
FOR x = 1 TO nTotParts
  IF VARTYPE(aTemp[x])!="C"
    LOOP
  ENDIF
  *-- get rid of any periods used in the part, and convert to uppercase
  cPart = STRTRAN(ALLTRIM(UPPER(aTemp[x])),".","")
  IF nTotParts = 1
    *-- Madonna? gonzomaximus? Treat single names as a lastname
    taParts[SURNAME] = cPart
  ELSE
    *-- Note: A data profile on a name file I parsed a few years back revealed
    *-- the following common name prefixes. Beef this list up if you have others!
    lIsPrefix = INLIST(cPart,"MR","MRS","MS","MISS","DR","PROF","SIR",;
                  "MASTER","REV","REVERAND","FATHER","SISTER","BROTHER",;
                  "BR","SIS","ATTORNEY","COL","COLONOL","REP","PRES")
    IF lIsPrefix
      taParts[PREFIX] = cPart
      LOOP
    ENDIF
    *-- Note: The same profile mentioned above revealed the following suffixes
    lIsSuffix = INLIST(cPart,"PHD","II","III","IV","V","JR","SR","ESQ","CPA","CEA","DD","MD","JD")
    IF lIsSuffix
      taParts[SUFFIX] = cPart
      LOOP
    ENDIF
    IF x = nTotParts
      IF EMPTY(taParts[SURNAME])
        taParts[SURNAME] = cPart
      ELSE
        IF EMPTY(taParts[FIRSTNAME])
          taParts[FIRSTNAME] = cPart
        ELSE
          taParts[MIDDLENAME] = cPart
        ENDIF
      ENDIF
    ELSE
      IF EMPTY(taParts[FIRSTNAME])
        taParts[FIRSTNAME] = cPart
      ELSE
        IF EMPTY(aParts[MIDDLENAME])
          taParts[MIDDLENAME] = cPart
        ELSE
          taParts[SURNAME] = cPart
        ENDIF
      ENDIF
    ENDIF
  ENDIF
NEXT
 
RETURN ALLTRIM(ALLTRIM(taParts[SURNAME]) + ", " +;
         ALLTRIM(taParts[FIRSTNAME]) + " " + ;
         ALLTRIM(taParts[MIDDLENAME]) )
ENDFUNC

So there you have it. The code above looks at the tcName variable and attempts to dump its parts into the taParts array. Some defined constants exist to help ease the “where the heck am I in this array” confusions (a practice I use for stuff like this all the time). As an added bonus, the code returns the name in last / first middle order, which is usually a good format for display purposes.