Category: Version: FM 8 or later

General, Level: Intermediate, SQL, Version: FM 8 or later

FileMaker’s Internal SQL Engine, part 2

Kevin Frank

When my kids were little, I sometimes found myself echoing that familiar parental refrain: “Just because you can do something, doesn’t mean you should.” And I think a related question can be asked with regard to FileMaker’s internal SQL engine: Given FileMaker’s almost infinite flexibility, why bother with SQL?

Three reasons come immediately to mind: 1) efficiency; 2) power; 3) SQL is the most widely used database language, and worth becoming familiar with. Number 3 is self-explanatory, and I’ll address #2 below, but what do I mean by #1? How is SQL more efficient? In a nutshell, it allows you manipulate data via text commands without having to “establish context” (by going to a particular layout or record), and also without having to add table occurrences or relationships to your Relationships Graph.

Let’s say I have a scheduling solution that ships with demo data. I want to ensure that all dates in the system are current, so that when the user looks at the calendar, they will see appointments for the current month, and dates for invoices, payments, purchases, etc., are contemporary as well. Now there’s no reason this can’t be handled the “traditional way”, but what a load of drudgery.

Go to layout
Show all records
Replace field contents of date field
(repeat as necessary for multiple date fields in a given table)
Go to another layout
Repeat ad nauseam…

You could easily end up producing a mountain of script steps to accomplish what could have instead been done with a single Set Variable script step:

…which contains multiple SQL calls.

Note: I am using the doSQL plug-in in these examples, but the code inside the parentheses would be the same, regardless of which SQL plug-in I chose to use.

A similar situation arises if we want to delete all test data prior to shipping a new version of our product. Sure, we could navigate to umpteen layouts, repeatedly issuing Delete All Records commands, but doesn’t this seem more elegant?

Incidentally, for a couple of the tables, rather than deleting all the records, I delete only the ones that meet the condition imposed by the WHERE clause. And of course the WHERE clause could be much more complex than the one I’m using here.

The two examples we’ve looked at demonstrate the economy of using internal SQL to accomplish tasks that would not pose particular challenges for the average FileMaker developer. But so far we haven’t waved a magic SQL wand to solve a difficult problem. I said in the second paragraph that “power” is one of the reasons to use internal SQL, so let’s look at a problem that is difficult to solve using traditional FileMaker methods. Not impossible, just difficult.

Consider a database with two tables, Donors and Donations, related in the usual manner. We would like to see who is donating during a given date range, how many times and how much.

Rather than using a standard summary report, we want to show the equivalent information inside a portal. So, we need an interface that will allow the user to enter a date range, and then see each unique donor, and the count and total dollar amout of donations made by the donor during the date range, like so:

As I say, it’s possible to do this using “pure” FileMaker, in fact I built this demo when FileMaker 7 was first released in 2004, but it took a fair amount of trial and error to get it working properly.

With internal SQL, on the other hand, this can be knocked out in a few easy steps.

1. Build a multiline key of donor ids for the specified date range.

2. Relate this key to the primary key in the Donors table, and base the portal on that relationship.

3. Define a calculated field in Donors to count the related donations for the specified date range.

4. Define another calculated field in Donors, to sum the related donation amounts for the specified date range.

That’s all there is to it, and if you don’t want to take the time to build it from scratch, you can download this demo: sql-summary-report-in-portal (requires either doSQL, or the SQL plug-in of your choice, but if you don’t use doSQL then obviously you’ll need to modify the plug-in calls accordingly).

Finally, the DateToSQL custom function in the above code samples, will be discussed in my next posting (and the Q custom function was explained here).

Level: Intermediate, SQL, Version: FM 8 or later

FileMaker’s Internal SQL Engine, part 1

Kevin Frank

Recently a client asked me if I could help her produce a somewhat unusual report. Normally reports are created to summarize, aggregate or otherwise analyze data; in this case she wanted to report on the database itself. Specifically, she wanted to see all the tables in her database, along with current record count and most recent modification timestamp per table. Ultimately we decided that a simple table view — rather than a formal report — was all she actually needed.

So, I created a new table, “tables”, and populated the table_name field manually, by typing in the table names. You might think that FileMaker’s “TableNames” function could help here, but the function actually returns table occurrence names, rather than table names. Once that was done, how to populate the other two fields?

The thought of creating 50 cartesian join relationships from my “tables” table to the other tables in the system wasn’t particularly appealing. Equally unappealing was the thought of the scripted convolutions I would need to go through to populate the table, though perhaps I would be able to use Evaluate() or GetField() to simplify things.

But fortunately we were using the MonkeyBread plug-in on this project, so another option readily presented itself. Why not use SQL instead? This would allow us to access data in any table without needing to create any additional relationships. So we did. To populate the record_count field was simply a matter of issuing this replace command:

MBS ( "FM.ExecuteSQL" ;
"SELECT COUNT ( * )
FROM " & Tables::table_name
)

In other words, count the records for each table. What about the most recent modification timestamp? That turned out to be a piece of cake also. Here’s the replace calculation for most_recent_ts:

GetValue (
MBS ( "FM.ExecuteSQL" ;
"SELECT zl_modification_ts
FROM " & Tables::table_name &
" ORDER BY zl_modification_ts DESC"
)
; 1 )

In plain English, for each table sort all the records by timestamp in descending order, and then grab just the first one (i.e., the most recent). The above syntax works, incidentally, because the modification timestamp field in every table is named zl_modification_ts. Consistency. It’s a good thing.

Unfortunately, FileMaker’s SQL implementation does not include the SQL “Limit” function, hence the use of GetValue above. But this illustrates a useful lesson — there’s nothing wrong with combining FileMaker syntax and SQL syntax… use whichever one makes the most sense at the moment.

Incidentally, the portions of code shown in grey above are specific to the MonkeyBread plug-in. If we used a different plug-in, the grey portion would be different, but the remainder of the code would not change. That’s because the numerous plug-ins that allow you to tap into FileMaker’s internal SQL engine are really just providing a conduit for your SQL code.

I’ll have more to say about FileMaker Internal SQL in upcoming postings. (For that matter, I also wrote about it the other day.) In the mean time…

Suggested reading:
1. FileMaker 11 OBDC and JDBC Guide (chapter 7: “Supported Standards”)
2. Wikipedia SQL Entry

Links to some of the SQL plug-in vendors (to the best of my knowledge, these are the only ones that are fully FM11-compliant):
1. MyFMButler (doSQL)
2. CNS (MMQuery)
3. Dracoventions (SQL Runner)

Level: Intermediate, SQL, Version: FM 8 or later

Filtered Relations, part 3

Kevin Frank

Today we’re going to take a third look at the challenge we encountered in part 1 of this series: Given a simple database consisting of Products, Invoices and Invoice Line Items, how can we show total sales per product filtered by date range?

Reminder: what makes this problem challenging is the fact that the date field lives in the Invoices table, but the product foreign key (fk_product) lives in the Line Items table.

To briefly recap, we solved this problem in part 1 at the cost of adding a couple fields and table occurrences, i.e., by leveraging the relational model. And in part 2 we solved this problem by displaying a related summary field inside a one-row filtered portal. Continue reading “Filtered Relations, part 3” →

Level: Beginner, Version: FM 8 or later

Dude, that code is sooooo FM3

Kevin Frank

Recently I saw some code that brought nostalgic tears to my eyes. The goal was to parse the file name from a reference in a container field called, appropriately enough, photo_ref. Here’s an example of what the data viewer showed when pointed at that field:

image:/C:/Client/XYZ Corp/photos/andrew wigan.jpeg

And this is the code the developer had written:

Middle (
   Photos::photo_ref ;
   Position ( Photos::photo_ref ; "/" ; 1 ;
      PatternCount ( Photos::photo_ref ; "/" ) ) + 1 ;
   99999
)

In a nutshell: count the slashes, and then retrieve everything to the right of the final slash. Here in a FileMaker 11 solution was code that could have been written in 1995.

To his credit, the code correctly returned “andrew wigan.jpeg”, but I had to wonder whether the developer was aware that there were several things he could have done to make his life easier (and his code more readable).

First, he could have simplified the code by using Let() to eliminate the multiple references to “Photos::photo_ref”.

Let ( a = Photos::photo_ref ;
Middle (
   a ;
   Position ( a ; "/" ; 1 ; PatternCount ( a ; "/" ) ) + 1 ;
   99999
)
)   //   end let

He could also have moved a few more things up into the Let portion of the calc.

Let ( [
a = Photos::photo_ref ;
b = PatternCount ( a ; "/" ) ;
c = Position ( a ; "/" ; 1 ; b ) + 1
] ;
   Middle ( a ; c ; 99999 )
)   //   end let

I find that to be a heck of a lot more readable than the code we started with. However, there’s a different approach that could be used to solve this problem, which strikes me as being both easier to understand and more elegant.

Let ( [
a = Photos::photo_ref ;
b = Substitute ( a ; "/" ; ¶ ) ;
c = ValueCount ( b )
] ;
   GetValue ( b ; c )
)   //   end let

In other words, convert the reference to a list, by transforming the slashes into hard returns, and then grab the bottom value from that list. Once you get comfortable with this technique, you will find many situations where it comes in handy.

For example, if you use the GetFieldName() function, you know that it returns both the table occurrence name as well as the field name itself, separated by “::” like so:

Invoice::id_customer

What if you just want to extract just the field name? You can use a simplified version of the technique we just finished discussing:

Let ( [
a = GetFieldName ( Invoice::id_customer ) ;
b = Substitute ( a ; "::" ; ¶ )
] ;
   GetValue ( b ; 2 )
) // end let

…and the result is “id_customer”.

Level: Any, Version: FM 8 or later

Logarithms I Have Known And Loved

Kevin Frank

They say you never forget your first time, especially if it’s your only time. Maybe that’s why my first (and, so far, only) logarithm stands out so vividly in my mind.

It was a quiet Thursday in October when the call came. A colleague was building a cat breeding database and wanted advice on how to solve a problem. If a given cat is assigned “position 1” in his or her family tree, the cat’s ancestors can be assigned tree positions like so:

2010-12-31-a

Additionally, each generation can be numbered as follows:

Continue reading “Logarithms I Have Known And Loved” →

Level: Intermediate, Version: FM 8 or later

Reorder Based on Summary Field

Kevin Frank

One of the useful things you can do with a FileMaker summary report is reorder it based on the contents of a summary field. Since this is a bit abstract, especially if you’ve never done it before, let’s look at a concrete example.

Demo file: 2010-12-18-salespeople-by-state

Your organization has sales reps in various US states. In fact, many of the states have multiple reps. Here’s a simple table showing some of the sales reps and the state they’re associated with:

You’d like to generate a report showing the number of reps in each state, so your first step is Continue reading “Reorder Based on Summary Field” →

General, Level: Beginner, Version: FM 8 or later

The Wondrous Bullet Character

Kevin Frank

Do you know how to create a bullet (•) character via the keyboard? It’s easy on the Mac: just press Option 8. It’s a bit more complicated in the Windows world, but once your fingers get the hang of it, you’ll be able to do it without much thought at all.

  1. Hold down the Alt key
  2. Using the number pad, type 0149
  3. Release the Alt key

Step 2 must be performed on the number pad; it won’t work if you use the numbers on the main portion of the keyboard. And if this seems unduly complicated, the upside is that you can create any character this way as long as you know its corresponding ASCII value.

Continue reading “The Wondrous Bullet Character” →

General, Level: Intermediate, Version: FM 8 or later

The Last Day of the Month, part 3

Kevin Frank

Grizzled FileMaker veterans are fond of saying things like, “You ought to know at least three different ways to accomplish any given task.” With that in mind, I hereby submit a third method for calculating the last day of a given month.

Let ( [
theDate = Get ( CurrentDate ) ;
monthNum = Month ( theDate ) ;
yearNum = Year ( theDate ) ;
febLastDay = 28 +
Case (
Mod ( yearNum ; 400 ) = 0 ; 1 ;
Mod ( yearNum ; 100 ) = 0 ; 0 ;
Mod ( yearNum ; 4 ) = 0 ; 1 ;
0
) ;
dayNum = Choose ( monthNum ; "" ; 31 ; febLastDay ; 31 ;
30 ; 31 ; 30 ; 31 ; 31 ; 30 ; 31 ; 30 ; 31
)
] ;

Date ( monthNum ; dayNum ; yearNum )

) // end let

If you read yesterday’s post, you may have noted a resemblance between this calculation and its predecessor, at least as far as the “Let” portion goes. The main difference is the addition of a dayNum variable populated via the Choose() function.

In case you’re not comfortable with Choose(), its format is

Choose (
test ; result if test = 0 { ; result if test = 1 ; result if test = 2... }
)

…where “test” is any non-negative whole number, and the results in braces are optional. At first this function may seem confusing but it turns out to be a very compact replacement for the Case() function, under a strictly defined set of circumstances.

Say, for example, in a table called “test”, you have a field called “score”, which can contain any integer between 0 and 9, and you want convert that value to its corresponding name (“zero,” “one,” “two”, etc.). You could certainly accomplish this with Case() and the statement might look like this:

Case (
test::score = 0 ; "zero" ;
test::score = 1 ; "one" ;
test::score = 2 ; "two" ;
test::score = 3 ; "three" ;
test::score = 4 ; "four" ;
test::score = 5 ; "five" ;
test::score = 6 ; "six" ;
test::score = 7 ; "seven" ;
test::score = 8 ; "eight" ;
test::score = 9 ; "nine"
)

The exact same result can be obtained far more economically thus:

Choose ( test::score ;
"zero" ; "one" ; "two" ; "three" ; "four" ;
"five" ; "six" ; "seven" ; "eight" ; "nine"
)

Essentially, Choose uses test::score as a pointer to the correct “result”, via what’s known as a zero-based index, so a test::score value of 0 corresponds to the first result, a test::score of 1 corresponds to the second result, etc.

In the case of our Last Day of the Month problem, there is no month number of 0, only 1 through 12, so our first result is "" to accommodate the non-existent zero result.

General, Level: Beginner, Version: FM 8 or later

The Last Day of the Month, part 2

Kevin Frank

Yesterday we explored a method to calculate the last day of the month, and by way of introduction I said,

You might be tempted to use a Case() statement, and test each month individually, but then you’d have to engage in some calculation gymnastics to accommodate Feb 29th in leap years.

This time around let’s look at that method, and those alleged “gymnastics”. They turn out to not be terribly convoluted; in fact, the most time-consuming part of this could be simply defining the rules for what constitutes a leap year.

But first let’s pretend there’s no such thing as a leap year or a leap day. In that case we could write our calculation as follows:

   Let ( [
      theDate = Get ( CurrentDate ) ;
      monthName = MonthName ( theDate ) ;
      yearNum = Year ( theDate )
   ] ;
   Case (
      monthName = "January" ; Date ( 1 ; 31 ; yearNum ) ;
      monthName = "February" ; Date ( 2 ; 28 ; yearNum ) ;
      <etc.>

Okay, that was a pleasant interlude, but now back to reality. Superficially, it appears that all years evenly divisible by 4 are leap years, but it’s a bit more complicated than that. A year is a leap year when:

  1. it is evenly divisible by 400
    or
  2. it is evenly divisible by 4 but not evenly divisible by 100

It’s important to know all the rules, because we don’t want our database breaking on February 29th in the year 2400, do we?

So how do we determine “even divisibility”? We do so by using the Mod() function, which returns the remainder when you divide one number by another. If the result of the Mod() operation is zero, then we know the first number is evenly divisible by the second.

For example, Mod ( 2000 ; 400 ) = 0, which passes test #1 above, and tells us that the year 2000 was a leap year. The year 1900 cannot pass either of the above tests, which tells us it was not a leap year. Bearing in mind that FileMaker uses calculation “short-circuiting” (stops calculating as soon as one of its logical tests evaluate as true), here’s a calc that returns a 1 for leap years and otherwise returns a zero:

   Let ( yearNum = Year ( Get ( CurrentDate ) ) ;
      Case (
         Mod ( yearNum ; 400 ) = 0 ; 1 ;
         Mod ( yearNum ; 100 ) = 0 ; 0 ;
         Mod ( yearNum ; 4 ) = 0 ; 1 ;
         0
      )
   )   //   end let

Okay, now that we’ve gotten to the heart of the matter, we can put it all together and write our calculation.

   Let ( [
      theDate = Get ( CurrentDate ) ;
      monthName = MonthName ( theDate ) ;
      yearNum = Year ( theDate ) ;
      possibleLeapDay =
         Case (
            Mod ( yearNum ; 400 ) = 0 ; 1 ;
            Mod ( yearNum ; 100 ) = 0 ; 0 ;
            Mod ( yearNum ; 4 ) = 0 ; 1 ;
            0
         )
   ] ;
   Case (
      monthName = "January" ; Date ( 1 ; 31 ; yearNum ) ;
      monthName = "February" ; Date ( 2 ; 28 + possibleLeapDay ; yearNum ) ;
      monthName = "March" ; Date ( 3 ; 31 ; yearNum ) ;
      monthName = "April" ; Date ( 4 ; 30 ; yearNum ) ;
      monthName = "May" ; Date ( 5 ; 31 ; yearNum ) ;
      monthName = "June" ; Date ( 6 ; 30 ; yearNum ) ;
      monthName = "July" ; Date ( 7 ; 31 ; yearNum ) ;
      monthName = "August" ; Date ( 8 ; 31 ; yearNum ) ;
      monthName = "September" ; Date ( 9 ; 30 ; yearNum ) ;
      monthName = "October" ; Date ( 10 ; 31 ; yearNum ) ;
      monthName = "November" ; Date ( 11 ; 30 ; yearNum ) ;
      monthName = "December" ; Date ( 12 ; 31 ; yearNum )
   )  
   )   //   end let

One of the nice things about yesterday’s elegant little calculation was that we didn’t have to think about leap year at all. We let the FileMaker calculation engine worry about that for us. But there’s something very different I like about today’s calculation… it’s supremely readable. It’s not elegant, and it’s not succinct, but it gets the job done. And that’s one of my favorite things about FileMaker: there is rarely only one way to solve a problem.