ExecuteSQL, Level: Intermediate, SQL, Version: FM 12 or later

FM 12 ExecuteSQL: A Cool Tool

It’s been a fun week experimenting with the ExecuteSQL function, and also seeing what others are doing with it. In some cases the experience has been eye opening, and I strongly recommend that anyone interested in ExecuteSQL grab the SQL Builder file from Eden Morris, available on FMForums (registration required) at http://fmforums.com/forum/files/file/25-sql-builder/

This attractive and powerful tool provides a point and click interface to build complex SQL queries, which are then rendered in FileMaker 12 ExecuteSQL syntax, e.g.,

The current version is 0.4, implying that updates may be forthcoming, so you might want to not only download the demo, but bookmark the above link as well. Very nicely done, Mr. Morris!

ExecuteSQL, Level: Intermediate, SQL, Version: FM 12 or later

FM 12 ExecuteSQL, part 2

One of the fun things about a new FileMaker release is figuring out how new features work, including subtle behavioral nuances. Today’s demo file, FM12 ExecuteSQL Help Example from CW, is based on Corn Walker’s re-working of one of the demos from part 1, and helps demonstrate how ExecuteSQL fits in with the FileMaker security model.

Continue reading “FM 12 ExecuteSQL, part 2”

ExecuteSQL, Level: Intermediate, SQL, Version: FM 12 or later

FM 12 ExecuteSQL, part 1

FileMaker 12 offers many intriguing new features, including new windowing capabilities, significant charting and container field improvements, and a brand new design surface. But to my way of thinking, all of these pale in comparison to the exciting possibilities offered by the new ExecuteSQL function.

What’s so great about ExecuteSQL? In a nutshell, it allows us to natively (without a plug-in) perform queries against FileMaker tables using SQL (structured query language). This means that we can now write code to answer questions that previously required adding new table occurrences and special-purpose fields to serve as relational predicates.

Incidentally, if you’re curious about SQL in general, here are some good starting points:

And of course you’ll want to grab a copy of the FileMaker 12 ODBC/JDBC Guide, which offers tantalizing glimpses of what FileMaker is capable of SQL-wise under the hood. Much of the information in Chapter 7 is directly relevant to the ExecuteSQL function.

Continue reading “FM 12 ExecuteSQL, part 1”

Level: Intermediate, SQL, Version: FM 11 or later

FileMaker 11 Internal SQL Changes, part 2

Welcome back for a second action-packed episode, as we continue exploring changes to the FileMaker 11 internal SQL parser. We ended part 1 by looking at the INSERT/SELECT construction which, as you may recall, provides the SQL equivalent of a FileMaker “add new records” import from one table to another.

Actually, when you use INSERT/SELECT, if you wish, the source and target tables can be the same table (or, rather, table occurrence), which once in a while can come in handy, and is something you cannot do via a regular FileMaker import. For that matter, you can also use INSERT/SELECT to map a single source field to multiple target fields, which is another thing a regular FileMaker import cannot do… but I digress.

Continue reading “FileMaker 11 Internal SQL Changes, part 2”

Level: Advanced, SQL, Version: FM 11 or later

FileMaker 11 Internal SQL Changes, part 1

March 9, 2010 is a date I recall quite clearly: not only was it the release date for  FileMaker 11; it was also the day a bunch of my FileMaker SQL code broke, due to changes in the FM 11 internal SQL parser.

If you’re new to this subject, or perhaps a bit rusty, I have written about internal SQL (a.k.a. FQL) on various occasions over the last year, and there are a number of important points that I won’t discuss today, because they have already been mentioned in one or more of these articles:

What follows is by no means exhaustive, but merely what I’ve documented. There is no comprehensive source of information about FQL, although chapter 7 of the FileMaker 11 OBDC and JDBC Guide is a good place to start (just remember that not everything you read there will apply to FQL).

Continue reading “FileMaker 11 Internal SQL Changes, part 1”

Level: Intermediate, SQL, Version: FM 10 or later

Custom Functions for Internal SQL

Recently I’ve mentioned several custom functions that can make your life easier when working with FileMaker’s internal SQL engine (a.k.a. FQL). This article’s accompanying demo file, SQL Sandbox, includes some sample data, an interface for testing SQL code, a number of code examples, and the following four custom functions:

Q – wraps an expression in single quotes. I created this because I hate visually parsing single quotation marks. As I wrote last month,

...first I want digress for a moment to complain about discuss single quotes, a.k.a. apostrophes. They can be difficult to see when they appear adjacent to double quotes — I liken them to an annoying cloud of gnats — so to avoid confusion, I wrote a custom function which I call simply “Q”:

In a nutshell, this CF supplies the surrounding single quotation marks so we don’t have to type them or think about them, and ironically, the above image illustrates the problem perfectly. A single quote surrounded by double quotes is extremely difficult to parse visually. But once you’ve written the CF, you may never have to deal with this problem ever again (until you edit another developer’s SQL code at any rate).

Additionally, this CF escapes any internal single quotes or apostrophes by prepending another one… e.g., O'Malley becomes O''Malley (yes, this is a good thing if you don’t want SQL code to break… don’t worry: the extra apostrophe won’t show up in your data).

GFN – like GetFieldName(), but only returns the field name itself — essential if you want to protect your code from the hazards of field renaming. Note: since this CF uses the GetFieldName function, it requires FM 10 or later.

Guess which one breaks when someone decides to rename creditCardType to ccType, or idCustomer to idContact? On a related note, in addition to protecting you from ex post facto field renaming, GFN also prevents you from misspelling or entering non-existent field names in the first place.

Also, FQL has over 250 reserved words, including “date”, “action” and “group” — you can see the complete set of reserved words on pages 55–57 of the FileMaker 11 OBDC and JDBC Guide. If you’ve used a reserved word as a field name in your FM database, you normally have to remember to wrap that field name in double quotes… GFN does away with all that by automatically wrapping your field name in quotes for you. There’s no harm done if the field name isn’t a reserved word, and that way you don’t need to worry about it.

(I once wasted an hour debugging a SQL statement referencing a field named “group”, before realizing “group” was a reserved word. An hour of my life, gone forever. But it inspired me to add the quote wrapper to GFN, so something good came out of it.)

GFN’s protective quote wrapper will also deliver you from certain other evils, e.g., in FM 11 (with its newly-rewritten SQL parser), the necessity to escape field names if they begin with an underscore, e.g., “_pk_serial”.

DateToSQL – takes a date in a variety of input formats, and outputs it in this format: DATE '2011-02-28', which is apparently the only format that is acceptable to both the new (FM 11) SQL parser, and the SQL parser used in FM 10 and earlier versions.

The input formats are: US-style dates (MM/DD/YYYY), European-style dates (DD/MM/YYYY), and SQL-style dates (YYYY-MM-DD). For example, say you want to grab a list of foreign keys for all customers who have placed an order in the last 30 days…

Assuming today’s date is 26 March 2011, the final line will be rendered as DATE '2011-02-24' and the SQL gods will smile on you. It’s true, they will.

The preceding code examples have consisted of simple SELECT statements; here’s a more complex UPDATE statement, utilizing all three of the CFs we’ve looked at so far. The aim here is to update the tax rate on any unbilled or future-dated invoice line items, if the user changes the default sales tax rate in Settings (this is a single-user system, so I have the luxury of not having to worry about potential record locks by other users).

And finally, we come to a custom function I don’t actually need very often, but that sure comes in handy when I do…

FoundSetToSQL – the concept of a found set does not exist in SQL, but sometimes we need a SQL command to operate only on the records in our current found set. What to do? Well, it turns out that we can say, “Hey, SQL, only process records if the value in a certain field matches one of the values in this list: a,b,c,d,e,…”, and we can build that list from values in the current FileMaker found set. Take for example this found set of 14 records:

FoundSetToSQL ( customers::id ) will return the following…

'000940','000540','001443','001545','000458','000639',
'000794','001629','000048','000973','000735','001139',
'001224','001449'

…as one unbroken string of text. And this can be used by the SQL “IN” operator as part of a WHERE clause, as per this statement, which generates a list of unique cities in the found set.

Warning: FoundSetToSQL is a) not fast with large found sets, and b) subject to the standard 10,000 iteration recursion limit… you don’t want to point it at a found set with more than 10,000 records. There are various ways to work around these limitations, but that is a topic for another day (the second half of this article addresses ways to build a multiline key of values from the current found set).

In closing, I will risk stating the obvious: these custom functions are not the last word on the subject by any means (especially FoundSetToSQL). Feel free modify and extend them as you see fit, and as always, use at your own risk.

FoundSetToSQL ( customers::id )’000940′,’000540′,’001443′,’001545′,’000458′,’000639′,’000794′,’001629′,’000048′,’000973′,’000735′,’001139′,’001224′,’001449′
General, Level: Intermediate, SQL, Version: FM 8 or later

FileMaker’s Internal SQL Engine, part 2

When my kids were little, I sometimes found myself echoing that familiar parental refrain: “Just because you can do something, doesn’t mean you should.” And I think a related question can be asked with regard to FileMaker’s internal SQL engine: Given FileMaker’s almost infinite flexibility, why bother with SQL?

Three reasons come immediately to mind: 1) efficiency; 2) power; 3) SQL is the most widely used database language, and worth becoming familiar with. Number 3 is self-explanatory, and I’ll address #2 below, but what do I mean by #1? How is SQL more efficient? In a nutshell, it allows you manipulate data via text commands without having to “establish context” (by going to a particular layout or record), and also without having to add table occurrences or relationships to your Relationships Graph.

Let’s say I have a scheduling solution that ships with demo data. I want to ensure that all dates in the system are current, so that when the user looks at the calendar, they will see appointments for the current month, and dates for invoices, payments, purchases, etc., are contemporary as well. Now there’s no reason this can’t be handled the “traditional way”, but what a load of drudgery.

Go to layout
Show all records
Replace field contents of date field
(repeat as necessary for multiple date fields in a given table)
Go to another layout
Repeat ad nauseam…

You could easily end up producing a mountain of script steps to accomplish what could have instead been done with a single Set Variable script step:

…which contains multiple SQL calls.

Note: I am using the doSQL plug-in in these examples, but the code inside the parentheses would be the same, regardless of which SQL plug-in I chose to use.

A similar situation arises if we want to delete all test data prior to shipping a new version of our product. Sure, we could navigate to umpteen layouts, repeatedly issuing Delete All Records commands, but doesn’t this seem more elegant?

Incidentally, for a couple of the tables, rather than deleting all the records, I delete only the ones that meet the condition imposed by the WHERE clause. And of course the WHERE clause could be much more complex than the one I’m using here.

The two examples we’ve looked at demonstrate the economy of using internal SQL to accomplish tasks that would not pose particular challenges for the average FileMaker developer. But so far we haven’t waved a magic SQL wand to solve a difficult problem. I said in the second paragraph that “power” is one of the reasons to use internal SQL, so let’s look at a problem that is difficult to solve using traditional FileMaker methods. Not impossible, just difficult.

Consider a database with two tables, Donors and Donations, related in the usual manner. We would like to see who is donating during a given date range, how many times and how much.

Rather than using a standard summary report, we want to show the equivalent information inside a portal. So, we need an interface that will allow the user to enter a date range, and then see each unique donor, and the count and total dollar amout of donations made by the donor during the date range, like so:

As I say, it’s possible to do this using “pure” FileMaker, in fact I built this demo when FileMaker 7 was first released in 2004, but it took a fair amount of trial and error to get it working properly.

With internal SQL, on the other hand, this can be knocked out in a few easy steps.

1. Build a multiline key of donor ids for the specified date range.

2. Relate this key to the primary key in the Donors table, and base the portal on that relationship.

3. Define a calculated field in Donors to count the related donations for the specified date range.

4. Define another calculated field in Donors, to sum the related donation amounts for the specified date range.

That’s all there is to it, and if you don’t want to take the time to build it from scratch, you can download this demo: sql-summary-report-in-portal (requires either doSQL, or the SQL plug-in of your choice, but if you don’t use doSQL then obviously you’ll need to modify the plug-in calls accordingly).

Finally, the DateToSQL custom function in the above code samples, will be discussed in my next posting (and the Q custom function was explained here).

Level: Intermediate, SQL, Version: FM 8 or later

FileMaker’s Internal SQL Engine, part 1

Recently a client asked me if I could help her produce a somewhat unusual report. Normally reports are created to summarize, aggregate or otherwise analyze data; in this case she wanted to report on the database itself. Specifically, she wanted to see all the tables in her database, along with current record count and most recent modification timestamp per table. Ultimately we decided that a simple table view — rather than a formal report — was all she actually needed.

So, I created a new table, “tables”, and populated the table_name field manually, by typing in the table names. You might think that FileMaker’s “TableNames” function could help here, but the function actually returns table occurrence names, rather than table names. Once that was done, how to populate the other two fields?

The thought of creating 50 cartesian join relationships from my “tables” table to the other tables in the system wasn’t particularly appealing. Equally unappealing was the thought of the scripted convolutions I would need to go through to populate the table, though perhaps I would be able to use Evaluate() or GetField() to simplify things.

But fortunately we were using the MonkeyBread plug-in on this project, so another option readily presented itself. Why not use SQL instead? This would allow us to access data in any table without needing to create any additional relationships. So we did. To populate the record_count field was simply a matter of issuing this replace command:

MBS ( "FM.ExecuteSQL" ;
"SELECT COUNT ( * )
FROM " & Tables::table_name
)

In other words, count the records for each table. What about the most recent modification timestamp? That turned out to be a piece of cake also. Here’s the replace calculation for most_recent_ts:

GetValue (
MBS ( "FM.ExecuteSQL" ;
"SELECT zl_modification_ts
FROM " & Tables::table_name &
" ORDER BY zl_modification_ts DESC"
)
; 1 )

In plain English, for each table sort all the records by timestamp in descending order, and then grab just the first one (i.e., the most recent). The above syntax works, incidentally, because the modification timestamp field in every table is named zl_modification_ts. Consistency. It’s a good thing.

Unfortunately, FileMaker’s SQL implementation does not include the SQL “Limit” function, hence the use of GetValue above. But this illustrates a useful lesson — there’s nothing wrong with combining FileMaker syntax and SQL syntax… use whichever one makes the most sense at the moment.

Incidentally, the portions of code shown in grey above are specific to the MonkeyBread plug-in. If we used a different plug-in, the grey portion would be different, but the remainder of the code would not change. That’s because the numerous plug-ins that allow you to tap into FileMaker’s internal SQL engine are really just providing a conduit for your SQL code.

I’ll have more to say about FileMaker Internal SQL in upcoming postings. (For that matter, I also wrote about it the other day.) In the mean time…

Suggested reading:
1. FileMaker 11 OBDC and JDBC Guide (chapter 7: “Supported Standards”)
2. Wikipedia SQL Entry

Links to some of the SQL plug-in vendors (to the best of my knowledge, these are the only ones that are fully FM11-compliant):
1. MyFMButler (doSQL)
2. CNS (MMQuery)
3. Dracoventions (SQL Runner)

Level: Intermediate, SQL, Version: FM 8 or later

Filtered Relations, part 3

Today we’re going to take a third look at the challenge we encountered in part 1 of this series: Given a simple database consisting of Products, Invoices and Invoice Line Items, how can we show total sales per product filtered by date range?

Reminder: what makes this problem challenging is the fact that the date field lives in the Invoices table, but the product foreign key (fk_product) lives in the Line Items table.

To briefly recap, we solved this problem in part 1 at the cost of adding a couple fields and table occurrences, i.e., by leveraging the relational model. And in part 2 we solved this problem by displaying a related summary field inside a one-row filtered portal. Continue reading “Filtered Relations, part 3”