FM Web Scraping FMPUG: Dallas Chapter Taylor Made Services: FileMaker Presentation March 6, 2009 Dallas Texas.

FM Web ScrapingFM Web Scraping

FMPUG: Dallas ChapterFMPUG: Dallas ChapterTaylor Made Services: Taylor Made Services: FileMaker PresentationFileMaker Presentation

March 6, 2009 • Dallas • TexasMarch 6, 2009 • Dallas • Texas

What is FM Web Scraping?What is FM Web Scraping?

FileMaker Web Scraping is the copying of source code from an http file.

It is the same thing as going to a web page and viewing the source code.

It is also the same thing as copying an http file, which is usually an html or txt file.

Scrape vs. ViewScrape vs. View

Use Web Viewer when all you Use Web Viewer when all you need is to view the information.need is to view the information.

Scrape a page when you need to Scrape a page when you need to store or manipulate data from that store or manipulate data from that page.page.

Skill Set NeededSkill Set Needed

Understanding of HTML PagesUnderstanding of HTML Pages

Familiarity with Calculations, and Familiarity with Calculations, and in particular, Text Functionsin particular, Text Functions

Knowledge of VariablesKnowledge of Variables

Ability to Create ScriptsAbility to Create Scripts

Web Viewer ObjectWeb Viewer Object

Create a LayoutCreate a Layout

Add a Web Viewer ObjectAdd a Web Viewer Object

Give the Object a NameGive the Object a Name

URL Code“scraped”From Web

Page

Web PageWith

Object Name

Create URLCreate URL Open Web Viewer SetupOpen Web Viewer Setup

Enter URL or Enter URL or URL calculation URL calculation into the Web into the Web AddressAddress

Create URLCreate URL Search for desired data in Search for desired data in

Web PageWeb Page

View the Source Code and View the Source Code and locate the desired datalocate the desired data

Text Calculation TimeText Calculation Time

Do you want to Store the Source Code?Do you want to Store the Source Code?

You can have it refresh the data each time it loads You can have it refresh the data each time it loads the web viewer.the web viewer.

Fresh Data, but doesn’t load quicklyFresh Data, but doesn’t load quickly

You can have a calculation or script step save the You can have a calculation or script step save the source code into a text field.source code into a text field.

Loads Quickly, but might not be “fresh” dataLoads Quickly, but might not be “fresh” data

Text Calculations from Source CodeText Calculations from Source Code Result can be scraped with:Result can be scraped with:

GetLayoutObjectAttribute(“html source” ; “content”)GetLayoutObjectAttribute(“html source” ; “content”)

Result can be a direct calculation from the Result can be a direct calculation from the source codesource code

Delay issuesDelay issues

Calculate warning if whole page not Calculate warning if whole page not loadedloaded

ChallengesChallenges Web pages do not load quickly Web pages do not load quickly

and script steps need delays built and script steps need delays built into them to allow the load times.into them to allow the load times.

Web Pages change!Web Pages change!

Build in Warning CalculationBuild in Warning Calculation

Is the URL available to all users?Is the URL available to all users?

Blank SlideBlank Slide