Maintaining Your iMIS Database - NiUG · • Determine backup strategy (CSI methodology) – Create a single Backup Device (file) – Create a single Log Device (file) – Nightly

Maintaining Your iMIS Database

Monday, May 22nd1:00PM to 1:30PM

Doug Morris, CSI

• Some SQL Knowledge• Access to the SQL Server• Lots of iMIS knowledge• A system that has been around for a while• Been blamed repeatedly when things go wrong• Lots of hands in his/her database (vendors, staff, etc.)• A strong desire to get out of here early and meet at the bar,

where the presenter will be forced to buy drinks

The perfect attendee has…

• How we got here• How we get there• How a SQL Server works…

– Backups, understanding SQL and best practices• Common database issues

– Address Corruption– Trans/Invoice/Orders mismatches– Name_Security Issues– ASP_Net Issues

• PCI Compliance options• Best practices for the future

Agenda

• Past bugs in iMIS• Multiple vendors touching the data – often with direct SQL

access• Multiple vendors touching the data, sometimes even

correctly (see #1 above)• Time

How we got here

• Use some of the what you learn today to go back and clean up your house

• Use all of what you learn today to help prevent future issues– Use approved methodologies to access the data– Let ASI know if/when you find an issue (and can duplicate it)

• Revisit this list in a year

How we get there

How a SQL Server works

File(s) on your server(.LDF)

File on your server(.MDF)

• Many types of operations are recorded in the transaction log. These operations include:– The start and end of each transaction.– Every data modification (insert, update, or delete). This

includes changes by system stored procedures or data definition language (DDL) statements to any table, including system tables.

– Every extent and page allocation or deallocation.– Creating or dropping a table or index.

Focus on the Transaction Log…

• At some point…. The server gets full• And it is your fault

If the transaction log has every transaction….

• Simple Recovery Mode– Transaction Log is truncated approximately every 15 seconds

• Full Recovery Mode– Your job to truncate the transaction log

You have two options

SQL Recovery Options

SIMPLE

• Set and forget• Transaction Log FILE can

still grow (e.g. during an upgrade)

• No restore to a point in time

FULL

• Transaction log must be truncated on a scheduled basis

• FILE can still grow (e.g. during an upgrade)

• Ability to restore to a point in time

Which version do you think most hosting companies like?Which version do you think is best for Live? Which version do you think is best for Test?

From SSMS, right mouse click on your database and choose Properties

Recovery Mode

Choose Options

• Confirm your settings match these

While we are here…

• Determine backup strategy– SQL Maintenance Plan

Transition to Full Recovery Mode

• Determine backup strategy (CSI methodology)– Create a single Backup Device (file)– Create a single Log Device (file)– Nightly backup the DB (this includes both DB and Log)– Every morning Backup the Transaction Log and Initialize the Log

Device (overwrite it)– Every hour, up until the nightly backup, add to the Transaction Log

Backup Device

Transition to Full Recovery Mode

Backup Options for a Full Recovery

CSI

• Only two files to worry about– DB Backup Device– Log Backup Device

• Need to recover backups if more than one day old

• Saves space on Server

Maintenance Plan

• Multiple days worth of backups

• SQL Takes care of cleanup• Additional space required

• Q1 – What time did you do that at? – “I don’t remember” (true story)

• Q2 – What time do you want us to restore to?• Q3 – What kind of beverage are you going to buy me for

saving your butt?

I just ruined my Name table…

• Always use BEGIN TRAN and COMMIT TRAN or ROLLBACK TRAN when updating ANY database – including test.

Tip of the Day

OMG!! Thank you Doug!

Tip of the Day (continued)

• (A) Full Recovery Mode• (B) Simple Recovery Mode• (C) BOTH

• Answer… (C) BOTH! Because the transaction log is only truncated after transactions are committed or rolled back.

Tip of the Day works with?

• Full Recovery mode should be a must for everyone in this room

• Backing up the Database log truncates it• Don’t turn it on until you

– Setup a proper backup plan– Confirm your server has the space for it– Confirm your hosting provider does not already offer this– Confirm your backup system is not doing some magic with SQL.

You do NOT want two backup plans running– Talk to your solution provider

• Server snapshots are great for only one thing – full recovery of the server at the time the snapshot was taken

In Summary

• iMIS supports three addresses (PURPOSES) that can be marked preferred mail/bill/ship

• iMIS supports unlimited additional addresses• iMIS saves the preferred mail, FULL_ADDRESS, CITY,

STATE_PROVINCE, COUNTRY in the Name table.• Only one address per ID can be marked preferred (mail, bill,

ship)• There always is a record in Name_Address for the first

PURPOSE – even if it is blank• Duplicate PURPOSES not allowed

Address Corruption – The basics of iMIS

• We have one address marked Preferred Mail and Bill

• No Street Address• A home address marked

Preferred Ship

What does it look like?

What does it look like?

Name Table

Name_Address Table

Pay close attention to ADDRESS_NUM_2

1. ID MismatchesLinking on ADDRESS_NUM_1,2,3, BILL_ADDRESS_NUM,

MAIL_ADDRESS_NUM, SHIP_ADDRESS_NUM, see if the ID matches!

Find the issues

--IDs for Name_Address and Name do not matchSelect Name.ID, Name.ADDRESS_NUM_1 from Name INNER JOIN Name_Address

ON Name_Address.ADDRESS_NUM = Name.ADDRESS_NUM_1WHERE Name.ID <> Name_Address.ID

Select Name.ID, Name.ADDRESS_NUM_2 from Name INNER JOIN Name_AddressON Name_Address.ADDRESS_NUM = Name.ADDRESS_NUM_2WHERE Name.ID <> Name_Address.ID

Select Name.ID, Name.ADDRESS_NUM_3 from Name INNER JOIN Name_AddressON Name_Address.ADDRESS_NUM = Name.ADDRESS_NUM_3WHERE Name.ID <> Name_Address.ID

Select Name.ID, Name.MAIL_ADDRESS_NUM from Name INNER JOIN Name_AddressON Name_Address.ADDRESS_NUM = Name.MAIL_ADDRESS_NUMWHERE Name.ID <> Name_Address.ID

Select Name.ID, Name.BILL_ADDRESS_NUM from Name INNER JOIN Name_AddressON Name_Address.ADDRESS_NUM = Name.BILL_ADDRESS_NUMWHERE Name.ID <> Name_Address.ID

Select Name.ID, Name.SHIP_ADDRESS_NUM from Name INNER JOIN Name_AddressON Name_Address.ADDRESS_NUM = Name.SHIP_ADDRESS_NUMWHERE Name.ID <> Name_Address.ID

ID Mismatches

2. BAD PREFERRED FlagUsing MAIL, BILL, and SHIP ADDRESS_NUM confirm the PREFERRED_MAIL, PREFERRED_BILL, and PREFERRED_SHIP is set correctly

Find the issues

--Find Addresses that do not have preferred mail set rightSelect Name.ID, Name.MAIL_ADDRESS_NUM from Name INNER JOINName_Address

ON Name_Address.ADDRESS_NUM = Name.MAIL_ADDRESS_NUMWHERE Name_Address.PREFERRED_MAIL = 0

--Find Addresses that do not have preferred bill set rightSelect Name.ID, Name.BILL_ADDRESS_NUM from Name INNER JOINName_Address

ON Name_Address.ADDRESS_NUM = Name.BILL_ADDRESS_NUMWHERE Name_Address.PREFERRED_BILL = 0

--Find Addresses that do not have preferred ship set rightSelect Name.ID, Name.SHIP_ADDRESS_NUM from Name INNER JOINName_Address

ON Name_Address.ADDRESS_NUM = Name.SHIP_ADDRESS_NUMWHERE Name_Address.PREFERRED_SHIP = 0

Bad PREFERRED flags in Name_Address

3. Missing records in Name_AddressThe Name table is pointing to records in Name_Address that do not exist!

Find the issues

--Find Missing Address Nums in Name_AddressSelect Name.ID, Name.ADDRESS_NUM_1 from Name LEFT JOIN Name_Address

ON Name.ADDRESS_NUM_1 = Name_Address.ADDRESS_NUMWHERE Name_Address.ADDRESS_NUM is null

Select Name.ID, Name.ADDRESS_NUM_2 from Name LEFT JOIN Name_AddressON Name.ADDRESS_NUM_2 = Name_Address.ADDRESS_NUMWHERE Name_Address.ADDRESS_NUM is null

Select Name.ID, Name.ADDRESS_NUM_3 from Name LEFT JOIN Name_AddressON Name.ADDRESS_NUM_3 = Name_Address.ADDRESS_NUMWHERE Name_Address.ADDRESS_NUM is null

Select Name.ID, Name.MAIL_ADDRESS_NUM from Name LEFT JOIN Name_AddressON Name.MAIL_ADDRESS_NUM = Name_Address.ADDRESS_NUMWHERE Name_Address.ADDRESS_NUM is null

Select Name.ID, Name.BILL_ADDRESS_NUM from Name LEFT JOIN Name_AddressON Name.BILL_ADDRESS_NUM = Name_Address.ADDRESS_NUMWHERE Name_Address.ADDRESS_NUM is null

Select Name.ID, Name.SHIP_ADDRESS_NUM from Name LEFT JOIN Name_AddressON Name.SHIP_ADDRESS_NUM = Name_Address.ADDRESS_NUMWHERE Name_Address.ADDRESS_NUM is null

Missing Name_Address Records

4. Duplicate PREFERRED Address RecordsMore than one Address per ID is marked Preferred MAIL/BILL/SHIP

Find the issues

--Find Duplicate PREFERRED_MAILSelect ID from Name_Address

WHERE Name_Address.PREFERRED_MAIL = 1GROUP BY IDHAVING COUNT(*)>1

--Find Duplicate PREFERRED_BILLSSelect ID from Name_Address

WHERE Name_Address.PREFERRED_BILL = 1GROUP BY IDHAVING COUNT(*)>1

--Find Duplicate PREFERRED_SHIPSelect ID from Name_Address

WHERE Name_Address.PREFERRED_SHIP = 1GROUP BY IDHAVING COUNT(*)>1

Duplicate Address Records (1)

5. Duplicate PREFERRED Address Records by PURPOSEMore than one Name_Address record per PURPOSE/ID

Find the issues

--Find Duplicate Address PurposesSelect ID, PURPOSE from Name_Address

GROUP BY ID, PURPOSEHAVING COUNT(*)>1

Duplicate Address Records (2)

1. Don’t run the rebuild Name_Addres Pointers script2. Do investigate each and every issue – if you see a pattern,

you can sometimes clean up easily3. REMEMBER BEGIN TRAN4. REMEMBER TO COMMITT THE TRANSACTIONS

Cleaning it all up

• Duplicate purposes• Bad/missing pointers• Bad/missing flags

Let’s work through some samples

• Leading/Trailing spaces in Strings (varchar fields) are no Bueno• You might/will have these if you have had iMIS for a long long time

(think SQL 4.21)– In those days, a blank field was a single space

• Disable triggers before runningDISABLE TRIGGER asi_Name_Delete on NamegoDISABLE TRIGGER asi_Name_Insert_Update on NamegoDISABLE TRIGGER InformzDeleteName on NamegoDISABLE TRIGGER InformzInsertName on NamegoDISABLE TRIGGER InformzUpdateName on Namego

Tip of the Day – Part Deux!

• Update the fieldBEGIN TRANUPDATE Name SET BT_ID=LTRIM(RTRIM(BT_ID))COMMIT TRAN

Tip of the Day – Part Deux!

• “Hey, my trial balance says a member owes money, but iMIS says she doesn’t. Can you figure out what is going on?– Invoice and Trans do not match

• “Hey, I’m looking at an event registration where it says the member owes money, but the AR/Cash and Trial balance do not show this?– Orders does not match Invoice

Trans/Invoice/Orders Corruption

• Invoice table is “Header”• Trans table is “the Detail”• Orders table is “just there to annoy you when out of whack”• ENT_ARCheckbal.SQL is the script of choice

Trans/Invoice/Orders Corruption

Trans/Invoice Corruption/Balance Issues

Trans/Invoice Corruption/Balance Issues• You need to determine

– Is the Invoice table just wrong? (easy fix)– Is the Trans table wrong?

• Are there missing transactions?• The Trans.AMOUNT for a given Transaction should sum to zero (does

it?) If not, your issue is with the Trans table

Fixing the Invoice Line

FRIK!

Fixing the Invoice Line

• From Travis Campbell at PAMS (Aussie dude)• This script looks for all duplicates where one as a security group of

anonymous and removes it.

BEGIN TRANdelete nsgfrom Name_Security_Groups nsginner join (

SELECT ID, COUNT(*) AS dupeCountFROM Name_Security_GroupsGROUP BY IDHAVING COUNT(*) > 1

) nsg1 on nsg.id = nsg1.idwhere nsg1.dupecount = '2' and SECURITY_GROUP = 'anonymous'COMMIT TRAN

Name Security Issues

Missing User Credentials(user definition is damaged)

• Cause someone put it in the Description

PCI Stuff

• Turn off “Allow non-referenced transactions” with PayFlow Pro– This will require process changes on your end.

• From the DB Maintenance Utility– Purge All Cardholder Info

• From iMIS, under AR/Cash –Setup Module

– Do not retain cardholder info

PCI Stuff (Best Practices)

• Any must have best practices?

From the audience…

Questions?

Doug MorrisComputer System Innovations, Inc. (CSI)[email protected]#dougcsi

Thank you!

Strategic Partners

Corporate PartnersAmericaneagle.com • Ascension Technology Solutions, LLC • Association Technologies, Inc. • Bursting Silver CadmiumCD • enSYNC Corporation • Informz • ISG Solutions • MemberPrime • RSM US LLP • Visual Antidote

Event PartnersArmstrong Enterprise Communications • Computer System Innovations • C Systems Global • iFINITY

Intuitive Business Concepts • Lane Services, LLC • Source of Knowledge (Official Recording Partner) • TGD Communications

Thank You To Our Sponsors