Search features and architecture in DNN 7.1

@DNNCon @ashishprasad

Don’t forget to include #DNNCon in your tweets!

7.1 Search and Lucene.Net

Ash Prasad

@DNNCon @ashishprasad

Don’t forget to include #DNNCon in your tweets!

• History and New Objectives • Architecture• Lucene / Lucene.Net• Crawlers, Entities, Controllers• Ranking, Synonyms, Ignore Words,

Stemming• Security Trimming• Module Integration, New Crawler

Agenda

@DNNCon @ashishprasad

Don’t forget to include #DNNCon in your tweets!

• Platform Edition• SQL Server• ISearchable

• Commercial Edition• Lucene 2.9.2• URL and Files

History of Search

Lucene

Scheduler

SQL

Scheduler Module

Module

ISearchable

@DNNCon @ashishprasad

Don’t forget to include #DNNCon in your tweets!

• Handle diverse Content • CMS, Social, Localized, 3rd Party

Modules)

• Consistent User Experience• Simple for Module Developers• Uniform Architecture • Feature based differentiation

Objectives of New Search

@DNNCon @ashishprasad

Don’t forget to include #DNNCon in your tweets!

Architecture

@DNNCon @ashishprasad

Don’t forget to include #DNNCon in your tweets!

• Java-based indexing and search technology

• Managed by Apache• NOSQL database• Near real-time, Spellchecking,

Highlighting, Ranking, Synonyms

• Many companies use Lucene directly or customize

• Facebook’s Graph search uses

similar ‘Inverted Index’

What’s Lucene

@DNNCon @ashishprasad

Don’t forget to include #DNNCon in your tweets!

• Line-by-line port from Java to C#• Maintains high-performance requirements• A bit behind Java releases• Who Uses Lucene.Net• Products - RavenDB, Orchard, Umbraco,

SubText• Commercial Sites – BBC UK Top Gear,

AutoDesk, Koders.Com

What’s Lucene.Net

@DNNCon @ashishprasad

Don’t forget to include #DNNCon in your tweets!

• Flexible Schema

• Consists of Documents• Which are collection of Fields

• Documents can have different set of Fields• Field(“ID”,”xxx-yyy-999”), Field(“Title”,

“My best doc”)• Field(“Owner”,”Ash”),

Field(“Locale”,”en-US”)

Lucene – A Document Store

@DNNCon @ashishprasad

Don’t forget to include #DNNCon in your tweets!

• Denormalized (No Referential Integrity)

• Deletion – Done through a flag• Compact reclaims deleted space

• Update is Delete + Insert • Boost = Ranking• Unicode compliant

Lucene – A Document Store (Contd.)

@DNNCon @ashishprasad

Don’t forget to include #DNNCon in your tweets!

Book consulted for Search

• Book on version 3.0

• ~ 500 pages• Very useful

@DNNCon @ashishprasad

Don’t forget to include #DNNCon in your tweets!

Search Phases

Content Acquisition• Crawling• ISearchable• ModuleSearchBase• URL• Doc / PDF

Content Indexing• Text Analysis• Ranking• Synonyms• Ignore Words• Stemming

Content Search• Querying• Sorting• Security Trimming• Boolean Search• Highlighting

@DNNCon @ashishprasad

Don’t forget to include #DNNCon in your tweets!

• Platform• Site Crawler• Module and Tab Metadata• Module Content

(ModuleSearchBase/ISearchable)

• Commercial Edition• File Crawler • Uses IFilter for extraction of text

PDF/Office files

• URL Crawler• Internal and External URLs

Crawlers

@DNNCon @ashishprasad

Don’t forget to include #DNNCon in your tweets!

• SearchType• Distinguishes Crawlers

• SearchDocument• Properties for a Content• Stored in the Index

• SearchQuery• Parameters to execute a Query

• SearchResult• Derived from SearchDocument

Search Entities

@DNNCon @ashishprasad

Don’t forget to include #DNNCon in your tweets!

Search Entities – Indexing vs. Querying

@DNNCon @ashishprasad

Don’t forget to include #DNNCon in your tweets!

• SearchController• For Querying

• InternalSearchController• For Adding / Updating / Deleting

• LuceneController• Interacts with Lucene

Controllers

@DNNCon @ashishprasad

Don’t forget to include #DNNCon in your tweets!

• Doc and/or Field can be boosted in Lucene

• DNN does Field boosts (Default - 10)• Title (50)• Tag (40)• Keyword (35)• Description (20)• Author (15)

• Configured manually by HostSettings

Ranking = Boosting

@DNNCon @ashishprasad

Don’t forget to include #DNNCon in your tweets!

• Synonyms are injected into Index

• Ignore Words are removed from Index

Synonyms and Ignore Words

@DNNCon @ashishprasad

Don’t forget to include #DNNCon in your tweets!

• Convert words to its root• PorterStemFilter is used• Country and Countries = countri• breathe, breathes, breathing,

breathed = breath• fishing, fished, fisher = fish

Stemming

@DNNCon @ashishprasad

Don’t forget to include #DNNCon in your tweets!

• Done through Collectors (Callback)

• Each Doc found is sent to Collector

• Collector rejects/accept per Permission

• Site Crawler - Module / Tab Permission

• File Crawler - Folder Permission• User Crawler – Profile

Permission

Security Trimming

@DNNCon @ashishprasad

Don’t forget to include #DNNCon in your tweets!

• ModuleSearchBase • New abstract class with just one

method• Defined in BusinessControllerClass• GetModifiedSearchDocuments• Returns New, Changed and Deleted

content• Delta based• Granular Permission, Localization, etc.

• ISearchable continues to work (no delta)

Module Integration

@DNNCon @ashishprasad

Don’t forget to include #DNNCon in your tweets!

• Define a new SearchType• Optionally use IsPrivate to hide

from site search

• Implement BaseResultController (2 methods)• HasViewPermission• GetDocUrl

• Create Scheduled Task• Call AddSearchDocuments to inject

content

New Crawler (How to)

@DNNCon @ashishprasad

Don’t forget to include #DNNCon in your tweets!

Demo

@DNNCon @ashishprasad

Don’t forget to include #DNNCon in your tweets!

• New Search uses Lucene.Net• Platform has Site Crawler • Commercial has URL and File

Crawlers• Modules to implement

ModuleSearchBase• New Crawler implements

BaseResultController

Recap

@DNNCon @ashishprasad

Don’t forget to include #DNNCon in your tweets!

THANKS TO ALL OF OUR GENEROUS SPONSORS!