A NEW FRAMEWORK FOR DOMAIN- SPECIFIC HIDDEN WEB CRAWLING BASED ON DATA EXTRACTION TECHNIQUE

The submission of this form generates a web page containing the results of the query, as shown in figure 1(b). A crawler must perform a similar filling process by selecting suitable values from the domain of each finite form element or by generating queries to infinite form element (e.g. text fields); our goal is to automate this process.
Text form Field Labels to the left Labels to the right Submit button
Figure 1(a): A label Form interface of https://www.wendangku.net/doc/183436328.html,
Result matches
Figure 1(b): Result page of https://www.wendangku.net/doc/183436328.html,
4

2. RELATED WORK
The notion of a Hidden (or Deep or Invisible) Web has been a subject of interest for a number of years. A number of researchers have discussed the problems of crawling the contents of hidden Web databases. The problem of automatically discovering and interacting with hidden Web search forms is a problem examined in [22, 2, 6, 1]. Raghavan and GarciaMolina [22] proposes HiWE, a task-specific hidden-Web crawler, the main focus of this work is to learn Hidden-Web query interfaces. Lin and Chen’s solution [14] aims to build up a catalogue of small search engines located in sites and choose which ones are more likely to answer the query. However, forms with more than a text field is not treated. Wang and Lochovsky [13] describes a system called, DeLa, which reconstructs (part of) a “hidden” back-end web database, and it uses the HiWE crawler. There are other approaches that focus on the data extraction. Lage et al. [12] claims to automatically generate agents to collect hidden web pages by filling HTML forms. Liddle et al. [19] performs a study on how Valuable information can be obtained behind web forms, but do not include a crawler to fetch them. Barbosa and Freire [15] experimentally evaluate methods for building multi-keyword queries that can return a large fraction of a document collection. Ntoulas et al. [3] differs from the previous studies, that, it provides a theoretical framework for analyzing the process of generating queries for a database problem of Hidden Web crawling. There exists a large body of work studying how to identify the most relevant database given a user query [16, catg1, catg2]. This body of work is often referred to as meta-searching or database selection problem over the Hidden Web. Based on the adaptation of the recent hidden web crawlers, we propose a framework which combines some of ideas presented in these recent hidden web crawlers in order to enhance the crawling performance. Based on the adaptation of the recent hidden web crawlers, we propose a framework which combines some of ideas presented in these recent hidden web crawlers in order to enhance the crawling performance
5

3. CRAWLING HIDDEN WEB PROBLEM
The problem is that the Current-day crawlers retrieve only Publicly Indexable Web (PIW) ignoring large amounts of high quality information ‘hidden’ behind search forms, This hidden Web is 500 times as large as PIW. The solution is to build a hidden Web crawler can crawl and extract content from hidden databases. Enable indexing, analysis, and mining of hidden Web content, and the content extracted by such crawlers can be used to categorize and classify the hidden databases. Figure 2 shows the idea of the hidden web crawler. Figure 2 (a) illustrates the sequence of steps that take place, when a user uses a search form to submit queries on a hidden database. Figure 2 (b) illustrates the same interaction, with the crawler now playing the role of the human browser combination. Our hidden web crawler goal is to automate the process of viewing, filling in and submitting the HW forms and analyzing the response pages.
Figure 2 (a): User form interaction
Figure 2 (b): Crawler form interaction
6

4. PROPOSED FRAMEWORK
As shown in figure 3, the proposed framework for hidden web crawling is divided into eight phases. Each phase has its function and its own algorithms, which helps our framework to provide a wide Varity of features. Two unique features that unify our crawler are 1) the classification phase for grouping HW and PIW pages into distinct classes, so that making our crawler performs well in both the domain-specific and random mode of crawling, and 2) the capability of dealing with both the single-attribute and multi-attribute databases, unlike most hidden web crawlers that ignore either of them. To support the entire scope of a hidden Web crawling, our framework is designed as a suite of algorithms and heuristics distributed into the eight proposed phases.
4.1 System Layout
Similar to the most recent hidden web crawlers, our framework consists of eight phases; each phase has its functions and algorithms. There are many modules in this architecture, and they are reflected in these eight phases, the function of each phase and its corresponding algorithm will be outlined below:
4.1.1 Phase 1: Collecting web pages
This phase is dedicated to control the entire crawling process and it consists of two basic functional modules: ? Crawler: when start up the crawler URL seed is initialized to a seed set of URLs, it decides which link to visit next, check if URL already visited, so that not to crawl same links again, it opens network connection to retrieve URLs from the web , then crawler hands the downloaded pages over to Html parser. ? Html parser: it takes the HTML pages that have been crawled and indexed, then it parses them looking for two main tags: first, Any HREF tags that refers to links pointed to another sites, they are extracted and then added to be visited by the crawler later.
7

Figure 3: The proposed Hidden web crawler Framework
8

Second, Any FORM tags that refer to presence of HTML forms in the page, then the pages containing these forms are stores in distinct dataset "FormUrl" for subsequent indexing. Thus, for each HTML page retrieved from the Web a logical tree representation of the structure of the page based on the document object model (DOM) [7] is constructed. Then, the tree is analyzed to find the nodes corresponding to the FORM elements. If one or more elements exist, then the pruned tree is constructed for each FORM element. For such tree construction we use only the subtree below the FORM element and the nodes on the path from the FORM to the root. The proposed algorithm in this phase is shown in figure 4, showing the main tasks of the two modules.
Crawler module
Get Next URL to be crawled from URL Seed URL Seed
"No URL"
Wait 10Sec
Phase1
"URL"
URL visited
Already visited
Not visited
URL validation
Dump URL
Valid URL
Open connection to URL
HTML parser module
HTML parser Check 2 main tags
Yes
HREF tag
No
FormURL Yes
FORM tag
No
URL visited
Phase 2
Figure 4: The proposed algorithm for phase1 (collecting web pages)
9

4.1.2 Phase 2: Finding relevant forms
The URLs with forms that already indexed in phase one, few of them are actually relevant forms. The main aim in this phase is to locate any form capable of returning data rich pages. The basic module in this phase is: Form Analyzer that responsible for identifying pages that contain any forms capable of returning data-rich pages (hidden web forms) and to eliminate pages containing register, purchase, or login forms. In practice, different types of pages are found; we distinguish five classes of the pages containing forms. As depicted in figure 5, type A represent all crawled pages (with and without forms). The largest subclass is type B contains pages (with all kind of forms) and we distinguish these pages from all crawled pages by using the algorithm described in phase 1. Type C is the proper subclass of type B and it contains the pages with forms that have at least one input control with text type (text form) and we will recognize this type by using the algorithm described below in figure 6. The registration and login forms are in this class, and these forms are irrelevant to the hidden web forms. Type D is the proper subclass of type C and it contains the pages with (Queryable form), after we apply our algorithm to discard all login or registration forms now we have this class that contains the forms that are textual AND Queryable. Finally, type E which is our ultimate target of the discovery process; it's a subclass of type D and presents pages that contain (Hidden web forms). The important fact here is that any Queryable form is textual form and not vice versa, and any HW form is Queryable AND textual form and not vice versa. Unlike text and Queryable forms, which can be recognized in this phase as shown below in the algorithm, the recognition of HW forms requires further forms analysis and processing. Thus, hidden web forms are recognized in phase 7 which will be discussed later.
10

Phase 1
Type A Pages
Forms detection
Type B Pages
Text forms detection
Type C Pages
Hidden Web Forms
Type E Pages
Phase 7
Type D Pages
Queryable forms detection
Figure 5: Stages of detecting hidden web forms
4.1.3 Phase 3: Classification phase
This main goal of this phase is to group the Hidden Web (HW) and publicable Indexable web (PIW) pages into distinct classes, so that making our crawler more specific. In order to increase the specificity of the crawler, the crawler classifies these pages then stores them in specialized databases according to their category. So that when a user applies a query to our crawler, it must effectively determine which searchable databases are most likely to contain the relevant information for which a user is looking. Thus, this phase give the crawler the capability to perform well in both the domain-specific and random mode of crawling. The basic module in this phase, which performs the phase task, is: The Classifier engine: its main function is to categorize the web pages collected in both of phase 1 (URL visited by the crawler) and phase 2 (URL with relevant Forms) to a set of pre-defined categories.
11

Phase 2
Get next URL with a form to be checked
GetNext (URLi)
Phase 1
FormUR
For every form tag in the URL
For each Fj in URLi FormSubDB= Get (Name, U, method)
FormSubDB
If Type ≠ text
(Label, checkbox, radio, select...etc)
1. ElmNo+=1 2. If Fj in TxtFormDB Fj → considered Else Fj → Rejected 3. Parse (Fj) 4. WordsDB= Get(W) 5. ExtrctLabel(Fj) Phase 4 6. WordsDB= Get (label)
Check every element in every form
For each Ck in Fj
If Type = text OR textarea
1. ElmNo+=1 2. Fj → TxtFormDB 3. TxtFormDB= Get (default, Maxlength) 4. TxtForm+=1 5. Parse(Fj) 6. ExtrctLabel(Fj) Phase 4 7. TxtFormDB= Get (label)
Detect text forms
TxtFormDB
WordsDB
No form Z:
Fj → QFormDB QForm+=1
X:
Check every Fj in TxtFormDB
Detect Queryable forms
A form exist
1. For each Ck in Fj 2. ChckType (type) 3. If type= password { Fj → Rejected Else Fj → QFormDB QForm+=1 } 4. No(password) Goto Y
QFormDB
1. For each Ck in Fj 2. Check (Maxlength) 3. If Maxlength < = 6 { Fj → Rejected Else Fj → QFormDB QForm+=1 } 4. No(Maxlength) Goto Z
Y:
1. For each Ck in Fj 2. ChckLabel (label) 3. If label= password { Fj → Rejected Else Fj → QFormDB QForm+=1 } 4. No(label) Goto X
Phase 7
To recognize HW forms
Figure 6: The proposed algorithm for phase2 (relevant form detection)
The technique for classifying pages over a set of categories starts by training a rule-based document classifier over those categories. Given a set of training, pre-classified documents [16], this tool returns a classifier that might consist of rules for example: Computers IF operating system Computers IF graphics windows Hobbies IF baseball
12

The first rule indicates that if a document contains the term operating system it should be classified in the “Computers” category. A document should also be classified into that category if it has the words graphics and windows. Similarly, if a document has the word baseball, it is a “Hobbies” document. The outcome of categorizing the PIW pages will help in the form filling and label matching operations, i.e. when the documents is classified and indexed in separate databases according to their classes, this will make form analyzing and processing more easily.
4.1.4 Phase 4: Form Analyzing
There are two main functions for this phase parsing hidden web forms and extracting the labels of the html forms. It consists of two functional modules: 1. Form parser: It is Responsible for Parsing hidden web forms to check whether they are single-attribute forms or multi-attribute forms, in order to make our crawler able to deal with all kind of forms and not to limit the crawler functions to any of the single-attribute or the multi-attribute forms. A textual database is a site that mainly contains plain-text documents, such as PubMed database, since these documents do not usually have welldefined structure, most textual databases provide a simple search interface where users type a list of keywords in a single search box, as shown in Figure 7(a).
Figure 7(a): A single-attribute search interface
In contrast, a structured database often contains multi-attribute relational data (e.g., a book on the Amazon Web site may have the fields Author=‘J.K. Rowling’, Title=‘Harry Potter' and ISBN=‘0590353403’) and supports multi-attribute search interfaces, as shown in Figure 7(b).
13

Author: Title: ISBN:
Search now
Figure 7(b): A multi-attribute search interface
After the parser identify the type of search interface, it indexes the forms in two distinct data structures, one for the Single-attribute forms and the other for the multi-attribute forms. The form parser hands the indexed forms over to the second module. 2. Label extractor module: It is responsible for extracting labels from the forms that have been indexed, In spite of having a tag in the HTML specification called label for the declaration of a label, it is almost not used and, therefore, there isn't any explicit mechanism identifying which label is related to each field. Thus, we propose an algorithm to acquire field labels by determining which labels are visually adjacent to a form field. There are two types of label extractor are used, one for the single-attribute forms and the other for the multi-attribute forms, each one has its own algorithm to extract the labels. In both cases, we do not use expensive complex parsing techniques, as in [22]. Rather a simple event driven Parser is used in both cases, and three buffers, one for the left hand side text, one for the right hand side text and one to the above text are used in the multi-attribute case, as it shown in figure 8. The extracted labels are indexed in two separate data structure; one contains the labels of the single-attribute forms and the other for the multiattribute forms. Extracting the labels of the html forms (single and multiple attribute forms) is a very important task, because these labels are the only clues we have to “learn” how to automatic fill in the forms.
14

(1) (2)
Phase 4
FORM PARSER
Classified HW Pages
S-A
S-A forms
Check form attributes
M-A
M-A forms
Phase 3
Y:
LABEL EXTRACTOR
M-A Label extraction S-A Label extraction
Only one label with single textbox Bl→ Ba Go to x
Keyword
To Detect label position
For each PWrd in Fj PWrdj → BL Select OR textarea ValueIn (Bl, Br, Ba) → LBf Clear (Bl, Br, Ba) Go to x Else ValueIn (Bl, Br, Ba) → LBf Clear (Bl, Br, Ba) Go to x
BR or TD
Check Tag Input
Checkbox or radio
Check Type
Z:
For each PWrd in Fj PWrdj → Buff
Else Check Tag Select OR textarea
ValueIn (Bl, Br, Ba) → LBf Br=GetInptValue () ValueIn (Br) →CRLB Clear (CRLB) Go to x
X:
Yes
Check