Data Mining Software - Discover Software Modernization

Data mining software is usually an application that one uses and covers mostly with one's knowledge in the discovery of software modernization. Mining data software involves the understanding of the software artifacts that exist and the mining data tools. This process has very close relations with reverse engineering. The knowledge that one gains from studying data software that exists is usually presented in forms of models and by doing these queries one can be in a position to make his personal data mining software. With the knowledge that someone gains it must be applicable and one must also know the mining data tools that are suppose to be used apart from the soft wares. One can be able to know very widely about the mining data tools that are there in mining data software by doing computer science as a course. Computer science covers widely on what are the procedures, steps of mining data software and how can use the mining data tools.

This software is mostly used in making of databases schemes. Making of databases is not as easy as many would think it requires one to have some knowledge about computer engineering and the basic concepts of computers.;This software is mostly used in data crawling because it can be in a position to store data and one can be able to retrieve the data when needed.

The softwares are not that cheap they come in different varieties and it will depend on which information or the database on which one is coming up with.

Data mining software are usually in different levels there is the data level, design level, application level, architectural level, call graph level and program level it will depend on which level one is covering and this come together with mining data tools.

Data software's have increased rapidly through the introduction of computers and ERP definition. Computers hackers have been able to get the softwares at a very low price and this has made data mining to become very easy and quick to use in the shops and supermarkets and also government institutions. One cannot do data crawling without having the basic knowledge about data mining soft wares because soft wares are the programmes that are usually installed into the computer and without the programmes then no data can be processed.

There are a lot of challenges that come with the use of the mining soft ware. One can easily crush the software he is using or the softwares can easily break they are normally sold on CDS one can easily break it or loose it.

High chances of losing the data that someone is coming up with is very high because computers easily crash due to some difficulties that they experience or a virus can easily crush the computer.

Mining software take a very large space and in most of the computers. The reason behind this is because, data crawling use graphics. Graphics usually occupy a lot of space in terms of the size of the local disk. One is suppose to look for a computer that has very good memory. Data crawling is something that needs to be updated each and every time something appears along the way.

Victor C. has many hobbies and interests. As well being a keen blogger and article writer for many sites, he has also recently created a site focusing on data mining tools. The site is constantly being updated.



Source: http://ezinearticles.com/?Data-Mining-Software---Discover-Software-Modernization&id=5054991

Data Entry Outsourcing - 6 Key Benefits of Outsourced Data Entry

The effective data typing services are must and have to outsource because of globalization. Without information, no company can go ahead and become successful. At every point of making decisions, proper information is essential. So data is one of the most important parts in any organization. There must be proper management to keep the business running smoothly and effectively.

If you want reliable source for data handling, hire typing service company to outsource data entry task. Currently, solutions for every type of business needs are available at reasonable rate. As business grow, it is very hard to manage huge information. So, companies are turning to data entry outsourcing.

Here are the key benefits of data entry outsourcing:

1. All-in-One: data entry firms are offering numbers of services like, data processing, scanning, information formatting, document conversion, indexing and others. They also understand your requirement and deliver the output required format such as Word, Excel, JPG, HTML, XML and Other.

2. Resolve the Issues: As company grows, there are many issues arise like information about employees, benefits, healthcare for them, tuning with rapidly changing technologies, latest business information and others. If organization outsources some of their responsibilities, various issues get resolved quickly and automatically.

3. Better Services: You can expect superior data management and high quality services from outsourcing companies. They have experienced and skilled professionals with latest technologies to deliver unexpected result and stay ahead of other.

4. Least Cost: You can lower down your capital cost of infrastructure and other cost of salary, stationery and other, if you outsource data typing task. Through offshore companies, you can easily save up to 60% on data typing services.

5. Higher Efficiency: If your employees are free from routine and uninteresting process of entering information, they can deliver better result. Ultimately, this can increase the job satisfaction level and efficiency. You can expect high output at lower costs.

6. Place of Outsourcing: You must think about the outsourcing country. India is chosen by various companies for data typing outsourcing. At India, you can get benefits of better quality, enough infrastructure, quick delivery, skilled experts at very low rates.

You can easily reduce tons of time-consuming and boring responsibilities by outsourcing.

Bea Arthur is a quality controller at Data Entry India that provides Data Entry Services, Data Conversion Services and Data Processing Services. They are having more than 17 years of experience in data entry outsourcing.




Source: http://ezinearticles.com/?Data-Entry-Outsourcing---6-Key-Benefits-of-Outsourced-Data-Entry&id=4253927

Backtesting & Data Mining

Introduction

In this article we'll take a look at two related practices that are widely used by traders called Backtesting and Data Mining. These are techniques that are powerful and valuable if we use them correctly, however traders often misuse them. Therefore, we'll also explore two common pitfalls of these techniques, known as the multiple hypothesis problem and overfitting and how to overcome these pitfalls.

Backtesting

Backtesting is just the process of using historical data to test the performance of some trading strategy. Backtesting generally starts with a strategy that we would like to test, for instance buying GBP/USD when it crosses above the 20-day moving average and selling when it crosses below that average. Now we could test that strategy by watching what the market does going forward, but that would take a long time. This is why we use historical data that is already available.

"But wait, wait!" I hear you say. "Couldn't you cheat or at least be biased because you already know what happened in the past?" That's definitely a concern, so a valid backtest will be one in which we aren't familiar with the historical data. We can accomplish this by choosing random time periods or by choosing many different time periods in which to conduct the test.

Now I can hear another group of you saying, "But all that historical data just sitting there waiting to be analyzed is tempting isn't it? Maybe there are profound secrets in that data just waiting for geeks like us to discover it. Would it be so wrong for us to examine that historical data first, to analyze it and see if we can find patterns hidden within it?" This argument is also valid, but it leads us into an area fraught with danger...the world of Data Mining

Data Mining

Data Mining involves searching through data in order to locate patterns and find possible correlations between variables. In the example above involving the 20-day moving average strategy, we just came up with that particular indicator out of the blue, but suppose we had no idea what type of strategy we wanted to test? That's when data mining comes in handy. We could search through our historical data on GBP/USD to see how the price behaved after it crossed many different moving averages. We could check price movements against many other types of indicators as well and see which ones correspond to large price movements.

The subject of data mining can be controversial because as I discussed above it seems a bit like cheating or "looking ahead" in the data. Is data mining a valid scientific technique? On the one hand the scientific method says that we're supposed to make a hypothesis first and then test it against our data, but on the other hand it seems appropriate to do some "exploration" of the data first in order to suggest a hypothesis. So which is right? We can look at the steps in the Scientific Method for a clue to the source of the confusion. The process in general looks like this:

Observation (data) >>> Hypothesis >>> Prediction >>> Experiment (data)

Notice that we can deal with data during both the Observation and Experiment stages. So both views are right. We must use data in order to create a sensible hypothesis, but we also test that hypothesis using data. The trick is simply to make sure that the two sets of data are not the same! We must never test our hypothesis using the same set of data that we used to suggest our hypothesis. In other words, if you use data mining in order to come up with strategy ideas, make sure you use a different set of data to backtest those ideas.

Now we'll turn our attention to the main pitfalls of using data mining and backtesting incorrectly. The general problem is known as "over-optimization" and I prefer to break that problem down into two distinct types. These are the multiple hypothesis problem and overfitting. In a sense they are opposite ways of making the same error. The multiple hypothesis problem involves choosing many simple hypotheses while overfitting involves the creation of one very complex hypothesis.

The Multiple Hypothesis Problem

To see how this problem arises, let's go back to our example where we backtested the 20-day moving average strategy. Let's suppose that we backtest the strategy against ten years of historical market data and lo and behold guess what? The results are not very encouraging. However, being rough and tumble traders as we are, we decide not to give up so easily. What about a ten day moving average? That might work out a little better, so let's backtest it! We run another backtest and we find that the results still aren't stellar, but they're a bit better than the 20-day results. We decide to explore a little and run similar tests with 5-day and 30-day moving averages. Finally it occurs to us that we could actually just test every single moving average up to some point and see how they all perform. So we test the 2-day, 3-day, 4-day, and so on, all the way up to the 50-day moving average.

Now certainly some of these averages will perform poorly and others will perform fairly well, but there will have to be one of them which is the absolute best. For instance we may find that the 32-day moving average turned out to be the best performer during this particular ten year period. Does this mean that there is something special about the 32-day average and that we should be confident that it will perform well in the future? Unfortunately many traders assume this to be the case, and they just stop their analysis at this point, thinking that they've discovered something profound. They have fallen into the "Multiple Hypothesis Problem" pitfall.

The problem is that there is nothing at all unusual or significant about the fact that some average turned out to be the best. After all, we tested almost fifty of them against the same data, so we'd expect to find a few good performers, just by chance. It doesn't mean there's anything special about the particular moving average that "won" in this case. The problem arises because we tested multiple hypotheses until we found one that worked, instead of choosing a single hypothesis and testing it.

Here's a good classic analogy. We could come up with a single hypothesis such as "Scott is great at flipping heads on a coin." From that, we could create a prediction that says, "If the hypothesis is true, Scott will be able to flip 10 heads in a row." Then we can perform a simple experiment to test that hypothesis. If I can flip 10 heads in a row it actually doesn't prove the hypothesis. However if I can't accomplish this feat it definitely disproves the hypothesis. As we do repeated experiments which fail to disprove the hypothesis, then our confidence in its truth grows.

That's the right way to do it. However, what if we had come up with 1,000 hypotheses instead of just the one about me being a good coin flipper? We could make the same hypothesis about 1,000 different people...me, Ed, Cindy, Bill, Sam, etc. Ok, now let's test our multiple hypotheses. We ask all 1000 people to flip a coin. There will probably be about 500 who flip heads. Everyone else can go home. Now we ask those 500 people to flip again, and this time about 250 will flip heads. On the third flip about 125 people flip heads, on the fourth about 63 people are left, and on the fifth flip there are about 32. These 32 people are all pretty amazing aren't they? They've all flipped five heads in a row! If we flip five more times and eliminate half the people each time on average, we will end up with 16, then 8, then 4, then 2 and finally one person left who has flipped ten heads in a row. It's Bill! Bill is a "fantabulous" flipper of coins! Or is he?

Well we really don't know, and that's the point. Bill may have won our contest out of pure chance, or he may very well be the best flipper of heads this side of the Andromeda galaxy. By the same token, we don't know if the 32-day moving average from our example above just performed well in our test by pure chance, or if there is really something special about it. But all we've done so far is to find a hypothesis, namely that the 32-day moving average strategy is profitable (or that Bill is a great coin flipper). We haven't actually tested that hypothesis yet.

So now that we understand that we haven't really discovered anything significant yet about the 32-day moving average or about Bill's ability to flip coins, the natural question to ask is what should we do next? As I mentioned above, many traders never realize that there is a next step required at all. Well, in the case of Bill you'd probably ask, "Aha, but can he flip ten heads in a row again?" In the case of the 32-day moving average, we'd want to test it again, but certainly not against the same data sample that we used to choose that hypothesis. We would choose another ten-year period and see if the strategy worked just as well. We could continue to do this experiment as many times as we wanted until our supply of new ten-year periods ran out. We refer to this as "out of sample testing", and it's the way to avoid this pitfall. There are various methods of such testing, one of which is "cross validation", but we won't get into that much detail here.

Overfitting

Overfitting is really a kind of reversal of the above problem. In the multiple hypothesis example above, we looked at many simple hypotheses and picked the one that performed best in the past. In overfitting we first look at the past and then construct a single complex hypothesis that fits well with what happened. For example if I look at the USD/JPY rate over the past 10 days, I might see that the daily closes did this:

up, up, down, up, up, up, down, down, down, up.

Got it? See the pattern? Yeah, neither do I actually. But if I wanted to use this data to suggest a hypothesis, I might come up with...

My amazing hypothesis:

If the closing price goes up twice in a row then down for one day, or if it goes down for three days in a row we should buy,

but if the closing price goes up three days in a row we should sell,

but if it goes up three days in a row and then down three days in a row we should buy.

Huh? Sounds like a whacky hypothesis right? But if we had used this strategy over the past 10 days, we would have been right on every single trade we made! The "overfitter" uses backtesting and data mining differently than the "multiple hypothesis makers" do. The "overfitter" doesn't come up with 400 different strategies to backtest. No way! The "overfitter" uses data mining tools to figure out just one strategy, no matter how complex, that would have had the best performance over the backtesting period. Will it work in the future?

Not likely, but we could always keep tweaking the model and testing the strategy in different samples (out of sample testing again) to see if our performance improves. When we stop getting performance improvements and the only thing that's rising is the complexity of our model, then we know we've crossed the line into overfitting.

Conclusion

So in summary, we've seen that data mining is a way to use our historical price data to suggest a workable trading strategy, but that we have to be aware of the pitfalls of the multiple hypothesis problem and overfitting. The way to make sure that we don't fall prey to these pitfalls is to backtest our strategy using a different dataset than the one we used during our data mining exploration. We commonly refer to this as "out of sample testing".

Scott Percival

October 2006

Scott Percival is the Director of Research for the FOREX Statistical Research Center at Market-geeks.com, a site which is devoted to using mathematics and the scientific method to study the behavior of prices in the FOREX market. Mr. Percival has a degree in Civil Engineering from Northeastern University, and has worked as a Registered Representative and trading instructor at Fidelity Investments. He is currently working toward the goal of becoming a full time FOREX trader.




Source: http://ezinearticles.com/?Backtesting-and-Data-Mining&id=341468

The Need for Specialised Data Mining Techniques for Web 2.0

Web 2.0 is not exactly a new version of the Web, but rather a way to describe a new generation of interactive websites centred on the user. These are websites that offer

interactive information sharing, as well as collaboration - a case in point being wikis and blogs - and is now expanding to other areas as well. These new sites are the result of new technologies and new ideas and are on the cutting edge of Web development. Due to their novelty, they create a rather interesting challenge for data mining.

Data mining is simply a process of finding patterns in masses of data. There is such a vast plethora of information out there on the Web that it is necessary to use data mining tools to make sense of it. Traditional data mining techniques are not very effective when used on these new Web 2.0 sites because the user interface is so varied. Since Web 2.0 sites are created largely by user-supplied content, there is even more data to mine for valuable information. Having said that, the additional freedom in the format ensures that it is much more difficult to sift through the content to find what is usable.The data available is very valuable, so where there is a new platform, there must be new techniques developed for mining the data. The trick is that the data mining methods must themselves be flexible as the sites they are targeting are flexible. In the initial days of the World Wide Web, which was referred to as Web 1.0, data mining programs knew where to look for the desired information. Web 2.0 sites lack structure, meaning there is no single spot for the mining program to target. It must be able to scan and sift through all of the user-generated content to find what is needed. The upside is that there is a lot more data out there, which means more and more accurate results if the data can be properly utilized. The downside is that with all that data, if the selection criteria are not specific enough, the results will be meaningless. Too much of a good thing is definitely a bad thing. Wikis and blogs have been around long enough now that enough research has been carried out to understand them better. This research can now be used, in turn, to devise the best possible data mining methods. New algorithms are being developed that will allow data mining applications to analyse this data and return useful. Another problem is that there are many cul-de-sacs on the internet now, where groups of people share information freely, but only behind walls/barriers that keep it away from the genera results.

The main challenge in developing these algorithms does not lie with finding the data, because there is too much of it. The challenge is filtering out irrelevant data to get to the meaningful one. At this point none of the techniques are perfected. This makes Web 2.0 data mining an exciting and frustrating field, and yet another challenge in the never ending series of technological hurdles that have stemmed from the internet. There are numerous problems to overcome. One is the inability to rely on keywords, which used to be the best method to search. This does not allow for an understanding of context or sentiment associated with the keywords which can drastically vary the meaning of the keyword population. Social networking sites are a good example of this, where you can share information with everyone you know, but it is more difficult for that information to proliferate outside of those circles. This is good in terms of protecting privacy, but it does not add to the collective knowledge base and it can lead to a skewed understanding of public sentiment based on what social structures you have entry into. Attempts to use artificial intelligence have been less than successful because it is not adequately focused in its methodology. Data mining depends on the collection of data and sorting the results to create reports on the individual metrics that are the focus of interest. The size of the data sets are simply too large for traditional computational techniques to be able to tackle them. That is why a new answer needs to be found. Data mining is an important necessity for managing the backhaul of the internet. As Web 2.0 grows exponentially, it is increasingly hard to keep track of everything that is out there and summarize and synthesize it in a useful way. Data mining is necessary for companies to be able to really understand what customers like and want so that they can create products to meet these needs. In the increasingly aggressive global market, companies also need the reports resulting from data mining to remain competitive. If they are unable to keep track of the market and stay abreast of popular trends, they will not survive. The solution has to come from open source with options to scale databases depending on needs. There are companies that are now working on these ideas and are sharing the results with others to further improve them. So, just as open source and collective information sharing of Web 2.0 created these new data mining challenges, it will be the collective effort that solves the problems as well.

It is important to view this as a process of constant improvement, not one where an answer will be absolute for all time. Since its advent, the internet has changed quite significantly as well as the way users interact with it. Data mining will always be a critical part of corporate internet usage and its methods will continue to evolve just as the Web and its content does.

There is a huge incentive for creating better data mining solutions to tackle the complexities of Web 2.0. For this reason, several companies exist just for the purpose of analysing and creating solutions to the data mining problem. They find eager buyers for their applications in companies which are desperate for information on markets and potential customers. The companies in question do not simply want more data, they want better data. This requires a system that can classify and group data, and then make sense of the results.While the data mining process is expensive to start with, it is well worth for a retail company because it provides insight into the market and thus enables quick decisions.The speed at which a company which has insightful information on the marketplace can react to changes, gives it a huge advantage over the competition. Not only can the company react quickly, it is likely to steer itself in the right direction if its information is based on updated data.Advanced data mining will allow companies not only to make snap decisions, but also to plan long range strategies, based on the direction the marketplace is heading. Data mining brings the company closer to its customers. The real winners here, are the companies that have now discovered that they can make a living by improving the existing data mining techniques. They have filled a niche that was only created recently, which no one could have foreseen and have done quite a, good job at it.



Source: http://ezinearticles.com/?The-Need-for-Specialised-Data-Mining-Techniques-for-Web-2.0&id=7412130

Data Mining vs Screen-Scraping

Data mining isn't screen-scraping. I know that some people in the room may disagree with that statement, but they're actually two almost completely different concepts.

In a nutshell, you might state it this way: screen-scraping allows you to get information, where data mining allows you to analyze information. That's a pretty big simplification, so I'll elaborate a bit.

The term "screen-scraping" comes from the old mainframe terminal days where people worked on computers with green and black screens containing only text. Screen-scraping was used to extract characters from the screens so that they could be analyzed. Fast-forwarding to the web world of today, screen-scraping now most commonly refers to extracting information from web sites. That is, computer programs can "crawl" or "spider" through web sites, pulling out data. People often do this to build things like comparison shopping engines, archive web pages, or simply download text to a spreadsheet so that it can be filtered and analyzed.

Data mining, on the other hand, is defined by Wikipedia as the "practice of automatically searching large stores of data for patterns." In other words, you already have the data, and you're now analyzing it to learn useful things about it. Data mining often involves lots of complex algorithms based on statistical methods. It has nothing to do with how you got the data in the first place. In data mining you only care about analyzing what's already there.

The difficulty is that people who don't know the term "screen-scraping" will try Googling for anything that resembles it. We include a number of these terms on our web site to help such folks; for example, we created pages entitled Text Data Mining, Automated Data Collection, Web Site Data Extraction, and even Web Site Ripper (I suppose "scraping" is sort of like "ripping"). So it presents a bit of a problem-we don't necessarily want to perpetuate a misconception (i.e., screen-scraping = data mining), but we also have to use terminology that people will actually use.

Todd Wilson is the owner of screen-scraper.com (http://www.screen-scraper.com/), a company which specializes in data extraction from web pages. While not scraping screens Todd is hard at work finishing up a doctoral degree in Instructional Psychology and Technology.




Source: http://ezinearticles.com/?Data-Mining-vs-Screen-Scraping&id=146813

Data Extraction Services For Better Outputs in Your Business

Data Extraction can be defined as the process of retrieving data from an unstructured source in order to process it further or store it. It is very useful for large organizations who deal with large amount of data on a daily basis that need to be processed into meaningful information and stored for later use. The data extraction is a systematic way to extract and structure data from scattered and semi-structured electronic documents, as found on the web and in various data warehouses.

In today's highly competitive business world, vital business information such as customer statistics, competitor's operational figures and inter-company sales figures play an important role in making strategic decisions. By signing on this service provider, you will be get access to critivcal data from various sources like websites, databases, images and documents.

It can help you take strategic business decisions that can shape your business' goals. Whether you need customer information, nuggets into your competitor's operations and figure out your organization's performance, it is highly critical to have data at your fingertips as and when you want it. Your company may be crippled with tons of data and it may prove a headache to control and convert the data into useful information. Data extraction services enable you get data quickly and in the right format.

Few areas where Data Extraction can help you are:

    Capturing financial data
    Generating better sales leads
    Conducting market research, survey and analysis
    Conducting product research and analysis
    Track, extract and harvest product pricing data
    Searching for specific job postings
    Duplicating an online database
    Acquiring real estate data
    Processing auction information
    Searching online newspapers for latest pricing information
    Extracting and summarize news stories from online news sources

Outsourcing companies provide custom made data extraction services to the client's requirements. The different types of data extraction services;

    Web extraction
    Database extraction

Outsourcing is the beneficial option for large organizations seeking to manage large information. Outsourcing this services helps businesses in managing their data effectively, which in turn enables business to experience an increase in profits. By outsourcing, you can certainly increase your competitive edge and save costs too!



Source: http://ezinearticles.com/?Data-Extraction-Services-For-Better-Outputs-in-Your-Business&id=2760257

Three Common Methods For Web Data Extraction

Probably the most common technique used traditionally to extract data from web pages this is to cook up some regular expressions that match the pieces you want (e.g., URL's and link titles). Our screen-scraper software actually started out as an application written in Perl for this very reason. In addition to regular expressions, you might also use some code written in something like Java or Active Server Pages to parse out larger chunks of text. Using raw regular expressions to pull out the data can be a little intimidating to the uninitiated, and can get a bit messy when a script contains a lot of them. At the same time, if you're already familiar with regular expressions, and your scraping project is relatively small, they can be a great solution.

Other techniques for getting the data out can get very sophisticated as algorithms that make use of artificial intelligence and such are applied to the page. Some programs will actually analyze the semantic content of an HTML page, then intelligently pull out the pieces that are of interest. Still other approaches deal with developing "ontologies", or hierarchical vocabularies intended to represent the content domain.

There are a number of companies (including our own) that offer commercial applications specifically intended to do screen-scraping. The applications vary quite a bit, but for medium to large-sized projects they're often a good solution. Each one will have its own learning curve, so you should plan on taking time to learn the ins and outs of a new application. Especially if you plan on doing a fair amount of screen-scraping it's probably a good idea to at least shop around for a screen-scraping application, as it will likely save you time and money in the long run.

So what's the best approach to data extraction? It really depends on what your needs are, and what resources you have at your disposal. Here are some of the pros and cons of the various approaches, as well as suggestions on when you might use each one:

Raw regular expressions and code

Advantages:

- If you're already familiar with regular expressions and at least one programming language, this can be a quick solution.

- Regular expressions allow for a fair amount of "fuzziness" in the matching such that minor changes to the content won't break them.

- You likely don't need to learn any new languages or tools (again, assuming you're already familiar with regular expressions and a programming language).

- Regular expressions are supported in almost all modern programming languages. Heck, even VBScript has a regular expression engine. It's also nice because the various regular expression implementations don't vary too significantly in their syntax.

Disadvantages:

- They can be complex for those that don't have a lot of experience with them. Learning regular expressions isn't like going from Perl to Java. It's more like going from Perl to XSLT, where you have to wrap your mind around a completely different way of viewing the problem.

- They're often confusing to analyze. Take a look through some of the regular expressions people have created to match something as simple as an email address and you'll see what I mean.

- If the content you're trying to match changes (e.g., they change the web page by adding a new "font" tag) you'll likely need to update your regular expressions to account for the change.

- The data discovery portion of the process (traversing various web pages to get to the page containing the data you want) will still need to be handled, and can get fairly complex if you need to deal with cookies and such.

When to use this approach: You'll most likely use straight regular expressions in screen-scraping when you have a small job you want to get done quickly. Especially if you already know regular expressions, there's no sense in getting into other tools if all you need to do is pull some news headlines off of a site.

Ontologies and artificial intelligence

Advantages:

- You create it once and it can more or less extract the data from any page within the content domain you're targeting.

- The data model is generally built in. For example, if you're extracting data about cars from web sites the extraction engine already knows what the make, model, and price are, so it can easily map them to existing data structures (e.g., insert the data into the correct locations in your database).

- There is relatively little long-term maintenance required. As web sites change you likely will need to do very little to your extraction engine in order to account for the changes.

Disadvantages:

- It's relatively complex to create and work with such an engine. The level of expertise required to even understand an extraction engine that uses artificial intelligence and ontologies is much higher than what is required to deal with regular expressions.

- These types of engines are expensive to build. There are commercial offerings that will give you the basis for doing this type of data extraction, but you still need to configure them to work with the specific content domain you're targeting.

- You still have to deal with the data discovery portion of the process, which may not fit as well with this approach (meaning you may have to create an entirely separate engine to handle data discovery). Data discovery is the process of crawling web sites such that you arrive at the pages where you want to extract data.

When to use this approach: Typically you'll only get into ontologies and artificial intelligence when you're planning on extracting information from a very large number of sources. It also makes sense to do this when the data you're trying to extract is in a very unstructured format (e.g., newspaper classified ads). In cases where the data is very structured (meaning there are clear labels identifying the various data fields), it may make more sense to go with regular expressions or a screen-scraping application.

Screen-scraping software

Advantages:

- Abstracts most of the complicated stuff away. You can do some pretty sophisticated things in most screen-scraping applications without knowing anything about regular expressions, HTTP, or cookies.

- Dramatically reduces the amount of time required to set up a site to be scraped. Once you learn a particular screen-scraping application the amount of time it requires to scrape sites vs. other methods is significantly lowered.

- Support from a commercial company. If you run into trouble while using a commercial screen-scraping application, chances are there are support forums and help lines where you can get assistance.

Disadvantages:

- The learning curve. Each screen-scraping application has its own way of going about things. This may imply learning a new scripting language in addition to familiarizing yourself with how the core application works.

- A potential cost. Most ready-to-go screen-scraping applications are commercial, so you'll likely be paying in dollars as well as time for this solution.

- A proprietary approach. Any time you use a proprietary application to solve a computing problem (and proprietary is obviously a matter of degree) you're locking yourself into using that approach. This may or may not be a big deal, but you should at least consider how well the application you're using will integrate with other software applications you currently have. For example, once the screen-scraping application has extracted the data how easy is it for you to get to that data from your own code?

When to use this approach: Screen-scraping applications vary widely in their ease-of-use, price, and suitability to tackle a broad range of scenarios. Chances are, though, that if you don't mind paying a bit, you can save yourself a significant amount of time by using one. If you're doing a quick scrape of a single page you can use just about any language with regular expressions. If you want to extract data from hundreds of web sites that are all formatted differently you're probably better off investing in a complex system that uses ontologies and/or artificial intelligence. For just about everything else, though, you may want to consider investing in an application specifically designed for screen-scraping.

As an aside, I thought I should also mention a recent project we've been involved with that has actually required a hybrid approach of two of the aforementioned methods. We're currently working on a project that deals with extracting newspaper classified ads. The data in classifieds is about as unstructured as you can get. For example, in a real estate ad the term "number of bedrooms" can be written about 25 different ways. The data extraction portion of the process is one that lends itself well to an ontologies-based approach, which is what we've done. However, we still had to handle the data discovery portion. We decided to use screen-scraper for that, and it's handling it just great. The basic process is that screen-scraper traverses the various pages of the site, pulling out raw chunks of data that constitute the classified ads. These ads then get passed to code we've written that uses ontologies in order to extract out the individual pieces we're after. Once the data has been extracted we then insert it into a database.




Source: http://ezinearticles.com/?Three-Common-Methods-For-Web-Data-Extraction&id=165416