System for collecting and monitoring Arabic weblogs

Background information

• System for collecting and monitoring Arabic weblogs is responsible for maintaining Arabic weblogs collection. In this collection, we are monitoring more than 12.000 Arabic weblogs from different weblog service providers. The system works on daily basis. It collects new Arabic weblogs and adds them to the collection; Besides, it ensures daily based update of All weblogs content.

System architecture

• This part explains the system architecture with respect to its functional modules. That is why we introduce the following modules:

• Collecting module: This module manages first steps of the system. It starts by looking up for blog URLs by connecting remotely to Arabic blog counts. In the same time, it accesses the local index that is storing Arabic blog URLs locally and retrieves information about all blogs. Then, It updates the blogs local index if there are new blog URLs retrieved remotely from the blog counts and not yet present in the local index.

• Monitoring module: This module is the backbone of the whole system. First, Monitoring module reads all entries of the local index. Entries contain URLs of all blogs besides other information related to the content. Second, It constructs for each blog URL a query string representing the blog feed. Third, it fires the query to the blog service provider in order to get the blog feed content. The blog feed content is an XML-like file that contains the last publications of each blog.

• Updating module: Updating comes as the final task of the system. This module obtains from the previous module (monitoring) the blog feed content and checks up its reference at the blogs local index. It compares the last publication date of the local content with the last publication date of the content provided earlier by the monitoring module. The difference of dates triggers this module to update the local index with the last publication date of the blog feed content and add the feed content to the Arabic weblogs collection.