4 changed files with 460 additions and 0 deletions
@ -0,0 +1,358 @@ |
|||||
|
# Java爬虫框架 |
||||
|
|
||||
|
基于MVC架构的Java爬虫框架，支持多态扩展，可轻松添加新的网站爬虫。 |
||||
|
|
||||
|
## 功能特性 |
||||
|
|
||||
|
- **MVC架构**：清晰的分层设计，职责分明 |
||||
|
- **多态扩展**：通过继承BaseCrawler实现新爬虫 |
||||
|
- **命令行界面**：支持交互式命令操作 |
||||
|
- **自动识别**：根据URL自动选择合适的爬虫 |
||||
|
- **日期提取**：支持从URL中提取发布日期 |
||||
|
|
||||
|
## 支持的网站 |
||||
|
|
||||
|
| 网站 | 域名 | 爬虫名称 | |
||||
|
|------|------|----------| |
||||
|
| 湖南大学官网 | `*.hnu.edu.cn` | HunanUniversityCrawler | |
||||
|
| 湖南大学新闻网 | `news.hnu.edu.cn` | HunanUniversityNewsCrawler | |
||||
|
| 中国天气网 | `*.weather.com.cn` | ChinaWeatherCrawler | |
||||
|
| 骑砍中文站 | `*.mountblade.com.cn` | MountBladeCrawler | |
||||
|
|
||||
|
## 快速开始 |
||||
|
|
||||
|
### 编译项目 |
||||
|
|
||||
|
```bash |
||||
|
javac -d target/classes src/main/java/com/crawler/**/*.java |
||||
|
``` |
||||
|
|
||||
|
### 运行程序 |
||||
|
|
||||
|
```bash |
||||
|
java -cp target/classes com.crawler.Main |
||||
|
``` |
||||
|
|
||||
|
### 命令行使用 |
||||
|
|
||||
|
``` |
||||
|
======================================== |
||||
|
Java爬虫框架 |
||||
|
======================================== |
||||
|
|
||||
|
======================================== |
||||
|
Java爬虫框架 - 命令行模式 |
||||
|
======================================== |
||||
|
输入 'help' 查看可用指令 |
||||
|
======================================== |
||||
|
> help |
||||
|
可用指令: |
||||
|
--------- |
||||
|
help : 显示所有可用指令 |
||||
|
list : 查看使用过的指令历史 |
||||
|
crawl : 运行爬虫，输入URL自动选择爬虫 |
||||
|
exit : 退出程序 |
||||
|
|
||||
|
> crawl |
||||
|
请输入要爬取的URL: https://www.mountblade.com.cn |
||||
|
使用爬虫: MountBladeCrawler |
||||
|
... |
||||
|
``` |
||||
|
|
||||
|
## 项目结构 |
||||
|
|
||||
|
``` |
||||
|
src/main/java/com/crawler/ |
||||
|
├── Main.java # 主入口 |
||||
|
├── model/ |
||||
|
│ ├── CrawlerData.java # 爬取数据模型（标题、链接、来源、发布日期） |
||||
|
│ └── CrawlerConfig.java # 爬虫配置（超时时间、User-Agent） |
||||
|
├── view/ |
||||
|
│ └── CrawlerView.java # 视图层（输出结果展示） |
||||
|
├── controller/ |
||||
|
│ └── CrawlerController.java # 爬虫控制器 |
||||
|
├── crawler/ |
||||
|
│ ├── Crawler.java # 爬虫接口 |
||||
|
│ ├── BaseCrawler.java # 爬虫抽象基类 |
||||
|
│ ├── CrawlerFactory.java # 爬虫工厂（自动选择爬虫） |
||||
|
│ └── impl/ |
||||
|
│ ├── ExampleCrawler.java # 通用爬虫 |
||||
|
│ ├── TestCrawler.java # 测试爬虫 |
||||
|
│ ├── HunanUniversityCrawler.java |
||||
|
│ ├── HunanUniversityNewsCrawler.java |
||||
|
│ ├── ChinaWeatherCrawler.java |
||||
|
│ └── MountBladeCrawler.java |
||||
|
└── command/ |
||||
|
├── Command.java # 命令接口 |
||||
|
├── BaseCommand.java # 命令抽象基类 |
||||
|
├── CommandHistory.java # 命令历史记录 |
||||
|
├── HelpCommand.java # 帮助命令 |
||||
|
├── ListCommand.java # 历史记录命令 |
||||
|
├── CrawlCommand.java # 爬虫命令 |
||||
|
├── ExitCommand.java # 退出命令 |
||||
|
└── CommandController.java # 命令控制器 |
||||
|
``` |
||||
|
|
||||
|
## 扩展新爬虫 |
||||
|
|
||||
|
只需继承 `BaseCrawler` 并重写两个方法： |
||||
|
|
||||
|
```java |
||||
|
package com.crawler.crawler.impl; |
||||
|
|
||||
|
import com.crawler.crawler.BaseCrawler; |
||||
|
import com.crawler.model.CrawlerData; |
||||
|
|
||||
|
import java.util.ArrayList; |
||||
|
import java.util.List; |
||||
|
import java.util.regex.Matcher; |
||||
|
import java.util.regex.Pattern; |
||||
|
|
||||
|
public class MyWebsiteCrawler extends BaseCrawler { |
||||
|
private static final String BASE_URL = "https://www.mywebsite.com"; |
||||
|
|
||||
|
@Override |
||||
|
public String getCrawlerName() { |
||||
|
return "MyWebsiteCrawler"; |
||||
|
} |
||||
|
|
||||
|
@Override |
||||
|
protected List<CrawlerData> parseHtml(String html) { |
||||
|
List<CrawlerData> results = new ArrayList<>(); |
||||
|
|
||||
|
// 使用正则表达式解析HTML |
||||
|
Pattern pattern = Pattern.compile("<a href=\"([^\"]+)\">([^<]+)</a>"); |
||||
|
Matcher matcher = pattern.matcher(html); |
||||
|
|
||||
|
while (matcher.find()) { |
||||
|
CrawlerData data = new CrawlerData(); |
||||
|
data.setTitle(matcher.group(2)); |
||||
|
data.setUrl(normalizeUrl(matcher.group(1))); |
||||
|
data.setSource(getCrawlerName()); |
||||
|
data.setPublishDate(extractDateFromUrl(matcher.group(1))); |
||||
|
results.add(data); |
||||
|
} |
||||
|
|
||||
|
return results; |
||||
|
} |
||||
|
|
||||
|
private String normalizeUrl(String url) { |
||||
|
if (url.startsWith("/")) { |
||||
|
return BASE_URL + url; |
||||
|
} |
||||
|
return url; |
||||
|
} |
||||
|
|
||||
|
private String extractDateFromUrl(String url) { |
||||
|
Pattern datePattern = Pattern.compile("/(\\d{4}-\\d{2}-\\d{2})/"); |
||||
|
Matcher matcher = datePattern.matcher(url); |
||||
|
return matcher.find() ? matcher.group(1) : null; |
||||
|
} |
||||
|
} |
||||
|
``` |
||||
|
|
||||
|
然后在 `CrawlerFactory.java` 中添加识别规则： |
||||
|
|
||||
|
```java |
||||
|
crawlerPatterns.put("MyWebsiteCrawler", |
||||
|
Pattern.compile(".*mywebsite\\.com.*", Pattern.CASE_INSENSITIVE)); |
||||
|
``` |
||||
|
|
||||
|
并在 `createCrawlerByName` 方法中添加： |
||||
|
|
||||
|
```java |
||||
|
case "MyWebsiteCrawler": |
||||
|
return new MyWebsiteCrawler(); |
||||
|
``` |
||||
|
|
||||
|
## 架构设计 |
||||
|
|
||||
|
### MVC模式 |
||||
|
|
||||
|
- **Model**：`CrawlerData`（数据模型）、`CrawlerConfig`（配置） |
||||
|
- **View**：`CrawlerView`（结果展示） |
||||
|
- **Controller**：`CrawlerController`（爬虫控制）、`CommandController`（命令控制） |
||||
|
|
||||
|
### 多态设计 |
||||
|
|
||||
|
- `Crawler` 接口定义标准方法 |
||||
|
- `BaseCrawler` 提供通用HTTP请求能力 |
||||
|
- 各爬虫实现类继承 `BaseCrawler` 并重写 `parseHtml` 方法 |
||||
|
|
||||
|
### 工厂模式 |
||||
|
|
||||
|
`CrawlerFactory` 根据URL模式自动选择合适的爬虫实现。 |
||||
|
|
||||
|
## 配置说明 |
||||
|
|
||||
|
`CrawlerConfig` 支持以下配置： |
||||
|
|
||||
|
- `timeout`：HTTP请求超时时间（默认30000毫秒） |
||||
|
- `userAgent`：User-Agent（默认模拟Chrome浏览器） |
||||
|
|
||||
|
## 命令列表 |
||||
|
|
||||
|
| 命令 | 功能 | |
||||
|
|------|------| |
||||
|
| `help` | 显示所有可用指令 | |
||||
|
| `list` | 查看使用过的指令历史 | |
||||
|
| `crawl` | 运行爬虫，输入目标URL，爬取后可保存结果 | |
||||
|
| `cache` | 缓存操作：save/load/list/delete | |
||||
|
| `exit` | 退出程序 | |
||||
|
|
||||
|
### cache 命令子操作 |
||||
|
|
||||
|
| 子操作 | 功能 | |
||||
|
|--------|------| |
||||
|
| `save` | 保存当前爬取数据到数据文件 | |
||||
|
| `load` | 从数据文件读取数据 | |
||||
|
| `list` | 列出 `data/` 目录中的所有文件 | |
||||
|
| `delete` | 删除指定的数据文件或所有文件 | |
||||
|
|
||||
|
### 数据目录 |
||||
|
|
||||
|
程序会自动创建 `data/` 目录用于保存爬取的数据文件。 |
||||
|
|
||||
|
### 爬取后自动保存 |
||||
|
|
||||
|
使用 `crawl` 命令爬取完成后，系统会自动询问是否保存结果： |
||||
|
|
||||
|
``` |
||||
|
爬虫运行完成，共获取 10 条数据 |
||||
|
======================================== |
||||
|
|
||||
|
是否保存爬取结果? (y/n): y |
||||
|
请输入保存路径 (默认: data/crawler_data.json): |
||||
|
数据已保存到: data/crawler_data.json |
||||
|
``` |
||||
|
|
||||
|
### 删除缓存文件示例 |
||||
|
|
||||
|
``` |
||||
|
> cache |
||||
|
请输入缓存操作 (save/load/list/delete): delete |
||||
|
======================================== |
||||
|
可选删除的文件: |
||||
|
======================================== |
||||
|
[1] crawler_data.json (1024 bytes) |
||||
|
[2] mountblade_data.json (2048 bytes) |
||||
|
[all] 删除所有文件 |
||||
|
======================================== |
||||
|
请输入要删除的文件序号或 'all': 1 |
||||
|
确定要删除 'crawler_data.json' 吗? (y/n): y |
||||
|
已删除: crawler_data.json |
||||
|
``` |
||||
|
|
||||
|
## 输出示例 |
||||
|
|
||||
|
``` |
||||
|
[12] |
||||
|
标题: 骑砍2《战帆》v1.2.4与本体v1.4.4测试版更新日志 |
||||
|
链接: https://www.mountblade.com.cn/news/Bannerlord/2026-05-13/3175.html |
||||
|
来源: MountBladeCrawler |
||||
|
发布日期: 2026-05-13 |
||||
|
---------------------------------------- |
||||
|
``` |
||||
|
|
||||
|
## 异常处理 |
||||
|
|
||||
|
项目采用分层异常体系设计，区分受检异常和非受检异常： |
||||
|
|
||||
|
### 异常分类 |
||||
|
|
||||
|
| 异常类型 | 说明 | 示例 | |
||||
|
|---------|------|------| |
||||
|
| **受检异常** | 可恢复异常，强制调用者处理 | `HttpRequestException`, `TimeoutException`, `HtmlParseException`, `DataExtractException` | |
||||
|
| **非受检异常** | 编程错误，不可恢复 | `InvalidUrlException`, `UnsupportedCrawlerException` | |
||||
|
|
||||
|
### 异常继承树 |
||||
|
|
||||
|
``` |
||||
|
CrawlerException (爬虫框架根异常 - 受检) |
||||
|
├── NetworkException (网络异常父类) |
||||
|
│ ├── HttpRequestException (HTTP请求失败) |
||||
|
│ └── TimeoutException (连接超时) |
||||
|
└── ParseException (解析异常父类) |
||||
|
├── HtmlParseException (HTML解析失败) |
||||
|
└── DataExtractException (数据提取失败) |
||||
|
|
||||
|
ConfigurationException (配置异常父类 - 非受检) |
||||
|
├── InvalidUrlException (无效URL) |
||||
|
└── UnsupportedCrawlerException (不支持的爬虫) |
||||
|
``` |
||||
|
|
||||
|
### 异常处理示例 |
||||
|
|
||||
|
```java |
||||
|
try { |
||||
|
List<CrawlerData> data = crawler.crawl(); |
||||
|
view.showData(data); |
||||
|
} catch (HttpRequestException e) { |
||||
|
view.showErrorMessage("HTTP请求失败: " + e.getStatusCode()); |
||||
|
} catch (TimeoutException e) { |
||||
|
view.showErrorMessage("连接超时，请稍后重试"); |
||||
|
} catch (HtmlParseException e) { |
||||
|
view.showErrorMessage("HTML解析失败: " + e.getSourceUrl()); |
||||
|
} catch (CrawlerException e) { |
||||
|
view.showErrorMessage("爬虫执行失败: " + e.getMessage()); |
||||
|
} |
||||
|
``` |
||||
|
|
||||
|
完整的异常设计文档请参考 [EXCEPTIONS.md](file:///C:/Users/黄志楷/Documents/ocix/学校相关/jwork/w12/EXCEPTIONS.md) |
||||
|
|
||||
|
## 数据序列化 |
||||
|
|
||||
|
项目提供基于Jackson的JSON序列化工具类，支持将爬取数据保存到文件和从文件读取。 |
||||
|
|
||||
|
### 使用示例 |
||||
|
|
||||
|
```java |
||||
|
import com.crawler.util.JsonSerializer; |
||||
|
import com.crawler.model.CrawlerData; |
||||
|
import java.util.List; |
||||
|
|
||||
|
List<CrawlerData> dataList = crawler.crawl(); |
||||
|
|
||||
|
JsonSerializer.serializeToFile(dataList, "output/crawler_data.json"); |
||||
|
|
||||
|
List<CrawlerData> loadedData = JsonSerializer.deserializeFromFile("output/crawler_data.json"); |
||||
|
``` |
||||
|
|
||||
|
### JsonSerializer 类方法 |
||||
|
|
||||
|
| 方法 | 功能 | |
||||
|
|------|------| |
||||
|
| `serializeToFile(List<CrawlerData>, String)` | 将数据列表序列化到指定文件 | |
||||
|
| `deserializeFromFile(String)` | 从文件反序列化数据列表 | |
||||
|
| `toJsonString(List<CrawlerData>)` | 将数据列表转换为JSON字符串 | |
||||
|
| `toJsonString(CrawlerData)` | 将单条数据转换为JSON字符串 | |
||||
|
| `fromJsonString(String)` | 从JSON字符串反序列化数据列表 | |
||||
|
| `fromJsonStringToSingle(String)` | 从JSON字符串反序列化单条数据 | |
||||
|
|
||||
|
### 输出格式示例 |
||||
|
|
||||
|
```json |
||||
|
[ |
||||
|
{ |
||||
|
"title": "新闻标题", |
||||
|
"content": "新闻内容", |
||||
|
"url": "https://example.com/news/1", |
||||
|
"source": "ExampleCrawler", |
||||
|
"publishDate": "2026-05-21" |
||||
|
} |
||||
|
] |
||||
|
``` |
||||
|
|
||||
|
## 技术栈 |
||||
|
|
||||
|
- Java 21+ |
||||
|
- Java HttpClient（内置HTTP客户端） |
||||
|
- Jackson（JSON序列化） |
||||
|
- 正则表达式（HTML解析） |
||||
|
|
||||
|
## 注意事项 |
||||
|
|
||||
|
1. 请遵守目标网站的robots.txt规则 |
||||
|
2. 不要频繁请求，避免给目标服务器造成压力 |
||||
|
3. 某些网站可能有反爬机制，可能需要添加额外的请求头 |
||||
|
4. 建议在爬取前获取网站的爬取授权 |
||||
@ -0,0 +1,79 @@ |
|||||
|
<?xml version="1.0" encoding="UTF-8"?> |
||||
|
<project xmlns="http://maven.apache.org/POM/4.0.0" |
||||
|
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" |
||||
|
xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 |
||||
|
http://maven.apache.org/xsd/maven-4.0.0.xsd"> |
||||
|
<modelVersion>4.0.0</modelVersion> |
||||
|
|
||||
|
<groupId>com.crawler</groupId> |
||||
|
<artifactId>crawler-framework</artifactId> |
||||
|
<version>1.0.0</version> |
||||
|
<packaging>jar</packaging> |
||||
|
|
||||
|
<name>crawler-framework</name> |
||||
|
<description>Java MVC Crawler Framework</description> |
||||
|
|
||||
|
<properties> |
||||
|
<maven.compiler.source>11</maven.compiler.source> |
||||
|
<maven.compiler.target>11</maven.compiler.target> |
||||
|
<project.build.sourceEncoding>UTF-8</project.build.sourceEncoding> |
||||
|
</properties> |
||||
|
|
||||
|
<dependencies> |
||||
|
<dependency> |
||||
|
<groupId>junit</groupId> |
||||
|
<artifactId>junit</artifactId> |
||||
|
<version>4.13.2</version> |
||||
|
<scope>test</scope> |
||||
|
</dependency> |
||||
|
<dependency> |
||||
|
<groupId>com.fasterxml.jackson.core</groupId> |
||||
|
<artifactId>jackson-databind</artifactId> |
||||
|
<version>2.15.2</version> |
||||
|
</dependency> |
||||
|
<dependency> |
||||
|
<groupId>com.fasterxml.jackson.datatype</groupId> |
||||
|
<artifactId>jackson-datatype-jsr310</artifactId> |
||||
|
<version>2.15.2</version> |
||||
|
</dependency> |
||||
|
</dependencies> |
||||
|
|
||||
|
<build> |
||||
|
<plugins> |
||||
|
<plugin> |
||||
|
<groupId>org.apache.maven.plugins</groupId> |
||||
|
<artifactId>maven-compiler-plugin</artifactId> |
||||
|
<version>3.8.1</version> |
||||
|
<configuration> |
||||
|
<source>11</source> |
||||
|
<target>11</target> |
||||
|
</configuration> |
||||
|
</plugin> |
||||
|
|
||||
|
<plugin> |
||||
|
<groupId>org.apache.maven.plugins</groupId> |
||||
|
<artifactId>maven-assembly-plugin</artifactId> |
||||
|
<version>3.3.0</version> |
||||
|
<configuration> |
||||
|
<descriptorRefs> |
||||
|
<descriptorRef>jar-with-dependencies</descriptorRef> |
||||
|
</descriptorRefs> |
||||
|
<archive> |
||||
|
<manifest> |
||||
|
<mainClass>com.crawler.Main</mainClass> |
||||
|
</manifest> |
||||
|
</archive> |
||||
|
</configuration> |
||||
|
<executions> |
||||
|
<execution> |
||||
|
<id>make-assembly</id> |
||||
|
<phase>package</phase> |
||||
|
<goals> |
||||
|
<goal>single</goal> |
||||
|
</goals> |
||||
|
</execution> |
||||
|
</executions> |
||||
|
</plugin> |
||||
|
</plugins> |
||||
|
</build> |
||||
|
</project> |
||||
Binary file not shown.
@ -0,0 +1,23 @@ |
|||||
|
<!DOCTYPE html> |
||||
|
<html> |
||||
|
<head> |
||||
|
<title>Test Page</title> |
||||
|
</head> |
||||
|
<body> |
||||
|
<h1>测试页面</h1> |
||||
|
<div class="news-list"> |
||||
|
<article> |
||||
|
<h2><a href="https://example.com/news1">新闻标题1 - 这是第一条测试新闻</a></h2> |
||||
|
<p>这是第一条新闻的内容摘要...</p> |
||||
|
</article> |
||||
|
<article> |
||||
|
<h2><a href="https://example.com/news2">新闻标题2 - 这是第二条测试新闻</a></h2> |
||||
|
<p>这是第二条新闻的内容摘要...</p> |
||||
|
</article> |
||||
|
<article> |
||||
|
<h2><a href="https://example.com/news3">新闻标题3 - 这是第三条测试新闻</a></h2> |
||||
|
<p>这是第三条新闻的内容摘要...</p> |
||||
|
</article> |
||||
|
</div> |
||||
|
</body> |
||||
|
</html> |
||||
Loading…
Reference in new issue