feat:🎉 add season and stock and manager info crawler

README.md

@@ -14,18 +14,36 @@
 
 <img src="./screenshot/fund_list.png" style="zoom:50%;" />
 
-#### 2. 晨星基金详情页数据
+#### 2. 晨星基金详情页数据--不变数据
 
 > 爬取基金详情页的数据， 根据`晨星列表数据` 数据，遍历爬取单支基金的详情页数据（包括名称，代码，分类，成立时间，基金公司）等维度，后续还有根据这些数据爬取基金的持仓信息，为后面筛选股票做好进一步铺垫
 
 <img src="./screenshot/fund_base.png" style="zoom:50%;" />
 
+#### 3. 晨星基金详情页数据--季度变动数据
+
+> 爬取基金详情页的数据， 根据第二部分`晨星基础数据` 数据，过滤掉货币，纯债基金，爬取单支基金的详情页数据（包括总资产，投资风格，各种风险信息，评级，股票，债券持仓比例等）等维度
+
+<img src="./screenshot/fund_season.png" style="zoom:50%;" />
+
+#### 4. 晨星基金详情页数据--十大持仓股票信息
+
+> 爬取基金详情页的数据， 根据第二部分`晨星基础数据` 数据，过滤掉没有持有股票的基金，爬取单支基金的十大持仓股票信息（包括每只股票的代码，名称，占比）等维度
+
+<img src="./screenshot/fund_stock.png" style="zoom:50%;" />
+
+#### 5. 晨星基金经理
+
+> 爬取基金详情页的数据，据此爬取基金经理数据
+<img src="./screenshot/fund_manager.png" style="zoom:50%;" />
+
 ### 技术点
 
 - `selenium` 模拟登录， 切换分页
 - `BeautifulSoup` 解析 HTML
 - `pandas` 处理数据
 - 工具 — 数据库用了`pymysql` , id 使用雪花 id，验证码识别使用了`pytesseract`
+- 多线程爬取
 
 ### 爬虫流程
 
@@ -44,5 +62,6 @@
 2. 切换分页如果是最后一页时，怎么处理下一页点击
 3. 晨星评级是用图片表示，如果转化成数字表示
 4. 如何保证循环当前页与浏览器当前页一致
+5. 多线程爬取时，线程锁
 
 以上问题，我都做了相对应的处理，如果有问题的话，欢迎提 issue，私聊，star。

output/fund_morning_season_catch.csv

@@ -0,0 +1,187 @@
+代码,晨星专属号,名称,类型,股票总仓位,页码,备注
+003529, 0P00018W2Q, 汇添富长添利定期开放债券C, 普通债券型基金(封闭), 0.00, 3120, li[16]
+004832, 0P0001CBPL, 汇添富鑫泽定期开放债券C, 普通债券型基金(封闭), 0.00, 3150, li[16]
+005214, 0P0001CLLW, 华夏鼎旺三个月定期开放债券C, 普通债券型基金(封闭), 0.00, 3150, li[16]
+005365, 0P0001D9DK, 华夏鼎顺三个月定期开放债券C, 普通债券型基金(封闭), 0.00, 3160, li[16]
+005500, 0P0001DCZC, 银华岁盈定期开放债券, 普通债券型基金(封闭), 0.00, 3170, li[16]
+005792, 0P0001DKU5, 华夏鼎福三个月定期开放债券C, 普通债券型基金(封闭), 0.00, 3190, li[16]
+005858, 0P0001D51Q, 汇添富鑫成定期开放债券C, 普通债券型基金(封闭), 0.00, 3190, li[16]
+005863, 0P0001EKME, 华夏鼎禄三个月定期开放债券C, 普通债券型基金(封闭), 0.00, 3190, li[16]
+005964, 0P0001D7EG, 中欧安财定期开放债券, 普通债券型基金(封闭), 0.00, 3200, li[16]
+007712, 0P0001I7FR, 中银康享3个月定期开放债券, 普通债券型基金(封闭), 0.00, 3230, li[16]
+008139, 0P0001J0ZR, 鑫元一年定期开放中高等级债券, 普通债券型基金(封闭), 0.00, 3230, li[16]
+008289, 0P0001JOVJ, 国寿安保泰祥纯债一年定期开放债券, 普通债券型基金(封闭), 0.00, 3230, li[16]
+008360, 0P0001KJ4Z, 鹏扬淳选一年定期开放债券, 普通债券型基金(封闭), 0.00, 3240, li[16]
+008463, 0P0001JOVH, 招商添瑞1年定期开放债券A, 普通债券型基金(封闭), 0.00, 3240, li[16]
+008464, 0P0001JOVI, 招商添瑞1年定期开放债券C, 普通债券型基金(封闭), 0.00, 3240, li[16]
+008508, 0P0001JKQ4, 中融聚锦一年定期开放债券, 普通债券型基金(封闭), 0.00, 3240, li[16]
+008766, 0P0001JVY7, 财通资管鸿盛12个月定期开放债券A, 普通债券型基金(封闭), 0.00, 3250, li[16]
+008767, 0P0001JVY8, 财通资管鸿盛12个月定期开放债券C, 普通债券型基金(封闭), 0.00, 3250, li[16]
+008826, 0P0001K2ER, 天弘成享一年定期开放债券, 普通债券型基金(封闭), 0.00, 3250, li[16]
+008925, 0P0001JV4Q, 鹏华尊达一年定期开放债券, 普通债券型基金(封闭), 0.00, 3260, li[16]
+001274, 0P000187Q2, 民生加银新动力灵活配置混合D, 灵活配置型基金, 33.04, 5640, li[16]
+011420, 0P0001LKWB, 广发全球科技三个月定期开放混合（QDII）人民币A, , 0.00, 2150, li[16]
+011421, 0P0001LKWD, 广发全球科技三个月定期开放混合（QDII）美元A, , 0.00, 2150, li[16]
+011422, 0P0001LKW8, 广发全球科技三个月定期开放混合（QDII）人民币C, , 0.00, 2150, li[16]
+011423, 0P0001LKW9, 广发全球科技三个月定期开放混合（QDII）美元C, , 0.00, 2150, li[16]
+970001, 0P0001LLLX, 天风证券天泽六个月定期开放债券A, 普通债券型基金(封闭), 0.00, 3280, li[16]
+970002, 0P0001LLLY, 天风证券天泽六个月定期开放债券C, 普通债券型基金(封闭), 0.00, 3280, li[16]
+003184, 0P0001A90J, 中证财通中国可持续发展100 (ECPI ESG) 指数增强C, 普通股票型, 94.67, 3300, li[16]
+968068, 0P0001K5PY, 惠理高息股票基金 P USD Acc, 亚洲不包括日本股票, 95.10, 2180, li[16]
+005658, 0P0001CNDG, 华夏沪深300ETF联接C, 普通股票型, 0.00, 3350, None
+240010, F00000043F, 华宝行业精选混合, 积极配置型, 80.19, 1460, None
+008719, 0P0001LJ1H, 德邦安顺混合A, 保守混合型基金, 0.00, 2350, li[16]
+008720, 0P0001LJ1I, 德邦安顺混合C, 保守混合型基金, 0.00, 2350, li[16]
+001774, 0P00016GLJ, 招商丰庆灵活配置混合C, 灵活配置型基金, 0.00, 5860, li[16]
+009247, 0P0001KEQ1, 易方达磐恒九个月持有期混合A, 保守混合型基金, 18.40, 2400, li[16]
+009248, 0P0001KEQ2, 易方达磐恒九个月持有期混合C, 保守混合型基金, 18.40, 2400, li[16]
+009810, 0P0001L65F, 易方达悦通一年持有期混合A, 保守混合型基金, 0.00, 2440, li[16]
+009811, 0P0001L65G, 易方达悦通一年持有期混合C, 保守混合型基金, 0.00, 2440, li[16]
+009812, 0P0001KZ92, 易方达悦兴一年持有期混合A, 保守混合型基金, 0.00, 2440, li[16]
+009813, 0P0001KZ93, 易方达悦兴一年持有期混合C, 保守混合型基金, 0.00, 2440, li[16]
+009902, 0P0001KN1O, 易方达悦享一年持有期混合A, 保守混合型基金, 18.54, 2450, li[16]
+009903, 0P0001KN1Q, 易方达悦享一年持有期混合C, 保守混合型基金, 18.54, 2450, li[16]
+010222, 0P0001LI8H, 摩根士丹利华鑫民丰盈和一年持有期混合, 保守混合型基金, 0.00, 2480, li[16]
+009981, 0P0001LFN3, 万家创业板指数增强A, 普通股票型, 0.00, 3740, li[16]
+009982, 0P0001LFN4, 万家创业板指数增强C, 普通股票型, 0.00, 3740, li[16]
+010120, 0P0001LF71, 九泰久福量化股票A, 普通股票型, 0.00, 3750, li[16]
+010121, 0P0001LF72, 九泰久福量化股票C, 普通股票型, 0.00, 3750, li[16]
+010253, 0P0001LJ9O, 兴银中证500指数增强A, 普通股票型, 0.00, 3770, li[16]
+010322, 0P0001LLTR, 摩根士丹利华鑫新兴产业股票, 普通股票型, 0.00, 3770, li[16]
+010854, 0P0001LC8R, 汇添富沪深300基本面增强指数A, 普通股票型, 0.00, 3790, li[16]
+010855, 0P0001LC8S, 汇添富沪深300基本面增强指数C, 普通股票型, 0.00, 3790, li[16]
+010953, 0P0001LGK3, 天弘国证A50指数A, 普通股票型, 0.00, 3800, li[16]
+010954, 0P0001LGK4, 天弘国证A50指数C, 普通股票型, 0.00, 3800, li[16]
+011205, 0P0001LJ9P, 兴银中证500指数增强C, 普通股票型, 0.00, 3810, li[16]
+009794, 0P0001K9CN, 太平智选一年定期开放股票, 普通股票型(封闭), 85.34, 4010, li[16]
+004051, 0P0001AIS0, 华夏新锦升灵活配置混合C, 灵活配置型基金, 65.19, 6410, li[16]
+009677, 0P0001L3FK, 浙商智多益稳健一年持有期混合A, 沪港深保守混合型, 0.00, 4200, li[16]
+009678, 0P0001L3FL, 浙商智多益稳健一年持有期混合C, 沪港深保守混合型, 0.00, 4210, li[16]
+009725, 0P0001K7SS, 东方红优质甄选一年持有期混合, 沪港深保守混合型, 12.68, 4210, li[16]
+009806, 0P0001KEF1, 东方红招盈甄选一年持有期混合A, 沪港深保守混合型, 12.37, 4220, li[16]
+009807, 0P0001KEF2, 东方红招盈甄选一年持有期混合C, 沪港深保守混合型, 12.37, 4220, li[16]
+009977, 0P0001L2WK, 银华招利一年持有期混合A, 沪港深保守混合型, 0.00, 4220, li[16]
+009978, 0P0001L2WL, 银华招利一年持有期混合C, 沪港深保守混合型, 0.00, 4230, li[16]
+010219, 0P0001LANH, 汇添富稳健添益一年持有期混合, 沪港深保守混合型, 0.00, 4240, li[16]
+010242, 0P0001L65L, 平安稳健增长混合A, 沪港深保守混合型, 0.00, 4240, li[16]
+010243, 0P0001L65M, 平安稳健增长混合C, 沪港深保守混合型, 0.00, 4240, li[16]
+010404, 0P0001L8MI, 博道盛利6个月持有期混合, 沪港深保守混合型, 0.00, 4240, li[16]
+010408, 0P0001L9PX, 安信浩盈6个月持有期混合, 沪港深保守混合型, 0.00, 4240, li[16]
+010480, 0P0001LFTD, 汇添富稳进双盈一年持有期混合, 沪港深保守混合型, 0.00, 4240, li[16]
+010683, 0P0001L9PO, 中融景颐6个月持有期混合A, 沪港深保守混合型, 0.00, 4260, li[16]
+010684, 0P0001L9PP, 中融景颐6个月持有期混合C, 沪港深保守混合型, 0.00, 4260, li[16]
+010688, 0P0001LDH8, 招商瑞德一年持有期混合A, 沪港深保守混合型, 0.00, 4260, li[16]
+010689, 0P0001LDH9, 招商瑞德一年持有期混合C, 沪港深保守混合型, 0.00, 4260, li[16]
+010707, 0P0001LFN0, 安信平稳合盈一年持有期混合A, 沪港深保守混合型, 0.00, 4260, li[16]
+010708, 0P0001LFN1, 安信平稳合盈一年持有期混合C, 沪港深保守混合型, 0.00, 4260, li[16]
+010725, 0P0001L3AI, 鹏华安享一年持有期混合A, 沪港深保守混合型, 0.00, 4270, li[16]
+010726, 0P0001L3AJ, 鹏华安享一年持有期混合C, 沪港深保守混合型, 0.00, 4270, li[16]
+010762, 0P0001L65A, 博时恒康一年持有期混合A, 沪港深保守混合型, 0.00, 4270, li[16]
+010763, 0P0001L65B, 博时恒康一年持有期混合C, 沪港深保守混合型, 0.00, 4270, li[16]
+010775, 0P0001L6OO, 博时恒旭一年持有期混合A, 沪港深保守混合型, 0.00, 4270, li[16]
+010776, 0P0001L6OP, 博时恒旭一年持有期混合C, 沪港深保守混合型, 0.00, 4270, li[16]
+010797, 0P0001LDHO, 长城优选回报六个月持有期混合A, 沪港深保守混合型, 0.00, 4270, li[16]
+010798, 0P0001LDHP, 长城优选回报六个月持有期混合C, 沪港深保守混合型, 0.00, 4280, li[16]
+010830, 0P0001LFTF, 国泰通利9个月持有期混合A, 沪港深保守混合型, 0.00, 4280, li[16]
+010831, 0P0001LFTG, 国泰通利9个月持有期混合C, 沪港深保守混合型, 0.00, 4280, li[16]
+010832, 0P0001LI8E, 国泰合益混合A, 沪港深保守混合型, 0.00, 4280, li[16]
+010833, 0P0001LI8F, 国泰合益混合C, 沪港深保守混合型, 0.00, 4280, li[16]
+010843, 0P0001LLTL, 富国天润回报混合A, 沪港深保守混合型, 0.00, 4280, li[16]
+010844, 0P0001LLTK, 富国天润回报混合C, 沪港深保守混合型, 0.00, 4280, li[16]
+010879, 0P0001L8CX, 南方宝升混合A, 沪港深保守混合型, 0.00, 4290, li[16]
+010880, 0P0001L8CY, 南方宝升混合C, 沪港深保守混合型, 0.00, 4290, li[16]
+010904, 0P0001L8VO, 博时双季鑫6个月持有期混合A, 沪港深保守混合型, 0.00, 4290, li[16]
+010905, 0P0001L8VQ, 博时双季鑫6个月持有期混合C, 沪港深保守混合型, 0.00, 4290, li[16]
+010924, 0P0001L8VP, 博时双季鑫6个月持有期混合B, 沪港深保守混合型, 0.00, 4290, li[16]
+011017, 0P0001LCI7, 鹏扬景明一年持有期混合, 沪港深保守混合型, 0.00, 4300, li[16]
+011044, 0P0001LDPM, 中银顺泽回报一年持有期混合A, 沪港深保守混合型, 0.00, 4300, li[16]
+011045, 0P0001LDPN, 中银顺泽回报一年持有期混合C, 沪港深保守混合型, 0.00, 4300, li[16]
+011052, 0P0001L4NZ, 鹏华弘裕一年持有期混合A, 沪港深保守混合型, 0.00, 4300, li[16]
+011053, 0P0001L4O0, 鹏华弘裕一年持有期混合C, 沪港深保守混合型, 0.00, 4300, li[16]
+011071, 0P0001L13N, 鹏华安悦一年持有期混合A, 沪港深保守混合型, 0.00, 4300, li[16]
+011072, 0P0001L13O, 鹏华安悦一年持有期混合C, 沪港深保守混合型, 0.00, 4300, li[16]
+011095, 0P0001LERZ, 博时恒泽混合A, 沪港深保守混合型, 0.00, 4310, li[16]
+011096, 0P0001LES0, 博时恒泽混合C, 沪港深保守混合型, 0.00, 4310, li[16]
+011118, 0P0001LM6W, 汇添富稳健睿选一年持有期混合A, 沪港深保守混合型, 0.00, 4310, li[16]
+011119, 0P0001LM6Z, 汇添富稳健睿选一年持有期混合C, 沪港深保守混合型, 0.00, 4310, li[16]
+011175, 0P0001LG30, 平安恒鑫混合A, 沪港深保守混合型, 0.00, 4310, li[16]
+011176, 0P0001LG31, 平安恒鑫混合C, 沪港深保守混合型, 0.00, 4310, li[16]
+011302, 0P0001LJ17, 易方达悦盈一年持有期混合A, 沪港深保守混合型, 0.00, 4310, li[16]
+011303, 0P0001LJ18, 易方达悦盈一年持有期混合C, 沪港深保守混合型, 0.00, 4310, li[16]
+011336, 0P0001LJ9C, 兴全汇吉一年持有期混合A, 沪港深保守混合型, 0.00, 4310, li[16]
+011337, 0P0001LJ9D, 兴全汇吉一年持有期混合C, 沪港深保守混合型, 0.00, 4310, li[16]
+011393, 0P0001LLIT, 中欧融益稳健一年持有期混合A, 沪港深保守混合型, 0.00, 4320, li[16]
+011394, 0P0001LLIV, 中欧融益稳健一年持有期混合C, 沪港深保守混合型, 0.00, 4320, li[16]
+011508, 0P0001LJPV, 易方达悦弘一年持有期混合A, 沪港深保守混合型, 0.00, 4320, li[16]
+011509, 0P0001LJPW, 易方达悦弘一年持有期混合C, 沪港深保守混合型, 0.00, 4320, li[16]
+011516, 0P0001LMPB, 嘉实浦盈一年持有期混合A, 沪港深保守混合型, 0.00, 4320, li[16]
+011517, 0P0001LMPC, 嘉实浦盈一年持有期混合C, 沪港深保守混合型, 0.00, 4320, li[16]
+011525, 0P0001LN6O, 中信保诚丰裕一年持有期混合A, 沪港深保守混合型, 0.00, 4320, li[16]
+011526, 0P0001LN6N, 中信保诚丰裕一年持有期混合C, 沪港深保守混合型, 0.00, 4320, li[16]
+011580, 0P0001LP0Z, 民生加银稳健配置6个月持有期混合（FOF）, 沪港深保守混合型, 0.00, 4320, li[16]
+008263, 0P0001IYR0, 东方红品质优选两年定期开放混合, 沪港深保守混合型(封闭), 25.10, 4330, li[16]
+008416, 0P0001IYR1, 鹏扬景瑞三年定期开放混合A, 沪港深保守混合型(封闭), 28.33, 4330, li[16]
+008417, 0P0001IYR2, 鹏扬景瑞三年定期开放混合C, 沪港深保守混合型(封闭), 28.33, 4330, li[16]
+009031, 0P0001JKQ8, 工银瑞信聚和一年定期开放混合A, 沪港深保守混合型(封闭), 19.94, 4330, li[16]
+009032, 0P0001JKQ9, 工银瑞信聚和一年定期开放混合C, 沪港深保守混合型(封闭), 19.94, 4330, li[16]
+009144, 0P0001JGSZ, 博时荣升稳健添利18个月定期开放混合A, 沪港深保守混合型(封闭), 5.25, 4330, li[16]
+009145, 0P0001JGT0, 博时荣升稳健添利18个月定期开放混合C, 沪港深保守混合型(封闭), 5.25, 4340, li[16]
+009691, 0P0001K9CH, 国泰浩益18个月封闭运作混合A, 沪港深保守混合型(封闭), 20.93, 4340, li[16]
+009692, 0P0001K9CI, 国泰浩益18个月封闭运作混合C, 沪港深保守混合型(封闭), 20.93, 4340, li[16]
+009718, 0P0001L3L9, 招商增浩一年定期开放混合A, 沪港深保守混合型(封闭), 0.00, 4340, li[16]
+009719, 0P0001L3LA, 招商增浩一年定期开放混合C, 沪港深保守混合型(封闭), 0.00, 4340, li[16]
+009842, 0P0001KF5S, 东方红明鉴优选两年定期开放混合, 沪港深保守混合型(封闭), 23.01, 4340, li[16]
+009927, 0P0001L725, 工银瑞信聚利18个月定期开放混合A, 沪港深保守混合型(封闭), 0.00, 4340, li[16]
+009928, 0P0001L726, 工银瑞信聚利18个月定期开放混合C, 沪港深保守混合型(封闭), 0.00, 4340, li[16]
+010700, 0P0001LE90, 东方红锦丰优选两年定期开放混合, 沪港深保守混合型(封闭), 0.00, 4350, li[16]
+169108, 0P0001JBT1, 东方红均衡优选两年定期开放混合, 沪港深保守混合型(封闭), 23.80, 4350, li[16]
+005950, 0P0001DCLC, 鑫元行业轮动灵活配置混合C, 沪港深灵活配置型, 79.31, 4400, li[16]
+010902, 0P0001LF6V, 博时成长领航灵活配置混合A, 沪港深灵活配置型, 0.00, 4480, li[16]
+010903, 0P0001LF6W, 博时成长领航灵活配置混合C, 沪港深灵活配置型, 0.00, 4480, li[16]
+011021, 0P0001LFEG, 汇添富互联网核心资产六个月持有期混合A, 沪港深灵活配置型, 0.00, 4480, li[16]
+011022, 0P0001LFEH, 汇添富互联网核心资产六个月持有期混合C, 沪港深灵活配置型, 0.00, 4480, li[16]
+011056, 0P0001LD48, 博时汇兴回报一年持有期灵活配置混合, 沪港深灵活配置型, 0.00, 4480, li[16]
+007207, 0P0001I7G1, 华夏常阳三年定期开放混合, 沪港深灵活配置型(封闭), 79.86, 4510, li[16]
+007639, 0P0001I3PR, 汇添富3年封闭运作竞争优势灵活配置混合, 沪港深灵活配置型(封闭), 68.57, 4510, li[16]
+008966, 0P0001J4VX, 博时成长优选两年封闭运作灵活配置混合A, 沪港深灵活配置型(封闭), 88.65, 4510, li[16]
+008967, 0P0001J4VY, 博时成长优选两年封闭运作灵活配置混合C, 沪港深灵活配置型(封闭), 88.65, 4510, li[16]
+009217, 0P0001JJ2U, 博时荣丰回报三年封闭运作灵活配置混合A, 沪港深灵活配置型(封闭), 89.04, 4520, li[16]
+009218, 0P0001JJ2V, 博时荣丰回报三年封闭运作灵活配置混合C, 沪港深灵活配置型(封闭), 89.04, 4520, li[16]
+009715, 0P0001K69T, 汇添富策略增长两年封闭运作灵活配置混合, 沪港深灵活配置型(封闭), 90.22, 4520, li[16]
+970015, 0P0001LM7F, 申万宏源红利成长灵活配置混合, 灵活配置型基金, 0.00, 6990, li[16]
+007501, 0P0001IF4Z, 万家科创主题3年封闭运作灵活配置混合C, 灵活配置型基金(封闭), 78.84, 7030, li[16]
+008890, 0P0001K18K, 中邮价值优选一年定期开放灵活配置混合, 灵活配置型基金(封闭), 89.45, 7040, li[16]
+009048, 0P0001JET9, 浦银安盛科技创新优选三年封闭运作灵活配置混合, 灵活配置型基金(封闭), 41.48, 7040, li[16]
+009595, 0P0001K0AL, 山西证券裕盛一年定期开放灵活配置混合, 灵活配置型基金(封闭), 93.91, 7040, li[16]
+010130, 0P0001KOAP, 海富通惠增多策略一年定期开放灵活配置混合A, 灵活配置型基金(封闭), 24.81, 7040, li[16]
+010131, 0P0001KOAQ, 海富通惠增多策略一年定期开放灵活配置混合C, 灵活配置型基金(封闭), 24.81, 7040, li[16]
+008480, 0P0001JC4Z, 永赢股息优选混合A, 沪港深积极配置型, 93.91, 4810, None
+008481, 0P0001JC50, 永赢股息优选混合C, 沪港深积极配置型, 93.91, 4810, None
+009342, 0P0001JV4P, 易方达优质企业三年持有期混合, 沪港深积极配置型, 94.64, 4900, li[16]
+009859, 0P0001L5TN, 银华乐享混合, 沪港深积极配置型, 0.00, 4950, li[16]
+007315, 0P0001L9PQ, 汇安嘉盈一年持有期债券A, 积极债券型, 0.00, 7300, li[16]
+010041, 0P0001LFER, 嘉实港股优势混合A, 沪港深积极配置型, 0.00, 4990, li[16]
+010042, 0P0001LFES, 嘉实港股优势混合C, 沪港深积极配置型, 0.00, 4990, li[16]
+009465, 0P0001LL7B, 东方可转债债券A, 积极债券型, 0.00, 7350, li[16]
+009466, 0P0001LL7A, 东方可转债债券C, 积极债券型, 0.00, 7350, li[16]
+009943, 0P0001LDHB, 浦银安盛稳健丰利债券A, 积极债券型, 0.00, 7360, li[16]
+009944, 0P0001LDHC, 浦银安盛稳健丰利债券C, 积极债券型, 0.00, 7370, li[16]
+010270, 0P0001L9PR, 汇安嘉盈一年持有期债券C, 积极债券型, 0.00, 7380, li[16]
+010473, 0P0001LDPU, 华富安华债券A, 积极债券型, 0.00, 7390, li[16]
+010474, 0P0001LDPV, 华富安华债券C, 积极债券型, 0.00, 7390, li[16]
+010475, 0P0001LFEI, 上投摩根安享回报一年持有期债券, 积极债券型, 0.00, 7390, li[16]
+010619, 0P0001LEKC, 华安添利6个月持有期债券A, 积极债券型, 0.00, 7390, li[16]
+010620, 0P0001LEKD, 华安添利6个月持有期债券C, 积极债券型, 0.00, 7390, li[16]
+010651, 0P0001L7ZV, 平安双季增享6个月持有期债券A, 积极债券型, 0.00, 7390, li[16]
+010652, 0P0001L7ZW, 平安双季增享6个月持有期债券C, 积极债券型, 0.00, 7400, li[16]
+010803, 0P0001LF73, 天弘庆享债券A, 积极债券型, 0.00, 7400, li[16]
+010804, 0P0001LF74, 天弘庆享债券C, 积极债券型, 0.00, 7400, li[16]
+010899, 0P0001LCTL, 上银慧恒收益增强债券, 积极债券型, 0.00, 7400, li[16]
+011168, 0P0001LGU5, 嘉实睿享安久双利18个月持有期债券, 积极债券型, 0.00, 7400, li[16]
+952020, 0P0001LLTN, 国君资管君得盈债券A, 积极债券型, 0.00, 7410, li[16]
+952320, 0P0001LLTM, 国君资管君得盈债券C, 积极债券型, 0.00, 7410, li[16]
+008743, 0P0001JAAS, 南方集利18个月定期开放债券A, 积极债券型(封闭), 8.36, 7440, li[16]
+008744, 0P0001JAAT, 南方集利18个月定期开放债券C, 积极债券型(封闭), 8.36, 7440, li[16]
+009158, 0P0001JS1A, 中金衡利1年定期开放债券, 积极债券型(封闭), 11.13, 7440, li[16]
+010493, 0P0001L1LX, 中航瑞昱一年定期开放债券A, 积极债券型(封闭), 0.00, 7450, li[16]
+010494, 0P0001L1LY, 中航瑞昱一年定期开放债券C, 积极债券型(封闭), 0.00, 7450, li[16]

output/fund_morning_season_error.csv

@@ -0,0 +1,14 @@
+代码,晨星专属号,名称,类型,页码,备注
+000435, 0P000101H1, 建信稳定添利债券A, 积极债券型, 1180, 页面跳转有问题
+004182, 0P0001ABLY, 建信瑞福添利混合A, 保守混合型基金, 2090, 页面跳转有问题
+004468, 0P0001ABLZ, 建信瑞福添利混合C, 保守混合型基金, 2090, 页面跳转有问题
+512340, 0P00015KX3, 南方中证500原材料指数ETF, 普通股票型, 1280, 页面跳转有问题
+000723, 0P0001421O, 建信稳定添利债券C, 积极债券型, 1290, 页面跳转有问题
+005747, 0P0001ECUP, 华安MSCI中国A股国际ETF联接A, 普通股票型, 3360, 页面跳转有问题
+005748, 0P0001ECUQ, 华安MSCI中国A股国际ETF联接C, 普通股票型, 3360, 页面跳转有问题
+006912, 0P0001FIFH, 长城久泰沪深300指数 C, 普通股票型, 3470, 页面跳转有问题
+001724, 0P00016FID, 申万菱信多策略灵活配置混合C, 灵活配置型基金, 5830, 页面跳转有问题
+512860, 0P0001E1FC, 华安MSCI中国A股国际ETF, 普通股票型, 3930, 页面跳转有问题
+005769, 0P0001DGGJ, 南方共享经济灵活配置混合, 沪港深保守混合型, 4100, 页面跳转有问题
+006362, 0P0001EG9S, 凯石涵行业精选混合A, 积极配置型, 7530, 页面跳转有问题
+006815, 0P0001FIB4, 凯石涵行业精选混合C, 积极配置型, 7560, 页面跳转有问题

src/acquire_fund_season.py

@@ -15,6 +15,7 @@ from utils import parse_cookiestr, set_cookies, login_site
 from fund_info_crawler import FundSpider
 from lib.mysnowflake import IdWorker
 from time import sleep, time
+from pprint import pprint
 import pymysql
 import pandas
 connect = pymysql.connect(host='127.0.0.1', user='root',
@@ -42,6 +43,23 @@ def login():
     return chrome_driver
 
 
+def generate_insert_sql(target_dict, table_name, ignore_list):
+    keys = ','.join(target_dict.keys())
+    values = ','.join(['%s'] * len(target_dict))
+    update_values = ''
+    for key in target_dict.keys():
+        if key in ignore_list:
+            continue
+        update_values = update_values + '{0}=VALUES({0}),'.format(key)
+    sql_insert = "INSERT INTO {table} ({keys}) VALUES ({values})  ON DUPLICATE KEY UPDATE {update_values}; ".format(
+        table=table_name,
+        keys=keys,
+        values=values,
+        update_values=update_values[0:-1]
+    )
+    return sql_insert
+
+
 if __name__ == '__main__':
     # 过滤没有股票持仓的基金
     sql_count = "SELECT COUNT(1) FROM fund_morning_base \
@@ -81,21 +99,19 @@ if __name__ == '__main__':
     def crawlData(start, end):
         chrome_driver = login()
         morning_cookies = chrome_driver.get_cookies()
-        print('end', end)
         page_start = start
         page_limit = 10
         while(page_start < end):
-            sql = "SELECT fund_morning_base.fund_code,\
-            fund_morning_base.morning_star_code, fund_morning_base.fund_name, fund_morning_base.fund_cat, \
-            fund_morning_snapshot.fund_rating_3,fund_morning_snapshot.fund_rating_5 \
-            FROM fund_morning_base \
-            LEFT JOIN fund_morning_snapshot ON fund_morning_snapshot.fund_code = fund_morning_base.fund_code \
-            WHERE fund_morning_base.fund_cat NOT LIKE '%%货币%%' \
-            AND fund_morning_base.fund_cat NOT LIKE '%%纯债基金%%' \
-            AND fund_morning_base.fund_cat NOT LIKE '目标日期' \
-            AND fund_morning_base.fund_cat NOT LIKE '%%短债基金%%' \
-            ORDER BY fund_morning_snapshot.fund_rating_5 DESC,fund_morning_snapshot.fund_rating_3 DESC, \
-            fund_morning_base.fund_cat, fund_morning_base.fund_code LIMIT %s, %s"
+            sql = "SELECT t.fund_code,\
+            t.morning_star_code, t.fund_name, t.fund_cat \
+            FROM fund_morning_base as t \
+            LEFT JOIN fund_morning_snapshot as f ON f.fund_code = t.fund_code \
+            WHERE t.fund_cat NOT LIKE '%%货币%%' \
+            AND t.fund_cat NOT LIKE '%%纯债基金%%' \
+            AND t.fund_cat NOT LIKE '目标日期' \
+            AND t.fund_cat NOT LIKE '%%短债基金%%' \
+            ORDER BY f.fund_rating_5 DESC,f.fund_rating_3 DESC, \
+            t.fund_cat, t.fund_code LIMIT %s, %s"
             lock.acquire()
             cursor.execute(
                 sql, [page_start, page_limit])    # 执行sql语句
@@ -118,26 +134,108 @@ if __name__ == '__main__':
                         csv_file.write(output_line)
                     lock.release()
                     continue
-                # each_fund.get_fund_manager_info()
-                each_fund.get_fund_morning_rating()
-                # each_fund.get_fund_season_info()
-                continue
+                # 开始爬取数据
+                each_fund.get_fund_season_info()  # 基本数据
+                each_fund.get_fund_manager_info()  # 基金经理模块
+                each_fund.get_fund_morning_rating()  # 基金晨星评级
+                each_fund.get_fund_qt_rating()  # 基金风险评级
+                if each_fund.stock_position['total'] != '0.00' and each_fund.total_asset != None:
+                    each_fund.get_asset_composition_info()
                 if each_fund._is_trigger_catch == True:
                     lock.acquire()
                     fund_infos = [each_fund.fund_code, each_fund.morning_star_code,
                                   each_fund.fund_name, record[3],
-                                  each_fund.stock_position['stock_total_position'],
+                                  each_fund.stock_position['total'],
                                   page_start, each_fund._catch_detail]
                     with open(result_dir + 'fund_morning_season_catch.csv', 'a') as csv_file:
                         output_line = ', '.join(str(x)
                                                 for x in fund_infos) + '\n'
                         csv_file.write(output_line)
                     lock.release()
-                fundDict = dict((name, getattr(each_fund, name))
-                                for name in vars(each_fund)
-                                if not (name.startswith('_') or getattr(each_fund, name) == None))
-
-                # print(current_thread().getName(), fundDict)
+                # 入库
+                lock.acquire()
+                snow_flake_id = IdWorker.get_id()
+                lock.release()
+                # 基金经理
+                if each_fund.manager.get('id'):
+                    manager_dict = {
+                        'id': snow_flake_id,
+                        'manager_id': each_fund.manager.get('id'),
+                        'name': each_fund.manager.get('name'),
+                        'brife': each_fund.manager.get('brife')
+                    }
+                    manager_sql_insert = generate_insert_sql(
+                        manager_dict, 'fund_morning_manager', ['id', 'manager_id', 'name'])
+                    lock.acquire()
+                    cursor.execute(manager_sql_insert,
+                                   tuple(manager_dict.values()))
+                    connect.commit()
+                    lock.release()
+                season_dict = {
+                    'id': snow_flake_id,
+                    'season_number': each_fund.season_number,
+                    'fund_code': each_fund.fund_code,
+                    'investname_style': each_fund.investname_style,
+                    'total_asset': each_fund.total_asset,
+                    'manager_id': each_fund.manager.get('id'),
+                    'manager_start_date': each_fund.manager.get('start_date'),
+                    'three_month_retracement': each_fund.three_month_retracement,
+                    'june_month_retracement': each_fund.june_month_retracement,
+                    'risk_statistics_alpha': each_fund.risk_statistics.get('alpha'),
+                    'risk_statistics_beta': each_fund.risk_statistics.get('beta'),
+                    'risk_statistics_r_square': each_fund.risk_statistics.get('r_square'),
+                    'risk_assessment_standard_deviation': each_fund.risk_assessment.get('standard_deviation'),
+                    'risk_assessment_risk_coefficient': each_fund.risk_assessment.get('risk_coefficient'),
+                    'risk_assessment_sharpby': each_fund.risk_assessment.get('sharpby'),
+                    'risk_rating_2': each_fund.risk_rating.get(2),
+                    'risk_rating_3': each_fund.risk_rating.get(3),
+                    'risk_rating_5': each_fund.risk_rating.get(5),
+                    'risk_rating_10': each_fund.risk_rating.get(10),
+                    'stock_position_total': each_fund.stock_position.get('total'),
+                    'stock_position_ten': each_fund.stock_position.get('ten'),
+                    'bond_position_total': each_fund.bond_position.get('total'),
+                    'bond_position_five': each_fund.bond_position.get('five'),
+                    'morning_star_rating_3': each_fund.morning_star_rating.get(3),
+                    'morning_star_rating_5': each_fund.morning_star_rating.get(5),
+                    'morning_star_rating_10': each_fund.morning_star_rating.get(10),
+                }
+                season_sql_insert = generate_insert_sql(
+                    season_dict, 'fund_morning_season', ['id', 'season_number', 'fund_code'])
+                lock.acquire()
+                cursor.execute(season_sql_insert,
+                               tuple(season_dict.values()))
+                connect.commit()
+                lock.release()
+                # 入库十大股票持仓
+                stock_position_total = each_fund.stock_position.get(
+                    'total', '0.00')
+                if float(stock_position_total) > 0:
+                    stock_dict = {
+                        'id': snow_flake_id,
+                        'season_number': each_fund.season_number,
+                        'fund_code': each_fund.fund_code,
+                        'stock_position_total': each_fund.stock_position.get('total'),
+                    }
+                    for index in range(len(each_fund.ten_top_stock_list)):
+                        temp_stock = each_fund.ten_top_stock_list[index]
+                        prefix = 'top_stock_' + str(index) + '_'
+                        code_key = prefix + 'code'
+                        stock_dict[code_key] = temp_stock['stock_code']
+                        name_key = prefix + 'name'
+                        stock_dict[name_key] = temp_stock['stock_name']
+                        portion_key = prefix + 'portion'
+                        stock_dict[portion_key] = temp_stock['stock_portion']
+                        market_key = prefix + 'market'
+                        stock_dict[market_key] = temp_stock['stock_market']
+                    stock_sql_insert = generate_insert_sql(
+                        stock_dict, 'fund_morning_stock_info', ['id', 'season_number', 'fund_code'])
+                    lock.acquire()
+                    # print('stock_sql_insert', stock_sql_insert)
+                    cursor.execute(stock_sql_insert,
+                                   tuple(stock_dict.values()))
+                    connect.commit()
+                    lock.release()
+                # pprint(fundDict)
             page_start = page_start + page_limit
             print(current_thread().getName(), 'page_start', page_start)
             sleep(3)
@@ -145,10 +243,27 @@ if __name__ == '__main__':
     threaders = []
     start = time()
     step_num = 2500
-    for i in range(1):
-        print(i * step_num, (i+1) * step_num)
-        t = Thread(target=crawlData, args=(
-            i * step_num, (i+1) * step_num))
+    # steps = [{
+    #     "start": 800,
+    #     "end": 2500
+    # }, {
+    #     "start": 2740,
+    #     "end": 5000
+    # }, {
+    #     "start": 5100,
+    #     "end": 7500
+    # }, {
+    #     "start": 8300,
+    #     "end": record_total
+    # }]
+    for i in range(4):
+        skip_num = 100
+        # print(i * step_num + skip_num, (i+1) * step_num)
+        # start = steps[i]['start']
+        # end = steps[i]['end']
+        start = i * step_num
+        end = (i + 1) * step_num
+        t = Thread(target=crawlData, args=(start, end))
         t.setDaemon(True)
         threaders.append(t)
         t.start()

src/fund_info_crawler.py

@@ -36,10 +36,13 @@ class FundSpider:
         self.total_asset = None  # 总资产
         self.investname_style = None  # 投资风格
         self.manager = dict()  # 基金经理,name,id,管理时间
-        self.three_month_retracement = 0.0  # 三个月最大回撤
-        self.bond_position = dict()  # 债券总仓位、前五大持仓
-        self.stock_position = dict()  # 股票总仓位、前十大持仓
-        self.risk_assessment = dict()  # 标准差 风险系数 夏普比
+        self.three_month_retracement = 0.0  # 最差六个月回报
+        self.june_month_retracement = 0.0  # 最差六个月回报
+        self.bond_position = dict(
+            {'total': '0.00'})  # 债券总仓位、前五大持仓
+        self.stock_position = dict(
+            {'total': '0.00'})  # 股票总仓位、前十大持仓
+        self.risk_assessment = dict()  # 三年风险评估 -- 标准差 风险系数 夏普比
         self.risk_statistics = dict()  # 阿尔法 贝塔 R平方值
         self.risk_rating = dict()  # 风险评价 -- 二年、三年、五年、十年
         self.morning_star_rating = dict()  # 晨星评级--三年，五年，十年
@@ -106,10 +109,10 @@ class FundSpider:
             self._is_trigger_catch = True
             self._catch_detail = parent_id + '-' + class_name
             print('error_fund_info:', self.fund_code,
-                  '-', self.morning_star_code, self.stock_position["stock_total_position"])
+                  '-', self.morning_star_code, self.stock_position["total"])
             file_name = './abnormal/' + self.fund_code + \
                 '-' + parent_id + "-no_such_element.png"
-            self._chrome_driver.save_screenshot(file_name)
+            # self._chrome_driver.save_screenshot(file_name)
             # driver.get_screenshot_as_file(file_name)
             # raise  # 抛出异常，注释后则不抛出异常
         return None
@@ -123,9 +126,9 @@ class FundSpider:
             self._is_trigger_catch = True
             self._catch_detail = id
             print('error_fund_info:', self.fund_code,
-                  '-', self.morning_star_code, self.stock_position["stock_total_position"])
+                  '-', self.morning_star_code, self.stock_position["total"])
             file_name = './abnormal/' + '-' + id + self.fund_code + "-no_such_element.png"
-            self._chrome_driver.save_screenshot(file_name)
+            # self._chrome_driver.save_screenshot(file_name)
             # driver.get_screenshot_as_file(file_name)
             # raise  # 抛出异常，注释后则不抛出异常
         return None
@@ -143,10 +146,10 @@ class FundSpider:
             self._is_trigger_catch = True
             self._catch_detail = xpath
             print('error_fund_info:', self.fund_code,
-                  '-', self.morning_star_code, self.stock_position["stock_total_position"])
+                  '-', self.morning_star_code, self.stock_position["total"])
             file_name = './abnormal/' + \
                 self.fund_code + '-' + xpath + "-no_such_element.png"
-            self._chrome_driver.save_screenshot(file_name)
+            # self._chrome_driver.save_screenshot(file_name)
             # driver.get_screenshot_as_file(file_name)
             # raise  # 抛出异常，注释后则不抛出异常
         return None
@@ -185,7 +188,7 @@ class FundSpider:
             print('error_fund_info:', self.fund_code,
                   '-', self.morning_star_code)
             file_name = './abnormal/manager-' + self.fund_code + "-no_such_element.png"
-            self._chrome_driver.save_screenshot(file_name)
+            # self._chrome_driver.save_screenshot(file_name)
             # driver.get_screenshot_as_file(file_name)
             # raise  # 抛出异常，注释后则不抛出异常
         return None
@@ -202,11 +205,9 @@ class FundSpider:
             rating_3 = get_star_count(rating_3_src)
             rating_5 = get_star_count(rating_5_src)
             rating_10 = get_star_count(rating_10_src)
-            print(rating_3, rating_5, rating_10)
             self.morning_star_rating[3] = rating_3
             self.morning_star_rating[5] = rating_5
             self.morning_star_rating[10] = rating_10
-            print(self.morning_star_rating)
         except NoSuchElementException:
             self._is_trigger_catch = True
             print('error_fund_info:', self.fund_code,
@@ -226,7 +227,6 @@ class FundSpider:
                 "li[7]/img").get_attribute('src')
             rating_10_src = qt_el.find_element_by_xpath(
                 "li[8]/img").get_attribute('src')
-            print(rating_3_src, rating_5_src, rating_10_src)
             # //*[@id="qt_rating"]/li[6]/img
             rating_2 = re.findall(
                 r"\d(?:stars\.gif)$", rating_2_src)[0][0]
@@ -236,12 +236,10 @@ class FundSpider:
                 r"\d(?:stars\.gif)$", rating_5_src)[0][0]
             rating_10 = re.findall(
                 r"\d(?:stars\.gif)$", rating_10_src)[0][0]
-            print(rating_2, rating_3, rating_5, rating_10)
-            self.risk_rating['2'] = rating_2
-            self.risk_rating['3'] = rating_3
-            self.risk_rating['5'] = rating_5
-            self.risk_rating['10'] = rating_10
-            print(self.risk_rating)
+            self.risk_rating[2] = rating_2
+            self.risk_rating[3] = rating_3
+            self.risk_rating[5] = rating_5
+            self.risk_rating[10] = rating_10
         except NoSuchElementException:
             self._is_trigger_catch = True
             print('error_fund_info:', self.fund_code,
@@ -249,22 +247,26 @@ class FundSpider:
             file_name = './abnormal/qt_rating-' + self.fund_code + "-no_such_element.png"
 
     def get_fund_season_info(self):
-        # 投资风格
-        self.investname_style = self.get_element_text_by_class_name(
-            'sbdesc', 'qt_base')
         # 总资产
         self.total_asset = self.get_element_text_by_class_name(
             "asset", 'qt_base')
-        # 三位最大回撤
+        # 投资风格
+        self.investname_style = self.get_element_text_by_class_name(
+            'sbdesc', 'qt_base')
+        # 最差三个月回报
         self.three_month_retracement = self.get_element_text_by_class_name(
             "r3", 'qt_worst')
+        # 最差六个月回报
+        self.june_month_retracement = self.get_element_text_by_class_name(
+            "r6", 'qt_worst')
         # 获取股票总仓位、前十大持仓、债券总仓位、前五大持仓
-        self.stock_position["stock_total_position"] = self.get_element_text_by_class_name(
-            "stock", 'qt_asset')
-        # self.stock_total_position["stock_total_position"] = float(
-        #     stock_total_position) / 100 if stock_total_position != '-' else None  # 股票的总仓位
-        self.bond_position["bond_total_position"] = self.get_element_text_by_class_name(
+        total = self.get_element_text_by_class_name(
             "stock", 'qt_asset')
+        self.stock_position["total"] = total if total != None else '0.00'
+        # self.total["total"] = float(
+        #     total) / 100 if total != '-' else None  # 股票的总仓位
+        self.bond_position["total"] = self.get_element_text_by_class_name(
+            "bonds", 'qt_asset')
 
         # 十大股票仓位
         ten_stock_position = None
@@ -274,7 +276,7 @@ class FundSpider:
                 r"\d+\.?\d*", ten_stock_position_text)
             if len(ten_stock_position_list) > 0:
                 ten_stock_position = ten_stock_position_list.pop(0)
-        self.stock_position["ten_stock_position"] = ten_stock_position
+        self.stock_position["ten"] = ten_stock_position
 
         # 五大债券仓位
         five_bond_position = None
@@ -284,7 +286,7 @@ class FundSpider:
                 r"\d+\.?\d*", five_bond_position_text)
             if len(five_bond_position_list) > 0:
                 five_bond_position = five_bond_position_list.pop(0)
-        self.bond_position["five_bond_position"] = five_bond_position
+        self.bond_position["five"] = five_bond_position
 
         # 获取标准差
         # standard_deviation = self._chrome_driver.find_element_by_id(
@@ -301,10 +303,6 @@ class FundSpider:
             sharpby = self.get_element_text_by_xpath(
                 'li[30]', 'qt_risk')
             self.risk_assessment["sharpby"] = sharpby
-            # sharpby = self._chrome_driver.find_element_by_id(
-            #     "qt_risk").find_elements_by_xpath('li').pop(29).text
-            # self.risk_assessment["sharpby"] = float(
-            #     sharpby) if sharpby != '-' else None
             # 获取阿尔法
             alpha = self.get_element_text_by_xpath(
                 'li[5]', 'qt_riskstats')
@@ -317,3 +315,20 @@ class FundSpider:
             r_square = self.get_element_text_by_xpath(
                 'li[11]', 'qt_riskstats')
             self.risk_statistics["r_square"] = r_square
+    # 获取持仓
+
+    def get_asset_composition_info(self):
+        # 判断是否含有股票持仓
+        li_elements = self._chrome_driver.find_element_by_id(
+            'qt_stock').find_elements_by_xpath("li")
+        for index in range(4, len(li_elements) - 1, 4):
+            temp_stock_info = dict()  # 一只股票信息
+            stock_base = li_elements[index].text.split('.')
+            temp_stock_info['stock_code'] = stock_base[0]
+            temp_stock_info['stock_market'] = None if len(
+                stock_base) == 1 else stock_base.pop()
+            temp_stock_info['stock_name'] = li_elements[index+1].text
+            # temp_stock_info['stock_value'] = li_elements[index+2].text
+            temp_stock_info['stock_portion'] = li_elements[index +
+                                                           3].text if li_elements[index+3].text != '-' else None
+            self.ten_top_stock_list.append(temp_stock_info)