diff --git a/Euler/README.md b/Euler/README.md index ba34219..6073e4b 100644 --- a/Euler/README.md +++ b/Euler/README.md @@ -1,9 +1,9 @@ -##Euler +## Euler [Euler](https://projecteuler.net/archives) solutions About [Euler](https://projecteuler.net/about) -##Problems +## Problems - [Level1 001-025](level1/) - [Level2 026-050](level2/) - [Level3 051-075](level3/) diff --git a/README.md b/README.md index 8efa6ae..761d27b 100644 --- a/README.md +++ b/README.md @@ -1,73 +1,87 @@ -##Machine Learning +## Machine Learning +## 课程 -##Coursera -####Andrew Ng--Stanford University +#### [机器学习](https://github.com/xxg1413/coursera/tree/master/Machine%20Learning-Andrew%20Ng) -- 吴恩达 -- 斯坦福大学 -- [Machine Learning](https://github.com/xxg1413/coursera/tree/master/Machine%20Learning-Andrew%20Ng) - - -####Hsuan-Tien Lin--Taiwan University -- [Machine Learning Foundations](https://github.com/xxg1413/coursera/tree/master/Machine%20Learning%20Foundations) -- [Machine Learning Techniques](https://github.com/xxg1413/coursera/tree/master/Machine%20Learning%20Techniques) +- 课程进度： 100% +- 作业： + - [Octave] - 使用Octave，默认版本 100% + - [Jupyter Notebook](https://github.com/xxg1413/coursera/tree/master/Machine%20Learning-Andrew%20Ng/Jupyter%20Notebook) - 使用numpy,pandas,scikit-learn,matplotlib,tensorflow等库的Python3版 12% +#### 机器学习 -- 林軒田 -- 台湾大学 +- [机器学习基石](https://github.com/xxg1413/coursera/tree/master/Machine%20Learning%20Foundations) +- [机器学习技法](https://github.com/xxg1413/coursera/tree/master/Machine%20Learning%20Techniques) -##Book Note -- [Building Machine Learning Systems](https://github.com/xxg1413/MachineLearning/tree/master/Building%20Machine%20Learning%20Systems) -- [Learning Data Mining with Python](https://github.com/xxg1413/MachineLearning/tree/master/Learning%20Data%20Mining%20with%20Python) -- [Machine Learning in Action](https://github.com/xxg1413/MachineLearning/tree/master/Machine Learning in Action/) -- [NumPy Beginner's Guide](https://github.com/xxg1413/MachineLearning/tree/master/NumPy Beginner's Guide/) -- [Programming Collective Intelligence](https://github.com/xxg1413/MachineLearning/tree/master/Programming Collective Intelligence/) -- [R for Everyone](https://github.com/xxg1413/MachineLearning/tree/master/R for Everyone/) -- [Web Scraping with Python](https://github.com/xxg1413/MachineLearning/tree/master/Web Scraping with Python/) -- [Learning Spark: Lightning-fast Data Analysis](https://github.com/xxg1413/MachineLearning/tree/master/Learning Spark/) -##Competitions -- [Kaggle](https://github.com/xxg1413/MachineLearning/tree/master/Kaggle/) -- [Euler](https://github.com/xxg1413/MachineLearning/tree/master/Euler/) -##课程 -####吴恩达--斯坦福大学 +#### [CS231n: 面向视觉识别的卷积神经网络--Fei-Fei Li & Andrej Karpathy & Justin Johnson -- 斯坦福大学](https://github.com/xxg1413/coursera/tree/master/CS231n:%20Convolutional%20Neural%20Networks%20for%20Visual%20Recognition) -- [机器学习](https://github.com/xxg1413/coursera/tree/master/Machine%20Learning-Andrew%20Ng) +- 课程进度： 第2集 +- 作业： + - 作业1 + - 作业2 + - 作业3 -####林軒田--台湾大学 -- [机器学习基石](https://github.com/xxg1413/coursera/tree/master/Machine%20Learning%20Foundations) -- [机器学习技法](https://github.com/xxg1413/coursera/tree/master/Machine%20Learning%20Techniques) +#### [机器学习中的神经网络--Geoffrey Hinton-多伦多大学](https://github.com/xxg1413/coursera/tree/master/Neural%20Networks%20for%20Machine%20Learning) +- 课程进度： Lecture2 +- 作业： + - Octave - 使用Octave，默认版本 + - Tensorflow 版本 -##读书 -- [机器学习系统设计](https://github.com/xxg1413/MachineLearning/tree/master/Building Machine Learning Systems/) -- [机器学习实战](https://github.com/xxg1413/MachineLearning/tree/master/Machine Learning in Action/) -- [Python数据分析基础教程:NumPy学习指南](https://github.com/xxg1413/MachineLearning/tree/master/NumPy Beginner's Guide/) +## 读书 +- [机器学习系统设计](https://github.com/xxg1413/MachineLearning/tree/master/Building%20Machine%20Learning%20Systems) + - 进度：100% + - 项目：寻找实际数据练习 + +- [机器学习实战](https://github.com/xxg1413/MachineLearning/tree/master/Machine%20Learning%20in%20Action) + - 进度：第2章 + - 项目： + +- [Python数据分析基础教程:NumPy学习指南](https://github.com/xxg1413/MachineLearning/tree/master/NumPy%20Beginner's%20Guide) + - 进度：100% + - 项目：numpy入门 + - [Python数据挖掘入门与实践](https://github.com/xxg1413/MachineLearning/tree/master/Learning%20Data%20Mining%20with%20Python) -- [集体智慧编程](https://github.com/xxg1413/MachineLearning/tree/master/Programming Collective Intelligence/) -- [R语言-实用数据分析和可视化技术](https://github.com/xxg1413/MachineLearning/tree/master/R for Everyone/) -- [Python网络数据采集](https://github.com/xxg1413/MachineLearning/tree/master/Web Scraping with Python/) - - -##竞赛 + - 进度：Chapter3 + - 项目： + +- [集体智慧编程](https://github.com/xxg1413/MachineLearning/tree/master/Programming%20Collective%20Intelligence/) + - 进度：100% + - 项目：使用库重写所有例子 + +- [R语言-实用数据分析和可视化技术](https://github.com/xxg1413/MachineLearning/tree/master/R%20for%20Everyone/) + - 进度：100% + - 项目： + +- [Python网络数据采集](https://github.com/xxg1413/MachineLearning/tree/master/Web%20Scraping%20with%20Python/) + - 进度：100% + - 项目：爬虫系统 + + + +## 竞赛 - [Kaggle练习题题解](https://github.com/xxg1413/MachineLearning/tree/master/Kaggle/) - [Euler项目题解](https://github.com/xxg1413/MachineLearning/tree/master/Euler/) -##入门 +## 入门 - [numpy](https://github.com/xxg1413/MachineLearning/tree/master/numpy-tutorial/) - [pandas](https://github.com/xxg1413/MachineLearning/tree/master/pandas-tutorial/) - [sklearn](https://github.com/xxg1413/MachineLearning/tree/master/sklearn-tutorial/) - [tensorflow](https://github.com/xxg1413/Tensorflow/tree/master/tutorial) -##Spark +## Spark - [Introduction to Apache Spark](https://github.com/xxg1413/edx/tree/master/Introduction%20to%20Apache%20Spark) - [Distributed Machine Learning with Apache Spark](https://github.com/xxg1413/edx/tree/master/Distributed%20Machine%20Learning%20with%20Apache%20Spark) - [Big Data Analysis with Apache Spark]() -##项目列表 -- Python爬虫:豆瓣书籍 +## 项目列表 diff --git a/Web Scraping with Python/Chapter10/javascript-redirect.py b/Web Scraping with Python/Chapter10/javascript-redirect.py new file mode 100644 index 0000000..6b86770 --- /dev/null +++ b/Web Scraping with Python/Chapter10/javascript-redirect.py @@ -0,0 +1,30 @@ +from selenium import webdriver +import time +from selenium.webdriver.remote.webelement import WebElement +from selenium.common.exceptions import StaleElementReferenceException + + +''' +我们可以用一种智能的方法来检测客户端重定向是否完成,首先从页面开始加载 时就“监视”DOM 中的一个元素, +然后重复调用这个元素直到 Selenium +抛出一个 StaleElementReferenceException 异常; +也就是说,元素不在页面的 DOM 里了,说明这时 网站已经跳转: +''' +def waitForLoad(driver): + elem = driver.find_element_by_tag_name("html") + count = 0 + while True: + count += 1 + if count > 20: + print("Timing out after 10 seconds and returning") + return + time.sleep(.5) + try: + elem == driver.find_element_by_tag_name("html") + except StaleElementReferenceException: + return + +driver = webdriver.PhantomJS(executable_path='') +driver.get("http://pythonscraping.com/pages/javascript/redirectDemo1.html") +waitForLoad(driver) +print(driver.page_source) \ No newline at end of file diff --git a/Web Scraping with Python/Chapter10/selenium-basic.py b/Web Scraping with Python/Chapter10/selenium-basic.py new file mode 100644 index 0000000..325308e --- /dev/null +++ b/Web Scraping with Python/Chapter10/selenium-basic.py @@ -0,0 +1,18 @@ +from selenium import webdriver +import time + + +''' +那些使用了 Ajax 或 DHTML 技术改变 / 加载内容的页面,可能有一些采集手段,但是用 Python 解决这个问题只有两种途径: +直接从 JavaScript 代码里采集内容,或者用 Python 的 第三方库运行 JavaScript,直接采集你在浏览器里看到的页面 + +PhantomJS 无头浏览器 + +把 Selenium 和 PhantomJS 结合在一 起,就可以运行一个非常强大的网络爬虫了, +可以处理 cookie、JavaScrip、header,以及 任何你需要做的事情。 +''' +driver = webdriver.PhantomJS(executable_path='') +driver.get("http://pythonscraping.com/pages/javascript/ajaxDemo.html") +time.sleep(3) +print(driver.find_element_by_id("content").text) +driver.close() \ No newline at end of file diff --git a/Web Scraping with Python/Chapter10/wait-for-load.py b/Web Scraping with Python/Chapter10/wait-for-load.py new file mode 100644 index 0000000..bc69109 --- /dev/null +++ b/Web Scraping with Python/Chapter10/wait-for-load.py @@ -0,0 +1,11 @@ +from selenium.webdriver.common.by import By +from selenium.webdriver.support.ui import WebDriverWait +from selenium.webdriver.support import expected_conditions as EC + +driver = webdriver.PhantomJS(executable_path='') +driver.get("http://pythonscraping.com/pages/javascript/ajaxDemo.html") +try: + element = WebDriverWait(driver, 10).until(EC.presence_of_element_located((By.ID, "loadedButton"))) +finally: + print(driver.find_element_by_id("content").text) + driver.close() \ No newline at end of file diff --git a/Web Scraping with Python/Chapter11/basic-image.py b/Web Scraping with Python/Chapter11/basic-image.py new file mode 100644 index 0000000..bc66c1f --- /dev/null +++ b/Web Scraping with Python/Chapter11/basic-image.py @@ -0,0 +1,6 @@ +from PIL import Image, ImageFilter + +kitten = Image.open("kitten.jpg") +blurryKitten = kitten.filter(ImageFilter.GaussianBlur) +blurryKitten.save("kitten_blurred.jpg") +blurryKitten.show() \ No newline at end of file diff --git a/Web Scraping with Python/Chapter11/captcha.jpg b/Web Scraping with Python/Chapter11/captcha.jpg new file mode 100644 index 0000000..82d9734 Binary files /dev/null and b/Web Scraping with Python/Chapter11/captcha.jpg differ diff --git a/Web Scraping with Python/Chapter11/captcha.txt b/Web Scraping with Python/Chapter11/captcha.txt new file mode 100644 index 0000000..d6d5faa --- /dev/null +++ b/Web Scraping with Python/Chapter11/captcha.txt @@ -0,0 +1,2 @@ +r S MAS + diff --git a/Web Scraping with Python/Chapter11/clean-image.py b/Web Scraping with Python/Chapter11/clean-image.py new file mode 100644 index 0000000..f29a41e --- /dev/null +++ b/Web Scraping with Python/Chapter11/clean-image.py @@ -0,0 +1,24 @@ +''' +利用 Pillow 库,我们可以创建一个 阈值过滤器来去掉渐变的背景色,只把文字留下来,从而让图片更加清晰 +''' + +from PIL import Image +import subprocess + +def cleanFile(filePath, newFilePath): + image = Image.open(filePath) + + #Set a threshold value for the image, and save + image = image.point(lambda x: 0 if x<143 else 255) + image.save(newFilePath) + + + #子进程调用tesseract Tesseract 最大的缺点是对渐变背景色的处理 + subprocess.call(["tesseract", newFilePath, "test"]) + + #Open and read the resulting data file + outputFile = open("test.txt", 'r') + print(outputFile.read()) + outputFile.close() + +cleanFile("text.png", "text_clean.png") \ No newline at end of file diff --git a/Web Scraping with Python/Chapter11/kitten.jpg b/Web Scraping with Python/Chapter11/kitten.jpg new file mode 100644 index 0000000..ea4ee7f Binary files /dev/null and b/Web Scraping with Python/Chapter11/kitten.jpg differ diff --git a/Web Scraping with Python/Chapter11/kitten_blurred.jpg b/Web Scraping with Python/Chapter11/kitten_blurred.jpg new file mode 100644 index 0000000..b40e3bf Binary files /dev/null and b/Web Scraping with Python/Chapter11/kitten_blurred.jpg differ diff --git a/Web Scraping with Python/Chapter11/output.txt b/Web Scraping with Python/Chapter11/output.txt new file mode 100644 index 0000000..51ab060 --- /dev/null +++ b/Web Scraping with Python/Chapter11/output.txt @@ -0,0 +1,3 @@ +This IS some text. wntten In Anal, that will be , +Tesseracl Here are some symbols: IWM . + diff --git a/Web Scraping with Python/Chapter11/page.jpg b/Web Scraping with Python/Chapter11/page.jpg new file mode 100644 index 0000000..66dee1c Binary files /dev/null and b/Web Scraping with Python/Chapter11/page.jpg differ diff --git a/Web Scraping with Python/Chapter11/page.txt b/Web Scraping with Python/Chapter11/page.txt new file mode 100644 index 0000000..fdd6a7f --- /dev/null +++ b/Web Scraping with Python/Chapter11/page.txt @@ -0,0 +1,47 @@ +WEI‘ nrrd Peace +Len Nlkelayevldu Iolfluy + +Readmg shmdd be ax +wlnvame asnossxble Wenfler +an mm m our cram: Llhvary + +— Leo Tmsloy was a Russian rwovelwst +I and moval phflmopher med lur +A ms Ideas 01 nonviolenx reswslance m 5 We range 0, “and” + +M. +known for ms genevosxly to the '°'”"“‘ ‘”' "*°’“‘* W‘"' + +reading dwnmuie,—a|\ +‘L °““"“‘ opnmizedfarreadabIh(y—so +readerscan tho the fnrmat + +ms best knawn nuvckave “War and + +Peace" (1869),w>v(h msmy regarded as an em ‘”“”L"‘5"‘°"‘ W‘ 599 ""19 +vamer man a novel, and "Anna Kavemna" mm) “” '“"" + +Nxswark was admued m hiswne by Doxlayevxky, + +Chxkoxa Turgenev, and Flauben and Water by Tm EaSyREad super + +wgmiz wow and )ame§Joyc vlsxon +onshuqhtlhrmlghlhevadherenzr(owrme He +tamures the essenze af Ne wwlh an R5 mancnges, +hardsmps andmys The maepm poV\raya\ at +numemus thataclerx lrom an xpheves m We wands +u unparalleled reahsm Aime dassm‘ + + + +su+u27o3ro2ar1 + +||I||| H + +cuvmumeu .21mna + + + diff --git a/Web Scraping with Python/Chapter11/read-web-images.py b/Web Scraping with Python/Chapter11/read-web-images.py new file mode 100644 index 0000000..d1616a0 --- /dev/null +++ b/Web Scraping with Python/Chapter11/read-web-images.py @@ -0,0 +1,36 @@ +import time +from urllib.request import urlretrieve +import subprocess +from selenium import webdriver + +#driver = webdriver.PhantomJS(executable_path='/Users/ryan/Documents/pythonscraping/code/headless/phantomjs-1.9.8-macosx/bin/phantomjs') +driver = webdriver.Firefox() +driver.get("http://www.amazon.com/War-Peace-Leo-Nikolayevich-Tolstoy/dp/1427030200") +time.sleep(2) + +driver.find_element_by_id("img-canvas").click() +#The easiest way to get exactly one of every page +imageList = set() + +#Wait for the page to load +time.sleep(10) +print(driver.find_element_by_id("sitbReaderRightPageTurner").get_attribute("style")) +while "pointer" in driver.find_element_by_id("sitbReaderRightPageTurner").get_attribute("style"): + #While we can click on the right arrow, move through the pages + driver.find_element_by_id("sitbReaderRightPageTurner").click() + time.sleep(2) + #Get any new pages that have loaded (multiple pages can load at once) + pages = driver.find_elements_by_xpath("//div[@class='pageImage']/div/img") + for page in pages: + image = page.get_attribute("src") + imageList.add(image) + +driver.quit() + +#Start processing the images we've collected URLs for with Tesseract +for image in sorted(imageList): + urlretrieve(image, "page.jpg") + p = subprocess.Popen(["tesseract", "page.jpg", "page"], stdout=subprocess.PIPE,stderr=subprocess.PIPE) + p.wait() + f = open("page.txt", "r") + print(f.read()) diff --git a/Web Scraping with Python/Chapter11/solve-captcha.py b/Web Scraping with Python/Chapter11/solve-captcha.py new file mode 100644 index 0000000..dc07652 --- /dev/null +++ b/Web Scraping with Python/Chapter11/solve-captcha.py @@ -0,0 +1,63 @@ +#!encoding=utf-8 + +''' +识别验证码 +''' + +from urllib.request import urlretrieve +from urllib.request import urlopen +from bs4 import BeautifulSoup +import subprocess +import requests +from PIL import Image +from PIL import ImageOps + +#清洗图片 +def cleanImage(imagePath): + image = Image.open(imagePath) + image = image.point(lambda x: 0 if x<143 else 255) + borderImage = ImageOps.expand(image,border=20,fill='white') + borderImage.save(imagePath) + + +html = urlopen("http://www.pythonscraping.com/humans-only") +bsObj = BeautifulSoup(html,"html.parser") + +#Gather prepopulated form values +imageLocation = bsObj.find("img", {"title": "Image CAPTCHA"})["src"] +formBuildId = bsObj.find("input", {"name":"form_build_id"})["value"] +captchaSid = bsObj.find("input", {"name":"captcha_sid"})["value"] +captchaToken = bsObj.find("input", {"name":"captcha_token"})["value"] + +#下载验证码图片 +captchaUrl = "http://pythonscraping.com"+imageLocation +urlretrieve(captchaUrl, "captcha.jpg") + +cleanImage("captcha.jpg") + +#调用tessract +p = subprocess.Popen(["tesseract", "captcha.jpg", "captcha"], stdout= + subprocess.PIPE,stderr=subprocess.PIPE) +p.wait() +f = open("captcha.txt", "r") + +#Clean any whitespace characters +captchaResponse = f.read().replace(" ", "").replace("\n", "") +print("Captcha solution attempt: "+captchaResponse) + + +#填充 测试发现识别率并不是很高 +if len(captchaResponse) == 5: + params = {"captcha_token":captchaToken, "captcha_sid":captchaSid, + "form_id":"comment_node_page_form", "form_build_id": formBuildId, + "captcha_response":captchaResponse, "name":"Ryan Mitchell", + "subject": "I come to seek the Grail", + "comment_body[und][0][value]": + "...and I am definitely not a bot"} + r = requests.post("http://www.pythonscraping.com/comment/reply/10", + data=params) + responseObj = BeautifulSoup(r.text) + if responseObj.find("div", {"class":"messages"}) is not None: + print(responseObj.find("div", {"class":"messages"}).get_text()) +else: + print("There was a problem reading the CAPTCHA correctly!") \ No newline at end of file diff --git a/Web Scraping with Python/Chapter11/test.txt b/Web Scraping with Python/Chapter11/test.txt new file mode 100644 index 0000000..51ab060 --- /dev/null +++ b/Web Scraping with Python/Chapter11/test.txt @@ -0,0 +1,3 @@ +This IS some text. wntten In Anal, that will be , +Tesseracl Here are some symbols: IWM . + diff --git a/Web Scraping with Python/Chapter11/text.png b/Web Scraping with Python/Chapter11/text.png new file mode 100644 index 0000000..0a57443 Binary files /dev/null and b/Web Scraping with Python/Chapter11/text.png differ diff --git a/Web Scraping with Python/Chapter11/text_clean.png b/Web Scraping with Python/Chapter11/text_clean.png new file mode 100644 index 0000000..9d3b82c Binary files /dev/null and b/Web Scraping with Python/Chapter11/text_clean.png differ diff --git a/Web Scraping with Python/Chapter12/headers.py b/Web Scraping with Python/Chapter12/headers.py new file mode 100644 index 0000000..2b760da --- /dev/null +++ b/Web Scraping with Python/Chapter12/headers.py @@ -0,0 +1,12 @@ +import requests +from bs4 import BeautifulSoup + +session = requests.Session() +headers = {"User-Agent":"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_9_5)AppleWebKit 537.36 (KHTML, like Gecko) Chrome", + "Accept":"text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8"} + +url = "https://www.whatismybrowser.com/developers/what-http-headers-is-my-browser-sending" +req = session.get(url, headers=headers) + +bsObj = BeautifulSoup(req.text) +print(bsObj.find("table",{"class":"table-striped"}).get_text) \ No newline at end of file diff --git a/Web Scraping with Python/Chapter12/honeypot-detection.py b/Web Scraping with Python/Chapter12/honeypot-detection.py new file mode 100644 index 0000000..01334bf --- /dev/null +++ b/Web Scraping with Python/Chapter12/honeypot-detection.py @@ -0,0 +1,14 @@ +from selenium import webdriver +from selenium.webdriver.remote.webelement import WebElement + +driver = webdriver.PhantomJS(executable_path='/Users/xxg/phantomjs-2.1.1-macosx/bin/phantomjs') +driver.get("http://pythonscraping.com/pages/itsatrap.html") +links = driver.find_elements_by_tag_name("a") +for link in links: + if not link.is_displayed(): + print("The link "+link.get_attribute("href")+" is a trap") + +fields = driver.find_elements_by_tag_name("input") +for field in fields: + if not field.is_displayed(): + print("Do not change value of "+field.get_attribute("name")) \ No newline at end of file diff --git a/Web Scraping with Python/Chapter12/selenium-cookies.py b/Web Scraping with Python/Chapter12/selenium-cookies.py new file mode 100644 index 0000000..5b53f2b --- /dev/null +++ b/Web Scraping with Python/Chapter12/selenium-cookies.py @@ -0,0 +1,18 @@ +from selenium import webdriver + +driver = webdriver.PhantomJS(executable_path='/Users/xxg/phantomjs-2.1.1-macosx/bin/phantomjs') +driver.get("http://pythonscraping.com") +driver.implicitly_wait(1) +print(driver.get_cookies()) + +savedCookies = driver.get_cookies() + +driver2 = webdriver.PhantomJS(executable_path='/Users/xxg/phantomjs-2.1.1-macosx/bin/phantomjs') +driver2.get("http://pythonscraping.com") +driver2.delete_all_cookies() +for cookie in savedCookies: + driver2.add_cookie(cookie) + +driver2.get("http://pythonscraping.com") +driver.implicitly_wait(1) +print(driver2.get_cookies()) \ No newline at end of file diff --git a/Web Scraping with Python/Chapter13/combined-test.py b/Web Scraping with Python/Chapter13/combined-test.py new file mode 100644 index 0000000..0d0e107 --- /dev/null +++ b/Web Scraping with Python/Chapter13/combined-test.py @@ -0,0 +1,27 @@ +from selenium import webdriver +from selenium.webdriver.remote.webelement import WebElement +from selenium.webdriver import ActionChains +import unittest + +class TestAddition(unittest.TestCase): + driver = None + def setUp(self): + global driver + driver = webdriver.PhantomJS(executable_path='/Users/xxg/phantomjs-2.1.1-macosx/bin/phantomjs') + url = 'http://pythonscraping.com/pages/javascript/draggableDemo.html' + driver.get(url) + + def tearDown(self): + print("Tearing down the test") + + def test_drag(self): + global driver + element = driver.find_element_by_id("draggable") + target = driver.find_element_by_id("div2") + actions = ActionChains(driver) + actions.drag_and_drop(element, target).perform() + + self.assertEqual("You are definitely not a bot!", driver.find_element_by_id("message").text) + +if __name__ == '__main__': + unittest.main() \ No newline at end of file diff --git a/Web Scraping with Python/Chapter13/drag-and-drop.py b/Web Scraping with Python/Chapter13/drag-and-drop.py new file mode 100644 index 0000000..9376814 --- /dev/null +++ b/Web Scraping with Python/Chapter13/drag-and-drop.py @@ -0,0 +1,15 @@ +from selenium import webdriver +from selenium.webdriver.remote.webelement import WebElement +from selenium.webdriver import ActionChains + +driver = webdriver.PhantomJS(executable_path='phantomjs/bin/phantomjs') +driver.get('http://pythonscraping.com/pages/javascript/draggableDemo.html') + +print(driver.find_element_by_id("message").text) + +element = driver.find_element_by_id("draggable") +target = driver.find_element_by_id("div2") +actions = ActionChains(driver) +actions.drag_and_drop(element, target).perform() + +print(driver.find_element_by_id("message").text) \ No newline at end of file diff --git a/Web Scraping with Python/Chapter13/github.png b/Web Scraping with Python/Chapter13/github.png new file mode 100644 index 0000000..8a2c973 Binary files /dev/null and b/Web Scraping with Python/Chapter13/github.png differ diff --git a/Web Scraping with Python/Chapter13/interactive-test.py b/Web Scraping with Python/Chapter13/interactive-test.py new file mode 100644 index 0000000..892c8bc --- /dev/null +++ b/Web Scraping with Python/Chapter13/interactive-test.py @@ -0,0 +1,27 @@ +from selenium import webdriver +from selenium.webdriver.remote.webelement import WebElement +from selenium.webdriver.common.keys import Keys +from selenium.webdriver import ActionChains + + +driver = webdriver.PhantomJS(executable_path='phantomjs/bin/phantomjs') +driver.get("http://pythonscraping.com/pages/files/form.html") + +firstnameField = driver.find_element_by_name("firstname") +lastnameField = driver.find_element_by_name("lastname") +submitButton = driver.find_element_by_id("submit") + +### METHOD 1 ### +firstnameField.send_keys("Ryan") +lastnameField.send_keys("Mitchell") +submitButton.click() +################ + +### METHOD 2 ### +actions = ActionChains(driver).click(firstnameField).send_keys("Ryan").click(lastnameField).send_keys("Mitchell").send_keys(Keys.RETURN) +actions.perform() +################ + +print(driver.find_element_by_tag_name("body").text) + +driver.close() \ No newline at end of file diff --git a/Web Scraping with Python/Chapter13/pythonscraping.png b/Web Scraping with Python/Chapter13/pythonscraping.png new file mode 100644 index 0000000..ee8e64c Binary files /dev/null and b/Web Scraping with Python/Chapter13/pythonscraping.png differ diff --git a/Web Scraping with Python/Chapter13/take-screenshot.py b/Web Scraping with Python/Chapter13/take-screenshot.py new file mode 100644 index 0000000..fb2ff10 --- /dev/null +++ b/Web Scraping with Python/Chapter13/take-screenshot.py @@ -0,0 +1,9 @@ +from selenium import webdriver +from selenium.webdriver.remote.webelement import WebElement +from selenium.webdriver import ActionChains + + +driver = webdriver.PhantomJS(executable_path='/Users/xxg/phantomjs-2.1.1-macosx/bin/phantomjs') +driver.implicitly_wait(5) +driver.get('http://github.com/xxg1413') +driver.get_screenshot_as_file('github.png') \ No newline at end of file diff --git a/Web Scraping with Python/Chapter13/wiki-selenium-test.py b/Web Scraping with Python/Chapter13/wiki-selenium-test.py new file mode 100644 index 0000000..6a4a20b --- /dev/null +++ b/Web Scraping with Python/Chapter13/wiki-selenium-test.py @@ -0,0 +1,8 @@ +from selenium import webdriver + + +driver = webdriver.PhantomJS(executable_path='/Users/xxg/phantomjs-2.1.1-macosx/bin/phantomjs') +driver.get("http://en.wikipedia.org/wiki/Monty_Python") +assert "Monty Python" in driver.title +print("Monty Python was not in the title") +driver.close() \ No newline at end of file diff --git a/Web Scraping with Python/Chapter13/wiki-unit-test.py b/Web Scraping with Python/Chapter13/wiki-unit-test.py new file mode 100644 index 0000000..2a1cc32 --- /dev/null +++ b/Web Scraping with Python/Chapter13/wiki-unit-test.py @@ -0,0 +1,53 @@ +from urllib.request import urlopen +from urllib.parse import unquote +import random +import re +from bs4 import BeautifulSoup +import unittest + +class TestWikipedia(unittest.TestCase): + + bsObj = None + url = None + + + def test_PageProperties(self): + global bsObj + global url + + url = "http://en.wikipedia.org/wiki/Monty_Python" + #Test the first 100 pages we encounter + for i in range(1, 100): + bsObj = BeautifulSoup(urlopen(url), "html.parser") + titles = self.titleMatchesURL() + self.assertEquals(titles[0], titles[1]) + self.assertTrue(self.contentExists()) + url = self.getNextLink() + print("Done!") + + #测试标题 + def titleMatchesURL(self): + global bsObj + global url + pageTitle = bsObj.find("h1").get_text() + urlTitle = url[(url.index("/wiki/")+6):] + urlTitle = urlTitle.replace("_", " ") + urlTitle = unquote(urlTitle) + return [pageTitle.lower(), urlTitle.lower()] + + def contentExists(self): + global bsObj + content = bsObj.find("div",{"id":"mw-content-text"}) + if content is not None: + return True + return False + + def getNextLink(self): + global bsObj + links = bsObj.find("div", {"id":"bodyContent"}).findAll("a", href=re.compile("^(/wiki/)((?!:).)*$")) + link = links[random.randint(0, len(links)-1)].attrs['href'] + print("Next link is: "+link) + return "http://en.wikipedia.org"+link + +if __name__ == '__main__': + unittest.main() diff --git a/Web Scraping with Python/Chapter14/selenium-socks.py b/Web Scraping with Python/Chapter14/selenium-socks.py new file mode 100644 index 0000000..a921567 --- /dev/null +++ b/Web Scraping with Python/Chapter14/selenium-socks.py @@ -0,0 +1,7 @@ +from selenium import webdriver +service_args = [ '--proxy=localhost:9150', '--proxy-type=socks5', ] +driver = webdriver.PhantomJS(executable_path='', service_args=service_args) + +driver.get("http://icanhazip.com") +print(driver.page_source) +driver.close() \ No newline at end of file diff --git a/Web Scraping with Python/Chapter14/socks.py b/Web Scraping with Python/Chapter14/socks.py new file mode 100644 index 0000000..52e0665 --- /dev/null +++ b/Web Scraping with Python/Chapter14/socks.py @@ -0,0 +1,8 @@ +import socks #pysocks +import socket +from urllib.request import urlopen + +socks.set_default_proxy(socks.SOCKS5, "127.0.0.1", 9150) +socket.socket = socks.socksocket +print(urlopen('http://icanhazip.com').read()) + diff --git a/Web Scraping with Python/Chapter7/clean-2-grams.py b/Web Scraping with Python/Chapter7/clean-2-grams.py new file mode 100644 index 0000000..1efd0f0 --- /dev/null +++ b/Web Scraping with Python/Chapter7/clean-2-grams.py @@ -0,0 +1,55 @@ +#!/usr/bin/env python3 +#!encoding=utf-8 + +from urllib.request import urlopen +from bs4 import BeautifulSoup +import re +import string +from collections import OrderedDict + + +def cleanInput(input): + ''' + 移除转义字符，过滤Unicode字符 + ''' + input = re.sub('\n+', " ", input) #替换换行符 + input = re.sub('\[[0-9]*\]', "", input)#除去数字 + input = re.sub(' +', " ", input) #除去多余的空格 + input = bytes(input, "UTF-8") #更改编码 + input = input.decode("ascii", "ignore") + + cleanInput = [] + input = input.split(' ') + + for item in input: + item = item.strip(string.punctuation) + if len(item) > 1 or (item.lower() == 'a' or item.lower() == 'i'): + cleanInput.append(item) + return cleanInput + + +def getNgrams(input, n): + input = cleanInput(input) + output = dict() + for i in range(len(input)-n+1): + newNGram = " ".join(input[i:i+n]) + if newNGram in output: + output[newNGram] += 1 + else: + output[newNGram] = 1 + return output + + + +html = urlopen("http://en.wikipedia.org/wiki/Python_(programming_language)") +bsObj = BeautifulSoup(html,"html.parser") +content = bsObj.find("div", {"id":"mw-content-text"}).get_text() + + +ngrams = getNgrams(content, 2) + +#使用OrderedDict 去重 +ngrams = OrderedDict(sorted(ngrams.items(), key=lambda t: t[1], reverse=True)) + + +print(ngrams) \ No newline at end of file diff --git a/Web Scraping with Python/Chapter8/6-degrees-finder.py b/Web Scraping with Python/Chapter8/6-degrees-finder.py new file mode 100644 index 0000000..7fc4263 --- /dev/null +++ b/Web Scraping with Python/Chapter8/6-degrees-finder.py @@ -0,0 +1,48 @@ +from urllib.request import urlopen +from bs4 import BeautifulSoup +import pymysql + + +conn = pymysql.connect(host='127.0.0.1', port=3306, user='root', passwd='root', db='mysql', charset='utf8') +cur = conn.cursor() +cur.execute("USE wikipedia") + +def getUrl(pageId): + cur.execute("SELECT url FROM pages WHERE id = %s", (int(pageId))) + if cur.rowcount == 0: + return None + return cur.fetchone()[0] + +def getLinks(fromPageId): + cur.execute("SELECT toPageId FROM links WHERE fromPageId = %s", (int(fromPageId))) + if cur.rowcount == 0: + return None + return [x[0] for x in cur.fetchall()] + +def searchBreadth(targetPageId, currentPageId, depth, nodes): + if nodes is None or len(nodes) == 0: + return None + if depth <= 0: + for node in nodes: + if node == targetPageId: + return [node] + return None + #depth is greater than 0 -- go deeper! + for node in nodes: + found = searchBreadth(targetPageId, node, depth-1, getLinks(node)) + if found is not None: + return found.append(currentPageId) + return None + +nodes = getLinks(1) +targetPageId = 123428 +for i in range(0,4): + found = searchBreadth(targetPageId, 1, i, nodes) + if found is not None: + print(found) + for node in found: + print(getUrl(node)) + break + else: + print("No path found") + diff --git a/Web Scraping with Python/Chapter8/count-2-grams.py b/Web Scraping with Python/Chapter8/count-2-grams.py new file mode 100644 index 0000000..e7b96a1 --- /dev/null +++ b/Web Scraping with Python/Chapter8/count-2-grams.py @@ -0,0 +1,39 @@ +#!encoding=utf-8 + +from urllib.request import urlopen +from bs4 import BeautifulSoup +import re +import string +import operator + + +def cleanInput(input): + input = re.sub('\n+', " ", input).lower() + input = re.sub('\[[0-9]*\]', "", input) + input = re.sub(' +', " ", input) + input = bytes(input, "UTF-8") + input = input.decode("ascii", "ignore") + cleanInput = [] + input = input.split(' ') + for item in input: + item = item.strip(string.punctuation) + if len(item) > 1 or (item.lower() == 'a' or item.lower() == 'i'): + cleanInput.append(item) + return cleanInput + +def getNgrams(input, n): + input = cleanInput(input) + output = {} + for i in range(len(input)-n+1): + ngramTemp = " ".join(input[i:i+n]) + if ngramTemp not in output: + output[ngramTemp] = 0 + output[ngramTemp] += 1 + return output + +content = str(urlopen("http://pythonscraping.com/files/inaugurationSpeech.txt").read(),'utf-8') +ngrams = getNgrams(content, 2) + +#计数排序 +sortedNGrams = sorted(ngrams.items(), key = operator.itemgetter(1), reverse=True) +print(sortedNGrams) \ No newline at end of file diff --git a/Web Scraping with Python/Chapter8/count-uncommon-2-grams.py b/Web Scraping with Python/Chapter8/count-uncommon-2-grams.py new file mode 100644 index 0000000..5c1e608 --- /dev/null +++ b/Web Scraping with Python/Chapter8/count-uncommon-2-grams.py @@ -0,0 +1,60 @@ +from urllib.request import urlopen +from bs4 import BeautifulSoup +import re +import string +import operator + +#定义常见词 +def isCommon(ngram): + commonWords = ["the", "be", "and", "of", "a", "in", "to", "have", "it", "i", "that", "for", "you", "he", "with", "on", "do", "say", "this", "they", "is", "an", "at", "but","we", "his", "from", "that", "not", "by", "she", "or", "as", "what", "go", "their","can", "who", "get", "if", "would", "her", "all", "my", "make", "about", "know", "will","as", "up", "one", "time", "has", "been", "there", "year", "so", "think", "when", "which", "them", "some", "me", "people", "take", "out", "into", "just", "see", "him", "your", "come", "could", "now", "than", "like", "other", "how", "then", "its", "our", "two", "more", "these", "want", "way", "look", "first", "also", "new", "because", "day", "more", "use", "no", "man", "find", "here", "thing", "give", "many", "well"] + for word in ngram: + if word in commonWords: + return True + return False + +def cleanText(input): + input = re.sub('\n+', " ", input).lower() + input = re.sub('\[[0-9]*\]', "", input) + input = re.sub(' +', " ", input) + input = re.sub("u\.s\.", "us", input) + input = bytes(input, "UTF-8") + input = input.decode("ascii", "ignore") + return input + + +def cleanInput(input): + input = cleanText(input) + cleanInput = [] + input = input.split(' ') + for item in input: + item = item.strip(string.punctuation) + if len(item) > 1 or (item.lower() == 'a' or item.lower() == 'i'): + cleanInput.append(item) + return cleanInput + + +def getNgrams(input, n): + input = cleanInput(input) + output = {} + for i in range(len(input)-n+1): + ngramTemp = " ".join(input[i:i+n]) + if ngramTemp not in output: + output[ngramTemp] = 0 + output[ngramTemp] += 1 + return output + +def getFirstSentenceContaining(ngram, content): + + sentences = content.split(".") + for sentence in sentences: + if ngram in sentence: + return sentence + return "" + +content = str(urlopen("http://pythonscraping.com/files/space.txt").read(), 'utf-8') +ngrams = getNgrams(content, 2) +sortedNGrams = sorted(ngrams.items(), key = operator.itemgetter(1), reverse = True) +print(sortedNGrams) + + + diff --git a/Web Scraping with Python/Chapter8/markov-generator.py b/Web Scraping with Python/Chapter8/markov-generator.py new file mode 100644 index 0000000..82261ee --- /dev/null +++ b/Web Scraping with Python/Chapter8/markov-generator.py @@ -0,0 +1,71 @@ +#!encoding=utf-8 + +from urllib.request import urlopen +from random import randint + +def wordListSum(wordList): + sum = 0 + for word, value in wordList.items(): + sum += value + return sum + +def retrieveRandomWord(wordList): + + randIndex = randint(1, wordListSum(wordList)) + for word, value in wordList.items(): + randIndex -= value + if randIndex <= 0: + return word + + +def buildWordDict(text): + ''' + 数据清洗 + ''' + #替换换行符，除去引号 + text = text.replace("\n", " ") + text = text.replace("\"", "") + + + + punctuation = [',','.',';',':'] + + for symbol in punctuation: + text = text.replace(symbol, " "+symbol+" ") #在标点符号前后加空格 + + + words = text.split(" ") + #Filter out empty words + words = [word for word in words if word != ""] #除去空单词 + + wordDict = {} + for i in range(1, len(words)): + if words[i-1] not in wordDict: + #Create a new dictionary for this word + wordDict[words[i-1]] = {} + if words[i] not in wordDict[words[i-1]]: + wordDict[words[i-1]][words[i]] = 0 + wordDict[words[i-1]][words[i]] += 1 + + return wordDict + +text = str(urlopen("http://pythonscraping.com/files/inaugurationSpeech.txt").read(), 'utf-8') +wordDict = buildWordDict(text) + +#生成长度为100的马尔可夫链 + +#单词数组的格式： {word_a : {word_b : 2, word_c : 1, word_d : 1}} +# 表示：word_a 出现4次 有两次是跟在word_b,一次word_c，一次word_d +#那么“word_a”可能就有带 50% 概率的箭头指向 “word_b”(四次中的两次),带 25% 概率的箭头指向“word_c”,还有带 25% 概率的箭头指向“word_d” + +#最后一个单词的考虑： 因为可能一个单词的后面没有单词 + +length = 100 +chain = "" +currentWord = "I" +for i in range(0, length): + chain += currentWord+" " + #print(wordDict[currentWord]) + currentWord = retrieveRandomWord(wordDict[currentWord]) + +print(chain) diff --git a/Web Scraping with Python/Chapter8/nltk-analysis.py b/Web Scraping with Python/Chapter8/nltk-analysis.py new file mode 100644 index 0000000..2f9f77c --- /dev/null +++ b/Web Scraping with Python/Chapter8/nltk-analysis.py @@ -0,0 +1,10 @@ +from nltk import word_tokenize, sent_tokenize, pos_tag +sentences = sent_tokenize("Google is one of the best companies in the world. I constantly google myself to see what I'm up to.") +nouns = ['NN', 'NNS', 'NNP', 'NNPS'] + +for sentence in sentences: + if "google" in sentence.lower(): + taggedWords = pos_tag(word_tokenize(sentence)) + for word in taggedWords: + if word[0].lower() == "google" and word[1] in nouns: + print(sentence) diff --git a/Web Scraping with Python/Chapter8/nltk-search.py b/Web Scraping with Python/Chapter8/nltk-search.py new file mode 100644 index 0000000..6697b23 --- /dev/null +++ b/Web Scraping with Python/Chapter8/nltk-search.py @@ -0,0 +1,6 @@ +from nltk.book import * +from nltk import ngrams +fourgrams = ngrams(text6, 4) +for fourgram in fourgrams: + if fourgram[0] == "coconut": + print(fourgram) \ No newline at end of file diff --git a/Web Scraping with Python/Chapter8/nltk-tokenize.py b/Web Scraping with Python/Chapter8/nltk-tokenize.py new file mode 100644 index 0000000..adb3448 --- /dev/null +++ b/Web Scraping with Python/Chapter8/nltk-tokenize.py @@ -0,0 +1,7 @@ +#!encoding=utf-8 + +from nltk import word_tokenize +from nltk import Text + +tokens = word_tokenize("Here is some not very interesting text") +text = Text(tokens) \ No newline at end of file diff --git a/Web Scraping with Python/Chapter9/basic-auth.py b/Web Scraping with Python/Chapter9/basic-auth.py new file mode 100644 index 0000000..4b839f4 --- /dev/null +++ b/Web Scraping with Python/Chapter9/basic-auth.py @@ -0,0 +1,7 @@ +import requests +from requests.auth import AuthBase +from requests.auth import HTTPBasicAuth + +auth = HTTPBasicAuth('ryan', 'password') +r = requests.post(url="http://pythonscraping.com/pages/auth/login.php", auth=auth) +print(r.text) \ No newline at end of file diff --git a/Web Scraping with Python/Chapter9/cookies.py b/Web Scraping with Python/Chapter9/cookies.py new file mode 100644 index 0000000..6e66f04 --- /dev/null +++ b/Web Scraping with Python/Chapter9/cookies.py @@ -0,0 +1,10 @@ +import requests + +params = {'username': 'Ryan', 'password': 'password'} +r = requests.post("http://pythonscraping.com/pages/cookies/welcome.php", params) +print("Cookie is set to:") +print(r.cookies.get_dict()) +print("-----------") +print("Going to profile page...") +r = requests.get("http://pythonscraping.com/pages/cookies/profile.php", cookies=r.cookies) +print(r.text) \ No newline at end of file diff --git a/Web Scraping with Python/Chapter9/file-submission.py b/Web Scraping with Python/Chapter9/file-submission.py new file mode 100644 index 0000000..9a7e413 --- /dev/null +++ b/Web Scraping with Python/Chapter9/file-submission.py @@ -0,0 +1,5 @@ +import requests + +files = {'uploadFile': open('../files/Python-logo.png', 'rb')} +r = requests.post("http://pythonscraping.com/pages/processing2.php", files=files) +print(r.text) \ No newline at end of file diff --git a/Web Scraping with Python/Chapter9/file-submission2.py b/Web Scraping with Python/Chapter9/file-submission2.py new file mode 100644 index 0000000..de3b2df --- /dev/null +++ b/Web Scraping with Python/Chapter9/file-submission2.py @@ -0,0 +1,6 @@ +import requests + +files = {'uploadFile': open('../files/Python-logo.png', 'rb')} +r = requests.post("http://pythonscraping.com/pages/processing2.php", + files=files) +print(r.text) \ No newline at end of file diff --git a/Web Scraping with Python/Chapter9/session-cookies.py b/Web Scraping with Python/Chapter9/session-cookies.py new file mode 100644 index 0000000..caa82fa --- /dev/null +++ b/Web Scraping with Python/Chapter9/session-cookies.py @@ -0,0 +1,12 @@ +import requests + +session = requests.Session() + +params = {'username': 'username', 'password': 'password'} +s = session.post("http://pythonscraping.com/pages/cookies/welcome.php", params) +print("Cookie is set to:") +print(s.cookies.get_dict()) +print("-----------") +print("Going to profile page...") +s = session.get("http://pythonscraping.com/pages/cookies/profile.php") +print(s.text) \ No newline at end of file diff --git a/Web Scraping with Python/Chapter9/simple-form.py b/Web Scraping with Python/Chapter9/simple-form.py new file mode 100644 index 0000000..dc7e141 --- /dev/null +++ b/Web Scraping with Python/Chapter9/simple-form.py @@ -0,0 +1,5 @@ +import requests + +params = {'firstname': 'Ryan', 'lastname': 'Mitchell'} +r = requests.post("http://pythonscraping.com/files/processing.php", data=params) +print(r.text) \ No newline at end of file diff --git a/Web Scraping with Python/README.md b/Web Scraping with Python/README.md index 310ebaf..eab6d70 100644 --- a/Web Scraping with Python/README.md +++ b/Web Scraping with Python/README.md @@ -1,9 +1,6 @@ ##Web Scraping with Python - - - #Python网络数据采集 - [第1章 初见网络爬虫](Chapter1/) - [第2章 复杂HTML解析]((Chapter2/)) @@ -11,16 +8,25 @@ - [第4章 使用API](Chapter4/) - [第5章 存储数据](Chapter5/) - [第6章 读取文档](Chapter6/) -- 第7章 数据清洗 -- 第8章 自然语言处理 -- 第9章 穿越网页表单与登录窗口采集 -- 第10章 采集JavaScript -- 第11章 图像识别与文字处理 -- 第12章 避开采集陷阱 -- 第13章 用爬虫测试网站 -- 第14章 远程采集 - +- [第7章 数据清洗](Chapter7/) +- [第8章 自然语言处理](Chapter8/) +- [第9章 穿越网页表单与登录窗口采集](Chapter9/) +- [第10章 采集JavaScript](Chapter10/) +- [第11章 图像识别与文字处理](Chapter11/) +- [第12章 避开采集陷阱](Chapter12/) +- [第13章 用爬虫测试网站](Chapter13/) +- [第14章 远程采集](Chapter14/) ##注意 -此书代码使用的是Python3 \ No newline at end of file +此书代码使用的是Python3 + +##实践 +- [1.获得wiki编辑的IP归属地数据](Chapter4/get_wiki_ip.py) +- [2.数据清洗:N-grams](Chapter7/2-clean2grams.py) +- [3.马尔可夫生成器](Chapter8/markov-generator.py) +- [4.识别验证码](Chapter11/solve-captcha.py) +- [5.网站截图](Chapter13/take-screenshot.py) + + +##项目 TODO diff --git a/Web Scraping with Python/code/python-scraping b/Web Scraping with Python/code/python-scraping new file mode 160000 index 0000000..6c3cdcc --- /dev/null +++ b/Web Scraping with Python/code/python-scraping @@ -0,0 +1 @@ +Subproject commit 6c3cdccc7cc1af01abce279273ad8cfd90f7c268