Scraping: https://www.nytimes.com/

Let’s try to scrape the frontpage of the NYT. We’re looking for

Headlines
Bylines
Article links

Getting started

We’ll start by importing the necessary libraries.

import requests
from bs4 import BeautifulSoup

And then move into downloading the page and importing it into BeautifulSoup.

response = requests.get('http://www.nytimes.com')
doc = BeautifulSoup(response.text, 'html.parser')

A lot of people call the analyzed page variable soup but for once in my life I actually go against the popular thing - I like to call it doc, since it helps me remember that it’s the entire document.

ATTEMPT ONE: Grabbing the tags directly

Let’s jump right into trying to grab the link.

Oh, look it’s an…. a tag. No special class or anything. What if we try to get all of the a tags on the page?

links = doc.find_all('a')

for link in links[:10]:
    print("This is a link")
    print(link.text)

This is a link
LEARN MORE »
This is a link
Skip to content
This is a link
Skip to navigation
This is a link
中文 (Chinese)
This is a link
Español
This is a link





This is a link
Today’s Paper
This is a link
Video
This is a link
World
This is a link
U.S.

Okay, that’s terrible. Do you know how many a tags are going to be on that page? Many many many. Many very useless ones.

Talking to parents

When you can’t uniquely identify something, sometimes you need to go up the tree to find its parent, the elements that are above it. We’ll be looking for an element that covers the entire story, then we’ll pick the link out of it.

Great, it looks like this:

<article class="story theme-summary lede" id="topnews-100000004994965" data- story-id="100000004994965" data-rank="0" data-collection-renderstyle="LedeSum">

I’m going to go out on a limb and say we should look for an article tag, but what about the class? story theme-summary lede gives us three options:

story
theme-summary
lede

story sounds promising, yeah?

story = doc.find_all('article', { 'class': 'story' })

for story in story[:10]:
    print("This is a story")
    print(story.text)

This is a story

Revised Trump Travel Ban Suffers Another Legal Setback
By ADAM LIPTAK 2:09 PM ET
A ruling by a federal appeals court was the latest in a string of decisions rejecting President Trump’s efforts to limit travel from several predominantly Muslim countries.The administration has already sought a Supreme Court review of a similar decision issued last month.

 Comments


This is a story

Putin Opponent Is Arrested Amid Protests in Russia
By NEIL MacFARQUHAR and ANDREW HIGGINS 1:26 PM ET
The police detained Aleksei A. Navalny, a Kremlin critic and the architect of a wave of antigovernment protests across Russia.

 Comments


This is a story

Why Cyberwar on ISIS Has Fallen Short of U.S. Hopes
By DAVID E. SANGER and ERIC SCHMITT 
The effectiveness of cyberweapons hit its limits against an enemy that exploits the internet to recruit, spread propaganda and use encrypted communications, all of which can be quickly reconstituted.

 Comments


This is a story

Trump Holdings Are Unconstitutional, Pioneering Suit Says
By SHARON LaFRANIERE 12:31 PM ET
The complaint, filed in federal court by Maryland and the District of Columbia, alleges that the president’s failure to shed his businesses has undermined public trust and violated constitutional bans on self-dealing.

This is a story

Opioid Addicts Find an Ally in Blue
By AL BAKER 
Police leaders are assigning themselves a big role in reversing a complex crisis, and not through mass arrests.

 Comments


This is a story

Addiction Drug Lacks Results, but It Has Powerful Friends 

This is a story

Role of Trump’s Lawyer Blurs Public and Private Lines
By REBECCA R. RUIZ and SHARON LaFRANIERE 




        Marc E. Kasowitz, the president’s private lawyer, has counseled administration staff and exerted broader influence over legal strategy.    

 Comments


This is a story

Mediator 
Trump Era, Unlike Watergate Era, Has Rival Sets of Facts
By JIM RUTENBERG 
Different versions of the Trump-Russia scandal are fighting for supremacy, leaving news veterans wondering, “What if Watergate happened now?”

 Comments


This is a story

Fired U.S. Attorney Says Trump Tried to Build Relationship With Him 

This is a story








Credit
	            Inez and Vinoodh for The New York Times. Stylist: Alex White. 	        


The Long, Lonely Road of Chelsea Manning
Her disclosure of classified documents in 2010 ushered in the age of leaks. Now, freed from prison, she talks about why she did it — and the isolation that followed.
By MATTHEW SHAER 



Comments

Seems to work well enough! Now that we have a parent, we can use that parent to grab the elements inside of the story. We’ll use .find and .find_all to get everything we need.

STEP ONE: Get the story
STEP TWO: Get the headline
STEP THREE: Get the byline
STEP FOUR: Get the link

If we examine the page, it looks like headlines might be h2 tags that have a story-heaing class.

story = doc.find_all('article', { 'class': 'story' })

for story in story[:10]:
    print("This is a story")
    headline = story.find('h2', { 'class': 'story-heading' })
    print(headline.text)

This is a story
Revised Trump Travel Ban Suffers Another Legal Setback
This is a story
Putin Opponent Is Arrested Amid Protests in Russia
This is a story
Why Cyberwar on ISIS Has Fallen Short of U.S. Hopes
This is a story
Trump Holdings Are Unconstitutional, Pioneering Suit Says
This is a story
Opioid Addicts Find an Ally in Blue
This is a story
Addiction Drug Lacks Results, but It Has Powerful Friends 
This is a story
Role of Trump’s Lawyer Blurs Public and Private Lines
This is a story
Trump Era, Unlike Watergate Era, Has Rival Sets of Facts
This is a story
Fired U.S. Attorney Says Trump Tried to Build Relationship With Him 
This is a story



---------------------------------------------------------------------------

AttributeError                            Traceback (most recent call last)

<ipython-input-6-c39c462cfba2> in <module>()
      4     print("This is a story")
      5     headline = story.find('h2', { 'class': 'story-heading' })
----> 6     print(headline.text)


AttributeError: 'NoneType' object has no attribute 'text'

An error strikes!

But we get an error!

---------------------------------------------------------------------------
AttributeError                            Traceback (most recent call last)
<ipython-input-57-9218ec61124f> in <module>()
      4     print("This is a story")
      5     headline = story.find('h2', { 'class': 'story-heading' })
----> 6     print(headline.text)

AttributeError: 'NoneType' object has no attribute 'text'

Hm, a story missing a headline? Let’s look at it a little closer. We could do this in a classy way, but let’s just brute force it by print out every article just before the error line.

story = doc.find_all('article', { 'class': 'story' })

for story in story[:10]:
    print("This is a story")
    headline = story.find('h2', { 'class': 'story-heading' })
    print(story)
    print(headline.text)

This is a story
<article class="story theme-summary lede" data-collection-renderstyle="LedeSum" data-rank="0" data-story-id="100000005155100" id="topnews-100000005155100">
<h2 class="story-heading"><a href="https://www.nytimes.com/2017/06/12/us/politics/trump-travel-ban-court-of-appeals.html">Revised Trump Travel Ban Suffers Another Legal Setback</a></h2>
<p class="byline">By ADAM LIPTAK <time class="timestamp" data-eastern-timestamp="2:09 PM" data-utc-timestamp="1497290947" datetime="2017-06-12">2:09 PM ET</time></p>
<p class="summary"><ul><li>A ruling by a federal appeals court was the latest in a string of decisions rejecting President Trump’s efforts to limit travel from several predominantly Muslim countries.</li><li>The administration has already sought a Supreme Court review of a similar decision issued last month.</li></ul></p>
<p class="theme-comments">
<a class="comments-link" href="https://www.nytimes.com/2017/06/12/us/politics/trump-travel-ban-court-of-appeals.html?hp&amp;target=comments#commentsContainer"><i class="icon sprite-icon comments-icon"></i><span class="comment-count"> Comments</span></a>
</p>
</article>
Revised Trump Travel Ban Suffers Another Legal Setback
This is a story
<article class="story theme-summary" data-collection-renderstyle="HpSum" data-end-time="" data-rank="0" data-start-time="" data-story-id="100000005159260" id="topnews-100000005159260">
<h2 class="story-heading"><a href="https://www.nytimes.com/2017/06/12/world/europe/russia-aleksei-navalny-kremlin-protests.html">Putin Opponent Is Arrested Amid Protests in Russia</a></h2>
<p class="byline">By NEIL MacFARQUHAR and ANDREW HIGGINS <time class="timestamp" data-eastern-timestamp="1:26 PM" data-utc-timestamp="1497288362" datetime="2017-06-12">1:26 PM ET</time></p>
<p class="summary">The police detained Aleksei A. Navalny, a Kremlin critic and the architect of a wave of antigovernment protests across Russia.</p>
<p class="theme-comments">
<a class="comments-link" href="https://www.nytimes.com/2017/06/12/world/europe/russia-aleksei-navalny-kremlin-protests.html?hp&amp;target=comments#commentsContainer"><i class="icon sprite-icon comments-icon"></i><span class="comment-count"> Comments</span></a>
</p>
</article>
Putin Opponent Is Arrested Amid Protests in Russia
This is a story
<article class="story theme-summary" data-collection-renderstyle="HpSum" data-end-time="" data-rank="0" data-start-time="" data-story-id="100000005146544" id="topnews-100000005146544">
<h2 class="story-heading"><a href="https://www.nytimes.com/2017/06/12/world/middleeast/isis-cyber.html">Why Cyberwar on ISIS Has Fallen Short of U.S. Hopes</a></h2>
<p class="byline">By DAVID E. SANGER and ERIC SCHMITT </p>
<p class="summary">The effectiveness of cyberweapons hit its limits against an enemy that exploits the internet to recruit, spread propaganda and use encrypted communications, all of which can be quickly reconstituted.</p>
<p class="theme-comments">
<a class="comments-link" href="https://www.nytimes.com/2017/06/12/world/middleeast/isis-cyber.html?hp&amp;target=comments#commentsContainer"><i class="icon sprite-icon comments-icon"></i><span class="comment-count"> Comments</span></a>
</p>
</article>
Why Cyberwar on ISIS Has Fallen Short of U.S. Hopes
This is a story
<article class="story theme-summary" data-collection-renderstyle="HpSum" data-end-time="" data-rank="0" data-start-time="" data-story-id="100000005159532" id="topnews-100000005159532">
<h2 class="story-heading"><a href="https://www.nytimes.com/2017/06/12/us/trump-lawsuit-private-businesses.html">Trump Holdings Are Unconstitutional, Pioneering Suit Says</a></h2>
<p class="byline">By SHARON LaFRANIERE <time class="timestamp" data-eastern-timestamp="12:31 PM" data-utc-timestamp="1497285082" datetime="2017-06-12">12:31 PM ET</time></p>
<p class="summary">The complaint, filed in federal court by Maryland and the District of Columbia, alleges that the president’s failure to shed his businesses has undermined public trust and violated constitutional bans on self-dealing.</p>
</article>
Trump Holdings Are Unconstitutional, Pioneering Suit Says
This is a story
<article class="story theme-summary" data-collection-renderstyle="HpSum" data-end-time="" data-rank="0" data-start-time="" data-story-id="100000005090510" id="topnews-100000005090510">
<h2 class="story-heading"><a href="https://www.nytimes.com/2017/06/12/nyregion/when-opioid-addicts-find-an-ally-in-blue.html">Opioid Addicts Find an Ally in Blue</a></h2>
<p class="byline">By AL BAKER </p>
<p class="summary">Police leaders are assigning themselves a big role in reversing a complex crisis, and not through mass arrests.</p>
<p class="theme-comments">
<a class="comments-link" href="https://www.nytimes.com/2017/06/12/nyregion/when-opioid-addicts-find-an-ally-in-blue.html?hp&amp;target=comments#commentsContainer"><i class="icon sprite-icon comments-icon"></i><span class="comment-count"> Comments</span></a>
</p>
</article>
Opioid Addicts Find an Ally in Blue
This is a story
<article class="story" data-collection-renderstyle="HpHeadline" data-rank="1" data-story-id="100000005137980" id="topnews-100000005137980">
<h2 class="story-heading"><i class="icon"></i><a href="https://www.nytimes.com/2017/06/11/health/vivitrol-drug-opioid-addiction.html">Addiction Drug Lacks Results, but It Has Powerful Friends</a> </h2>
</article>
Addiction Drug Lacks Results, but It Has Powerful Friends 
This is a story
<article class="story theme-summary" data-collection-renderstyle="HpSumSmallMedia" data-rank="0" data-story-id="100000005148452" id="topnews-100000005148452">
<h2 class="story-heading"><a href="https://www.nytimes.com/2017/06/11/us/politics/trump-lawyer-marc-kasowitz.html">Role of Trump’s Lawyer Blurs Public and Private Lines</a></h2>
<p class="byline">By REBECCA R. RUIZ and SHARON LaFRANIERE </p>
<div class="thumb">
<a href="https://www.nytimes.com/2017/06/11/us/politics/trump-lawyer-marc-kasowitz.html"><img alt="" src="https://static01.nyt.com/images/2017/06/10/us/10dc-kasowitz/10dc-kasowitz-thumbStandard.jpg"/></a>
</div>
<p class="summary">
        Marc E. Kasowitz, the president’s private lawyer, has counseled administration staff and exerted broader influence over legal strategy.    </p>
<p class="theme-comments">
<a class="comments-link" href="https://www.nytimes.com/2017/06/11/us/politics/trump-lawyer-marc-kasowitz.html?hp&amp;target=comments#commentsContainer"><i class="icon sprite-icon comments-icon"></i><span class="comment-count"> Comments</span></a>
</p>
</article>
Role of Trump’s Lawyer Blurs Public and Private Lines
This is a story
<article class="story theme-summary" data-collection-renderstyle="HpSum" data-end-time="" data-rank="0" data-start-time="" data-story-id="100000005158161" id="topnews-100000005158161">
<h3 class="kicker">Mediator </h3>
<h2 class="story-heading"><a href="https://www.nytimes.com/2017/06/11/business/media/comey-trump-watergate.html">Trump Era, Unlike Watergate Era, Has Rival Sets of Facts</a></h2>
<p class="byline">By JIM RUTENBERG </p>
<p class="summary">Different versions of the Trump-Russia scandal are fighting for supremacy, leaving news veterans wondering, “What if Watergate happened now?”</p>
<p class="theme-comments">
<a class="comments-link" href="https://www.nytimes.com/2017/06/11/business/media/comey-trump-watergate.html?hp&amp;target=comments#commentsContainer"><i class="icon sprite-icon comments-icon"></i><span class="comment-count"> Comments</span></a>
</p>
</article>
Trump Era, Unlike Watergate Era, Has Rival Sets of Facts
This is a story
<article class="story" data-collection-renderstyle="HpHeadline" data-rank="1" data-story-id="100000005158566" id="topnews-100000005158566">
<h2 class="story-heading"><i class="icon"></i><a href="https://www.nytimes.com/2017/06/11/us/politics/preet-bharara-trump-contacts.html">Fired U.S. Attorney Says Trump Tried to Build Relationship With Him</a> </h2>
</article>
Fired U.S. Attorney Says Trump Tried to Build Relationship With Him 
This is a story
<article class="story theme-feature" data-collection-renderstyle="LargeMediaHeadlineSumCentered" data-rank="1" data-story-id="100000005159290" id="topnews-100000005159290">
<figure class="media photo">
<div class="image">
<a href="https://www.nytimes.com/2017/06/12/magazine/the-long-lonely-road-of-chelsea-manning.html"><img alt="" src="https://static01.nyt.com/images/2017/06/18/magazine/18manning6-copy/18manning6-copy-largeHorizontal375-v2.jpg"/></a>
</div>
<figcaption class="caption" itemprop="description">
<span class="caption-text"></span>
<span class="credit" itemprop="copyrightHolder">
<span class="visually-hidden">Credit</span>
	            Inez and Vinoodh for The New York Times. Stylist: Alex White. 	        </span>
</figcaption>
</figure>
<h1 class="story-heading"><a href="https://www.nytimes.com/2017/06/12/magazine/the-long-lonely-road-of-chelsea-manning.html">The Long, Lonely Road of Chelsea Manning</a></h1>
<p class="summary">Her disclosure of classified documents in 2010 ushered in the age of leaks. Now, freed from prison, she talks about why she did it — and the isolation that followed.</p>
<p class="byline">By MATTHEW SHAER </p>
<ul class="refer theme-news-headlines">
<li>
<article class="story" id="topnews-100000005159290">
<h2 class="refer-heading"><span class="theme-comments"><a class="comments-link" href="https://www.nytimes.com/2017/06/12/magazine/the-long-lonely-road-of-chelsea-manning.html?target=comments#commentsContainer"><i class="icon sprite-icon comments-icon"></i><span class="comment-count">Comments</span></a></span></h2>
</article>
</li>
</ul>
</article>



---------------------------------------------------------------------------

AttributeError                            Traceback (most recent call last)

<ipython-input-7-a704a72bd6ca> in <module>()
      5     headline = story.find('h2', { 'class': 'story-heading' })
      6     print(story)
----> 7     print(headline.text)


AttributeError: 'NoneType' object has no attribute 'text'

The error seems to happen with this one piece here:

<h1 class="story-heading"><a href="https://www.nytimes.com/2017/03/17/nyregion/norman-podhoretz-still-picks- fights-and-drops-names.html">Legendary New York Intellectuals Are His Ex- Friends</a></h1>
<p class="summary">Norman Podhoretz, the former editor at Commentary magazine, looks back at the fierce, argumentative parties of New York’s intelligentsia.</p>
<p class="byline">By JOHN LELAND </p>
<p class="theme-comments">
<a class="comments-link" href="https://www.nytimes.com/2017/03/17/nyregion/norman-podhoretz-still-picks- fights-and-drops-names.html?hp&amp;target=comments#commentsContainer"><i class="icon sprite-icon comments-icon"></i><span class="comment-count"> Comments</span></a>
</p>
</article>

Oh look, it uses an h1 instead of an h2, but it’s still a story-heading. Let’s change our code to look for a story-heading class regardless of tag name.

story = doc.find_all('article', { 'class': 'story' })

for story in story:
    print("This is a story")
    headline = story.find(class_='story-heading')
    print(headline.text)

This is a story
Revised Trump Travel Ban Suffers Another Legal Setback
This is a story
Putin Opponent Is Arrested Amid Protests in Russia
This is a story
Why Cyberwar on ISIS Has Fallen Short of U.S. Hopes
This is a story
Trump Holdings Are Unconstitutional, Pioneering Suit Says
This is a story
Opioid Addicts Find an Ally in Blue
This is a story
Addiction Drug Lacks Results, but It Has Powerful Friends 
This is a story
Role of Trump’s Lawyer Blurs Public and Private Lines
This is a story
Trump Era, Unlike Watergate Era, Has Rival Sets of Facts
This is a story
Fired U.S. Attorney Says Trump Tried to Build Relationship With Him 
This is a story
The Long, Lonely Road of Chelsea Manning
This is a story



---------------------------------------------------------------------------

AttributeError                            Traceback (most recent call last)

<ipython-input-8-d8d1f3203fa2> in <module>()
      4     print("This is a story")
      5     headline = story.find(class_='story-heading')
----> 6     print(headline.text)


AttributeError: 'NoneType' object has no attribute 'text'

Another error! Let’s print out again.

story = doc.find_all('article', { 'class': 'story' })

for story in story:
    print("This is a story")
    headline = story.find(class_='story-heading')
    print(story)
    print(headline.text)

This is a story
<article class="story theme-summary lede" data-collection-renderstyle="LedeSum" data-rank="0" data-story-id="100000005155100" id="topnews-100000005155100">
<h2 class="story-heading"><a href="https://www.nytimes.com/2017/06/12/us/politics/trump-travel-ban-court-of-appeals.html">Revised Trump Travel Ban Suffers Another Legal Setback</a></h2>
<p class="byline">By ADAM LIPTAK <time class="timestamp" data-eastern-timestamp="2:09 PM" data-utc-timestamp="1497290947" datetime="2017-06-12">2:09 PM ET</time></p>
<p class="summary"><ul><li>A ruling by a federal appeals court was the latest in a string of decisions rejecting President Trump’s efforts to limit travel from several predominantly Muslim countries.</li><li>The administration has already sought a Supreme Court review of a similar decision issued last month.</li></ul></p>
<p class="theme-comments">
<a class="comments-link" href="https://www.nytimes.com/2017/06/12/us/politics/trump-travel-ban-court-of-appeals.html?hp&amp;target=comments#commentsContainer"><i class="icon sprite-icon comments-icon"></i><span class="comment-count"> Comments</span></a>
</p>
</article>
Revised Trump Travel Ban Suffers Another Legal Setback
This is a story
<article class="story theme-summary" data-collection-renderstyle="HpSum" data-end-time="" data-rank="0" data-start-time="" data-story-id="100000005159260" id="topnews-100000005159260">
<h2 class="story-heading"><a href="https://www.nytimes.com/2017/06/12/world/europe/russia-aleksei-navalny-kremlin-protests.html">Putin Opponent Is Arrested Amid Protests in Russia</a></h2>
<p class="byline">By NEIL MacFARQUHAR and ANDREW HIGGINS <time class="timestamp" data-eastern-timestamp="1:26 PM" data-utc-timestamp="1497288362" datetime="2017-06-12">1:26 PM ET</time></p>
<p class="summary">The police detained Aleksei A. Navalny, a Kremlin critic and the architect of a wave of antigovernment protests across Russia.</p>
<p class="theme-comments">
<a class="comments-link" href="https://www.nytimes.com/2017/06/12/world/europe/russia-aleksei-navalny-kremlin-protests.html?hp&amp;target=comments#commentsContainer"><i class="icon sprite-icon comments-icon"></i><span class="comment-count"> Comments</span></a>
</p>
</article>
Putin Opponent Is Arrested Amid Protests in Russia
This is a story
<article class="story theme-summary" data-collection-renderstyle="HpSum" data-end-time="" data-rank="0" data-start-time="" data-story-id="100000005146544" id="topnews-100000005146544">
<h2 class="story-heading"><a href="https://www.nytimes.com/2017/06/12/world/middleeast/isis-cyber.html">Why Cyberwar on ISIS Has Fallen Short of U.S. Hopes</a></h2>
<p class="byline">By DAVID E. SANGER and ERIC SCHMITT </p>
<p class="summary">The effectiveness of cyberweapons hit its limits against an enemy that exploits the internet to recruit, spread propaganda and use encrypted communications, all of which can be quickly reconstituted.</p>
<p class="theme-comments">
<a class="comments-link" href="https://www.nytimes.com/2017/06/12/world/middleeast/isis-cyber.html?hp&amp;target=comments#commentsContainer"><i class="icon sprite-icon comments-icon"></i><span class="comment-count"> Comments</span></a>
</p>
</article>
Why Cyberwar on ISIS Has Fallen Short of U.S. Hopes
This is a story
<article class="story theme-summary" data-collection-renderstyle="HpSum" data-end-time="" data-rank="0" data-start-time="" data-story-id="100000005159532" id="topnews-100000005159532">
<h2 class="story-heading"><a href="https://www.nytimes.com/2017/06/12/us/trump-lawsuit-private-businesses.html">Trump Holdings Are Unconstitutional, Pioneering Suit Says</a></h2>
<p class="byline">By SHARON LaFRANIERE <time class="timestamp" data-eastern-timestamp="12:31 PM" data-utc-timestamp="1497285082" datetime="2017-06-12">12:31 PM ET</time></p>
<p class="summary">The complaint, filed in federal court by Maryland and the District of Columbia, alleges that the president’s failure to shed his businesses has undermined public trust and violated constitutional bans on self-dealing.</p>
</article>
Trump Holdings Are Unconstitutional, Pioneering Suit Says
This is a story
<article class="story theme-summary" data-collection-renderstyle="HpSum" data-end-time="" data-rank="0" data-start-time="" data-story-id="100000005090510" id="topnews-100000005090510">
<h2 class="story-heading"><a href="https://www.nytimes.com/2017/06/12/nyregion/when-opioid-addicts-find-an-ally-in-blue.html">Opioid Addicts Find an Ally in Blue</a></h2>
<p class="byline">By AL BAKER </p>
<p class="summary">Police leaders are assigning themselves a big role in reversing a complex crisis, and not through mass arrests.</p>
<p class="theme-comments">
<a class="comments-link" href="https://www.nytimes.com/2017/06/12/nyregion/when-opioid-addicts-find-an-ally-in-blue.html?hp&amp;target=comments#commentsContainer"><i class="icon sprite-icon comments-icon"></i><span class="comment-count"> Comments</span></a>
</p>
</article>
Opioid Addicts Find an Ally in Blue
This is a story
<article class="story" data-collection-renderstyle="HpHeadline" data-rank="1" data-story-id="100000005137980" id="topnews-100000005137980">
<h2 class="story-heading"><i class="icon"></i><a href="https://www.nytimes.com/2017/06/11/health/vivitrol-drug-opioid-addiction.html">Addiction Drug Lacks Results, but It Has Powerful Friends</a> </h2>
</article>
Addiction Drug Lacks Results, but It Has Powerful Friends 
This is a story
<article class="story theme-summary" data-collection-renderstyle="HpSumSmallMedia" data-rank="0" data-story-id="100000005148452" id="topnews-100000005148452">
<h2 class="story-heading"><a href="https://www.nytimes.com/2017/06/11/us/politics/trump-lawyer-marc-kasowitz.html">Role of Trump’s Lawyer Blurs Public and Private Lines</a></h2>
<p class="byline">By REBECCA R. RUIZ and SHARON LaFRANIERE </p>
<div class="thumb">
<a href="https://www.nytimes.com/2017/06/11/us/politics/trump-lawyer-marc-kasowitz.html"><img alt="" src="https://static01.nyt.com/images/2017/06/10/us/10dc-kasowitz/10dc-kasowitz-thumbStandard.jpg"/></a>
</div>
<p class="summary">
        Marc E. Kasowitz, the president’s private lawyer, has counseled administration staff and exerted broader influence over legal strategy.    </p>
<p class="theme-comments">
<a class="comments-link" href="https://www.nytimes.com/2017/06/11/us/politics/trump-lawyer-marc-kasowitz.html?hp&amp;target=comments#commentsContainer"><i class="icon sprite-icon comments-icon"></i><span class="comment-count"> Comments</span></a>
</p>
</article>
Role of Trump’s Lawyer Blurs Public and Private Lines
This is a story
<article class="story theme-summary" data-collection-renderstyle="HpSum" data-end-time="" data-rank="0" data-start-time="" data-story-id="100000005158161" id="topnews-100000005158161">
<h3 class="kicker">Mediator </h3>
<h2 class="story-heading"><a href="https://www.nytimes.com/2017/06/11/business/media/comey-trump-watergate.html">Trump Era, Unlike Watergate Era, Has Rival Sets of Facts</a></h2>
<p class="byline">By JIM RUTENBERG </p>
<p class="summary">Different versions of the Trump-Russia scandal are fighting for supremacy, leaving news veterans wondering, “What if Watergate happened now?”</p>
<p class="theme-comments">
<a class="comments-link" href="https://www.nytimes.com/2017/06/11/business/media/comey-trump-watergate.html?hp&amp;target=comments#commentsContainer"><i class="icon sprite-icon comments-icon"></i><span class="comment-count"> Comments</span></a>
</p>
</article>
Trump Era, Unlike Watergate Era, Has Rival Sets of Facts
This is a story
<article class="story" data-collection-renderstyle="HpHeadline" data-rank="1" data-story-id="100000005158566" id="topnews-100000005158566">
<h2 class="story-heading"><i class="icon"></i><a href="https://www.nytimes.com/2017/06/11/us/politics/preet-bharara-trump-contacts.html">Fired U.S. Attorney Says Trump Tried to Build Relationship With Him</a> </h2>
</article>
Fired U.S. Attorney Says Trump Tried to Build Relationship With Him 
This is a story
<article class="story theme-feature" data-collection-renderstyle="LargeMediaHeadlineSumCentered" data-rank="1" data-story-id="100000005159290" id="topnews-100000005159290">
<figure class="media photo">
<div class="image">
<a href="https://www.nytimes.com/2017/06/12/magazine/the-long-lonely-road-of-chelsea-manning.html"><img alt="" src="https://static01.nyt.com/images/2017/06/18/magazine/18manning6-copy/18manning6-copy-largeHorizontal375-v2.jpg"/></a>
</div>
<figcaption class="caption" itemprop="description">
<span class="caption-text"></span>
<span class="credit" itemprop="copyrightHolder">
<span class="visually-hidden">Credit</span>
	            Inez and Vinoodh for The New York Times. Stylist: Alex White. 	        </span>
</figcaption>
</figure>
<h1 class="story-heading"><a href="https://www.nytimes.com/2017/06/12/magazine/the-long-lonely-road-of-chelsea-manning.html">The Long, Lonely Road of Chelsea Manning</a></h1>
<p class="summary">Her disclosure of classified documents in 2010 ushered in the age of leaks. Now, freed from prison, she talks about why she did it — and the isolation that followed.</p>
<p class="byline">By MATTHEW SHAER </p>
<ul class="refer theme-news-headlines">
<li>
<article class="story" id="topnews-100000005159290">
<h2 class="refer-heading"><span class="theme-comments"><a class="comments-link" href="https://www.nytimes.com/2017/06/12/magazine/the-long-lonely-road-of-chelsea-manning.html?target=comments#commentsContainer"><i class="icon sprite-icon comments-icon"></i><span class="comment-count">Comments</span></a></span></h2>
</article>
</li>
</ul>
</article>
The Long, Lonely Road of Chelsea Manning
This is a story
<article class="story" id="topnews-100000005159290">
<h2 class="refer-heading"><span class="theme-comments"><a class="comments-link" href="https://www.nytimes.com/2017/06/12/magazine/the-long-lonely-road-of-chelsea-manning.html?target=comments#commentsContainer"><i class="icon sprite-icon comments-icon"></i><span class="comment-count">Comments</span></a></span></h2>
</article>



---------------------------------------------------------------------------

AttributeError                            Traceback (most recent call last)

<ipython-input-9-dfdf9ca56478> in <module>()
      5     headline = story.find(class_='story-heading')
      6     print(story)
----> 7     print(headline.text)


AttributeError: 'NoneType' object has no attribute 'text'

It looks like it failed on this one.

<article class="story">
<h3 class="kicker">
<a href="http://wordplay.blogs.nytimes.com">Wordplay »</a>
</h3>
</article>

Now we have a choice to make: do we care about this? I… don’t. If we want to skip through to the next element in a loop, we can use continue.

Let’s say hey, if you don’t have a headline, we’re going to skip you.

story = doc.find_all('article', { 'class': 'story' })

for story in story:
    print("This is a story")
    headline = story.find(class_='story-heading')
    if not headline:
        continue
    print(headline.text)

This is a story
Revised Trump Travel Ban Suffers Another Legal Setback
This is a story
Putin Opponent Is Arrested Amid Protests in Russia
This is a story
Why Cyberwar on ISIS Has Fallen Short of U.S. Hopes
This is a story
Trump Holdings Are Unconstitutional, Pioneering Suit Says
This is a story
Opioid Addicts Find an Ally in Blue
This is a story
Addiction Drug Lacks Results, but It Has Powerful Friends 
This is a story
Role of Trump’s Lawyer Blurs Public and Private Lines
This is a story
Trump Era, Unlike Watergate Era, Has Rival Sets of Facts
This is a story
Fired U.S. Attorney Says Trump Tried to Build Relationship With Him 
This is a story
The Long, Lonely Road of Chelsea Manning
This is a story
This is a story
California Today: Talking to a Tony Winner
This is a story


      Listen to ‘The Daily’
    

This is a story
How to Save on Summer Travel
This is a story
Ways Your iPhone Will Change After Apple’s iOS 11
This is a story
Gargantuan Job: Dismantling a Million Tons of North Sea Oil Rigs
This is a story
The Best (and Worst) of the 2017 Tony Awards
This is a story
‘Dear Evan Hansen’ Wins 6 Tonys; Midler Is Best Leading Actress
This is a story
Television Review: An Uneven Night for Kevin Spacey as Host
This is a story
Fashion Review: The Tony Awards Get the Red Carpet Blues
This is a story
‘We Are Not Unusual Anymore’: 50 Years of Mixed-Race Marriage in U.S.
This is a story
Ahead of ‘Brexit,’ Britons Face Economic Pinch at Home
This is a story
For Democrats, Spotlight Turns to Race to Replace Christie
This is a story
This is a story
Death Penalty in Pakistan for Social Media Blasphemy 1:45 PM ET
This is a story
Theresa May Battles to Hold On as U.K. Prime Minister 12:54 PM ET
This is a story
Uber’s No. 2 Executive Leaves Firm 12:32 PM ET
This is a story
Jeffrey Immelt to Retire as General Electric Chief 1:56 PM ET
This is a story
Bill Cosby Sex Assault Trial: The Defense Rests 1:07 PM ET
This is a story
Morning Briefing: Australia Edition
This is a story
Morning Briefing: Asia Edition
This is a story
Morning Briefing: Europe Edition
This is a story
Morning Briefing
This is a story

Australia to Tighten Parole Laws After Deadly Siege

This is a story

Australian Politics Is Open to Foreign Cash, and China Has Much to Gain

This is a story

Australia Makes Riesling Its Own

This is a story

Take Our Quiz on Solving Global Warming

This is a story

Something Old and New at the French Open

This is a story

How a Dark Web Mastermind Went Down

This is a story
Making Google the Censor
This is a story
Editorial: Mr. Trump Goes After the Inspectors 
This is a story
Blow: Impeachment Anxiety 
This is a story
Krugman: They Don’t Need No Information 
This is a story
Sexual Abusers Shouldn’t Be Allowed to Run the Clock 
This is a story
A Night of Terror, a Year of Racism 
This is a story
The True Meaning of ‘Idiot’ 
This is a story
Loving, 50 Years Later
This is a story
Finally, Something Isn’t the Matter With Kansas 
This is a story
America Needs Closure on the Travel Ban 
This is a story
Follow us on Twitter » 
This is a story

Bar Code: A New Infatuation Poised for a Puzzle Craze

This is a story

Agree With ‘The 25 Best Films’? Tell Us How You Really Feel

This is a story

Bar Code: A New Infatuation Poised for a Puzzle Craze

This is a story

Play Today’s Puzzle 

This is a story

Play Today’s Puzzle 

This is a story
This is a story
This is a story
At Palace, the Unbearable Being of Lines
This is a story
A Fierce Famine Stalks Africa
This is a story
The Lice Ladies of Brooklyn Go Corporate
This is a story
The Revolutionary Specters of Russian Letters
This is a story
Meet the Musicians Who Modeled for Gucci
This is a story
What We’re Reading
This is a story
Ideas Sparked by Flow of Twitter Conversations
This is a story
Making Google the Censor
This is a story
What Yotam Ottolenghi Can’t Do Without
This is a story
Will Iran Descend Into Chaos?
This is a story
From the Subways to the Sofa: Pet Rats
This is a story
The Presence of Violence at the Kabul Bureau
This is a story

        U.S. Cyberweapons, Used Against Iran and North Korea, Are a Disappointment Against ISIS        
This is a story


            Reviving a Lost Language of Canada Through Film        

This is a story


            Emmanuel Macron’s Party on Track to Claim Majority in France’s Parliament        

This is a story

        Jeffrey Immelt to Retire as General Electric Chief        
This is a story


            Emil Michael Leaves Uber        

This is a story


            Uber Weighs Leave of Absence for Chief Executive        

This is a story

        Charles M. Blow: The Resistance: Impeachment Anxiety        
This is a story


            Editorial: Mr. Trump Goes After the Inspectors        

This is a story


            Op-Ed Contributor: Making Google the Censor        

This is a story

        ‘We Are Not Unusual Anymore’: 50 Years of Mixed-Race Marriage in U.S.        
This is a story


            For Flint, an 1882 Play About Bad Water Is ‘Exactly What We’re Living’        

This is a story


            Heroin Use? Juvenile Record? For Recruits, Police Forgive Past Sins        

This is a story

        Uber Weighs Leave of Absence for Chief Executive        
This is a story


            Emil Michael Leaves Uber        

This is a story


            U.S. Cyberweapons, Used Against Iran and North Korea, Are a Disappointment Against ISIS        

This is a story

        Tony Awards Briefing: ‘Dear Evan Hansen’ Wins 6 Tony Awards; Bette Midler Is Best Leading Actress        
This is a story


            The Best (and Worst) of the 2017 Tony Awards        

This is a story


            Et Tu, Delta? Shakespeare in the Park Sponsors Withdraw From Trump-Like ‘Julius Caesar’        

This is a story

        Supreme Court Bars Favoring Mothers Over Fathers in Citizenship Case        
This is a story


            Trump Loses Travel Ban Ruling in Appeals Court        

This is a story


            Ivanka Trump Says Father Felt ‘Vindicated’ by Comey Testimony        

This is a story

        On the Runway: The Tony Awards Get the Red Carpet Blues        
This is a story


            Styles Q. & A.: Virgil Abloh Projects Politics Into His Florence Fashion Show        

This is a story


            Death of a Downtown Icon: Remembering Benjamin Cho        

This is a story

        The 25 Best Films of the 21st Century So Far.        
This is a story


            Review: ‘The Mummy,’ With Tom Cruise, Deserves a Quick Burial        

This is a story


            Review: Mom, Dad, the Kid and the Dog. What Could Be Scarier?        

This is a story

        When Opioid Addicts Find an Ally in Blue        
This is a story


            L.I.R.R. Announces Travel Options for ‘Summer of Hell’        

This is a story


            What’s the Insurance Capital Without Aetna? Hartford May Find Out        

This is a story

        Keeping Score: All Should Rise for Yankees’ Gary Sanchez, Too        
This is a story


            Penguins 2, Predators 0 | Pittsburgh wins series, 4-2: Pittsburgh Penguins Beat the Predators to Repeat as Stanley Cup Champions        

This is a story


            Give Baseball Scouts a (Perfect) 80 for Tradition        

This is a story

        The Best (and Worst) of the 2017 Tony Awards        
This is a story


            Tony Awards Briefing: ‘Dear Evan Hansen’ Wins 6 Tony Awards; Bette Midler Is Best Leading Actress        

This is a story


            Et Tu, Delta? Shakespeare in the Park Sponsors Withdraw From Trump-Like ‘Julius Caesar’        

This is a story

        On Nuclear Waste, Finland Shows U.S. How It Can Be Done        
This is a story


            Basics: The Liver: A ‘Blob’ That Runs the Body        

This is a story


            Trilobites: These Hungry Goats Learned to Branch Out        

This is a story

        Edith Shiffert, a Poet Inspired by Nature and Her Life in Japan, Dies at 101        
This is a story


            Sad Day for Gotham: Adam West, Who Played Batman, Dies at 88        

This is a story


            Andimba Toivo ya Toivo, Namibian Independence Leader, Dies at 92        

This is a story

        Television Review: An Uneven Night for Kevin Spacey as Tonys Host        
This is a story


            Oliver Stone’s ‘Putin Interviews’: Flattery, but Little Skepticism        

This is a story


            Televsion Review: ‘Claws’ on TNT: That Manicure Comes With a Murder        

This is a story

        Basics: The Liver: A ‘Blob’ That Runs the Body        
This is a story


             The New Old Age: Hearing Aids at the Mall? Congress Could Make It Happen        

This is a story


            More Than 10 Percent of World’s Population Is Obese, Study Finds        

This is a story

        The Getaway: How Much Are Frequent Flier Miles Really Worth?        
This is a story


            U.S. Islands: On a Georgia Island, a Lot of Good Food and Plenty of Nothing        

This is a story


            Essay: How to Ease Travel Anxiety in an Era of Terror: Travel More        

This is a story

        Nonfiction: How the Dark Web’s Dread Pirate Roberts Went Down        
This is a story


            Q. and A.: Tell Us 5 Things About Your Book: A Reporter Inside the World of Jihad        

This is a story


            Fiction: Arundhati Roy’s Return to the Form That Made Her Famous        

This is a story

        To Understand Betsy DeVos’s Educational Views, View Her Education        
This is a story


            David L. Kirp: Ending the Curse of Remedial Math        

This is a story


            Success Academy Wins Round in Fight Over Preschool Oversight        

This is a story

        Hungry City: At Bep Ga in Chinatown, It All Begins With Chicken        
This is a story


            City Kitchen: A Juicy Tenderloin, a Pesto Swirl        

This is a story


            A Good Appetite: The Simplest Shortbread You Can Bake        

This is a story

        Opinion: Is Putin Getting What He Wanted With Trump?        
This is a story


            Editorial: It’s the Olympics for Trump Apologists        

This is a story


            Frank Bruni: Can Democrats Save Themselves?        

This is a story

        The Elusive $1,500 Rental        
This is a story


            Exclusive: Where a Rockefeller Raised His Family        

This is a story


            Ask Real Estate: Creating New Windows Without Ruining the Building        

This is a story

        Your Rabbi? Probably a Democrat. Your Baptist Pastor? Probably a Republican. Your Priest? Who Knows.        
This is a story


            The New Health Care: There’s No Magical Savings in Showing Prices to Doctors        

This is a story


            Economic View: Why Women Don’t See Themselves as Entrepreneurs        

This is a story

        Feature: The Long, Lonely Road of Chelsea Manning        
This is a story


            Feature: America’s Hidden H.I.V. Epidemic        

This is a story


            Feature: The Bounty Hunter of Wall Street        

This is a story

        Driven: Video Review: With New Ioniq, Hyundai Goes After Prius        
This is a story


            Wheels: In Portland, a Rare Outpost for Die-Hard Saab Fans        

This is a story


            Bits: Electronic Setups of Driverless Cars Vulnerable to Hackers        

This is a story

        6 Designers (Straight Out of Fashion School) to Know        
This is a story


            Some Things T Editors Are Really Into Right Now        

This is a story


            8 Male Models to Watch This Season        

This is a story

        Bar Code: A New Infatuation Poised for a Puzzle Craze        
This is a story


            Ahead of the Tonys, Watching Everything        

This is a story


            Agree With ‘The 25 Best Films’? Tell Us How You Really Feel        

This is a story
Homes That Sold for Around $450,000
This is a story
Search for Homes for Sale or Rent
This is a story
Mortgage Calculator

Maybe we can also say hey, let’s get rid of the whitespace on the headlines by using .strip()

story = doc.find_all('article', { 'class': 'story' })

for story in story:
    headline = story.find(class_='story-heading')
    if not headline:
        continue
    print(headline.text.strip())

Revised Trump Travel Ban Suffers Another Legal Setback
Putin Opponent Is Arrested Amid Protests in Russia
Why Cyberwar on ISIS Has Fallen Short of U.S. Hopes
Trump Holdings Are Unconstitutional, Pioneering Suit Says
Opioid Addicts Find an Ally in Blue
Addiction Drug Lacks Results, but It Has Powerful Friends
Role of Trump’s Lawyer Blurs Public and Private Lines
Trump Era, Unlike Watergate Era, Has Rival Sets of Facts
Fired U.S. Attorney Says Trump Tried to Build Relationship With Him
The Long, Lonely Road of Chelsea Manning
California Today: Talking to a Tony Winner
Listen to ‘The Daily’
How to Save on Summer Travel
Ways Your iPhone Will Change After Apple’s iOS 11
Gargantuan Job: Dismantling a Million Tons of North Sea Oil Rigs
The Best (and Worst) of the 2017 Tony Awards
‘Dear Evan Hansen’ Wins 6 Tonys; Midler Is Best Leading Actress
Television Review: An Uneven Night for Kevin Spacey as Host
Fashion Review: The Tony Awards Get the Red Carpet Blues
‘We Are Not Unusual Anymore’: 50 Years of Mixed-Race Marriage in U.S.
Ahead of ‘Brexit,’ Britons Face Economic Pinch at Home
For Democrats, Spotlight Turns to Race to Replace Christie
Death Penalty in Pakistan for Social Media Blasphemy 1:45 PM ET
Theresa May Battles to Hold On as U.K. Prime Minister 12:54 PM ET
Uber’s No. 2 Executive Leaves Firm 12:32 PM ET
Jeffrey Immelt to Retire as General Electric Chief 1:56 PM ET
Bill Cosby Sex Assault Trial: The Defense Rests 1:07 PM ET
Morning Briefing: Australia Edition
Morning Briefing: Asia Edition
Morning Briefing: Europe Edition
Morning Briefing
Australia to Tighten Parole Laws After Deadly Siege
Australian Politics Is Open to Foreign Cash, and China Has Much to Gain
Australia Makes Riesling Its Own
Take Our Quiz on Solving Global Warming
Something Old and New at the French Open
How a Dark Web Mastermind Went Down
Making Google the Censor
Editorial: Mr. Trump Goes After the Inspectors
Blow: Impeachment Anxiety
Krugman: They Don’t Need No Information
Sexual Abusers Shouldn’t Be Allowed to Run the Clock
A Night of Terror, a Year of Racism
The True Meaning of ‘Idiot’
Loving, 50 Years Later
Finally, Something Isn’t the Matter With Kansas
America Needs Closure on the Travel Ban
Follow us on Twitter »
Bar Code: A New Infatuation Poised for a Puzzle Craze
Agree With ‘The 25 Best Films’? Tell Us How You Really Feel
Bar Code: A New Infatuation Poised for a Puzzle Craze
Play Today’s Puzzle
Play Today’s Puzzle
At Palace, the Unbearable Being of Lines
A Fierce Famine Stalks Africa
The Lice Ladies of Brooklyn Go Corporate
The Revolutionary Specters of Russian Letters
Meet the Musicians Who Modeled for Gucci
What We’re Reading
Ideas Sparked by Flow of Twitter Conversations
Making Google the Censor
What Yotam Ottolenghi Can’t Do Without
Will Iran Descend Into Chaos?
From the Subways to the Sofa: Pet Rats
The Presence of Violence at the Kabul Bureau
U.S. Cyberweapons, Used Against Iran and North Korea, Are a Disappointment Against ISIS
Reviving a Lost Language of Canada Through Film
Emmanuel Macron’s Party on Track to Claim Majority in France’s Parliament
Jeffrey Immelt to Retire as General Electric Chief
Emil Michael Leaves Uber
Uber Weighs Leave of Absence for Chief Executive
Charles M. Blow: The Resistance: Impeachment Anxiety
Editorial: Mr. Trump Goes After the Inspectors
Op-Ed Contributor: Making Google the Censor
‘We Are Not Unusual Anymore’: 50 Years of Mixed-Race Marriage in U.S.
For Flint, an 1882 Play About Bad Water Is ‘Exactly What We’re Living’
Heroin Use? Juvenile Record? For Recruits, Police Forgive Past Sins
Uber Weighs Leave of Absence for Chief Executive
Emil Michael Leaves Uber
U.S. Cyberweapons, Used Against Iran and North Korea, Are a Disappointment Against ISIS
Tony Awards Briefing: ‘Dear Evan Hansen’ Wins 6 Tony Awards; Bette Midler Is Best Leading Actress
The Best (and Worst) of the 2017 Tony Awards
Et Tu, Delta? Shakespeare in the Park Sponsors Withdraw From Trump-Like ‘Julius Caesar’
Supreme Court Bars Favoring Mothers Over Fathers in Citizenship Case
Trump Loses Travel Ban Ruling in Appeals Court
Ivanka Trump Says Father Felt ‘Vindicated’ by Comey Testimony
On the Runway: The Tony Awards Get the Red Carpet Blues
Styles Q. & A.: Virgil Abloh Projects Politics Into His Florence Fashion Show
Death of a Downtown Icon: Remembering Benjamin Cho
The 25 Best Films of the 21st Century So Far.
Review: ‘The Mummy,’ With Tom Cruise, Deserves a Quick Burial
Review: Mom, Dad, the Kid and the Dog. What Could Be Scarier?
When Opioid Addicts Find an Ally in Blue
L.I.R.R. Announces Travel Options for ‘Summer of Hell’
What’s the Insurance Capital Without Aetna? Hartford May Find Out
Keeping Score: All Should Rise for Yankees’ Gary Sanchez, Too
Penguins 2, Predators 0 | Pittsburgh wins series, 4-2: Pittsburgh Penguins Beat the Predators to Repeat as Stanley Cup Champions
Give Baseball Scouts a (Perfect) 80 for Tradition
The Best (and Worst) of the 2017 Tony Awards
Tony Awards Briefing: ‘Dear Evan Hansen’ Wins 6 Tony Awards; Bette Midler Is Best Leading Actress
Et Tu, Delta? Shakespeare in the Park Sponsors Withdraw From Trump-Like ‘Julius Caesar’
On Nuclear Waste, Finland Shows U.S. How It Can Be Done
Basics: The Liver: A ‘Blob’ That Runs the Body
Trilobites: These Hungry Goats Learned to Branch Out
Edith Shiffert, a Poet Inspired by Nature and Her Life in Japan, Dies at 101
Sad Day for Gotham: Adam West, Who Played Batman, Dies at 88
Andimba Toivo ya Toivo, Namibian Independence Leader, Dies at 92
Television Review: An Uneven Night for Kevin Spacey as Tonys Host
Oliver Stone’s ‘Putin Interviews’: Flattery, but Little Skepticism
Televsion Review: ‘Claws’ on TNT: That Manicure Comes With a Murder
Basics: The Liver: A ‘Blob’ That Runs the Body
The New Old Age: Hearing Aids at the Mall? Congress Could Make It Happen
More Than 10 Percent of World’s Population Is Obese, Study Finds
The Getaway: How Much Are Frequent Flier Miles Really Worth?
U.S. Islands: On a Georgia Island, a Lot of Good Food and Plenty of Nothing
Essay: How to Ease Travel Anxiety in an Era of Terror: Travel More
Nonfiction: How the Dark Web’s Dread Pirate Roberts Went Down
Q. and A.: Tell Us 5 Things About Your Book: A Reporter Inside the World of Jihad
Fiction: Arundhati Roy’s Return to the Form That Made Her Famous
To Understand Betsy DeVos’s Educational Views, View Her Education
David L. Kirp: Ending the Curse of Remedial Math
Success Academy Wins Round in Fight Over Preschool Oversight
Hungry City: At Bep Ga in Chinatown, It All Begins With Chicken
City Kitchen: A Juicy Tenderloin, a Pesto Swirl
A Good Appetite: The Simplest Shortbread You Can Bake
Opinion: Is Putin Getting What He Wanted With Trump?
Editorial: It’s the Olympics for Trump Apologists
Frank Bruni: Can Democrats Save Themselves?
The Elusive $1,500 Rental
Exclusive: Where a Rockefeller Raised His Family
Ask Real Estate: Creating New Windows Without Ruining the Building
Your Rabbi? Probably a Democrat. Your Baptist Pastor? Probably a Republican. Your Priest? Who Knows.
The New Health Care: There’s No Magical Savings in Showing Prices to Doctors
Economic View: Why Women Don’t See Themselves as Entrepreneurs
Feature: The Long, Lonely Road of Chelsea Manning
Feature: America’s Hidden H.I.V. Epidemic
Feature: The Bounty Hunter of Wall Street
Driven: Video Review: With New Ioniq, Hyundai Goes After Prius
Wheels: In Portland, a Rare Outpost for Die-Hard Saab Fans
Bits: Electronic Setups of Driverless Cars Vulnerable to Hackers
6 Designers (Straight Out of Fashion School) to Know
Some Things T Editors Are Really Into Right Now
8 Male Models to Watch This Season
Bar Code: A New Infatuation Poised for a Puzzle Craze
Ahead of the Tonys, Watching Everything
Agree With ‘The 25 Best Films’? Tell Us How You Really Feel
Homes That Sold for Around $450,000
Search for Homes for Sale or Rent
Mortgage Calculator

Next step: Adding more pieces

Now we need to add in the links and the bylines. We’ll start with the links by pulling in any a tags.

story = doc.find_all('article', { 'class': 'story' })

for story in story:
    headline = story.find(class_='story-heading')
    if not headline:
        continue
    print(headline.text.strip())
    link = story.find('a')
    print(link['href'])

Revised Trump Travel Ban Suffers Another Legal Setback
https://www.nytimes.com/2017/06/12/us/politics/trump-travel-ban-court-of-appeals.html
Putin Opponent Is Arrested Amid Protests in Russia
https://www.nytimes.com/2017/06/12/world/europe/russia-aleksei-navalny-kremlin-protests.html
Why Cyberwar on ISIS Has Fallen Short of U.S. Hopes
https://www.nytimes.com/2017/06/12/world/middleeast/isis-cyber.html
Trump Holdings Are Unconstitutional, Pioneering Suit Says
https://www.nytimes.com/2017/06/12/us/trump-lawsuit-private-businesses.html
Opioid Addicts Find an Ally in Blue
https://www.nytimes.com/2017/06/12/nyregion/when-opioid-addicts-find-an-ally-in-blue.html
Addiction Drug Lacks Results, but It Has Powerful Friends
https://www.nytimes.com/2017/06/11/health/vivitrol-drug-opioid-addiction.html
Role of Trump’s Lawyer Blurs Public and Private Lines
https://www.nytimes.com/2017/06/11/us/politics/trump-lawyer-marc-kasowitz.html
Trump Era, Unlike Watergate Era, Has Rival Sets of Facts
https://www.nytimes.com/2017/06/11/business/media/comey-trump-watergate.html
Fired U.S. Attorney Says Trump Tried to Build Relationship With Him
https://www.nytimes.com/2017/06/11/us/politics/preet-bharara-trump-contacts.html
The Long, Lonely Road of Chelsea Manning
https://www.nytimes.com/2017/06/12/magazine/the-long-lonely-road-of-chelsea-manning.html
California Today: Talking to a Tony Winner
https://www.nytimes.com/2017/06/12/us/california-today-talking-to-a-tony-winner.html
Listen to ‘The Daily’
https://www.nytimes.com/column/the-daily
How to Save on Summer Travel
https://www.nytimes.com/interactive/2017/05/16/travel/frugal-traveler-summer-travel-chat.html
Ways Your iPhone Will Change After Apple’s iOS 11
https://www.nytimes.com/2017/06/12/smarter-living/apple-iphone-ios11.html
Gargantuan Job: Dismantling a Million Tons of North Sea Oil Rigs
https://www.nytimes.com/2017/06/08/business/energy-environment/oil-north-sea-shell.html
The Best (and Worst) of the 2017 Tony Awards
https://www.nytimes.com/2017/06/12/theater/the-best-and-worst-of-the-2017-tony-awards.html
‘Dear Evan Hansen’ Wins 6 Tonys; Midler Is Best Leading Actress
https://www.nytimes.com/2017/06/11/theater/tony-awards.html
Television Review: An Uneven Night for Kevin Spacey as Host
https://www.nytimes.com/2017/06/12/theater/tony-awards-review-recap-kevin-spacey.html
Fashion Review: The Tony Awards Get the Red Carpet Blues
https://www.nytimes.com/2017/06/12/fashion/tony-awards-red-carpet-trends.html
‘We Are Not Unusual Anymore’: 50 Years of Mixed-Race Marriage in U.S.
https://www.nytimes.com/2017/06/11/us/mixed-race-marriage-loving-decision-anniversary.html
Ahead of ‘Brexit,’ Britons Face Economic Pinch at Home
https://www.nytimes.com/2017/06/11/business/economy/britain-economy-election-brexit.html
For Democrats, Spotlight Turns to Race to Replace Christie
https://www.nytimes.com/2017/06/09/nyregion/democrats-new-jersey-governor-philip-murphy.html
Death Penalty in Pakistan for Social Media Blasphemy 1:45 PM ET
https://www.nytimes.com/2017/06/12/world/asia/pakistan-blasphemy-sentence.html
Theresa May Battles to Hold On as U.K. Prime Minister 12:54 PM ET
https://www.nytimes.com/2017/06/12/world/europe/uk-theresa-may-britain.html
Uber’s No. 2 Executive Leaves Firm 12:32 PM ET
https://www.nytimes.com/2017/06/12/technology/uber-travis-kalanick-emil-michael.html
Jeffrey Immelt to Retire as General Electric Chief 1:56 PM ET
https://www.nytimes.com/2017/06/12/business/ge-immelt.html
Bill Cosby Sex Assault Trial: The Defense Rests 1:07 PM ET
https://www.nytimes.com/2017/06/12/arts/television/bill-cosby-trial-day-6.html
Morning Briefing: Australia Edition
https://www.nytimes.com/newsletters/morning-briefing-australia
Morning Briefing: Asia Edition
https://www.nytimes.com/newsletters/morning-briefing-asia
Morning Briefing: Europe Edition
https://www.nytimes.com/newsletters/morning-briefing-europe
Morning Briefing
https://www.nytimes.com/newsletters/morning-briefing
Australia to Tighten Parole Laws After Deadly Siege
https://www.nytimes.com/2017/06/09/world/australia/australia-melbourne-terror-parole.html
Australian Politics Is Open to Foreign Cash, and China Has Much to Gain
https://www.nytimes.com/2017/06/06/world/australia/china-political-influence-campaign-finance.html
Australia Makes Riesling Its Own
https://www.nytimes.com/2017/06/08/dining/wine-review-australian-riesling.html
Take Our Quiz on Solving Global Warming
https://www.nytimes.com/interactive/2017/06/09/climate/drawdown-climate-solutions-quiz.html
Something Old and New at the French Open
https://www.nytimes.com/2017/06/12/sports/tennis/something-old-something-new-at-french-open.html
How a Dark Web Mastermind Went Down
https://www.nytimes.com/2017/06/12/books/review/american-kingpin-nick-bilton.html
Making Google the Censor
https://www.nytimes.com/2017/06/12/opinion/making-google-the-censor.html
Editorial: Mr. Trump Goes After the Inspectors
https://www.nytimes.com/2017/06/12/opinion/mr-trump-goes-after-the-inspectors.html
Blow: Impeachment Anxiety
https://www.nytimes.com/2017/06/12/opinion/the-resistance-impeachment-anxiety.html
Krugman: They Don’t Need No Information
https://krugman.blogs.nytimes.com/2017/06/11/they-dont-need-no-information/
Sexual Abusers Shouldn’t Be Allowed to Run the Clock
https://www.nytimes.com/2017/06/12/opinion/sexual-abusers-shouldnt-be-allowed-to-run-the-clock.html
A Night of Terror, a Year of Racism
https://www.nytimes.com/2017/06/12/opinion/a-night-of-terror-a-year-of-racism.html
The True Meaning of ‘Idiot’
https://www.nytimes.com/2017/06/12/opinion/trump-and-the-true-meaning-of-idiot.html
Loving, 50 Years Later
https://www.nytimes.com/interactive/2017/06/12/opinion/loving-virginia-50-year-anniversary.html
Finally, Something Isn’t the Matter With Kansas
https://www.nytimes.com/2017/06/12/opinion/finally-something-isnt-the-matter-with-kansas.html
America Needs Closure on the Travel Ban
https://www.nytimes.com/2017/06/11/opinion/america-needs-closure-on-the-travel-ban.html
Follow us on Twitter »
https://twitter.com/nytopinion
Bar Code: A New Infatuation Poised for a Puzzle Craze
https://www.nytimes.com/2017/06/12/insider/bar-code-a-new-infatuation-poised-for-a-puzzle-craze.html
Agree With ‘The 25 Best Films’? Tell Us How You Really Feel
https://www.nytimes.com/2017/06/10/insider/agree-with-the-25-best-films-tell-us-how-you-really-feel.html
Bar Code: A New Infatuation Poised for a Puzzle Craze
http://www.nytimes.com/section/insider
Play Today’s Puzzle
http://www.nytimes.com/crosswords
Play Today’s Puzzle
http://www.nytimes.com/crosswords
At Palace, the Unbearable Being of Lines
https://www.nytimes.com/2017/06/12/fashion/palace-skateboards-british-supreme.html
A Fierce Famine Stalks Africa
https://www.nytimes.com/2017/06/12/opinion/a-fierce-famine-stalks-africa.html
The Lice Ladies of Brooklyn Go Corporate
https://www.nytimes.com/2017/06/08/nyregion/the-lice-ladies-of-brooklyn-go-corporate.html
The Revolutionary Specters of Russian Letters
https://www.nytimes.com/2017/06/12/opinion/the-revolutionary-specters-of-russian-letters.html
Meet the Musicians Who Modeled for Gucci
https://www.nytimes.com/2017/05/30/t-magazine/fashion/gucci-cruise-musicians-james-righton-lucio-corsi-francesco-bianconi.html
What We’re Reading
https://www.nytimes.com/2017/06/06/insider/what-were-reading.html
Ideas Sparked by Flow of Twitter Conversations
https://www.nytimes.com/2017/06/07/technology/personaltech/jenna-wortham-tech-twitter.html
Making Google the Censor
https://www.nytimes.com/2017/06/12/opinion/making-google-the-censor.html
What Yotam Ottolenghi Can’t Do Without
https://www.nytimes.com/2017/06/06/dining/tahini-cookies-cake-recipes.html
Will Iran Descend Into Chaos?
https://www.nytimes.com/2017/06/11/opinion/will-iran-descend-into-chaos.html
From the Subways to the Sofa: Pet Rats
https://www.nytimes.com/2017/06/08/nyregion/new-yorkers-adopting-rats-as-pets.html
The Presence of Violence at the Kabul Bureau
https://www.nytimes.com/2017/06/08/insider/first-thought-at-the-timess-kabul-bureau-we-are-under-attack.html
U.S. Cyberweapons, Used Against Iran and North Korea, Are a Disappointment Against ISIS
https://www.nytimes.com/2017/06/12/world/middleeast/isis-cyber.html
Reviving a Lost Language of Canada Through Film
https://www.nytimes.com/2017/06/11/world/americas/reviving-a-lost-language-of-canada-through-film.html
Emmanuel Macron’s Party on Track to Claim Majority in France’s Parliament
https://www.nytimes.com/2017/06/11/world/europe/france-legislative-elections-emmanuel-macron.html
Jeffrey Immelt to Retire as General Electric Chief
https://www.nytimes.com/2017/06/12/business/ge-immelt.html
Emil Michael Leaves Uber
https://www.nytimes.com/2017/06/12/technology/uber-travis-kalanick-emil-michael.html
Uber Weighs Leave of Absence for Chief Executive
https://www.nytimes.com/2017/06/11/technology/uber-holder-report.html
Charles M. Blow: The Resistance: Impeachment Anxiety
https://www.nytimes.com/2017/06/12/opinion/the-resistance-impeachment-anxiety.html
Editorial: Mr. Trump Goes After the Inspectors
https://www.nytimes.com/2017/06/12/opinion/mr-trump-goes-after-the-inspectors.html
Op-Ed Contributor: Making Google the Censor
https://www.nytimes.com/2017/06/12/opinion/making-google-the-censor.html
‘We Are Not Unusual Anymore’: 50 Years of Mixed-Race Marriage in U.S.
https://www.nytimes.com/2017/06/11/us/mixed-race-marriage-loving-decision-anniversary.html
For Flint, an 1882 Play About Bad Water Is ‘Exactly What We’re Living’
https://www.nytimes.com/2017/06/11/us/flint-water-crisis-theater-ibsen.html
Heroin Use? Juvenile Record? For Recruits, Police Forgive Past Sins
https://www.nytimes.com/2017/06/12/us/police-departments-american-history.html
Uber Weighs Leave of Absence for Chief Executive
https://www.nytimes.com/2017/06/11/technology/uber-holder-report.html
Emil Michael Leaves Uber
https://www.nytimes.com/2017/06/12/technology/uber-travis-kalanick-emil-michael.html
U.S. Cyberweapons, Used Against Iran and North Korea, Are a Disappointment Against ISIS
https://www.nytimes.com/2017/06/12/world/middleeast/isis-cyber.html
Tony Awards Briefing: ‘Dear Evan Hansen’ Wins 6 Tony Awards; Bette Midler Is Best Leading Actress
https://www.nytimes.com/2017/06/11/theater/tony-awards.html
The Best (and Worst) of the 2017 Tony Awards
https://www.nytimes.com/2017/06/12/theater/the-best-and-worst-of-the-2017-tony-awards.html
Et Tu, Delta? Shakespeare in the Park Sponsors Withdraw From Trump-Like ‘Julius Caesar’
https://www.nytimes.com/2017/06/11/arts/delta-airline-trump-public-theater-julius-caesar.html
Supreme Court Bars Favoring Mothers Over Fathers in Citizenship Case
https://www.nytimes.com/2017/06/12/us/politics/supreme-court-citizenship-ginsburg-gorsuch.html
Trump Loses Travel Ban Ruling in Appeals Court
https://www.nytimes.com/2017/06/12/us/politics/trump-travel-ban-court-of-appeals.html
Ivanka Trump Says Father Felt ‘Vindicated’ by Comey Testimony
https://www.nytimes.com/2017/06/12/us/politics/ivanka-trump-comey-donald-trump-fox-and-friends.html
On the Runway: The Tony Awards Get the Red Carpet Blues
https://www.nytimes.com/2017/06/12/fashion/tony-awards-red-carpet-trends.html
Styles Q. & A.: Virgil Abloh Projects Politics Into His Florence Fashion Show
https://www.nytimes.com/2017/06/12/fashion/mens-style/virgil-abloh-pitti-uomo-florence-italy.html
Death of a Downtown Icon: Remembering Benjamin Cho
https://www.nytimes.com/2017/06/11/fashion/death-of-a-downtown-icon-benjamin-cho.html
The 25 Best Films of the 21st Century So Far.
https://www.nytimes.com/interactive/2017/06/09/movies/the-25-best-films-of-the-21st-century.html
Review: ‘The Mummy,’ With Tom Cruise, Deserves a Quick Burial
https://www.nytimes.com/2017/06/07/movies/the-mummy-review-tom-cruise.html
Review: Mom, Dad, the Kid and the Dog. What Could Be Scarier?
https://www.nytimes.com/2017/06/08/movies/it-comes-at-night-review.html
When Opioid Addicts Find an Ally in Blue
https://www.nytimes.com/2017/06/12/nyregion/when-opioid-addicts-find-an-ally-in-blue.html
L.I.R.R. Announces Travel Options for ‘Summer of Hell’
https://www.nytimes.com/2017/06/12/nyregion/lirr-train-schedule-penn-station-repairs.html
What’s the Insurance Capital Without Aetna? Hartford May Find Out
https://www.nytimes.com/2017/06/11/nyregion/hartford-aetna-relocation.html
Keeping Score: All Should Rise for Yankees’ Gary Sanchez, Too
https://www.nytimes.com/2017/06/12/sports/baseball/all-should-rise-for-yankees-gary-sanchez-too.html
Penguins 2, Predators 0 | Pittsburgh wins series, 4-2: Pittsburgh Penguins Beat the Predators to Repeat as Stanley Cup Champions
https://www.nytimes.com/2017/06/11/sports/hockey/stanley-cup-pittsburgh-penguins-nashville-predators.html
Give Baseball Scouts a (Perfect) 80 for Tradition
https://www.nytimes.com/2017/06/12/sports/baseball/give-baseball-scouts-a-perfect-80-for-tradition.html
The Best (and Worst) of the 2017 Tony Awards
https://www.nytimes.com/2017/06/12/theater/the-best-and-worst-of-the-2017-tony-awards.html
Tony Awards Briefing: ‘Dear Evan Hansen’ Wins 6 Tony Awards; Bette Midler Is Best Leading Actress
https://www.nytimes.com/2017/06/11/theater/tony-awards.html
Et Tu, Delta? Shakespeare in the Park Sponsors Withdraw From Trump-Like ‘Julius Caesar’
https://www.nytimes.com/2017/06/11/arts/delta-airline-trump-public-theater-julius-caesar.html
On Nuclear Waste, Finland Shows U.S. How It Can Be Done
https://www.nytimes.com/2017/06/09/science/nuclear-reactor-waste-finland.html
Basics: The Liver: A ‘Blob’ That Runs the Body
https://www.nytimes.com/2017/06/12/health/liver-bodily-function.html
Trilobites: These Hungry Goats Learned to Branch Out
https://www.nytimes.com/2017/06/12/science/goats-trees-seeds-morocco.html
Edith Shiffert, a Poet Inspired by Nature and Her Life in Japan, Dies at 101
https://www.nytimes.com/2017/06/11/arts/edith-shiffert-a-poet-inspired-by-nature-and-her-life-in-japan-dies-at-101.html
Sad Day for Gotham: Adam West, Who Played Batman, Dies at 88
https://www.nytimes.com/2017/06/10/arts/sad-day-for-gotham-adam-west-who-played-batman-dies-at-88.html
Andimba Toivo ya Toivo, Namibian Independence Leader, Dies at 92
https://www.nytimes.com/2017/06/10/world/africa/andimba-toivo-ya-toivo-dead.html
Television Review: An Uneven Night for Kevin Spacey as Tonys Host
https://www.nytimes.com/2017/06/12/theater/tony-awards-review-recap-kevin-spacey.html
Oliver Stone’s ‘Putin Interviews’: Flattery, but Little Skepticism
https://www.nytimes.com/2017/06/09/arts/television/review-the-putin-interviews-showtime-oliver-stone.html
Televsion Review: ‘Claws’ on TNT: That Manicure Comes With a Murder
https://www.nytimes.com/2017/06/11/arts/television/claws-on-tnt-that-manicure-comes-with-a-murder.html
Basics: The Liver: A ‘Blob’ That Runs the Body
https://www.nytimes.com/2017/06/12/health/liver-bodily-function.html
The New Old Age: Hearing Aids at the Mall? Congress Could Make It Happen
https://www.nytimes.com/2017/06/12/health/hearing-aids-congress.html
More Than 10 Percent of World’s Population Is Obese, Study Finds
https://www.nytimes.com/2017/06/12/health/obesity-study-10-percent-globally.html
The Getaway: How Much Are Frequent Flier Miles Really Worth?
https://www.nytimes.com/2017/06/12/travel/how-much-are-frequent-flier-miles-worth.html
U.S. Islands: On a Georgia Island, a Lot of Good Food and Plenty of Nothing
https://www.nytimes.com/2017/06/06/travel/cumberland-island-georgia-national-seashore.html
Essay: How to Ease Travel Anxiety in an Era of Terror: Travel More
https://www.nytimes.com/2017/06/08/travel/easing-anxiety-travelers-global-terror.html
Nonfiction: How the Dark Web’s Dread Pirate Roberts Went Down
https://www.nytimes.com/2017/06/12/books/review/american-kingpin-nick-bilton.html
Q. and A.: Tell Us 5 Things About Your Book: A Reporter Inside the World of Jihad
https://www.nytimes.com/2017/06/11/arts/tell-us-5-things-about-your-book-inside-the-world-of-jihad.html
Fiction: Arundhati Roy’s Return to the Form That Made Her Famous
https://www.nytimes.com/2017/06/09/books/review/arundhati-roys-return-to-the-form-that-made-her-famous.html
To Understand Betsy DeVos’s Educational Views, View Her Education
https://www.nytimes.com/2017/06/10/us/politics/betsy-devos-private-schools-choice.html
David L. Kirp: Ending the Curse of Remedial Math
https://www.nytimes.com/2017/06/10/opinion/sunday/cuny-ending-the-curse-of-remedial-math.html
Success Academy Wins Round in Fight Over Preschool Oversight
https://www.nytimes.com/2017/06/09/nyregion/success-academy-preschool-appeal.html
Hungry City: At Bep Ga in Chinatown, It All Begins With Chicken
https://www.nytimes.com/2017/06/08/dining/bep-ga-review-vietnamese-restaurant-chinatown.html
City Kitchen: A Juicy Tenderloin, a Pesto Swirl
https://www.nytimes.com/2017/06/09/dining/beef-tenderloin-recipe.html
A Good Appetite: The Simplest Shortbread You Can Bake
https://www.nytimes.com/2017/06/09/dining/shortbread-cookie-recipe.html
Opinion: Is Putin Getting What He Wanted With Trump?
https://www.nytimes.com/2017/06/10/opinion/sunday/is-putin-getting-what-he-wanted-with-trump.html
Editorial: It’s the Olympics for Trump Apologists
https://www.nytimes.com/2017/06/10/opinion/sunday/its-the-olympics-for-trump-apologists.html
Frank Bruni: Can Democrats Save Themselves?
https://www.nytimes.com/2017/06/10/opinion/sunday/can-democrats-save-themselves.html
The Elusive $1,500 Rental
https://www.nytimes.com/2017/06/09/realestate/finding-an-affordable-new-york-city-apartment.html
Exclusive: Where a Rockefeller Raised His Family
https://www.nytimes.com/2017/06/09/realestate/david-rockefeller-upper-east-side-home-for-sale.html
Ask Real Estate: Creating New Windows Without Ruining the Building
https://www.nytimes.com/2017/06/10/realestate/creating-new-windows-without-ruining-the-building.html
Your Rabbi? Probably a Democrat. Your Baptist Pastor? Probably a Republican. Your Priest? Who Knows.
https://www.nytimes.com/interactive/2017/06/12/upshot/the-politics-of-americas-religious-leaders.html
The New Health Care: There’s No Magical Savings in Showing Prices to Doctors
https://www.nytimes.com/2017/06/12/upshot/theres-no-magical-savings-in-showing-prices-to-doctors.html
Economic View: Why Women Don’t See Themselves as Entrepreneurs
https://www.nytimes.com/2017/06/09/upshot/why-women-dont-see-themselves-as-entrepreneurs.html
Feature: The Long, Lonely Road of Chelsea Manning
https://www.nytimes.com/2017/06/12/magazine/the-long-lonely-road-of-chelsea-manning.html
Feature: America’s Hidden H.I.V. Epidemic
https://www.nytimes.com/2017/06/06/magazine/americas-hidden-hiv-epidemic.html
Feature: The Bounty Hunter of Wall Street
https://www.nytimes.com/2017/06/08/magazine/the-bounty-hunter-of-wall-street.html
Driven: Video Review: With New Ioniq, Hyundai Goes After Prius
https://www.nytimes.com/2017/06/08/automobiles/autoreviews/video-review-with-new-ioniq-hyundai-goes-after-prius.html
Wheels: In Portland, a Rare Outpost for Die-Hard Saab Fans
https://www.nytimes.com/2017/06/08/automobiles/saab-autos-cars.html
Bits: Electronic Setups of Driverless Cars Vulnerable to Hackers
https://www.nytimes.com/2017/06/07/technology/electronic-setups-of-driverless-cars-vulnerable-to-hackers.html
6 Designers (Straight Out of Fashion School) to Know
https://www.nytimes.com/2017/06/09/t-magazine/fashion/ben-duncan-chungin-no-matt-dyer-new-young-designers.html
Some Things T Editors Are Really Into Right Now
https://www.nytimes.com/2017/06/09/t-magazine/givenchy-audrey-hepburn-editors-picks.html
8 Male Models to Watch This Season
https://www.nytimes.com/2017/06/09/t-magazine/fashion/male-models-mens-fashion-week-masao-parris-mathieu-franke-zack-hunt.html
Bar Code: A New Infatuation Poised for a Puzzle Craze
https://www.nytimes.com/2017/06/12/insider/bar-code-a-new-infatuation-poised-for-a-puzzle-craze.html
Ahead of the Tonys, Watching Everything
https://www.nytimes.com/2017/06/10/insider/tony-awards-theater-desk.html
Agree With ‘The 25 Best Films’? Tell Us How You Really Feel
https://www.nytimes.com/2017/06/10/insider/agree-with-the-25-best-films-tell-us-how-you-really-feel.html
Homes That Sold for Around $450,000
https://www.nytimes.com/2017/06/11/realestate/homes-that-sold-for-around-450000.html
Search for Homes for Sale or Rent
http://www.nytimes.com/real-estate/find-a-home
Mortgage Calculator
https://www.nytimes.com/real-estate/mortgage-calculator

Adding in bylines

Bylines look like this:

<p class="byline">By PETER BAKER and STEVEN ERLANGER <time class="timestamp" datetime="2017-03-17" data-eastern-timestamp="12:36 PM" data-utc- timestamp="1489768575">12:36 PM ET</time></p>

So… let’s just grab the element inside of story that has the class of byline!

story = doc.find_all('article', { 'class': 'story' })

for story in story:
    headline = story.find(class_='story-heading')
    if not headline:
        continue
    print(headline.text.strip())
    link = story.find('a')
    print(link['href'])
    byline = story.find(class_='byline')
    print(byline.text)

Revised Trump Travel Ban Suffers Another Legal Setback
https://www.nytimes.com/2017/06/12/us/politics/trump-travel-ban-court-of-appeals.html
By ADAM LIPTAK 2:09 PM ET
Putin Opponent Is Arrested Amid Protests in Russia
https://www.nytimes.com/2017/06/12/world/europe/russia-aleksei-navalny-kremlin-protests.html
By NEIL MacFARQUHAR and ANDREW HIGGINS 1:26 PM ET
Why Cyberwar on ISIS Has Fallen Short of U.S. Hopes
https://www.nytimes.com/2017/06/12/world/middleeast/isis-cyber.html
By DAVID E. SANGER and ERIC SCHMITT 
Trump Holdings Are Unconstitutional, Pioneering Suit Says
https://www.nytimes.com/2017/06/12/us/trump-lawsuit-private-businesses.html
By SHARON LaFRANIERE 12:31 PM ET
Opioid Addicts Find an Ally in Blue
https://www.nytimes.com/2017/06/12/nyregion/when-opioid-addicts-find-an-ally-in-blue.html
By AL BAKER 
Addiction Drug Lacks Results, but It Has Powerful Friends
https://www.nytimes.com/2017/06/11/health/vivitrol-drug-opioid-addiction.html



---------------------------------------------------------------------------

AttributeError                            Traceback (most recent call last)

<ipython-input-13-e0ae1796bc1c> in <module>()
      9     print(link['href'])
     10     byline = story.find(class_='byline')
---> 11     print(byline.text)


AttributeError: 'NoneType' object has no attribute 'text'

So we get another one of those “missing byline” errors, yeah? Well, maybe not everything has a byline. It doesn’t mean we should skip the whole thing, let’s just skip the byline for that one.

stories = doc.find_all('article', { 'class': 'story' })

for story in stories:
    headline = story.find(class_='story-heading')
    if not headline:
        continue
    print(headline.text.strip())
    link = story.find('a')
    print(link['href'])
    byline = story.find(class_='byline')
    if byline:
        print(byline.text)

Revised Trump Travel Ban Suffers Another Legal Setback
https://www.nytimes.com/2017/06/12/us/politics/trump-travel-ban-court-of-appeals.html
By ADAM LIPTAK 2:09 PM ET
Putin Opponent Is Arrested Amid Protests in Russia
https://www.nytimes.com/2017/06/12/world/europe/russia-aleksei-navalny-kremlin-protests.html
By NEIL MacFARQUHAR and ANDREW HIGGINS 1:26 PM ET
Why Cyberwar on ISIS Has Fallen Short of U.S. Hopes
https://www.nytimes.com/2017/06/12/world/middleeast/isis-cyber.html
By DAVID E. SANGER and ERIC SCHMITT 
Trump Holdings Are Unconstitutional, Pioneering Suit Says
https://www.nytimes.com/2017/06/12/us/trump-lawsuit-private-businesses.html
By SHARON LaFRANIERE 12:31 PM ET
Opioid Addicts Find an Ally in Blue
https://www.nytimes.com/2017/06/12/nyregion/when-opioid-addicts-find-an-ally-in-blue.html
By AL BAKER 
Addiction Drug Lacks Results, but It Has Powerful Friends
https://www.nytimes.com/2017/06/11/health/vivitrol-drug-opioid-addiction.html
Role of Trump’s Lawyer Blurs Public and Private Lines
https://www.nytimes.com/2017/06/11/us/politics/trump-lawyer-marc-kasowitz.html
By REBECCA R. RUIZ and SHARON LaFRANIERE 
Trump Era, Unlike Watergate Era, Has Rival Sets of Facts
https://www.nytimes.com/2017/06/11/business/media/comey-trump-watergate.html
By JIM RUTENBERG 
Fired U.S. Attorney Says Trump Tried to Build Relationship With Him
https://www.nytimes.com/2017/06/11/us/politics/preet-bharara-trump-contacts.html
The Long, Lonely Road of Chelsea Manning
https://www.nytimes.com/2017/06/12/magazine/the-long-lonely-road-of-chelsea-manning.html
By MATTHEW SHAER 
California Today: Talking to a Tony Winner
https://www.nytimes.com/2017/06/12/us/california-today-talking-to-a-tony-winner.html
By JONAH ENGEL BROMWICH 8:30 AM ET
Listen to ‘The Daily’
https://www.nytimes.com/column/the-daily
How to Save on Summer Travel
https://www.nytimes.com/interactive/2017/05/16/travel/frugal-traveler-summer-travel-chat.html
Ways Your iPhone Will Change After Apple’s iOS 11
https://www.nytimes.com/2017/06/12/smarter-living/apple-iphone-ios11.html
By LIAM STACK 
Gargantuan Job: Dismantling a Million Tons of North Sea Oil Rigs
https://www.nytimes.com/2017/06/08/business/energy-environment/oil-north-sea-shell.html
By STANLEY REED 
The Best (and Worst) of the 2017 Tony Awards
https://www.nytimes.com/2017/06/12/theater/the-best-and-worst-of-the-2017-tony-awards.html
By ERIK PIEPENBURG and SCOTT HELLER 11:32 AM ET
‘Dear Evan Hansen’ Wins 6 Tonys; Midler Is Best Leading Actress
https://www.nytimes.com/2017/06/11/theater/tony-awards.html
Television Review: An Uneven Night for Kevin Spacey as Host
https://www.nytimes.com/2017/06/12/theater/tony-awards-review-recap-kevin-spacey.html
Fashion Review: The Tony Awards Get the Red Carpet Blues
https://www.nytimes.com/2017/06/12/fashion/tony-awards-red-carpet-trends.html
‘We Are Not Unusual Anymore’: 50 Years of Mixed-Race Marriage in U.S.
https://www.nytimes.com/2017/06/11/us/mixed-race-marriage-loving-decision-anniversary.html
By JENNIFER MEDINA 
Ahead of ‘Brexit,’ Britons Face Economic Pinch at Home
https://www.nytimes.com/2017/06/11/business/economy/britain-economy-election-brexit.html
By PETER S. GOODMAN 
For Democrats, Spotlight Turns to Race to Replace Christie
https://www.nytimes.com/2017/06/09/nyregion/democrats-new-jersey-governor-philip-murphy.html
By NICK CORASANITI 
Death Penalty in Pakistan for Social Media Blasphemy 1:45 PM ET
https://www.nytimes.com/2017/06/12/world/asia/pakistan-blasphemy-sentence.html
Theresa May Battles to Hold On as U.K. Prime Minister 12:54 PM ET
https://www.nytimes.com/2017/06/12/world/europe/uk-theresa-may-britain.html
Uber’s No. 2 Executive Leaves Firm 12:32 PM ET
https://www.nytimes.com/2017/06/12/technology/uber-travis-kalanick-emil-michael.html
Jeffrey Immelt to Retire as General Electric Chief 1:56 PM ET
https://www.nytimes.com/2017/06/12/business/ge-immelt.html
Bill Cosby Sex Assault Trial: The Defense Rests 1:07 PM ET
https://www.nytimes.com/2017/06/12/arts/television/bill-cosby-trial-day-6.html
Morning Briefing: Australia Edition
https://www.nytimes.com/newsletters/morning-briefing-australia
Morning Briefing: Asia Edition
https://www.nytimes.com/newsletters/morning-briefing-asia
Morning Briefing: Europe Edition
https://www.nytimes.com/newsletters/morning-briefing-europe
Morning Briefing
https://www.nytimes.com/newsletters/morning-briefing
Australia to Tighten Parole Laws After Deadly Siege
https://www.nytimes.com/2017/06/09/world/australia/australia-melbourne-terror-parole.html
Australian Politics Is Open to Foreign Cash, and China Has Much to Gain
https://www.nytimes.com/2017/06/06/world/australia/china-political-influence-campaign-finance.html
Australia Makes Riesling Its Own
https://www.nytimes.com/2017/06/08/dining/wine-review-australian-riesling.html
Take Our Quiz on Solving Global Warming
https://www.nytimes.com/interactive/2017/06/09/climate/drawdown-climate-solutions-quiz.html
Something Old and New at the French Open
https://www.nytimes.com/2017/06/12/sports/tennis/something-old-something-new-at-french-open.html
How a Dark Web Mastermind Went Down
https://www.nytimes.com/2017/06/12/books/review/american-kingpin-nick-bilton.html
Making Google the Censor
https://www.nytimes.com/2017/06/12/opinion/making-google-the-censor.html
By DAPHNE KELLER 
Editorial: Mr. Trump Goes After the Inspectors
https://www.nytimes.com/2017/06/12/opinion/mr-trump-goes-after-the-inspectors.html
Blow: Impeachment Anxiety
https://www.nytimes.com/2017/06/12/opinion/the-resistance-impeachment-anxiety.html
Krugman: They Don’t Need No Information
https://krugman.blogs.nytimes.com/2017/06/11/they-dont-need-no-information/
Sexual Abusers Shouldn’t Be Allowed to Run the Clock
https://www.nytimes.com/2017/06/12/opinion/sexual-abusers-shouldnt-be-allowed-to-run-the-clock.html
A Night of Terror, a Year of Racism
https://www.nytimes.com/2017/06/12/opinion/a-night-of-terror-a-year-of-racism.html
The True Meaning of ‘Idiot’
https://www.nytimes.com/2017/06/12/opinion/trump-and-the-true-meaning-of-idiot.html
Loving, 50 Years Later
https://www.nytimes.com/interactive/2017/06/12/opinion/loving-virginia-50-year-anniversary.html
Finally, Something Isn’t the Matter With Kansas
https://www.nytimes.com/2017/06/12/opinion/finally-something-isnt-the-matter-with-kansas.html
America Needs Closure on the Travel Ban
https://www.nytimes.com/2017/06/11/opinion/america-needs-closure-on-the-travel-ban.html
Follow us on Twitter »
https://twitter.com/nytopinion
Bar Code: A New Infatuation Poised for a Puzzle Craze
https://www.nytimes.com/2017/06/12/insider/bar-code-a-new-infatuation-poised-for-a-puzzle-craze.html
Agree With ‘The 25 Best Films’? Tell Us How You Really Feel
https://www.nytimes.com/2017/06/10/insider/agree-with-the-25-best-films-tell-us-how-you-really-feel.html
Bar Code: A New Infatuation Poised for a Puzzle Craze
http://www.nytimes.com/section/insider
Play Today’s Puzzle
http://www.nytimes.com/crosswords
Play Today’s Puzzle
http://www.nytimes.com/crosswords
At Palace, the Unbearable Being of Lines
https://www.nytimes.com/2017/06/12/fashion/palace-skateboards-british-supreme.html
A Fierce Famine Stalks Africa
https://www.nytimes.com/2017/06/12/opinion/a-fierce-famine-stalks-africa.html
The Lice Ladies of Brooklyn Go Corporate
https://www.nytimes.com/2017/06/08/nyregion/the-lice-ladies-of-brooklyn-go-corporate.html
The Revolutionary Specters of Russian Letters
https://www.nytimes.com/2017/06/12/opinion/the-revolutionary-specters-of-russian-letters.html
Meet the Musicians Who Modeled for Gucci
https://www.nytimes.com/2017/05/30/t-magazine/fashion/gucci-cruise-musicians-james-righton-lucio-corsi-francesco-bianconi.html
What We’re Reading
https://www.nytimes.com/2017/06/06/insider/what-were-reading.html
Ideas Sparked by Flow of Twitter Conversations
https://www.nytimes.com/2017/06/07/technology/personaltech/jenna-wortham-tech-twitter.html
Making Google the Censor
https://www.nytimes.com/2017/06/12/opinion/making-google-the-censor.html
What Yotam Ottolenghi Can’t Do Without
https://www.nytimes.com/2017/06/06/dining/tahini-cookies-cake-recipes.html
Will Iran Descend Into Chaos?
https://www.nytimes.com/2017/06/11/opinion/will-iran-descend-into-chaos.html
From the Subways to the Sofa: Pet Rats
https://www.nytimes.com/2017/06/08/nyregion/new-yorkers-adopting-rats-as-pets.html
The Presence of Violence at the Kabul Bureau
https://www.nytimes.com/2017/06/08/insider/first-thought-at-the-timess-kabul-bureau-we-are-under-attack.html
U.S. Cyberweapons, Used Against Iran and North Korea, Are a Disappointment Against ISIS
https://www.nytimes.com/2017/06/12/world/middleeast/isis-cyber.html
Reviving a Lost Language of Canada Through Film
https://www.nytimes.com/2017/06/11/world/americas/reviving-a-lost-language-of-canada-through-film.html
Emmanuel Macron’s Party on Track to Claim Majority in France’s Parliament
https://www.nytimes.com/2017/06/11/world/europe/france-legislative-elections-emmanuel-macron.html
Jeffrey Immelt to Retire as General Electric Chief
https://www.nytimes.com/2017/06/12/business/ge-immelt.html
Emil Michael Leaves Uber
https://www.nytimes.com/2017/06/12/technology/uber-travis-kalanick-emil-michael.html
Uber Weighs Leave of Absence for Chief Executive
https://www.nytimes.com/2017/06/11/technology/uber-holder-report.html
Charles M. Blow: The Resistance: Impeachment Anxiety
https://www.nytimes.com/2017/06/12/opinion/the-resistance-impeachment-anxiety.html
Editorial: Mr. Trump Goes After the Inspectors
https://www.nytimes.com/2017/06/12/opinion/mr-trump-goes-after-the-inspectors.html
Op-Ed Contributor: Making Google the Censor
https://www.nytimes.com/2017/06/12/opinion/making-google-the-censor.html
‘We Are Not Unusual Anymore’: 50 Years of Mixed-Race Marriage in U.S.
https://www.nytimes.com/2017/06/11/us/mixed-race-marriage-loving-decision-anniversary.html
For Flint, an 1882 Play About Bad Water Is ‘Exactly What We’re Living’
https://www.nytimes.com/2017/06/11/us/flint-water-crisis-theater-ibsen.html
Heroin Use? Juvenile Record? For Recruits, Police Forgive Past Sins
https://www.nytimes.com/2017/06/12/us/police-departments-american-history.html
Uber Weighs Leave of Absence for Chief Executive
https://www.nytimes.com/2017/06/11/technology/uber-holder-report.html
Emil Michael Leaves Uber
https://www.nytimes.com/2017/06/12/technology/uber-travis-kalanick-emil-michael.html
U.S. Cyberweapons, Used Against Iran and North Korea, Are a Disappointment Against ISIS
https://www.nytimes.com/2017/06/12/world/middleeast/isis-cyber.html
Tony Awards Briefing: ‘Dear Evan Hansen’ Wins 6 Tony Awards; Bette Midler Is Best Leading Actress
https://www.nytimes.com/2017/06/11/theater/tony-awards.html
The Best (and Worst) of the 2017 Tony Awards
https://www.nytimes.com/2017/06/12/theater/the-best-and-worst-of-the-2017-tony-awards.html
Et Tu, Delta? Shakespeare in the Park Sponsors Withdraw From Trump-Like ‘Julius Caesar’
https://www.nytimes.com/2017/06/11/arts/delta-airline-trump-public-theater-julius-caesar.html
Supreme Court Bars Favoring Mothers Over Fathers in Citizenship Case
https://www.nytimes.com/2017/06/12/us/politics/supreme-court-citizenship-ginsburg-gorsuch.html
Trump Loses Travel Ban Ruling in Appeals Court
https://www.nytimes.com/2017/06/12/us/politics/trump-travel-ban-court-of-appeals.html
Ivanka Trump Says Father Felt ‘Vindicated’ by Comey Testimony
https://www.nytimes.com/2017/06/12/us/politics/ivanka-trump-comey-donald-trump-fox-and-friends.html
On the Runway: The Tony Awards Get the Red Carpet Blues
https://www.nytimes.com/2017/06/12/fashion/tony-awards-red-carpet-trends.html
Styles Q. & A.: Virgil Abloh Projects Politics Into His Florence Fashion Show
https://www.nytimes.com/2017/06/12/fashion/mens-style/virgil-abloh-pitti-uomo-florence-italy.html
Death of a Downtown Icon: Remembering Benjamin Cho
https://www.nytimes.com/2017/06/11/fashion/death-of-a-downtown-icon-benjamin-cho.html
The 25 Best Films of the 21st Century So Far.
https://www.nytimes.com/interactive/2017/06/09/movies/the-25-best-films-of-the-21st-century.html
Review: ‘The Mummy,’ With Tom Cruise, Deserves a Quick Burial
https://www.nytimes.com/2017/06/07/movies/the-mummy-review-tom-cruise.html
Review: Mom, Dad, the Kid and the Dog. What Could Be Scarier?
https://www.nytimes.com/2017/06/08/movies/it-comes-at-night-review.html
When Opioid Addicts Find an Ally in Blue
https://www.nytimes.com/2017/06/12/nyregion/when-opioid-addicts-find-an-ally-in-blue.html
L.I.R.R. Announces Travel Options for ‘Summer of Hell’
https://www.nytimes.com/2017/06/12/nyregion/lirr-train-schedule-penn-station-repairs.html
What’s the Insurance Capital Without Aetna? Hartford May Find Out
https://www.nytimes.com/2017/06/11/nyregion/hartford-aetna-relocation.html
Keeping Score: All Should Rise for Yankees’ Gary Sanchez, Too
https://www.nytimes.com/2017/06/12/sports/baseball/all-should-rise-for-yankees-gary-sanchez-too.html
Penguins 2, Predators 0 | Pittsburgh wins series, 4-2: Pittsburgh Penguins Beat the Predators to Repeat as Stanley Cup Champions
https://www.nytimes.com/2017/06/11/sports/hockey/stanley-cup-pittsburgh-penguins-nashville-predators.html
Give Baseball Scouts a (Perfect) 80 for Tradition
https://www.nytimes.com/2017/06/12/sports/baseball/give-baseball-scouts-a-perfect-80-for-tradition.html
The Best (and Worst) of the 2017 Tony Awards
https://www.nytimes.com/2017/06/12/theater/the-best-and-worst-of-the-2017-tony-awards.html
Tony Awards Briefing: ‘Dear Evan Hansen’ Wins 6 Tony Awards; Bette Midler Is Best Leading Actress
https://www.nytimes.com/2017/06/11/theater/tony-awards.html
Et Tu, Delta? Shakespeare in the Park Sponsors Withdraw From Trump-Like ‘Julius Caesar’
https://www.nytimes.com/2017/06/11/arts/delta-airline-trump-public-theater-julius-caesar.html
On Nuclear Waste, Finland Shows U.S. How It Can Be Done
https://www.nytimes.com/2017/06/09/science/nuclear-reactor-waste-finland.html
Basics: The Liver: A ‘Blob’ That Runs the Body
https://www.nytimes.com/2017/06/12/health/liver-bodily-function.html
Trilobites: These Hungry Goats Learned to Branch Out
https://www.nytimes.com/2017/06/12/science/goats-trees-seeds-morocco.html
Edith Shiffert, a Poet Inspired by Nature and Her Life in Japan, Dies at 101
https://www.nytimes.com/2017/06/11/arts/edith-shiffert-a-poet-inspired-by-nature-and-her-life-in-japan-dies-at-101.html
Sad Day for Gotham: Adam West, Who Played Batman, Dies at 88
https://www.nytimes.com/2017/06/10/arts/sad-day-for-gotham-adam-west-who-played-batman-dies-at-88.html
Andimba Toivo ya Toivo, Namibian Independence Leader, Dies at 92
https://www.nytimes.com/2017/06/10/world/africa/andimba-toivo-ya-toivo-dead.html
Television Review: An Uneven Night for Kevin Spacey as Tonys Host
https://www.nytimes.com/2017/06/12/theater/tony-awards-review-recap-kevin-spacey.html
Oliver Stone’s ‘Putin Interviews’: Flattery, but Little Skepticism
https://www.nytimes.com/2017/06/09/arts/television/review-the-putin-interviews-showtime-oliver-stone.html
Televsion Review: ‘Claws’ on TNT: That Manicure Comes With a Murder
https://www.nytimes.com/2017/06/11/arts/television/claws-on-tnt-that-manicure-comes-with-a-murder.html
Basics: The Liver: A ‘Blob’ That Runs the Body
https://www.nytimes.com/2017/06/12/health/liver-bodily-function.html
The New Old Age: Hearing Aids at the Mall? Congress Could Make It Happen
https://www.nytimes.com/2017/06/12/health/hearing-aids-congress.html
More Than 10 Percent of World’s Population Is Obese, Study Finds
https://www.nytimes.com/2017/06/12/health/obesity-study-10-percent-globally.html
The Getaway: How Much Are Frequent Flier Miles Really Worth?
https://www.nytimes.com/2017/06/12/travel/how-much-are-frequent-flier-miles-worth.html
U.S. Islands: On a Georgia Island, a Lot of Good Food and Plenty of Nothing
https://www.nytimes.com/2017/06/06/travel/cumberland-island-georgia-national-seashore.html
Essay: How to Ease Travel Anxiety in an Era of Terror: Travel More
https://www.nytimes.com/2017/06/08/travel/easing-anxiety-travelers-global-terror.html
Nonfiction: How the Dark Web’s Dread Pirate Roberts Went Down
https://www.nytimes.com/2017/06/12/books/review/american-kingpin-nick-bilton.html
Q. and A.: Tell Us 5 Things About Your Book: A Reporter Inside the World of Jihad
https://www.nytimes.com/2017/06/11/arts/tell-us-5-things-about-your-book-inside-the-world-of-jihad.html
Fiction: Arundhati Roy’s Return to the Form That Made Her Famous
https://www.nytimes.com/2017/06/09/books/review/arundhati-roys-return-to-the-form-that-made-her-famous.html
To Understand Betsy DeVos’s Educational Views, View Her Education
https://www.nytimes.com/2017/06/10/us/politics/betsy-devos-private-schools-choice.html
David L. Kirp: Ending the Curse of Remedial Math
https://www.nytimes.com/2017/06/10/opinion/sunday/cuny-ending-the-curse-of-remedial-math.html
Success Academy Wins Round in Fight Over Preschool Oversight
https://www.nytimes.com/2017/06/09/nyregion/success-academy-preschool-appeal.html
Hungry City: At Bep Ga in Chinatown, It All Begins With Chicken
https://www.nytimes.com/2017/06/08/dining/bep-ga-review-vietnamese-restaurant-chinatown.html
City Kitchen: A Juicy Tenderloin, a Pesto Swirl
https://www.nytimes.com/2017/06/09/dining/beef-tenderloin-recipe.html
A Good Appetite: The Simplest Shortbread You Can Bake
https://www.nytimes.com/2017/06/09/dining/shortbread-cookie-recipe.html
Opinion: Is Putin Getting What He Wanted With Trump?
https://www.nytimes.com/2017/06/10/opinion/sunday/is-putin-getting-what-he-wanted-with-trump.html
Editorial: It’s the Olympics for Trump Apologists
https://www.nytimes.com/2017/06/10/opinion/sunday/its-the-olympics-for-trump-apologists.html
Frank Bruni: Can Democrats Save Themselves?
https://www.nytimes.com/2017/06/10/opinion/sunday/can-democrats-save-themselves.html
The Elusive $1,500 Rental
https://www.nytimes.com/2017/06/09/realestate/finding-an-affordable-new-york-city-apartment.html
Exclusive: Where a Rockefeller Raised His Family
https://www.nytimes.com/2017/06/09/realestate/david-rockefeller-upper-east-side-home-for-sale.html
Ask Real Estate: Creating New Windows Without Ruining the Building
https://www.nytimes.com/2017/06/10/realestate/creating-new-windows-without-ruining-the-building.html
Your Rabbi? Probably a Democrat. Your Baptist Pastor? Probably a Republican. Your Priest? Who Knows.
https://www.nytimes.com/interactive/2017/06/12/upshot/the-politics-of-americas-religious-leaders.html
The New Health Care: There’s No Magical Savings in Showing Prices to Doctors
https://www.nytimes.com/2017/06/12/upshot/theres-no-magical-savings-in-showing-prices-to-doctors.html
Economic View: Why Women Don’t See Themselves as Entrepreneurs
https://www.nytimes.com/2017/06/09/upshot/why-women-dont-see-themselves-as-entrepreneurs.html
Feature: The Long, Lonely Road of Chelsea Manning
https://www.nytimes.com/2017/06/12/magazine/the-long-lonely-road-of-chelsea-manning.html
Feature: America’s Hidden H.I.V. Epidemic
https://www.nytimes.com/2017/06/06/magazine/americas-hidden-hiv-epidemic.html
Feature: The Bounty Hunter of Wall Street
https://www.nytimes.com/2017/06/08/magazine/the-bounty-hunter-of-wall-street.html
Driven: Video Review: With New Ioniq, Hyundai Goes After Prius
https://www.nytimes.com/2017/06/08/automobiles/autoreviews/video-review-with-new-ioniq-hyundai-goes-after-prius.html
Wheels: In Portland, a Rare Outpost for Die-Hard Saab Fans
https://www.nytimes.com/2017/06/08/automobiles/saab-autos-cars.html
Bits: Electronic Setups of Driverless Cars Vulnerable to Hackers
https://www.nytimes.com/2017/06/07/technology/electronic-setups-of-driverless-cars-vulnerable-to-hackers.html
6 Designers (Straight Out of Fashion School) to Know
https://www.nytimes.com/2017/06/09/t-magazine/fashion/ben-duncan-chungin-no-matt-dyer-new-young-designers.html
Some Things T Editors Are Really Into Right Now
https://www.nytimes.com/2017/06/09/t-magazine/givenchy-audrey-hepburn-editors-picks.html
8 Male Models to Watch This Season
https://www.nytimes.com/2017/06/09/t-magazine/fashion/male-models-mens-fashion-week-masao-parris-mathieu-franke-zack-hunt.html
Bar Code: A New Infatuation Poised for a Puzzle Craze
https://www.nytimes.com/2017/06/12/insider/bar-code-a-new-infatuation-poised-for-a-puzzle-craze.html
Ahead of the Tonys, Watching Everything
https://www.nytimes.com/2017/06/10/insider/tony-awards-theater-desk.html
Agree With ‘The 25 Best Films’? Tell Us How You Really Feel
https://www.nytimes.com/2017/06/10/insider/agree-with-the-25-best-films-tell-us-how-you-really-feel.html
Homes That Sold for Around $450,000
https://www.nytimes.com/2017/06/11/realestate/homes-that-sold-for-around-450000.html
Compiled by C.J. HUGHES 
Search for Homes for Sale or Rent
http://www.nytimes.com/real-estate/find-a-home
Mortgage Calculator
https://www.nytimes.com/real-estate/mortgage-calculator

Looking a lot better! Now the only problem is “By LOUIS LUCERO II 1:00 PM ET” instead of having “LOUIS LUCERO II” or even better “LOUIS LUCERO II”.

Scraping - NYTimes