Persine 🔎

Persine is an automated tool to study and reverse-engineer algorithmic recommendation systems, like YouTube videos and Amazon products.

J Soma

slides at jonathansoma.com/projects/persine

Hi, I’m Soma

Lede Program, Columbia Journalism School

js4571@columbia.edu @dangerscarf

investigate.ai littlecolumns.com ledeprogram.com

Persine

github.com/jsoma/persine

persine.readthedocs.io

Who’s to blame for a bad vaccine rollout? Algorithms!

Who’s to blame for a radicalization? Algorithms!

Who’s to blame for a racism? Algorithms!

Researching algorithms

FOIA
Logistic regression a tutorial if you’re curious
Source code analysis
Researching
Reporting

Enter: Persine

“Persine is an automated tool to study and reverse-engineer algorithmic recommendation systems, like YouTube videos and Amazon products. It has a simple interface and encourages reproducible results.”

from persine import PersonaEngine

engine = PersonaEngine()

with engine.persona() as persona:
   persona.run_batch([
      "https://www.youtube.com/watch?v=hZw23sWlyG0",
      "youtube:next_up#3",
      "youtube:homepage"
   ])
   persona.history.to_csv("history.csv")
   persona.recommendations.to_csv("recs.csv")

You only need to know three lines here. In this code, we say hey, Persine let’s open up the browser. You’re going to be a brand-new person in this browser, it’s not going to be connected to your history or your cookies or anything like that, it’s brand new, blank slate. Google or Amazon or whoever kind of has no idea who you are.

So we say hey, go watch this one YouTube video. We just pop the URL in, right there. And then click the ‘next up’ video, watch that, click the next up video, watch that. So you’re clicking ‘next up’ three times. And then at the end, visit the youtube homepage.

Pretty boring, right? Not awfully exciting.

But what happened behind the scenes is in every step Persine was tracking what YouTube was doing. It was scraping every single recommendation that YouTube showed us. The sidebar recommendations, the homepage recommendations, sponsored content, it was tracking all of that. It paid attention to what the ‘next up’ video was that YouTube suggested we watch, along with everything else on the page. So for a new user, without any history, knowing only that we’re interested in this one video, we can ask: what does YouTube show us?

And as long as we’re technical enough to open a CSV file or two, Persine can tell us.

Let’s run Persine!

Run Persine on Google Colab

What can it do?

YouTube and Amazon

YouTube, the Great Radicalizer by Zeynep Tufekci (NYT)

Does watching a politically charged video prompt YouTube to send you down a rabbit hole of radicalization?

Children’s YouTube is still churning out blood, suicide and cannibalism by K.G. Ophanides (Wired)

Do kids watch super weird videos if YouTube is left on autoplay?

YouTube’s Algorithm Accused of Facilitating Paedophile Rings by Tim Cross (Videoweek)

Does watching bikini haul videos lead you to young girls in suggestive swimsuits?

‘Fiction is outperforming reality’: how YouTube’s algorithm distorts truth by Paul Lewis (The Guardian)

Does YouTube promote conspiracy theories?

The New Zealand shooting shows how YouTube and Facebook spread hate and violent images — yet again (Washington Post)

How well does YouTube police its content, and what kind of content is policed?

And Amazon, too!

Persine also works on Amazon, although it’s not as well tested, and Amazon’s website is kind of insane. You can see what’s surfaced in search results, what’s listed as recommendations on product pages or on the Amazon homepage.