How we collect data

This dashboard is generated every six hours using the latest data within CAT Lab’s research software. The output of this dashboard is based on the last 3 days (we are limiting how often we update the dashboard for ethics reasons listed below). It uses these sources:

• snapshots every 2 minutes of two key reddit algorithms:
  • reddit’s HOT algorithm, the default popularity ranking on the site
  • reddit’s TOP algorithm
• data about posts, currently queried from the reddit API at the time of report generation

With every report, this software also publishes research-quality datasets to covid-algotracker on github, along with information about the settings, keywords, and software behind the data.

This report is based on 3 days of data up to the following snapshots:

• Last HOT snapshot: 2020-06-02 19:59:21 UTC
• Last TOP snapshot: 2020-06-02 19:55:37 UTC

How we identify COVID-19 posts

This report identified a post as related to COVID-19 if the lower-case title or the text of the submission matched any of the following English language terms (code here). Terms will be updated as the pandemic evolves.

##  [1] "coronavirus"       "covid"             "corona"           
##  [4] "wuhan"             "pandemic"          "outbreak"         
##  [7] "epidemic"          "cdc"               "social distanc"   
## [10] "self-isolate"      "isolation"         "self isolate"     
## [13] "quarantine"        "sanitizer"         "toilet paper"     
## [16] "tp"                "wipes"             "world health"     
## [19] "community spread"  "herd immunity"     "respirators"      
## [22] "shelter in place"  "lockdown"          "n95"              
## [25] "n-95"              "work from home"    "virus"            
## [28] "ventilator"        "fauci"             "stimulus"         
## [31] "flatten the curve" "chloroquine"       "hcq"              
## [34] "serological"       "immunity test"     "containment"      
## [37] "antibody test"     "stay-at-home"      "stay at home"     
## [40] "reopen"            "re-open"

How algorithms promote reddit posts over time

Reddit’s public rankings are a classic recommender system. Using information from upvotes, downvotes, post age, and maybe other data, reddit’s algorithms create a ranked list suggestions to readers. According to Christian Sandvig, feeds like this outsource human knowledge and attention to machines. Hybrid human-machine curation can be valuable and difficult during crises, as Alex Leavitt has described in extensive research on reddit.

This dashboard reviews posts that appeared anywhere in the top 100 recommendations for TOP and HOT. On average over the last 3 days, COVID-19 articles that appeared anywhere in the rankings stayed on HOT for 4.8 hours and on TOP for 13.6 hours.

HOT and TOP aren’t the only algorithms that influence what people see. For several years now, reddit has provided a personalized newsfeed algorithm to logged in users, based on the communities that people subscribe to. The company has also developed BEST and POPULAR algorithms, which can add if there’s interest.

Here’s what that trajectory looks like for one randomly-sampled post that rose and fell from prominence. In the chart, a rank position of 0 means that the article is at the very top. People typically have to scroll down to see more than 3-4 posts. A rank position of -99 means that you would have to scroll past 99 other posts to see that one. In this example, notice that the HOT rankings are more volatile than TOP:

Currently-promoted COVID-19 posts on reddit

Top communities promoted by reddit’s algorithms

Reddit’s algorithms recommended these subreddits at least 3 times over the last 3 days.

Top domains promoted by reddit’s algorithms

These domains were recommended at least 3 times by HOT or TOP over the last 3 days.

Ethics

This dashboard summarizes and archives information that is currently public on reddit’s front page. To reduce risks to communities, this dashboard does not link directly to discussions. All author information is removed from the archival datasets.

You can read more about CAT Lab’s ethics values and processes in our post on strategies for ethical, accountable online behavior research.

Don’t use this dashboard for interventions (yet)

At CAT Lab, we believe that powerful digital interventions ought to be tested to discover if they help rather than harm, and to enable public accountability. Interventions that form feedback loops with algorithms can have unpredictable side-effects, and we advise against proceeding without evaluation. To add extra friction to interventions, we are only updating the dashboard every six hours. The research data will still provide fine-grained information for people working to study and model reddit’s algorithm behavior by the minute.

We are currently talking to funders about a project we developed with reddit communities and public health experts to test public health information interventions with people and algorithms. If you have ideas for funding opportunities, please contact J. Nathan Matias at nathan.matias@cornell.edu

Questions and bugs

If you find a bug or have a question, please post it to the issues page for this project on github. Thanks!

About CAT Lab

The Citizens and Technology Lab at Cornell University works with communities to study the social impacts of digital technologies and discover effective ideas for change. CAT Lab is led by Dr. J. Nathan Matias, an assistant professor in the department of Communication.

Working alongside communities and volunteers, we discover practical knowledge that also contributes to science, holds companies accountable, and is guided by the people most affected. Communities bring their problems, deep knowledge, and desire for change. We bring expertise in scientific research and a software platform for coordinating citizen behavioral science.

Over a dozen communities with tens of millions of people have worked with CAT Lab since 2016 on reddit, Wikipedia, and Twitter. We have tested practical ways to prevent harassment, fight misinformation, broaden inclusion, manage civic discourse, and protect freedom of expression. Our discoveries have directly influenced community practice, corporate policies, and government discussions worldwide. These industry-independent findings are regularly published by the world’s top scientific journals in the social sciences and computer science.

References and further reading

• Leavitt, A., & Robinson, J. J. (2017). Upvote My News: The Practices of Peer Information Aggregation for Breaking News on reddit.com. Proceedings of the ACM on Human-Computer Interaction, 1(CSCW), 1-18.
• Leavitt, A., & Robinson, J. J. (2017, February). The role of information visibility in network gatekeeping: Information aggregation on reddit during crisis events. In Proceedings of the 2017 ACM conference on computer supported cooperative work and social computing (pp. 1246-1261).
• Larson, H. J. (2018). The biggest pandemic risk? Viral misinformation. Nature, 562(7726), 309-310.
• Matias, J. N. (2019). Preventing harassment and increasing group participation through social norms in 2,190 online science discussions. Proceedings of the National Academy of Sciences, 116(20), 9785-9789.
• Matias, J. N. (pre-print, 2018). Nudging Algorithms by Influencing Human Behavior. Effects of Encouraging Fact-Checking on News Algorithms. https://osf.io/m98b6/
• Matias, J. N., & Mou, M. (2018, April). CivilServant: Community-led experiments in platform governance. In Proceedings of the 2018 CHI conference on human factors in computing systems (pp. 1-13).
• Vaughan, E., & Tinker, T. (2009). Effective health risk communication about pandemic influenza for vulnerable populations. American Journal of Public Health, 99(S2), S324-S332.

License

The Citizens and Technology Lab makes this report and associated code available under a Creative Commons Attribution-NonCommercial-NoDerivatives 4.0 International Public License.