Unchecked: The architecture of disinformation
Misinformation and disinformation thrive in today’s technology landscape, and arguably present the greatest threat to modern society. Information architecture – the practice of designing and managing digital spaces – has an opportunity to intervene. This podcast looks at disinformation from an information architecture perspective, and considers ways to expand the practice of IA to address this new reality.
•••
What is Information Architecture? Information architecture is the practice of designing virtual structures – the shape and form of online spaces and digital products. When you click on a navigation menu or follow the steps in a process, you're experiencing the information architecture of a web site or digital product.
•••
What is disinformation? Understanding disinformation is the purpose of this podcast. We are trying to figure out exactly what it is and what it means. If information architecture is the practice of designing virtual spaces, then disinformation is something that can occupy that space to disrupt the user's experience. Alternatively, it is a way of manipulating the space (like flooding it with irrelevant facts) to achieve an end unrelated to the space's original intention.
Unchecked: The architecture of disinformation
Social Networks and Disinformation with Renée DiResta (S02 E03)
Use Left/Right to seek, Home/End to jump to start or end. Hold shift to jump forward or backward.
SYNOPSIS
Renée DiResta, associate research professor at Georgetown University’s McCourt School of Public Policy and author of Invisible Rulers, joins Rachel and Dan to talk about how influencers, algorithms and crowds work together to decide what feels real and true online. She explains how monetization programs reward provocation, how opaque curation fuels stories of shadow banning, and how her MySky project gives users more control over their own feeds and more insight into how they work. Rachel describes the lens of Show Me the Money, and Dan introduces the lens of Authenticity.
NOTE: GreenEarth is now MySky
Throughout the episode Renée refers to her user-adjustable social media feed project with Jonathan Stray as "GreenEarth". Since the interview was recorded, they changed the project name to MySky.
INTERVIEW WITH RENÉE DIRESTA
- Renée DiResta
- Renée's book: Invisible Rulers: The People Who Turn Lies into Reality
- MySky (project with Jonathan Stray for bridging-based, user-adjustable feed for Bluesky)
- “It’s not social media anymore, it’s parasocial media” (essay by danah boyd, cited by Renée)
- Facebook “Trending Topics” controversy (the 2016 Gizmodo report on Facebook’s human curators)
- For Expertise to Matter, Nonpartisan Institutions Need New Communications Strategies (Carnegie Endowment paper by Rachel Kleinfeld and Renée DiResta)
LENSES
Show Me the Money
Systems are shaped by visible structures like navigation and taxonomy, and also by invisible ones, especially the incentive and monetization rules underneath them. Those rules decide what content gets made, surfaced and rewarded just as much as any content model does.
- How do the system’s monetization rules or incentive structures act as an information model, shaping what content people see and how it flows?
- What does the system count, and are people gaming it? If so, why?
- Who benefits from the incentives in the system: the business, creators, sellers, advertisers or users?
Authenticity
Systems reward authenticity differently. Some expect participants to show up as their true selves, and others reward a curated or performed version of themselves. When trusted experts don’t offer authenticity, people go looking for it elsewhere, which creates space for voices that only perform it.
- What does the system expect of participants when it comes to being their true selves?
- Does the system reward honest self-representation, or does it reward a curated persona?
- How does the system tie contributions to identity, and when is anonymity beneficial or harmful?
_____________________________________________________
Personnel
- Dan Brown, Host
- Rachel Price, Host
Music
- Turtle Up Fool, by Elliot
_____________________________________________________
Unchecked is a production of Curious Squid
Curious Squid is a digital design consulting firm specializing in information architecture, user experience, and product design
There's a saying by Charlie Munger, show me the incentives and I'll show you the outcome, right? If you have a monetization program where you are compensated based on engagement, which is what X's was, you had to have a certain number of blue check followers, right? Which meant, of course, by that point, just subscribers, people who were paying X. That was where the revenue share was going to come from. And you had certain numbers of engagements from the blue checks in the replies. Well, all of a sudden you're really incentivized to be kind of a provocative asshole who just works to get their base riled up because every time they leave one of those comments, you get another couple cents or whatever it is, you know.
SarahYou're listening to Unchecked, the podcast about the architecture of disinformation with Dan Brown and Rachel Price.
DanSo I've been reading this book. I finished a book.
RachelYou finished a book? Dan.
DanYou're joking, but I it is very hard for me to finish a book these days.
RachelOh, I didn't know. Yeah, I didn't mean to be insensitive.
DanI read, it's it's fine. I am aware of this. I read very slowly, and I also have limited time during the day when I can read, and I'm also constantly fighting against a short attention span. And my dog, who is constantly bothering me to go for a walk. But I read a book called Yesteryear, and I think I recommended it to you. Trevor Burrus, Jr.
RachelYou did. I am on a 16-week wait at my local library. Just saying, use your local libraries, folks.
DanAgain, I would love to, but I don't read fast enough. So I often bump up against due dates. I do listen to a lot of audiobooks, which suits me and my the time I have much better. But anyway, I did read yesteryear. I even wondered about getting the author on because the premise of the book is that a social media influencer who lives this kind of trad wife lifestyle on Instagram is suddenly thrust back into 1855 and has to figure out why she's there and how she ended up there. And the story ends up being not just her life in 1855, but sort of the events that led up to her becoming a trad wife influencer. The reason why I finished it so fast, it was like one of the first page turners I've read in a long time. I was like genuinely excited about where it was going. I don't read that guy.
RachelI have been holding on to that joke for almost two years.
DanI will not read books by people with the same name as me.
RachelContinue your story about yesteryear, though. I'm sorry.
DanAaron Powell So I genuinely enjoyed it. And since I don't follow Tradwives on Instagram, and in fact I have since, influenced in part by you, have cut off social media almost altogether. I've erased, I don't longer on Instagram or Threads or Blue Sky. I still peek into Facebook on occasion because of neighborhood stuff, which is very frustrating. But I'm thinking about getting rid of that too. And so the confluence of all these things is what made me super excited to talk to Rene D'Aresta today. Renee wrote a book called Invisible Rulers. It's about more than just social media, but it talks about the kind of entire ecosystem that we have been immersed in, I guess, for the last 10 years with social media and influencers and a very divisive society and things like that.
RachelI was trying to look back at like the very first notes I made, because usually we open by asking, like, why on earth did we bring so and so on to the podcast? And I was trying to find the very first notes I left about Renee, and I'm struggling, but I remember being kind of giddy when I was like, oh my God, Renee DeResta. Do you think we could get Renee De Resta and these notes here? And my first note was really consider contacting Renee DeResta, but I don't know if she'll say yes because she's been on podcast with Prince Harry. I am not as uh I might be as interesting as Prince Harry, but I don't know. That's really putting it in the world.
DanWell, now she she has now confirmed, yes, that you are definitely as interesting as Prince Harry.
RachelWe got the signal. Okay, great. But she has been a very outspoken scholar and person and mother about misinformation in her own words. Well, not exactly her words, but she's like, you know, I basically I live on the internet, my life is on the internet. And she's done a lot of really incredible, and I think kind of vulnerable thinking and talking about the reality of misinformation with this wonderfully well-informed kind of academic view balanced by a wonderfully well-informed, pragmatic reality view of what it's like to be a consumer, a recipient of misinformation, a mother in a world full of misinformation, a human who likes to hang out on the internet in a world of misinformation. And so I think for me, you didn't ask me why we brought Renee on the show, but that is why we really wanted Renee to join us. And it was such a great conversation.
DanOh, yeah.
RachelI just want to like go talk about it now.
DanLet's do it. Okay. She studies adversarial abuse online, ranging from state actors running influence operations to spammers and scammers to issues related to child safety. In October 2024, she joined the Georgetown University McCourt School of Public Policy as an associate research professor. Prior to that, she was the technical research manager at the Stanford Internet Observatory. Renee, thank you so much for joining us.
RenéeThank you for having me.
DanYou know, the last time I wore these headphones, I was listening to your book, Invisible Rulers. Oh, that's awesome. Thank you. Yeah, which was very well read. I you found a great narrator for it.
RenéeShe was great. Yep. She's a professional. I I can't do that.
DanYeah. So for those who have not read it, maybe you can give us the elevator pitch of the main message of Invisible Rulers. What's the book about?
RenéeYeah, it's about influence and propaganda. I wanted to write a book sort of updating manufacturing consent, which sounds very ambitious. And the publishers were like, nobody wants to read that. Um so I wrote it, but I actually wanted to write about influencers specifically and the role that they play in the information ecosystem and the unique power that they have as being incredibly persuasive figures that everybody feels like they kind of know. You know, my friend Dana Boyd wrote this essay that I loved about how it's not social media anymore, it's parasocial media. And how increasingly what we're seeing online is not content put out by our friends at all. It's content put out by these people that we follow who are these sort of hybrid entertainers, but they're just kind of like us, maybe a little prettier, maybe they go on better trips, you know, maybe in the political realm they say the thing that you can't say, right? Is a big sort of meme, but it's real. And I wanted to write about the role that they play as propagandists specifically. And I don't use that term as a pejorative, I use it as information with an agenda intended to persuade and move people. And the fact that now on social media we're increasingly surrounded by that, by all of these people constantly trying to persuade us to do things, whether it's to buy something or to follow a candidate.
DanThat makes sense. That's great. And you also describe some other aspects of the ecosystem. And I wonder if you could just give us the quick summary of besides influencers, you also describe the algorithm and the crowd. Can you talk about how they all fit together?
RenéeYep. So the influencer was for me the focus of the book, the kind of main character, if you will. But the way that I see influencers as being part of a system uh alongside algorithms and crowds. And you can't have one without the other. So the influencer needs a following. I use the term crowd to refer to the sort of fandoms online because sometimes they're not really fandoms in the traditional sense. They're just people who follow this person. Sometimes they're not even following on a platform like X. They might just be there aligned around a trend for a moment in a fleeting way, and then they kind of dissipate and go on to the next trend. But the online crowd is incredibly powerful. It is the thing that signals through engagements like likes or shares or retweets that something is worth paying attention to. So you have this relationship between the influencer and the audience that is incredibly potent. And as I mentioned, they're signaling things to an algorithm. That's the third piece. So the algorithm sits as a sort of mediating function between the humans in this interaction. And it is deciding, oh, you got enough likes for me to push your video out to more people. In the first six seconds, you had enough people not scroll away. So let me go show that to some more people. So, in a very unexpected way that a lot of people don't think about, the algorithm really is serving as a third entity in this relationship. So I wanted to write about how influencers, algorithms, and crowds together form the system of persuasion, form the system of deciding what is real, deciding what is true. And that's what the book focuses on. What does it mean when we are active participants in this ourselves, even if we're not necessarily thinking about it in those terms?
DanRight. I like that you distinguish crowds from just regular fans because the crowd gives the influencer its power, but is also, in a sense, holds the influencer accountable. And if the influencer doesn't keep giving the kinds of content that the crowd expects, the crowd can very much turn on the influencer too.
RenéeYes. And that is a thing that you particularly see in political influencers. It's called audience capture, where, you know, maybe somebody starts off their content making career or hobby producing content about music, right? They're just out there, they're talking about music, they go, they go on their live, they kind of riff, they they have oftentimes a very real back and forth with the audience. That's something I think is underappreciated, just how much if you're on a stream, if you're watching a live, you're engaging with the person directly. It's not like broadcast media where you're sitting there watching the TV or reading the article. Maybe it makes you mad, but you can't say anything directly back. With this relationship, with this culture of commenting, of streaming, of going live, you can actually say it right back. And so you do see the influencers very cognizant. Uh, if they say something that's gonna alienate their fans, that is going to maybe be a bridge too far, move them in the wrong direction, that costs them views. It costs them engagement, which costs them further views in the future because the algorithm decides, hey, maybe this person isn't like the hotness, maybe we're not gonna push them out anymore. Maybe this video isn't that great. We're not gonna share it with more people. So they lose both engagement and potentially money, revenue, right? Uh, brand deals. So there's a really complex interaction that happens there. And again, maybe you start talking about music, but then you make a video talking about the COVID vaccine. And all of a sudden, you get more engagement than you've ever had in your life. All of a sudden, you become a COVID vaccine content creator, right? Because you're incentivized to do that. And that's literally exactly what we saw during COVID. So I write a little bit about that relationship between the influencer and the crowd from the social standpoint, too.
DanOne of the themes of your work is decentralization. Can you talk to us about what does that mean in the context of the modern information landscape? I mean, what the promise of the internet, certainly when I got started in the 90s, was massive decentralization. And I feel like that's changing a little bit.
RenéeYeah, yeah, yeah. No, I we're probably about the same age. I remember that too, you know, just going and finding like random websites at a GeoCities webpage, you know, there's and you could like join these specific communities, but you were still kind of isolated, right? You built your page, you did your thing. Maybe you were in like the music kind of subsection of GeoCities and you could kind of scroll by number if I recall. They even had a little map you could see at the time, where you could kind of like hop to other people's pages. But discoverability was a real challenge, right? It was very hard to find good stuff. You could click into a million websites and it would be garbage. This is why search engines became so powerful, why Google became such a juggernaut. But when you had this problem of discoverability, one of the solutions to it is centralization and algorithmic curation. And that's what you started to see on social media actually was this period, this kind of heyday where the great advance of social media was that all of your friends were on it too. This was like, you know, Friendster, if you remember back in the day or MySpace. But what the social media platforms that became the quote unquote winners did was they also created feeds where they would curate for you the best of your friends' posts. And that was the era of kind of mass centralization. Everybody on three platforms. If you were not on those three platforms, if you got kicked off of those three platforms, you know, you were like unpersoned, right? It was a huge deal. The trend now, since around 2018, is actually a little bit more back to the decentralization era. And it's not exactly the same. There are still these algorithmic curators and things that are shaping feeds, that are doing a lot of this filtering out among the proliferation of content what is best for you. It's very personalized now, but you're starting to see actually different sites that people are going to. So in 2018, conservatives were dissatisfied with some of the policies on social media. So you saw the rise of the right-wing platforms like Parlor, like Gab. Eventually, a couple years later, you would have Truth Social after January 6th. On the left, when Elon bought Twitter, you saw a similar exodus. People went to Mastodon, they went to Threads, they went to Blue Sky. And what was interesting about two of those three, about Threads and Blue Sky, is they are actually technically decentralized at an infrastructure level, if you will, in that they are protocol-based platforms, which means that anyone can spin up a server, right? Anyone can spin up a particular instance of Mastodon. You can fork Blue Sky. And, you know, there's a great community called Black Sky, catering to the Black audience on, you know, that felt that Twitter was no longer for them. So you can make these spaces. We now have the infrastructure and tools to do it. And so you're starting to see a little bit more again in both the protocol-based platforms that actually allow for decentralization, in addition to what I described, the sort of social decentralization.
RachelKind of at the top of this, you talked about like the crowds relationship with the algorithm or just recognizing the algorithm as one of the entities in this relationship. And then looking at the reaction to maybe too much centralization and how we kind of rebounded a little bit more to decentralization. Advanced users of these systems have probably always been aware of the algorithm, but I'm going to guess most mainstream users of systems for a long time didn't think about the algorithm. Did something happen, or was there a moment in time when the algorithm became a known entity?
RenéeYeah, this is a really interesting thing because I would argue that a lot of it was actually when algorithmic curation became politicized. And what I mean by that was that there was this moment, Facebook had this situation happen. I call it trending topics gate. I refer to it in the book. Trending topics, you might think, just means a thing that a lot of people are talking about. But it turns out that that's not always what trending topics means. So Facebook back in the day actually did have human curators that were trying to keep random spam from trending, right? Because sometimes clickbait was really excellent at getting a lot of people clicking on it. And so the algorithm would be like, oh, everybody wants to know about how, for example, the Pope endorsed Donald Trump. Now, this is a thing that never happened. These were Macedonian teenagers that were just very, very good at gaming engagement, getting people to click. You see this all the time now, particularly in political content, but you can see it in celebrity content also, where people just make BS, right? And the thing that happens with that is the algorithm pushes it out to more people and it trends. So Facebook decided that its editorial team would curate the trends so that if it was spam, they would filter it back out. Now, unfortunately, what happened was that a lot of the spam targeted right-wing political users. This was around the time of the first Trump campaign. And it was targeting specifically older, less savvy participants in the Facebook community who would click through. This was the era of fake news. That was the term that came to be given to this stuff. So when it became known that Facebook was doing this curation, because one of the members of the team left and sort of had a whistleblower story at Gizmoto announcing that this was being done, people felt very deceived. Glenn Beck posted a ton about it. They got letters from senators saying, you know, you have to come in and explain why you're censoring conservative viewpoints. It became a very large story. It was a massive story. And Mark Zuckerberg had to do an apology tour. Glenn Beck and others visited Facebook, right? They had to explain why they had done this because it was reframed not as anti-spam or anti-manipulation practice, but it was very much framed around political viewpoint censorship. And that became a wildly effective narrative. And once that happened, they began to realize that they could apply that same narrative to a bunch of other platforms, a bunch of other types of narratives. And that became something where the idea of shadow banning began to emerge. And I remember being on Twitter in maybe 2018 and just having conversations with people that who were absolutely convinced that Twitter was shadow banning them because they were conservatives and they had heard that word, right? Their media was talking about it constantly. It had become a very effective story, funny enough, for their media to get engagement. And in that ecosystem, I would talk to people who would say it on Twitter and I would say, like, why do you think you're shadow banned? Because these were random accounts. These are not influencers. These are people with maybe 25 followers. And they would say, Because my friends don't see all of my posts. So they had this combination of hearing that viewpoint-based censorship was a thing and that the platforms were out to get them, right? That was that was a narrative that was being told to them. They were aware of these stories like Facebook's trending topics gate, an anti-spam effort that was, you know, egregiously sort of twisted into something it was not. And then they had this experience personally of my friends don't see all of my posts, not realizing that that's because the algorithmic curators are choosing what posts are good. And sometimes your posts are not. And that's the reality of it, right? This happens to me too. I post on threads, not everybody sees everything. Sometimes you're like, wow, that one really hit. And then sometimes you've got something that you think is fantastic and nobody engages at all, right? Yeah. And if you are aware of it, if you understand it as like the algorithmic gods are smiling on this one and not on that one. And maybe my popular friend hit the retweet button on this one, and that's why it got that initial lift, and then the algorithm pushed it out to sell more people. Like, I understand what's happening, but I could also just as easily scream that I'm being censored or shadow banned, you know? Yeah, you can't disprove shadow banning. You can't disprove it. Right, exactly. So this is where I wrote I read a lot about the need for more transparency around these things. I want to be clear, I do think that platforms have a right to create and curate the type of environment they want for their users. I actually am very happy with most of the decentralization, honestly, because it gives users places to go where they can feel that they are in a place that is made for them with their rules. And I would like them to have more direct control at a personal level, right? Where I can change my own sliders myself to say, I don't want to see this BS fake trend. I want to see these more stuff from friends and family. But as it stands right now, because users don't have that control and because there is no transparency, these stories really proliferate. And the combination of not understanding what quote unquote the algorithm does and how your engagement with it shapes, you know, your your experience and your and your friends' experiences, it leads to a lot of resentment and misinterpretation.
DanSo would you say algorithms are here to stay?
RenéeOh, for sure. So I actually have a project. It is in beta now. You can see it on Blue Sky. The reason it's on Blue Sky is because, again, AT proto, open architecture, we can just build, right? We don't have to ask permission. We actually have a project. It's called Green Earth. And if you search for Green Earth on Substack, you'll find the blog about it. Green Earth is a feed that gives users that control, right? We actually wanted to go and try to build something. This is me and Jonathan Stray at Berkeley, our kind of co-PIs on this project. And we decided we were just going to build a recommender system that users could use. And the thing that is different about it is you actually have sliders where you can make exactly those changes. You can say, I want to move my slider all the way to the left and only see stuff from friends. I want to move my slider all the way to the right and only see like random news articles that are being pushed to me. Because, you know, whatever you use Blue Sky for, maybe you have a lot of friends on there, you want the friend experience, maybe you don't, you just want it as another information source. It gives you a lot more leverage. And one of the reasons why we can do these kinds of things now is actually because of large language models, right? It is much, much easier to classify content, to ask a model, what is this post about? And then the downstream effect for users is you can engage directly with your feed creator and say, I want to see more pictures of cats. I want a feed that is almost exclusively cats. And you can communicate that in a natural language way. You can just say exactly that sentence, I want to see more cats. And then LLMs can handle that essentially curation and identification function. It's more complicated than that. You can read about the mechanics on the blog. I don't want to bore your listeners. But the technology now actually lets us do that. And there's no reason why large platforms can't do this. It's just that we are piloting it for Blue Sky because we can build it and just immediately deploy it there. We don't have to ask for permission.
DanSo let me say that back to you. Green Earth allows users to curate the content that they see by curating the algorithm. Yep. And by moving sliders, feeding it prompts to tell the algorithm the kinds of content it would prefer to see.
RenéeYeah, yeah. So one thing I really like about Blue Sky is, you know, look, I understand it has a very particular political valence and that that alienates some people. I absolutely get it. But you can change that experience for yourself by filtering out some of those, you know, some of those topics you might not want to be immersed in, right? You can actually do that if you want to. Now, the thing that's been nice about Blue Sky, even before we launched our feed, which again, as I said, is it's in beta. We're doing a bigger kind of release and announcement push in the next um couple weeks. So this is your sneak preview here. But the feeds experience prior, I really liked it. Like you can subscribe even before we got to the part where you could have LLM-enabled, much more personalized experiences. One of the things that was great about Blue Sky was that you could subscribe to a bunch of different feeds. And as you moved through them, just by kind of swiping from one to the next, you would see your the posts in your news feed, so to speak, change. And then I could see, like, hey, you know what? I really don't want to be immersed in politics this morning. I want to post about my garden. Right. I have a garden and I'm gonna go to the gardening feed and I'm gonna post. And I know that the only people who really care about this post are the gardening feed people. I in turn can just scroll the gardening feed and see nothing but this is what you know, the vegetable somebody grew. And it's kind of nice, right? It means that I have that control, I have that agency. I can go and I can do a thing, make my experience tangibly better. In that morning, I just don't feel like seeing politics. Maybe there's a cat feed, there's an art feed, there's a lot of these like identity-based community feeds, people who really care about seeing content that they consider to be representative of an identity that they may have. And I think this is great. Again, it's it's not for me necessarily. Those feeds aren't made for me, but they exist so that people who want them can have that experience.
DanAnd now have a million questions. Okay, so one of the things that we've thought about with respect to disinformation is that disinformation is, if not caused, exacerbated by, let's call them, media bubbles, right? The sort of idea that I live in a space and I don't see anything else. I was convinced the 2024 election was going to go in a completely different direction because of my the bubble that I was in. And disinformation may be exacerbated by that bubble because I'm not getting a wide variety of sources. What I'm hearing about from you about sort of curated algorithms is that I might create an even stronger bubble for myself.
RenéeYes, this is a really interesting point. So I'm glad you brought it up. Green Earth started, and at its core is what is called bridging-based ranking or bridging-based curation. There are a bunch of different papers on this. I again, I don't want to go too deep in the weeds here, but the idea of bridging is that if you believe that you should be able to give users more agency and more transparency, which I do, and I've I've emphasized this for the last 10 years. And we can talk specifically about disinformation and moderation on centralized platforms and happy to have that conversation. But in an ideal world, right, you are going to have people who move into identity-based or hard partisan spaces. That is a reality. And that you'd be surprised, it's fewer than you might think, right? Hard partisans who are completely unwilling to engage content, you know, from outside of their bubble. It's actually a much smaller percentage, but they're very loud. They're very noisy. You hear about them constantly. This is like, because for an influencer, that's what you have to be, right? You have to be performing that identity, feeding that niche at all times. You can't be nuanced. But again, if we were to curate away from that, the idea of bridging-based content, bridging-based feeds, is that rather than ranking content based on engagement as a default, you can rank according to a bunch of other types of metrics, right? Is this something that, and again, um, large language models are actually quite good at assessing this? I can talk about community notes also if you want to, because it's another area where you where bridging-based ratings are kind of core to the idea. You can surface something that is a hot topic, right? That is an area of disagreement, but without being disagreeable. So I can offer you my opinions on like algebra in San Francisco public schools, which is a thing I used to rant about on Twitter all the time, right? And I might be a little bit more caustic about it than a much more polite person. So maybe my commentary on it would be downranked because of my highly belligerent tone when I talk about that topic. But other people who are better at communicating in non-offensive ways, their content would be surfaced. And that's where you have that opportunity to sort of surface that issue of how should we be thinking about math education for our kids. And maybe one person's tone is really intended for an activist community to galvanize them, to make them do a thing. But the other is actually out there just explaining the issue, presenting the topic, offering you a way to think about this in the world. And maybe that is the thing that a feed should be emphasizing and amplifying. And if you want to go and engage around it and be an activist around it or whatever else, maybe you have an absolutely passionate opinion about it, you can absolutely do that. You can put that out there and an engagement-based feed, that kind of thing will surface. But in a feed that's based more around bridging, you'll see content that is presented a little bit more in a nuanced way, with a little bit more of a, again, disagreement without being disagreeable kind of tone. So Green Earth, really, the first feed that we launched, we have a couple of them so people can engage. Again, you can see this happen in your own feed. You can see what changes when you like move from one feed to the next. We launched a few different feeds, but with the bridging-based feed, it is actually this idea of engaging with content and debates with people who might also want to engage in content and debates without it being, you know, the libs of TikTok model of pointing at somebody and screaming and attacking them for holding an opinion.
RachelYeah, I think we've come to accept, maybe tacitly accept, that most algorithmic feeds on most of the products we use are optimizing for money, which is like engagement and then clicks and then money. Yep. And it sounds like this kind of feed you're talking about, the one you're working on, is optimizing for something very different.
RenéeYeah. And I think that again, there's there's room for both. I mean, the point should be I really do believe user choice actually matters. I think that otherwise all of the power is concentrated in the hands of the centralized platforms, right? And that's where I do believe that community-based, by community-based, I mean like sort of user, user community. Or again, one of the things that we see on Blue Sky is actually community moderation, also, where there's labelers and volunteers participate in the labeling process where they decide if you subscribe to this labeler, let's say you are a member of a marginalized identity, right? And you don't want to see certain types of posts, there are actually now labelers that allow warning labels or interstitials or even a complete excavation from the feed to shape your experience. And you, a member of the community, can subscribe to that labeler. Whereas I, who might not be a member of the community and these posts don't really affect me very much, don't subscribe. And we have a different experience then because again, it's not a centralized moderator making a call. It is a much more personalized, community-based approach of the community has decided that this type of language or this type of content or this type of thing is not suitable for them, even if a centralized moderator might argue that for free expression purposes, this stuff has to be, you know, or should be allowed to stay up. And that's where you see, again, this question of can you give users better tools to make their experience more amenable to them, more suitable for them?
DanThe intervention you're describing is at the algorithm point of your triangle. Are there interventions on the influencer and crowd side that you can imagine?
RenéeYes, totally. So for the influencer, it's really the economic motivation. I mean, there is just no way to get around that. So I'll point to X as my sort of canonical example here, but I can give an example from YouTube also. When X began monetizing the feed, right? When you could turn your account on essentially for monetization. Now, I do think that revenue sharing is nice, right? These are people who are they're keeping people on the platform, right? Why not compensate creators? But anytime, you know, there's a saying by Charlie Munger, show me the incentives and I'll show you the outcome, right? If you have a monetization program where you are compensated based on engagement, which is what X's was, you had to have a certain number of blue check followers, right? Which meant, of course, by that point, just subscribers, people who were paying X. That was where the revenue share was going to come from. And you had certain numbers of engagements from the blue checks in the replies. Well, all of a sudden you're really incentivized to be kind of a provocative asshole who just works to get their base riled up because every time they leave one of those comments, you get another couple cents or whatever it is, you know. So you did see Nikita Beer, who was the head of product over at X at the time, really working to deal with some of the negative externalities this caused. For example, there was a proliferation of American political commentators who were not in the US at all, right? They're like in Nigeria. And all of a sudden, there's this question of like, why are we letting Nigerian profiteers rile up American audiences purely for money? Like, let's be absolutely candid about what's happening here. I have a report that's going to come out that documents the extent to which this happens on Facebook now, just using AI. There are literally Bangladeshi pages that are putting out content constantly about data centers, which is a real issue that Americans are very divided and heated about right now. But there are these people in Bangladesh who are literally like noodle merchants, you know, who just are also on here posting screenshots of the amount, you know, of their earnings from Facebook because Facebook pays them directly. Back in the olden days, when I was talking about trending topics, the way the Macedonian teenagers made money was they created Flickbait and then they monetized the websites off platform. Now you're just monetizing from Facebook directly. Like it's surreal, actually, what the incentives have created. Now, I think that Americans, you know, and people who are directly impacted should be able to fight about data centers or whatever it is. I think this is an absolutely valid use for social media. I think that the problem is when you've created an incentive for outside participants to rile up, you know, and and create false perceptions of majority opinion or create false perceptions of minority opinion. Even that is where you start to have problems. YouTube has the same issue, right? You have um, I had an influencer reach out to me. The story came out in the New York Times. She's a content creator, she posted something charming about her anniversary, right? They made a video, it went viral. All of a sudden, scammers in Pakistan, right? These people happen to be overseas also, began making videos about her and her husband, about how, you know, they were secretly not really happy. I maybe one had an affair. I don't remember the specifics of the video. Maybe he was like cheating on her, but they were doing it because the algorithm had, you know, she had trended. And so people were searching for her, searching for her content. And this incentivized these hangers on to make content about her because it would be returned in the keyword process. And that's the sort of thing you see. So the question for, you know, how do you deal with influencers? It's really like what motivation does your economic model create? And is it the one that you want? And if it's not the one you want, if you have these externalities, are you doing the moderation work to prevent this kind of manipulation and harassment from happening? And the answer right now is no. I get so amped up about this. I really hate the economic incentive speech.
DanYeah, I get that. What about crowds? Do you think there's something we can do about crowds?
RenéeSo that I see as sort of an education process, right? Like my why do you think you're a shadow banned conversations? Um I saw that as like not trying to be provocative, but just this sort of like fascination with, you know, you have 20 followers. Why would the platform bother shadow banning you? You've not done anything, really.
RachelIt's a it's a very strange kind of Because it's psychologically an easier explanation.
RenéeIt is a hundred percent, yes. Yes. And that's where I think making people understand it's, you know, media literacy is a kind of a loaded term with a little bit of controversy. Not, and I don't mean political controversy from the right, though there's that also. I actually mean even on the left, this question of what does media literacy mean? Is it teaching people to recognize sources? You know, I would argue that algorithmic literacy is a um, here is how this works. Let me just explain to you how it works. We try to do this. The reason, again, that I like the I promise I'm not just shamelessly plugging my project here. But the thing I like with the green earth sliders is like you can just see it. There it is. You do this thing, this change happens, boom, right? We we don't have to write a whole piece about like in a recommender system, this is how posts are filtered. And I think it was Meta that put out when it was trying to actually deal with these allegations of shadow banning and and topical bias and so on and so forth, they made a very nice, I think it was an animation where you could see here is the universe of posts. And then first we filter on this, and then we filter on that, and then we filter on this other thing, and then it goes through a couple of different classifiers where this, that, and the other thing are decided. Is it a what's called a quote unquote borderline post? Like those actually are the ones that might not get pushed out as often. And it really tried to kind of break it down for users, but of course, Meta just kind of like put it up on their blog. Whereas I think if this had been something that, you know, Hank Green does a very good job with this, making content just to explain to people how does this work, smarter every day, did a few series on this, right? Just trying to explain to people what is the algorithmic process behind it and how should you think about that.
DanYeah, the whole impetus for Rachel's and my conversations here is we tend to put a lot of pressure on the crowd or consumers of content to be smarter or more aware or less ignorant about how they're being manipulated. And I think that's kind of unfair.
RenéeYeah, I think again, there's that sort of basic level of awareness, right? Let me explain to you how this works, just so that you understand when you see something, why you might be seeing it. We also try now to surface that in the post, right? Like when you look at a feed, and you can do this on Blue Sky, when you look at a post, instead of it just saying, like, you might be seeing this because you're female between 18 and 49 living in the United States, which tells you nothing, right? Nothing. We do try to get into here are the breakdowns in the weighting that went into this post. This is oh, interesting. And that, and that I think again, our feed does kind of come with like a look, guys, we're doing research on this, like absolutely transparently. All these other companies are doing research on you too. We're actually gonna tell you that we are very interested in how people engage with this. Do you ever move beyond the defaults? Do you ever change anything? Do people click that button? Do they actually want to know? We're very interested in those questions as researchers, right? This is, of course, what the platforms are doing in mass. Every time you hover, they are gathering information about that, right? To refine what they serve you. We want to understand it a little bit more from an open science standpoint. But this question of getting back to what you should expect the users to know and understand, I think that the ethics are actually quite clear, right? That if you are profiting off of a user base, you have an obligation to make them understand what they're there for, what they're doing, what you know, what you are doing with them and their data, right? Make making that transaction a little bit more legible. The other thing I will say, and this comes up a lot in the context of work I do on generative AI and manipulation, you can't tell anymore. There is nothing. I think three, three, four years ago, I remember doing some of the very, very, very first work on generative adversarial network-based persona creation, right? So prior to diffusion models that made incredibly realistic images of people, you would have just these faces and we would be able to detect them actually quite quickly. And so I remember doing a project with NPR where I actually didn't even really think it was all that interesting of fine, but we found this network of like thousands of fake accounts on LinkedIn. And this reporter was like, we want to do a PSA, we want to explain to people how to know if fake people are requesting them on LinkedIn. And it was like, okay, cool. So we we did this, they made this beautiful interactive where you could see like the teeth are gonna be a little bit off. The ears often blend into the hairline, the earrings are asymmetrical, glasses just blend right into the face. So we we sort of listed out all of the tells. And then that became a thing that people would want to do PSAs on for a while. List out the tells, even in the early days of diffusion models. The fingers, sometimes there are six, the numbers, the they're not real Roman numeral characters or any known human alphabet characters for that matter. So that has ended really candidly, right? Like I can't tell a lot of the time. For me, when I get asked to do an authentication on a video or something, there's like groups of us that work collaboratively on it with models, with human verification, with was anybody at this place at this time? It's a whole rubric of content credentials, provenance research. But the average person cannot tell at this point. And I can't either. And so that's the that's the reality that we're in now. It's very telling. So it really is like up to the platforms, I think, to decide what is my ethical obligation here, what is the public demand of me? That's the other piece. The public does have agency here in the in the standpoint from the standpoint of like, you know, you can complain about it. You can make clear that you don't think this is acceptable. And I think people need to be doing that more.
DanI really like the idea of being very transparent about how the algorithm arrived at prioritizing this content. It's almost like no one really knew that they had to pay attention to their blood pressure until we started going to the doctor and they told us what our blood pressure was. Those numbers now sort of led to a need to be literate, at least partially, in what these things mean. So we can pay attention to them too.
RenéeYeah, I think that's a great example, right? You don't want to be inundated with it. I feel like I'm very sensitive because this is what I, you know, I spend my my for the last 12 years, I think that's all I've done. You know, I see things in my feed sometimes and I'm like, all right, they changed something. You know, if I go over to X, I'm like, something is different here. Threads is very interesting because Threads really rewards a commenter culture, right? The algorithm is very different than on many other platforms where you will often see posts because one of your friends has engaged or commented on it. And they consider that to be incredibly high signal. That is very, very heavily weighted, much more heavily than recency. So you might see something that's three days old, but your friend commented, and that's why all of a sudden it's in your feed. And helping people understand what that means, like, okay, that means that this platform is really not gonna be very effective for breaking news in a lot of ways. You're gonna have to do a little bit of work to figure out what is new in terms of how things trend on there. It's just a different experience. You might not see it in your feed, you might have to go to the trend directly. I don't think most people think in those terms. I think they open their feed and they just want to have an entertaining experience. You know, you've got 20 minutes to kill, you're gonna read some threads, you're gonna read some tweets, you're gonna watch a video on YouTube. Speaking of fandoms, a BTS is one of the most and has been for the last 20 years, like those, you know, army can make things trend. And I had a, I have a daughter who's nine who loves K-pop, right? And I and so for the last maybe year, she watches YouTube and it lands on my account. And so I open up my YouTube and it is just clip after clip after clip after clip. And so once you start engaging with something, man, are you gonna see more of it? Yeah. And if you want to see this illustrated, just engage with like one K-pop video. Your whole feed will change. So I think when people become more aware that what they're seeing is very much personalized to them and to their behavior, and that it is not what the majority of America or their friends are thinking about or talking about. It also sort of shows you how you can form a false perception of what issues are really hot or what really matters.
RachelYeah, I feel like a lot of this is about lifting the veil and showing you where you are. Like, are you in a town hall right now? And I'm obviously speaking metaphor, but like, are you in a third place hanging out with your friends, or are you in the, you know, town square and there's stump speeches happening all around you? Are you actually in the mall? Right. And like you're just trying to and I feel like this is such a thing that's become so blended in social media spaces, and there are not strong cues for where you are and what's happening because there's so many variations of where you are and what's happening in every scroll of the feed. I feel like the human brain can only, you know, at some point most people just kind of stop wondering and just kind of start accepting what's whatever's happening in front of them.
RenéeNo, it's very true. I see it, again, since I I'll I'll use threads as an example. Like it does really reward that comment or culture. Once you comment, though, you are it like you have really given a very strong signal of engagement there. I'm very involved in the vaccine conversation. I've been a pro-vaccine activist for over a decade now, and like a very direct one. And that means that again, because this is sort of like the incentive story, I am incentivized to comment on those posts because that's how people see my content, right? That that is the the sort of perverse transaction on that platform. I have to comment on things in order for people to see my stuff. Otherwise, the algorithm will not push my stuff out. This is absolutely like transactional, but that is how they have built their community and it is what it is. And what they push me constantly is anti-vaxxers asking if they should vaccinate their kids. And I'm always like, man, I can't leave a constructive comment on this. So I gotta go find something else to comment on, otherwise, I'm just gonna be contributing to like rage bait, you know? And then sometimes you're like, well, I mean, this is where the feed is, this is what's this is where the conversation is. Like, I maybe I should just go with it, right? So I think once you are consciously aware of what it's putting in front of you, and this is from an from a, you know, and I'm not an influencer, I'm not a creator, I I don't get paid. None of my none of my socials are monetized. I get paid for writing or for research, you know, I have a salary at an academic institution. But that means that I have to think very carefully about okay, what is the value of engaging in this way at this moment? Right. And is it an emotional response where I'm like, man, I really want to tell this person exactly what I think? Or is it something where I should like not do that, like try to go find something more positive to do with my next 20 minutes?
RachelSo you're telling us that you are human is what I'm curious.
RenéeNo, I'm like, I look, I've been on the internet since middle school, right? And in like chat rooms and um, you know, BBSs and and uh, you know, I too know how to troll, right? But it's again, it's always that question of what do I want to do in this moment? And I think this question of authenticity comes up quite a lot because I think people actually do like to see that you're human, right? They, this is something where I I actually really believe, and I've written about this, you know, we wrote a whole paper about it with Carnegie Endowment, my colleague Rachel Kleinfeld and I, we have another one coming out in September, like a very, very detailed paper on this, basically making the argument that academics and institutions and sort of quote unquote experts are often too precious about this stuff. They're very afraid to engage authentically and honestly. They don't know what that looks like, they don't know how to do it. I remember giving a talk at a social media lab, right? At a social media research lab once. And I got this very interesting question from the lab head, who's maybe 10 years older than me, who asked, How do I schedule my time on social media? He was like, I see you post. How do you set your time? How do you think about when you go and do that? And I thought it was such a fascinating question because I was like, I am always online. Like I just talk, right? You know, do I regret it sometimes?
DanLike, yes, you know, schedule the time when I'm not online. I need to schedule that. Right, right.
RenéeI mean, for me, I'm like, I'm also like a night owl, so I'll you know, one in the morning sometimes where I'm like, oh man, people are gonna tell that I'm still up at 132, like reading my socials or whatever. But that's absolutely something that happens. And I've just chosen. To be like, well, I really there's really very little gap between me in real life and me on the socials, to be totally honest. And I've just chosen to do it that way. But not everybody feels comfortable like that.
DanI think academics have this double-edged sword because they do need to be out there more. But the ones that I've seen who are out there end up either being accused of being an AI-generated thing, or they're accused of sort of selling out, or their legitimacy is questioned. There's this sort of tension between the influencer persona and the expert or academic persona. And that tension makes it really hard to have a foot in both of those places.
RenéeSo again, if I'm replying or commenting or talking about hobbies or interests or whatever, there's very little cost to that. Besides, you know, I remember once I wrote a Substack post about how the Epstein files didn't prove pizza game, right? I just wanted to get this because they had really been picked up as like, oh, wayfair was real. It was trafficking children and all these conspiracy theories that people were tying to like four words, four uses of the word pizza in the Epstein files. And so I wrote a post about it on Substack, and I had this woman get very, very mad at me and my mentions, pet O PR, this that, the other. You know, they some people like again, I'm public on the internet, people come for you sometimes. It's just part of the job. But this particular one, I had maybe two or three interactions with her, got fed up and muted her, right? Sort of got her out of my out of my field of vision. She went and she looked on social media for every time I had used a curse word and she aggregated them all and sent them to my dean. And I was like, you know, so I'm sorry, what? Oh yeah. Oh, people will just like absolutely, right? Because because they want it to be a liability for you, right? Like if you say the word like, you know, the B word or like bullshit or whatever. Literally, she took every curse word and she sent it to my dean. That is anonymously cute, actually, when you said that. It was so funny because I'm like, I'm like in my mid-40s now. I was like, you're gonna yell at me for using like mean words. Come on, come on. But it was very funny to me because it was like, again, these were just casual interactions. And all of a sudden, I was being accused of like, not accused, but like just, I mean, she obviously did this just to be malicious, but it was like impropriety. I have not conducted myself, you know, in this power academic way. And I was like, look, there's a difference between would I use these words in a lecture? No. Would I write them in an article? Maybe I actually do on my substack right in exactly that tone. Because I feel like there's a a difference in, you know, I remember hearing a lecture by actually it was the creator of 4chan, it was Chris Poole, way back in the day in Silicon Valley. I was a venture capitalist when I first moved out there. So I had a junior VC role. It was like 2011. And he he said something that I've never forgotten, which is it's not who you share with, it's who you share as. And he was giving this talk specifically about the value of anonymity, right? And why people want to have the ability to share as someone that is not necessarily tied to their named, you know, identity. And it was sort of fascinating to me because I also thought, okay, well, then you do have this interesting gap, right? Where you're you're doing something over here anonymously. And I absolutely support anonymous speech, but it does then create that incentive for people to try to like unmask you, right? Or figure out who you are. And then that becomes a whole story. So this question of how should quote unquote experts or institutionalists or academics or people who have deep domain knowledge communicate, I'm very careful when I'm talking about my areas of expertise, right? If I am putting something out there about my work, right, or about something where I am speaking as an expert in the field, I weigh my language very, very carefully. Like I will spend quite a long time actually trying to figure out how to frame something to make sure I'm not being misunderstood. And I think that that's a responsibility I have, because otherwise people could pick up that post and say, well, she says this, right? And I don't, I don't like that, but that's the responsibility that I think I have when I'm weighing in on is this a Russian influence operation? Is this the whole bunch of different topics? But if I'm just like casually commenting about news or things that are going by, it's that interesting balance between how can I speak in the language of the internet without sounding stilted, or just talk as me because I don't spend all that much time thinking about it. It's just an off-the-cuff comment sometimes, knowing that that this is sometimes the cost of doing it. And so you're not rewarded, you know, I will never be tenured. I have a different relationship with academia than the average person because I came in from tech, I came in from the outside. But people who work their way up through PhD programs and things wouldn't want to jeopardize it by, you know, they're they don't get credit from their institutions for being popular on social media. They get credit for writing papers. And that's a very, very big divide. And so the incentives again there are for certain types of engagement rather than maybe what in the olden days we would have called like public intellectual or kind of casual informed commentator content.
DanWe're getting close to the end. So we're gonna end with our typical question. Even though this has not been as depressing an interview as many of the others that we have had, we always like to end on we're in a very complicated place in terms of the information ecosystem. You're doing some work to try and untangle some of that, but what are some of the other things that give you hope about the future?
RenéeYeah, so it's an interesting question. There are really interesting things that we can do now with technology. And I spend a lot of time personally in that area. I do think for all of the manipulative downsides of LLMs, all of the challenges, they actually do open up some interesting options for making the internet better or creating more, you know, giving users agency in a way that I could only like yell about five years ago, and now we can build it, we can go do it. So there's that. I know that that people have very polarized opinions on AI in general, but I think it's different when you're just pumping out slop or using it for manipulative kind of content generation versus, you know, what you what you actually can do with them is uh there's some really, really powerful things. On the social front, I have a hard time with optimism there, I will say. Um just because that requires the better angels of influential people to decide to change their behavior. I saw a video last night of uh Zoram Mamdani, mayor of New York. You know, his politics are not mine, but I like him quite a bit as a communicator. And he bikes around the city, bikes around New York, and he wasn't wearing a helmet. And this video that made that landed in my Instagram feed was him kind of watching all of these comments come pouring in, like, Mr. Mayor, you're not wearing your helmet. I don't know if you saw this video, you're not wearing your helmet, you're not wearing your helmet. And he just said, like, you're right, I have to do better. And I was like, God, it's such a simple sentence. Imagine if they said it more often, you know. Right. And I was like, ah, this is like, this is why I think he's so good at what he does. Again, independent of my opinions of his politics. Like, I just really like that model of accountability. And I think that it has to be incentivized by people modeling it. The comments were so positive. Again, just because people were like, wow, what a refreshing thing to see from a politician. And that's the kind of stuff where I would just like to see more of that. And this was something very small and niche, like wearing a bike helmet, but it has to come from like the tone, the tenor has to come from those people who have power making that change. And they need to see that feedback from the crowd, but they also need to do it themselves. So I realize I didn't answer your question about hope. It's frustrating to me that as, you know, again, as a technologist, but I do look at tech as something that has a lot of potential to change systems. But it's the human piece that is really the the hang up at this point.
DanYeah. Renee, that was great. Thank you so much for joining us. I really appreciate it.
RenéeThank you. Thanks for having me.
RachelSo that conversation with Renee was a lot packed into not very much time.
DanYes. And we could have kept going.
RachelWe could have kept going for days. Yeah. So, Dan, I'm going to put you on the spot. What are you left thinking about?
DanI want to reflect on the fact that for a lot of the interviews that we've done, we're sort of talking about a system, and you and I need to sort of apply a lot of our expertise to kind of look deeply into the system to try and understand what's going on there. We've spoken to people where I felt like it was more meta. We were getting kind of more theory stuff. And this interview, we were looking like directly at it. We were looking directly into the belly of the beast, so to speak. And Renee has spent so much time and so much of her academic effort, her effort, like trying to understand these kinds of systems. And it was really, really great to just sort of talk about these things as directly as we ever have.
RachelYes. I think there's often several layers of practice between where the quote-unquote misinformation experts sit and think and where the reality of how misinformation works is. You know, there's like a there's a big chasm there. And one of the things I love about talking with Renee is that she is really using the systems that she's studying and also thinking about the reality of how businesses work and how customers are kept and how people actually want to use the internet, not how they hypothetically or theoretically use products. And so all of that really influences the pragmatism with which she approaches the subject. And also, obviously, I think one thing I love about her is when you look at a lot of the stuff she writes about, it's all very structurally oriented, like looking at the structures in place that lead to these situations or that could mitigate these situations. So clearly, that is one reason I really love Renee.
DanShe applied a systems thinking lens to these systems, right? And so we got to see that in action.
RachelYeah.
DanDo you want to go first? Do you have a good lens for us?
RachelI have a great lens.
DanI shouldn't even ask.
RachelYeah, why did you ask it that way? As if, like, no, I have a bad lens.
DanWould you like to share your excellent lens with us?
RachelOkay. My lens is show me the money.
DanOh, god. Good one.
RachelRight? That's what I thought. No. Um okay. So show me the money. This asks us to wonder how does the system's design or structure actually reveal the incentive structure of the system or the business as a whole, right? Like systems often are operating in service to a business. So I guess this lens is kind of maybe specifically looking at that. But the idea here is like nobody designs a taxonomy or a navigation system for which accounts should be paid to keep users engaged. Like no one's like crafting that taxonomy, but it still has the same effect as like the visible system design, right? Incentives decide what content people see and when and how it flows. And so this lens asks us to examine like how are the monetization rules or the incentive structures acting as kind of information models or content models. And one reason I really like this lens is because actually Dan and I were just talking about this off-mic is that information architects are usually brought in to do some IA on the visible structure of a product. Right. But I'm usually actually much more interested in the invisible structures or the things users aren't seeing. And I think the economic structure underneath a system is doing just as much work as visible structural decisions in a UI. Of course. And so I want to go hang out there. And so this lens asks us to go hang out there.
DanAaron Powell Hang out in the space that is even less visible than information architecture maybe normally is that talks about the underlying structure of the system that creates a set of incentives, which is the mechanisms by which businesses earn money. Did I get that right?
RachelAaron Powell Yeah. So in our conversation with Renee, she talked about how X's blue check kind of produces these incentives, right? To be, I think in her words she said kind of a provocative asshole.
DanYes.
RachelBecause outrage in the replies equals more paid engagement, which equals more revenue share, right? Yes. So you have this financial incentive to be an asshole. So this is a business rule, right? I'm guessing there was no design decision or visible IA decision that was logged anywhere that was like, hey, being an asshole is a good thing, right? But so this business rule that says being an asshole makes more money is functioning as like a waiting rule. And it's functioning as a lever that determines what content is getting showed and how and how it's flowing. That's doing the same job as like a style guide or a content model. Yep. But it's so far upstream. Right. But it's still IA. It's still structuring the information a certain way.
DanVery much so. It's funny, I was I pulled something different out. It's connected to what you were saying. But there are behaviors that are incented. I mean, I guess it's all about making money, but at the end, part of the incentive is not just that the business gets what it wants, but that the individual also gets what they want in this completely artificial incentive structure that we've created. She was talking about how she gets drawn into vaccine-related conversations. And the more provocative she can be, the more likes and traffic and eyeballs and stuff she gets. And that sort of filters in through favorites or going viral or things like that, which yes, benefits the business, but also taps into a desire that we all have to be whatever recognized or acknowledged. Does this lens accommodate a broader view than just the incentives the business sets up to benefit itself?
RachelYeah. We have a lot of products that are like two-sided marketplaces. You might have revenue for the business, you might have revenue for influencers, you might have revenue in the case of Etsy, like you might have revenue for the people selling stuff on the platform, you might have revenue for advertisers that is influenced here. So I think it's really about chasing down like what incentives are happening outside of the visible structure, and how is that actually providing some pretty foundational structure to what's going on? Yes. In the same way you can talk about like shipping your org chart, uh that's not necessarily I mean, uh if you go upstream enough, that is about incentives. But it's this idea of like there are things outside of traditional engineering and product and design decisions that are absolutely influencing the IA of the experience your end customers are having.
DanIs there something that you recall from our conversation or that you can think of now, just to put you on the spot, that would be a clue about incentives that people can look at a structure or a system and sort of go, aha, there must be something going on behind the scenes here.
RachelAaron Powell That's a good question because this ties into a different idea ahead for a lens that I think is actually like a facet of this. So if you can look at a system and see like what might be gamed here and why. So in social products, we pretty much know that engagement, right, or likes or favorites or reshare, like anything that gets someone to kind of upvote your post is a thing that can be gamed. And the question is why would you game it? Sometimes, maybe rarely, there's a non-monetary reason to game a thing, but usually it's gonna have something to do, like there's some personal benefit to gaming that. And I think that's a really strong clue. Like if there's a thing to be gamed, what is that thing actually doing? Why would a human want to game it? So, in a different example, I'm thinking about how Meta, when meta employees were told that AI usage was going to be tracked and part of their performance review, like how much they were using AI was going to determine their performance ranking, right? Okay, why do we care about how it affects your performance? Because your performance ranking affects how much you get paid, whether you get a bonus, what stocks you get, and often whether or not you get laid off. And so this engineer at Meta, I think it was an engineer, made this informal, unblessed leaderboard of AI usage of individuals at the company. So you could see who was using the most token. And now we come to the idea of token maxing, right? Like, why would you game how many tokens you're using? You could come in cold to that leaderboard and be like, okay, this thing is telling me how many tokens that people are using. Why would someone game the number of tokens they're using? If you pull on that thread, in that case, it's another monetary like survival incentive. It comes down to money. And so I think to me, there's probably other signals, but I immediately go to like, what is the thing we're counting? And are people gaming it? And if so, why are they gaming it?
DanAaron Powell Just to say this out loud, one of my takeaways from the conversation with Renee is that she spends a lot of time looking at social networks. And I wonder whether how artificial social networks are. I mean, they're very much part of our reality these days, but if you just turn them all off, the world would keep spimping. Yep, we'd be fine. I think we would be just fine.
RachelMaybe even better.
DanTrevor Burrus, Jr.: So the question is so definitely better. So we've created these weird little artificial petri dishes for ourselves where we participate and start to take things too seriously. But it is kind of an experiment too, because we can learn things about how systems and incentives work inside those petri dishes that we can then apply to let's call them less artificial systems too.
RachelYeah. So the other thing that I've been thinking about is in these mainstream social media networks, there's like one metric that you follow, and that's engagement, which is like likes and shares and whatever. But what Renee talked about was this really good example that contrasts that with like a disclosed incentive layer rather than a hidden one, which is the green earth tool that she's been working on, where she talks about like scoring content on dimensions other than just raw engagement. Because engagement can be gamed, and it actually doesn't really tell you that much about why a thing is getting engagement. But with the green earth example, you know, she was talking about a user could kind of move those levers around. And there's all these attributes of content rather than just engagement. I think in that example, like the disclosed, transparent, manipulatable ranking algorithm that Green Earth uses is kind of an attempt to document what like that incentive layer and and disclose it and even kind of diversify it a bit and make it about more than one thing. So I think like there can be incentive layers that aren't inherently evil and about money, right? But most social platforms, the incentive layer is money, and that's like the kind of status quo we've gotten used to.
DanRight. And even if you user feel a different pull, a different incentive, ultimately that's going to be tied to dollars for the business. I'm glad you also brought this up because there was sort of an overarching theme to Renee's potential solutions for some of these problems, which it involves more transparency, right? Kind of a greater exposure of how an algorithm works. What are the incentives behind the business, right? Exposing some of those things so the people participating it have a sense of what is controlling what they're seeing on the screen or what they're drawn to do. It would have been easy for me to choose that as the lens, but I also felt like Renee talked about that so explicitly and so much and has an example of this through Green Earth that we didn't need to necessarily bring it up here.
RachelAaron Powell So you're saying my lens is redundant, Dan?
DanNo, I'm saying your lens is great because it builds on the foundation of transparency. Yeah. Okay. What was your lens? Aaron Powell So in that spirit, I was trying to think of something that was trying to build on top of that. And there was one thing that she said was almost a throwaway, but it comes back to authenticity. And I think what is understanding the system's perspective on authenticity. And this maybe feels like a deep cut, but I also really liked as I started thinking about it, I kind of started to snowball in this idea that systems reward or incent authenticity differently. Some systems like people to be very true to themselves, and some systems don't have that expectation. And I think it can tell us a lot about a system as to where the authenticity of its participants sit. Where I really started snowballing was looking at the influencer part of the ecosystem that Renee created for us. That in some ways, those influencers are a reaction to academics or experts, people who speak with a level of authority, but lack authenticity and therefore essentially created a space for influencers to emerge because there is a craving, a desire for authenticity. And I'd never thought about it in this way before, but now reflecting on Renee's conversation, I was like, authenticity here is lacking in our experts, and therefore we want to see it somewhere else. Oh my God, I love this lens. Aaron Powell So now influencers have emerged who I don't think they're authentic, but they pretend at authenticity in a way that creates a stronger bond with the crowd, as uh Rune refers to them.
RachelAaron Powell My only feedback on this lens so far is that I would have called it a victim of your own success.
DanOh, it's we can still we're work shopping. We're work shopping the names.
RachelUh no, I this is such a good observation.
DanAaron Powell And this has come up when we talk to Laura and Amelia about nutrition and wellness, right? There's this there's a disconnect. We were focused on the path that that information needs to follow. To get from actual scientific research to choices that I make for what food do I put on the table, right? But both of them acknowledge the sort of insertion of these influencers and the utility that those influencers bring, presumably, is that they're bringing things down to earth. They're more realistic, they're more authentic, in quotes, than what the academics have positioned.
RachelOkay. So give me some concrete application of how you want me to use this lens.
DanI mean, I think in the most simplest terms, when we create spaces for people to participate in, what is our expectation of them being their true selves? Does the system need them to be completely honest with who they are? Or does the system like Instagram have an expectation, reward the idea that maybe they are not entirely honest with who they are?
RachelThis makes me think of, oh, I can't remember it. For a minute, I used this social media app that you didn't post anything when you wanted to, like everyone posted at the same time, and it would take two photos. I take one photo from your selfie camera and one photo from your front camera. And the idea was that you would just share with your friends exactly what was going on in your life at that moment in time that was like the random moment of the day the app chose to like capture what everyone was doing so that they're not composed. Right. And there were a lot of limitations on the design of how you could use it to compose anything. Like you had a very short time frame, you couldn't take multiple photos. So in order for something to be composed, you'd have to be truly like an advanced expert at like speed composition. Right. But there are very different constraints there than there are on Instagram, which was is extremely curated if you want it to be.
DanRight. Yes.
RachelOkay.
DanAnd often it's. Yeah. And then to be very frank with you, I don't know because I've never thought about authenticity in the work that I do. But I do feel like that's the whole point of this, is like we're trying to find ways to look at our work differently that we haven't really had to before. I don't work on social networks. I hope I never have to work on a social network. But I do feel like I work on systems where people participating have a choice as to how they represent themselves and how they contribute to that space.
RachelSo this doesn't have to just be about social media. I can think of a couple examples. In workplaces where I've worked, you might have Google Docs, right? Where you need to be logged in to see this document. If you leave a comment, it's associated with you and your name. Like you cannot hide behind an anonymous comment on a Google Doc. However, if you are in Figma in a Fig Jam and you're not logged in, which is common in a lot of workplaces, you know, because not everyone is going to have a Figma seat, you can leave comments as anonymous rabbit or like happy frog, whatever. And there are situations in which it is beneficial in team collaboration where you do want to be able to leave anonymous comments, you know, for different reasons. And so this is interesting because this is these are both like not social media, not all about user-generated content, not these traditional like Reddit versus Facebook question, but two different systems in which we collaborate and communicate in a business setting, one of which you can hide behind an avatar and a fake name, the other one is literally tied to your like professional profile in your company.
DanYes, part of your permanent record, so to speak. Yeah, that's really cool. How does it come back to disinformation? Again, I Who knows? Who knows? But when I look at the systems that perpetuate that are essential for perpetuating disinformation or that contribute to disinformation, I sense that authenticity or the lack thereof is such an important ingredient there that we don't spend enough time thinking about the tension that authentic we talk about authenticity a lot, but and how we crave it or whatever, but we don't, I don't feel like we've spent sufficient time really unpacking what it means, what role should it play in our system, and what tensions does it create.
RachelAbsolutely. I think, yeah, it's come up a lot peripherally. We have not really sat and wrestled with it. We should find someone who wants to talk to us about that.
DanYeah. Maybe Renee will do that again.
RachelRenee, come back.
DanBack. And that was Unchecked. Thanks so much for listening. We really want to hear from you. If you've got ideas for topics or guests or stories, drop us a line at unchecked at curious-squid.com. If you made use of the lenses that we described today in your practice, we want to hear about that too. Hey, check the show notes for any of the links that we talked about today. And it would really mean a lot to us if you shared this episode with a friend and rated and reviewed us on your favorite podcast platform. Thank you.