Tutorials
Analyze past conversations
Introduction
This tutorial is written at a beginner level, and uses cURL and the command line to make a request to the Twitter API v2 recent search endpoint.
Twitter data is used by developers, students, and researchers to study various topics. You may want to study the conversation around a topic using hashtags, set of keywords, etc. from the last few days. For example, you may want to study the sentiment of Tweets about a baseball game over the weekend. To retrieve data such as this, you will use the recent search endpoint, which provides Tweets that match a query that you submit from over the last 7 days.
In this guide, you will learn how to:
- Identify which past conversation you wish to study
- Connect and authenticate to the recent search endpoint to receive Tweets
- Analyze the data for past conversations
Before you start, you will need the following:
- An approved developer account. Sign up here.
- A developer App that is attached to a Project created in the developer portal.
A Bearer Token, which can be generated within your developer App.
Steps to consider
Step 1: Identify which past conversation you wish to study
As a first step, you need to identify the type of conversation you wish to study. For example, you may decide to study all Tweets about the climate for the last few days. In order to do this, you might be looking for Tweets with a certain hashtag or words. Once you have identified these words, you will need to convert these into a query (also referred to as rule or filter). You can define the rule to include/exclude keywords, images, etc. You can also specify certain account names to get Tweets from. Check out this guide to see which operators are available, and to learn how to build a query to get that data you need.
In this example, we will search for all Tweets with images about the hashtag #Caturday, excluding retweets. Your query will look something like this:
#caturday has:images -is:retweet
Step 2: Build your query, and make your request
Once you have decided what data you want and what operators you will use, you can connect to the recent search endpoint. The example is a good starting place. To properly authenticate this request, replace $BEARER_TOKEN with your Bearer Token. You will also need to replace $QUERY , which we'll describe next.
curl --request GET \
--url 'https://api.twitter.com/2/tweets/search/recent?query=$QUERY' \
--header 'Authorization: Bearer $BEARER_TOKEN'
Since your search query is added as a query parameter, you will need to HTTP encode it. To do so, you can copy the query that you built in step one and paste it into a tool such as meyerweb.com's URL Decoder/Encoder. If done correctly, your encoded query will look like this:
%23caturday%20has%3Aimages%20-is%3Aretweet
Once you have added your Bearer Token and encoded query, your request will look like the following. You can copy and paste this request into your command line tool and submit your request with 'Return' or 'Enter'.
curl --request GET \
--url 'https://api.twitter.com/2/tweets/search/recent?query=%2523caturday%2520has%253Aimages%2520-is%253Aretweet' \
--header 'Authorization: Bearer $BEARER_TOKEN'
And here is what the response will look like:
{
"data": [
{
"id": "1420134874049130507",
"text": "These 3 Fitness Bands Will Motivate You To Get Fit https://t.co/r0Ob33fIMx #SnGlobe\n#TransformationTuesday\n#WednesdayWisdom\n#TBT\n#FridayFeeling\n#Caturday\n#SundayFunday https://t.co/3c0NFE44LS"
},
{
"id": "1420127938490650634",
"text": "@RolfatWarwick @KarenScoda Alfred speaks: “How about sharpening your claws on it instead? It looks like the doll appearing on dark roads before the next murder in the episode where DCI #Taggart enrolls himself in a health farm 😹 - give it a good scratch to prevent bad things happening?” #Caturday https://t.co/P3y4CZPLET"
},
{
"id": "1420124471432675328",
"text": "Si Benji seneng bener dibawa keluar meskipun dia tidak bisa melihat apapun 🧡 #Caturday https://t.co/RjRyGVoUaN"
},
{
"id": "1420122787126054925",
"text": "What's a cat to do but lounge on freshly washed and folded towels.... grrrr I was only got 3 minutes. #Caturday #CatsOfTwitter #Cat https://t.co/zAPl7dxwhz"
},
{
"id": "1420121105176698889",
"text": "#CatsofTwittter #Cats #Caturday #WednesdayMotivation https://t.co/rrodb7iuWi"
},
{
"id": "1420119706216935424",
"text": "Garena Free Fire Season 39 Elite Pass leak suggest addition of scarecrow themed items https://t.co/qW4hF7ksjh #SnGlobe\n#TransformationTuesday\n#WednesdayWisdom\n#TBT\n#FridayFeeling\n#Caturday\n#SundayFunday https://t.co/zTgoXMTGNA"
},
{
"id": "1420115918059933696",
"text": "Just chillin on the couch after my morning run. ❤ \n\n#fitness #Caturday #catsofinstagram #twitch #smallstreamer #witch #run #TikTok #instagood #foryoupage #FYP https://t.co/cWFvc8MkwR"
},
{
"id": "1420114278108045312",
"text": "A relaxing day in...\nhttps://t.co/ohvoDhjLmn 🎬\n\n#CatsOfTwitter #cats #Caturday #CooLSharpee https://t.co/l8z16ZR2NL"
},
{
"id": "1420113937509625859",
"text": "Especially for cat lovers #Catsoftwitter #Caturday #Cats\n\nCrafty has designed mugs for all Cat Dads and Cat Moms 😀 enjoy \n\n👉 https://t.co/tKLy7H8kV7\n\n#MHHSBD #TuesdayMotivations #Summer #TuesdayVibe #TuesdayThoughts #UK #AD #TuesdayMorning #Handmade #Pets #Elevenseshour https://t.co/EmmwSudZqj"
},
{
"id": "1420105238229438469",
"text": "@NftSheikh There are two ways to avoid getting disappointed.\nExpect everything - or expect nothing.\n#NFT #NFTCommunity #nftcollector #nftart #NFTartist #cryptoart #nftdrop $SIGN $WAVES #NFTshare #nftgallery #Caturday\nExpecting. Available on @signArtApp \nCollect Here: https://t.co/b7riFddzJN https://t.co/a812hA7V0w"
}
],
"meta": {
"newest_id": "1420134874049130507",
"oldest_id": "1420105238229438469",
"result_count": 10,
"next_token": "b26v89c19zqg8o3fpdm6gnxhsjxxwoscerv2yg6kvkfb1"
}
}
This request will return 10 Tweets objects by default. In order to get additional Tweets that match your query, you need to paginate through the results. To do so, you will need to find the next_token within the meta object in your latest response and pass the value of that response field as the value of a newly added next_token query parameter in your next request. You will receive a new next_token value in the meta object until you've paginated through all results.
Here is an example of what a request might look like if you include the next_token from the response example above:
curl --request GET \
--url 'https://api.twitter.com/2/tweets/search/recent?next_token=b26v89c19zqg8o3fpdm6gnxhsjxxwoscerv2yg6kvkfb1&query=%2523caturday%2520has%253Aimages%2520-is%253Aretweet' \
--header 'Authorization: Bearer $BEARER_TOKEN'
You can also increase the number of Tweets that you receive per API request using the max_results query parameter with your API call.
By default, you will get back the id and text of each Tweet. However, you can customize what additional fields you want to return as part of the Tweet payload. You can include these additional fields in the payload by adding additional fields and expansions to your query. Learn more about how to do this by reviewing our guide on using fields and expansions.
Instead of making cURL calls from your command line tools, you can also use a REST client such as Postman to get data from the recent search endpoint. Our Postman collection will help you get started quickly in calling the recent search endpoint.
Step 3: Analyzing the data for past conversations
The Twitter API provides a good opportunity to study historical data. Examples of analysis that you might be interested in doing with Twitter data include:
- Mapping Tweets by location
- Sentiment analysis on Tweets about past events
- Identifying influencers on Twitter
There are various libraries available in programming languages such as Python, R, etc. that let you analyze the data obtained from Twitter API.
In Python, you can use libraries such as pandas, numpy, etc. that let you do data analysis and wrangling. You can also use libraries such as matplotlib to build visualizations from Twitter data.
In R, you can use something like Tidyverse (which is a collection of R packages for data science) for data analysis. You can use ggplot2, which is part of Tidyverse to build data visualization using Twitter data.
There are a variety of visualizations that you can build using these libraries mentioned above. Some examples of such visualizations that can be built using Twitter data include:
- Histograms for displaying Tweet frequency
- Map for displaying Tweets by Geo-Location
- Time series analysis of Trends
Check out this sample App that demonstrates how to use the recent search endpoint to analyze sentiments of your own Tweets.
Next steps
- Check out our API reference to learn more about what’s available.
- Learn more about building search queries.
- Get inspired by reading our other tutorials.
Ready to build your solution?