SocialMining started in 2017 as a research project at the City University of New York (CUNY) through the CUNY Research Scholar Program. It was developed by Tahir Mitchell and Yahdira Ixcoy under the guidance of Professor Jinzhong Niu.
The research was sparked by a 2015 study, "Dawn of the Selfie Era," which looked at Instagram photos across five major cities. It was an interesting premise, but it only covered a handful of places. We wanted to see what would happen if you took that concept and expanded it globally.
Do people in different countries express emotion differently in public photos? Are there measurable differences between weekdays and weekends? How do smiles and expressions shift across seasons or times of day?
To find out, we built an automated pipeline that pulls public geotagged photos, detects faces, estimates emotion and smile intensity, and maps the results. Rather than keeping the findings tucked inside an academic paper, the goal was to build an open dashboard so anyone can explore the data directly.
The numbers so far
2,859,725
Images Crawled
405,445
Faces Analyzed
* Not every scanned photo has a clear frontal face, so the number of analyzed faces is lower than the total images retrieved.
How it works
When an image comes through the pipeline, here is what gets measured:
Smile intensity (0 to 100): Measures mouth curvature and width relative to the rest of the face. A score of 0 represents a neutral face or frown, while higher scores reflect a wider smile with visible teeth.
Emotion: The model analyzes facial contours and estimates the dominant emotion, such as happy, neutral, sad, or surprised.
Demographics: Dlib detects frontal faces, and Face++ and DeepFace estimate age range, gender, and broad racial or ethnic groupings.
Do people just smile because of the camera?
A fair question people often ask is whether the "say cheese" effect ruins this kind of study. If people are conditioned to smile whenever a camera appears, does the data tell us anything meaningful?
Across more than 360,000 faces in our database, it turns out people do not just smile all the time:
Over 40% of faces don't smile. They score under 20 on the smile index.
More than 56% show non-happy expressions. Neutral faces make up about 34%, with the rest registering sadness, surprise, or other states. The global average smile sits around 52 out of 100.
Flickr is mostly candid. Unlike selfie-centric apps, Flickr contains a large amount of street photography, cultural events, travel photography, and crowd scenes where subjects are unposed.
Smiling is a scale, not yes or no. A polite social grin might score a 30, while a broad grin scores 85. The pipeline picks up those nuances.
Cultural norms differ. Cross-cultural psychology research shows that societies have different expectations for the camera. Some cultures encourage big smiles, while in others, a composed, neutral face is the standard.
The focus is on relative differences. The research does not attempt to diagnose what someone was secretly feeling inside. The goal is to look at statistical differences between countries, days of the week, and seasons.
How the system is built
The pipeline runs in containers on an Ubuntu server. Python handles photo retrieval from Flickr directly into memory, Dlib handles face detection, and Face++ and DeepFace run the inferences. Everything lands in MySQL, and a weekly GitHub Actions build publishes updated static data to this site.
Figure 1: How data flows from Flickr to this website.
What to expect from the data
We set up this project to be transparent and responsible. Here is what to keep in mind:
Clear limitations: These models make mistakes. Demographic predictions (especially ethnicity) are estimates made by neural networks, not official census records. Sample sizes also vary widely between popular travel spots and rural areas.
Strict privacy: No photos are ever saved to disk. When the crawler pulls an image, it loads it directly into computer memory (RAM), analyzes it, and discards it immediately. No photos, thumbnails, or user accounts are ever saved to disk or written to the database. All stored data is numeric and grouped by country.
Open methods: The source code and database structure are public on GitHub. There are no hidden weighting formulas or secret adjustments.
Tools and resources used
Flickr REST API for public geotagged photo search (not affiliated with or endorsed by SmugMug or Flickr)