Exploring player growth via rankings page

I've had an idea for a project brewing in my head for a while. I thought I'd go the route of writing it out before diving into it, and have a little development journal/log to go along with it. I'm also fishing for feedback and ideas from other curious folk here.

I made this data visualization a few months back of level 200 characters, which summarizes the leveling for level 200 characters using the publicly available data on the website. I'd like to expand on this. I'm curious about the history of the population, and the trajectory of the average player on the server from different eras. Here are the types of questions that I'd like to answer with the data that I plan on scraping from the website (mostly a brain dump):

• How often does the ranking change at certain positions over the course of several months?
• What was the approximate size of the server over its history?
• Given the history of a player, what is the estimated time until their next level?
• Are players who started in 2020 more or less likely to stick around than players who started in 2019?
• In what ways did the 2020 summer balance patch affect the buccaneer/paladin populations?
• Has the most popular class changed over time?
• What is the effect of events on the frequency of levels?
• What is the distribution of player levels today? Last year? Two years ago?
• What is the cumulative experience gained by everyone, ever?

As a disclaimer since I am web developer -- I do not have direct access to player data. I plan on building something with publicly available data. The constraints make it more interesting anyways IMO.

With 192k players in the ranking at the time of writing, it wouldn't be polite to scrape this continuously. I plan on a budget of 2000 scrapes per week or ~1% sample of data. This should take about <10 minutes at no more than 10 requests a second.

There are many ways to sample data, but this is what I have in plan:

• Stratified sampling by class and level. Balanced data makes it easier to compare sub-populations. For each class, determine the prior distribution of levels. Bucket the distribution in a way where each bucket has the same number of pages from the rankings page, and scrape positions and levels from the page representing those buckets.
• Fixed characters in the ranking page. Take an initial sample of characters, and track their positions over time. How much do the rankings move?

Once I get this data (with timestamps), I'll dump it in a bucket as a sqlite/csv file and make it available for analysis. I'll probably make a website too once I figure out what it should look like.

Hopefully this will be interesting!