MapleLegends Online Users over Time

I built a small tool to plot the number of online users over time. The maplelegends API is pinged every 15 minutes and inserted into a database that powers this chart.

You can play with the chart here. An up-to-date copy of the data export can be found here.



There was an issue with the server in June that made me curious enough to start collecting data. The server let people log in but not to log out, which caused the number of users to spike to ~2800 before it was restarted.



It's interesting to look at the history. It's a testament to how well this server is run with the new online population that hovers around 2k users at it's peak.

My only regret is that I don't have any data earlier than 2020-06-10, but at least this doesn't require any maintenance on my part now that it's set up.

Spoiler: programming details

The overview of gathering and displaying the data looks something like this:


[source]

Pinging the MapleLegends API is straightforward. This is the same one that powers the !online command and the online box on the home page (I think).

Code:

async function get_user_stats() {
  let online = await fetch(base_url + "/api/get_online_users").then((res) =>
    res.json()
  );
  let unique = await fetch(base_url + "/api/get_unique_users").then((res) =>
    res.json()
  );
  return {
    timestamp: new Date().toISOString(),
    ...online,
    ...unique,
  };
}

[source]

The data is then inserted into BigQuery table. The infrastructure is defined using terraform, which looks something like this:

Code:

resource "google_bigquery_dataset" "maplelegends" {
  dataset_id = "maplelegends"
  location   = "US"
}

resource "google_bigquery_table" "userstats" {
  dataset_id = google_bigquery_dataset.maplelegends.dataset_id
  table_id   = "userstats"
  time_partitioning {
    type = "DAY"
  }
  schema = <<EOF
[
  {
    "name": "timestamp",
    "type": "TIMESTAMP",
    "mode": "REQUIRED",
    "description": "iso8601 timestamp of the measurement"
  },
  {
    "name": "usercount",
    "type": "INTEGER",
    "description": "Number of users online"
  },
  {
    "name": "monthly",
    "type": "INTEGER",
    "mode": "REQUIRED",
    "description": "Number of monthly users"
  },
  {
    "name": "daily",
    "type": "INTEGER",
    "mode": "REQUIRED",
    "description": "Number of daily users"
  },
  {
    "name": "weekly",
    "type": "INTEGER",
    "mode": "REQUIRED",
    "description": "Number of weekly users"
  }
]
EOF
}

[source]

Now to insert the data into the table, a cloud function is exposed via an express endpoint.

Code:

exports.insertStats = async (req, res) => {
  let stats = await get_user_stats();
  const bigquery = new BigQuery();
  const datasetId = "maplelegends";
  const tableId = "userstats";
  await bigquery.dataset(datasetId).table(tableId).insert([stats]);
  res.status(200).send(JSON.stringify(stats));
};

The cloud function points to the code above, and is triggered every 15 minutes by a scheduled job.

Code:

resource "google_cloudfunctions_function" "insert_maplelegends_userstats" {
  name                = "insert-maplelegends-userstats"
  runtime             = "nodejs10"
  available_memory_mb = 128
  source_repository {
    url = "https://source.developers.google.com/${google_sourcerepo_repository.bitbucket.id}/moveable-aliases/main/paths/functions/insert-maplelegends-userstats"
  }
  entry_point           = "insertStats"
  trigger_http          = true
  service_account_email = google_service_account.app_engine.email
}

resource "google_cloud_scheduler_job" "trigger_insert_maplelegends_userstats" {
  name        = "trigger-insert-maplelegends-userstats"
  description = "start an insertion into bigquery"
  schedule    = "*/15 * * * *"
  http_target {
    http_method = "POST"
    uri         = google_cloudfunctions_function.insert_maplelegends_userstats.https_trigger_url
    oidc_token {
      service_account_email = google_service_account.app_engine.email
    }
  }
}

[source]

Once this is up and running, it's easy to run queries against the data.

Code:

SELECT timestamp, usercount
FROM maplelegends.userstats
ORDER BY timestamp DESC

I define another cloud function that queries the table and exports a JSON file into a public cloud storage bucket. This is used by the chart on the site, and is found at the following location:

https://storage.googleapis.com/geospiza/query/maplelegends-online-count.json

The site itself uses Plot.ly to render the time-series data. A note that running this locally requires the use of an add-on like CORS Everywhere because the JSON file is separated from the HTML/JS.

Most the code here can be found at the site's source.