1. Hello!

    First of all, welcome to MapleLegends! You are currently viewing the forums as a guest, so you can only view the first post of every topic. We highly recommend registering so you can be part of our community.

    By registering to our forums you can introduce yourself and make your first friends, talk in the shoutbox, contribute, and much more!

    This process only takes a few minutes and you can always decide to lurk even after!

    - MapleLegends Administration-
  2. Experiencing disconnecting after inserting your login info? Make sure you are on the latest MapleLegends version. The current latest version is found by clicking here.
    Dismiss Notice

MapleLegends Online Users over Time

Discussion in 'Programming' started by geospiza, Aug 4, 2020.

  1. geospiza
    Offline

    geospiza Web Developer Staff Member Web Developer

    212
    449
    215
    Apr 16, 2020
    10:25 PM
    geospiza
    Dark Knight
    146
    Funk
    I built a small tool to plot the number of online users over time. The maplelegends API is pinged every 15 minutes and inserted into a database that powers this chart.

    You can play with the chart here. An up-to-date copy of the data export can be found here.

    upload_2020-8-3_18-2-12.png

    There was an issue with the server in June that made me curious enough to start collecting data. The server let people log in but not to log out, which caused the number of users to spike to ~2800 before it was restarted.

    upload_2020-8-3_18-6-24.png

    It's interesting to look at the history. It's a testament to how well this server is run with the new online population that hovers around 2k users at it's peak.

    My only regret is that I don't have any data earlier than 2020-06-10, but at least this doesn't require any maintenance on my part now that it's set up.

    The overview of gathering and displaying the data looks something like this:

    upload_2020-8-3_18-18-31.png
    [source]

    Pinging the MapleLegends API is straightforward. This is the same one that powers the !online command and the online box on the home page (I think).

    Code:
    async function get_user_stats() {
      let online = await fetch(base_url + "/api/get_online_users").then((res) =>
        res.json()
      );
      let unique = await fetch(base_url + "/api/get_unique_users").then((res) =>
        res.json()
      );
      return {
        timestamp: new Date().toISOString(),
        ...online,
        ...unique,
      };
    }
    
    [source]

    The data is then inserted into BigQuery table. The infrastructure is defined using terraform, which looks something like this:

    Code:
    resource "google_bigquery_dataset" "maplelegends" {
      dataset_id = "maplelegends"
      location   = "US"
    }
    
    resource "google_bigquery_table" "userstats" {
      dataset_id = google_bigquery_dataset.maplelegends.dataset_id
      table_id   = "userstats"
      time_partitioning {
        type = "DAY"
      }
      schema = <<EOF
    [
      {
        "name": "timestamp",
        "type": "TIMESTAMP",
        "mode": "REQUIRED",
        "description": "iso8601 timestamp of the measurement"
      },
      {
        "name": "usercount",
        "type": "INTEGER",
        "description": "Number of users online"
      },
      {
        "name": "monthly",
        "type": "INTEGER",
        "mode": "REQUIRED",
        "description": "Number of monthly users"
      },
      {
        "name": "daily",
        "type": "INTEGER",
        "mode": "REQUIRED",
        "description": "Number of daily users"
      },
      {
        "name": "weekly",
        "type": "INTEGER",
        "mode": "REQUIRED",
        "description": "Number of weekly users"
      }
    ]
    EOF
    }
    
    [source]

    Now to insert the data into the table, a cloud function is exposed via an express endpoint.

    Code:
    exports.insertStats = async (req, res) => {
      let stats = await get_user_stats();
      const bigquery = new BigQuery();
      const datasetId = "maplelegends";
      const tableId = "userstats";
      await bigquery.dataset(datasetId).table(tableId).insert([stats]);
      res.status(200).send(JSON.stringify(stats));
    };
    
    The cloud function points to the code above, and is triggered every 15 minutes by a scheduled job.

    Code:
    resource "google_cloudfunctions_function" "insert_maplelegends_userstats" {
      name                = "insert-maplelegends-userstats"
      runtime             = "nodejs10"
      available_memory_mb = 128
      source_repository {
        url = "https://source.developers.google.com/${google_sourcerepo_repository.bitbucket.id}/moveable-aliases/main/paths/functions/insert-maplelegends-userstats"
      }
      entry_point           = "insertStats"
      trigger_http          = true
      service_account_email = google_service_account.app_engine.email
    }
    
    resource "google_cloud_scheduler_job" "trigger_insert_maplelegends_userstats" {
      name        = "trigger-insert-maplelegends-userstats"
      description = "start an insertion into bigquery"
      schedule    = "*/15 * * * *"
      http_target {
        http_method = "POST"
        uri         = google_cloudfunctions_function.insert_maplelegends_userstats.https_trigger_url
        oidc_token {
          service_account_email = google_service_account.app_engine.email
        }
      }
    }
    
    [source]

    Once this is up and running, it's easy to run queries against the data.

    Code:
    SELECT timestamp, usercount
    FROM maplelegends.userstats
    ORDER BY timestamp DESC
    
    I define another cloud function that queries the table and exports a JSON file into a public cloud storage bucket. This is used by the chart on the site, and is found at the following location:

    https://storage.googleapis.com/geospiza/query/maplelegends-online-count.json

    The site itself uses Plot.ly to render the time-series data. A note that running this locally requires the use of an add-on like CORS Everywhere because the JSON file is separated from the HTML/JS.

    Most the code here can be found at the site's source.
     
    • Great Work Great Work x 4
    • Like Like x 2
    • Informative Informative x 2
  2. abe27342
    Offline

    abe27342 Orange Mushroom

    35
    10
    48
    Mar 14, 2020
    Male
    10:25 PM
    420
    Hero
    >at least this doesn't require any maintenance on my part now that it's set up.

    famous last words
     
    • Funny Funny x 3

Share This Page