Exploring player growth via rankings page

I'm in the middle of retraining the models and fixing up a few bugs. Turns out that I can compare the model fit using the model's log-likelihood. There's a measure called the Akaike information critereon that can be used to compare different models.

I tried 4 different models. There was the first model where the features I used were the seconds since the last level. The second model used log transformed values. The third used experience per second as the measure (by dividing experience to level from the experience table by the seconds to level). The fourth and last model used the log transform of that value.

This is the plot of the experience per second to level to 165 from a population of people at least level 175.



Log transforming this gives it a shape that's closer to a normal distribution.

Code:

aic: 11560.684924517518  (seconds)
aic: 11309.26298955611  (seconds, log-transformed)
aic: 11383.190899434145  (experience per second)
aic: 11308.935167475034  (experience per second, log-transformed)

I also did things like removing the overall account age, since it didn't seem to be contributing all that much to model fit.

I also tried playing around with l1/l2 regularization. These are penalty terms that enforce a particular solution. I did a random grid search over penalty weight and the ratio of l1/l2 terms.

Code:

p: 0.0000    l1 ratio: 0.0000    aic: 20756.375317352864
p: 0.0013    l1 ratio: 0.1274    aic: 20757.720481014894
p: 0.0902    l1 ratio: 0.9282    aic: 21003.535073800413
p: 0.0286    l1 ratio: 0.9439    aic: 20849.653643143458
p: 0.2151    l1 ratio: 0.8272    aic: 21201.889532110898
p: 0.0095    l1 ratio: 0.4721    aic: 20777.427744340894
p: 0.0321    l1 ratio: 0.0521    aic: 20780.380295308492
p: 0.0012    l1 ratio: 0.4995    aic: 20759.30579250059
p: 0.0263    l1 ratio: 0.7155    aic: 20828.254761777283
p: 0.8946    l1 ratio: 0.5667    aic: 21456.85113920379
...

After about 100 trials, I determined that ignoring this gave me the best result for higher level ranges. I'm going to ignore this for simplicity.

Finally, I played with an alternative model called the accelerated time to failure model. It assumes that the covariates either accelerate or decelerate the progression of some event. Anyhow, I just plugged the numbers in and determined that it didn't do much better than the model above. I ended up going with the a model retrained on log-transformed experience per second, which seems to do better than the super naive thing I did for prototyping.

While waiting for this to run, I made a few plots of the original models. I plot the median survival time (and the 90th and 95th percentiles) for each level. Each level has it's own individual model.



The spikes at the lower levels are caused by the much larger sample of players, which ends up skewing the median to be higher than you would expect. Cutting out the first 30 levels leads to a pretty reasonable plot that shows each level taking progressively more time.

 

Here's the time it takes to get to level 70 split across classes.

Spoiler

Code:

with jobs as (
    select distinct name, job
    from geospiza.mlrank.ranking
),
ttl as (
    select
        name,
        level,
        min(level) over (partition by name) as min_level,
        job,
        timestamp_diff(timestamp, min(timestamp) over (partition by name order by level), second) as diff
    from geospiza.mlrank.levels
    join jobs
    using (name)
),
quantiles as (
    select distinct
        job,
        level,
        count(*) as n_characters,
        round(min(diff)/(60*60*24), 2) as min_days,
        round(approx_quantiles(diff, 100)[offset(25)]/(60*60*24), 2) as p25_days,
        round(approx_quantiles(diff, 100)[offset(50)]/(60*60*24), 2) as p50_days,
        round(approx_quantiles(diff, 100)[offset(75)]/(60*60*24), 2) as p75_days,
        round(max(diff)/(60*60*24), 2) as max_days,
    from ttl
    where diff is not null and min_level = 2 and diff > 0 and level = 70
    group by 1, 2
    order by 1, 2
)
select * from quantiles