OwlRepo - a repository of transcribed owl searches

I've been thinking about the best way to keep the repository up to date because out of date prices are useless to most folks. I'll address this in two ways:

• Remove anything older than a month from the front page.
• Add recommendations for owls to upload.

The first bullet is to encourage owl use when the price goes out of date (as a seller). Prices go out of date quickly. Historical values for all items will be accessible from the items page though.

The second is to optimize owl usage. Not everything needs to be owled, only the high value and high volume items. Mii's (excellent!) scroll guide
only tracks things above 500k, and is color coded for how recent an update was. I want a page to tell me what to upload and when so I don't use more owls than I can afford. I put together a model that predicts the number of search results and the 25th percentile of prices for things that don't exist in the database yet, from the most recent uploads. The results are not great, but it's better than brute forcing and wasting effort on low value things.

Here's an example spreadsheet of predicted prices for 2020-07-17: https://docs.google.com/spreadsheets/d/1YpmGZJrcuX5uUwObc_SVwxjlERJH3z9P70FN84Mu_wY/edit?usp=sharing. I'll be adding this to the API like all of the other things.

On another note, the historical data that's being collected is pretty interesting. I've heard that it's been even harder to sell Zakum, CWK, and HT services nowadays...

Spoiler: python code

Code:

import requests
import pandas as pd
import matplotlib.pyplot as plt

req = requests.get("https://owlrepo.com/api/v1/query/search_item_listing")
listing = pd.DataFrame(req.json())

df = listing[listing["search_item"] == "Scroll for Gloves for ATT 60%"]
df.index = pd.to_datetime(df.search_item_timestamp)
results = df[df["percent_complete"] > 0.6][["search_results", "p25", "p50", "p75"]]
print(results)

results.plot(y=["p25", "p50", "p75"])
plt.show()

results.plot(y=["search_results"])
plt.show()

 

I added a recommendation page alongside a few minor tweaks. I had a lot of fun with this one, although it's tricky to define "valuable". There are also other items like skill books (e.g. MW20) and ores that are ignored here.

Spoiler: recommendation page

In the "Other" drop down, select Recommendations. This uses the new endpoint for sorting scrolls based on predicted value. This gets updated 5 minutes after all of the other queries.

Code:

https://owlrepo.com/api/v1/query/scrolls_predict

If you have extra owls, have at it

 

Thanks for the feedback!

I've run into the unknown case before. There's a case that's easy to fix by tweaking a regex and a case that's hard to fix where the name goes off into the margins of the UI. The latter can be fixed for scrolls, now that I have data from the library that can be used for spelling correction (like "acoidability" scrolls). All of the screenshots are stored on the site, so it's easy enough to reprocess uploads.

I'm not sure how to deal with the unobtainable scrolls (spikes on shoes 10%?), but I'm sure its a small list. The spreadsheet in one of the earlier comments has all of the known scrolls. If you can make a list (or annotate an existing one), I can incorporate it.

I had no idea about the bottomwear dex scrolls, but it's a problem in more than one way because there's no way to differentiate them from an owl screenshot. This is the same problem with weapons and getting their stats. This can be fixed in the unseen list though by parsing out the stats from the library descriptions, but there's no way of mapping historical data to the set of scrolls for forecasting.

 

Thanks for the list, I've incorporated this into the unseen scrolls with minor modifications. "Scroll for Earring for INT 30%" no longer shows up, so it's working as expected.

Spoiler: unobtainable_scrolls.txt

Code:

# https://forum.maplelegends.com/index.php?threads/owlrepo-a-repository-of-transcribed-owl-searches.32316/page-2#post-223116
# Thanks to Luscious for compiling this list. If there is at least one type
# of scroll available with the same name, it is omitted from this list.
Scroll for Earring for INT 30%
Scroll for Earring for DEF 60%
Scroll for Earring for DEF 10%
Scroll for Earring for DEF 100%
Scroll for Bottomwear for Jump 60%
Scroll for Bottomwear for Jump 100%
Scroll for Overall for STR 100%
Scroll for Two-Handed BW for Accuracy 100%
Scroll for Two-Handed BW for Accuracy 70%
Scroll for Helmet for Accuracy 10%
Scroll for Helmet for Accuracy 100%
Scroll for Helmet for Accuracy 60%
Scroll for Helmet for INT 10%
Scroll for Cape for Cold Protection 10%
Scroll for Knuckle for Accuracy 100%
Scroll for Knuckle for Accuracy 70%
Scroll for Knuckle for Accuracy 10%
Scroll for One-Handed BW for Accuracy 100%
Scroll for One-Handed Axe for Accuracy 10%
Scroll for One-Handed Axe for Accuracy 100%
Scroll for One-Handed Axe for Accuracy 70%
Scroll for Two-Handed Axe for Accuracy 100%
Scroll for Two-Handed Axe for Accuracy 70%
Scroll for Earring for HP 10%
Scroll for Earring for HP 70%
Dark scroll for Cape for Magic Def. 70%
Scroll for Overall Armor for DEF 30%
Scroll for Shield for LUK 60%
Scroll for Earring for DEX 100%
Dark scroll for Helmet for DEF 70%
Scroll for Earring for LUK 100%
Scroll for Gloves for HP 100%
Scroll for Spear for Accuracy 100%
Scroll for Two-Handed Sword for Accuracy 70%
Scroll for Two-Handed Sword for Accuracy 10%
Scroll for Pole-Arm for Accuracy 100%
Scroll for One-Handed Sword for Accuracy 100%
Scroll for One-Handed Sword for Accuracy 70%
Scroll for One-Handed Sword for Accuracy 10%
Scroll for Helmet for DEX 100%
Scroll for Topwear for LUK 10%
Scroll for Shield for Magic Att. 10%
Scroll for Shield for Magic Att. 100%
Scroll for Bottomwear for HP 10%
Scroll for Spikes on Shoes 10%
Scroll for Bottomwear for DEX 100%
Scroll for Bottomwear for DEX 100%
Scroll for Shield for HP 100%
Scroll for Topwear for HP 100%
Dark scroll for Topwear for HP 70%
Scroll for Shield for STR 100%
Scroll for Shield for LUK 100%
Scroll for Shoes for Avoidability 100%
Scroll for Helmet for HP 100%
Scroll for Topwear for STR 100%
Scroll for Shield for HP 100%

I also added a page for tracking personal upload history which is stored on device. Ugly as it is, it's a good base for future improvements like downloading bundles of cropped images and transcripts.

Spoiler: Personal Statistics

There have been 5.2k owl pages and almost 40k search results uploaded since the site has gone online. I've been spending 5-10 minutes a day uploading ~10 items a day (with a handy script for pulling owls from the screenshots folder), so overall I'm happy with how easy it's been to collect data.

There's enough data to retrain the OCR models now, which should help the trustworthiness of the results. It's kind of an onerous task, but one that I eventually want to get out of the way. I'm interested in the error rates, and how much retraining will help improve things. Getting the error rate down means that that bundles might actually be useful for weighing averages (e.g. 100x scrolls for 1k and 2x scrolls for 50k gives different results for the average price if you count the bundle).

I'm also quickly running out of vote cash on my owl mule, but I'm done collecting data on the 10 most frequent items this week. Afterwards I'll be following the recommendations to be efficient with my owling (which means avoiding weapons unless I'm actively seeking them out). I'm hoping the repo gets to the point where most items have at least 3 data points spread several days apart, which should make for nice charts.

 

Thanks for the suggestion! Charting is one of the next improvements that I'd like to work on, now that there's some history to work with. The charts page is horribly formatted right now, and isn't as powerful as it was when this first started. I'm definitely interested in getting an overview of the important items, so it's great to get context from those who knows a lot more than me .

The list you've recommended is well thought out. The original 3 that I tracked day to day were glove att 60, earring int 10, and dark crystal ores. The first two were from the monsters I was grinding at from level 50-80ish, while the ore was interesting because there's quite a bit of it being sold. Now that I've experienced more of the content, I can definitely see value in tracking those major scrolls in particular.

The brown work gloves might not be too useful to track through this tool, but I have noticed that the price for clean gloves have gone from 12m to 18m recently.



Without stats, it's hard to glean useful things directly from the price (although the volume being traded could be a useful indicator).

Some of the items don't have much history built up, but what I could do is make them appear on the upload recommendations more often. Prioritizing items (based on data) is something that I'd like this tool to be good at, so this is good input. Currently, the recommendation list defaults to 30 days. For the major scrolls like gfa60, this could be as low as every 3 days.

 

It's because the text goes off the end of the UI. Here are the relevant images from the uploads:





Inside the code somewhere, there's a bit of magic to extract the text out:

Code:

    try:
        item = re.match(".*results for (.*) that you.*", lines[0]).group(1)
    except:
        logging.error(f"could not parse search item: {lines[0]}")

The text is being chopped off, so it defaults to "unknown". That being said, I have most of the tools to fix this. Hopefully not too long from now, but I'll make sure to go back and fix all of the unknown uploads.

 

I put together charts with a search box and finally got rid of the embedded Data Studio chart. It very rough, but it'll get better with time.

Spoiler: work in progress charting

Here's a video of it in action. Once this is set up, it should be easier to set up a dashboard with predefined items.

 

I'm finally free from owling the top 10 items. I'll be using my best judgement from this point forward.

I put together a separate report with these "core" items that utilizes the tables generated by new uploads. Kind of neat to see the relationship between the volume of trades and the median prices after months of collecting data. It's actually not the volume since bundle isn't reliable enough to use yet, but close enough.

Spoiler: etc dev notes

I'm also actively working on the backend which makes it easier for me to maintain and develop against. The biggest piece is reimplementing the Google Cloud stuff using terraform, which will result in prod/nonprod environments. I'm happy to hand out keys to the development environment once this is done.

I've been removing references to the code-name (ml-market) and making the code modular since there are a surprising number of components involved.

One thing that I may implement is adding an optional user id to uploads, so it's possible to build trust through a good upload history. I'd like to be able to identify the data that I upload into BigQuery using the id. It's totally possible that I over-engineer this and require uploads to be signed with the server public key and a private key generated in the client. This makes it much harder to fake the id and easier to assign trustworthiness to uploads based on history.

 

I think I've fixed this now. I used the dark scrolls for one-handed magic attack to test that this works, so it should be possible to track most scrolls now. Here are a few examples:

https://owlrepo.com/listing/67d00884-9b29-492b-9601-69dbe47f52f0
https://owlrepo.com/listing/025c44ff-9983-4f12-b2b3-9280850b781d
https://owlrepo.com/listing/fcc476c0-e060-4742-bfed-fdcd573c0f42

Spoiler: dev notes

I have a list of ~15 uploads that I need to be fixed, but I'll put this on the backlog for right now. These include the face for avoidability scrolls. I need to write the code to resubmit a task_ids, which is why it hasn't been done yet.

Code:

WITH
  unknown AS (
  SELECT
    DISTINCT task_id
  FROM
    owlrepo.upload_flat
  WHERE
    search_item IS NULL )
SELECT
  CONCAT("https://owlrepo.com/listing/", task_id) AS uri
FROM
  unknown

This list for the curious, which has all of the unknown search items.

Spoiler: unknown items

https://owlrepo.com/listing/bb61294c-75e6-4ccb-b8ec-fac1035a0820
https://owlrepo.com/listing/5ae42d61-91c2-42f6-a5d5-36089ab4e842
https://owlrepo.com/listing/53862f31-8bb6-419a-aa44-d5d488e12425
https://owlrepo.com/listing/cf456b92-4276-4748-8423-3fc15d124342
https://owlrepo.com/listing/6c715ca0-5d0a-453a-9550-6959bf55f055
https://owlrepo.com/listing/ec771f0e-29d0-4ef2-b027-fe718c811f66
https://owlrepo.com/listing/8e576293-5025-42a3-98a7-0cb9531685ec
https://owlrepo.com/listing/bd1517a7-faba-4e0b-960a-11bf4f0dd77c
https://owlrepo.com/listing/6eb17ce5-d5d2-4f6c-a2ee-61eb3d4e9726
https://owlrepo.com/listing/3b307d4c-f92c-40cf-a93b-f6777e2ffb60
https://owlrepo.com/listing/87d920fc-233b-4dbf-9e17-0695923900be
https://owlrepo.com/listing/25dc5254-e3a4-4eea-8f9c-4825a0c62cbd
https://owlrepo.com/listing/6de1f6f3-7376-44b3-b1cb-987711042b55
https://owlrepo.com/listing/179a228b-053a-4319-a357-de46131146f3
https://owlrepo.com/listing/ed8d556e-ba14-412a-9a69-f48e110e0cc3

I've also set up a proper development and production stack using terraform. It looks like a more refined version of an unrelated thing I did to measure the number of online players in the server: MapleLegends Online Users over Time.

I made the swap over to the new servers yesterday, and it's mostly been smooth sailing. If you're interested in doing any sort of development on the site, the source code and the development instances are actually in a good state now.

 

There are some new plots that might be useful to know about. When you click an item on the front page, it will show the plot underneath the item results. A chart will show up when there is only one type of item in the table. For example, here's "[Mastery Book 30] Genesis":



Obviously, something weird happened here between August 1st and August 10th. The median price of the Gen30 skill book jumped from 100m to 180m, which was preceded by a buyout of the market. The "[Mastery Book 20] Maple Warrior" data shows a very similar trend, but the plot is currently trash because of outlier value on the 14th.



I pulled out an old notebook to plot this data by hand for comparison.



We can see that someone bought out the Horntail skill books, and moved the market to a new value for a short time. This was an unexpected observation, but it makes sense in retrospect since these had been on the frequently owled list for a few weeks.

I'm happy with the way these price/volume charts turned out on the items page, and they look half decent on mobile too. It's not perfect though. I'll be on the lookout on how to filter the time-series data so the chart doesn't look all wonky in the future. My attempts using MAD (see below) on the 75th percentile didn't seem to work well for some reason, but overall the plotting stuff needs to be reworked a bit.


I also added some box plots to the listing page where you can see all of the results from a transcribed owl. Here's a few from a recent batch that contains a bunch of common scrolls (as suggested in this thread). https://owlrepo.com/listing/10363539-3f13-44a9-b494-d13adab686d9



I ended up filtering data here because the box plots would be heavily skewed by someone selling a thing for 1B when it's really only worth 100k. This summary is generated on device, so it can take a few seconds to load on mobile for pages like GFA60 where there are hundreds of results.


I'm trying to capture as much of the current state of things before the market goes wild with the next patch/event. If you find this kind of data interesting, I encourage you to screenshot your owls and contribute .

 

This should work with Chrome and Firefox, from my testing. It can take up to 10-15 seconds to load the page depending on when it was last loaded. The files are also kind of big. Do you mind opening up the console on the page, reloading, and sending me an image of it so I can diagnose a bit more?

 

I took advantage of the most recent server maintenance to do a measurement of transactions in the free market. View this notebook for the data and code.


[source]

Starting 2020-09-04 midnight server time, I transcribed owl searches for Honsters and GFA60s every 6 hours. By looking at the difference between consecutive search results, it's possible to get an estimate of items that are sold. Because all stores are reset during the maintenance, it means that search results that disappear are from sales rather than stores closing.

Honsters are interesting because they happen to be a cash crop in the FM. I buy many of my NLC potions from the FM because of the convenient.

From "The Junk Yard has closed":

There were a total of 34516 Honsters added and 20350 Honsters removed from the market during 2020-09-04. This is almost 61m mesos exchanging hands. The overall profit is 4m, resulting in a profit margin of 6.6% after taxes.

The top 5 sellers for this period are captured in this table:


These are almost all mules, but there is quite a bit to be made here.

The GFA60 market is also pretty robust. On the site, the charts only plot the number of search results. We can also look at the sum of the bundles (or the total number of scrolls).



There were about 146 scrolls sold totaling ~54m of profit. I noticed a few big sellers on the market, who've consolidated many scrolls.

The top 5 sellers are listed here:


Half of the mesos being exchanged here were all done by a single seller who made 29m.

From the table of differences for each sample period and seller, I was able to plot the normalized histogram of prices for items added and removed from the market.



The scrolls that are removed (i.e. sold) have a smaller mean price than the scrolls that are added. This is intuitive: of course you're going to buy the cheapest scrolls.

I've been thinking about running this again but for a longer period of time, but this is more challenging once stores closing after 24 hours are taken into account. Not sure how I would deal with it, but I think its possible with a few days of carefully collected data and some modeling.