1. Hello!

    First of all, welcome to MapleLegends! You are currently viewing the forums as a guest, so you can only view the first post of every topic. We highly recommend registering so you can be part of our community.

    By registering to our forums you can introduce yourself and make your first friends, talk in the shoutbox, contribute, and much more!

    This process only takes a few minutes and you can always decide to lurk even after!

    - MapleLegends Administration-
  2. Experiencing disconnecting after inserting your login info? Make sure you are on the latest MapleLegends version. The current latest version is found by clicking here.
    Dismiss Notice

OwlRepo - a repository of transcribed owl searches

Discussion in 'Items & Mesos' started by geospiza, Jul 3, 2020.

  1. OP
    OP
    geospiza
    Offline

    geospiza Web Developer Staff Member Web Developer

    212
    449
    215
    Apr 16, 2020
    8:31 AM
    geospiza
    Dark Knight
    146
    Funk
    Actually, I found out how to reproduce this bug. When default sort in the settings is set to "Search Item", it results in wand for matt 60% being the first result. If you have Search Item as the default sort, it means you've been around for a while!

    Setting the sort to "Updated" or any other column and refreshing solves this. I probably won't get around to a proper fix until my next session, so this workaround will have to do until now.

    upload_2021-3-2_22-17-14.png
     
    • Agree Agree x 1
    • Friendly Friendly x 1
  2. Vezka
    Offline

    Vezka Mano

    13
    6
    20
    Feb 19, 2021
    Male
    4:31 PM
    Vezka
    Cleric
    174
    Perhaps add in more ores, like power crystal & stuff?
    Boss spawn items?
     
  3. OP
    OP
    geospiza
    Offline

    geospiza Web Developer Staff Member Web Developer

    212
    449
    215
    Apr 16, 2020
    8:31 AM
    geospiza
    Dark Knight
    146
    Funk
    Good suggestions.

    I've heard power, dex, luk, black, mithril and steel are the only ores worth anything, which is why black crystal and mithril are included. I'll give that its own section.

    Spawn items are a good idea; zombie teeth and dragon scales made it in, but things like cracked dimension and eye of fire that exist in search are worth adding too.
     
  4. OP
    OP
    geospiza
    Offline

    geospiza Web Developer Staff Member Web Developer

    212
    449
    215
    Apr 16, 2020
    8:31 AM
    geospiza
    Dark Knight
    146
    Funk
    Fixed a stupid bug with the search caused by a weird mix of libraries being used. I like the fuzzy search because it's lenient with the inputs, although it seems like it was not so awesome for some.

    I also included suggestions above, and included a search bar in the guide:


    [​IMG]


    I've finished migrating the website to Sapper using server-side rendering instead of just static files. I tried some stuff with preloading content that made the pages feel sluggish (i.e. fetch data from a bucket, transform it in the server, and attach it the top of the page before serving).

    The server was written in Python and ran on App Engine, but I wanted to use it for the Sapper server. The old API server that handles JWT authentication, uploads, and transcriptions now lives in Google Cloudrun (serverless docker). I have some middleware that proxies v1 requests to the Cloudrun, so the whole setup is pretty slick. I moved the app engine configs from a private repo into the client repo since it's so simple now.

    Next thing on my list reducing time to refreshes on the page from 1 hour to 5 minutes; I thought of a method that should be robust enough to handle denial of service attacks.
     
  5. Rainily
    Offline

    Rainily Red Snail

    6
    4
    16
    Feb 22, 2021
    Male
    3:31 PM
    Rainily
    Cleric
    58
    Tavern
    Seems like I'm getting 500 errors trying to upload now
     
  6. OP
    OP
    geospiza
    Offline

    geospiza Web Developer Staff Member Web Developer

    212
    449
    215
    Apr 16, 2020
    8:31 AM
    geospiza
    Dark Knight
    146
    Funk
    Sad noises, it looks I somehow mixed the backend service accounts:

    2021-03-05 21:54:58.478 PST
    details = "The principal (user or service account) lacks IAM permission "iam.serviceAccounts.actAs" for the resource "[email protected]" (or the resource may not exist)."

    I'll try to get this resolved soon, I have a large batch of stuff that I'm trying to upload too.
     
  7. OP
    OP
    geospiza
    Offline

    geospiza Web Developer Staff Member Web Developer

    212
    449
    215
    Apr 16, 2020
    8:31 AM
    geospiza
    Dark Knight
    146
    Funk
    This should be fixed now, here's an example of something I just uploaded: https://owlrepo.com/listing/a76a9741-fc65-4ece-bc60-011ffe813e8b

    I misconfigured something which was falling back a hard-coded value that I used for development. What an embarassing mistake. Thanks for reporting this!
     
  8. Rainily
    Offline

    Rainily Red Snail

    6
    4
    16
    Feb 22, 2021
    Male
    3:31 PM
    Rainily
    Cleric
    58
    Tavern
    It happens, I've been there MapleF12

    Thanks for OwlRepo, I use it all the time and try to upload as much as I can! It's a super cool project, nice work.
     
    • Friendly Friendly x 1
  9. theKatia
    Offline

    theKatia Capt. Latanica

    315
    874
    283
    Apr 29, 2017
    Female
    5:31 PM
    Asteracea
    Bowmaster
    130
    Funk
    Hey there, I've been using owlrepo a bunch lately and tried to give back and upload a few times... but my uploads don't seem to make it through, or I just can't tell?
    Here's an example of what I'm seeing:
    Screen Shot 2021-03-08 at 23.29.56.png
    Screen Shot 2021-03-08 at 23.30.19.png
    It's been more than 10 minutes, it's been almost a week. They all seem to get stuck on this status, and don't appear on the front page in 'recent uploads'.

    I've been working on making lots of space, for anyone interested I've been locating my shops at 7-7 top ^.^ skillbooks and ores at the moment.
     
    • Useful Useful x 1
  10. welch123
    Offline

    welch123 King Slime

    27
    5
    41
    Nov 4, 2017
    Male
    New York, NY
    11:31 AM
    nucky, Jacinda
    good website unlike that trash fruitscroll.ml
     
    • Disagree Disagree x 2
  11. OP
    OP
    geospiza
    Offline

    geospiza Web Developer Staff Member Web Developer

    212
    449
    215
    Apr 16, 2020
    8:31 AM
    geospiza
    Dark Knight
    146
    Funk
    I'm guessing you're uploading from a Mac? This is what I see in the logs:

    2021-03-08 13:29:06.630 PST WARNING:root:time data '2021-03-08 23.23.31' does not match format '%Y-%m-%d %I.%M.%S %p'
    2021-03-08 13:29:06.630 PST ERROR:root:could not parse timestamp Screen_Shot_2021-03-08_at_23.23.31.png

    It looks like there's a setting for using a 24 hour clock on Mac that I didn't consider? All of my screenshots have had AM/PM in there. I'll have to fix the server to accept the format (but I'm also wondering why the site it let you submit it...).
     
    • Agree Agree x 1
  12. OP
    OP
    geospiza
    Offline

    geospiza Web Developer Staff Member Web Developer

    212
    449
    215
    Apr 16, 2020
    8:31 AM
    geospiza
    Dark Knight
    146
    Funk
    Fruit has done a great job with their scroll site (defunct now though :(). It traditionally takes a lot of effort to maintain a scroll guide (C++'s guide, Precel's guide, Mi's guide), either from surfing shops or manually inputting data into spreadsheet. I appreciate the work that's gone into it, even if I didn't use it myself.
     
    • Friendly Friendly x 2
    • Like Like x 1
  13. theKatia
    Offline

    theKatia Capt. Latanica

    315
    874
    283
    Apr 29, 2017
    Female
    5:31 PM
    Asteracea
    Bowmaster
    130
    Funk
    Yep, I'm on Mac.
    Are we really that much of a minority that this hasn't come up before? lol
     
  14. OP
    OP
    geospiza
    Offline

    geospiza Web Developer Staff Member Web Developer

    212
    449
    215
    Apr 16, 2020
    8:31 AM
    geospiza
    Dark Knight
    146
    Funk
    Other than me (using AM/PM for time), there's a single upload by someone on a Mac. So pretty rare overall. You should be good to try re-uploading once you see v3.4.0 go live by the way.
     
  15. OP
    OP
    geospiza
    Offline

    geospiza Web Developer Staff Member Web Developer

    212
    449
    215
    Apr 16, 2020
    8:31 AM
    geospiza
    Dark Knight
    146
    Funk
    upload_2021-3-15_21-19-12.png

    OnionF5
    This is the first time someone has contributed more than I have.
     
    • Like Like x 1
  16. Poke.
    Offline

    Poke. Red Snail

    5
    1
    16
    Oct 17, 2018
    11:31 PM
    Hey there. I really love the idea of this project! It's incredibly well made and it's evident from your consistent updates that a considerable amount of effort has been put into it. Being a former *cough* "FM hoe", I used to spend most of my time loitering around the FM or every once in a while going through each shop, listing down prices and storing them in a .csv file as reference to sell my stuff.

    Due to that, I've always thought about training a model that would be able to recognize Maple's built-in text in order to list down items and its current prices. It's just that I had trouble envisioning the data collection part, and at the same time; wanted to stop spending so much time manually recording down prices. Seeing that this idea has actually materialized and has been executed so well made me incredibly motivated to work on something similar to this!

    If you don't mind, I'm not quite good in programming; but I would love to be able to help out with the development of the project!
    I see that you've listed some tasks we could contribute to if we'd like, but it seems to be a year old and I was wondering if these tasks still stand. I'm keen with tasks relating to machine learning or data visualization. I might not have experience with some of the tasks and I'm a little busy recently, but I'm willing to learn frameworks required to implement it on my free time!

    Regardless, a big thanks for making this amazing project! This is a great contribution to the community, and personally inspired me to work on a programming project as well! Thank you! MapleF14
     
  17. OP
    OP
    geospiza
    Offline

    geospiza Web Developer Staff Member Web Developer

    212
    449
    215
    Apr 16, 2020
    8:31 AM
    geospiza
    Dark Knight
    146
    Funk
    Hi Poke, happy to see the energy for contributions. The client is on github if you're interested, it should by default point to a non-production backend if you decide to do any development there. The list of tasks has been completed for the most part, but there's plenty to do if you feel like helping with more than just owls.
    • Visualizations and plots: The main page has the box plot on search, and there's a separate page for visualizations (https://owlrepo.com/charts). It uses plotly-js. I'm open to change; for example the charts page could include an interface for storing a custom set of items to watch.
    • Local storage: This is complete, the personal page has all of the upload ids. Also uploads include a public fingerprint that can be verified, so this could be done server-side too now. I've been using localforage for local browser storage.
    • Recommendations for contributions: I've been using the recommendation page to figure what to upload for the day. It's very simple (sort things older than N days old with more than X items in descending order). I think this could be improved significantly in terms of UI, and perhaps in recommending things that are likely to change in price. Basically, I'd like to optimize the value of the ~6 owls I use a day.
    • Fine-tuning OCR: I haven't done this, it's a lot of work. The core functionality isn't bad right now though.
    Improving the visualizations or the recommendations would be much appreciated. The charts page is unfinished, and the recommendations page has a somewhat clunky UI and a naive algorithm. If you're interested in the data side of things, I recommend loading the data locally. Here's some starter python code:

    Code:
    import pandas as pd
    
    # most recent per item
    index_df = pd.read_json("https://www.owlrepo.com/api/v2/query/search_item_index")
    
    # listing of all uploads, deduplicated
    listing_df = pd.read_json("https://www.owlrepo.com/api/v2/query/search_item_listing")
    
    I have some other ideas that could be fun to implement too. One I was thinking about was having a notepad-like tool where you can tab-complete items from owlrepo and have the price show up on the side. The idea is that you'd be able to use this to fill up your shop so it's easier to keep track of when you have to relist items.
     
  18. Poke.
    Offline

    Poke. Red Snail

    5
    1
    16
    Oct 17, 2018
    11:31 PM
    I would love to give a shot with the recommendation task! Could we have a chat regarding the requirements of the task? Given that I just got back to Maple and am quite inactive, my domain knowledge is quite outdated and am not entirely sure what should be prioritized on. SlimeSweat2

    Also, mind asking what's the accuracy for the OCR model? What kind of work does it entail?
     
  19. OP
    OP
    geospiza
    Offline

    geospiza Web Developer Staff Member Web Developer

    212
    449
    215
    Apr 16, 2020
    8:31 AM
    geospiza
    Dark Knight
    146
    Funk
    Feel free to ping me on discord (geospiza#5912). A good first task would be something small to get used to the code-base; getting an idea of the requirements will come with a little bit of familiarity. IMO, a good first task would be to add another section for mastery books and the custom etc. items in the guide (e.g., white scrolls, chaos scrolls, apples). Currently, they're lumped together with weapons like this:

    upload_2021-5-7_23-23-10.png

    That should be a good exercise; it shouldn't be too challenging if you're familiar with git/npm/javascript framework of the day.

    One direction this could go is to add a new column for the variability of the time series. The repo is at a point where most of the scrolls have at least 3 data points (many of the items on the guide have ~8 or so data points). It might be as simple as adding the MAD/standard deviation and sorting by the column and preferring to sort on a larger spread given the last time it was owled, and its relative value.

    At a higher level, it'd be nice to have a recommendation algorithm that incorporates how frequently something should be owled. These could be manual rules (white scrolls should be owled weekly instead of monthly), or it could be more intelligent (white scroll has been volatile recently, owl it again in another 2 days). These are decisions that I make manually now, but it would be great to have a system that someone else could use without too much context outside of "these are the scrolls you need to owl to keep the guide up-to-date, given a day's worth of vote cash."

    The OCR model probably goes something like this:
    1. correcting transcripts for a (diverse) set of images, implement a scoring function
    2. following the tesseract guide for retraining the top layers of the OCR model (an lstm), involving image annotation and whatever arcane build scripts they have
    3. improving on training processing using some kind of data augmentation (splicing images together to create "more" data)
    I'm using the base tesseract model, which works pretty well. I haven't actually quantified the model's accuracy, but you can take a look at the raw table from a recent upload I did to see how it does (https://owlrepo.com/listing/235c5f29-3523-4089-887c-293550c8db41). Weapons should always be in bundles of 1, and the high price of skis doesn't play too well with the OCR stuff. The mistakes are generally pretty obvious and get filtered out via outlier detection, but it would be nice to have some numbers to back this up. I have all the images stored in the same bucket as the transcribed data (uncompressed, which is eating up a lot of space...), so it probably wouldn't be too hard to add the raw images into the client to get a side-by-side comparison.

    Hope this clears things up; let me know if you have more questions!
     
  20. OP
    OP
    geospiza
    Offline

    geospiza Web Developer Staff Member Web Developer

    212
    449
    215
    Apr 16, 2020
    8:31 AM
    geospiza
    Dark Knight
    146
    Funk
    • Funny Funny x 3

Share This Page