Generating summaries of forum posts

I've been playing with Hugging Face transformer models recently, and had the idea of summarizing a few of the forum posts here.

Scripts can be found here, I have the repository private because it contains a lot of the raw forum text: https://gist.github.com/geospiza-fortis/1db40d17b20a8a5d5fd63212ab704399

You have to be logged in to view the entire thread, so I wrote a scraper using selenium. I parsed the dumped html with beautiful soup into json files. I played around with the the transformers library until I got summaries that seemed reasonable. My first naive implementations took about 30 minutes to generate a summary, but I realized I wasn't parallelizing things on my GPU. It takes about 10 seconds now to generate a summary for each thread.

While the summaries generally seem okay, there's room to improve on how the threads get split up. Because the summarizing model can only accept ~1000 words at a time, I have to divide and conquer e.g. split up the thread into chunks and then summarize those summaries. There's a lot of details that get missed, especially just using one size. The summaries tend to come out a little better when I do them by hand, but that doesn't scale when doing a large batch of them. Overall though, I'm pleased with the results.

Here are some samples from the automated summaries, some of which may be interesting, others that may not. Here's one for example. I'll post some more examples in the second post (to put it behind being logged in).

## better-anti-leech-system.4153

thread: https://forum.maplelegends.com/index.php?threads/better-anti-leech-system.4153

- original: 8715 words
- medium: 661 words (7.6% of original)
- short: 83 words (1.0% of original)

I suggest implementing a system that prevents leeching completely. Bishops are fast at killing, together with one, buying/gaining leech. A huge part of the Maple World gets basically never explored, even where there are maps that are good for certain classes, in terms of training.

Leching isn't removing the competitive aspect, YOU are giving up because you don't like the parameters of the competition. We should have different rankings for people who don't leech (get other people to play the game for them)

Spoiler: medium summary

Bishops are fast at killing, together with one, buying/gaining leech. A huge part of the Maple World gets basically never explored, even where there are maps that are good for certain classes, in terms of training. I suggest implementing a system that prevents leeching completely.

The game will change a lot without leeching, it will require everyone to explore all aspects and parts of the game and make it even more community-based. Leeching might have been a big part of v62, but it doesn't make a good part. Just because v62 isn't developed anymore and it has been here for a long time, we can still make it better.

Leching is game-breaking. Everyone groups up in maps where Bishops are fast at killing, together with one, buying/gaining leech. A huge part of the Maple World gets basically never explored, even where there are maps that are good for certain classes, in terms of training. If you want to leech, it should take you a longer time to level because you spend less effort.

The patch made the party exp share rates 80/20 instead of 60/40. It did nerf the leeching system BUT it also nerfed partying, which is going the wrong way. Just because v62 isn't developed anymore and it has been here for a long time, we can still make it better ourselves.

Leching will only improve the community by forcing people to find other more efficient ways to train (e.g. parties). I prefer to not play this game solo. train while exploring the world, do what you enjoy doing.

I don't envy anyone getting leech, I just don't understand them. My guess is that people just want to be high up on the rankings and that they don't actually enjoy getting leeched. Leeching is a part of the game.... why not make the party exp to something like 65/35 or 70/30 ~ ?

I don't understand why leeching is even in the game. With changing the rates to 80/20 people still don't have to do anything to get exp, even if partying would be faster. I find it highly unfair that someone uses the strength of their class for their own gains. I suggest any item that is involved with a quest to be marked untradeable. There is no reason why you can't just go and kill things yourself and collect items for yourself.

V62 Maplestory is challenging, but amidst the challenge people help each other, and your comment completely kills that. It's not as if they have an unfair advantage- everyone can obtain it if they decide to buy it. Leeching isn't removing the competitive aspect of the game.

Leching isn't removing the competitive aspect, YOU are giving up because you don't like the parameters of the competition. We should have different rankings for people who don't leech (get other people to play the game for them)

Leching is allowed and not banned in the ToS, and is a game mechanic. Hermits can kill hh, anego, bf all they want. Without leeching, they get more exp for themselves (unless we are starting to count HS)

If you don't like it, don't play it. This is a different server. Not all servers use the same rules. I came here because I was able to use both of my computers and leech my own characters. That's how I enjoy the game. What's wrong with making improvements?

The problem with this topic is that it shares similar if not the same issues as the Bishop topic. If you are leeching, you aren't playing the game. You're either alt-tabbed watching a movie or doing something else, or maybe not even home.

The only problem I see with this is that people just won't AFK when they leech or they will come back to their screen after x minutes, move 2 inches down the rope, and AFK again. With a 2 people in your party, HS goes from +10% EXP to +50% instantly.