1. This site uses cookies. By continuing to use this site, you are agreeing to our use of cookies. Learn More.
  2. Welcome to Pokemon Perfect, Guest!

    Our motto is Pokémon Practice makes Pokémon Perfect. We are a competitive-battling community that encourages the development of players and their ideas, and fosters positive and respectful attitudes. We love Collaboration (working together), Competition (getting stronger), and Communication (being informed).

    You are free to post everywhere, unless the thread explicitly states otherwise (usually in the case of a vote), and there are no private forums whatsoever. We just require you to not make multiple accounts. Let us greet you by posting a thread in the Introduce Yourself! forum.

  3. Tiers

    View Introduction to Tiers if you don't know what tiers are. Pokémon Perfect tiers are named differently to those on Smogon. A numeral followed by the letter U, e.g. 1U, 2U, 3U, represents a main tier on Pokémon Perfect – the '1' of '1U' representing the tier level. For a tier to be a main tier, it must be balanced (nothing is too powerful and game-breaking) and diverse enough (include a variety of Pokémon and strategies). A numeral followed by the letter P, e.g. 1P, 2P, 3P contain all Pokémon that are deemed overpowered in the respective 1U, 2U, 3U tiers. The 1st tier level allows Pokémon that are banned in the 2nd level, and this process continues down. Read the tier list, and in-depth explanations of the tiers naming system and tiering system. Also check out our analyses for all tiers.

  4. Tournaments

    RBY 1U Seasons and its master tournaments are responsible for starting up the community, and tournaments continue to play a big role in maintaining interest in the forums. Signups Open gives you a list of tournaments you can join, and Ongoing lists tournaments that you might want to follow. Additionally, you can tap to find out approximate Schedules for tournaments.

    For historical threads, check out Signups Closed, Finished tournaments and Results. We also have Nominations, Voting and Event threads for exhibitions – past and present.

All Gens Concept: Using Bots To Measure Metagame Understanding

Discussion in 'Analysis and Research' started by Disaster Area, Jun 10, 2019.

  1. Disaster Area

    Disaster Area Little Catto of Furr and Power Member

    Joined:
    May 4, 2014
    Messages:
    5,547
    Likes Received:
    2,423
    Interesting idea popped into my head at work:
    we can use bots such as Technical Machine to, in a certain sense, somewhat more objectively measure metagame understanding, both in an individual and collective level, of a tier.

    The thing about bots is that, for more advanced ones anyway, there are ways of changing the level of "difficulty" of the bot. For Technical Machine, I think this basically amounts to the depth of its depth search (but I don't fully recall the details). Let's run with the analogy of a depth search, looking N number of turns ahead.

    The idea then would be that, after a reasonable number of trials at depth search N, the player either has above 50%, below 50% or near to 50% winrate versus the bot. If the player has a <50% winrate than their level of understanding is less than N. Similarly, if it's >50% then their level of understanding is >N. If a player has >50% winrate at N and <50% winrate at N+1 then their level of understanding is N + 1/2.

    On a collective level, you can think of it as like there being a leaderboard to see the highest N for which a player can get close to 50% winrate. For example, say, in RBY OU, Troller did 50 games at N=3 and got a 62% winrate, then at N=4 he played 50 games and got a 38% winrate, then his level of understanding would be 3.5. And say that only marcoasd was able to top him, getting a 56% winrate in 50 games at N=4 but then 32% in 50 games at N=5 then, his level of understanding would be 4, and that would be the "depth to which we (as a playerbase) have understood RBY OU/1U".

    I would note that the more variability in the teams that can be brought, and the more luck based a tier, the more games need to be played before the result is meaningful. I wonder if vapicuno or someone could come up with some model to determine when enough games have been played to have an adequately accurate result, based on those sorts of factors.

    I don't know, just thought it was interesting to think about the fact that bots in some way actually allow us to measure our individual and collective understanding of the game, in a certain sense.
     
  2. magic9mushroom

    magic9mushroom BEST END. Member

    Joined:
    May 1, 2013
    Messages:
    652
    Likes Received:
    391
    And then people ghost with a bot in tournaments and ladders and the competitive scene becomes a joke. Fun!
     
    Lilyhollow likes this.
  3. Ilesaural

    Ilesaural Manque pierda. Season Host

    Joined:
    Sep 1, 2017
    Messages:
    135
    Likes Received:
    128
    I have thought a lot about this idea, but I think it's impossible to do, cause in competitive pokemon you are playing against another person and it will be hard for a bot to do some predictions.

    Anyway, I'm fully interested of how this will work, and I will follow this project :)
     
  4. Ortheore

    Ortheore Host Emeritus

    Joined:
    May 16, 2013
    Messages:
    2,029
    Likes Received:
    1,272
    So I don't know anything about TM or much about pokemon bots in general. How well developed are they? Can they stack up to top players? Also the logistics of this would be interesting. How many matches against an AI would be necessary? I mean, I guess if it's used as a training tool you could feasibly gather enough data. Also as a means of determining player skill it does have some issues in that I think it would be likely to have a very specific playstyle, which means some players would be favoured, you're not being tested against a variety of playstyles and also the AI could potentially be gamed. That said, I suppose these are all things that could be addressed if developed sufficiently

    If a bot is at an elite level, that would create a very interesting situation. I guess the closest comparison would be chess, where bots have been at a GM level for over two decades now. At least, it would be an interesting situation except...
    This is why we can't have nice things. What solutions exist? To me the only way around it would be to make the game less accessible- disable the API, shift all information exchanges into formats difficult for programs to handle. Even then, if the playerbase were sufficiently motivated to cheat that'd become a game of whack-a-mole. That's probably the biggest factor saving things- because this game is basically only played at an amateur level (barring the small prizes given out for seasons and some tours) there probably won't be a cheating arms race, because there's just not enough incentive.

    Also, since using AIs would still be worth pursuing you would also want a dedicated server for AI matches

    I really want to see the competitive scene shift into irl tours (I was actually planning on running some in Sydney, but some personal stuff got in the way and I decided to shelve the idea), since they offer a very different dynamic but also it's much harder to cheat. I think cheating is a massive deal that greatly limits the scope of competitive pokemon, certainly there's currently no way of advancing beyond an amateur stage. It would probably take a grassroots irl movement that got lucky and picked up momentum. That said, I'm not claiming irl tournaments are some kind of panacea either- I have no idea of what (reasonable) measures might be implemented to prevent teams leaking between matches, though I haven't done enough research for that either (look at Hearthstone?).
    I just think it'd be really cool if pro pokemon was a thing

    Also no, I'm not counting VGC, as that's basically an entirely separate community (fun fact, I ran a survey on what tours to run and one of the responses was openly hostile to the idea of running non-VGC formats i.e. OU and Ubers). For the longest time I treated VGC with contempt because I regarded it as a phony, contrived format that sees nintendo trying to trample over the existing competitive community while being completely out of touch. I still believe all of those points, it's just I care less about them now, as I do see VGC's merits- bring 6 pick 4 is really interesting from a teambuilding perspective, especially if you like to experiment, while doubles is faster and on a per turn basis is more deep than singles- not saying either one is overall more deep than the other though, as singles is balanced out by the fact that matches last longer, and doubles has fuck-all long term planning compared to singles
     
  5. Disaster Area

    Disaster Area Little Catto of Furr and Power Member

    Joined:
    May 4, 2014
    Messages:
    5,547
    Likes Received:
    2,423
    there are some threads about technical machine on smogon, tjdaas can point you their way

    it's been around since gen 4 - in fact in while DPP was still current the bot actually managed to beat its own creator in a game of DPP OU. However it wasn't made to work with PS until a few months ago

    also for what it's worth I think all the
    is scaremongering since we've literally had this bot which is of a competitive level for longer than I've been playing Pokemon on the internet and that hasn't happened or come anywhere close to happening yet. Like literally, the bots old website mentions that it could be run on POKEMON LAB lmao. So with regards to
    I don't know except that we've had this bot for a decade or something and noone has abused it yet and it's frankly not something people even talk about, so whatever it was that was keeping it from being abused like people are hypothesising is apparently working perfectly well.
     
  6. vapicuno

    vapicuno Member

    Joined:
    Jan 3, 2019
    Messages:
    6
    Likes Received:
    24
    I haven't actually heard of the bot so thanks for letting me know of it. I find the idea interesting, though I see two challenges. If we were to use bots to measure metagame understanding, the bot must first be supplied data about the metagame. Also, it assumes that depth is a good indicator of understanding.

    One could say bots could be used to gather metagame data, but that's subject to bias because there's a huge amount of incomplete data. Statisticians who deal with data imputation will know more than me about this, but I don't know if there could be systematic bias. For example, the last pokemon revealed isn't just a random pick from the list of tier mons, it's usually something specific.

    That brings me into the next point: Pokemon can be strategic, and in so I'm not sure if depth is a good measure of mastery. You may know way in advance that you want to prepare Curselax for the endgame when you just see two passive mons on your opponent's team. You can fumble in midgame, be 1-5 down and still win against a computer that reads you 10 moves ahead using a material-based metric. In that case, are you a 10-move player when you aren't so good at making good short term decisions?

    What I do like about this idea is that with a mixed strategy nash equilibrium, you can probably decouple metagame understanding from player psychology. No more "this person tends to be aggressive at the start and play more safely towards the end" or vice versa. That doesn't work against a bot who always makes plays optimally up to a certain depth. A good bot can even learn to cteam you to test your understanding of varied teams from the point of view of the person using the team, not just playing against the team.

    Cool ideas.
     
    Disaster Area likes this.
  7. Disaster Area

    Disaster Area Little Catto of Furr and Power Member

    Joined:
    May 4, 2014
    Messages:
    5,547
    Likes Received:
    2,423
    My understanding regarding Technical Machine is that it does use a mixed nash equilibrium approach... when I wrote this I did have Technical Machine specifically in mind precisely because it's so sophisticated and plays in a way which mimics "optimal pokemon" insofar as it's possible (mixed nash equilibriums and data-driven prediction of what the opponent's team is, though as it stands I don't think it can build its own teams?)
     
  8. Hipmonlee

    Hipmonlee Member

    Joined:
    Oct 11, 2018
    Messages:
    13
    Likes Received:
    24
    Pretty sure TM doesnt do any mixed strategies yet, I imagine it wouldnt be too hard of a thing to add though.
     
    Disaster Area likes this.

Share This Page