Writing the bot for Forge of Empires MMO game

Prologue.
FoE bot appeared pretty much accidentally. I was applying for Senior Flash developer position at Innogames at the beginning of 2014. I successfully passed first 2 interviews and had to prepare myself to the last important one. I did not have games in my portfolio and decided to impress the company by cracking some of their online games. I had one weekend for that but eventually succeeded in much less time. It took me just few hours to find the core algorithm which made this bot possible (however, it took more time to build the rest of functionality). Eventually, I did not pass the last interview but decided to try my bot as some kind of experiment. It went successfully – I was at the top of rating list 2 times, a year ago and half a year ago, without spending a cent, surrounded with users who spent thousands of dollars. It took me quite a lot of time to finish this article, approx. 1 year. I also have to say that Innogames knew about the bot but seemed to ignore the fact. First, I told about it to some lead developers and top managers who were present on my third interview. Next, I mentioned it on another interview in half a year to some Forge of Empires developer. I still can use my bot today, nothing changed on their side. Eventually, it just happened that I had a conversation with another top manager from Innogames today and decided to finish the article. And yes, Innogames, I have found your custom header: “X-JoinUs: If you found it and want help us improving our Games, go to career.innogames.com and mention this header!” – which is there for at least last half of a year.
I am not providing compiled FoE bot and have no plans to do this in the future. Please, do not ask for it neither in comments nor in private messages. The code for FoE bot is available on GitHub here The code is no longer available. However, it is not 100% functional. There are few parts which use the code from the game itself and I removed them from the repo. Following article has some instructions how to find that missing code. I will also not provide any instructions on how to compile the bot or use it. I never had plans to distribute it or to teach people how to use it. The code itself is quite a mess, so is pretty much an example of how NOT to write the apps.

List of features.

Current Forge of Empires Bot consists of built in browser which you use to login into the game and play it just like you used to. It also has several automated features which work behind the scene as well as number of manual ones.

Automated functionality:

  1. Picks up gold from the buildings. It will pick it up when the production is ready.
  2. Initiates the shortest (5 minutes) production of resource and picks it up.
  3. Initiates the shortest (4 hours) production of goods and picks it up.
  4. Initiates and pick the production of military units (a bit buggy, does not work sometimes)
  5. Saves the list of your events. It is used for some statistics.

Manual functionality:

  1. Automated motivation. You can motivate all your neighbors, guild members and friends in single click. This was build long before similar functionality was introduced by Innogames. You can choose what building’s Era is your primary target (however, this is hardcoded now).
  2. Similar to above, but uses new automotivation functionality inside the game. Still, a huge time saver – few clicks instead of dozens.
  3. Invite friends. Sends messages to different users with request to add you as the friend. The filter to whom send the messages is hardcoded.
  4. Invites managers. You can remove all your friends there. Then, you can send friend requests (not messages to ask to add you, but explicit friend requests) based on hardcoded filter. Your neighbors and guild members are skipped.
  5. Guild manager. Shows some statistics of your guild based on your own events: how many times the user motivated your city or traded with your, how many forge points the user invested in Great Buildings of your guild, the date he/she joined the guild.
  6. Plunder manager. Shows the neighbors whom you attacked and can plunder with the list of goods production buildings and the amount of goods in current production cycle. Then, you select what you want from the list and the bot will automatically try to plunder until success (happens in 90% of cases).

    plunder-manager

  7. Battlefield. Uses the bug in the game when I can move units to any place in the field during the battle.
  8. Autobattle. Actually, not absolutely automatic in vast majority of cases. However, can bring you hundreds of thousands of resources and gold and lead you to the winner on World map. Uses another bug in the game when it is possible to initiate same battle on world map multiple times. Current implementation has some hardcoded scenarios for few sectors. You can perform the whole battle just by single click of the button, or even perform same battle in a loop by single click. However, this works for selected sectors only and with specific armies.
  9. Sending messages. This functionality does not have UI trigger and is hardcoded and initiated manually from the code. However, it is really powerful feature for spamming. The app can send same message to thousands of users.
  10. City designer, sort of. The quicker way to build lovely gardens in your city.

Update 08 May 2015:

foe3

foe9

foe10

Update 10 May 2015:

Town Halls museum would never appear without this bot.

Forge of Empires is MMO game developed by German company InnoGames. They claim to have over 14 million registered users in it (at least, this was true at the beginning of 2014 year).

The first step is to find out the structure of the data which is used for game-server communication. Fiddler is one of the best tool for this purpose (this application is for Windows only, so you should find the alternative if you are on Mac). It is debugging proxy which you can use in many situations when you need to check or modify HTTP(s) traffic.

fiddler-general

Fiddler’s window consists of several parts. The list of captured requests is to the left, and tools are to the right. We are interested in Inspectors and Composer. Sometimes AutoResponder may also be very handy. The data of the particular request is populated in Inspectors upon selection from the list. Inspectors themselves consist of top part for request and bottom part for response. Inspectors have different views which allow either to see particular part of request/response (e.g. Headers) or format the data (e.g. JSON, XML). Fiddler will also suggest you to decode the data if required. Sometimes you need to temporary stop capturing new requests. The easiest way to do it is by clicking in the bottom left corner on “Capturing” word. The word will disappear, but you can click the same place when you need to re-enable capturing process.

Let’s start investigating the communication process. We can perform some actions in the game, like picking up productions, moving buildings, reading list of top users. We can see that all requests have same simple url. The url does not change with different requests and looks like following:
http://ru5.forgeofempires.com/game/json?h=457ee95a4af
All payload is sent as JSON. The communication is performed via HTTP, the data is not encoded at all and you can easily study its structure.

fiddler-composer

We can perform some simple tests right now and decide whether we are going to move forward. Composer tool in Fiddler will help us in this task. Select it and then simply drug and drop any request from the list into the composer. All fields will be populated with data from the request so you have its clone. Picking up the production from city entities is one of the basic actions in the game. It is also nice first target for the bot because shorter production period gives more income. We need to answer several questions: can we perform API requests outside of the game; can we modify the request; can we modify the response.

To do this, delete all captured request in the Fiddler first, just to keep things clean. Simply select everything and click Delete button. Go to the game, pick up the production from some building, then go back to Fiddler and drag and drop the request to Composer tool. Click on Execute button. Select new request in the list and go to Inspectors. Great, we see absolutely valid response there – no errors. OK, we can send requests to the server outside from the app and the backend eats it. Now, we need to find out if we can really use this. Go back to the browser and refresh the page. Nothing changed, the amount of resources stays the same. Let’s wait till resources from same building can be collected. Then, do not pick them up. Instead, go back to Fiddler and click Execute button in Composer again. Now, everything worked as we wanted – the production was picked up, which you can see by refreshing the browser again. So, we used 100% the same request like before, and it worked. Thus, we already have something. Let’s move forward.

We are going to alter the request now and see if this works. Each request contains 4 variables which are different in different requests: requestClass, requestData, requestId and requestMethod. Also, only 2 of these properties are different for similar requests (for example, picking up gold from the building): requestData and requestId. It is easy to find out that requestId is integer incremented with each new request. It also looks like it is not important for us because we successfully performed several requests with same id. Let’s pick up the gold from another building and investigate the request. We see that the only part that has been changed is some integer in requestData. It looks like this is the id of the building. Paste it into our request in Composer, changing the id there, and then click Execute. Oops, we got error response this time. There is another part of the payload which is different in different requests and we missed it – some hash prepended to JSON object. Obviously, it is generated inside the app. We need to get the app file now, decompile and investigate it.

fiddler-hash-tag

Fiddler also helps as to find the url for main swf file. Look at our previous requests in Inspectors>Headers. There is a header
Referer: http://foeru.innogamescdn.com/swf/Main.swf?1429602617
which is what we are looking for. We have the file now. Then, we need some decompiler. I suggest following open-source flash decompiler: JPEXS Free Flash Decompiler (https://github.com/jindrapetrik/jpexs-decompiler).
I have to say that I really love the code of the game. I think it is written very well and has very nice and clear organization structure. I believe this plays crucial role when some new developer starts working on the app. However, at the same time it makes the process of investigating the code easier. Also, it looks like the code is not obfuscated at all (well, I think this was true a year ago. I had another conversation with some FoE developer last summer and it seems that they started to obfuscate the code now, though not much. I am not sure if this is a result of our conversation). This was not hard to find the place where hash code is generated. However, I will not post it here. After all, this is Innogames’ commercial product and I want to respect this fact. You will not find this part on GitHub as well. You need to find it yourself if you want to compile working bot. The hash algorithm consists of JSON payload, some hardcoded salt and another dynamic salt. It is clear that the second dynamic salt goes from the request’s url. It is some session id
http://ru5.forgeofempires.com/game/json?h=457ee95a4af
At this point, we can test everything and answer the question if we can build our own requests. Create some simple Actionscript 3 project to generate required payload. Paste generated data into Composer (leave all other fields as they are. The only thing that you may need to change is the session id in the url). Click Execute. Everything works. At this point, we have working core methods to build our bot. It took me just few hours to investigate the algorithm.

There is one drawback of our bot. It’s not absolutely automatic. We need to change session id manually each time it is changed (not often, but still we need to do this). Also, we hardcoded ids of city entities. They are changed each time we add or remove some building and we need to provide these new ids manually. Really, not an elegant solution.

Let’s check some simple places to find if the session id exists there. My first idea is to look at headers. Fiddler will help us here as well. Clear the list of requests and update the page with the game in the browser. Then check several requests which goes before the one with session id in the url. Nothing there. Next logical place to check is the html code of the page where the game is embedded. We are lucky this time. The session id is there in plain text located in flash vars.
'string_gatewayUrl' : 'http://ru5.forgeofempires.com/game/json?h=998898c26ea'
We need to load the game inside the bot to get the session id dynamically. This is easy to achieve with HTMLLoader

The last thing that we need is to get the ids of buildings dynamically. Just look at some first requests in Fiddler. There is one response from the server with relatively large payload. It contains the data to build the city. Everything that happens next is just routine procedure of investigating different JSON data structures and getting the one you need.

There are a lot of other features which can be added to the bot. Some of them can be pretty interesting, for example, the statistics of top users or guilds etc. Also, it’s relatively easy to build guild manager which will show so long-requested statistics who add goods to the guild’s treasury. However, there is no firm plans to support this bot in the future and to add new features.

Some images of my first city and positions in ratings:

my-city

The code for the FoE bot is available on GitHub. The code is no longer available. Please, read License and Readme before using it.

Some ideas how to find such bots or prevent their existence may come in the future (or may not 🙂 ).

25 Comments

  1. Niks

    Как я могу связаться с вами?

    • ingwe

      I have got your e-mail. However, as I have mentioned in the article – I will not provide any assistance. There is nothing really special about compiling that code on GitHub. However, you should start from some basics if you really want to get some good results.

      • Niks

        Возможно вы сможете ответить на некоторые вопросы, которые не будут выходить за рамки границ ваших принципов.

        • ingwe

          You may ask them here in comments. I will reply if I can/decide to. However, better ask in English, so everybody can read them.

          • Niks

            Как вы добились несколько ратуш разных эпох? Это заложено в функционал бота или это баги игры? Расскажите подробнее что еще возможно?
            Do you have some town halls of different eras ? It is in the functional robot or it bugs the game? Tell us more what else is possible?

    • Niks

      Спасибо я уже сделал это и теперь имею при компиляции всего 2 ошибки. Они связаны с тем что части кода нет как ы писали в статье, или причина другая?
      Изначально я хотел скомпилировать то что уже есть а позже искать недостающий код в файлах игры.
      Thank you, I’ve already done it . Now, when I get a compilation error of 2 .
      They are related to the code that you need from the game as you wrote in the article , or have other reasons?
      Initially, I wanted to compile what is already there and later seek the missing code in the game files
      C:\bot\src\UI.as
      Error:(19, 25) [FOEBot (module bot)]: Error code: 1017: The definition of base class HBox was not found.
      C:\bot\src\Collector.as
      Error:(11, 24) [FOEBot (module bot)]: Error code: 1172: Definition com.adobe.crypto:MD5 could not be found.

    • Niks

      Вы помните фильм Автостопом по галактике? Там был ответ на вопрос смысла жизни вселенной и вообще.. 42.
      В общем она, Идея то биш решила мне выдать новую ошибку на то что вы скинули. Я поместил это в папку с ботом, где уже были идентичные файлы.
      Error:(54, 3) [FOEBot (module bot)]: Unable to transcode ../../../../assets/pf_ronda_seven.swf.
      Error:(54, 3) [FOEBot (module bot)]: unable to resolve ‘../../../../assets/pf_ronda_seven.swf’ for transcoding
      Говорит она мне. Гугл говорит возможно это из за версии air sdk. А может карма у меня плохая.

  2. Niks

    Появилась проблема в intellij idea. Не могу найти нужную версию sdk и некий keystore файл.
    There was a problem in intellij idea. I can not find the required version of sdk and a keystore file.

    Build configuration ‘FOEBot’ (module ‘bot’): warning: AIR version in the app descriptor FOEBot-app.xml (4.0) doesn’t match AIR SDK version: 17.0
    Build configuration ‘FOEBot’ (module ‘bot’): signing options: keystore file not found: C:\..\..\..\Certificates\AIR\IngweLand.p12

    • ingwe

      Google for how to set up the SDK for the project and how to sign Adobe AIR apps. These are basic topics.

  3. Niks

    Спасибо я уже сделал это и теперь имею при компиляции всего 2 ошибки. Они связаны с тем что части кода нет как ы писали в статье, или причина другая?
    Изначально я хотел скомпилировать то что уже есть а позже искать недостающий код в файлах игры.
    Thank you, I’ve already done it . Now, when I get a compilation error of 2 .
    They are related to the code that you need from the game as you wrote in the article , or have other reasons?
    Initially, I wanted to compile what is already there and later seek the missing code in the game files
    C:\bot\src\UI.as
    Error:(19, 25) [FOEBot (module bot)]: Error code: 1017: The definition of base class HBox was not found.
    C:\bot\src\Collector.as
    Error:(11, 24) [FOEBot (module bot)]: Error code: 1172: Definition com.adobe.crypto:MD5 could not be found.

    • Niks

      Судя по ридми мои ошибки недостающий код пока не при чем.
      Я не програмист простой пльзователей, и это довольно трудно для меня. И я не спал уже 26 часов =)
      Judging from my mistakes readme missing code is not to blame.
      I’m not a programmer plzovateley simple , and it’s pretty hard for me . And I have not slept in 26 hours =)

  4. Niks

    Илья подскажите что должен делать этот класс class HBox ?

  5. Niks

    Не могу найти private function generateKey в swf из игры. Вы можете дать какую нибудь подсказку?

  6. Niks

    Оно находится в parser token strim?

  7. Niks

    Я Чувствую что он там. Но не могу ничего сделать потому что вообще не знаю ничего о программировании. ((( Столько усилий и все напрасно.
    Илья, что можно сделать чтобы получить этот файл servise.as?

  8. Niks

    Илья помогите пожалуйста.

  9. Yevgeniy

    Thanks for your job. Great work! It helps me to made bot without browser. It’s amazing!
    I wrote it in delphi 7. And also I add to the bot auto research function.
    Sorry for my bad english.

  10. Сергей

    Спасибо за теорию. Сделал бот [link removed]

  11. Is all this compiling of code going to cause a mishap in-game?

  12. Alex

    I was looking for an API to interface with the game and perform some simple tasks (a city planner of sorts) when I stumbled upon this article.

    After reading your article end to end, I have to admit that you have done a very great work in getting to the workings of the game. I thus commend you greatly for your in-depth investigation and wonderful description of the workings of the game.

    Unfortunately, I have not got enough time on my hands now to create a bot like you did.
    Hopefully, I will come back to it before the year is up.

    Is it okay with you if I contact you after completing my implementation?

    • Pete

      What happened to city_entities portion of the Startup.getData map? Where does the latest FoE game get them from and how – using Socket?
      I was able to display and render my maps as images until now – I can still get the city map data – all entities and their positions, but not the info about each entity – its size for example…
      Anyone has an idea how this is done now via API? Can you help?
      I just checked the City Planner and it can still get the current map for any user so either it’s not using the Startup.getData for it or has a way to get at city_entities info?

      • ingwe

        The data which you are looking for is there, but just in a bit different place. It’s not that hard to find it 😉

Trackbacks for this post

  1. What is wrong with you, Innogames?

Leave a Comment