Milepost GCC-Web Services

7:17 AM Nik 0 Comments

So finally after banging my head for almost two days against the Web Service of cTuning and CCC Framework, I have finally managed to get the combinations correct and am able to receive the Compiler Optimization flags now from the cTuning web services. There are a few glitches here and there, I think it requires that the platform, compiler and environment that one is using should be already present in the database. So I opened up the database and retrieved some records and used the platform, env and compiler ids from there to retrieve the predicted compiler flags and I am quite pleased to say that it is working now. Although, I see that using the flags generated, I am getting a worse runtime than by using normal -o2, -o3 levels :)..

Also, I read on the cTuning website that currently only compiler optimization flags are being predicted by the ML based compiler and optimization passes are not being predicted but there are plans to incorporate this too. I will try to follow up the authors of Milepost and find whether that part has been done, or is in the process.

Milepost GCC - Plugins

5:26 AM Nik 0 Comments

Here are some of the plugins that I have installed and a summary of what they do:

• save_executed_passes.legacy
  This plugin saves the executed passes per function in external files
  "ici_passes_function.txt"
• save_executed_passes_with_time.legacy
  This plugin times the execution of the passes split-compilation.
• substitute-passes.legacy
  This plugin substitutes original GCC pass order with the one read from either external files "ici_passes_function.txt", 1 global file "ici_passes_all.txt" or environment variable ICI_PASSES_ALL (passes are separated by comma) thus allowing external manipulation with passes (adding, removing or reordering).
• extract_program_static_features
  This plugin extracts program static features per function as vectors and saves them into "ici_features_function.txt"
  

The installation of these plugins was not very intuitive but I managed by tweaking the code here and there. The plugins are working fine now with GCC. The only problem I now face with Milepost is how to predict the optimization flags using the static program features which are extracted by the last plugin mentioned above. I believe (I may be wrong here) that we need to access the cDatabase and push our features in there, which then returns a set of optimization flags which can be used to compile with GCC. There are two ways to do this: using Web Services or using the CCC Framework (http://ctuning.org/wiki/index.php/CDatabase:Documentation:API). I have currently tried both but there are issues.

I cannot find a php script using which I can send a request to the "predict_opt" Web Service. Without this, its not possible to use the Web Service. The script for adding the optimization cases is available though.
With the CCC framework, I am unable to access the Web Service using sockets. This is the part where I am currently working. I have posted a request on the Discussions of the Community for making available the php script. Meanwhile, I'll try to get the CCC framework to work and return the predicted flags.

Milepost GCC - Introduction

5:00 AM Nik 1 Comments

Continuing with the work of the previous week, this week I started reading up about Milepost GCC. Some of the information can be found on this link: http://ctuning.org/wiki/index.php/CTools and the following paper: http://gcc-ici.sourceforge.net/papers/fmtp2008.pdf.

I am putting up here some of the extracts which I thought were important and which highlight the key concept of the Machine Learning based compiler:

1. Considerable speed-ups can be already obtained after iterative compilation on all platforms. However, this is a time-consuming process and different speed-ups across different platforms motivate the use of machine learning to automatically build specialized compilers and predict the best optimization flags or sequences of passes for different architectures.
   
   
2. Research has shown a great potential to improve program execution time or reduce code size by carefully selecting global compiler flags or transformation parameters using iterative compilation. The quality of generated code can also be improved by selecting different optimization orders. Milepost GCC's approach combines the selection of optimal optimization orders and tuning parameters of transformations at the same time.
   
   
3. Here is a description of some of the related tools and frameworks:
   
   
   • Continuous Collective Compilation Framework (CCC): A tool that generates the training examples for the Machine Learning tools. It does this by evaluating different compilation optimizations, storing execution time, code size and other metrics in a database.
   • Interactive Compilation Interface (ICI): The ICI provides opportunities for external control and examination of the compiler. The new version of ICI expands on the capabilities of its predecessor permitting the pass order to be modified. This version of ICI is used in the Milepost GCC to automatically learn good sequences of optimization passes.
   • PLUGINS: The features of the program (program structure) are extracted from Milepost GCC via a plugin and are also stored in the database. They are driven through shared libraries.
     
     
4. Working of Milepost GCC:
   
   
   • The plugins are invoked by the new -fici GCC flag or by setting ICI_USE environment variable to 1. When GCC detects these options, it loads a plugin (dynamic library) with a name specified by ICI_PLUGIN environment variable.
   • The machine learnt model predicts the best GCC optimization to apply to an input program based on its program structure or program features.
   • To extract these static program features an additional GCC pass 'ml-feat' is implemented. This pass can be called using a extract_program_static_features plugin after any arbitrary pass starting from FRE when all the GCC data necessary to produce features is ready.

There is still a lot of material to be read. I will give some details about the plugins I have installed in the next post. I believe Milepost GCC is an ingenious idea, however, as a user, I would have been happier if the documentation had been better. Currently I have to read the READMEs scattered around in the installation directories, and they are not very clear and specific.

Mozilla Builds

4:29 AM Nik 2 Comments

I have started working on a student project in Mozilla which requires using Milepost GCC to speed up Firefox. I started with doing a few builds of Firefox, both by using Mercurial and downloading the source code archive directly. I faced a few problems during the build process. After taking a lot of time for the build, it would crash on me giving errors like "vsnprintf not found", "fprintf not found" etc. After trying all possible options, I googled the errors and even found a related bug already filed:
https://bugzilla.mozilla.org/show_bug.cgi?id=485019
But the bug is marked "Resolved Invalid" (I have no idea what that means). Anyways, once I found that it was the breakpad which was causing problems, I disabled the crashreporter in the mozconfig file and that worked. So I managed to do some optimization and debug builds.

After that I moved onto Profile Guided Optimization Builds, which I believe Mozilla is currently using for its releases. A PGO build consists of two passes: a first pass to build binaries, then a second pass to re-build optimized binaries using profile information extracted by running the first binaries. I managed to successfully build firefox 3.6 with PGO.

I have moved on currently to studying Milepost GCC for now, but I'll be back to the builds to examine the makefiles and the code changes between normal and PGO builds, because I'll need to know how the builds actually work.

Where it all begins

7:49 AM Nik 1 Comments

So finally the day has come when I start blogging.. Had it been left to me, my hands wouldn't be typing this write now.. But well, duty calls, and thanks to Mozilla, I am now writing these seemingly monotonous words which must be making no sense to you all.. But I hope to improve soon and bring you some interesting tidbits.. So do keep up!