Documenting the life of…

Release Readiness Dashboard
Qb query Builder

The Release Readiness Dashboard chose Elastic Search as the source of Bugzilla data because it was fast and provided a convenient means of retrieving historic information for plotting trends. However, the native queries used to request data from Elastic Search clusters are long, ugly, and horrible JSON objects.

To solve the issue of dealing with ugly Elastic Search queries, the administrator of Bugzilla’s Elastic Search cluster has set up a helpful Javascript library for web developers. It acts as a middle layer for requesting data from the Elastic Search cluster using Qb queries instead of native Elastic Search queries. The RRDashboard uses the Qb Javascript library.

Qb queries, similar to Elastic Search queries, are JSON formatted objects. One might now ask the question of what Qb queries are, and how it has an advantage over native Elastic Search queries if both are just about as obscure as the other. Qb queries for one, are much shorter than native Elastic Search queries. They are also a lot more read-able.

Coming from an obscure background, the problem with writing Qb queries still persists. The release management team will have significant difficulty in adopting the RRDashboard if they are unable to form Qb queries that request the same data they are currently pulling using the Bugzilla search interface. I am now ready to introduce the Qb query builder.

When the release management team builds a query using the Bugzilla search interface, a long Bugzilla URL is generated containing the query parameters. As an input to the Qb query builder, this Bugzilla URL can be directly translated into a Qb query after specifying some other required parameters like the cluster to query against. In the background, here is what actually happens.

Upon submitting the form on the Qb query builder, a big Javascript event handler is triggered. The search parameters are extracted from the submitted Bugzilla URL, then re-appended to the Bugzilla search interface’s URL. Using a CURL request, we receive an enormous HTML string containing the Bugzilla search interface’s page filled with user-specified search parameters. Scraping then occurs on each individual page element to extract the search parameters, which are then appended into a JSON object formatted as a Qb query.

The benefit of such an implementation is that this is as simple as it will ever get for an end-user to work between Bugzilla and Qb queries. Unfortunately, there are also significant limitations to this implementation.

Scraping HTML of the Bugzilla search interface implies that the Qb query tool is very dependent on how the views of Bugzilla are generated. If the HTML layout of the Bugzilla search interface changes, the Qb query builder will lose its accuracy and start missing search parameters in the output Qb queries. A possible alternative is directly parsing the Bugzilla URL without doing a CURL request to get the HTML. However, this can be a potentially complex task to handle and will require more time beyond what I current have as an intern.

Another limitation to the Qb query builder is that it can be buggy. Bugzilla’s search interface is a mature product that can handle many different combinations of search parameters. The Qb query builder, on the other hand, is new and may not be able to handle as many varieties of querying as Bugzilla can. As a result, until the Qb query builder matures enough to handle all the corner cases that Bugzilla may throw at it, users of the tool may occasionally have to modify the output Qb queries manually to get what they want.

Ideas for Web Applications

Every once or twice in a 4 month term, I get a sudden rush of inspiration and ideas for cool things to take on as projects. This term at Mozilla there has been a lot of exposure with open sourcing, scraping the web for data and software tooling and automation. This lapse of ideas mostly revolve around those areas.

However, I usually get drained after the phase is done which is probably why I don’t feel like doing anything right now. I’ve already created repositories for these projects on GitHub and will contribute to whichever then interests me the most after this lazy phase is done.

  1. Tradester
  2. “Wouldn’t it be awesome to have a financial trading algorithm that anyone can write into and use freely?”. This thought came to me a few years ago when I finished first year at university. I wanted to start an open sourced script for an automated trading algorithm. The hope was for it to encompass all the smartness of traders everywhere into one algorithm for all who are smart enough to take advantage of it. Then I found MetaTrader4, a niche language which I never found the willpower to learn. So the idea died, or in terms of recent medical research, got put in animated suspension.

    Recently, I discovered that online brokers like Oanda and Robinhood (hopefully soon) are starting to offer REST APIs as one of their services to traders. Then it hit me that MT4 could be dropped completely by using common scripting languages like PHP instead. I.e. deploy algorithm script on any web hosting service, set-up CRON jobs, start automated trading. It also helps that web development languages are more popular than the obscure MT4, which is important considering that this will be open sourced.

  3. StoryLine
  4. This started about a year ago. The idea was for a common social platform where story writers could come together and collaboratively create new stories that were open sourced, like the GitHub for people who wrote in languages that were not meant for the computer.

    The project died mid-way when I realized that the product was taking a very bad approach to begin. It was made to try and take on a lot of stories and writers at the same time, which led to a very confusing user experience because of all the empty “social” views. I was working through the version control (back-end) component of StoryLine and Terry on the user interfaces when the project was killed.

    I recently found inspiration from looking at the WordPress model. The code is open sourced, so anyone can deploy and install their own independent instance of WordPress. At the same time, non-technical users can simply use wordpress.com where deployment is made simple. WordPress makes money from advertising on the wordpress.com sites and from writers who wish to export wordpress.com sites to their own hosting servers.

    Instead of a common platform that acts as GitHub for writers, StoryLine is likely to be better off as a deployable web application for all end-users. Each instance of StoryLine hosts a single, independent story. At some point later after maturity, a unified tool for users to create stories easily on awesome-name.storyline.com can be set up, much like the existing wordpress.com.

  5. Languify
  6. I love fontawesome. It’s open sourced, so simple, and adds so much value when used in the right context. Languify, like fontawesome, is an open sourced CSS library that contains commonly used words and phrases in different languages. Languify enables developers to create views that can be adapted to any language they want, just by the loading of a .css file.

    For example, <span class="lf lf-hello></span> shows “hello” when the “en.css” file is included on the page. When “en.css” is replaced with “zh.css”, the HTML page will display “你好” instead of “hello”. Aside from the Languify css file being loaded, no other changes to a HTML page are needed to have it adapted to a community that reads a different language.

    The repository currently contains some css files which have the words manually filled in. It is not scalable and I see it getting messy very quickly, especially if people start contributing to the library. For this project to get anywhere, a front facing tool has to be made to manage the addition/editing of words in the language library. The tool will then automatically generate the [language].css files that will be pushed into Github.

    Those are the project ideas that I will be building upon.
    If anything interests you, reach me!

Release Readiness Dashboard
Rules for Scoring

Looking back at groups of queries, we know that each individual group is a metric that the Release Management team cares about when deciding on the release readiness of that particular version. One of the key objectives of the dashboard is being able to automatically compute a release readiness score based on important metrics. In the screenshot above, we notice that two of the groups have titles with colored backgrounds (green), indicating the status of that group.

Statuses of each group on the dashboard are denoted by 3 primary colors of green, yellow, and red. To automatically determine the status of individual groups, computational rules must be scripted. In a future implementation, individual group statuses in each version will also be able to be aggregated for computing an overall release readiness score for the particular version.

To script a rule for any group, hover over the group and click on the tachometer like in the above screenshot. This will bring up a modal containing the boilerplate for scripting rules for that group as shown below. The user copies the JavaScript in the text area, and pastes it into a js file as shown in the specified directory. Scripting can begin immediately after that, with inline comments available to provide guidance if necessary. Note that if a user scripting a rule does not have deployment rights, the stack owner must be contacted at the end to upload the script file.

In the background, every time a version view is loaded the server checks for the existence of /assets/rules/rule_[group_id].js for each group that will be loaded. If found, the script file is loaded on the view and executed when data is returned from the Elastic Search cluster that mirrors Bugzilla. The resulting status color from executing the rule is then applied to its corresponding groups’ title. Using an architecture like that, we reap the benefits of:

  • Scripting of rules enabled for both default and custom groups.
  • Flexible scripting for defining rules with any level of complexity.
  • Upload into /assets/rules/rule_[group_id].js if group has rule, delete the file otherwise. i.e. scripting rules for a group is optional.
  • Anyone is free to access the boilerplate and start scripting a rule, but only the stack owner has deployment rights.

Mouse / Moose
Taste of Good Old Times

Today I meet Erasmus again after almost 4 long years. Back in Singapore I remember we would hang out with the usuals every other day for lunch/dinner/12am meals. How time has gone by since then. I heard about how everyone was doing well and we had a great time catching up.

Ever since moving to Canada I have often wondered about whether old friends will meet again and despite the years, still have great time hanging out. I occasionally get dreams about meeting my old friends and feeling awkward in the moment because we’ve all changed and been disconnected for so long. The image of these friends frozen in my mind exactly as they were when I last saw them years ago.

Yes, indeed we have changed. The friends we once thought were the best people ever have grown, matured, started new careers, gotten new experiences and built new lives. But friends will always be friends. Meeting with Erasmus today felt good, and sort of heartwarming. It’s like watching Toy Story 3 after many years since Toy Story 2, but way better because this is real life.

Release Readiness Dashboard
Groups of Queries

One of the key requirements for the Release Readiness Dashboard was to have multiple widgets of information. Each widget was to contain visuals for at least one data set. In addition, there was also a need for a numeric view that only displays the current count in cases when a view of historic data is not necessary. The above design was the result of those requirements.

Interpreted in terms of what is stored in the database, each grid on the view represents a group that can have multiple queries. I.e. a one-to-many relationship. Groups can be specified to be a plot, number, or both, eventually affecting how it will be loaded on the front view.

Queries are stored in the database as a JSON formatted string with Qb structure, and used to dynamically request for data from Elastic Search each time a page is loaded. No Bugzilla data is stored locally on the dashboard’s server. After the data is retrieved from Elastic Search and depending on the group’s type, a plot (or number) is then generated on the fly and appended onto the view.

Moving forward, another requirement was to have default groups that will display for every version of a specific product. Being linked to through a group.version_id meant that a new group had to be created every time a new version is being set up. The solution to this problem: Polymorphic objects.

With each group as a polymorphic object, we are able to link rows in the same group table to different entities like product and version. In terms of changes to the schema, we simply remove group.version_id and replace it with a generic group.entity and group.entity_id. Note that an actual FOREIGN KEY constraint cannot be reinforced here.

We are now able to define default groups by linking it to a product as a polymorphic object. However, because a single product group and its corresponding queries can now used across different versions, some version dependent text in fields like query.query_qb must be changed dynamically before it can be useful.

To solve this issue, soft tags like <version_tag> and <version_title> are stored in place of the version dependent text and replaced by the corresponding version’s values when being requested on the server.

So here we have it; Multiple groups that each display multiple data sets:

  • With the option to display the group as a plot or a current count.
  • With the option to display the group for an individual version (custom group) or for all versions of a product (default group).

 

 

Release Readiness Dashboard
Running with the Train

Simply put, the release readiness dashboard is an overview of various versions of Mozillian products Firefox, Fennec, and Firefox OS. It provides a trend (or current number) of the various kinds of bugs that the release management team cares about when determining whether or not a particular version is on track to be ready before the scheduled release date. The product versions appearing on the dashboard are displayed automatically by following the train model described in the rapid release calendar.

My first attempt to capture the train model in the dashboard’s database worked but was very inefficient and had serious redundancy issues. The following tables and fields are a part of the initial database schema:

  1. product – id, title
  2. version – id, title, product_id, central, aurora, beta, release, deprecate

For example, if the version that is currently on central is wanted then current timestamp is checked for being in between version.central and version.aurora, which are both timestamp data. Based on that logic, the scripts are able to automatically identify currently active versions.

However, this means an awful lot of timestamp comparisons and also a lot of the same dates being stored (in different coloumns of succeeding rows). How to better capture the train model so that the database is not storing redundant data?

After some brainstorming and discussions with both Bhavana and Lukas, a solution for a normalized data schema was found. See the above for the finalized solution in the form of an ER diagram.

On every reasonable period, a CRON job will be run. In the script, we check for a new train cycle by reading from an external data source. If a new cycle is found, it is insert as a new row in the cycle table. Following that, new rows are made in the version table for each product that will have a version entering the central channel. Now, the mapping begins. The newly created versions are mapped to the newly created cycle as being in the central channel. Previous versions are also bumped up a channel and mapped to the newly created cycle. The rows in the version_channel_cycle can be interpreted in English as the particular version that is in the particular channel for a particular cycle.

When retrieving data, this means that we only have to do in between timestamp comparisons once in the cycle table, then we get the cycle.id which is used for the other database queries. Convenient. In terms of storing data, this also means inserting rows every time a new cycle is entered, as opposed to updating version rows over and over again, which can get messy. The same dates are also stored only once now, so the redundancy is gone.

All things considered I’m quite happy with the new database schema. Big thanks to Lukas for pushing me to rethink my initial design and walking me through when I was lost.

Perhaps at some point in future I will write more about the other aspects of the database that deal with the storage and use of groups and queries. This project just keeps getting more awesome. Excited to see it go live.

 

LA Bamba to Los Angeles

This happened last weekend. Three friends and I made a road trip to Los Angeles. It was a fairly spontaneous one; all plans cooked and confirmed in under a week. We started the drive on Friday evening when we all finished our day at work and arrived 7 hours later at 2am. Here are some pictures to capture the moments at Universal Studios Hollywood, Walk of Fame, Venice beach, and other random bambas. Good times.

Notes:

  • Los Angeles feels quite a lot warmer than San Francisco, especially at night.
  • Venice beach is beautiful. There’s mountains, the ocean, a long coast.
  • Lots of houses are built on hills. Fancy and inconvenient at the same time.
  • Universal Studios is really nice and all, but the rides could be more exciting.
  • Jurassic Park’s soundtrack is the definition of epic. Gets stuck in my head.
  • Yup, it just started playing in my head again.
  • Places are more accessible in the bay area. Feels more organized.
  • LA is done now. What’s next?

Why I picked CodeIgniter over Laravel 4

Every web artist and artisan knows how awesome Laravel4 is by now. Blade templating, Object relational mapping, routing, built-in restful controllers? Sweeet. Having developed with CodeIgniter before, Laravel 4 was the coolest web thing ever when I first got to play around with it. I even wrote a 12 page academic report about how awesome Laravel 4 is.

Because everything seemed to make sense… until today when I had to spend over a full day trying to set it up.

I have just been put into a circumstance where I have to build in an environment I am quite unfamiliar with. Mozilla has a Platform-as-a-Service tool where employees like myself can freely deploy web applications. An instance of Stackato has been installed into Mozillian servers to provide this service.

Now, what is Stackato? To be honest, I’m still not really sure although I was already able to deploy a sample app with it.

Back on topic; Laravel 4. I was a big fan, and I still am because I do love all the developer features that come with the package. But just to get settled down with Laravel 4 there is so much setup to go through before one can even start writing code for the actual app.

  1. Installing Laravel 4? Nope, install composer first. Chances are that your PHP extensions or modules or something is fucked up somewhere too, so go find the problem; Configuration.
  2. Using Github? Sure, but you’ll be committing a tonne of junk like cached files and vendor add-ons until you figure out the stuff that should be git ignored; Configuration again.
  3. Putting the project online? You’ll have to find out how to point the document root into ../root/public. More configuration.

If one had a standard setup like cPanel + VPS hosting service, then it’s not so bad at all. But it really doesn’t help when Stackato and Laravel 4 (PHP Artisan and Composer) both use command line interfaces at the same time. Working with command lines for hours can be a real bitch.

Yeah, the first 15 minutes of a CLI feels good because you know what you’re doing, you’re feeling fresh and frankly a bit like a genius hacker. Then after that you slowly start to feel like you’re chasing a cloud that you are in the middle of inside your own head.

So that’s how I spent a good chunk of my time today and tonight. With all the doubts I’m having at this point, even if everything suddenly fell nicely in place I would still be extremely uncomfortable working on top of, and eventually delivering, an app that I’m so shaky about.

Taking two steps back, do I really need Laravel 4? My dashboard app will have only a few different views and API calls, so blade templating and routing will both only be marginally beneficial. My database is not super complex either so I don’t really need ORM.

Laravel 4, you’re out.

CodeIgniter, I choose you.

 

A day in San Francisco

Things I liked/enjoyed:

  • The Caltrain. It is very comfortable, is very fast, is on time, seems reliable.
  • Grilled onion cheeseburgers from McDonalds. Yes, I had McDonalds again.
  • Local public transit in SF. Two different and intertwining local transit services in one city. Convenient.
  • Going to San Francisco without any plans, and deciding only after getting there to go see the Golden Gate Bridge.
  • The Golden Gate Bridge. It looks super awesome especially when lit up at night. And I never knew it was built with panels hanging off cables on both sides. Loved the model that illustrated the engineering of it.
  • The people of San Francisco. Having no plans means asking for a lot of directions. People were super helpful. So many times the person I approach stops whatever he/she is doing, and for about 5 minutes try to help me find my way. So many times a person I didn’t approach overhears my question to someone else and blurts out a helpful answer / tip. Today couldn’t have been possible at all if it weren’t for those guys. Thank you kind friendly people of SF!
  • Reaching home after a nice, long, fulfilling day.

Things I didn’t like:

  • Streets and buses sometimes reeked very strongly of weed. Sketchy.
  • Local public transit in SF. Two different and intertwining local transit services in one city. Confusing.
  • The hilly landscape of the city. Actual roads are slightly different from the maps because of elevation.

Mozillian – Week 1

Firstly, I love the place I’m living at because there’s a gym, a hot spa, blah blah, and my bedroom is awesome. Mountain View is an awesome city to be in. The weather hovers around 16′C, and the streets are well maintained. In fact, the streets are very posh and upscale when compared to what I’m used to. Downtown MV is nicely decorated, full of techie looking people, and the air smells of internet.

There’s the online stories about geniuses that work at silicon valley giants who are the key innovators behind everything awesome that we use on the computer today. Here’s the thing: THEY ARE REAL. So the first day of work was of course, slightly intimidating because I know I didn’t know much.

But that feeling didn’t stay long. Almost everyone I’ve met so far will freely share their domain knowledge if you ask nicely. Nobody, regardless of awesomeness, snobs or condescends. Generally speaking, anything I want to learn can be learned from someone. And they all love what they do. Passion for technology on this scale is hard to come across. My mentor has also been super nice in helping me out with answering my bazillion questions. Thanks Bhavana!

Mozilla uses IRC as the primary communication tool (I know right, old school cool tingz), so everyone hangs out in online chat rooms. Takes me back to the day when I was 12; sitting in the computer lab typing “:D” a million times in a virtual chat room full of classmates no more than 5 tables away from me. I was young, computers were slow, webpages had ridiculous styles. I digress.

Anyway, I joined the Release Management team (#RelMan ftw). Mozilla’s software products, like every other piece of software, unfortunately has bugs. The RelMan people identify the bugs that are critical in nature and track them, making sure that they get fixed before the version is eventually released. In event that a bug still persists despite an approaching release date, the team has to make a call on whether or not the feature/patch with the bug will be shipped in that release.

Of course there is more to release management than that, and an entire book can probably be written on the subject and I won’t be the author (at least not yet), but I hope the provided description gives a general idea of what release management is about.

When monitoring bugs and making decisions, there is a whole array of data that has to be interpreted by the team. At the moment, it is my understanding that the operations and processes are relatively manual. Each set of data is manually pulled up and analyzed separately. There is opportunity to perform better with the introduction of a dashboard that automates the data-pulling, data manipulation into useful information, and finally presenting it all nicely to the team. My job is to design and build that dashboard.

Oh so you think Mozilla’s going to put me in trouble for telling you about RelMan? False. Mozilla actually encourages employees to share their work with the world. Even the IRC chat rooms that employees use to discuss work are public. I’m not quite used to this level of openness either but that’s cool.

Anyway it’s 2 in the morning and I am exhausted.

Brain dump:

  • I don’t think that last gif fits here. I don’t know why I included it.
  • My toothbrush sucks; might as well stick a blade in my mouth.
  • $5 for 20 chicken McNuggets?! Awesome job McDonalds!
  • When building software, it’s usually 3 choose 2; Features, Quality, Time.
  • Nearest McDonalds to my place is a 40 minute walk? FFFFFF you McDs.
  • If you don’t understand the context, nevermind. FFFFFF is just white.
  • I loved today’s company lunch. We had very awesome dimsum.
  • The best sushi, is not all-you-can-eat sushi.