1. No category

fjord Documentation
Release 1.0.dev
Mozilla
January 20, 2015
Contents
1
Overview
2
Contents
2.1 Join this project! . . . . . . . . . . . . . .
2.2 Getting started . . . . . . . . . . . . . . .
2.3 Development workflow . . . . . . . . . . .
2.4 Project Conventions (git, l10n, Python, etc.)
2.5 Testing in Fjord . . . . . . . . . . . . . . .
2.6 Maintaining the db . . . . . . . . . . . . .
2.7 Maintaining ElasticSearch . . . . . . . . .
2.8 Maintaining l10n . . . . . . . . . . . . . .
2.9 Maintaining Vendor Library . . . . . . . .
1
.
.
.
.
.
.
.
.
.
3
3
4
10
11
14
16
18
19
21
3
Part 2: Using the Site
3.1 Feedback URLs . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
3.2 API . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
3.3 Heartbeat API . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
25
25
26
35
4
Part 3: Maintaining The Site
4.1 Maintaining Fjord and Input . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
4.2 Service Level Agreement . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
41
41
44
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
i
ii
CHAPTER 1
Overview
Fjord is the software that runs Mozilla Input. The site has two sides to it: one for gathering product feedback and user
sentiment and the other for exposing that data and analytics on it. The Mozilla Input feedback site helps us understand
what 450 million users are thinking and makes it possible for us to catch issues we wouldn’t otherwise be able to know
about.
This is at heart a Django-powered site. It uses a MySQL database for data storage and an Elasticsearch cluster for
search, analytics and facets.
Project details:
wiki https://wiki.mozilla.org/Firefox/Input
code https://github.com/mozilla/fjord
new bugs https://bugzilla.mozilla.org/enter_bug.cgi?product=Input&rep_platform=all&op_sys=all
irc #input on irc.mozilla.org
mailing list https://mail.mozilla.org/listinfo/input-dev
get involved https://wiki.mozilla.org/Webdev/GetInvolved/input.mozilla.org
1
fjord Documentation, Release 1.0.dev
2
Chapter 1. Overview
CHAPTER 2
Contents
Want to contribute to Input? Then this is the section for you! Start with Join this project! which will guide you through
the rest.
2.1 Join this project!
Fjord is the software that runs Input (input.mozilla.org) which is one of several tools the Mozilla community uses
to gather user sentiment on Mozilla products. This data helps us track down problems, guide new development and
generally make Mozilla products better for users using them.
Interested in helping out? I hope so!
Generally we need help with:
1. Writing code, fixing bugs and writing tests.
2. Localizing.
3. Testing.
4. Improving the Fjord documentation.
5. Making Fjord easier for other people to contribute to.
Before contributing to Fjord, you should make sure you have done the following things:
1. Create a GitHub account. If you don’t want to, that’s ok, but it does make the workflow a little harder. If you’re
new to git or development, we suggest you create a GitHub account since it’ll reduce some of the complexities
in submitting changes.
2. Join the input-dev mailing list. We send status reports, development environment change notices and other
things to the mailing list. Plus it keeps an archive and it makes it possible for us to work together even if we’re
not all in the same time zone. Plus this is the best way to get help when you’re having trouble.
3. Say “hi!” on the #input IRC channel on irc.mozilla.org. If no one says anything back, it’s possible no one
is awake. However, if people are awake, someone will definitely reply and this is one of the best ways to get
familiar with who’s who and one of the places you can go to get help when you’re having trouble.
After you’ve done all that, it’s time to set up a development environment. For that, see the Getting started.
After that, check out our GetInolved page which lists mentored bugs.
If you have problems, please let us know! See Overview.
3
fjord Documentation, Release 1.0.dev
2.2 Getting started
•
•
•
•
•
•
•
•
•
Getting the requirements
Download Fjord
Fix the version mismatch between Virtualbox and the Virtualbox Guest Additions
Building a vm and booting it
Setting up Fjord
Editing code and running it
Getting more sample data
Virtual machine maintenance commands
manage.py commands
– generatedata
– runserver
– collectstatic
– test
– migrate
– shell
– esreindex and esstatus
– update_product_details
– ihavepower
• Helpful tips
– Flushing the cache
– Issues with commit timestamps
• Where to go from here?
This guide walks through getting a development environment set up to allow you to contribute to Fjord.
Note: This definitely works for Mac OSX and Linux and probably works for Windows, too. If you have issues, please
let us know.
2.2.1 Getting the requirements
1. Download and install git if you don’t have it already: http://git-scm.com/
Note: Windows users: When you install git, make sure to choose “Checkout as-is, commit Unix-style line
endings”. If you don’t, then you’ll end up with Windows-style line endings in your checkout and Fjord won’t
work in the virtual machine.
2. Download and install VirtualBox if you don’t have it already: https://www.virtualbox.org/
3. Download and install Vagrant if you don’t have it already: http://www.vagrantup.com/
2.2.2 Download Fjord
If you have a GitHub account, you should fork the Fjord repository and then clone your fork:
git clone https://github.com/<USERNAME>/fjord.git
If you do not have a GitHub account, that’s ok! Clone the official Fjord repository this way:
4
Chapter 2. Contents
fjord Documentation, Release 1.0.dev
git clone https://github.com/mozilla/fjord.git
This creates a directory called fjord/. We’ll call that the “Fjord repository top-level directory”.
2.2.3 Fix the version mismatch between Virtualbox and the Virtualbox Guest Additions
It’s likely there will be a version mismatch between the version of Virtualbox you’re using and the version of the
Virtualbox Guest Additions in the image you’re going to use.
If you’re using Mac OSX or Linux, you can do this from the Fjord repository top-level directory:
./bin/vagrant_fix_guest_additions.sh
If you’re not using Mac OSX or Linux or that doesn’t work, then you can do it by hand:
# Installs the VirtualBox Guest plugin
vagrant plugin install vagrant-vbguest
# Creates and launches the vm without provisioning it
vagrant up --no-provision
# Runs a command in the vm to remove the packages in the vm that
# are the cause of the mismatch
vagrant ssh -c ’sudo apt-get -y -q purge virtualbox-guest-dkms virtualbox-guest-utils virtualbox-gues
# Halts the vm so you can move onwards
vagrant halt
After that, the versions of the two things should be the same and you should be good to go to the next step.
2.2.4 Building a vm and booting it
This will build a VirtualBox vm using Vagrant. The vm will have Ubuntu Linux 14.04 installed in it. Fjord works in
this environment.
Run this commands in the Fjord repository top-level directory:
vagrant up
It takes a while the first time you do this since it has to create the virtual machine and provision it. First it downloads
an Ubuntu Linux 14.04 image (~300mb), then it installs some software in this image like MySQL and Elasticsearch.
Then it sets things up so that Fjord runs in this VM using the files on your machine. This allows you to use whatever
editor you like on your machine to edit code that runs in the VM without having to copy files around.
2.2.5 Setting up Fjord
After you’ve created a vm, we have some minor setup to do:
# Create a shell on the guest virtual machine
vagrant ssh
# Change into the fjord/ directory
cd ~/fjord
2.2. Getting started
5
fjord Documentation, Release 1.0.dev
First we download all the product detail data:
# Update product details
./manage.py update_product_details -f
Then we set up the database:
# Create the database and a superuser
./manage.py syncdb
# Run the db migrations
./manage.py migrate
It will ask if you want to create a superuser. You totally do! Create a superuser that you’ll use to log into Fjord.
The username and password don’t matter, but the email address does. You must choose an email address that is your
Persona identity. If you don’t have a Persona identity, you can create one at the Persona site.
Note: You can create a superuser at any time by doing:
./manage.py createsuperuser
Additionally, you can convert any account into a superuser account by doing:
./manage.py ihavepower <email-address>
After that, let’s generate some data in the database so that we have something to look at. We’ll then need to index that
data so it shows up in searches.
# Generate sample data
./manage.py generatedata
# Index the sample data into Elasticsearch
./manage.py esreindex
That’s it!
2.2.6 Editing code and running it
Fjord is a Django project. We use the Django runserver to run the website to test it.
First, if you haven’t got a running virtual machine, launch it with:
vagrant up
Then, ssh into the virtual machine:
vagrant ssh
This gives you a shell in the virtual machine that lets you run all the Django commands, run the test suite, etc.
To launch the Django runserver, use the vagrant ssh shell and do:
cd ~/fjord
./manage.py runserver 0.0.0.0:8000
Then on your host computer, use your browser and go to http://127.0.0.1:8000. You should see Fjord.
6
Chapter 2. Contents
fjord Documentation, Release 1.0.dev
2.2.7 Getting more sample data
Sample data is tied to a specific moment in time. You’ll need to run the generatedata command every time you need
fresh data.
The generatedata command only generates data and saves it to the db. After running generatedata, you’ll need to add
that data to the Elasticsearch index:
./manage.py generatedata
./manage.py esreindex
Note: You can call generadata as many times as you like.
2.2.8 Virtual machine maintenance commands
Command to run on host
vagrant up
vagrant ssh
vagrant halt
vagrant status
vagrant destroy
Explanation
Launches the vm
SSHs to the vm
Halts the vm
Status of the vm
Destroys the vm (not recoverable!)
See more in the Vagrant documentation. If you have questions, let us know.
2.2.9 manage.py commands
You can see the complete list of ./manage.py commands by typing:
./manage.py
For each command, you can get help by typing:
./manage.py <COMMAND> --help
We use the following ones pretty often:
Command
generatedata
runserver
collectstatic
test
migrate
shell
esreindex
esstatus
update_product_details
ihavepower
Explanation
Generates fake data so Fjord works
Runs the Django server
Collects static files and “compiles” them
Runs the unit tests
Migrates the db with all the migrations specified in the repository
Opens a Python REPL in the Django context for debugging
Reindexes all the db data into Elasticsearch
Shows the status of things in Elasticsearch
Updates the product details with the latest information
Turns a user account into a superuser
generatedata
You can get sample data in your db by running:
./manage.py generatedata
2.2. Getting started
7
fjord Documentation, Release 1.0.dev
This will generate 5 happy things and 5 sad things so that your Fjord instance has something to look at.
If you want to generate a lot of random sample data, then do:
./manage.py generatedata --with=samplesize=1000
That’ll generate 1000 random responses. You can re-run that and also pass it different amounts. It’ll generate random
sample data starting at now and working backwards.
runserver
Vagrant sets up a forward between your host machine and the guest machine. You need to run the runserver in a way
that binds to all the ip addresses.
Run it like this:
./manage.py runserver 0.0.0.0:8000
collectstatic
When you’re running the dev server (i.e. ./manage.py runserver ...), Fjord compiles the LESS files to CSS
files and serves them individually. When you’re running Fjord in a server environment, you run:
./manage.py collectstatic
to compile the LESS files to CSS files and then bundle the CSS files and JS files into single files and minify them. This
reduces the number of HTTP requests the browser has to make to fetch all the relevant CSS and JS files for a page. It
makes our pages load faster.
However, a handful of tests depend on the bundles being built and will fail unless you run collectstatic first.
test
The test suite will create and use this database, to keep any data in your development database safe from tests.
Before you run the tests, make sure you run collectstatic:
./manage.py collectstatic
I run this any time I run the tests with a clean database.
The test suite is run like this:
./manage.py test
For more information about running the tests, writing tests, flags you can pass, running specific tests and other such
things, see the test documentation.
migrate
Over time, code changes to Fjord require changes to the database. We create migrations that change the database from
one version to the next. Whenever there are new migrations, you’ll need to apply them to your database so that your
database version is the version appropriate for the codebase.
To apply database migrations, do this:
8
Chapter 2. Contents
fjord Documentation, Release 1.0.dev
./manage.py migrate
For more information on the database and migrations, see Maintaining the db.
shell
This allows you to open up a Python REPL in the context of the Django project. Do this:
./manage.py shell
esreindex and esstatus
Fjord uses Elasticsearch to index all the feedback responses in a form that’s focused on search. The front page
dashboard and other parts of the site look at the data in Elasticsearch to do what they do. Thus if you have no data in
Elasticsearch, those parts of the site won’t work.
To reindex all the data into Elasticsearch, do:
./manage.py esreindex
If you want to see the status of Elasticsearch configuration, indexes, doctypes, etc, do:
./manage.py esstatus
update_product_details
Event data like Firefox releases and locale data are all located on a server far far away. Fjord keeps a copy of the
product details local because it requires this to run.
Periodically you want to update your local copy of the data. You can do that by running:
./manage.py update_product_details
ihavepower
If you create an account on Fjord and want to turn it into a superuser account that can access the admin, then you need
to grant that account superuser/admin status. To do that, do:
./manage.py ihavepower <email-address>
2.2.10 Helpful tips
Flushing the cache
We use memcached for caching. to flush the cache, do:
echo "flush_all" | nc localhost 11211
2.2. Getting started
9
fjord Documentation, Release 1.0.dev
Issues with commit timestamps
The Ubuntu image that we are using, has UTC as the configured timezone. Due to this, if you are in a different
timezone and make commits from the VM, the commit timestamps will have a different timezone when compared to
the timezone on the host computer. To have matching timezone on the host and the VM, run:
sudo dpkg-reconfigure tzdata
and select your current timezone as the timezone for the VM.
2.2.11 Where to go from here?
Project Conventions (git, l10n, Python, etc.) covers project conventions for Python, JavaScript, git usage, etc.
Development workflow covers the general workflow for taking a bug, working on it and submitting your changes.
Maintaining the db covers database-related things like updating your database with new migrations, creating migrations, etc.
Maintaining ElasticSearch covers Elasticsearch-related things like maintaining your Elasticsearch index, reindexing,
getting status, deleting the index and debugging tools.
Maintaining l10n covers how we do localization in Fjord like links to the svn repository where .po files are stored,
Verbatim links, getting localized strings, updating strings in Verbatim with new strings, testing strings with Dennis,
linting strings, creating new locales, etc.
Testing in Fjord covers testing in Fjord like running the tests, various arguments you can pass to the test runner to
make debugging easier, running specific tests, writing tests, the smoketest system, JavaScript tests, etc.
Maintaining Vendor Library covers maintaining vendor/ and the Python library dependencies in there.
2.3 Development workflow
2.3.1 Taking a bug, working on it, sending in changes
You found a bug you want to work on! Great! Here’s how it works:
1. Comment in the bug that you want to take it. We’ll talk with you and if it sounds like it’s a good fit, we’ll assign
it to you.
2. Create a new branch. I use the bug number, but it doesn’t matter what the branch is named. For example:
git checkout -b 1026503-vagrant-work
3. Work on the changes. Commit your changes to your branch.
4. Go through your changes and make sure they respect our project conventions. They’re listed in Project Conventions (git, l10n, Python, etc.).
FIXME - We should add a script that lints PEP-8, jslint, etc issues so we don’t have to think about those.
5. Send the changes in.
If you have a GitHub account:
(a) Push your branch to your fork on GitHub.
(b) Create a Pull Request.
If you don’t have a GitHub account:
10
Chapter 2. Contents
fjord Documentation, Release 1.0.dev
(a) Use git format-patch to generate a patch:
git format-patch --stdout master > bug_<id>.patch
(b) Attach the patch to the bug.
6. Then we’ll talk about the changes and when they’re good to land, someone with commit permissions will land
them.
2.4 Project Conventions (git, l10n, Python, etc.)
• Coding conventions
– Git pre-commit hook
– Python conventions
– JavaScript
• Git conventions
– Git commit messages
• Localization conventions
– Strings
– Good Practices
– Testing strings
This document contains coding conventions, and things to watch out for, etc.
2.4.1 Coding conventions
Git pre-commit hook
We have a Git pre-commit hook that makes it easier to make sure you’re checking in linted code. To set it up, run:
$ ./bin/hooks/lint.pre-commit
That’ll set up the pre-commit hook. After that, every time you commit something in Git, it’ll run the hook first and if
everything is fine continue with the commit. If things are not fine, it’ll notify you and stop.
Python conventions
Follow PEP-8 for code and PEP-257 for docstrings.
If you don’t have an editor that checks PEP-8 issues and runs pyflakes for you, it’s worth setting it up.
You can also use the linting script bin/flake8_lint.sh:
$ ./bin/flake8_lint.sh <files-to-lint>
JavaScript
Use jshint for JavaScript code. This is automatically done in the pre-commit hook.
Use jsdoc for JavaScript function documentation.
2.4. Project Conventions (git, l10n, Python, etc.)
11
fjord Documentation, Release 1.0.dev
2.4.2 Git conventions
Git commit messages
Git commit messages should have the following form:
[bug xxxxxxx] Short summary
Longer explanation with paragraphs and lists and all that where
each line is under 72 characters.
* bullet 1
* bullet 2
Etc. etc.
Summary line should be capitalized, short and shouldn’t exceed 50 characters. Why? Because this is a convention
many git tools take advantage of.
If the commit relates to a bug, the bug should show up in the summary line in brackets.
Lines shouldn’t exceed 72 characters.
See these guidelines for some more explanation.
2.4.3 Localization conventions
Strings
You can localize strings both in Python code as well as Jinja templates.
In Python:
from tower import ugettext as _, ugettext_lazy as _lazy
yodawg = _lazy(’The Internet’)
def myview(request):
return render(’template.html’, {msg=_(’Hello World’), msg2=yodawg})
_lazy is used when we are not in scope of a request. This lets us evaluate a string, such as yodawg, at the last possible
second when we finally can draw upon the request’s context. E.g. in a template:
{{ msg2 }}
Will be evaluated to whatever The Internet is in the requester’s locale.
In Jinja we can use the following syntax for localized strings:
<h1>{{ _(’Hello’) }}</h1>
{% trans link=’http://mozilla.org’ %}
<p>Go to this <a href="{{ link }}">site</a>.</p>
{% endtrans %}
Good Practices
Let’s say you have some template:
12
Chapter 2. Contents
fjord Documentation, Release 1.0.dev
<h1>Hello</h1>
<p>
Is it <a href="http://about.me/lionel.richie">me</a> you’re
looking for?
</p>
Let’s say you are told to translate this. You could do the following:
{% trans %}
<h1>Hello</h1>
<p>
Is it <a href="http://about.me/yo">me</a> you’re looking for?
</p>
{% endtrans %}
This has a few problems, however:
1. It forces every localizer to mimic your HTML, potentially breaking it.
2. If you decide to change the HTML, you need to either update your .po files or buy all your localizers a nice gift
because of all the pain you’re inflicting upon them.
3. If the URL changes, your localizer has to update everything.
Here’s an alternative:
<h1>_(’Hello’)</h1>
<p>
{% trans about_url=’http://about.me/lionel.richie’ %}
Is it <a href="{{ about_url }}">me</a> you’re looking for?
{% endtrans %}
</p>
or if you have multiple paragraphs:
<h1>_(’Hello’)</h1>
{% trans about_url=’http://about.me/lionel.richie’ %}
<p>
Is it <a href="{{ about_url }}">me</a> you’re looking for?
</p>
<p>
I can see it in your eyes.
</p>
{% endtrans %}
Here are the advantages:
1. Localizers have to do minimal HTML.
2. The links and even structure of the document can change, but the localizations can stay put.
Be mindful of work that localizers will have to do.
See also:
http://playdoh.readthedocs.org/en/latest/userguide/l10n.html#localization-l10n Localization (l10n) in the Playdoh docs
2.4. Project Conventions (git, l10n, Python, etc.)
13
fjord Documentation, Release 1.0.dev
Testing strings
Fjord comes with bin/test_locales.sh script which makes it pretty easy to test that strings in the user interface
are getting gettext’d. It creates a faux “Pirate” translation of the strings in the xx locale.
You need to install polib for the script to work:
$ pip install polib
After that, cd into the project directory and do:
$ bin/test_locales.sh
After that runs, you can see what happened by doing:
$ ./manage.py runserver 0.0.0.0:8000
and going to http://127.0.0.1:8000/xx/.
2.5 Testing in Fjord
Tests in Fjord allow us to make changes and be reasonably sure that the system continues to work. Further, they make
it easier to verify correctness for behavioral details.
2.5.1 Running tests
Setup
Before you run the tests, you have to run the collectstatic command. This compiles LESS files to CSS files and
creates the bundles that some of the tests require to run. If you don’t do this, then a few of the tests will fail.
To run collectstatic, do:
./manage.py collectstatic
You don’t have to do this often. I’d do it the first time and then any time you run the tests with a fresh db.
FIXME: This is annoying and it’d be nice to get it fixed.
Running tests and arguments
To run the test suite, do:
./manage.py test
The NOSE_ARGS setting in fjord/settings/local.py sets some default arguments so you don’t see tons and
tons and tons and tons and tons and tons and tons and tons and tons and tons and tons and tons and tons of debugging
output. Ew.
If you ever need to see that debugging, comment the arguments out.
The --nocapture flag is important if you want to be able to drop into PDB from within tests.
Other helpful flags when debugging are:
-x: Fast fail. Exit immediately on failure. No need to run the whole test suite if you already know something is
broken.
14
Chapter 2. Contents
fjord Documentation, Release 1.0.dev
--pdb: Drop into PDB on an uncaught exception. (These show up as E or errors in the test results, not F or failures.)
--pdb-fail: Drop into PDB on a test failure. This usually drops you right at the assertion.
The test suite will create a new database named test_%s where %s is whatever value you have for
settings.DATABASES[’default’][’NAME’].
If you know there haven’t been any schema changes, you can run the tests with REUSE_DB=1 in the environment.
This will reuse the existing database:
REUSE_DB=1 ./manage.py test -s --noinput --logging-clear-handlers
Further, running the test suite can be a good way to suss out Python warnings and 2to3 problems. To do that, run the
test suite like this:
python -t -3 -Wd ./manage.py test
Argument explanations:
-t Issue warnings about inconsistent tab usage.
-3 Warns about Python 3.x incompatabilities that 2to3 can’t fix.
-Wd Enables default warnings.
See python -h for details and other arguments you can use.
Running specific tests
You can run part of the test suite by specifying the directories of the code you want to run, like:
./manage.py test fjord/feedback/tests
You can specify specific tests:
./manage.py test fjord.feedback.tests.test_ui:TestFeedback.test_happy_url
See the output of ./manage.py test --help for more arguments.
2.5.2 Writing New Tests
Code should be written so it can be tested, and then there should be tests for it.
When adding code to an app, tests should be added in that app that cover the new functionality. All apps have a tests
module where tests should go. They will be discovered automatically by the test runner as long as the look like a test.
• Avoid naming test files test_utils.py, since we use a library with the same name.
test__utils.py instead.
Use
• If you’re expecting reverse to return locales in the URL, use LocalizingClient instead of the default
client for the TestCase class.
• We
use
FactoryBoy
to
generate
model
instances
instead
of
using
fixtures.
fjord.feedback.tests.ResponseFactory generates fjord.feedback.models.Response
instances.
• To add a smoketest, see the README.rst file in the smoketests/ directory.
2.5. Testing in Fjord
15
fjord Documentation, Release 1.0.dev
2.5.3 Writing New JavaScript Tests
JavaScript tests are not run in our normal unit test suite. Instead we have a different test system.
We test JavaScript utility functions using QUnit.
These tests are located in fjord/base/static/tests/.
Running tests
Launch the server with:
./manage.py runserver
Then go to:
http://127.0.0.1:8000/static/tests/index.html
(You might have to use a different protocol, host and port depending on how you have Fjord set up.)
Adding tests
To add a new test suite, add a couple of script lines to index.html in the relevant place and then create a new
test_FILENAMEHERE.js file with your QUnit tests.
2.5.4 Changing tests
Unless the current behavior, and thus the test that verifies that behavior is correct, is demonstrably wrong, don’t change
tests. Tests may be refactored as long as its clear that the result is the same.
2.5.5 Removing tests
On those rare, wonderful occasions when we get to remove code, we should remove the tests for it, as well.
If we liberate some functionality into a new package, the tests for that functionality should move to that package, too.
2.6 Maintaining the db
2.6.1 Updating the db
If someone pushes changes that change the db, you’ll need to apply the new migrations to your db. Do this:
./manage.py syncdb
./manage.py migrate
2.6.2 Creating a schema migration
We use South for database migrations.
To create a new migration the automatic way:
1. make your model changes
16
Chapter 2. Contents
fjord Documentation, Release 1.0.dev
2. run:
./manage.py schemamigration <app> --auto
where <app> is the app name (base, feedback, analytics, ...).
3. run the migration on your machine:
./manage.py migrate
4. run the tests to make sure everything works
5. add the new migration files to git
6. commit
See also:
http://south.readthedocs.org/en/latest/tutorial/index.html South tutorial
2.6.3 Creating a data migration
Creating data migrations is pretty straight-forward in most cases.
To create a data migration the automatic way:
1. run:
./manage.py datamigration <app> <name>
where <app> is the app name (base, feedback, analytics, ...) and <name> is the name of the migration
2. edit the data migration you just created to do what you need it to do
3. add the new migration file to git
4. commit
See also:
http://south.readthedocs.org/en/latest/tutorial/part3.html#data-migrations South tutorial: data migrations
Backwards migrations
Make sure to write backwards code if you can. If there’s no way to undo the migration, then do this:
def backwards(self, orm):
raise RuntimeError("Cannot reverse this migration.")
Data migrations for data in non-Fjord apps
If you’re doing a data migration that adds data to an app that’s not part of Fjord, but is instead a library (e.g. djangowaffle), then create the data migration in the base app and make sure to freeze the library app so that it’s available.
For example, this creates a waffle flag:
./manage.py datamigration base create_gengo_switch --freeze waffle
2.6. Maintaining the db
17
fjord Documentation, Release 1.0.dev
2.7 Maintaining ElasticSearch
2.7.1 Things to know about ElasticSearch
Input uses ElasticSearch to power search.
Input uses ElasticUtils to interface with ElasticSearch.
2.7.2 Command line tools
esreindex: indexing
Do a complete reindexing of everything by:
./manage.py esreindex
This will delete the existing index specified by ES_INDEXES, create a new one, and reindex everything in your
database.
If you need to get stuff done and don’t want to wait for a full indexing, you can index a percentage of things.
For example, this indexes 10% of your data ordered by id:
./manage.py esreindex --percent 10
This indexes 50% of your data ordered by id:
./manage.py esreindex --percent 50
I use this when I’m fiddling with mappings and the indexing code.
You can also specify which models to index:
./manage.py esreindex --models feedback_simple
See --help for more details:
./manage.py esreindex --help
Note: If you kick off indexing with the admin, then indexing gets done in chunks by celery tasks. If you need to halt
indexing, you can purge the tasks with:
./manage.py celeryctl purge
If you purge the tasks, you need to cancel outstanding records. Run this sql in mysql:
UPDATE search_record SET status=2 WHERE status=0 or status=1;
If you do this often, it helps to write a shell script for it.
esstatus: health/statistics
You can see ElasticSearch index status with:
./manage.py esstatus
This lists the indexes, tells you which ones are set to read and write, and tells you how many documents are in the
indexes by mapping type.
18
Chapter 2. Contents
fjord Documentation, Release 1.0.dev
esdelete: deleting indexes
You can use the search admin to delete the index.
On the command line, you can do:
./manage.py esdelete <index-name>
2.7.3 Live Indexing
When you add data to the database, it needs to be added to the index. If the setting ES_LIVE_INDEX is True, then
this will be handled automatically in the post_save hook as long as celery tasks are being handled.
For celery tasks to be handled, you have to either have CELERY_ALWAYS_EAGER set to True, or have at least one
celery worker running, and RabbitMQ working.
Index Maintenance
If you don’t want live indexing, you can also reindex everything using the admin or using the esreindex command-line
tool, as detailed above.
2.7.4 Debugging tools
See ElasticUtils documentation for debugging tools and tips.
2.8 Maintaining l10n
2.8.1 Things to know about l10n
1. translations are done in Verbatim
2. translators are looking at our -dev environment to see the strings in use
3. the -dev environment should have a cronjob kicking off every 10 minutes that updates the svn locale repository
so that translators can see the latest stuff
svn url https://svn.mozilla.org/projects/l10n-misc/trunk/input/locale
viewvc svn http://viewvc.svn.mozilla.org/vc/projects/l10n-misc/trunk/input/locale/
Verbatim https://localize.mozilla.org/projects/input/
2.8.2 Installing localizations from svn
Localizations are not stored in this repository–they’re in Mozilla’s subversion repository.
To get the localizations, do:
cd locale
svn checkout https://svn.mozilla.org/projects/l10n-misc/trunk/input/locale locale
To see the translated strings, you need to first compile the .po files to .mo files:
2.8. Maintaining l10n
19
fjord Documentation, Release 1.0.dev
./bin/compile-mo.sh locale/
2.8.3 Extract/merge/sync strings with svn
First, ping Kadir (or whoever else) and tell him/her we’re merging strings (i.e. extracting them from our codebase and
committing the changes to svn).
Then, update the svn repository:
cd locale
svn update
cd ..
Then, extract the strings from the codebase:
./manage.py extract
./manage.py merge
Then, commit the new strings to svn:
cd locale
svn commit -m ’Input strings update YYYY/MM/DD’
Then, tell him/her that the strings have been extracted. He/she will update Verbatim for all locales.
Note: There are also instructions in ./locale/README.txt.
After that, send an email to the dev-l10n-web mailing list using this template:
Hi!
I’ve just pushed new strings to Verbatim for Input.
https://localize.mozilla.org/projects/input/
< explanation of strings changes here >
To give you an idea of how many untranslated strings there are, I
have computed this list:
< copy and paste output of "./bin/stats-po.sh locale/" >
If you have any questions, please reply to the list.
Thank you so much for your help!
Input development team
Sometimes we have to create bugs for each individual locale to update the translations. It’s nice to copy/paste the
specific strings that need to be translated.
Sometimes the translator says they’ve finished, but the string updates aren’t in svn. If that happens, tell the translator
that they have to push the “Commit to VCS” button/link.
2.8.4 Creating new localization locales
FIXME: This section is probably wrong.
20
Chapter 2. Contents
fjord Documentation, Release 1.0.dev
In your projects settings.py add the new locale to KNOWN_LANGUAGES.
Then run:
$ ./manage.py merge -c
which will create directories for any locales that are missing in locale/.
See also:
http://playdoh.readthedocs.org/en/latest/userguide/l10n.html#creating-new-locales Playdoh docs on creating
new locales
2.9 Maintaining Vendor Library
To help make setup faster and deployment easier, we pull all of our pure-Python dependencies into the “vendor library”.
This chapter talks about that.
• vendor
• Updating vendor
– Updating an existing library with git submodules
– Adding a new library with git submodules
– Updating a library with pip
– Adding a library with pip
2.9.1 vendor
The vendor/ directory contains git submodules and unpacked source for dependencies.
If you cloned Fjord with --recursive, you already have the the vendor library.
If you didn’t clone with --recursive, or you need to update the submodules, run:
$ git submodule update --init --recursive
Note: Aliasing that to something short (e.g. gsu) is recommended.
2.9.2 Updating vendor
From time to time we need to update libraries, either for new versions of libraries or to add a new library. There are
two ways to do that, but the easiest and prefered way is pure git.
Updating an existing library with git submodules
Using git submodules is prefered because it is much easier to maintain, and it keeps the repository size small. Upgrading is as simple as updating a submodule.
If the library is in vendor/src, it was pulled directly from version control, and if that version control was git,
updating the submodule is as easy as:
2.9. Maintaining Vendor Library
21
fjord Documentation, Release 1.0.dev
$
$
$
$
$
$
cd vendor/src/<LIBRARY-DIR>
git fetch origin
git checkout <REFSPEC>
cd ../..
git add vendor/src/<LIBRARY-DIR>
git commit -m "[bug xyz] Update <LIBRARY>"
Easy peasy!
Adding a new library with git submodules
Run:
$
$
$
$
$
$
$
cd vendor/src
git clone <LIBRARY-REPO>
cd <LIBRARY-DIR>
git checkout <LIBRARY-REPO-REV>
cd ../../..
vendor/addsubmodules.sh
vim vendor/vendor.pth
# back to fjord project root
<Add the new library’s dir to vendor.pth>
$ git add vendor/vendor.pth
$ git commit -m "[bug xyz] Add <LIBRARY>"
Updating a library with pip
The easiest way to update a library with pip is to remove it completely and then install the new version.
Do:
$ cd vendor/packages
$ git rm -r <LIBRARY-DIR>
$ cd ../..
This puts you in the repository root directory.
After removing the old version, go ahead and install the new one:
$ pip install --no-install --build=vendor/packages \
--src=vendor/src -I <LIBRARY>
Finally, add the new library to git:
$ cd vendor
$ git add packages
$ git commit -m "Adding version <VERSION> of <LIBRARY>"
Warning: Caveat developer! Sometimes a library has dependencies that are already installed in the vendor repo.
You may need to remove several of them to make everything work easily.
Adding a library with pip
Adding a new library with pip is easy using pip:
22
Chapter 2. Contents
fjord Documentation, Release 1.0.dev
$ pip install --no-install --build=vendor/packages \
--src=vendor/src -I <LIBRARY>
$ cd vendor
$ git add <LIBRARY>
$ vim vendor.pth
<Add the new library’s path>
$ git commit -m "Adding <LIBRARY>"
Make sure you add any dependencies from the new library, as well.
Note: Need to add a specific version of the library? You can tell pip to install a specific version using ==. For
example:
$ pip install --no-install --build=vendor/packages \
--src=vendor/src -I pyes==0.16
2.9. Maintaining Vendor Library
23
fjord Documentation, Release 1.0.dev
24
Chapter 2. Contents
CHAPTER 3
Part 2: Using the Site
Are you an Input user? This is the section for you!
3.1 Feedback URLs
3.1.1 Generic URL
The basic URL is this:
https://input.mozilla.org/feedback
Users sent to that URL will be redirected to a locale-based URL based on the HTTP headers their browser sends.
The locale-base URL version of this URL will show the product picker letting the user pick the product they want to
leave feedback for.
However, you probably don’t want to use the Generic URL—you know what product you’re asking users to leave
feedback for, so you should provide the product in the url.
3.1.2 Product URLs
Product urls have this form:
https://input.mozilla.org/feedback/%PRODUCT%/%VERSION%/
https://input.mozilla.org/%LOCALE%/feedback/%PRODUCT%/%VERSION%/
This allows you to specify the locale, product and version.
locale You can provide this if you know what locale you want the user to leave feedback for.
Alternatively, you can leave this out and the user will see the feedback form in a locale based on the HTTP
headers of the request.
product The product is the slug of the product that is set up on Input. If you haven’t had your product set up on
Input, you must do that first. Please file a bug.
version Any string that represents the product version. e.g. 3.0, 33.1.1, etc.
Examples:
Firefox desktop:
25
fjord Documentation, Release 1.0.dev
https://input.mozilla.org/feedback/firefox
https://input.mozilla.org/en-US/feedback/firefox
https://input.mozilla.org/pt-BR/feedback/firefox/29.0
Firefox for Android:
https://input.mozilla.org/feedback/android
https://input.mozilla.org/en-US/feedback/android
https://input.mozilla.org/de/feedback/android/29.0
Firefox OS:
https://input.mozilla.org/feedback/fxos
https://input.mozilla.org/en-US/feedback/fxos
https://input.mozilla.org/es/feedback/fxos/1.3.0.0
3.1.3 Source/Campaign
If you’re writing an email, blog post, tweet, whatever and link to Input feedback form, you should specify the
utm_source and utm_campaign querystring parameters. For example:
https://input.mozilla.org/feedback/firefox?utm_source=wiki&utm_campaign=wiki_example
For example, you have a link in the Help menu of the product, you would probably use:
utm_source=helpmenu
For example, if you wrote up a blog post on a blog called “Sam’s Dev Blog” asking people for their feedback because
you’ve made some change, you would probably use:
utm_source=samsdevblog&utm_campaign=20141124_feedback_request
Note: When you specify utm_campaign the resulting feedback is considered non-organic feedback because you
requested the feedback. This is distinguished from organic feedback where the user leaves feedback as they so desire.
3.2 API
26
Chapter 3. Part 2: Using the Site
fjord Documentation, Release 1.0.dev
• Getting product feedback: GET /api/v1/feedback/
– Filters
– Examples
• Posting product feedback: POST /api/v1/feedback/
– Testing clients using the API
– Required fields
– Optional fields
– Extra context
– Minimal example with curl
– Responses
• Getting event data: /api/v1/events/
– Filters
– Curl example
• Posting heartbeat feedback: /api/v1/hb/
– Testing clients using the API
– Required fields
– Curl examples
Fjord has several POST APIs and one GET API for data.
3.2.1 Getting product feedback: GET /api/v1/feedback/
URL /api/v1/feedback/
Method HTTP GET
Payload format No payload–everything is done in the querystring
Response JSON
Doing a GET without any querystring arguments will return the most recent 1,000 publicly visible responses for all
products.
Warning: This API endpoint is throttled. If you hit it too often, you’ll get throttled. You just need to wait a bit
for the throttle to expire.
Note: This API endpoint is still in flux. The documentation is rudimentary at best. Amongst other things, it’s difficult
to discover valid values for things.
If you’re using this API endpoint and have issues, please open a bug.
Filters
q String. Text query.
Example:
?q=crash
happy Boolean. 0 or 1. Restricts results to either happy feedback or sad feedback.
Example:
?happy=0
3.2. API
27
fjord Documentation, Release 1.0.dev
platforms Strings. Comma separated list of platforms. For valid platforms, see https://input.mozilla.org/.
Examples:
?platforms=linux
?platforms=linux,os x
locales Strings. Comma separated list of locales. For valid locales, see https://input.mozilla.org/.
Examples:
?locales=en-US
?locales=en-US,es,es-BR
products Strings. Comma separated list of products. For valid products, see https://input.mozilla.org/.
Examples:
?products=Firefox
?products=Firefox OS,Firefox
versions Strings.
Comma separated list of versions for a specific product.
https://input.mozilla.org/.
For valid versions, see
You must specify a product in order to specify versions.
Examples:
?products=Firefox&versions=31.0
?products=Firefox&versions=31.0,32.0
date_start Date in YYYY-MM-DD format. Specifies the start for the date range you’re querying for.
Example:
?date_start=2014-08-12
date_end Date in YYYY-MM-DD format. Specifies the end for the date range you’re querying for.
Defaults to today.
Example:
?date_end=2014-08-12
date_delta String. 1d, 7d, 14d, etc. The number of days from date_start or date_end.
Example:
# Shows the last 7 days ending today
?date_delta=7d
# Shows 14 days ending 2014-08-12
?date_end=2014-08-12&date_delta=14d
# Shows 22 days starting 2014-08-12
?date_start=2014-08-12&date_delta=22d
max
Integer. Defaults to 1,000. Maximum is 10,000. Minimum is 1. The maximum number of responses you
want to get back.
Example:
28
Chapter 3. Part 2: Using the Site
fjord Documentation, Release 1.0.dev
# Retrieve at most 500 responses
?max=500
# Retrieve at most 10000 responses
?max=10000
Examples
Show all the happy responses for Firefox for the last 7 days for the English locale:
?happy=1&products=Firefox&locales=en-US&date_delta=7d
Show sad responses for Windows platforms for the last day:
?happy=0&platforms=Windows 7,Windows XP, Windows 8.1,Windows 8,Windows Vista,Windows NT&date_delta=1d
3.2.2 Posting product feedback: POST /api/v1/feedback/
URL /api/v1/feedback/
Method HTTP POST
Payload format JSON—make sure to have Content-type:
application/json header
Testing clients using the API
Warning: DO NOT TEST YOUR CLIENT AGAINST OUR PRODUCTION SERVER. IT WILL MAKE
CHENG, MATT, TYLER AND I CROSS.
Seriously. Please don’t test your client against our production server.
Test your client against our stage server which runs the same code that our production server does. The url for the our
stage server is:
https://input.allizom.org/
^^^^^^^
Please make sure to use the correct domain!
Required fields
happy Boolean. All feedback is either happy or sad. This denotes whether this feedback is happy (True) or sad
(False).
Valid values: true, false
description String. Max length: None (but 10,000 characters is probably a good cutoff). This is the feedback text.
Example: "OMG! I love Firefox!"
Note: The form this field is on should have some informational text stating that data in this field will be publicly
available and that the user should not include personally identifyable information.
Example informational text:
3.2. API
29
fjord Documentation, Release 1.0.dev
The content of your feedback will be public, so please be sure
not to include any personal information.
product String. Max length: 20. The name of the product the user is giving feedback on.
Examples:"Firefox for Android", "Firefox OS"
Note: This must be a valid product in the system. Before you start posting to Input, please talk to the User
Advocacy folks or an Input admin to have your product added.
Optional fields
channel String. Max length: 30. The channel of the product the user is giving feedback on.
Examples: "stable", "beta"
version String. Max length: 30. The version of the product the user is giving feedback on as a string.
Examples: "22b2", "1.1"
platform String. Max length: 30. The name of the operating system/platform the product is running on.
Examples: "OS X", "Windows 8", "Firefox OS", "Android", "Linux"
locale String. Max length: 8. The locale the user is using.
Examples: "en-US", "fr", "de"
country String. Max length: 30. The country of origin for the device.
Examples: "Peru", "Mexico"
Note: This is only relevant to Firefox OS phones.
manufacturer String. Max length: 255. The manufacturer of the device the product is running on.
Examples: "Geeksphone", "Samsung"
device String. Max length: 255. The model name of the device the product is running on.
Examples: "Peak", "Galaxy Tab 10.1"
category String. Max length: 50. The category classification for this feedback response.
Examples: "ui", "performance", "bookmarks"
url String. Max length: 200. If the feedback relates to a specific webpage, then the url is the url of the webpage it
refers to.
Examples: "https://facebook.com/", "https://google.com/"
email String. The email address of the user. This allows us to contact the user at some later point to respond to the
user’s feedback or ask for more information.
Example: "[email protected]"
Note: The form this field is in should state that email addresses will not be publicly available.
Example informational text:
While your feedback will be publicly visible, email addresses
are kept private. We understand your privacy is important.
30
Chapter 3. Part 2: Using the Site
fjord Documentation, Release 1.0.dev
user_agent String. Max length: 255. The user agent of the client if applicable. For example if the user is using a
Firefox OS device, this would be the user agent of the browser used to send feedback.
Example: ’Mozilla/5.0 (Mobile; rv:18.0) Gecko/18.0 Firefox/18.0’
source String. Max length: 100. If this response was initiated by a blog post, wiki page, search, newsletter, tweet or
something like that, this is the source that initiated the response. It has the same semantics as the utm_source
querystring parameter:
https://support.google.com/analytics/answer/1033867
Example: ’Hacks blog’
Note: Don’t set the source if the user is leaving feedback of their own accord. Only set the source if you have
prompted or asked the user to leave feedback.
campaign String. Max length: 100. If this response was initiated by a marketing campaign, this is the name of the
campaign. It has the same semantics as the utm_campaign querystring parameter:
https://support.google.com/analytics/answer/1033867
Example: ’show the firefox love post’
Note: Don’t set the campaign if the user is leaving feedback of their own accord. Only set the campaign if you
have prompted or asked the user to leave feedback.
Extra context
You can provide additional context in the form of key/value pairs by adding additional data to the JSON object.
Any fields that aren’t part of the required or optional fields list will get thrown into a JSON object and dumped in the
feedback response context.
For example, if the product were the Firefox devtools and you want feedback responses to include the theme (dark or
light) that the user was using, you could add this to the JSON object:
{
"happy": true,
"description": "devtools are the best!",
"product": "Devtools",
"theme": "dark"
}
That last key will get added to the feedback response context.
Note: Obviously, don’t use a key that’s already the name of a field. Also, since this is not future proof, you might
want to prepend a unique string to any keys you add.
Note: It’s important you don’t add ids or data that allows you to correlate feedback responses to things in other data
sets. That violates our privacy policy.
3.2. API
31
fjord Documentation, Release 1.0.dev
Minimal example with curl
$ curl -v -XPOST ’https://input.allizom.org/api/v1/feedback’ \
-H ’Accept: application/json; indent=4’ \
-H ’Content-type: application/json’ \
-d ’
{
"happy": true,
"description": "Posting by api!",
"product": "Firefox"
}’
Responses
After posting feedback, you’ll get one of several responses:
HTTP 201 Feedback was posted successfully.
HTTP 400 Feedback has errors. Details will be in the response body.
Possibilities include:
• missing required fields
• email address is malformed
• data is in the wrong format
HTTP 429 There has been too many feedback postings from this IP address and the throttle trigger was hit. Try again
later.
3.2.3 Getting event data: /api/v1/events/
URL /api/v1/events/
Method HTTP GET
Payload format No payload–everything is done in the querystring
Response JSON
Doing a GET without any querystring arguments will return all the event data.
Note: This API endpoint is still in flux. The documentation is rudimentary at best. Amongst other things, it’s difficult
to discover valid values for things.
If you’re using this API endpoint and have issues, please open a bug.
Note: The event data is pretty sparse at the moment and only contains releases for Firefox and Firefox for Android.
We plan to add to this data.
Filters
products Strings. Comma separated list of products. For valid products, see https://input.mozilla.org/.
Examples:
32
Chapter 3. Part 2: Using the Site
fjord Documentation, Release 1.0.dev
?products=Firefox
?products=Firefox,Firefox for Android
date_start Date in YYYY-MM-DD format. Specifies the start for the date range you’re querying for.
Example:
?date_start=2014-08-12
date_end Date in YYYY-MM-DD format. Specifies the end for the date range you’re querying for.
Defaults to today.
Example:
?date_end=2014-08-12
Curl example
Minimal example:
curl -v https://input.mozilla.org/api/v1/events/?products=Firefox
3.2.4 Posting heartbeat feedback: /api/v1/hb/
URL /api/v1/hb/
Method HTTP POST
Payload format JSON–make sure to have Content-type:
application/json header
Testing clients using the API
Warning: DO NOT TEST YOUR CLIENT AGAINST OUR PRODUCTION SERVER. IT WILL MAKE
CHENG, MATT, TYLER AND I CROSS.
Seriously. Please don’t test your client against our production server.
Test your client against our stage server which runs the same code that our production server does. The url for the our
stage server is:
https://input.allizom.org/
^^^^^^^
Please make sure to use the correct domain!
Required fields
locale String. Max length: 8. The locale of the user interface that the user is using
Examples:"en-US", "fr", "de"
platform String. Max length: 30. The name of the operating system/platform the product is running on.
Examples: "OS X", "Windows 8", "Firefox OS", "Android", "Linux"
3.2. API
33
fjord Documentation, Release 1.0.dev
product String. Max length: 30. The name of the product the user is giving feedback on.
Examples:"Firefox for Android", "Firefox OS"
Note: This must be a valid product in the system. Before you start posting to Input, please talk to the User
Advocacy folks or an Input admin to have your product added.
channel String. Max length: 30. The channel of the product the user is giving feedback on.
Examples:"stable", "beta"
version String. Max length: 30. The version of the product the user is giving feedback on as a string.
Examples:"22b2", "1.1"
String. The operating system the user is using
poll String. Max length: 50. Alpha-numeric characters and - only. The slug of the poll this heartbeat response is for.
Examples:"is-firefox-fast"
Note: The poll must be created on the Input system you’re testing against and enabled. Otherwise you’ll get
errors.
Before you start posting to Input, please talk to the User Advocacy folks or an Input admin to have your product
added.
answer String. Max length: 10. The answer value.
Examples: "true", "false", "4"
Extra data Any additional fields you provide in the POST data will get glommed into a JSON object and stuck in the
db.
Curl examples
Minimal example:
curl -v -XPOST $URL \
-H ’Accept: application/json; indent=4’ \
-H ’Content-type: application/json’ \
-d ’
{
"locale": "en-US",
"platform": "Linux",
"product": "Firefox",
"version": "30.0",
"channel": "stable",
"poll": "ou812",
"answer": "42"
}’
Here’s an example providing “extra” data:
curl -v -XPOST $URL \
-H ’Accept: application/json; indent=4’ \
-H ’Content-type: application/json’ \
-d ’
{
"locale": "en-US",
34
Chapter 3. Part 2: Using the Site
fjord Documentation, Release 1.0.dev
"platform": "Linux",
"product": "Firefox",
"version": "30.0",
"channel": "stable",
"poll": "ou812",
"answer": "42",
"favoritepie": "cherry",
"favoriteUAperson": "tyler"
}’
The extra fields are plucked out and put in a JSON object and stored in the db like this:
{"favoritepie": "cherry", "favoriteUAperson": "tyler"}
3.3 Heartbeat API
• General
• Fields
– Required
– Optional fields
– Extra content
• Responses
– HTTP 201: Success
– HTTP 400: Bad request
– HTTP 500: Server error
• Examples
– Minimal example
– Example flow
This covers the Heartbeat v2 API. The v1 API was phased out.
The Heartbeat v2 API is for sending data from Heartbeat survey addons to Input. The data contains personally
identifyable information so we treat it as such. It is stored in the database and is not publicly accessible.
If you’re interested in Hearbeat data, speak to the User Advocacy group.
3.3.1 General
URL /api/v2/hb/
Method HTTP POST
Payload format JSON—make sure to have Content-type:
application/json header
Response JSON
3.3.2 Fields
Required
These fields are required and have no defaults. If you do not provide them, then you’ll get back an HTTP 400 with a
message stating you missed a required field.
3.3. Heartbeat API
35
fjord Documentation, Release 1.0.dev
field
type
reinsponse_version
teger
notes
This is the version of the packet specification. Any time the values of the packet or the
calculation of the values for the packet change, we should increase the
response_version. This allows us to distinguish between different packet builds when
doing analysis.
string Max length: 50; The version of the experiment addon.
experiment_version
person_id
string Max length: 50; Id representating a person’s browser. Global across surveys and flows.
survey_id
string This is the survey name as it is set up in Input. This is the only field that has a foreign key to
another table in Input and must be set up before running a survey.
flow_id
string Max length: 50; Id uniquely identifying a flow attempt for this survey.
quesstring Max length: 50; Id uniquely identifying a question for this survey. This allows us to tweak
tion_id
the question text for a survey without launching a new survey.
updated_ts inMilliseconds since the epoch for when this packet was created.
teEvery time you update data for a flow attempt, it should include a new and more recent
ger updated_ts.
quesstring Max length: None; Default: ""; The actual question asked. This can be localized.
tion_text
variastring Max length: 100; Default: ""
tion_id
Optional fields
field
score
max_score
flow_began_ts
flow_offered_ts
flow_voted_ts
flow_engaged_ts
platform
channel
version
locale
build_id
partner_id
profile_age
profile_usage
type
float
float
integer
integer
integer
integer
string
string
string
string
string
string
integer
JSON
addons
JSON
extra
is_test
JSON
boolean
notes
Default: null; The answer the user submitted.
Default: null; The maximum of the answer range.
Default: 0; Milliseconds since the epoch of when the flow began
Default: 0
Default: 0; Milliseconds since the epoch of when the user submitted an answer.
Default: 0
Max length: 50; Default: ""; Operating system and version that the user is using.
Max length: 50; Default: ""; The channel of the browser that the user is using.
Max length: 50; Default: ""; The version of the browser that the user is using.
Max length: 50; Default: ""; The locale that the user’s browser’s interface is in.
Max length: 50; Default: ""
Max length: 50; Default: ""
Default: null; Age in days of the user’s browser profile.
Default: {}; JavaScript object of key/value pairs with information about profile
usage.
Default: {}; JavaScript object of key/value pairs with information about the user’s
addons.
Default: {}; Any extra context you want to provide.
Default: false; Whether (true) or not (false) this is a test packet.
Extra content
Provide extra content in the extra field–there is no slop for this API.
3.3.3 Responses
All response bodies are in JSON.
36
Chapter 3. Part 2: Using the Site
fjord Documentation, Release 1.0.dev
HTTP 201: Success
The answer was posted successfully.
Example curl:
curl -v -XPOST ’https://input.mozilla.org/api/v2/hb/’ \
-H ’Accept: application/json; indent=4’ \
-H ’Content-type: application/json’ \
-d ’
{
"person_id": "c1dd81f2-6ece-11e4-8a01-843a4bc832e4",
"survey_id": "lunch",
"flow_id": "20141117_attempt1",
"experiment_version": "1",
"response_version": 1,
"question_id": "howwaslunch",
"question_text": "how was lunch?",
"variation_id": "1",
"updated_ts": 1416011156000,
"is_test": true
}’
yields this response:
HTTP/1.0 201 CREATED
<uninteresting headers omitted>
Content-Type: application/json; indent=4; charset=utf-8
{
"msg": "success!"
}
HTTP 400: Bad request
Answer has errors. Details will be in the response body.
Possibilities include:
• non-existent survey_id
• disabled survey
• missing required fields
• data is in the wrong format
Example curl:
curl -v -XPOST ’https://input.mozilla.org/api/v2/hb/’ \
-H ’Accept: application/json; indent=4’ \
-H ’Content-type: application/json’ \
-d ’
{
"person_id": "c1dd81f2-6ece-11e4-8a01-843a4bc832e4",
"survey_id": "nonexistent",
"flow_id": "20141114_attempt2",
"response_version": 1,
"experiment_version": "1",
"question_id": "howwaslunch",
3.3. Heartbeat API
37
fjord Documentation, Release 1.0.dev
"updated_ts": 1416011156000,
"is_test": true
}’
yields this response:
HTTP/1.0 400 BAD REQUEST
<uninteresting headers omitted>
Content-Type: application/json; indent=4; charset=utf-8
{
"msg": "bad request; see errors",
"errors": {
"survey_id": [
"Object with name=nonexistent does not exist."
],
"question_text": [
"This field is required."
],
"variation_id": [
"This field is required."
]
}
}
Each field with errors will have its own slot in the “errors” section. If there are multiple errors for that field, it’ll show
multiple errors.
HTTP 500: Server error
Tell Will. He has some ‘splaining to do.
3.3.4 Examples
Minimal example
Anything less than this will kick up “required” type errors.
curl -v -XPOST ’https://input.mozilla.org/api/v2/hb/’ \
-H ’Accept: application/json; indent=4’ \
-H ’Content-type: application/json’ \
-d ’
{
"person_id": "c1dd81f2-6ece-11e4-8a01-843a4bc832e4",
"survey_id": "lunch",
"flow_id": "20141117_attempt1",
"experiment_version": "1",
"response_version": 1,
"question_id": "howwaslunch",
"question_text": "how was lunch?",
"variation_id": "1",
"updated_ts": 1416011156000,
"is_test": true
}’
38
Chapter 3. Part 2: Using the Site
fjord Documentation, Release 1.0.dev
Example flow
(I’m totally making things up here, but maybe this is what it could look like?)
Began:
curl -v -XPOST $URL \
-H ’Accept: application/json; indent=4’ \
-H ’Content-type: application/json’ \
-d ’
{
"person_id": "c1dd81f2-6ece-11e4-8a01-843a4bc832e4",
"survey_id": "lunch",
"flow_id": "20141117_attempt5",
"experiment_version": "1",
"response_version": 1,
"question_id": "howwaslunch",
"updated_ts": 1416011156000,
"question_text": "how was lunch?",
"variation_id": "1",
"score": null,
"max_score": null,
"flow_began_ts": 1416011100000,
"flow_offered_ts": 0,
"flow_voted_ts": 0,
"flow_engaged_ts": 0,
"platform": "",
"channel": "",
"version": "",
"locale": "",
"build_id": "",
"partner_id": "",
"profile_age": null,
"profile_usage": {},
"addons": {},
"extra": {},
"is_test": true
}’
Voted, but not engaged, yet:
curl -v -XPOST $URL \
-H ’Accept: application/json; indent=4’ \
-H ’Content-type: application/json’ \
-d ’
{
"person_id": "c1dd81f2-6ece-11e4-8a01-843a4bc832e4",
"survey_id": "lunch",
"flow_id": "20141117_attempt7",
"experiment_version": "1",
"response_version": 1,
"question_id": "howwaslunch",
"updated_ts": 1416011180000,
"question_text": "how was lunch?",
"variation_id": "1",
"score": 5.0,
"max_score": 10.0,
"flow_began_ts": 1416011100000,
"flow_offered_ts": 1416011120000,
3.3. Heartbeat API
39
fjord Documentation, Release 1.0.dev
"flow_voted_ts": 1416011130000,
"flow_engaged_ts": 0,
"platform": "Windows 7",
"channel": "stable",
"version": "33.1",
"locale": "en-US",
"build_id": "e3b0971e-6ecf-11e4-af44-843a4bc832e4",
"partner_id": "Phil, Prince of Heck",
"profile_age": 365,
"profile_usage": {"avgperday": "5"},
"addons": {"count": 4, "badones": "plenty"},
"extra": {"moonphase": "waning gibbous"},
"is_test": true
}’
40
Chapter 3. Part 2: Using the Site
CHAPTER 4
Part 3: Maintaining The Site
Are you one of those super sekret gnomes that maintain the site through wintry blizzards and fiery summers? This is
the section for you!
4.1 Maintaining Fjord and Input
•
•
•
•
•
•
•
Onboarding a new developer
Triaging bugs for contributors
Adding support for a new product
Adding a new locale
Making strings changes to feedback forms
Dealing with errors in translated strings
EXPERIMENTAL: Update training data
4.1.1 Onboarding a new developer
1. Watch the “Input” product and “__Any__” component in Bugzilla in your Bugzilla Component Watching preferences
2. Hop on the #input IRC channel on irc.mozilla.org
3. Create an IT bug for getting added to error email lists
4. Ask an existing developer with an admin account on Input to create an admin account for you
5. Get access to push changes to https://github.com/mozilla/fjord
6. Update the wiki page
7. Read the documentation:
manual http://fjord.rtfd.org/
wiki page https://wiki.mozilla.org/Firefox/Input
8. Set up your development environment by reading through and doing all the things in the Getting started
9. If you haven’t read through the Webdev Bootcamp, now is a good time
10. Bask in the tranquil rhythmic sounds of the Fjord
41
fjord Documentation, Release 1.0.dev
4.1.2 Triaging bugs for contributors
Bugs designated for contributors fall into two groups:
1. good first bugs
2. good next bugs
Good first bugs should ideally touch one aspect of the code base, be self-contained, and not require a lot of knowledge
about how the system works.
Good first bugs are denoted in Bugzilla with:
• [good first bug] in the Whiteboard field
• a mentor listed in the Mentors field
• a comment explaining the problem and what needs to be done to fix it
Good next bugs are everything else and are denoted in Bugzilla with:
• a mentor listed in the Mentors field
• a comment explaining the problem and what needs to be done to fix it
It’s good to go through bugs once a month to see if any new bugs popped up that’d be good for contributors as well as
for bugs that currently exist that probably should get closed out.
Comment template:
<required skills>
<explanation of how to implement>
<note any wry twists>
Marking this as a mentored bug. If you’re interested in working on it,
please read through our "Join this project" guide [1], complete the
list of things you need to do before contributing and then add a comment
to this bug asking to have it assigned to you.
If you have any questions, please ask on the mailing list or in the
#input channel on irc.mozilla.org.
[1] http://fjord.readthedocs.org/en/latest/welcome.html
Once a week, go through any assigned mentored bugs and ping them to see how they’re coming along.
4.1.3 Adding support for a new product
To add support for a new product via product urls for the feedback form or the Input API, add a new entry to the
Products table.
4.1.4 Adding a new locale
To add a new locale so that https://input.mozilla.org/%NEWLOCALE% works and so that people can see
the site in that language and submit feedback:
1. In your local repository, run:
42
Chapter 4. Part 3: Maintaining The Site
fjord Documentation, Release 1.0.dev
$ ./manage.py update_product_details
2. Check lib/product_details_json/languages.json to see if the language is there.
(a) If not, you need to file a bug to get it added. See https://bugzilla.mozilla.org/show_bug.cgi?id=839506 for
example.
Once the locale is listed in lib/product_details_json/languages.json, proceed.
3. Update locale/ and check to see if the locale is listed there.
(a) If you don’t have a locale/ directory or don’t know how to update it, see Maintaining l10n.
(b) If the locale isn’t in the locale/ directory, ask Milos to add Input to the list of translated projects for
that locale. See https://bugzilla.mozilla.org/show_bug.cgi?id=860754 for better language because I only
vaguely understand how the Verbatim side works.
Once the locale is in svn and locale/, proceed.
4. Add the locale code to the PROD_LANGUAGES list in fjord/settings/base.py.
5. Commit the changes to fjord/settings/base.py and product details stuff to git.
4.1.5 Making strings changes to feedback forms
Feedback forms must be fully translated in specific target locales before they can be pushed to production.
Process for making feedback form strings changes:
1. create a new branch
2. make the changes you need to make
3. submit a pull request
4. after the pull request is approved (r+), extract and merge strings from that branch
5. send an email to the dev-l10n-web mailing list—see Extract/merge/sync strings with svn for details
Now you have to wait until the target locales have fully translated the new strings.
Use the bin/l10n_status.py script to tell you whether things are good to go or not.
Once they’re good to go, you can land the changes in master and push to stage and production.
For the list of target locales per form, see Cheng, Matt or Will.
4.1.6 Dealing with errors in translated strings
When we deploy a new version of Fjord, it updates the .po files and picks up newly translated strings.
.po files that have errors will not get compiled to .mo files and thus won’t go to production and thus won’t cause
fires.
Note that this doesn’t mean that this locale will have no translations—we’ll use the previously compiled .mo file.
If there is no .mo file, then the deployment will compile a .mo file even if there are errors with the figuring that a
problematic .mo file is better than nothing and that this should be an exceedingly rare occurrence.
If .po files have errors, then those errors are noted in the postatus.txt files:
• dev: https://input-dev.allizom.org/media/postatus.txt
• stage: https://input.allizom.org/media/postatus.txt
4.1. Maintaining Fjord and Input
43
fjord Documentation, Release 1.0.dev
• prod: https://input.mozilla.org/media/postatus.txt
If there are errors in those files, we need to open up a bug in Mozilla Localizations -> locale code with the specifics.
Bug description template:
We found errors in the translated strings for Input (https://input.mozilla.org/).
The errors are as follows:
<paste errors here>
Until these errors are fixed, we can’t deploy updates to the strings for this locale.
If you have any questions, let me know.
4.1.7 EXPERIMENTAL: Update training data
Note: This is here solely to support the spicedham prototype integration. It’s liable to change.
Create a directory structure:
- corpus
\
|- abuse
| \- bunch of [respid].json files
|
\- not_abuse
\- bunch of [respid].json files
Each file is the JSON for a single response.
4011021.json.
The files are titled <response-id>.json.
For example,
To generate training data:
$ python manage.py trainabuse corpus/
To generate new fixture:
$ python manage.py dumpdata --indent=2 --format=json \
flags flags.Store > fjord/flags/fixtures/abuse_training.json
To load the fixture:
$ python manage.py loaddata flags abuse_training.json
4.2 Service Level Agreement
If you notice issues with this SLA, please update it.
Last updated February 26th, 2013
4.2.1 Feedback submitters
We support feedback from the following things:
44
Chapter 4. Part 3: Maintaining The Site
fjord Documentation, Release 1.0.dev
1. Firefox Desktop stable using the feedback form
2. Firefox for Android stable using the feedback form
3. Firefox for Android stable about:feedback
This requires some funky code because we changed the way feedback posting worked between the old code
base and the new one.
We need to update Firefox for Android so it posts to the new codebase, then we can ditch a bunch of the funky
code.
4. FirefoxOS: v1.0
4.2. Service Level Agreement
45