Sphinx Doc, JSON highlight and Sphinx extensions: kung-fu

At the moment Pygments which used by Sphinx Doc haven't support for JSON code highlight which is really sad.

I've not found any useful information how to do it quickly. So here is my way:

1. I've found custom pygments lexer which support JSON: pygments-json . I will be part of pygments soon
2. It wasn't clear to me how to add custom pygments lexer to sphinx, my google-fu isn't good today
3. A bit more googling gave me Sphinx Extensions API , especially add_lexer method of Sphinx instance

Ok, now it's clear to me how to add new lexer. I've created ext/hijson folder within source, to __init__.py I've added setup function:

	def setup(sphinx):
	from pygson.json_lexer import JSONLexer
	sphinx.add_lexer("json", JSONLexer())

view raw __init__.py hosted with ❤ by GitHub

Also here is how to add support of ext folder as folder with custom extensions

sys.path.insert(0, os.path.join(os.path.abspath('.'), "ext"))

view raw conf.py hosted with ❤ by GitHub

So, pip install pygments-json and use

.. code-block:: json

Nice and smooth:

PyFlakes-clean import of local_settings.py

Any project have own bunch of settings for different environments: for dev, production, staging. Everyone use it on daily basis. But annoying thing is that PyFlakes, great code analysis tool, warn about that. And it's reasonable.

So, to have this functionality, but without warning I use this pattern:

try:
    import local_settings
    for setting in dir(local_settings):
        if setting.startswith('_'):
            continue
        locals()[setting] = getattr(local_settings, setting)
except ImportError:
    pass

Simple and useful!


Update: I've added ignore of attributes which starting with underscore. Thanks to Igor Davydenko

Unstable HTTP services: what we can do to easily handle that?

The story: I have several HTTP service providers which works quite unstable. Yes, I had to have in mind this during development. But we're all thought that issues are "temporary" and will gone when we going to production. We accurately added logging.error in every place and move on with other stuff.

But our expectation about temporary nature of service behavior will never happen. Service sometimes work slowly, sometimes return HTTP errors and so on. We receive tons of exceptions every day. We had to do something with that.

The solution: Here is safe_exec decorator which help solve this problem. You can specify how many times you want to try execute function, what's timeout between them and what exceptions are expected during execution decorated function. For example, urllib2.urlopen may generate urllib2.URLError or  urllib2.HTTPError.

import logging
import time

from functools import wraps

__all__ = ("safe_exec",)


def safe_exec(exceptions, shakes=3, timeout=1, title="", **kwargs):
    """
    Decorator to safely execute function or method
    within `shakes` trying.

    In case provide argument `default` exception will not
    be raised and will return provided value.
    """
    def wrap(func):
        if not isinstance(exceptions, tuple):
            raise TypeError(
                "First argument of safe_exec should be tuple of exceptions"
            )

        @wraps(func)
        def wrapped(*args, **kwargs):
            result = None
            orig_exception = None
            for shake in range(shakes):
                try:
                    result = func(*args, **kwargs)
                    break
                except exceptions, orig_exception:
                    logging.warn("%s: Sorry, can't execute %s, shake #%d",
                        title,
                        func.__name__,
                        shake,
                        exc_info=True
                    )
                    time.sleep(timeout)
            else:
                logging.error(
                    "%s: Can't execute `%s` after %d shakes",
                    title,
                    func.__name__,
                    shakes
                )

                if "default" in kwargs:
                    return kwargs.get("default")

                raise orig_exception

            return result
        return wrapped
    return wrap

Sample usage:

import urllib2

@safe_exec((urllib2.URLError, urllib2.HTTPError), shakes=2)
def download(url):
    return urllib2.urlopen(url).read()

download("http://slow-resource.com/")

Testing with Jenkins, Selenium and Continuous Deployment

My presentation at KharkivPy#4 about Selenium, Jenkins and Continuous Deployment. A lot of people asket me about code samples. There's no samples. There's no python code at all in this presentation. The main purpose of this presentation is that you can build full cycle testing for your project easy enough.

For example by using tools like Fabdeploy or even raw Fabric

Testing with Jenkins, Selenium and Continuous Deployment

View more presentations from Max Klymyshyn

Update: video of this presentation in russian

Headless JavaScript testing, Sinon.js, Fake Timers and Rhino

Recently I've started to use TDD approach with development of client-side code/JavaScript. The code on front-end become more complex and it have to be tested along with back-end code.

One of first things which I've found was Sinon.js - the mock library to make my life easier. Btw, I'm using Jasmine for writing tests. So, everything looks great until I've tried to integrate my tests into our CIA (Jenkins).

The tests which working great within browser failed using rhino/env.js. That was weird and unfortunately tracebacks were useless. I've started to go deeper and found an article which make things clear to me: Sinon.js have issues with timers.

Update: I've fixed issue with Sinon.JS Fake Timers and Rhino.js

I've made small stub to replace functionality of fake timeouts from Sinon.js and wow, tests are passed!

// setTimeout/clearTimeout stub using Underscore.js
var FakeTimeout = function () {
  var self = this,
  timers = [],
  counter = 1,
  timeoutOrig = setTimeout,
  clearOrig = clearTimeout;

  // add new timeout to the queue
  this.setTimeout = function (f, timeout) {
    var id = counter++;
    timers.push({
      'callback': f,
      'timeout': timeout,
      'id': id
    });
    return id;
  };

  // cleanup timeout
  this.clearTimeout = function (id) {
    timers = _.filter(timers, function (item) {
      return item.id !== id;
    });
  };

  // tick the clock by timeout
  this.tick = function (timeout) {
    var timerToRemove = [];
    _.each(timers, function (timer) {
      if (timer.timeout - timeout <= 0) {
        timer.callback();
        timerToRemove.push(timer);
      } else {
        timer.timeout = timer.timeout - timeout;
      }
    });
    timers = _.difference(timers, timerToRemove);
  };

  // install fake timers
  this.install = function () {
    setTimeout = _.bind(this.setTimeout, this);
    clearTimeout = _.bind(this.clearTimeout, this);
  };

  // restore the original timers
  this.restore = function () {
    setTimeout = timeoutOrig;
    clearTimeout = clearOrig;
  };
};

Here is how to use it:

// Usage:
var timeout = new FakeTimeout();

// Somewhere in your code
timeout.install();

// ... some functionality which use setTimeout/clearTimeout
timeout.tick(1000);

// restore the state of setTimeout/clearTimeout
timeout.restore()

The code require Underscore.js 

The conclusion is to keep it simple. I don't get why you guys need so much code for such simple thing.