Integrating OData and DocumentDb with Azure Functions

This is the finale in a series of posts that aimed to provide an alternative to Azure Mobile Apps based on Azure Functions and DocumentDb. I started with discussing how to run a CRUD HTTP API, then moved onto DocumentDb, handled inserts and replacements. Now it’s time to fetch data. Azure Mobile Apps uses a modified OData v3 query string to perform the offline sync and online querying of data. This is mostly because ASP.NET (which was the basis for the original service) has a nice OData library for it. OData is painful to use in our context, however. Firstly, there are some necessary renames – the updatedAt field is actually the DocumentDb timestamp, for example. The other thing is that there is no ready made library for turning an OData query string into a DocumentDb SQL statement. So I don’t have an “easy” way of fulfilling the requirement.

Fortunately, the Azure Mobile Apps Node SDK has split off a couple of libraries for more general use. The first is azure-query-js. This is a library for converting between a set of OData query parameters and an internal query structure. The second is azure-odata-sql, which is for turning a normalized OData query into SQL, based on Microsoft SQL or SQLite syntax. Neither of these libraries is particularly well documented, but they are relatively easy to use based on the examples used within the Azure Mobile Apps SDKs. We are going to need to modify the azure-odata-sql library to generate appropriate SQL statements for DocumentDB, so I’ve copied the source to the library into my project (in the directory odata-sql). My first stab at the getAllItems() method looks like this:

var OData = require('azure-query-js').Query.Providers.OData;
var formatSql = require('../odata-sql').format;

function getAllItems(req, res) {
    // DoumentDB doesn't support SKIP yet, so we can't do TOP either without some problems
    var query = OData.fromOData(
        settings.table,
        req.query.$filter,
        req.query.$orderby,
        undefined, //parseInt(req.query.$skip),
        undefined, //parseInt(req.query.$top),
        req.query.$select,
        req.query.$inlinecount === 'allpages',
        !!req.query.__includeDeleted);

    var sql = formatSql(OData.toOData(query), {
        containerName: settings.table,
        flavor: 'documentdb'
    });
    
    res.status(200).json({ query: req.query, sql: sql, message: 'getAll' });
}

As noted here, DocumentDB hasn’t added full support for SKIP/TOP statements, so we can’t use those elements. Once the support is available within DocumentDB, I just need to include that support in the odata-sql library and change the two parmeters to the fromOData() call.

So, what does this do? Well, first, it converts the request from the browser (or client SDK) from the jumble of valid OData query params into a Query object. That Query object is actually a set of functions to do the parsing. Then we use the toOData() method (from the azure-query-js library) to convert that Query object into a normalized OData query. Finally, we use a custom SQL formatter (based on the azure-odata-sql) library to convert it to a SQL statement. If you run this, you should get something like the following out of it:

getall-1

I can now see the SQL statements being generated. The only problem is that they are not actually valid SQL statements for DocumentDB. They are actually perfectly valid for Microsoft SQL Server or SQL Azure. We need to adjust the odata-sql library for our needs. There are a couple of things needed here. Our first requirement is around the updatedAt field. This is not updatedAt in DocumentDB – it’s _ts, and it’s a number. We can do this using regular expressions like this:

if (req.query.$filter) {
    while (/updatedAt [a-z]+ '[^']+'/.test(req.query.$filter)) {
        var re = new RegExp(/updatedAt ([a-z]+) '([^']+)'/);
        var results = re.exec(req.query.$filter);
        var newDate = moment(results[2]).unix();
        var newString = `_ts ${results[1]} ${newDate}`;
        req.query.$filter = req.query.$filter.replace(results[0], newString);
    }
}

I could have probably shrunk this code somewhat, but it’s clear as to what is going on. We loop around the filter while there is still an updatedAt clause, convert the date, then replace the old string with the new string. We need to do similar things with the $select and $orderby clauses as well – left out because I’m trying to make this simple.

In terms of the odata-sql library, most of what we want is in the helpers.js library. Specifically, in the case of DocumentDB, we don’t need the square brackets. That means the formatMember() and formatTableName() methods must be adjusted to compensate.

I found it easier to step through the code by writing a small test program to test this logic out. You can find it in todoitem\test.js. With Visual Studio Code, you can set breakpoints, watch variables and do all the normal debugging things to really understand where the code is going and what it is doing.

Now that the SQL looks good, I need to execute the SQL commands. I’ve got a version of queryDocuments() in the driver:

    queryDocuments: function (client, collectionRef, query, callback) {
        client.queryDocuments(collectionRef._self, query).toArray(callback);
    },

This is then used in the HTTP trigger getAllItems() method. I’ve included the whole method here for you:

function getAllItems(req, res) {
    // Adjust the query parameters for DocumentDB
    if (req.query.$filter) {
        while (/updatedAt [a-z]+ '[^']+'/.test(req.query.$filter)) {
            var re = new RegExp(/updatedAt ([a-z]+) '([^']+)'/);
            var results = re.exec(req.query.$filter);
            var newDate = moment(results[2]).unix();
            var newString = `_ts ${results[1]} ${newDate}`;
            req.query.$filter = req.query.$filter.replace(results[0], newString);
        }
    }
    // Remove the updatedAt from the request
    if (req.query.$select) {
        req.query.$select = req.query.$select.replace(/,{0,1}updatedAt/g, '');
    }

    // DoumentDB doesn't support SKIP yet, so we can't do TOP either
    var query = OData.fromOData(
        settings.table,
        req.query.$filter,
        req.query.$orderby,
        undefined, //parseInt(req.query.$skip),
        undefined, //parseInt(req.query.$top),
        req.query.$select,
        req.query.$inlinecount === 'allpages',
        !!req.query.__includeDeleted);

    var sql = formatSql(OData.toOData(query), {
        containerName: settings.table,
        flavor: 'documentdb'
    });

    // Fix up the object so that the SQL object matches what DocumentDB expects
    sql[0].query = sql[0].sql;
    sql[0].parameters.forEach((value, index) => {
        sql[0].parameters[index].name = `@${value.name}`;
    });

    // Execute the query
    console.log(JSON.stringify(sql[0], null, 2));
    driver.queryDocuments(refs.client, refs.table, sql[0])
    .then((documents) => {
        documents.forEach((value, index) => {
            documents[index] = convertItem(value);
        });

        if (sql.length == 2) {
            // We requested $inlinecount == allpages.  This means we have
            // to adjust the output to include a count/results field.  It's
            // used for paging, which DocumentDB doesn't support yet.  As
            // a result, this is a hacky way of doing this.
            res.status(200).json({
                results: documents,
                count: documents.length
            });
        } else {
            res.status(200).json(documents);
        }
    })
    .catch((error) => {
        res.status(400).json(error);
    });
}

Wrapping Up

So, there you have it. A version of the Azure Mobile Apps service written with DocumentDB and executing in dynamic compute on Azure Functions.

Of course, I wouldn’t actually use this code in production. Firstly, I have not written any integration tests on this, and there are a bunch of corner cases that I would definitely want to test. DocumentDB doesn’t have good paging support yet, so you are getting all records all the time. I also haven’t looked at all the OData methods that can be converted into SQL statement to ensure DocumentDB support. Finally, and this is a biggie, the service has a “cold start” time. It’s not very much, but it can be significant. In the case of a dedicated service, you spend that cold start time once. In the case of a dynamic compute Azure Function, you can spend that time continually. This isn’t actually a problem with DocumentDB, since I am mostly passing through the REST calls (adjusted). However, it can become a problem when using other sources. One final note is that I keep all the records in memory – this can drive up the memory requirements (and hence cost) of the Azure Function on a per-execution basis.

Until next time, you can find the source code for this project on my GitHub repository.

Updating Documents in DocumentDb

In my last few posts, I’ve been working on an Azure Mobile Apps replacement service. It will run in Azure Functions and use DocumentDb as a backing store. Neither of these requirements are possible in the Azure Mobile Apps server SDK today. Thus far, I’ve created a CRUD HTTP API, initialized the DocumentDb store and handled inserts. Today is all about fetching, but more importantly it is about replacing documents and handling conflict resolution.

The DocumentDb Driver

Before I get started with the code for the endpoint, I need to add some more functionality to my DocumentDb promisified driver. In the document.js file, I’ve added the following:

module.exports = {
    createDocument: function (client, collectionRef, docObject, callback) {
        client.createDocument(collectionRef._self, docObject, callback);
    },

    fetchDocument: function (client, collectionRef, docId, callback) {
        var querySpec = {
            query: 'SELECT * FROM root r WHERE r.id=@id',
            parameters: [{
                name: '@id',
                value: docId
            }]
        };

        client.queryDocuments(collectionRef._self, querySpec).current(callback);
    },

    readDocument: function (client, docLink, options, callback) {
        client.readDocument(docLink, options, callback);
    },

    replaceDocument: function(client, docLink, docObject, callback) {
        client.replaceDocument(docLink, docObject, callback);    
    }
};

My first attempt at reading a document used the readDocument() method. I would construct a docLink using the following:

var docLink = `${refs.table._self}${refs.table._docs}${docId}`;

However, this always resulted in a 400 Bad Request response from DocumentDb. The reason is likely that the _self link uses the shorted (and obfuscated) URI, whereas the Document Id I am using is a GUID and is not obfuscated. If you take a look at the response from DocumentDb, there is an id field and a _rid field. The _rid field is used in the document links. Thus, instead of using readDocument(), I’m using a queryDocuments() call on the driver to search for the Id. I’ve also promisified these calls in the normal manner using the Bluebird library.

Fetching a Record

The Azure Mobile Apps SDK allows me to GET /tables/todoitem/id – where id is the GUID. With the driver complete, I can do the following in the Azure Function table controller:

function getOneItem(req, res, id) {
    driver.fetchDocument(refs.client, refs.table, id)
    .then((document) => {
        if (typeof document === 'undefined')
            res.status(404).json({ 'error': 'Not Found' });
        else
            res.status(200).json(convertItem(document));
    })
    .catch((error) => {
        res.status(error.code).json(convertError(error));
    });
}

When doing this, I did notice that some semantics seem to have changed in the Azure Functions SDK. I can no longer use context.bindings.id and has to switch to using req.params.id. Aside from this small change in the router code, this code is relatively straight forward. I established the convertItem() and convertError() methods in my last article.

Replacing a Record

The more complex case is replacing a record. There is a little bit of logic around conflict resolution:

  • If there is an If-Match header, then ensure the version of the current record matches the If-Match header, otherwise return a 412 response.
  • If there is no If-match header, but the new record contains a version, return a 409 response.
  • Otherwise update the record

Because we want the version and updatedAt fields to be controlled as well, we need to ensure the new object does not contain those values when it is submitted to DocumentDb:

function replaceItem(req, res, id) {
    driver.fetchDocument(refs.client, refs.table, id)
    .then((document) => {
        if (typeof document === 'undefined') {
            res.status(404).json({ 'error': 'Not Found' });
            return;
        }

        var item = req.body, version = new Buffer(document._etag).toString('base64')
        if (item.id !== id) {
            res.status(400).json({ 'error': 'Id does not match' });
            return;
        }

        if (req.headers.hasOwnProperty('if-match') && req.header['if-match'] !== version) {
            res.status(412).json({ 'current': version, 'new': item.version, 'error': 'Version Mismatch' })
            return;
        }

        if (item.hasOwnProperty('version') && item.version !== version) {
            res.status(409).json({ 'current': version, 'new': item.version, 'error': 'Version Mismatch' });
            return;
        }

        // Delete the version and updatedAt fields from the doc before submitting
        delete item.updatedAt;
        delete item.version;
        driver.replaceDocument(refs.client, document._self, item)
        .then((updatedDocument) => {
            res.status(200).json(convertItem(updatedDocument));
            return;
        });
    })
    .catch((error) => {
        res.status(error.code).json(convertError(error));
    });
}

I’m using the same Base64 encoding for the etag in the current document to ensure I can do a proper match. I could get DocumentDb to do all this work for me – the options value in the driver replaceDocument() method allows me to specify an If-Match. However, to do that, I would need to still fetch the record (since I need the document link), so I may as well do the checks myself. This also keeps some load off the DocumentDb, which is helpful.

While this is almost there, there is one final item. If there is a conflict, the server version of the document should be returned. That means the 409 and 412 responses need to return convertItem(document) instead – a simple change.

Deleting a Record

Deleting a record does not delete a record. Azure Mobile Apps uses soft delete (whereby the deleted flag is set to true). This means that I need to use replaceDocument() again for deletions:

function deleteItem(req, res, id) {
    driver.fetchDocument(refs.client, refs.table, id)
    .then((document) => {
        if (typeof document === 'undefined') {
            res.status(404).json({ 'error': 'Not Found' });
            return;
        }

        var item = convertItem(document);
        delete item.updatedAt;
        delete item.version;
        item.deleted = true;
        driver.replaceDocument(refs.client, document._self, item)
        .then((updatedDocument) => {
            res.status(200).json(convertItem(updatedDocument));
            return;
        });
    })
    .catch((error) => {
        res.status(error.code).json(convertError(error));
    });
}

This brings up a point about the GetOneItem() method. It does not take into account the deleted flag. I need it to return 404 Not Found if the deleted flag is set:

function getOneItem(req, res, id) {
    driver.fetchDocument(refs.client, refs.table, id)
    .then((document) => {
        if (typeof document === 'undefined' || document.deleted === true)
            res.status(404).json({ 'error': 'Not Found' });
        else
            res.status(200).json(convertItem(document));
    })
    .catch((error) => {
        res.status(error.code).json(convertError(error));
    });
}

It’s a simple change, but important in getting the protocol right.

What’s left?

There is only one method I have not written yet, and it’s the biggest one of the set – the getAllItems() method. That’s because it deals with OData querying, which is no small task. I’ll be tackling that in my next article. Until then, get the current codebase at my GitHub repository.

Creating Documents in DocumentDB with Azure Functions HTTP API

Thus far in my story of implementing Azure Mobile Apps in a dynamic (consumption) plan of Azure Functions using DocumentDB, I’ve got the basic CRUD HTTP API stubbed out and the initialization of my DocumentDB collection done. It’s now time to work on the actual endpoints that my Azure Mobile Apps SDK will call. There are five methods to implement:

  • Insert
  • Update / Replace
  • Delete
  • Fetch a single record
  • Search

I’m going to do these in the order above. Before I do that, I need to take a look at what DocumentDB provides me. Azure Mobile Apps requires five fields to work properly:

  • id – a string (generally a GUID).
  • createdAt – the date the record was created, in ISO-8601 format.
  • updatedAt – the date the record was updated, in ISO-8601 format.
  • deleted – a boolean, if the record is deleted.
  • version – an opaque string for conflict resolution.

DocumentDB provides some of this for us:

  • id – a string (generally a GUID).
  • _ts – a POSIX / unix timestamp of the number of seconds since the epoch since the record was last updated.
  • _etag – a checksum / version identifier.

When we create a record, we need to convert the document that DocumentDB returns to us into the format that Azure Mobile Apps provides. I use the following routine:

/**
 * Given an item from DocumentDB, convert it into something that the service can used
 * @param {object} item the original item
 * @return {object} the new item
 */
function convertItem(item) {
    if (item.hasOwnProperty('_ts')) {
        item.updatedAt = moment.unix(item._ts).toISOString();
        delete item._ts;
    } else {
        throw new Error('Invalid item - no _ts field');
    }

    if (item.hasOwnProperty('_etag')) {
        item.version = new Buffer(item._etag).toString('base64');
        delete item._etag;
    } else {
        throw new Error('Invalid item - no _etag field');
    }

    // Delete all the known fields from documentdb
    if (item.hasOwnProperty('_rid')) delete item._rid;
    if (item.hasOwnProperty('_self')) delete item._self;
    if (item.hasOwnProperty('_attachments')) delete item._attachments;

    return item;
}

I’m using the moment library to do date/time manipulation. This is a very solid library and well worth learning about. In addition to the convertItem() method, I also need something to convert the error values that come back from DocumentDB. They are not nicely formed, so some massaging is in order:

/**
 * Convert a DocumentDB error into something intelligible
 * @param {Error} error the error object
 * @return {object} the intelligible error object
 */
function convertError(error) {
    var body = JSON.parse(error.body);
    if (body.hasOwnProperty("message")) {
        var msg = body.message.replace(/^Message:\s+/, '').split(/\r\n/);
        body.errors = JSON.parse(msg[0]).Errors;

        var addl = msg[1].split(/,\s*/);
        addl.forEach((t) => {
            var tt = t.split(/:\s*/);
            tt[0] = tt[0].replace(/\s/, '').toLowerCase();
            body[tt[0]] = tt[1];
        });

        delete body.message;
    }

    return body;
}

I had to work through the error object several times experimenting with the actual response to come up with this routine. This seems like the right code by experimentation. Whether it holds up during normal usage remains to be seen.

I’ve already written the createDocument() method in the DocumentDB driver:

module.exports = {
    createDocument: function (client, collectionRef, docObject, callback) {
        client.createDocument(collectionRef._self, docObject, callback);
    }
};

This is then promisifyed using the bluebird promise library. With this work done, my code for inserts becomes very simple:

function insertItem(req, res) {
    var item = req.body;

    item.createdAt = moment().toISOString();
    if (!item.hasOwnProperty('deleted')) item.deleted = false;

    driver.createDocument(refs.client, refs.table, item)
    .then((document) => {
        res.status(201).json(convertItem(document));
    })
    .catch((error) => {
        res.status(error.code).json(convertError(error));
    });
}

The item that we need to insert comes in on the body. We need to add the createdAt field and the deleted field (if it isn’t already set). Since this is an insert, we call createDocument() in the driver. If it succeeds, we return a 201 Created response with the new document (converted to the Azure Mobile Apps specification). If not, we return the error from DocumentDB together with the formatted object.

We can test inserts with Postman. For example, here is a successful insert:

insert-1

DocumentDB creates the id for me if it doesn’t exist. I convert the _ts and _etag fields to something more usable by the Azure Mobile Apps SDK on the way back to the client. If I copy the created object and push it again, I will get a conflict:

insert-2

Notice how DocumentDB does all the work for me? All I need to do is some adjustments on the output to get my insert operation working. I can use the Document Browser within the Azure Portal to look at the actual records.

In the next post, I’m going to move onto Update, Delete and Fetch all in one go.

Working with DocumentDb

In my last post, I introduced working with HTTP CRUD APIs with Azure Functions. My intent in all this is to create a proof of concept service that emulates the Azure Mobile Apps service, but using Azure Functions and the dynamic (or consumption-based) SKU. This means that you pay for the API only when it is being used, but it scales seamlessly as your needs grow. In addition, I’m going to make the backing store for this API a NoSQL store based on another Azure resource – DocumentDb.

Fortunately for me, DocumentDb has a nice Node.js driver. I’m going to promisify the callback-based SDK with bluebird. There are a number of samples available for the DocumentDb driver. For instance, here is my docdb-driver/database.js file:

module.exports = {
    createDatabase: function (client, databaseId, callback) {
        client.createDatabase({ id: databaseId }, callback);
    },

    deleteDatabase: function (client, databaseId, callback) {
        client.deleteDatabase(`dbs/${databaseId}`, callback);
    },

    findDatabaseById: function (client, databaseId, callback) {
        var qs = {
            query: 'SELECT * FROM root r WHERE r.id = @id',
            parameters: [
                { name: '@id', value: databaseId }
            ]
        };

        client.queryDatabases(qs).toArray(function (err, results) {
            if (err) {
                callback(err, null);
            } else {
                callback(null, (results.length === 0) ? null : results[0]);
            }
        });
    },

    listDatabases: function (client, callback) {
        client.readDatabases().toArray(callback);
    },

    readDatabase: function (client, database, callback) {
        client.readDatabase(database._self, callback);
    },

    readDatabases: function (client, databaseId, callback) {
        client.readDatabase(`dbs/${databaseId}`, callback);
    }
};

This is based on callbacks, rather than promises. So my docdb-driver/index.js file uses promisify to convert them to promises:

var Promise = require('bluebird');
var collection = require('./collection');
var database = require('./database');
var docops = require('./document');

var dbCache = {};

var createDatabase = Promise.promisify(database.createDatabase);
var findDatabaseById = Promise.promisify(database.findDatabaseById);

function ensureDatabaseExists(client, database) {
    if (database in dbCache) {
        return Promise.resolve(dbCache[database]);
    }

    return findDatabaseById(client, database).then((dbRef) => {
        if (dbRef == null) {
            return createDatabase(client, database).then((result) => {
                dbCache[database] = result;
                return result;
            });
        }
        dbCache[database] = dbRef;
        return dbRef;
    });
}

module.exports = {
    createCollection: Promise.promisify(collection.createCollection),
    listCollections: Promise.promisify(collection.listCollections),
    readCollection: Promise.promisify(collection.readCollection),
    readCollectionById: Promise.promisify(collection.readCollectionById),
    getOfferType: Promise.promisify(collection.getOfferType),
    changeOfferType: Promise.promisify(collection.changeOfferType),
    deleteCollection: Promise.promisify(collection.deleteCollection),

    createDatabase: createDatabase,
    deleteDatabase: Promise.promisify(database.deleteDatabase),
    ensureDatabaseExists: ensureDatabaseExists,
    findDatabaseById: findDatabaseById,
    listDatabases: Promise.promisify(database.listDatabases),
    readDatabase: Promise.promisify(database.readDatabase),
    readDatabases: Promise.promisify(database.readDatabases),

    createDocument: Promise.promisify(docops.createDocument)
};

I’m going to extend this driver package over time. Sometimes I use the straight API from the DocumentDb driver (see the readDatabase() method). Sometimes, however, I want to do something extra. The ensureDatabaseExists() method is an example of this. I want to find the database in the service and create it only if it doesn’t exist.

Back to the Azure Function I’m developing. DocumentDb mainly stores “documents” – JSON blobs of associated data. It organizes these documents into “collections” and collections into a “database”. In the Azure Mobile Apps equivalent, the collection would be a table and the individual rows or entities would be documents. My first requirement is to ensure that the database and collection are initialized properly (in todoitem/index.js):

var DocumentDb = require('documentdb');
var driver = require('../docdb-driver');

/**
 * Global Settings Object
 */
var settings = {
    host: process.env['DocumentDbHost'],
    accountKey: process.env['DocumentDbAccountKey'],
    database: 'AzureMobile',
    connectionPolicy: undefined,
    consistencyLevel: 'Session',
    pricingTier: 'S1',
    table: 'todoitem'
};

// Store any references we receive here as a cache
var refs = {
    initialized: false
};

/**
 * Routes the request to the table controller to the correct method.
 *
 * @param {Function.Context} context - the table controller context
 * @param {Express.Request} req - the actual request
 */
function tableRouter(context, req) {
    var res = context.res;
    var id = context.bindings.id;

    initialize(context).then(() => {
        switch (req.method) {
            case 'GET':
                if (id) {
                    getOneItem(req, res, id);
                } else {
                    getAllItems(req, res);
                }
                break;

            case 'POST':
                insertItem(req, res);
                break;

            case 'PUT':
                replaceItem(req, res, id);
                break;

            case 'DELETE':
                deleteItem(req, res, id);
                break;

            default:
                res.status(405).json({ error: "Operation not supported", message: `Method ${req.method} not supported`})
        }
    });
}

/**
 * Initialize the DocumentDb Driver
 * @param {Function.Context} context - the table controller context
 * @param {function} context.log - used for logging
 * @returns {Promise}
 */
function initialize(context) {
    if (refs.initialized) {
        context.log('[initialize] Already initialized');
    }

    context.log(`[initialize] Creating DocumentDb client ${settings.host} # ${settings.accountKey}`);
    refs.client = new DocumentDb.DocumentClient(
        settings.host,
        { masterKey: settings.accountKey },
        settings.connectionPolicy,
        settings.consistencyLevel
    );

    context.log(`[initialize] EnsureDatabaseExists ${settings.database}`);
    return driver.ensureDatabaseExists(refs.client, settings.database)
        .then((dbRef) => {
            context.log(`[initialize] Initialized Database ${settings.database}`);
            refs.database = dbRef;
            return driver.listCollections(refs.client, refs.database);
        })
        .then((collections) => {
            context.log(`[initialize] Found ${collections.length} collections`);
            const collection = collections.find(c => { return (c.id === settings.table); });
            context.log(`[initialize] Collection = ${JSON.stringify(collection)}`);
            if (typeof collection !== 'undefined') return collection;
            context.log(`[initialize] Creating collection ${settings.table}`);
            return driver.createCollection(refs.client, settings.pricingTier, refs.database, settings.table);
        })
        .then((collectionRef) => {
            context.log(`[initialize] Found collection`);
            refs.table = collectionRef;
            refs.initialized = true;
        });

    context.log('[initialize] Finished Initializing Driver');
}

Let's take this in steps. Firstly, I set up the settings. The important things here are the DocumentDbHost and the DocumentDbAccountKey. If you have created a DocumentDb within the Azure Portal, click on the Keys menu item. The DocumentDbHost is the URI field and the DocumentDbAccountKey is the PRIMARY KEY field. If you are running the Azure Function locally, then you will need to set these as environment variables before starting the func host. If you are running the Azure Function within Azure, you need to make these App Settings. An example of setting these locally in PowerShell:

$env:DocumentDbHost = "https://mydocdb.documents.azure.com:443/"
$env:DocumentDbAccountKey = "fuCZuSomeLongStringaLNKjIiMSEyaojsP05ywmevI7K2yCY9dYLRuCQPd3dMnvg=="
func run test-func --debug

When you use Postman (for example, a GET http://localhost:7071/tables/todoitem), you will see the initialize() method gets called. This method returns a Promise that, when resolved, will then allow the request to be continued. In the initialize() method, I short-circuit the initialization if it has already been initialized. If it has not been initialized, I fill in the refs object. This object will be used by the inidividual CRUD operations, so it needs to be filled in. The client, database, and collection that we need are found or created. At the end, we have resolve the promise by setting the initialized flag to true (thus future calls will be short circuited).

There is a race condition here. If two requests come in to a “cold” function, they will both go through the initialization together and potentially the “create database” and “create collection” will be duplicated, causing an exception in one of the requests. I’m sure I could fix this, but it’s a relatively rare case. Once the datbase and collection are created, the possibility of the condition goes away.

If you run this (either locally or within Azure Functions), you will see the following output in the log window:

function-docdb

If you’ve done something wrong, you will see the exception and you can debug it using the normal methods. Want a primer? I’ve written a blog post about it.

In the next post, I’ll cover inserting, deleting and updating records. Until then, check out the code on my GitHub repository.

Writing HTTP CRUD in Azure Functions

Over the last two posts, I’ve introduced writing Azure Functions locally and deploying them to the cloud. It’s time to do something useful with them. In this post, I’m going to introduce how to write a basic HTTP router. If you follow my blog and other work, you’ll see where this is going pretty quickly. If you are only interested in Azure Functions, you’ll have to wait a bit to see how this evolves.

Create a new Azure Function

I started this blog by installing the latest azure-functions-cli package:

npm i -g azure-functions-cli

Then I created a new Azure Function App:

mkdir dynamic-tables
cd dynamic-tables
func new

Finally, I created a function called todoitem:

dec15-01

Customize the HTTP Route Prefix

By default, any HTTP trigger is bound to /api/_function_ where function is the name of your function. I want full control over where my function exists. I’m going to fix this is the host.json file:

{
    "id":"6ada7ae64e8a496c88617b7ab6682810",
    "http": {
        "routePrefix": ""
    }
}

The routePrefix is the important thing here. The value is normally “/api”, but I’ve cleared it. That means I can put my routes anywhere.

Set up the Function Bindings

In the todoitem directory are two files. The first, function.json, describes the bindings. Here is the version for my function:

{
    "disabled": false,
    "bindings": [
        {
            "name": "req",
            "type": "httpTrigger",
            "direction": "in",
            "authLevel": "function",
            "methods": [ "GET", "POST", "PATCH", "PUT", "DELETE" ],
            "route": "tables/todoitem/{id:alpha?}"
        },
        {
            "type": "http",
            "direction": "out",
            "name": "res"
        }
    ]
}

This is going to get triggered by a HTTP trigger, and will accept five methods: GET, POST, PUT, PATCH and DELETE. In addition, I’ve defined a route that contains an optional string for an id. I can, for example, do GET /tables/todoitem/foo and this will be accepted. On the outbound side, I want to respond to requests, so I’ve got a response object. The HTTP Trigger for Node is modelled after ExpressJS, so the req and res objects are mostly equivalent to the ExpressJS Request and Response objects.

Write the Code

The code for this function is in todoitem/index.js:

/**
 * Routes the request to the table controller to the correct method.
 *
 * @param {Function.Context} context - the table controller context
 * @param {Express.Request} req - the actual request
 */
function tableRouter(context, req) {
    var res = context.res;
    var id = context.bindings.id;

    switch (req.method) {
        case 'GET':
            if (id) {
                getOneItem(req, res, id);
            } else {
                getAllItems(req, res);
            }
            break;

        case 'POST':
            insertItem(req, res);
            break;

        case 'PATCH':
            patchItem(req, res, id);
            break;

        case 'PUT':
            replaceItem(req, res, id);
            break;

        case 'DELETE':
            deleteItem(req, res, id);
            break;

        default:
            res.status(405).json({ error: "Operation not supported", message: `Method ${req.method} not supported`})
    }
}

function getOneItem(req, res, id) {
    res.status(200).json({ id: id, message: "getOne" });
}

function getAllItems(req, res) {
    res.status(200).json({ query: req.query, message: "getAll" });
}

function insertItem(req, res) {
    res.status(200).json({ body: req.body, message: "insert"});
}

function patchItem(req, res, id) {
    res.status(405).json({ error: "Not Supported", message: "PATCH operations are not supported" });
}

function replaceItem(req, res, id) {
    res.status(200).json({ body: req.body, id: id, message: "replace" });
}

function deleteItem(req, res, id) {
    res.status(200).json({ id: id, message: "delete" });
}

module.exports = tableRouter;

I use a tableRouter method (and that is what our function calls) to route the HTTP call to the write CRUD method I want to execute. It’s up to me to put whatever CRUD code I need to execute and respond to the request in those additional methods. In this case, I’m just returning a 200 status (OK) and some JSON data. One key piece is differentiating between a GET /tables/todoitem and a GET /tables/todoitem/foo. The former is meant to return all records and the latter is meant to return a single record. If the id is set, we call the single record GET method and if not, then we call the multiple record GET method.

What’s the difference between PATCH and PUT? In REST semantics, PATCH Is used when you want to do a partial update of a record. PUT is used when you want to send a full record. This CRUD recipe uses both, but you may decide to use one or the other.

Running Locally

As with the prior blog post, you can run func run test-func --debug to start the backend and get ready for the debugger. You can then use Postman to send requests to your backend. (Note: Don’t use func run todoitem --debug – this will cause a crash at the moment!). You’ll get something akin to the following:

dec15-02

That’s it for today. I’ll be progressing on this project for a while, so expect more information as I go along!

Deploying Azure Functions Automatically

In my last post, I went over how to edit, run and debug Azure Functions on your local machine. Eventually, however, you want to place these functions in the cloud. They are, after all, designed to do things in the cloud on dynamic compute. There are two levels of automation you can use:

  1. Continuous Deployment
  2. Automated Resource Creation

Most of you will be interested in continuous deployment. That is, you create your Azure Functions app once and then you just push updates to it via a source code control system. However, a true DevOps mindset requires “configuration as code”, so we’ll go over how to download an Azure Resource Manager (ARM) template for the function app and resource group.

Creating a Function App in the Portal

Creating an Azure Functions app in the portal is a straight forward process.

  1. Create a resource group.
  2. Create a function app in the resource group.

Log into the Azure portal. Select Resource Groups in the left-hand menu (which may be under the “More Services” link in your case), then click on the + Add link in the top bar to create a new resource group. Give it a unique name (it only has to be unique within your subscription), select a nominal location for the resource group, then click on Create:

create-rg-1

Once the resource group is created, click into it and then select + Add inside the resource group. Enter “Function App” in the search box, then select the same and click on Create

create-func-1

Fill in the name and select the region that you want to place the Azure Functions in. Ensure the “Consumption Plan” is selected. This is the dynamic compute plan, so you only pay for resources when your functions are actually being executed. The service will create an associated storage account for storing your functions in the same resource group.

Continuous Deployment

In my last blog post, I created a git repository to hold my Azure Function code. I can now link this git repository to the Function App in the cloud as follows:

  • Open the Function App.
  • Click the Function app settings link in the bottom right corner.
  • Click the Go to App Service Settings button.
  • Click the Deployment Credentials menu option.
  • Fill in the form for the deployment username and password (twice).
  • Click Save at the top of the blade.

You need to know the username and password of your git repository in the cloud that is attached to your Function App so that you can push to it. You’ve just set those credentials.

  • Click the Deployment options menu option.
  • Click Choose Source.
  • Click Local Git Repository
  • Click OK.

I could have just as easily linked my function app to GitHub, Visual Studio Team Services or BitBucket. My git repository is local to my machine, so a local git repository is suitable for this purpose.

  • Click the Properties menu option.
  • Copy the GIT URL field.

I now need to add the Azure hosted git repository as a remote on my local git repository. To do this, open a PowerShell console, change directory to the function app and type the following:

git remote add azure <the-git-url>
git push azure master

This will push the contents of the git repository up to the cloud, which will then do a deployment of the functions for you. You will be prompted for your username and password that you set when setting up the deployment credentials earlier.

create-func-2

Once the deployment is done, you can switch back to the Function App in the portal and you will see that your function is deployed. An important factor is that you are now editing the files associated with the function on your local machine. You can no longer edit the files in the cloud, as any changes would be overwritten by the next deployment from your local machine. To remind you of this, Azure Functions displays a helpful warning:

create-func-3

If you edit your files on the local machine, remember to push them to the Azure remote to deploy them.

Saving the ARM Template

You are likely to only need the Azure Function process shown above. However, in case you like checking in the configuration as code, here is how you do it. Firstly, go to your resource group:

create-rg-2

Note the menu item entitled Automation script – that’s the one you want. The portal will generate an Azure Resource Manager (ARM) template plus a PowerShell script or CLI script to run it. Click on Download to download all the files – you will get a ZIP file.

Before extracting the ZIP file, you need to unblock it. In the File Explorer, right-click on the ZIP file and select Properties.

create-rg-3

Check the Unblock box and then click on OK. You can now extract the ZIP file with your favorite tool. I just right-click and select Extract All….

Creating a new Azure Function with the ARM Template

You can now create a new Azure Function App with the same template as the original by running .\deploy.ps1 and filling in the fields. Yep – it’s that simple!

Creating and Debugging Azure Functions Locally

I’ve written about Azure Functions before as part of my Azure Mobile Apps series. Azure Functions is a great feature of the Azure platform that allows you to run custom code in the cloud in a “serverless” manner. In this context, “serverless” doesn’t mean “without a server”. Rather, it means that the server is abstracted away from you. In my prior blog post, I walked through creating an Azure Function using the web UI, which is a problem when you want to check your Azure Functions in to source code and deploy them as part of your application.

UPDATE: Azure App Service have released a blog on the new CLI tools.

This is the first in a series of blog posts. I am going to walk through a process by which you can write and debug Azure Functions on your Windows 10 PC, then check the code into your favorite SCCM and deploy in a controlled manner. In short – real world.

Getting Ready

Before you start, let’s get the big elephant out of the way. The actual runtime Windows only. Sorry, Mac Users. The run-time relies on the 4.x .NET Framework, and you don’t have that. Boot into Windows 10. You can still create functions locally, but you will have to publish them to the cloud to run them. There is no local runtime on a Mac.

To get your workstation prepped, you will need the following:

  • Node
  • Visual Studio Code
  • Azure Functions Runtime

Node is relatively easy. Download the Node package from nodejs.org and install it as you would any other package. You should be able to run the node and npm programs from your command line before you continue. Visual Studio Code is similarly easily downloaded and installed. You can download additional extensions if you like. If you write functions in C#, I would definitely download the C# extension.

The final bit is the Azure Functions Runtime. This is a set of tools produced by the Azure Functions team to create and run Functions locally and are based on Yeoman. To install:

npm install -g yo generator-azurefunctions azure-functions-cli

WARNING There is a third-party module called azure-functions which is not the same thing at all. Make sure you install the right thing!

After installing, the func command should be available:

func-1

Once you have these three pieces, you are ready to start working on Azure Functions locally.

Creating an Azure Functions Project

Creating an Azure Functions project uses the func command:

mkdir my-func-application
cd my-func-application
func init

Note that func init creates a git repository as well – one less thing to do! Our next step is to create a new function. The Azure Functions CLI uses Yeoman underneath, which we can call directly using yo azurefunctions:

func-2

You can create as many functions as you want in a single function app. In the example above, I created a simple HTTP triggered function written in JavaScript. This can be used as a custom API in a mobile app, for example. The code for my trigger is stored in test-func\index.js:

module.exports = function(context, req) {
    context.log('Node.js HTTP trigger function processed a request. RequestUri=%s', req.originalUrl);

    if (req.query.name || (req.body && req.body.name)) {
        context.res = {
            // status: 200, /* Defaults to 200 */
            body: "Hello " + (req.query.name || req.body.name)
        };
    }
    else {
        context.res = {
            status: 400,
            body: "Please pass a name on the query string or in the request body"
        };
    }
    context.done();
};

and the binding information is in test-func\function.json:

{
    "disabled": false,
    "bindings": [
        {
            "authLevel": "function",
            "type": "httpTrigger",
            "direction": "in",
            "name": "req"
        },
        {
            "type": "http",
            "direction": "out",
            "name": "res"
        }
    ]
}

Running the function

To run the Azure Functions Runtime for your function app, use func run test-func.

The runtime is kicked off first. This monitors the function app for changes, so any changes you do in the code will be reflected as soon as you save the file. If you are running something that can be triggered manually (like a cron job), then it will be run immediately. For my HTTP trigger, I need to hit the HTTP endpoint – in this case, http://localhost:7071/api/test-func.

Note that the runtime is running with the version of Node that you installed and it is running on your local machine. Yet it still can be triggered by whatever you set up. If you set up a blob trigger from a storage account, then that will trigger. You have to set up the environment properly. Remember that App Service (and Functions) app settings appear as environment variables to the runtime. When you run locally, you will need to manually set up the app settings by setting an environment variable of the same name. Do this before you use func run for the first time.

Debugging the function

Running the function is great, but I want to debug the function – set a breakpoint, inspect the internal state of the function, etc. This can be done easily in Visual Studio Code as the IDE has an integrated Node debugger.

  • Run func run test-func --debug
  • Go into the debugger within Visual Studio Code, and set a breakpoint
  • Switch to the Debugger tab and hit Start (or F5)

    func-3

  • Trigger the function. Since I have a HTTP triggered function, I’m using Postman for this:

    func-4

  • Note that the function is called and I can inspect the internals and step through the code:

    func-5

You can now step through the code, inspect local variables and generally debug your script.

Next up

In the next article, I’ll discuss programmatically creating an Azure Function app and deploying our function through some of the deployment mechanisms we have available to us in Azure App Service.