Using ModernHttpClient with Azure Mobile Apps

Azure Mobile Apps comes with a great client SDK that creates a client connection on its own. In the .NET library, that’s provided by System.Net.Http.HttpClient. If you take a look at the Xamarin implementation of System.Net.Http.HttpClient (for example, for Android), you should note that it uses raw I/O to a TCP/IP socket – the underlying building block of all internet communication that dates way back to the origins of the Internet and TCP/IP.

So what, you might ask? Well, doing raw I/O may not be the best way of communicating with a backend on all platforms. Both iOS and Android contains in-built raw libraries for doing HTTP and they deal with the underlying OS libraries to be more efficient. In the Android case, that’s HttpUrlConnection and in the iOS case, that’s NSURLConnection. In addition, there are other libraries that offer improvements for simple operations. Your client may be doing other (non-Mobile Apps) operations for this.

Fortunately, there is an answer. Paul Betts has produced a nice client alternative as a Portable Class Library called modernhttpclient and it’s open-source. The Portable Class Library (or PCL) is important because it means it can run on Xamarin as well as Windows .NET projects.

Now, I’m not saying you HAVE to use ModernHttpClient. Far from it – Azure Mobile Apps is perfectly fine as-is and you won’t have any problems. If you want to use ModernHttpClient, then go for it – it “just works” – almost.

ModernHttpClient is a drop-in replacement for System.Net.Http.HttpClient. You add the NuGet package and away you go. It “just works”. If you really want to go all-in, then the suggestion is to replace the HttpClient constructor with this:

var httpClient = new HttpClient(new NativeMessageHandler());

Here is where the “almost” comes in. Azure Mobile Apps constructs the httpClient with a set of handlers that it creates. You can add things to the list of handlers in the MobileServiceClient constructor, but you can’t affect the actual constructor of the HttpClient that the MobileServiceClient uses.

Fortunately, this is easy to integrate. Normally, you would create a MobileServiceClient like this:

var client = new MobileServiceClient("https://mysite.azurewebsites.net");

Instead, you add the message handler on the end:

var client = new MobileServiceClient("https://mysite.azurewebsites.net",
    new ModernHttpClient.NativeMessageHandler());

This will then adjust the HttpClient accordingly.

Multiple Message Handlers?

If you have read all the documentation and my blog series, you will note that this is acting as just another message handler. You can actually specify multiple message handlers, but – and this is important – the NativeMessageHandler() MUST be the last one listed. For instance, let’s say you’ve defined a Delegating Handler to do some logging. You would need to do the following:

var handlers = new HttpMessageHandler[]
{
    new LoggingDelegatedHandler(),
    new NativeMessageHandler()
};
var client = new MobileServiceClient("https://mysite.azurewebsites.net", handlers);

Consider the request as going through layers of an onion, with the request actually being done by the inner-most layer. The order of the layers is listed in the array with the outer-most layer listed first.

Adjusting the HTTP Request with Azure Mobile Apps

Azure Mobile Apps provides an awesome client SDK for dealing with common mobile client problems – data access, offline sync, Notification Hubs registration and authentication.  Sometimes, you want to be able to do something extra in the client.  Perhaps you need to adjust the headers that are sent, or perhaps you want to understand the requests by doing diagnostic logging.  Whatever the reason, Azure Mobile Apps is extensible and can easily handle these requirements.

Android (Native)

You can implement a ServiceFilter to manipulate requests and responses in the HTTP pipeline.  The general recipe is as follows:

ServiceFilter filter = new ServiceFilter() {
    @Override
    public ListenableFuture handleRequest(ServiceFilterRequest request, NextServiceFilterCallback next) {

        // Do pre-HTTP request requirements here
        request.addHeader("X-Custom-Header", "Header Value");  // Example: Adding a Custom Header
        Log.d("Request to ", request.getUrl());                // Example: Logging the request

        ListenableFuture responseFuture = next.onNext(request);

        Futures.addCallback(responseFuture, new FutureCallback() {
            @Override
            public void onFailure(Throwable exception) {
                // Do post-HTTP response requirements for failures here
                Log.d("Exception: ", exception.getMessage());  // Example: Logging an error
            }

            @Override
            public void onSuccess(ServiceFilterResponse response) {
                // Do post-HTTP response requirements for success here
                if (response != null && response.getContent() != null) {
                    Log.d("Response: ", response.getContent());
                }
            }
        });
        
        return responseFuture;
    }
};

MobileServiceClient client = new MobileServiceClient("https://xxx.azurewebsites.net", this).withFilter(filter);

You can think of the ServiceFilter as a piece of middleware that wraps the existing request/response from the server.

iOS (Native)

Similar to the Android case, you can wrap the request in a filter. For iOS, the same code (once translated) works in both Swift and Objective-C. Here is the Swift version:

class CustomFilter: NSObject, MSFilter {

    func handleRequest(request: NSURLRequest, next: MSFilterNextBlock, response: MSFilterResponseBlock) {
        var mutableRequest: NSMutableURLRequest = request.mutableCopy()

        // Do pre-request requirements here
        if !mutableRequest.allHTTPHeaderFields["X-Custom-Header"] {
            mutableRequest.setValue("X-Custom-Header", forHTTPHeaderField: "Header Value")
        }

        // Invoke next filter
        next(customRequest, response)
    }
}

// In your client initialization code...
let client = MSClient(applicationURLString: "https://xxx.azurewebsites.net").clientWithFilter(CustomFilter())

The .clientWithFilter() method clones the provided client with the filters.

JavaScript & Apache Cordova

As you might exepct given the Android and iOS implementations, the JavaScript client (and hence the Apache Cordova implementation) also uses a filter – this is just a function that the request gets passed through:

function filter(request, next, callback) {
    // Do any pre-request requirements here
    console.log('request = ', request);                     // Example: Logging
    request.headers['X-Custom-Header'] = "Header Value";    // Example: Adding a custom here
    
    next(request, callback);
}

// Your client initialization looks like this...
var client = new WindowsAzure.MobileServiceClient("https://xxx.azurewebsites.net").withFilter(filter);

Xamarin / .NET

The equivalent functionality in the .NET world is a Delegating Handler. The implementation and functionality are basically the same as the others:

public class MyHandler: DelegatingHandler
{
    protected override async Task SendAsync(HttpRequestMessage message, CancellationToken token)
    {
        // Do any pre-request requirements here
        request.Headers.Add("X-Custom-Header", "Header Value");

        // Request happens here
        var response = await base.SendAsync(request, cancellationToken);

        // Do any post-request requirements here

        return response;
    }
}

// In your mobile client code:
var client = new MobileServiceClient("https://xxx.azurewebsites.net", new MyHandler());

General Notes

There are some HTTP requests that never go through the filters you have defined. A good example for this is the login process. However, all requests to custom APIs and/or tables get passed through the filters.

You can also wrap the client multiple times. For example, you can use two separate filters – one for logging and one for the request. In this case, the filters are executed in an onion-like fashion – The last one added is the outer-most. The request goes through each filter in turn until it gets to the actual client, then the response is passed through each filter on its way out to the requestor.

Finally, note that this is truly a powerful method allowing you to change the REST calls that Azure Mobile Apps makes – including destroying the protocol that the server and client rely on. Certain headers are required for Azure Mobile Apps to work, including tokens for authentication and API versioning. Use wisely.

(Cross-posted to the Azure App Service Blog)

30 Days of Zumo.v2 (Azure Mobile Apps): Day 30: Catching Up

I’ve been doing this blog series – 29 blog articles – since late March. In the cloud world, that’s a lifetime, so let’s take a look at what new things you need to be aware of when developing Mobile apps.

Microsoft acquired Xamarin

There are really three distinct technologies you need to understand. Native development uses the platform specific tools (XCode or Android Development Studio) and languages (Swift, Objective-C and Java) to develop the mobile app. These offer the highest level of compatibility with the features of the mobile phone. However, there is zero code reuse between them. A common developer theme is DRY – Don’t Repeat Yourself. I’m developing two apps – one for Android and one for iOS – that do the same thing. Why can’t I reuse the models and business logic within my code? I’m not a fan of native development for that reason.

Apache Cordova places your app in a web view. I like this when there is an associated website that is already responsive. It allows you to wrap your existing web code (which is likely a SPA these days) into an app. You get access to the same libraries as your web code and you can leverage a good chunk of the talent you have for developing web code when developing your mobile app. It does have drawbacks, though. The UI is defined inside of HTML/CSS ultimately, which means it’s relatively hard to get a platform consistent UI. The UI will either look like an Android app or like an iOS app, even on the alternate platform. In addition, the code runs inside a Web view. This is a sandboxed area that has limitations. The limitations are becoming less important. Basically, if you want to access the bleeding edge of hardware, this is not the platform for you. Finally, I find performance lags that of native apps – there is a memory and CPU overhead. This may not be important in small apps, but as the apps grow, so does the memory and CPU requirements – the ceiling is lower.

The final segment is cross-platform apps and that’s where Xamarin comes in. Xamarin allows you to write apps in a common language – C# or F# with a .NET runtime. However, the apps compile to native code. This has two advantages. Firstly, you get native performance – the CPU and memory considerations I mentioned with Hybrid Apps are at the same level as for native apps. Secondly, you get access to the entire API surface that the device offers. Unlike native apps, you can encapsulate your models and business logic in a shared library, allowing for code re-use.

In the past (pre-acquisition), I was loath to suggest Xamarin for small projects because of the cost associated with it. With the acquisition, Microsoft made the technology free. For small groups, you can use Xamarin with Visual Studio Community. For bigger groups, the various Visual Studio offerings provide the appropriate licensing. In addition, Microsoft made Xamarin Studio free for Mac users, which – in my experience – is the platform of choice for most of the mobile developers. This means you don’t have to any excuse to not use Xamarin for developing your native apps.

Those of you that have looked into Xamarin before will know that there are two “modes” for developing Xamarin apps. Xamarin.iOS and Xamarin.Android provide distinct advantages by providing access to the platform features directly, whereas Xamarin.Forms provides a cross-platform UI capability as well. I am recommending that teams use Xamarin.Forms for cross-platform enterprise apps (where differences in the look and feel of the app are less important), but using the Xamarin.iOS / Xamarin.Android for consumer apps. This allows you to tune the UI to be very platform specific.

File Sync for Node Backends

This is really recent. The team released the azure-mobile-apps-files NPM package to allow the implementation of File Sync with node backends. It’s definitely still in preview – like most of the other functionality in File Sync, but it means you don’t have to make a choice of backend based solely on features available. The Azure Mobile Apps Node SDK has been one of our most active release tracks, and I expect more features being pulled into the alternate Node implementation.

Apache Cordova Offline Sync

I believe I’ve mentioned this several times, but it bears repeating. Azure Mobile Apps development – all the server and client SDKs – are open source. All the issues are on GitHub Issues, the repositories are on GitHub and the team develops in the open. That’s why I can safely say that Apache Cordova Offline Sync is coming “Real Soon”. If you want to view any of the repositories, you can do a search on the Azure GitHub organization page. Got a bug or an idea? File an issue. We don’t bite!

RIP Azure Mobile Services

Unfortunately, I had to close down Azure Mobile Services. I’m rather sad about that since I think it was an awesome service. However, it was reliant on older technology that couldn’t easily be upgraded. As a result, we were handling customer issues dealing primarily with new libraries not working on older versions of Node. To aid the process, we’ve done two things. The first is a single-click migration of the Mobile Service to Azure App Service. It doesn’t change any of the code (and you can clone the git repository and update the code any way you see fit). My recommendation, however, is to upgrade your code to Azure Mobile Apps and utilize all the good things we’ve been doing. Upgrading also allows you to use any version of Node that you wish, which is a constant request.

To aid in upgrading your site to Azure Mobile Apps, we’ve also released a compatibility layer for node sites (which accommodates about 75% of all Mobile Services). This will take your existing mobile services site (that you have cloned after migration) and turn it into an Azure Mobile Apps site. Afterwards, your code-behind files for table controllers and APIs are all compatible with Azure App Service. Then all you have to do is publish your site, upgrade the SDK in your clients (including changing the URL to point to your new App Service) and publish those new clients to the app store. At this point you will be running both sites (the v1 site and the v2 site) in parallel.

There is a small question of database compatibility. You can fix this with Views (I mentioned the process for this in Day 19 – it needs to be adjusted for the new situation). However, once that is done, you are ready to rock the new environment.

One of the biggest gotchas I see are people running Azure Mobile Services but referencing newer SDKs & Documentation, or people running Azure Mobile Apps and referencing older SDKs and Documentation. We get it – it’s confusing. MAke sure your libraries, documentation and lingo are all in line on both the backend and frontend / client.

When Will .NET Core be supported?

I’ve lost count how many times I’ve been asked this. ASP.NET Core is an awesome technology and I’m looking forward to stability and GA for it. However, that isn’t the only technology we use in the stack. We also use Entity Framework, System.Web.OData, Automapper, and others. Until the whole stack is compatible with .NET Core, we won’t be releasing a .NET Core version of the Azure Mobile Apps Server SDK. Rest assured, we are in touch with the right teams and it’s definitely on our radar.

Contacting the team?

One of the great things about this team is how involved they are in the community. There are a multitude of methods to get in touch. If you have a problem, then the best way is to post a question on Stack Overflow. This reaches the widest possible audience. You can also reach us via the Azure Forums, on Twitter, or via comments on our documentation (although we prefer that you post on Stack Overflow if you have an issue). You can also open an issue on one of our GitHub repositories.

Finally, the App Service team has a new team blog. I’ll be publishing further Azure Mobile Apps posts there instead of here. This blog is going back to (mostly) my own side work.

I hope you’ve enjoyed reading the last two months worth of posts as much as I’ve enjoyed writing them. I’ve included an Index – take a look under Pages to your right.

30 Days of Zumo.v2 (Azure Mobile Apps): Day 29: Post-Processing Files with Azure Functions

There are times when you don’t want things to happen as a result of an API call – maybe it’s a cron job that cleans out deleted records; or maybe it’s a post-processing job that looks at images you have uploaded and runs some process on them (such as facial recognition, cognitive services or just plain image handling). Whatever the case, this isn’t handled within Azure Mobile Apps easily. It’s time for a new service.

That service is Azure Functions. Azure Functions is a dynamic compute facility that runs “code” when triggered by an event. It responds to triggers by running code. I introduced a whole bunch of things there, so let’s look at each one.

Dynamic Compute is awesome and it’s the central theme of functions. You may have heard of the “serverless” movement. It’s the new word for an old idea – you don’t have to manage the resources yourself – the platform does it for you. If you had used the Parse server or another MBaaS in the past – same concept. You never had to worry about the VM running the code in those services. Dynamic Compute grabs the resources necessary to run the job and then releases the resources after it is run. You only pay for the resources that you consume. It’s like the ultimate endpoint of the promise of cloud computing.

Events or Triggers can be anything. Yes, they could be a HTTP endpoint (although I am not recommending that right now – I’ll explain at the end). They could also be a timer, an object placed into a storage queue or a file uploaded to blob storage.

Functions are not Web Apps. You can’t use the same project you used for your mobile backend to define your functions. Right now, the best way to define your functions is through the Azure Portal.

Creating a Function for cleaning up the TodoItem table

Let’s create a simple function based on a timer. Our mobile TodoItem table has soft-delete enabled. That means that instead of deleting the records directly, the records are marked as deleted by setting the deleted column to 1. I need to clean up those, so let’s define a simple function that runs every Sunday morning at 3am and deletes all the records marked for deletion. It’s just a SQL statement change if you want to be more prescriptive about this and delete all the records marked for deletion that are older than 7 days, for example.

Start by going to https://functions.azure.com, clicking on the big friendly Get Started button and logging in with your Azure credentials. Click on the + NEW button then select Web + Mobile, then Function App. Give the function app a name, select a resource group and app service plan and click on create.

Screen Shot 2016-06-01 at 7.11.08 PM

Note that creating a function app creates a storage area as well. This is required and has to do with a common storage area for your functions since you don’t actually know where they will run. It’s not serverless per-se (which is a silly term, IMHO – there is always a server), but that is what the term means. You don’t define a server for this app to run on, but it needs a little bit of shared storage to hold your code, logs, etc.

Once the function app has been deployed, the function app blade will open. It’s very different from the other blades with a unique management experience.

Screen Shot 2016-06-01 at 7.17.16 PM

In this case, I want to create something akin to a cron job – a timer job. I’m going to write my timer job in C#, but you could write something similar in Node.js, for example. You can also click on the + New function, in which case you get access to the full range of scenarios. The Timer Function is in the Core set of scenarios near the end. Give the function a name. The Cron schedule I picked is 0 0 3 ? * SUN *, which is one that is compatible with Quartz – a great .NET library for doing cron-type things.

Screen Shot 2016-06-01 at 7.22.09 PM

Once I click on Create, I am in a code editor. Here is the code for a basic function that deletes all the records marked as deleted:

#r "System.Configuration"
#r "System.Data"

using System;
using System.Configuration;
using System.Data.SqlClient;
using System.Threading.Tasks;

public static async Task Run(TimerInfo myTimer, TraceWriter log)
{
    var connStr = ConfigurationManager.ConnectionStrings["sqldb"].ConnectionString;
    
    using (var conn = new SqlConnection(connStr))
    {
        conn.Open();
        var sqlCmd = "DELETE FROM TodoItems WHERE deleted = 'True'";
        using (var cmd = new SqlCommand(sqlCmd, conn))
        {
            var rows = await cmd.ExecuteNonQueryAsync();
            log.Info($"{rows} rows deleted.");
        }
    }
}

A couple of notes. Firstly, note the #r at the top – these bring in references – just as if you were adding a reference to a project in visual studio. Secondly, it’s not in a class. You could wrap this in a namespace, and then a static class and add a main function to make it a console app. In fact, this is a great way of testing locally. However, you don’t really need it. Once you click on Save, the function is compiled and will run on the schedule you set.

If you are using the ASP.NET version of the Mobile Apps Server SDK, then your database table will be plural – i.e. if you have a model called TodoItem, the database table will be TodoItems. If you are using the Node.js version of the Mobile Apps Server SDK, the table and the table controller are the same thing – make sure you have the right SQL statement for your backend!

Before I can run this, I need a SQL Connection String. I’ve named it (imaginatively) “sqldb”. To set that connection string:

  • Click on function app settings at the top of the page
  • Click on Go to App Service Settings
  • In the new blade, click on Application Settings (you will probably have to scroll down)
  • Scroll down to Connection Strings, enter sqldb (or whatever you used) in the Name field. Enter your SQL Connection String in the Value box
  • Click on Save

I tend to use cut-and-paste from my Mobile App for this step so I get the SQL Connection String right.

Back in your Function app (and more importantly, in your Function)… At the bottom of the page, there is a Run button, just waiting to be clicked. It’s at this point I realized that my cron statement was wrong. Note that the log tells you exactly what is wrong pretty much straight away. When you are creating a timer Function, your cron string needs to be correct. If you are wondering what a correct cron string looks like, check out Wikipedia. My cron string should be 0 0 3 * 0 (five elements – sec min hr month day-of-week). Don’t try to use any of the extended formats – just the basics. You can change this in the Integrate tab.

Now you can click on Run. You will see a 202 Accepted in the panel to the right of the Run button (the Output panel) and the logs will capture any exceptions thrown by the function.

A more complicated example: Post-processing Images

Now that I’ve got the basics covered, let’s do something a little more complicated – image processing. In this example, I’m going to run a function whenever something is uploaded to a specific container in my blob storage. There is a mechanism whereby you can insert a record into an Azure Mobile Apps table by calling the table controller. However I would need to set up an “app key”. App keys are insecure and you definitely don’t want them to be out in the wild. However, for the purposes of this project, your app key isn’t going into the wild – it’s staying entirely within Azure. It’s still not secure. You are setting up a back door, and back doors have a tendency to leak out or get misused. As a result, I’m going to do the same mechanism as above to insert data – I’m going to use a SQL Command to insert directly into the SQL database.

Before continuing, you should set up a new Mobile Apps Table Controller called Image with a model that contains Filename, Height and Width fields. If you’ve followed along, I covered this way back in Day 17. If you can’t figure it out, check out the Backend for the file-upload project on my GitHub repository. I’ve added the code there.

I’m going to add a new Function to my function app. This time I’m going to select a BlobTrigger for C#. After naming my function, I need to select my storage account and the path to the storage. The path can be a specific file within a container, or it can be the container name itself. I’m going to use the container name. So, here is my trigger definition:

cap1

Switching over to the code:

using System;

public static void Run(string myBlob, TraceWriter log)
{
    log.Info($"C# Blob trigger function processed: {myBlob}");
}

The myBlob value will get filled with the contents of the file. That’s a problem for me because my files are likely images. There are alternatives to a string – you can use a Stream or a CloudBlockBlob, for example. The Stream can be used if you don’t care about the additional metadata and only care about the contents of the file. I’m going to use a CloudBlockBlob.

Looking for the variables you can use as parameters for the Run method? Check out the cheat sheet (link downloads a PDF)

Note that the Log contains the contents of your blob, which is likely to be binary if you are uploading images, then that goes into the log immediately. Time to write some code for the image processing:

#r "System.Configuration"
#r "System.Data"
#r "System.Drawing"
#r "Microsoft.WindowsAzure.Storage"

using System;
using System.Configuration;
using System.Data.SqlClient;
using System.Drawing;
using System.IO;
using System.Threading.Tasks;
using Microsoft.WindowsAzure.Storage.Blob;

public static async Task Run(CloudBlockBlob blob, TraceWriter log)
{
    log.Info($"Processing file: {blob.Name}");

    // Create an image object from a stream
    Bitmap image = new Bitmap(await blob.OpenReadAsync());
    
    // Get the information for the table
    int width = image.Width;
    int height = image.Height;
    string id = Guid.NewGuid().ToString();
    
    // Update the Images database.  Note that we have a bug - if I am using
    // the same filename, then this code adds a new record - it doesn't update
    // the old record.
    var connStr = ConfigurationManager.ConnectionStrings["sqldb"].ConnectionString;
    log.Info($"Using Connection String {connStr}");
    
    using (var conn = new SqlConnection(connStr))
    {
        conn.Open();
        var sqlCmd = "INSERT INTO Images (Id, Filename, Height, Width, Deleted) VALUES(@id, @name, @height, @width, @deleted)";
        var cmd = new SqlCommand(sqlCmd, conn);
        
        cmd.Parameters.AddWithValue("@id", id);
        cmd.Parameters.AddWithValue("@name", blob.Name);
        cmd.Parameters.AddWithValue("@height", height);
        cmd.Parameters.AddWithValue("@width", width);
        cmd.Parameters.AddWithValue("@deleted", false);
        
        log.Info($"Executing INSERT:('{id}','{blob.Name}',{height},{width})");
        await cmd.ExecuteNonQueryAsync();
    }
}

This is fairly basic code. I load the stream into a Bitmap object, grab the metadata and insert into the database table. I do need to ensure that the database table exists (which I did in this example by adding an Image table to my application).

Wrap Up

Functions is an awesome new feature for mobile developers. It allows you to do some relatively complex processing usign ad-hoc resources that are only used (and paid for) when required. The possibilities are really astounding. For instance, Cognitive Services – being able to determine what a photo is of and doing automatic tagging of the image – or Machine Learning are possible. These were really only the province of those that wanted to get into the backend processing. In addition to the triggers, you can also add input and output bindings. This allows you, for example, to automatically push the blob Uri into a queue or deliver a push notification or create a Mobile Apps record – without the additional code that goes along with it.

I mentioned at the beginning why I would not recommend using Functions as a HTTP endpoint (despite all the resourcing being there). A lot of endpoints incur a startup cost. For example, connecting to a quiet App Service spins up the service, which connects to your connected database, ensures all the tables are created, etc. Since dynamic compute spins up resources for every request, you are going to incur this cost every request. This limitation is a small one – there are plenty of reasons to use Functions without making it into a web service as well.

Functions is still in preview, but I can’t wait to see what happens with it. The team is active everywhere and it’s free right now, so give it a spin.

Want the code for the changes I made for images? They are at my GitHub Repository.

30 Days of Zumo.v2 (Azure Mobile Apps): Day 28 File Handling (v2)

I covered the mechanics of uploading to Blob Storage in my last blog post. However, there were certain caveats with that – you had to associate a file to a record by hand-coding the association (not really friendly) and there was no offline support. Azure Mobile Apps has a File Management SDK that is in preview. It requires the ASP.NET Server SDK (so no Node support) and a .NET client (Xamarin or UWP, for example), so no iOS native, Android native or Cordova support. The client and server support will be rounded out by the time it hits GA. Today, I’m going to expand my file-upload solution to support cross-platform offline-capable attachment of images to the Todo Item.

To start:

  1. I’ve created an Azure Mobile App
  2. I’ve connected a SQL Azure instance to the Mobile App
  3. I’ve connected a Storage Account instance to the Mobile App
  4. I’ve configured Authentication / Authorization with Google Authentication
  5. I’ve deployed an ASP.NET TodoItem from my last post

Short version – I’ve done everything that makes the file handling v1 work. I’m starting from the ending point of the last tutorial. You can find the code on my GitHub repository for both the server and the client (open the file-upload solution).

The Server Side

Before I can serve files, I need a few more endpoints on the table controller:

  • POST /tables/{table}/{id}/StorageToken will give me a SAS token for the connected Azure Storage account that allows me to upload a new file.
  • GET /tables/{table}/{id}/MobileServiceFiles will give me a list of the files associated with a particular table record, together with some meta-data on where they are.
  • DELETE /tables/{table}/{id}/MobileServicesFiles/{name} will delete the named file.

My client application will also interact with the Azure Storage service directly to upload or download the actual files. This allows me to take advantage of any Azure Storage features I want (like upload-restart or progress bars, for example).

To implement these endpoints, I need to add the Microsoft.Azure.Mobile.Server.Files NuGet package to my server (don’t forget it’s a prerelease package at the moment, so you will only see if if you check the Include prerelease box). I can then add the StorageController – a new type of controller that sits alongside your TableController:

using Backend.DataObjects;
using Microsoft.Azure.Mobile.Server.Files;
using Microsoft.Azure.Mobile.Server.Files.Controllers;
using System.Net.Http;
using System.Threading.Tasks;
using System.Web.Http;

namespace Backend.Controllers
{
    public class TodoItemStorageController : StorageController<TodoItem>
    {
        [HttpPost]
        [Route("tables/TodoItem/{id}/StorageToken")]
        public async Task<HttpResponseMessage> StorageToken(string id, StorageTokenRequest value)
            => Request.CreateResponse(await GetStorageTokenAsync(id, value));

        [HttpGet]
        [Route("tables/TodoItem/{id}/MobileServiceFiles")]
        public async Task<HttpResponseMessage> GetFiles(string id)
            => Request.CreateResponse(await GetRecordFilesAsync(id));

        [HttpDelete]
        [Route("tables/TodoItem/{id}/MobileServiceFiles/{name}")]
        public Task Delete(string id, string name) => base.DeleteFileAsync(id, name);
    }
}

Each method implements one of the three endpoints that make up the required interface. Most of the logic is abstracted into the provided StorageController base object. I also have to ensure that the routes are registered. If I had started with an MVC app, this may already be done. In this case, I’m going to register the routes that are defined with these standard attributes in my App_Start/Startup.MobileApp.cs file:

        public static void ConfigureMobileApp(IAppBuilder app)
        {
            var config = new HttpConfiguration();

            // Register the StorageController routes
            config.MapHttpAttributeRoutes();

            new MobileAppConfiguration()
                .UseDefaultConfiguration()
                .ApplyTo(config);

Once this is done, I can publish the project and move on to the client.

What about authentication

If you want authentication on the files (and you should, especially if your main table is authenticated), then you can just add the [Authorize] attribute to the class or individual routes. If you want to do more complex processing (for example, only allow uploads and downloads to records you own), you can look up the record with the DomainManager or DbContext. Let’s take a look at an example:

    [Authorize]
    public class TodoItemStorageController : StorageController<TodoItem>
    {
        IDomainManager<TodoItem> domainManager;

        public TodoItemStorageController() : base()
        {
            var context = new MobileServiceContext();
            domainManager = new EntityDomainManager<TodoItem>(context, Request);
        }

        [HttpPost]
        [Route("tables/TodoItem/{id}/StorageToken")]
        public async Task<HttpResponseMessage> StorageToken(string id, StorageTokenRequest value)
        {
            var principal = this.User as ClaimsPrincipal;
            var sid = principal.FindFirst(ClaimTypes.NameIdentifier).Value;
            var item = await domainManager.LookupAsync(id);
            if (item.UserId.Equals(sid))
                return Request.CreateErrorResponse(HttpStatusCode.BadRequest, "Not Authorized");

            return Request.CreateResponse(await GetStorageTokenAsync(id, value));
        }

In this case, I’m storing the SID in the UserId field of the record – I compare the record to the authentication coming in via the request and make a decision on what to send back based on that. You can get the email address the same way as you would from the Personal Table article if you are storing authentication that way.

The Client Implementation

I’m using a Xamarin Forms implementation of my client. In terms of the UX, I’m going to make it simple. I want to add an “Attach” button to my ItemDetail page. When the user clicks on the Attach button, it pops up a file dialog (or photo gallery dialog) that allows the attachment of a file to the item. That item (once uploaded) will then appear in the list on the Item Detail page.

Step 1: Add the NuGet Packages

You should already have the Microsoft.Azure.Mobile.Client.SQLiteStore package since you are doing offline sync. The Files SDK uses this for storing the meta data on the files. In addition, you will want the Microsoft.Azure.Mobile.Client.Files package and a package that implements some sort of storage interface. I’m going to use the PCLStorage package for this in the iOS and Android platforms and Windows.Storage in the UWP platform. You could use something else, as long as it exposes an interface you can use. I like the PCLStorage package because it allows targeting of just about everything.

Don’t forget that the Microsoft.Azure.Mobile.Client.Files package is a prerelease package. The client also depends on WindowsAzure.Storage, which is a dependency. Xamarin support is only available in the preview versions of WindowsAzure.Storage (at time of writing).

Step 2: Create a Platform dependency service

There are a couple of platform-dependent things I need to do:

  1. Take a photo or select an image or file
  2. Associate a piece of storage to use for files associated with a record

Storage is one of those things that is different on each platform. Fortunately, Xamarin Forms has a mechanism for dealing with platform specific code called DependencyService – I’ve already used this in my application for the Login Provider, and I’m going to use it again for the Files Provider.

In the Shared Project, I’ve added the following:

using Microsoft.WindowsAzure.MobileServices.Files;
using Microsoft.WindowsAzure.MobileServices.Files.Metadata;
using Microsoft.WindowsAzure.MobileServices.Sync;
using System.Threading.Tasks;

namespace XamarinTodo.Services
{
    public interface IFileProvider
    {
        Task<IMobileServiceFileDataSource> GetFileDataSource(MobileServiceFileMetadata metadata);
        Task<string> GetImageAsync();
        Task DownloadFileAsync<T>(IMobileServiceSyncTable<T> table, MobileServiceFile file, string filename);
        Task<string> CopyItemFileAsync(string itemId, string filePath);
        Task<string> GetLocalFilePathAsync(string itemId, string fileName);
        Task DeleteLocalFileAsync(MobileServiceFile fileName);
    }
}

I also need an implementation within each platform project. Here, for example, is the UWP edition:

using Microsoft.WindowsAzure.MobileServices.Files;
using Microsoft.WindowsAzure.MobileServices.Files.Metadata;
using Microsoft.WindowsAzure.MobileServices.Files.Sync;
using Microsoft.WindowsAzure.MobileServices.Sync;
using Plugin.Media;
using System;
using System.IO;
using System.Threading.Tasks;
using Windows.Storage;
using XamarinTodo.Services;

[assembly: Xamarin.Forms.Dependency(typeof(XamarinTodo.UWP.Services.UWPFileProvider))]
namespace XamarinTodo.UWP.Services
{
    public class UWPFileProvider : IFileProvider
    {
        /// <summary>
        /// Copy the newly found file into our temporary area
        /// </summary>
        /// <param name="itemId">The ID of the item that this file will be associated with</param>
        /// <param name="filePath">The path to the original file</param>
        /// <returns>The path to the copied file</returns>
		public async Task<string> CopyItemFileAsync(string itemId, string filePath)
		{
			string fileName = Path.GetFileName(filePath);
			string targetPath = await GetLocalFilePathAsync(itemId, fileName);

            var sourceFolder = await StorageFolder.GetFolderFromPathAsync(Path.GetDirectoryName(filePath));
            var sourceFile = await sourceFolder.GetFileAsync(Path.GetFileName(filePath));
            var sourceStream = await sourceFile.OpenStreamForReadAsync();

            var targetFolder = await StorageFolder.GetFolderFromPathAsync(Path.GetDirectoryName(targetPath));
            var targetFile = await targetFolder.CreateFileAsync(Path.GetFileName(filePath), CreationCollisionOption.ReplaceExisting);
			using (var targetStream = await targetFile.OpenStreamForWriteAsync())
			{
				await sourceStream.CopyToAsync(targetStream);
			}

			return targetPath;
		}

        /// <summary>
        /// Delete an existing mobile apps associated file
        /// </summary>
        /// <param name="file">The file to delete</param>
        /// <returns>Task (Async)</returns>
		public async Task DeleteLocalFileAsync(MobileServiceFile file)
		{
			string localPath = await GetLocalFilePathAsync(file.ParentId, file.Name);
            var storageFolder = await StorageFolder.GetFolderFromPathAsync(Path.GetDirectoryName(localPath));
            try
            {
                var storageFile = await storageFolder.GetFileAsync(Path.GetFileName(localPath));
                await storageFile.DeleteAsync();
            }
            catch (FileNotFoundException) { }
            // UnauthorizedAccessException is still thrown, but should never happen
		}

        /// <summary>
        /// Download a file from blob storage and store it in local storage
        /// </summary>
        /// <typeparam name="T">The type of the table controller</typeparam>
        /// <param name="table">The sync table reference</param>
        /// <param name="file">The file to download</param>
        /// <param name="filename">The local storage location of the file</param>
        /// <returns></returns>
        public async Task DownloadFileAsync<T>(IMobileServiceSyncTable<T> table, MobileServiceFile file, string filename)
        {
            var path = await GetLocalFilePathAsync(file.ParentId, file.Name);
            await table.DownloadFileAsync(file, path);
        }

        public async Task<IMobileServiceFileDataSource> GetFileDataSource(MobileServiceFileMetadata metadata)
        {
            var path = await GetLocalFilePathAsync(metadata.ParentDataItemId, metadata.FileName);
            return new PathMobileServiceFileDataSource(path);
        }

        /// <summary>
        /// Ask the user for an image location
        /// </summary>
        /// <returns>The path to the image (or null if cancelled)</returns>
        public async Task<string> GetImageAsync()
        {
            try
            {
                await CrossMedia.Current.Initialize();
                var file = await CrossMedia.Current.PickPhotoAsync();
                return file.Path;
            }
            catch (TaskCanceledException) { }
            return null;
        }

        /// <summary>
        /// Get the local storage path for a specific file attached to a specific item, creating the folder if necessary
        /// </summary>
        /// <param name="itemId">The ID of the item the file is attached to</param>
        /// <param name="fileName">The name of the file</param>
        /// <returns></returns>
	public async Task<string> GetLocalFilePathAsync(string itemId, string fileName)
	{
            var localStateFolder = ApplicationData.Current.LocalFolder;
            var tableStorageName = "TodoItemFiles";

            // Get a Storage Folder for the tableStorageName, creating it if necessary
            var tableStorageFolder = await CreateFolderIfNotExistsAsync(localStateFolder, tableStorageName);

            // Get a StorageFolder for the item, creating it if necessary
            var itemStorageFolder = await CreateFolderIfNotExistsAsync(tableStorageFolder, itemId);

            // Return the fully qualified path name of the file
            return Path.Combine(itemStorageFolder.Path, fileName);
	}

        private async Task<StorageFolder> CreateFolderIfNotExistsAsync(StorageFolder folder, string name)
            => await folder.CreateFolderAsync(name, CreationCollisionOption.OpenIfExists);
    }
}

There are subtle differences between the three implementations, primarily in the GetImageAsync() and GetItemFilesPathAsync() implementations. Note that, as with the LoginProvider implementation, Android has an additional piece in the MainActivity.cs file to initialize the UI Context properly. In addition, there is a line in the iOS projects AppDelegate.cs file to initialize SQLitePCL – this is platform dependent code, after all.

This code is concerned with how to deal with storage and picking images. There are platform-specific routines for

  • Getting an image (either from a camera or a pictures library)
  • Copying the image to a specific storage location
  • Uploading and downloading files from Blob Storage

It’s a good idea to become exceedingly good at debugging platform specific code as this is likely where your bugs are going to lie. I’ve also noticed that the file sync service can get into a “bad state” where two tables within the SQLite offline sync cache are not “in sync”. In this case, the best idea is to wipe out the SQLite database and the common storage area for files. In UWP, for example, this is located in the LocalState directory for the application – doing a search for the xamarintodo.db file will find that location as it’s in a hidden area (the AppData directory in your home directory).

Step 3: Implement a File Sync Service

Shockingly, the Azure Mobile Apps SDK does not handle file transfers – that is done via the Azure Storage SDK and platform specific SDKs. The Azure Mobile Apps File Sync SDK does provide the logic for doing the offline synchronization of the metadata and handling the coordination of the upload and downloads. The first step to this is to implement a file sync service for a particular table. This is done in the shared project by implementing the IFileSyncHandler interface (which is a part of the Azure Mobile Apps File Sync SDK). Here is my implementation of the file sync handler for the TodoItem table:

using Microsoft.WindowsAzure.MobileServices.Files;
using Microsoft.WindowsAzure.MobileServices.Files.Metadata;
using Microsoft.WindowsAzure.MobileServices.Files.Sync;
using System.Threading.Tasks;
using Xamarin.Forms;
using XamarinTodo.Helpers;

namespace XamarinTodo.Services
{
    class TodoItemFileSyncHandler : IFileSyncHandler
    {
        private IFileProvider fileProvider;
        private ICloudService cloudService;

        public TodoItemFileSyncHandler()
        {
            fileProvider = DependencyService.Get<IFileProvider>();
            cloudService = ServiceLocator.Instance.Resolve<ICloudService>();
        }

        public Task<IMobileServiceFileDataSource> GetDataSource(MobileServiceFileMetadata metadata)
            => fileProvider.GetFileDataSource(metadata);

        public async Task ProcessFileSynchronizationAction(MobileServiceFile file, FileSynchronizationAction action)
        {
            if (action == FileSynchronizationAction.Delete)
                await FileHelper.DeleteLocalFileAsync(file);
            else
                await cloudService.DownloadItemFileAsync(file);
        }
    }
}

Note that I’ve added a method to the ICloudService (implicitly) here. This is defined in ICloudService.cs:

using Microsoft.WindowsAzure.MobileServices.Files;
using System.Collections.Generic;
using System.Threading.Tasks;
using XamarinTodo.Models;

namespace XamarinTodo.Services
{
    public interface ICloudService
    {
        Task InitializeAsync();

        Task<IEnumerable<TodoItem>> GetAllItemsAsync();

        Task<TodoItem> UpsertItemAsync(TodoItem item);

        Task<bool> DeleteItemAsync(TodoItem item);

        Task SynchronizeServiceAsync();

        Task LoginAsync();

        Task LogoutAsync();

        Task<StorageTokenViewModel> GetStorageToken();

        Task DownloadItemFileAsync(MobileServiceFile file);
    }
}

The additional lines are highlighted. Finally, I need to update my AzureCloudService to do the synchronization – this means implementing the new DownloadItemFileAsync method:

        public async Task DownloadItemFileAsync(MobileServiceFile file)
        {
            var item = await itemTable.LookupAsync(file.ParentId);
            var path = await FileHelper.GetLocalFilePathAsync(file.ParentId, file.Name);
            var fileProvider = DependencyService.Get<IFileProvider>();
            await fileProvider.DownloadFileAsync(itemTable, file, path);
        }

Note that the DownloadItemFileAsync() actually calls the file provider (which is platform specific code) to do the actual download. That’s because the file is downloaded to local storage (which is, you guessed it, platform specific).

I also need to do some adjustments to the InitializeAsync() and SynchronizeServiceAsync() methods. First, the InitializeAsync() method – I need to initialize the file sync service:

        public async Task InitializeAsync()
        {
            if (isInitialized)
                return;

            var store = new MobileServiceSQLiteStore("xamarintodo.db");
            store.DefineTable<TodoItem>();

            // Initialize File Sync
            MobileService.InitializeFileSyncContext(new TodoItemFileSyncHandler(), store);

            // Initialize the sync context
            await MobileService.SyncContext.InitializeAsync(store,
                new MobileServiceSyncHandler(),
                StoreTrackingOptions.NotifyLocalAndServerOperations);

            // Get a reference to the sync table
            itemTable = MobileService.GetSyncTable<TodoItem>();

            isInitialized = true;
        }

The changes to the SyncContext.InitializeAsync() call register call backs for specific operations to trigger the file sync handler. FInally, I actually want to sync files as well, so in my SynchronizeServiceAsync() method, there is another addition:

        public async Task SynchronizeServiceAsync()
        {
            await InitializeAsync();
            await MobileService.SyncContext.PushAsync();
            await itemTable.PushFileChangesAsync();
            await itemTable.PullAsync("allitems", itemTable.CreateQuery());
        }

Update your Model: There is a bug in the current version of the File Sync SDK whereby the Id property can not be in a base class. I did have the Azure Mobile Apps properties in an EntityData class, but copied the fields from EntityData to the actual class. This will be fixed in the GA release.

Step 4: Update the UI

I’m going to add some UI to my Pages.ItemDetail page. This is a XAML page with a ViewModel. First off, I need a view-model to hold the information on the images. This needs to implement the INotifyPropertyChanged interface as I am going to make it a part of an ObservableCollection so I can add it to a ListView later on:

using Microsoft.WindowsAzure.MobileServices.Files;
using System.ComponentModel;
using XamarinTodo.Services;
using Xamarin.Forms;
using System.Threading.Tasks;
using System.Diagnostics;

namespace XamarinTodo.Models
{
    public class TodoItemImage : INotifyPropertyChanged
    {
		private IFileProvider fileProvider;
        private string name, uri;
        private TodoItem todoitem;

        public TodoItemImage(MobileServiceFile file, TodoItem item)
        {
            Name = file.Name;
            File = file;
            todoitem = item;
            fileProvider = DependencyService.Get<IFileProvider>();
#pragma warning disable CS4014 // Because this call is not awaited, execution of the current method continues before the call is completed
            InitializeUriAsync();
#pragma warning restore CS4014 // Because this call is not awaited, execution of the current method continues before the call is completed
        }
        private async Task InitializeUriAsync()
        {
            Uri = await fileProvider.GetLocalFilePathAsync(todoitem.Id, Name);
        }
        public MobileServiceFile File { get; }

        public string Name
        {
            get { return name; }
            set { name = value;  OnPropertyChanged(nameof(Name)); }
        }

        public string Uri
        {
            get { return uri; }
            set { uri = value;  OnPropertyChanged(nameof(Uri)); }
        }

        public event PropertyChangedEventHandler PropertyChanged;

        private void OnPropertyChanged(string name)
        {
            PropertyChanged?.Invoke(this, new PropertyChangedEventArgs(name));
        }
    }
}

I’ve got some UI XAML in my ItemDetail.xaml file:

<?xml version="1.0" encoding="utf-8" ?>
<ContentPage x:Class="XamarinTodo.Pages.ItemDetail"
             xmlns="http://xamarin.com/schemas/2014/forms"
             xmlns:x="http://schemas.microsoft.com/winfx/2009/xaml">
    <ContentPage.Content>
        <StackLayout Padding="10" Spacing="10">
            <Label Text="What should I be doing?" />
            <Entry Text="{Binding Item.Text}" />
            <Label Text="Completed?" />
            <Switch IsToggled="{Binding Item.Complete}" />
            <ListView x:Name="imagesList"
                      IsPullToRefreshEnabled="false"
                      ItemsSource="{Binding Images}">
                <ListView.ItemTemplate>
                    <DataTemplate>
                        <ImageCell ImageSource="{Binding Uri}" Text="{Binding Name}" />
                    </DataTemplate>
                </ListView.ItemTemplate>
            </ListView>
            <StackLayout VerticalOptions="CenterAndExpand" />
            <StackLayout Orientation="Vertical" VerticalOptions="End">
                <StackLayout HorizontalOptions="FillAndExpand" Orientation="Horizontal">
                    <Button BackgroundColor="#A6E55E"
                            Command="{Binding SaveCommand}"
                            Text="Save"
                            TextColor="White" />
                    <Button BackgroundColor="#A6E55E"
                            Command="{Binding AddImageCommand}"
                            Text="Add Image"
                            TextColor="White" />
                    <Button BackgroundColor="Red"
                            Command="{Binding DeleteCommand}"
                            Text="Delete"
                            TextColor="White" />
                </StackLayout>
            </StackLayout>
        </StackLayout>
    </ContentPage.Content>
</ContentPage>

Lines 11-19 add a ListView for the images and lines 27-30 add a button to add an image to the item. In the ItemDetailViewModel.cs, I need to do some work, namely:

  1. Set up the Images and Load those images
  2. Create the AddImageCommand so I can add an image

The former is pretty straight forward:

        public ItemDetailViewModel(TodoItem item = null)
        {
            if (item != null)
            {
                Item = item;
                Title = item.Text;
            }
            else
            {
                Item = new TodoItem { Text = "New Item", Complete = false };
                Title = "New Item";
            }
            cloudService = ServiceLocator.Instance.Resolve<ICloudService>();

            // Load the Images async
            Images = new ObservableCollection<TodoItemImage>();
            LoadImagesAsync();
        }

        public TodoItem Item { get; set; }
        public ObservableCollection<TodoItemImage> Images { get; set; }

        public async Task LoadImagesAsync()
        {
            IEnumerable<MobileServiceFile> files = await cloudService.GetItemImageFilesAsync(Item);
            Images.Clear();
            foreach (var file in files)
            {
                Images.Add(new TodoItemImage(file, Item));
            }
        }

I need to implement the GetItemImageFilesAsync() method in the cloud service – more on that later. The command needs to pick or take a photo and upload it. I’ve already created helpers for picking or taking the photo (when I created the platform dependent code). The code for the AddImageCommand looks like this:

        Command c_addimage;
        public Command AddImageCommand
        {
            get { return c_addimage ?? (c_addimage = new Command(async () => ExecuteAddImageCommand())); }
        }

        async Task ExecuteAddImageCommand()
        {
            if (IsBusy) return;
            IsBusy = true;

            try
            {
                var image = await DependencyService.Get<IFileProvider>().GetImageAsync();
                if (image != null)
                {
                    MobileServiceFile file = await cloudService.AddItemImageAsync(Item, image);
                    Images.Add(new TodoItemImage(file, Item));
                }
            }
            catch (Exception ex)
            {
                UserDialogs.Instance.ShowError(ex.Message);
            }
            finally
            {
                IsBusy = false;
            }
        }

The important code is highlighted – the non-highlighted code is basic boilerplate to implement a standard Command in my app. I’m going to need to add a second method to my cloud service class – AddImageAsync() – to assist with the actual addition of the image to the table sync context.

That’s two additional methods. Add their signatures to the ICloudService:

        Task<MobileServiceFile> AddItemImageAsync(TodoItem item, string image);

        Task<IEnumerable<MobileServiceFile>> GetItemImageFilesAsync(TodoItem item);

The AddItemImageAsync() method will attach the provided image to the provided item. The GetItemImageFilesAsync() method returns a list of all the files attached to the specified item. Of course, I need a concrete implementation of these methods in AzureCloudService.cs:

        public async Task<MobileServiceFile> AddItemImageAsync(TodoItem item, string image)
        {
            var path = await fileProvider.CopyItemFileAsync(item.Id, image);
            var fileName = Path.GetFileName(path);
            return await itemTable.AddFileAsync(item, fileName);
        }

        public async Task<IEnumerable<MobileServiceFile>> GetItemImageFilesAsync(TodoItem item)
            => await itemTable.GetFilesAsync(item);

The Problems

All my problems in this post could be grouped into one problem: Platform Differences. For example, the Universal Windows project that I was developing first has a weird permissions structure that took some time to understand. I didn’t actually get to implementing the FileProvider for Android and iOS (beyond the basic version that was a lot of cut-and-paste code from the tutorial) and those implementations don’t work yet. However, I hope to implement those really soon – check back on the master branch in a few weeks if you are interested.

File Sync is in preview and will likely get a lot of love from the developers prior to GA. Right now, it’s a great method to work through a basic problem – how do you upload and attach files to records when you are working offline.

As always, you can get my code on my GitHub repository.

30 Days of Zumo.v2 (Azure Mobile Apps): Day 27: File Handling (v1)

One of the oft requested features is file upload. It comes in two forms. Firstly, the simple version: “I want to upload a file”. Secondly, the database version: “I want to upload a file and link it to a record within an offline table”. Both of these have valid use cases, so I should cover both of them. For the linkage to the offline table edition, it would be nice for the file upload to be offline capable as well.

On to actual file uploads to Azure. The key here is: Don’t use Azure Mobile Apps to upload files. I know what you are thinking…. if I’m not using Azure Mobile Apps for file upload, how do I do it? Well, the information lies in how Azure Storage works and what you need to do to upload a file. I’ll still be using Azure Mobile Apps as you will see.

Azure Storage is an awesome flexible (and cheap) cloud storage solution. Like other cloud storage solutions, it offers multiple modes – Blob, File, Table and Queue Storage. I’m only going to be looking at Blob Storage today. You can read about the various types in the introduction from Azure. The key fact about Azure Storage is that it already has a lot of the features you need – things like restartable uploads, for example – in the REST interface. It also has a bunch of SDKs available for just about all the major platforms – including UWP, iOS, Android and Xamarin.

So, what’s the problem? Can’t I just go ahead and use the SDK? Well, sort of. There is a process here. For each request, you need to:

  1. Ensure that a storage account exists (just like you would your SQL Azure instance)
  2. Find or Create a Storage Container for each user (or some other segmentation that makes sense)
  3. Create a Storage Access Key that allows an authenticated user to write to the storage container
  4. Then use the Azure Storage SDK to upload the file

Let’s take a look at each one:

Create a Storage Account

You only need to create a storage account once per backend. Once it’s created, it’s linked for the duration. Creating a storage account is pretty much identical to any other service within Azure:

  1. Go to your Resource Group. You will want to store the Storage Account with your other items so that when you delete the Resource Group, the storage gets deleted as well.
  2. Click on the + Add button at the top of the Resource Group blade to add a resource:

day-27-pic-1

  1. Enter “Storage” in the search box and hit Enter. The right Storage Account template should be first in the list:

day-27-pic-2
4. Click on the right “Storage Accounnt” template, then click on the Create button.
5. Fill in the blade details. Enter a name and place the storage account in the same region as your mobile backend. Also select the right replication scheme and storage account type. Once done, click on Create.

day-27-pic-4

Let’s take a moment to talk about replication schema and storage account type. There are two types of storage account – General Purpose allows access to all four types of storage – blob, file, table and queue. Blob Storage only allows access to, well, blob storage. There is no downside to selecting General Purpose. You also have four different replication schemes. Azure Storage maintains three copies of your data. You get to choose where the copies are stored:

  • Locally Redundant Storage – the copies are stored in the same data center
  • Zone Redundant Storage – the copies are stored in different data centers in the same region
  • Geo-Redundant Storage – six copies (instead of three) distributed across multiple regions
  • Read-Access Geo-Redundant – a copy is maintained in a different region but is read-only

If you are just testing, go with LRS – it’s the cheapest. If you are going into production, make sure your replication strategy matches your needs. Also note that there may be one-time charges for converting from LRS to one of the others. There are also special notes on Zone Redundant Storage, so read the docs on this carefully.

You’ve also got an option between Standard and Premium performance. You can get up to 50Gbps throughput with premium storage, at a premium cost. Premium storage is really designed for VM disks, so you are unlikely to need it for this use case.

Now that I have a storage account, I can move on to the next step.

Link Storage Account to Mobile Backend

The next step is to link the storage account to the mobile backend App Service. To do this:

  1. Select your App Service
  2. Click on All Settings, then Data Connections
  3. Click on + Add
  4. In the drop-down (that normally shows SQL Database), select Storage
  5. Click on Connection String – it should be set correctly, (but correct if you need to), then click on OK
  6. Click on OK to add the connection string.

The storage account connection string will appear as the environment variable CUSTOMCONNSTR_MS_AzureStorageAccountConnectionString – which is a mouthful. You can check it in the environment section of Kudu.

Create a Shared Access Signature

The shared access signature is a URI that encompasses everything needed to access storage in a delegated manner. You can decide to limti the rights and give a time limit on the shared access signature. One must be created for each file to be uploaded, so this makes for a great custom API example. In this code, I’m going to create a storage container for the user (if one does not already exist), then I’m going to securely create a write key for the storage container. How securely? Well, I’m going to ensure that the user is authenticated. I’m also going to limit the amount of time that the storage access key is valid. You can tune the amount of time provided, but make it small – 30 minutes is plenty to upload a file. Here is my Custom API:

using System.Web.Http;
using Microsoft.Azure.Mobile.Server.Config;
using Microsoft.WindowsAzure.Storage;
using Microsoft.WindowsAzure.Storage.Blob;
using System;
using System.Threading.Tasks;
using System.Security.Claims;
using System.Diagnostics;

namespace Backend.Controllers
{
    [Authorize]
    [MobileAppController]
    public class GetStorageTokenController : ApiController
    {
        public GetStorageTokenController()
        {
            ConnectionString = Environment.GetEnvironmentVariable("CUSTOMCONNSTR_MS_AzureStorageAccountConnectionString", EnvironmentVariableTarget.Process);
            Debug.WriteLine($"[GetStorageTokenController$init] Connection String = {ConnectionString}");
            StorageAccount = CloudStorageAccount.Parse(ConnectionString);
            BlobClient = StorageAccount.CreateCloudBlobClient();
        }

        public string ConnectionString { get; set; }

        public CloudStorageAccount StorageAccount { get; set; }

        public CloudBlobClient BlobClient { get; set; }

        // GET api/GetStorageToken
        public async Task<StorageTokenViewModel> Get()
        {
            // Get the container name for the user
            Debug.WriteLine($"[GetStorageTokenController] Get()");
            var claimsPrincipal = User as ClaimsPrincipal;
            var sid = claimsPrincipal.FindFirst(ClaimTypes.NameIdentifier).Value.Substring(4); // strip off the sid: from the front
            string containerName = $"container-{sid}";
            Debug.WriteLine($"[GetStorageTokenController] Container Name = {containerName}");


            // Create the container if it does not yet exist
            CloudBlobContainer container = BlobClient.GetContainerReference(containerName);
            Debug.WriteLine($"[GetStorageTokenController] Got Container Reference");
            // This will throw a StorageException, which results in a 500 Internal Server Error on the outside
            try
            {
                await container.CreateIfNotExistsAsync();
                Debug.WriteLine($"[GetStorageTokenController] Container is created");
            }
            catch (StorageException ex)
            {
                Debug.WriteLine($"[GetStorageTokenController] Cannot create container: {ex.Message}");
            }

            // Create a blob URI - based on a GUID
            var blobName = Guid.NewGuid().ToString("N");
            Debug.WriteLine($"[GetStorageTokenController] Blob Name = {blobName}");
            var blob = container.GetBlockBlobReference(blobName);
            Debug.WriteLine($"[GetStorageTokenController] Got Blob Reference");

            // Create a policy for the blob access
            var blobPolicy = new SharedAccessBlobPolicy
            {
                // Set start time to five minutes before now to avoid clock skew.
                SharedAccessStartTime = DateTime.UtcNow.AddMinutes(-5),
                // Allow Access for the next 60 minutes (according to Azure)
                SharedAccessExpiryTime = DateTime.UtcNow.AddMinutes(60),
                // Allow read, write and create permissions
                Permissions = SharedAccessBlobPermissions.Read | SharedAccessBlobPermissions.Write | SharedAccessBlobPermissions.Create
            };
            Debug.WriteLine($"[GetStorageTokenController] Got Blob SAS Policy");

            return new StorageTokenViewModel
            {
                Name = blobName,
                Uri = blob.Uri,
                SasToken = blob.GetSharedAccessSignature(blobPolicy)
            };

        }
    }

    public class StorageTokenViewModel
    {
        public string Name { get; set; }
        public Uri Uri { get; set; }
        public string SasToken { get; set; }
    }
}

make sure you add the NuGet WindowsAzure.Storage package to your project – this contains the Azure Storage SDK. You can test this with Postman like this:

day-27-pic-5

Thinking of using this information to store that file? Remember that the SAS token is secure – it’s only valid for a certain time (just 60 minutes), so it won’t be valid by the time you read this.

Uploading a File

Now that I’ve got a custom API for generating a SAS token, all I need to do is use the client.invokeApi method to generate a SAS token for my upload, then use the Azure Storage SDK to actually upload the file. To test this, I’ve got a new Xamarin Forms application – it does pretty much the same as the UWP and Cordova applications that I’ve been using. Xamarin Forms is used for cross-platform development, so the resulting applications are native applications and I can compile for iOS, Android and Universal Windows simultaneously. I’ve added a button to the ItemList.xaml UI for handling a file upload:

          <StackLayout Orientation="Horizontal" HorizontalOptions="FillAndExpand" Padding="10">
            <Button BackgroundColor="Teal" Command="{Binding AddNewItemCommand}" Text="Add New Item" TextColor="White" />
            <Button BackgroundColor="Purple" Command="{Binding UploadFileCommand}" Text="Upload File" TextColor="White" />
          </StackLayout>

The handler for this is located in the ItemListViewModel.cs file:

        Command c_uploadFile;
        public Command UploadFileCommand
        {
            get { return c_uploadFile ?? (c_uploadFile = new Command(async () => await ExecuteUploadFileCommand())); }
        }

        async Task ExecuteUploadFileCommand()
        {
            if (IsBusy) return;
            IsBusy = true;

            try
            {
                await UserDialogs.Instance.ActionSheetAsync("Upload File", "Cancel", "Upload File");
            }
            catch (Exception ex)
            {
                UserDialogs.Instance.ShowError(ex.Message);
            }
            finally
            {
                IsBusy = false;
            }
        }

This code doesn’t actually do anything other than pop up a dialog box with the appropriate buttons for me. I’m not even that interested in the dialog box. This is just serving as a temporary holding area so I can move through the process of uploading a file. This is my base code before I’ve added any of the upload file logic. My first step is to get the SAS token. I’ve copied the StorageTokenViewModel from my Backend project to my shared Xamarin Forms project (it’s in the Models directory). Getting the storage token is now easy. I’ve expanded the ICloudService to include the following definition:

        Task<StorageTokenViewModel> GetStorageToken();

I’ve also added appropriate implementations in the cloud service implementations. The important one is in AzureCloudService.cs:

        public Task<StorageTokenViewModel> GetStorageToken()
        {
            return MobileService.InvokeApiAsync<StorageTokenViewModel>("GetStorageToken", HttpMethod.Get, null);
        }

I can now do the first part in my try block of my ItemListViewModel.cs:

            try
            {
                var storageToken = await cloudService.GetStorageToken();
                await UserDialogs.Instance.AlertAsync($"URI = {storageToken.Uri}{storageToken.SasToken}", "Got SAS Token", "OK");
            }

When you run this code, wait for the item list to refresh and then click on the Upload File button – you will get something akin to this:

day-27-pic-6

I’m going to need some assistance with a mobile file picker. I’m going to use the Xamarin Media Plugin from James Montemagno. James has produced a number of high quality plugins for Xamarin, so check out his collection. The easiest way to install this plugin is to get it from NuGet. You can add it directly within Visual Studio. Don’t forget to add the NuGet package to all the Xamarin projects. Once this is done, adding a photo picker is simple:

                // Xamarin Media Plugin to get a picture
                await CrossMedia.Current.Initialize();
                var file = await CrossMedia.Current.PickPhotoAsync();

I’m going to assume that requesting the storage token will always work if I am online, so I added the photo picker before the call to GetStorageToken(). My final piece of code uploads the file I’ve picked to the SAS location I’ve received from the custom API.

You will want to add the WindowsAzure.Storage SDK v7.0.1-preview edition. This is a specific version that is compatible with Xamarin Forms. You can find it in the NuGet Package Manager in Visual Studio by checking the “Include Prerelease” checkbox. As always, add the NuGet package to all the Xamarin Forms projects.

Here is the completed code with the additional upload code highlighted:

            Uri storageUri = null;
            try
            {
                // Xamarin Media Plugin to get a picture
                await CrossMedia.Current.Initialize();
                var file = await CrossMedia.Current.PickPhotoAsync();

                // Get the storage token from the custom API
                var storageToken = await cloudService.GetStorageToken();
                storageUri = new Uri($"{storageToken.Uri}{storageToken.SasToken}");

                // Store the MediaFile to the storage token
                var blob = new CloudBlockBlob(storageUri);
                var stream = file.GetStream();
                await blob.UploadFromStreamAsync(stream);
                UserDialogs.Instance.SuccessToast("File Upload Complete", null, 1500);
            }
            catch (Exception ex)
            {
                UserDialogs.Instance.Alert(storageUri.ToString(), ex.Message, "OK");
            }

It’s important to note that you don’t upload files – you upload a stream of data. Fortunately, the MediaFile type has taken that into account and provides a handy method for converting the file into a stream. The file upload is done asynchronously and I’ve added a toast for successful upload. If the upload fails, the SAS URI is displayed so you can debug further.

Looking at your uploaded file

You can look at your files easily enough by using the Resource browser in the portal:

day-27-pic-7

Ensure you are selecting the storage account that is linked to your mobile backend. The container name (in my case) is the SID of the user, so it’s not obvious who this belongs to. The alternative would be to use GetIdentitiesAsync to get a more reasonable username (potentially munging the email address). Note that you can set up the storage account to provide read-only access to the files generally. In this case, the Uri property on the StorageTokenViewModel object holds the URI that you can use to access the photo later.

Caveats

The main caveat is that I can only upload a file while I am online. I don’t have any checks for online vs. offline. That’s easily added by leveraging another Xamarin plugin from James Montemagno, but it doesn’t solve the underlying problem that this solution is online only. I also have not done any of the improve suggested by the Azure Storage SDK – like progress reporting, restart upload and so on. That would take a whole UX design to include that, so I have not bothered.

I could also deal with a bunch of deficiencies in the blob URI generation within the custom URI. Right now, all URIs are generated as GUIDs, which isn’t the most friendly. To allow easier browsing, I should at least be looking at appending the suffix extension of the original file (especially since I have it at that point) and making the container specification easier. This is definitely left as an exercise for the reader.

Next time, I’ll look at the offline aspect of file upload and attaching pictures to specific items. Until then, my code is in the file-upload solution on my GitHub repository.

30 Days of Zumo.v2 (Azure Mobile Apps): Day 26: Relationship Advice

No, I’m not going to fix the relationship with your significant other. One of the common questions I get asked is about relationships between tables. The basic advice is easy – don’t do it. The actual relationship advice is a little more complex than that and today I’m going to cover what is involved in setting up table relationships, what to do and what to avoid.

The Node.js / Easy Tables Version

This one is easy. The Node.js Server SDK for Azure Mobile Apps does not support relationships in any way, shape or form. If you want to use relationships, use an ASP.NET backend.

The ASP.NET Version

There are two relationships that are supported

  1. A one-way 1:1 relationship
  2. A one-way 1:Many relationship

One way relationships are where a table has a foreign key for another table, but the reverse is not true. Do not do the normal “two-way” relationships as you will run into circular serialization problems.

These are both configured with Entity Framework using foreign keys. You need to ensure at you set up your data transfer objects first. Let’s take an example. Let’s say I have my normal TodoItem table, but I want to add a tag to each record. The data transfer objects are configured thus:

using Microsoft.Azure.Mobile.Server;
using System.ComponentModel.DataAnnotations.Schema;

namespace Mobile.Backend.DataObjects
{
    public class Tag : EntityData
    {
        public string TagName { get; set; }
    }

    public class TodoItem : EntityData
    {
        public string Text { get; set; }

        public bool Complete { get; set; }

        #region Relationships
        public string TagId { get; set; }

        [ForeignKey("TagId")]
        public Tag Tag { get; set; }
        #endregion
    }
}

The ForeignKey annotation is the key to this – it sets up the linkage between the two tables. Note that the TagId references the Id of the Tag table which is, of course, a GUID in string form when you are in an Azure Mobile Apps DTO. I still need to add a table controller for the Tag entity, and add a DbSet to the MobileServiceContext – so now I have two controllers.

The Client Side

On the client side, I need to be able to add a Tag to the TodoItem. The Tag must exist before the TodoItem exists. Let’s take a look at one of the TodoItem records when I request it through Postman – here is the response:

[
  {
    "deleted": false,
    "updatedAt": "2016-05-25T03:01:53.33Z",
    "createdAt": "2016-05-25T03:01:53.33Z",
    "version": "AAAAAAAAB9Q=",
    "id": "183df2a2-abed-4f60-bde5-ebb157343462",
    "tagId": "e6782065-6ec4-4042-a0b7-3dcfc1c334a0",
    "complete": false,
    "text": "First item"
  },
  {
    "deleted": false,
    "updatedAt": "2016-05-25T03:01:53.345Z",
    "createdAt": "2016-05-25T03:01:53.345Z",
    "version": "AAAAAAAAB9Y=",
    "id": "e1291e16-c8e1-440f-ae6e-4ec9835302c4",
    "tagId": null,
    "complete": false,
    "text": "Second item"
  }
]

My first record has a linkage to a specific tag – this is shown via the tagId being filled in. My second record does not have a linkage to a specific tag. On the client, you will need to do the join of the tables yourself. The client does not know about relationships. Each table is independent. This means that your model (assuming a C#/.NET client, like UWP or Xamarin) should look like this:

using ClientApp.Helpers;

namespace ClientApp.Models
{
    public class Tag : EntityData
    {
        public string TagName { get; set; }
    }

    public class TodoItem : EntityData
    {
        public string Text { get; set; }
        public bool Complete { get; set; }

        public string TagId { get; set; }
    }
}

In general, things will just work. However, that’s because you will naturally create a Tag before using it. You must ensure that you insert the tag before inserting the Todoitem that references it. Again, you do the join in code – don’t expect a Tag item to magically be populated in your TodoItem as a linked reference – it won’t happen.

1:Many Relationships

You can also do one-way 1:Many relationships with the ASP.NET Server SDK. An example of a one-way 1:Many relationship is that my TodoItem may have multiple tags. As long as I am not doing the reverse (give me all todo items that use this tag, which would indicate a two-way relationship), you should be ok. In this case, the Tags would be an array. Let’s take a look at the new models on the server side:

using Microsoft.Azure.Mobile.Server;
using System.Collections.Generic;

namespace Mobile.Backend.DataObjects
{
    public class Tag : EntityData
    {
        public string TagName { get; set; }
    }

    public class TodoItem : EntityData
    {
        public string Text { get; set; }
        public bool Complete { get; set; }
        public virtual ICollection<Tag> Tags { get; set; }
    }
}

Sending the same query as before will result in the following:

[
  {
    "deleted": false,
    "updatedAt": "2016-05-25T21:17:05.034Z",
    "createdAt": "2016-05-25T21:17:05.03Z",
    "version": "AAAAAAAAB9I=",
    "id": "1b635919-9880-4353-a4c2-f6c346a124ff",
    "complete": false,
    "text": "First item"
  },
  {
    "deleted": false,
    "updatedAt": "2016-05-25T21:17:05.106Z",
    "createdAt": "2016-05-25T21:17:05.106Z",
    "version": "AAAAAAAAB9g=",
    "id": "25bcc657-6299-4634-9ce0-aa523b89227b",
    "complete": false,
    "text": "Second item"
  }
]

Note that there is no tag. OData requires you to send the $expand parameter to expand the collection – something like https://localhost:14683/tables/TodoItem?$expand=Tags. The Azure Mobile Apps Client SDK (currently) doesn’t support $expand, so I have to do something on the server to support this. The first thing is to provide a new annotation called the [ExpandProperty]:

using System;
using System.Linq;
using System.Web.Http.Controllers;
using System.Web.Http.Filters;

namespace Mobile.Backend.Helpers
{
    [AttributeUsage(AttributeTargets.Method, AllowMultiple = true)]
    public class ExpandPropertyAttribute : ActionFilterAttribute
    {
        string propertyName;

        public ExpandPropertyAttribute(string propertyName)
        {
            this.propertyName = propertyName;
        }

        public override void OnActionExecuting(HttpActionContext actionContext)
        {
            base.OnActionExecuting(actionContext);
            var uriBuilder = new UriBuilder(actionContext.Request.RequestUri);
            var queryParams = uriBuilder.Query.TrimStart('?').Split(new[] { '&' }, StringSplitOptions.RemoveEmptyEntries).ToList();
            int expandIndex = -1;
            for (var i = 0; i < queryParams.Count; i++)
            {
                if (queryParams[i].StartsWith("$expand", StringComparison.Ordinal))
                {
                    expandIndex = i;
                    break;
                }
            }

            if (expandIndex < 0)
            {
                queryParams.Add("$expand=" + this.propertyName);
            }
            else
            {
                queryParams[expandIndex] = queryParams[expandIndex] + "," + propertyName;
            }

            uriBuilder.Query = string.Join("&", queryParams);
            actionContext.Request.RequestUri = uriBuilder.Uri;
        }
    }
}

When applied to a table controller method, this will automatically add the $expand property. Use it like this on the TodoItem controller:

        // GET tables/TodoItem
        [ExpandProperty("Tags")]
        public IQueryable<TodoItem> GetAllTodoItems() => Query();

        // GET tables/TodoItem/48D68C86-6EA6-4C25-AA33-223FC9A27959
        [ExpandProperty("Tags")]
        public SingleResult<TodoItem> GetTodoItem(string id) => Lookup(id);

You can specify this multiple times so that you can expand multiple properties. You generally only need to add this to the “Read” methods – GetAll and GetSingle. Now, when I do the same Postman query as before, I get the following:

[
  {
    "deleted": false,
    "updatedAt": "2016-05-25T21:17:05.034Z",
    "createdAt": "2016-05-25T21:17:05.03Z",
    "version": "AAAAAAAAB9I=",
    "id": "1b635919-9880-4353-a4c2-f6c346a124ff",
    "complete": false,
    "text": "First item",
    "tags": [
      {
        "tagName": "Urgent",
        "id": "3257f559-1c09-40a8-a633-df60f0970ac7",
        "version": "AAAAAAAAB9Q=",
        "createdAt": "2016-05-25T21:17:05.059Z",
        "updatedAt": "2016-05-25T21:17:05.064Z",
        "deleted": false
      },
      {
        "tagName": "Question",
        "id": "7c2f4467-973f-464e-88a8-584c7becd37c",
        "version": "AAAAAAAAB9Y=",
        "createdAt": "2016-05-25T21:17:05.091Z",
        "updatedAt": "2016-05-25T21:17:05.091Z",
        "deleted": false
      }
    ]
  },
  {
    "deleted": false,
    "updatedAt": "2016-05-25T21:17:05.106Z",
    "createdAt": "2016-05-25T21:17:05.106Z",
    "version": "AAAAAAAAB9g=",
    "id": "25bcc657-6299-4634-9ce0-aa523b89227b",
    "complete": false,
    "text": "Second item",
    "tags": []
  }
]

Note that the tag result is embedded in the result. For client platforms that support models, you can now do something similar to the following:

using ClientApp.Helpers;

namespace ClientApp.Models
{
    public class Tag : EntityData
    {
        public string TagName { get; set; }
    }

    public class TodoItem : EntityData
    {
        public string Text { get; set; }
        public bool Complete { get; set; }

        public List<Tag> Tags { get; set; }
    }
}

The tags will be included in the todoitem. The data is denormalized. Say you have a tag called “Urgent” and it is used six times – there will be six copies of the “Urgent” tag – not one with references. While this is not a problem for most applications, if your data is large you actually want to deal with this a different way. This may include using a TagId list instead and then linking the models when you do a refresh, for example.

Wrap Up

I hope you take away from this a few points:

  1. Relationships is strictly an ASP.NET function.
  2. Relationships are hard and you must work at them to ensure the proper outcome.
  3. 1:1 and 1:Many relationships are doable if they are one-way (i.e. table1 refers to table2, but the reverse is not true)
  4. You have to deal with expansion in the 1:Many case, and data is denormalized (which may not be desirable)
  5. Avoid Many:Many relationships
  6. You really need to understand how Entity Framework implements these relationships

The final thing to remember is that the client does not know about relationships. That is a SQL concept and SQL is spoken only on the server. Finally – my advice is to try and avoid relationships on your mobile backend – use projections mechanisms (such as SQL Views and/or automatic expansion with the ExpandProperty attribute) to denormalize the data prior to it getting to the client.

You can see the 1:Many code for the backend in the relationship-backend solution on my GitHub Repository.