The woes of coding by assumption

In a project we have a scheduler type service that polls an Azure Queue for work and does it, e.g. sending some emails. I've been confused about our number of storage transactions ever since I joined the project. It was at least 100 times more than my back of an envelope calculation estimated there should be.

I installed Application Insights to try it out and, after I published, it informed me that one of the apps (the scheduler) was sending 250+ requests per second to Azure Queues. What the hell right?

I took at look at the code and it was the sort of thing I was expecting:

void Start()
{
    _timer = new Timer(PollQueue, null, _queuePollingInterval, Timeout.Infinite);
}

void PollQueue(object state)
{
    // check the queue for new work and do it.
    if (_timer != null) _timer.Change(_queuePollingInterval, Timeout.Inifinte);
}

All looks good right? The entry point called Start() which in turn sets up a timer to called PollQueue(null) after the specified _queuePollingInterval which was defined as:

int _queuePollingInterval = TimeSpan.FromSeconds(1).Milliseconds;

Spot the mistake now? _queuePollingInterval == 0. It was looping as fast as possible which seems to have been about 250 times a second! We should have been using the TotalMilliseconds property.

int _queuePollingInterval = Convert.ToInt32(TimeSpan.FromSeconds(1).TotalMilliseconds)

This sort of thing should have been caught at the time of writing but someone assumed they knew what the Milliseconds property would return and didn't bother checking. That simple change knocked 15% cpu utilization off the service and cut our storage transactions by about 99.6%.

Comments

Post a Comment

Popular posts from this blog

Trimming strings in action parameters in ASP.Net Web API

In standard MVC it's very easy to create a model binder to trim all of the strings passed into action methods. The approach I used for that was from here:  http://blogs.taiga.nl/martijn/2011/09/29/custom-model-binders-and-request-validation/ . It works well and it trims the properties of models too. The problem came when I tried to do the same thing for Web API. I added a similar StringTrimmingBinder and registered it as a service using the very non-discoverable API: var provider = new SimpleModelBinderProvider( typeof ( string ),  new StringTrimmingModelBinder()); GlobalConfiguration.Configuration.Services.Insert( typeof (ModelBinderProvider),  0,  provider); As a side note, I have no idea how you're supposed to find out how to register a model binder without opening up Bing and Googling it. Anyway, this worked great when the parameters to the action were actually strings but doesn't work with complex types as parameters. Now, I've n
Read more

Full text search in Entity Framework 6 using command interception

Image
Background One of the main problems we encountered using Entity Framework was its lack of support for full text search. Our search function is based around a series of processors that attempt to match the incoming phrase against a set of patterns that we're expecting (10+). For example, there's one for dates and times, recent financial quarters as well as more domain specific ones. Everything is cached and so is really fast, except when it isn't: Here's the monitoring chart for our SQL DB in Azure That's a spike to 100% resource utilisation because we fall through to a phrase search as a last resort (which cannot be cached) - it's exceedingly rare but it still hurts. Before EF, the app used full text search and it was one of the casualties when the app was migrated to EF4 several years ago. The lack of full text search has been getting more and more painful as the amount of data has grown. We really needed full text search back... Command intercept
Read more

Composing Expressions in C#

This post follows on from  Reusing predicates in Entity Framework . In my previous post I introduced the "and" extension method which allowed me to create a new entity framework safe expression from two existing expressions. Here's how we make that happen: The code is from a blogpost from Colin Meek from 2008:  InvocationExpression and LINQ to Entities . I'll just dump my version of the code here. The idea is exactly the same (even using the visitor as an immutable stack): public class InvocationExpander : ExpressionVisitor { private readonly ParameterExpression _parameter; private readonly Expression _expansion; private readonly InvocationExpander _previous; public static T Expand<T>(T expr) where T : LambdaExpression { return (T) new InvocationExpander().Visit(expr); } public InvocationExpander() { } private InvocationExpander(ParameterExpression parameter, Expression expansion, Invocati
Read more