StringBuilder Limit

Just a quick one today. One of the joys of being a software developer is that it’s a constant learning experience, even after 20+ years in the field, I’m still learning something new on, at least, a weekly basis.

Today, whilst combing through some error logs to find the root of an OutOfMemoryException, I found the cause to be none other than that old hero: StringBuilder, which led me to wonder about it’s limitations: “String Builders don’t have limits.”, I confidently scoffed. Only to have a moment of doubt and a quick google for the documentation and here’s what I learned today …

And lo and behold, StringBuilders do indeed have limitations, which makes perfect sense when you really sit back to think of it. The limitation in earlier .Net (implementations is enforced by the MaxCapacity property of the class, which is an Int32, i.e. around 2.1bn bytes. On a side-note, Microsoft Documentation indicates that versions of .Net will allow you to overflow this using .Append().

But of course a chain is only as weak as its weakest link, there’s a beautiful answer on Stack Overflow from Eric Lippert, explaining that you only get 2Gb of user-addressable virtual address space per process on a 32 bit Windows machine.

So onto the next link in our chain; “Where are we running?”. Which in turn finally leads us to our breaking point, this particular software is running in a Kubernetes Pod and is actually constrained in it’s memory by very design, after all you don’t want a run-away pod to consume all of the available resources in your cluster right? In this case, our pod was limited to a very generous 1Gb.

A StringBuilder has limits but you’re almost certainly going to bump into other limits before you even come close, and frankly if you’ve got a StringBuilder running at capacity, you’re almost certainly doing it wrong in the first place.

In this case, I fixed the bug by validating my input to prevent a stupidly large date range. Problem Solved!

BPMN For Architecture Diagrams

Some really interesting thoughts here from one of my colleagues about Business Process Model and Notation (BPMN), this is one of those problems which I never realised I had, until I saw the answer.

Mountain Warehouse Tech

Intro

During my career, I’ve seen a lot of in-house diagrams that are used to explain the top-level architecture of large complex systems. Usually, by looking at them you can understand the main idea, but…:

They may be not super intuitive
2 different diagrams made by different persons can use the same symbols (arrows, gear signs etc) in different ways

I think the main problem is that people don’t have a convention for a notation and existing ones like UML and IDEF0, DFD and others are designed to illustrate very specific parts of architecture, but not top-level flow.

It’s not the top problem of development, but seeing this issue in different places (even in big companies) I would like to introduce a tool that might be helpful. It is BPMN. In this article, I won’t do a full tutorial for the notation (there will be useful links at the end)…

View original post 657 more words

20 years on: Why co-location is no longer a requirement for Agile

How many times have you heard someone tell you that co-location and face-to-face interaction are some of the most important features of Agile? I’m guessing the answer is: A Lot.

Agile has 12 central tenets, which can be read in less than 60 seconds, this is the Agile Manifesto, the only tenet one that refers to location is here:

The most efficient and effective method of conveying information to and within a development team is face-to-face conversation.
Agile Manifesto

I imagine that just about everyone agrees with this one. How many times have you had back and forth emails with someone which were going nowhere until one of you either picked up the phone, or just walked across the office to get any progress?

But you have to remember that the manifesto was written 20 years ago and was being worked on in it’s embryonic form for 10 years before that. So we’re talking from 1991 to 2001. Do you remember where we were technologically speaking in 2001? In the UK we were on the cusp of moving from 56k modems to something called ADSL, WindowsXP was a flagship OS, everyone was still using Nokia 3310’s and Skype still had another 2 years before it would be invented.

So looking back technologically, it’s no surprise that the only answer to face-to-face communication was co-location, because there was no alternative.

If we zoom forward 2 decades to the present day, and in part thanks to Covid which has driven a technological surge in the past 24 months, you can see we have a plethora of face-to-face solutions including Teams, Zoom, Skype, WhatsApp, Google Meet and many more. There are also a huge amount of collaborative tools, remote teams can use tools such as Miro for online whiteboarding and design, there are online polling tools, calendar tools … everything is online and collaborative in 2021 and there’s no excuse for not using these tools.

I think I might expand all of this out into another blog post on the pros and cons of remote working in the future, as I feel I could happily write an essay on this subject.

On my team the switch to remote working was revelatory, the levels and ease of communication using Teams meant that we were communicating more often, and doing so more effectively. Without the constant background distractions of an office environment we could focus on the tasks in front of us making the most efficient use of our time. We also built new and stronger relationships with our testing team and our business analyst team as they fell into the practice of using Teams.

The downsides of co-location are also considerable, but people generally don’t focus on this because they’ve always been accepted as the cost of doing business. Hours of commuting time lost, expensive office space and maintenance, staff expenses. The general lack of efficiency of co-location is quite staggering in terms of lost time and lost money and is this really the environment they need, or is it just accepted because it has always been this way?

Build projects around motivated individuals.
Give them the environment and support they need,
and trust them to get the job done.
Agile Manifesto

So next time someone tells you that Agile teams must be co-located, don’t just accept it as gospel. Just because something was the right answer 20 years ago, that doesn’t mean it still holds true today.

Code Hoarding

Life as a software developer is probably not what most people think it is, many imagine coders as we are portrayed in films or series; sun-shy nerds sitting in darkened rooms hammering away at keyboards in a code trance creating the software that will change the world! Yeah not really, in fact being able to create projects and software from scratch, is something of a rarity, for example recruiters often throw around the term “Greenfield Project” in job adverts to garner more interest in their roles.

I would say being a dev is more a hybrid of working at a nursing home and being an archaeologist.

A new product, once out of its embryonic state, will start out hale and healthy like an Olympic athlete but as the years roll on, there will be added requirements, edge cases and general bumps and bruises and the underlying technologies change as the world moves on. Before you know it the product has reached a state of senescence, with some surviving the ravages of the years better than others. For many of us, keeping these old geezers alive is a full time occupation.

That’s where the archaeology part comes in. The chances are, you weren’t around when your product was created and your job involves scouring the code base and understanding how things work, in order to fix and maintain them. It’s not quite Indiana Jones territory, more like dusting through layers of sediment to really grok the architecture and data structures. If you go in there with a hammer you’re likely to destroy things before you’ve even had a chance to understand what you’re doing.

One of my colleagues has recently been given one of these monumental tasks, a piece of software that lies at the critical intersection of just about every product we currently have, and to make matters worse, it’s on life support and has been for a number of years. So needless to say, he’s moving very cautiously. I was chatting with him about this recently and he commented:

“I knew the code base was bad, but I had no idea that <original dev> was a Code Hoarder as well!”.

I’d never heard this phrase before, but instantly recognised what he was talking about without need for explanation. A Code Hoarder is someone who cannot bring themselves to throw code away, instead they will either comment out huge blocks of code or worse, just bypass the obsolete code without ever removing it.

You can imagine the complications this causes another developer, when already facing the daunting task of getting their head around 100,000s lines of code, their life is further complicated by the fact that it’s littered with these unintentional booby traps.

I’m incredibly passionate about this and think that this mindset is one of the skills that shows the difference between experienced developers and more junior ones and that’s not just developing something to the point where it “works”, but considering the future maintenance of the code. If there are 50 lines of code REMed out in the middle of a function, I have to stop and consider why? What purpose is there to leave this behind? Why didn’t they just delete it? Whatever the answers are, they’re sure to get lost in the mists of time. I mean, we’re all using source control right?

So next time you’re about to comment out a huge swathe of your code, or render a function unused, ask yourself “Shouldn’t I just delete this instead?”.

Don’t be a Code Hoarder.

Validating Classes using Data Annotations

We’ve been beefing up our validation routines recently as it became clear that we needed to add a lot more protection to our Validation Engines in our NetCore project.

We use EFCore Code-First to generate our database in which we have a large number of Data Annotations such as:

    [MaxLength(50)]
    public string Surname { get; set; }

Our testers have started really getting into min/max testing so they’ve been happily trying to put oversized data into all of our fields, and guess what? We’re seeing Internal Server Errors all over the place.

So we’ve been faced with a challenge about where to implement our validation and how and even what level. Whilst investigating solutions, I realized that we can re-use the same Data Annotations on our EF Entities as part of our validation routines, which means that in addition to our more complicated business logic, we can check for things such as Surname being less than 50 characters, all using generics.

Below is a sample function which will work with any class marked up with Data Annotations and add them to a custom class we use for catching and reporting validation errors. The key points to note here are the ValidationContext and TryValidateObject().

            var modelValidator = new ValidationContext(toValidate);
            var modelFailures = new List<ValidationResult>();
            if (!Validator.TryValidateObject(toValidate, modelValidator, modelFailures, validateAllProperties: true))
            {
                foreach (var failure in modelFailures)
                {
                    foreach (var memberName in failure.MemberNames)
                    {
                        result.ValidationErrors.Add(memberName, failure.ErrorMessage);
                    }
                }
            }

We then convert the ValidationErrors class into a ModelState so that we can return a 400 error.

Optional One to One Relationships in EFCore

My company is currently working on replacing our venerable WinForms application with a shinier ASP.Net MVC version and rather than just re-write everything as a slavish “lift and shift” approach, we’ve gone for the bolder approach of completely re-designing the whole thing from top to bottom. Cue buzzwords such as “Greenfield” and “Blue-sky thinking” etc etc.

As part of this, we made an early decision to move to an Entity Framework Code First approach and try to keep as close to the cutting edge of technology as possible, so whilst we started with EF Core 3.1, we’re currently on EF Core 5. As someone who has been mucking about with SQL Databases since the 90s, it’s hard to let go of the SQL and just let EF manage and control everything, but once you embrace the mindset and just let it go, the benefits have been substantial.

However, one thing that has been niggling at me is managing 1:1 relationships and especially optional ones. It wasn’t long before we designed our first 1:1 relationships and instantly hit some problems.

Let’s look at the below diagram. Here we have Requirements which have an optional 1:1 (aka zero or one) relationship with a set of Features. When a Product is created from these requirements, it will also have a set of Features, nothing complicated here, all we’re doing is putting the set of common fields in a separate table, if we happen to have them.

So the first thing we did here was try to add a simple navigation property to Requirements, Products and Features and hoped EFCore could work this out by magic; something like:

public virtual FeatureEntity Features { get; set; }

Your migration will generate fine but as soon as you try to apply it and code against it, you’ll see this fun error:

The dependent side could not be determined for the one-to-one relationship between ….

Which makes perfect sense if you think about it, how on earth should we expect EF to work this out? Don’t get me wrong here, EF is about as close to computer witchcraft as I’ve seen in a long time, but you’ve got to give it a helping hand in places. Unfortunately this error leads you down the path of trying to use the fluent API to try and configure the relationship by hand, but I very much disagree here, the fluent API should only be used when you can’t express what you’re trying to achieve via Data Annotations.

What we actually want to do here is configure the foreign key properties in the dependant table. So in our FeatureEntity we add our navigation properties to Requirement and Product but we additionally add RequirementId and ProductId.

Because I’m following a naming convention of {TableName}Id, EF can work everything else out itself. No need for a ForreignKey data annotation.

[Table("Features")]
public class FeatureEntity
{
    [Key]
    public Guid Id { get; set; }

    public Guid? RequirementId { get; set; }
    public Guid? ProductId { get; set; }

    public virtual RequirementEntity Requirement { get; set; }
    public virtual ProductEntity Product { get; set; }

}

After this all you have to do is add your Navigation properties to Requirement and Product (nothing extra required here) and EF will magic together the rest for you.

The final point of note here is the optionality of the relationship. As you can see above, I’ve declared that both of these FKs are optional by using ? after the type. It’s as simple as that.

If you review your migration and ContextModelSnapshot, you’ll see that the FKs will all be unique and indexed and EF has worked out the details for you using the HasOne()/WithOne()/HasForeignKey() structure for you.

No custom OnModelCreating, No Fluent API. Simples.

Hello World

“Hello World!”, you’ll often see this chirpy piece of text in relation to computer programming and I think it’s highly appropriate that my first post on this blog is in keeping with that tradition, but where does this come from?

Before I started to write this post I thought I knew the answer with a laser-like certainty, but as is usually the case in these situations, when we’re at our most certain, we’re also at our most mistaken. Though the exact origins of Hello World remain lost in the mists of time, its use as a test phrase is widely believed to have begun with Brian Kernigham’s 1972 book, A Tutorial Introduction to the Language B. In which, the first known version of the program was used to illustrate external variables.

But I, as many of you was introduced to this expression back when I, albeit unknowingly, started on my journey to becoming a software developer. I can’t remember the exact year but I was definitely under the age of 10 when my mother suggested I might want a computer as my Christmas present. I wasn’t sure about this and I suspect I was persuaded into the idea. I’ll have to check with my mum here to get the facts, but this is how I remember it.

Roll on Christmas day, and I was confronted with my very first computer, a Sinclair ZX Spectrum as shown in all of it’s glory above. This was the very cutting edge of what would become Personal Computers and I won’t bore you with the history or details, beyond knowing that in order to even do anything with a Spectrum, you had to start typing code. Yup that’s right, if you ever owned a Spectrum, an Amstrad, Commodore or BBC Micro, you were basically being groomed into a career as a software developer without even knowing it.

Touch any key and you’re instantly transported to a command line

Within minutes you’re writing your first program

I imagine that confronted with such a thing, you would be turning straight to the manual, and this is where the magic starts. After some extensive blurb about connecting up the myriad of wires, the first thing your manual tells you to do is get your new friend to say “Hello”.

Again, I’m surprised here, I would have bet a large portion of my fortune that the first thing I typed was “Hello World”, but having dutifully tracked down the very same manual on the internet, it turns out the first program I ever wrote was not “Hello World”, but in actual fact it was “Hello”.

The rest as they say is history: you quickly learned how to Load/Run games from cassette and that’s where most left it. Yet the more inquisitive in our midst would have absorbed that manual and been writing their own programs in short order and that is essentially the story of how I wrote my first program and set my feet upon the path of becoming a software developer.

These days, “Hello World” (See? I do get back to the point eventually) is used as the standard first piece of code written in computer languages, in fact there was a strong trend for a language to be judged on how simple it was to send this innocent and chirpy piece of text to the screen. Funnily enough, there are very few languages that can match the BASIC of the 80s for simplicity of this code.

I’ll leave you with what was probably my second program: