Saturday, December 17, 2022

Data Validation

I deal with a lot of bugs and production issues. A lot of them can ultimately be traced back to poor data validation. It's hard to get right, and it's an area that developers often overlook. Here are some things you need to think about when validating different data types. You can use this as a checklist.

All Data Types

  • Is it required or optional?
  • Is it nullable and is that the same as a not-supplied or optional field?

Integers

  • What are the min/max values it can have?
  • Can it ever be negative or zero?
  • Is the data type size (16, 32, 64 bits) appropriate?
  • Can it overflow during calculations?
  • Can it underflow during calculations?
  • How do you handle remainders during division or modulus?
  • How do you handle rounding when combining with floating point numbers?

Floating Point

  • How do you handle accuracy errors inherit to IEEE 754?
  • (all the same issues as integers)

Date/Time

  • What are the min/max values it can have?
  • Is the value an absolute point in time or relative?
  • Do leap years or leap seconds matter?
  • Does daylight savings time matter?
  • What time zone is the input (source)?
  • What time zone is the output (destination)?

Strings

  • What are the min/max lengths it can have?
  • How are you counting the lengths; glyphs, code points, code units, or bytes?
  • What is the character encoding of the input (source)?
  • What is the character encoding of the output (destination)?
  • Can it handle a BOM?
  • What is the EOL character (CRLF vs LF)?
  • What are the allowed or forbidden characters?
  • What normalization should you do?
    • Trim leading/trailing spaces
    • Remove multiple spaces in the middle
    • Change to upper/lower case
  • Is an empty string, or a string with only whitespace, distinct from null?

Lists

  • Should all values be unique? (Consider using a Set)
  • Is an empty list distinct from null?
  • Should the list be sorted? When is the sort performed?
  • Do you append to the list repeatedly? (Consider a LinkedList)
  • Do you need to read from arbitrary locations? (Consider using a Dictionary/Map)
  • What is the maximum number of elements in the list?
  • Do you have the ability to detect and handle a situation where repeated appends may cause an out-of-memory error? For example, when reading from a very large file.
  • Can any of the elements be null?

Objects

  • Are there any cross-field/conditional validations?

Sunday, July 25, 2021

Naming Classes and Variables

Naming classes and variables can be difficult. Sometimes you are working on an application and there is a piece of data that is very domain/problem specific and there just isn't really a word that accurately describes what it is. There are things you can do to make your code easier to understand.

1. Avoid nonsensical names.

This one is fairly obvious. Don't call your variable "var1", "abcdef", and other silly names. Perhaps you can manage it in a small program, but as you start developing larger applications, it will become more difficult to keep track of everything.

2. Avoid overly generic names.

Some might argue that this falls into the same category as my first point. However, a lot of programmers think these names are okay. They aren't completely nonsensical, but they also don't tell you anything.

The "Manager" suffix is often a red flag. It usually includes lots of disparate methods. If you have a Manager class, you may need to refactor the code rather than just rename it. If you are lucky, it may just be that it's a "Repository" or "Validator" class that was poorly named. However, my experience is that a Manager class usually isn't just improperly named, but improperly designed.

The verb "check" can be vague if one isn't specifying what is being checked. Prefer the verb "validate" if you are doing data validation.

The word "data" can almost always be replaced with something better.

Bad Names        Better Names
checkPerson        validatePerson
checkFile        isFileReadable, isFileWritable
getFileData        getFileContents, getFileAttributes

3. Avoid abbreviations unless they are well known.

Developers want to avoid excess typing, but code is read more often than it is written and abbreviations make it harder to read. Modern IDEs have code completion, so it's rarely necessary to type out the full name to reference it again. The problem with abbreviations are that they can be used inconsistently. For example, someone might shorten "keyboard" to kb, kbd, kboard, or other names . Depending on the domain, the abbreviations could be easily confused for another term. If you are writing software to track patient appointments with doctors, does "doc" refer to the doctor or a document? Does "desc" mean descending order or a description? Additionally, not all developers speak English as their first language. They may not understand what the abbreviation means and it's not always obvious to non-native speakers how words are shortened.

Bad Names        Better Names
attr        attribute
cnt        count
nrml        normal

Acceptable Abbreviations
  • URL, HTTP, FTP

 4. Avoid "of" 

This is more of a personal preference, but I find that it is better to avoid using "of" in variable names. The names without "of" tend to be shorter. I also find that those who use "of" in names do so inconsistently so it's hard to remember what variables or methods are using "of" and which aren't. It's easier just to avoid it outright.

Bad Names        Better Names
numberOfRecords        recordCount
dateOfBirth        birthDate
typeOfProduct        productType

5. Be consistent

Names can be inconsistent for several reasons. Sometimes it's for backwards compatibility or the application changed many hands. However, I once inherited a project that only ever had one developer before me, but it still had very little consistency. For example, this application had to store addresses and mark whether it was residential or business. Across the database column name, class field, and JSON serialization, it had different names. It was either addressType, addressCode, or addressCategory. Three names for the same field! It gets worse though because the field could either be represented by a numeric ID (1 for Residential, 2 for Business) or the actual labels. It was never consistent in regard to whether those variable names were storing the ID or storing the name.

This inconsistency can also be confusing if there really are two separate fields. Maybe addressType specifies Residential or Business and addressCode specifies shipping/delivery options to that location. A lot of time is wasted trying to determine whether differently named fields are actually the same or not. If I see addressCategory, I waste more time trying to figure out whether that represents an addressType or an addressCode.



Sunday, August 11, 2019

Stop Calling Your Tasks "User Stories"


There are a lot of things I hate about scrum, or at the very least, I hate how teams are implementing it, if you want to get really pedantic and tell me "that's not true scrum" in response.

The goal of a user story isn't to track your work, it's to elicit user requirements.

Stop calling your technical work user stories. They aren't. A big problem with scrum is that user stories often can't be completed in a two week sprint. This will either lead to wild point swings when measuring velocity or having stories split to fit them in. It's perfectly fine to split a user story, but much too often it's being done on technical grounds because you didn't finish some technical tasks necessary to implement it. Let me give an example of how a real user story can be split.

  • Story: I want to add a comment system to my blog

That might be a pretty big task. It's also pretty vague and high level. What we should do is sit down with the user and ask them to be more specific. This story might instead become an epic and get broken down into the following stories.

  • Story 1: Create a registration form that asks for username, password, and email address.
  • Story 2: Create a login form that asks for the username and password for someone to login after registering.
  • Story 3: Display a comment box at the bottom of each blog post.
  • Story 4: Display the posted comments below each post.
  • Story 5: Allow users to sort comments by oldest / newest.
  • Story 6: Allow some registered users to be marked as moderators by the blog owner.
  • Story 7: Allow moderators to see a "delete" button next to comments to remove offensive comments.

Notice anything about the above stories? They are all things the user can understand. Now, of course there are going to be applications within specific domains where a random person on the street isn't going to understand everything. I wouldn't expect a non-animator to know what a "keyframe" is, but if I'm writing software for animators, it's fine to use that term in the story.

They are called "user stories" because they are from the perspective of a user. The goal is to figure out what they want. Maybe they aren't sure yet and things will change with time (hint: That's what being agile is about), but they are all things the user can understand. The goal is to determine their requirements. You break down stories for two reasons; to get more information, or to define points of deliverability. If the user really wants comments on their blog ASAP and is okay with not having moderators for a little while, then you can push out a release between those stories.

Here is what typically happens instead. Someone starts on the user registration story. They don't get finished or don't think they can finish in two weeks and they split it like this.

  • Story 1: Add Angular FormBuilder to registration page.
  • Story 2: Send POST request to backend to make sure username is not already taken.
  • Story 3: Salt password and hash with SHA-256.

What the heck is an Angular? What is a POST request? I have high blood pressure, I don't need more salt. What's a SHA and why do I need 256 of them?

The problem with doing it this way is that now users have no idea what's going on in the project, the developers lose sight of the requirements, and it's hard to know when something is in a deliverable state.

Please, do track your tasks. It's important to know what they are. Add them as dependencies to your user stories, but don't call the tasks themselves stories because they aren't.

When you actually have proper user stories, you will find that sometimes even a very specific story still can't get done in two weeks. Stop trying to fit them into a two week window. Track tasks instead to measure your velocity. The tasks can be much smaller than the stories so you will get better estimates and you won't be constantly shuffling tickets around and obscuring your progress because your scrum master or overpaid consultant thinks being agile is about following arbitrary rules.

Monday, June 6, 2016

OpenBSD's Dirty Little Secret

OpenBSD is often advertised by its developers as being a highly secure operating system. One of their goals is to be "the #1 most secure operating system".

The home page proudly states:
 Only two remote holes in the default install, in a heck of a long time!
That sounds fantastic! But is OpenBSD really all that secure in practice? The statement on the front page needs to be read carefully.
Only two remote holes in a default install.
When you consider that nearly every network service is turned off by default, it's not really that impressive. The OpenBSD team also does not consider denial of service attacks to be a security exploit; only a reliability issue. The event that lead to the second remote hole in a default install was denied as a problem until proof of concept code showing arbitrary code execution was provided.

Most security vulnerabilities are going to be from services not part of a default install. When vulnerabilities are found, it's important to have a patch available. How does OpenBSD handle this? First, you have to consider that there are two ways software is typically added to a default OpenBSD system. Ports and packages. According to the FAQ:
In general, you are highly advised to use packages over building an application from ports.
Okay, so the OpenBSD team themselves are telling people to use packages. But are they kept secure? Nope.
When serious bugs or security flaws are discovered in third party software, they are fixed in the -stable branch of the ports tree. Note that binary packages for -release and -stable are not updated.
There is a solution to this problem if you are willing to trust a non-official source. M:Tier provides binary updates for the latest OpenBSD release.

Security is about understanding risk. If you think "I'm using OpenBSD, therefore I'm secure", then you don't understand the risks. OpenBSD has been an innovator in many areas. I give them credit for this. But as a user, you need to carefully understand what OpenBSD means when they talk about security and what it actually provides.

Wednesday, March 23, 2016

Micro-Classes are an Anti-Pattern


A micro-class, micro-method, or micro-function is a unit of code that accomplishes such a small amount of work that it can be said to have no useful functionality on its own. Instead, the work it's meant to perform is repeatedly delegated to other micro-units and obscures what task is really being performed.

It is important to note that abusing micro-classes is NOT the same as merely breaking down a problem into smaller parts (a good programming practice). Instead, the use of micro-classes takes that idea to such an extreme that even the simplest of tasks are divided and spread out so thinly that it becomes difficult to follow the logic of the program. Think of it like a book being broken down into chapters and sub-chapters versus a choose your own adventure book where you constantly have to flip pages. Reading code that makes heavy use of micro-classes is like having to flip to a different page for every sentence.

Because I primarily work as a Java developer, I will be using the term micro-class. However, my arguments apply equally to micro-functions for those not using OOP.

Micro-classes are a bad idea. We all know that God objects are an anti-pattern. I think the industry has been very successful in getting that point across. I even meet a lot of beginner developers who are familiar with the harm caused by really long methods that try to do everything. However, I think that now things are going too far in the other direction. There are now many software developers who go to such extremes to break things up into smaller units that the code base as a greater whole becomes incomprehensible.

Why Micro-Classes are Harmful


1. Micro-classes harm architecture.

When creating an architecture for your application, you think about what packages and layers you need. If you divide your program into these microscopic units, you will eventually find yourself in a situation where a micro-class doesn't have a logical place to live. You either stick them into some "util" package or, if you are really creative, you spin it off into an entirely separate project and you have your application code masquerading as a "library", despite not containing anything useful outside of the one application that will ever import it. Hard coded assumptions about the calling application are often embedded in this faux library. Additionally, the project is now composed of so many of these little utilities classes, that it becomes difficult to find the places where the real work happens.

2. Micro-classes harm API familiarity.

Code is easier to read when it uses APIs that programmers are already familiar with. In applications that abuse micro-classes, the common APIs are hidden at the lower layers and often unneccessarily wrapped. This creates a code base where the business logic is composed entirely from an unfamiliar API which only exists in that one application. This makes code harder to read because one must learn and understand this new and unfamiliar API in order to understand what the code is doing.

3. Micro-classes make change more difficult.

The typical argument in favor of micro-classes is that they enable change. I think there are flaws with this argument. The first flaw is that most systems are not well designed and only allow for particular kinds of changes. As soon as a requirement change occurs that was not anticipated, then the change is not any easier to make. Secondly, even in situations where micro-classes do enable the kind of change one would expect, the micro-class style of programming requires very advanced knowledge of the system in order to successfully make the change. In theory, one might be able to only change a couple lines of code. In reality, most programmers outside of the original developer won't understand the system well enough to do it. Instead, they struggle through it and end up with ugly hacks to accommodate the requirements change. I put forward the argument that while a more cohesive system requires a greater number of source lines to be modified, they are often contained in a fewer number of places and the change is actually easier to make.

4. Code which relies on micro-classes has a high level of indirection.

When you read code composed of micro-classes, it is difficult to understand and explain to others. In an application composed of micro-classes the steps needed to accomplish simple tasks are spread out across the system. I've worked on programs where it was not uncommon for me to end up with 20+ tabs open in my editor as I navigated through the web of files to see how a relatively simple piece of functionality worked.

Signs of a Micro-Class


How can you tell the difference between a micro-class and a proper class which neatly sub-divides a larger problem? Sometimes the distinction isn't always clear. I would even give some leniency and say that a few micro-classes in an application are acceptable. Their overuse is what causes major maintenance problems.

1. Micro-classes are often used only once. If you find a class which is never used by more than one caller, it may be a micro-class.

2. Micro-classes tend to reinvent the wheel. They will act as nothing more than wrapper classes which do little but provide a slightly different interface for doing the same functionality as another class.

3. Micro-classes tend to abuse design patterns. While design patterns aren't necessarily bad, one should have a rational for using them. Code that abuses micro-classes often does so in order to implement a design pattern for a relatively simple piece of functionality where the design pattern was overkill and not needed.

4. Micro-classes tend to be poorly named or invent new vocabulary. Because the code gets spread so thin, developers must think of new names for all the additional classes and functions. The names are often poor at describing the purpose of the class.

Conclusion


I'm not arguing in favor of god objects. I think there is a healthy middle ground between 2000 line methods and 500 methods of 4 lines each. At some point, developers need to learn when to stop sub-diving a problem and perform the task they want to accomplish inline. I also don't believe this is a violation of the Single Responsibility Principle (SRP). It's just taking the idea of what constitutes a "single responsibility" and viewing it a higher level.

Monday, June 20, 2011

Why I don't like Dynamic Typing

1.) It requires more testing.  Type information doesn’t go away.  Dynamic languages still have types.  They are just being hidden from you.  If you pass a string to a method expecting an integer, you’ll either get bad data or a crash.  Static typing can check for this discrepancy at compile time.  This means you don’t have to test all possible code paths manually.

2.) Dynamic typing isn’t as clear.  Static typing can be a form of documentation.  It tells additional information about a variable’s intent without having to load it up with ugly prefixes as is sometimes done with Hungarian notation.

3.) It makes using IDE tools more difficult.  Things like autocomplete and refactoring tools aren’t as common in IDEs for dynamic languages because they are much harder to do without type information that can be predetermined.

4.) It’s slow.  Many optimizations can be made when the type is known ahead of time.  Having the runtime check type information every time a variable is accessed is horribly slow.  Looking at the Computer Language Shootout shows dynamically typed languages like Python and Ruby perform 50x or even 100x worse on many of the benchmarks.

5.) It provides little benefit.  Dynamic typing doesn’t offer anything that can’t be done in a statically typed language with templates and type inference.

6.) Some dynamically typed languages lead to impossible to spot assignment errors.  I put this one last on the list since not all dynamically typed languages are like this.  However, some, like Python, make no distinction between variable declaration and assignment.  If you try to reassign an existing variable and make a typo in the variable’s name, Python will provide absolutely no warning.  It just creates a new variable.  This can lead to extremely frustrating errors that are very difficult to track down.