When Do You Trust a Wi-Fi Predictive Model? The Battle of Accuracy vs. Precision

Most non-engineers tend to use the terms accuracy and precision interchangeably, but they are actually very distinct concepts.    Precision is based on the computational power of the software and the underlying mathematics.  Most mathematical models (such as Wi-Fi predictive modeling software using ray traces to compute absorption and reflections of walls and objects, as well as free space path loss over distance) will generally provide a high level of precision.  Accuracy is based on the level of complexity (and underlying assumptions) of the mathematical model, as well as the quality of the input parameters.

One of the tricks of performing any type of engineering modeling or simulation in Wi-Fi (or, for that matter, in any engineering discipline) is that you first need to estimate; i.e. you need know what the answer should be BEFORE you actually start the model, at least to a rough order of magnitude.  The engineering model serves only to add both accuracy and precision to your estimation.  

 

Without having a good estimate, however, the results of the predictive model can be extremely precise, while simultaneously being wholly inaccurate! 

The true art and skill of wireless design, therefore, is in understanding the initial estimate. A Wi-Fi predictive model, like any mathematical model, is a very sophisticated idiot!  The model will precisely compute what you tell it to. However, if the inputs to the model are wrong, the model doesn’t know that, and so the model will compute a very precise, and very wrong, answer.  Another common way of saying this is "garbage in, garbage out".

It should also be noted that there are also underlying assumptions in the model itself in the way it works that will impact and limit both precision and accuracy.  For Wi-Fi predictive modeling packages (e.g. Ekahau, Tamograph), the following assumptions and simplifications are typically used:

• uniform walls with known dB loss values and known reflectivity percentages
• the models only account for absorption and reflectivity, and generally not diffraction, scattering, or other effects (and when using the “attenuation zones”, the model does not even calculate reflectivity, but only attenuation as a a dB/ft loss coefficient).  This is done to keep computation times reasonable
• we generally ignore the effects of furniture, wall decorations, appliances, mirrors, people, etc
• the antenna signal propagation patterns modeled are based on either measurements or design predictions of the antenna manufacturer, which will have its own underlying errors and assumptions
• we are generally only positioning / placing the access points with an accuracy of several feet at best, and these placements may change slightly during installation, based on how cabling is run

Thus, there are always intrinsic errors and simplifying assumptions in any model.  For most purposes, however, this limited level of accuracy is quite sufficient, so long as these simplifying assumptions are “reasonable”. We generally don’t need an exact model that predicts the absolutely correct signal level at every spot that we will get in the environment. What we need is a model that is “close enough”, so as to tell us how many APs are required, along with their locations and critical settings, such as channel and transmit power.  We can easily be off by +/- 5 dB in any particular location, and still it would be sufficient for most purposes.

So How Does One Create an Initial Estimate?

Unfortunately, the art of good estimation only comes with a lot of experience, usually a lot of negative experience where a project has gone horribly, horribly wrong and you need to fix it, often by trial and error.  

That said, there are several guidelines that can be used to point an engineer in the right direction to develop a reasonable initial estimate.   For Wi-Fi design, these are the guidelines I employ:

1. Understand Your Requirements:  How is the network going to be used?  Approximately how many, and what type, of client devices?  Where are they going to be located?  What areas do (or do not) require signal coverage?  How many years is this network going to be deployed, and how is usage likely to change over that time?   
   
   
   These requirements vary substantially by vertical market, and can vary even by individual project.  Most Wi-Fi engineers tend to specialize in one vertical market or a small set of closely related vertical markets, so trends on previous projects will typically be applicable to your current one.  If, like me, your specialty is "SMB" which spans multiple vertical markets, you need to develop a good understanding of how Wi-Fi gets used across many different environments.
   
   
2. Understand Your Constraints:  What are the building materials?  Can APs be deployed in rooms or can they only go in the hallways?  What kind of budget is allowed on this project?  Are there other Wi-Fi networks or non-Wi-Fi sources of interference in the environment?   Are aesthetics a concern (nobody likes seeing external antennas)? 
   
   
   Your requirements define what the network has to do.  Your constraints are what the network has to work around.  The distinction is subtle but important.  Requirements are always independent of each other and are generally inviolate. Constraints can be highly coupled and, in many cases, self-contradictory, but are also potential areas of compromise and push-back.
    
3. Understand Your Solutions:  Wi-Fi engineers generally have one, or at most a few, preferred Wi-Fi vendors that they use in most deployments, and a smaller subset of products from that vendor or vendors that they use.  Every product will have its own capabilities, limitations, and idiosyncrasies. Understanding how these products have behaved on previous projects (especially if challenges had to be overcome) will help in knowing how many will be needed on the next product.  
   
   
   Most AP vendors will provide and promote a set of Best Practices, guidelines, case studies, etc. on how to best deploy their products in various scenarios.  It is important that the Wi-Fi engineer be able to separate the marketing hype from the technical capabilities of a product, and know how to best tune the configuration settings (i.e. "nerd knobs") for that product for their environment.
   
   
4. Think Creatively; Don't be Constrained by Your Previous Solutions: This may seem contradictory with the prior point, but this is a key aspect of any successful Wi-Fi design.  My graduate thesis advisor, Professor Nam P. Suh (Mechanical Engineering Department Chair at MIT and later President of KAIST), was a controversial advocate of using a "clean sheet of paper" for every engineering design project, no matter the engineering discipline.   He promoted fully understanding your requirements and constraints up front, and then being unconstrained in terms of picking the right solution for the right problem.   Unfortunately, this approach is harder to achieve in practice than it sounds.
   
   
   AP vendors are specialized and generally target particular verticals - an AP or vendor that is very good for one environment is often very poorly suited for another.   This can work both ways, in terms of either lacking key features or having too many features and too much complexity, with a correspondingly high price tag. However, a Wi-Fi engineer (and the organization he or she works for) usually is heavily invested in particular vendors with education, experience, and infrastructure. Switching AP vendors can often be a logistical nightmare that most organizations won't employ unless forced to.  
   
   
   That said, there are often creative solutions that don't require a radical departure to a new vendor. The use of external directional antennas in some environments can provide some unique solutions that may be appropriate in some environments, especially in areas such as outdoor parks, parking lots, marinas, warehouses, etc.  Additionally, if you understand the "nerd knobs" (pursue your CWNA and beyond if interested), some settings can be tuned for particular environments to optimize performance.  
   
   
5. Establish Good "Rules of Thumb": It can be useful to start with simple (and even "overly simple") assumptions to establish a starting point, so long as it is understood that this is ONLY a starting point and the estimate will need to be refined from there based on actual requirements and constraints.  
   
   
   A very good rule of thumb in Wi-Fi is to start by using a fixed transmit power level for 2.4 GHz and 5 GHz (I personally like using 14 dBm for 2.4 GHz and 20 dBm for 5 GHz in most environments), so that all of the APs have roughly the same coverage area on both bands and are not too much stronger than the transmit power of typical client devices such as smartphones and tablets.  This allows the APs to be spaced roughly evenly in the environment, making the development of a static channel plan simpler.
   
   
   For example, a common question I get is "how much area in square feet will an access point cover"?  The answer to this is not simple, as it depends upon building materials, building geometry, whether 2.4 GHz or 5 GHz is being discussed, expected client density, etc.   As one colleague of mine puts it, "Wi-Fi is not a can of paint!"   That said, a square footage estimate for a design primarily driven by coverage (vs. high capacity) can be a good starting point.  For an indoor environment (e.g. offices, private homes) with standard drywall, one omni-directional AP per 1500 - 2000 sq ft is not unreasonable.  Similarly, for open outdoor environments, an area of approximately 10,000 - 15,000 sq ft (i.e. a radius of approximately 50' - 75')  would be reasonable.
   
6. The Estimate and the Model Must be Consistent: While one wants to make the initial estimate as good as possible, if we could estimate extremely accurately up front, there would be no need for predictive modeling.  You should expect that your estimate will be off by 20% - 25%.  The intent of the predictive model is to refine your estimate in terms of its accuracy and precision.  If the estimate and the predictive model are wildly divergent, then you have a problem somewhere and need to figure it out.
   
   For example, if you estimate that a project will require 10 APs, and the predictive model tells you that you need 8, or 12, then your estimate was pretty good.   If your model tells you that you only need 5 APs, or that you need 20 APs, then either there is a fundamental flaw in your estimate, or there is a fundamental flaw in your model or its inputs.   

An Example of Predictive Modeling Gone Awry

This is an actual case that came up a few weeks ago to provide Wi-Fi coverage in a 300’ x 300’ RV park with approximately 30-35 RV trailers, no trees, and excellent line of sight everywhere.  

In this case, our sales agent estimated and already sold the property six outdoor omni-directional access points.  The customer then asked us where to place the access points.  The customer was planning on installing 20' poles and running fiber from the MDF at the clubhouse to the poles, but needed to know where the poles should be placed.  My initial reaction was that this estimate was reasonable (and perhaps even slightly high) for such a small space.  I didn’t need the model to tell me this, but rather I know that from applying several years of experience deploying these types of networks.  

However, when I handed this to a junior engineer to model, the engineer came back with a predictive model solution requiring 12 APs, and even then, the coverage was marginal, especially on 5 GHz.

2.4 GHz predicted signal coverage (original model)

5 GHz predicted signal coverage (original model)

From prior experience deploying RV parks, I took one look at this and knew the answer was wrong. Quantifying WHY it was wrong, however, took some detective work.  In the end, this came down to misunderstandings of requirements and constraints, as well as how things should be modeled:

1. This model assumed 11 dBm power levels on both bands, which we typically only use for high density deployments when many APs need to be co-located in one space for handling user capacity.  With only 30 RVs, even with 3-4 devices per RV, high user density of devices is clearly not a requirement.   
   
   
2. This model assumes that the APs are going to be mounted between the RVs at a height of about 4', and not on poles approximately 12' - 15' in the air, several feet above the tops of the RVs.
   
   
3. The RVs themselves were modeled as solid blocks of metal using the area attenuation zones.  This mode of modeling assumes just an attenuation loss of dB / ft, and not any reflectivity of signal.  While the RVs themselves have an exterior shell of metal or fiberglass, these are thin and have large windows through which the RF can penetrate.  It is my general assertion, therefore, that the walls of the RVs can be ignored from a modeling perspective.  Thus, we are essentially covering an empty field.  Even if you debate the wisdom of this assertion, the next simplifying assumption is to model the RVs as thin, hollow shells of metal that reflect 90% of the signal, not solid metal blocks that do not reflect at all.

Correcting for these misunderstandings and ignoring the outer shell of the RVs leads to only requiring four access points.

5 GHz predicted signal coverage (open field, 20 dBm transmit power)

One can debate that the outer shells of the RVs, even if thin, will have an appreciable signal attenuation, and for appropriate talkback of client devices in the RVs to the APs, more APs would be required.  This can be accomplished by lowering the transmit power levels of the AP, say to 15 dBm, in which case, six APs are needed.

5 GHz predicted signal coverage (open field, 15 dBm tramsmit power)

Personally, I would have used two outdoor APs with sector antennas on the main building to cover the entire RV park, with one indoor AP to provide coverage inside the main building, since it would be in the shadow of the sector antennas.  Such a solution would’ve been both cheaper and easier to install; not only are there fewer APs, but there is no need for any poles or fiber runs.   

5 GHz predictive signal coverage using sector antennas

Given the constraint that the customer already purchased the outdoor APs, and we felt we had already confused the customer enough, we presented the six AP solution with the appropriate configuration settings.

Aside from the 12 AP solution which had flawed requirements and assumptions, none of the other design options presented here are “wrong”.  There are several approaches to doing a design, and therefore several different solutions that will work, i.e. meet the true requirements and expectations of the customer, including adequate coverage, adequate capacity, minimal co-channel interference, etc. Different design solutions are thus “better” or “worse” in comparison to each other, usually based on parameters like cost, ease of installation, ease of maintainability, etc.  

There is the old adage “If all you have is a hammer, everything looks like a nail.”  The design approach we have and the solution we select will generally be constrained by the potential solutions you have to work with (i.e. the types and models of APs, antennas, etc.).   Nonetheless, understanding your requirements and constraints up front, along with the use of rules of thumb and experience, is essential to creating an initial estimate of a design solution, which then can be plugged into a predictive model for refinement.