Dealing with data warehouses, data chains, and even data lakes can be awkward in an agile environment. While adding a single metric to a dashboard can be very natural, nobody builds a dimension table a few columns at a time. This embarrassment caused many variables in how agile methodology is applied to individual analytics databases.
Sometimes people keep their ‘agility’ shell, while defining delivery as a component along the way rather than offering something of tangible value to the end userAnd Such as appropriate design, or perhaps requirements document. Problems can arise if an organization chooses to see a single report or dashboard delivered, or even a metric as the “thing of value” for an agile team sprint – when that team’s efforts include data that is not yet in the data market or data warehouse. Or, perhaps the organization is sophisticated enough to split the database side of the development effort and create a separate Kanban stream that is used as a resource for multiple agile teams serving code of one type or another. Business users may request specific data from Kanban; This assumes that business users are making enough ad hoc query to say that having the required data available in the database is a given for them. Conveniently, agile teams are waiting for the same new data to appear. If Kanban can keep up with the data needs of smart teamsAnd Then things can progress smoothly.
Another option, though rarely seen, is to redefine the customer’s story/feature. The data entering the analytics database is not exactly the same size as the data leaving the dashboard’s analytics database. If the users of the business are defined as the actual dashboard developers rather than the end users of the dashboard business, then the value received for them will obviously be the availability of a new data source. Then, dashboard developers with this new data will be empowered to start working on their tasks, stories, and/or features that involve using that new data. The source may have dozens of data items added to the database while a particular report may only need a few data points. This way, the data entering the database is a different amount of work than the data leaving the dashboard. With this kind of approach, the mental block seems to be that dashboard developers are not seen as members of the business and therefore not resources they need to gain value. It’s a strange idea to think of, because everyone works for the company, right?
The reason why no one builds a dimension table one column at a time is that adding extra data items when someone checks out a new source is often trivial. While going back and refactoring already existing code to add in an extra column or two is also a small effort, it’s a lot more work than adding it all in the beginning. It is normal for data items to be pulled outAnd It is likely that there will be a needAnd Once in that initial iteration. Pulling data versus using data in analysis are very separate tasks with separate focal points and even very separate toolkits and skills. Often the idea of some agile teams trying to tackle fetching a new data source and then using that source in a new or improved dashboard in one sprint is a messy mistake that only creates opportunities for the team to fail.
As an industry, we need to embrace a way to stay agile even when we do things very differently than creating a web interface or an app. And perhaps we also need to embrace the idea that business and IT is not a “us against them” dynamic. We are all part of the business.