Friday, May 17, 2024

Why my application is slow? - Database Connections

There are four pillars of great software, It needs to be robust, fast, modifiable, and secure. When you are in application development, speed is absolutely important. Users demand responsiveness, and any sluggishness can lead to frustration. One of the primary culprits behind slow applications is the improper management of database connections. Understanding how these connections operate and implementing strategies to optimize them can significantly enhance application performance.

Every database server uses a connection pool, a collection of connections that applications use for read and write operations. However, the number of connections within this pool is limited often governed by the CPU and memory capacity of the server. Each connection can only handle one operation at a time. So, the efficiency of managing these connections directly impacts the overall performance of the application.

Write Efficient Queries: One basic approach to mitigate connection limitations is to write efficient queries with appropriate indexing. Fast queries enable swift release of connections, allowing them to return to the pool for reuse by other operations. By optimizing queries, developers can ensure that database connections are utilized efficiently, thereby maximizing throughput.

Read Replicas: Read replicas for queries offer another way to expand connection availability. By distributing read operations across replicas, additional connections become accessible. But we must also understand, that while read nodes reduce the load on the primary database, they also increase synchronization delays between the master and replicas, potentially impacting data freshness.

Implement Multi-Master Databases: We can implement a multi-master database architecture that also addresses connection constraints. By enabling multiple nodes to accept write operations, this configuration distributes the workload and enhances scalability. Yet, the adoption of a multi-master setup introduces complexities related to data conflict resolution and consistency maintenance. Without proper implementation, the risk of data inconsistencies can impact the integrity of the system.

Adopt Microservice Architecture: You might consider adopting a microservice architecture, where each microservice has its own dedicated database. This decentralization gives us access to multiple databases, so more available connections. However, this introduces additional complexity, particularly regarding synchronization and transactions between microservices that require careful orchestration to maintain consistency across the system.

We need to optimize database connections to achieve optimal application performance. By following the best practices, we can effectively mitigate connection limitations and enhance throughput. However, it is important to have a balance between scalability and complexity, ensuring that the chosen strategies align with the specific requirements and constraints of the application. Ultimately, by prioritizing effective connection management, developers can unlock the full potential of their applications, delivering a seamless and responsive user experience.

#performance #scalability

Friday, June 7, 2013

An unwilling customer

When a software project is outsourced or offshored to a faraway land, its obvious  that it has gone through a very tough decision making process for the stakeholders of the company. When the decision is done you will find not everyone is equally happy. Especially the technical team is  always bit reluctant. They have their reasons, starting from discomfort of working with people from different culture, to fear of losing importance in the company.

But in most cases this technical team is being asked to assess the outsource service provider and later work with them. This is a challenge other than the technical challenge itself, for the offsite team to gain trust and respect from the onsite team and successfully work together and deliver the  project.

This challenge can be addressed though few core steps.

By building personal relationship
This one is the most important. Without a strong personal relationship with the onsite team most offsite effort
Selise team challenges the onsite team
would ultimately fail. Even at the time of Skype and Hangout, I believe this personal relationship cannot be built with a strong foundation without visiting each other.  Each project needs planning and budget which allows each team to visit the other with a regular interval. Its always  a good idea to start a project onsite with a team combining onsite and offsite members and after one or two sprints break away.

By adopting the standards and tools of the OnSite team
If the onsite team has a set of standards regarding development which they follow, off-site team should adopt to those standards. Standards could vary from coding standards to development process standards, configuration management standards to quality standards. It good to use the tools used by the onsite team if they are habituated with it for long.  It creates additional comfort for the onsite team having known tools in a new operating mode.

By introducing transparency
Its almost impossible to visualize what the other team is doing at any given moment if you do not have a transparent way to produce and deliver this information. Let the onsite team have access to off-site teams calendar, scrum board, burndown chart and issue tracker.  Let the product owner of onsite team join your daily scrum, if not possible at least have a weekly meeting. Do the sprint demo in front of the whole onsite team over webex.

By addressing the pointing finger attitude
Pointing finger is a disease in every corporate house. When you  see Onsite team is trying to burn you,  the smartest way of handling the situation is to take responsibility proactively. Justifying the failure even with most strongest excuse mostly fires back. In case of a failure.   always take responsibility, analyse the cause of the failure and propose concrete steps which can be taken to improve future delivery.

A reluctant onsite team can cause major project damage or even put the whole operation in jeopardy if program management from both side fails to find and act accordingly. Distributed development is not easy,it has its own challenges and like any other great challenges it has be dealt with open mind and brave heart.

Tuesday, May 28, 2013

CQRS Experience 01

Did you hear about CQRS? Are you using it?

We, at Selise, are building a large enterprise application with CQRS. I would like to share some of my experience with it.

I would take a moment to explain it first in really basic words, you can checkout and dig more later. The core principle is  CQS - Command Query Separation which tells you to separate your data read and data write operations. The method returns data should not modify the data, on the other hand the method that modifies the data (like inserting, updating or deleting) should not return any data. Which means your AddNew Method cannot return the newly added row ID back to you. CQRS ( Command Query Responsibility Segregation or Separation)  is an application CQS concept. We can call it a pattern. Its not an architecture or framework. 

To go a bit deeper with CQRS lets agree to some arguments.

 Argument 1: In any large user base application you will have more read than writes. 

 Argument 2: Reading data to display in the UI from a normalized database is slower than a denormalized database. Joining and retrieving data is expensive, its much faster if you could retrieve all your UI data for a single page as a collection row from a single table of the database without joins. The structure of this table should be more like the View-Model of your application.  

 Argument 3: When user modifies a data, it is not always necessary to synchronously update the database. We can live with some delay and let the database change happen asynchronously. Update of the UI also could be asynchronous. For example while adding a new row to a grid, the UI to add item could make an async request to server to add the row and UI can get back to grid. The grid seems to be not showing the row which just now has been added but the UI is responsive, you can make another add row or click on a different link meanwhile.  If you are still at the grid, after a few sec the row appears. 

Asynchronous database operation

On the next episode we will see how this arguments will drive us to CQRS.

[Read more:Wikipedia - Command–query separation, Martin Fowler on CQRS, Udi Dahan - Clarified CQRS, Greg Young's Blog on CQRS]

Saturday, May 25, 2013

Rockstar Triangle

My entire career I have worked with very talented developers. I have seen their journey, how they evolved from a cowboy coder to a better programmer. But my observation is, being a great software developer is more than how good programmer you are. I call these great software developers “Rockstars”. I believe a rockstar developer can be defined with 3 core attributes.
  • Analytical Skill
  • Technical Know-how
  • Experience of best practices.

I put them on three side of a triangle. Let me I call it Rockstar triangle.

The area inside the rockstar triangle tells you how great software developer you are. The more area you have in there, it's good for you.

Now how do I define these three? Let’s start with Analytical Skill.  It’s part of our cognitive ability and hardwired to the brain. Wikipedia has the following definition “Analytical skill is the ability to visualize, articulate, and solve both complex and uncomplicated problems and concepts and make decisions that are sensical based on available information. Such skills include demonstration of the ability to apply logical thinking to gathering and analyzing information, designing and testing solutions to problems, and formulating plans.”

Why a great software developer needs great analytical skill? Well, that’s what we do, “apply logical thinking to gathering and analyzing information, designing and testing solutions to problems, and formulating plans.”” That’s the core, if you are not strong at analytical, go ahead, there is lot of other nice professions out there, be happy with them.

How would you know if you have great analytical skill? If you have it you probably already know. If you want to check as a programmer if you have it enough, one way to test your skill is to try some simple ACM problems.

Technical Know-how is the second biggest deal. You need a very strong, in depth, transparent vision of the technology you are working with. Don’t get me wrong, I am not talking about “Working Knowledge” that we play with our day to day business.  It's more of a complete understanding of the topic including how it was built, how it works and how it can be used. I think you can relate to the fact that in most cases we just deal with the last part - how it can be used. I always ask myself when I come up with a new technology, framework or component, could I build it myself. Until your answer is yes, well, you need to dig deeper.

Finally the best practices. You must learn and make them a part of your habit. By best practices I mean the design patterns, architectural patterns, SOLID principles, coding standards, code smells and refactoring, agile techniques, unit testing, dependency injections, AOP, NuGets, xml documentation of code and so on.  All this have to be inherent to you, anything you build should have these in them without much conscious effort. They are called practices; you have to practice them to get a grip on them.

Many developers works all around world who are building software. Most of them are just doing this in a way that they can. A rockstar would build software in a way that it's should have been built. The question is how far you will walk to achieve the perfection?  

Monday, May 20, 2013

Offshoring 01: Software Offshoring to Bangladesh

Bangladesh is at this moment could be the best place to setup an offshoring operation for any software company. In this blog series I will try to explain why and how you could setup an offshoring operation and look at the major challenges you will have to overcome.

Why Bangladesh? I summarize them into 2 reasons, Cost and Talent pool. Let’s have a closer look.


Bangladesh is still cheap. Office space rent in Dhaka is 50% of Bangalore, India, 33% of Kuala Lumpur, Malaysia. Later we will take detail look at each of the infrastructural costs but in summery Bangladesh is cheap in all respect comparing to India or any other surrounding emerging economy. Developers salaries are still quite low as the living cost is low. Service cost is minimal, administration salaries are almost invisible. You could run an 50 employee team in Dhaka with full infrastructure with the cost of a 10 employee office in US.

Talent Pool

Scrum Team @ Selise
Bangladesh has produced a huge pool of Computer Science graduates over last 20 year and still continuing. It has over 30 public and more than 100 private universities where 45 of them are located in Dhaka. Many of the them are specialized Science and Technology university and almost all of them have dedicated Computer science departments. As the industry is growing many of the talented minds are driven towards CS and that gives you access to a huge pool of experienced developers, QC, UxD and project managers.

Bangladesh is a gold mine now, its a window of 5 to 10 years. Prices will go up, all the big players like Google and Microsoft will be here, talent pool will get drained in this time. If you can move in now you will rule the place.

Many are already in. Samsung, Vizrt, Metatude , Orbitax, Cefalo, Selise,  IMH health,  ImpleVista  are just a few of them. Some of them are here for over a decade.

On the next episode we will talk about finding the right partner to start your venture.